SlideShare uma empresa Scribd logo
1 de 49
Baixar para ler offline
Lessons learned while
 building omroep.nl
    Bart Zonneveld (@bartzon)
    Sjoerd Tieleman (@tieleman)
Nederlandse Publieke Omroep
Dutch Public broadcasting Company


  AVRO    Joodse Omroep   NMO   Teleac

   BNN        KRO         NOS   TROS

   BOS       LLiNK        NPS   VARA

   EO         MAX         OHM   VPRO

  HUMAN      NCRV         RKK   ZvK

   IKON       NIO         RVU
Rails sites
• Beetlejuice       • Radio 1
• Centrale          • Z@PP
  navigatie
• Omroep.nl         • Z@ppelin
• Nederland 1       • Zelda
• Nederland 3       • Various tools
Lessons learned while building Omroep.nl
Team
• 2 coders
• 1 designer
• 1 editor
• 1 project manager
  6 months, CMS built from scratch
Requirements

• Handle 30.000-40.000 pageviews per day
• Handle traffic spikes
• Flexible, multi user CMS
• Loads of external data
Daily spread
Some numbers
+----------------------+-------+-------+---------+---------+-----+-------+
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Controllers          | 1864 | 1535 |        41 |     185 |   4 |     6 |
| Helpers              |   797 |   631 |       1 |      75 | 75 |      6 |
| Models               | 1303 | 1055 |        40 |     153 |   3 |     4 |
| Libraries            |   814 |   620 |      15 |      79 |   5 |     5 |
| Integration tests    |     0 |     0 |       0 |       0 |   0 |     0 |
| Functional tests     |     0 |     0 |       0 |       0 |   0 |     0 |
| Unit tests           |     0 |     0 |       0 |       0 |   0 |     0 |
| Model specs          | 1932 | 1573 |         0 |       9 |   0 |   172 |
| View specs           | 7322 | 5950 |         0 |     153 |   0 |    36 |
| Controller specs     | 7292 | 5846 |         0 |     175 |   0 |    31 |
| Helper specs         |   900 |   676 |       0 |       2 |   0 |   336 |
| Library specs        |    56 |    45 |       2 |      12 |   6 |     1 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total                | 22280 | 17931 |      99 |     843 |   8 |    19 |
+----------------------+-------+-------+---------+---------+-----+-------+
       Code LOC: 3841    Test LOC: 14090     Code to Test Ratio: 1:3.7
Moar numbers


• 410 Cucumber scenarios, 600 step definitions
• 2235 RSpec specifications


              so it must be bug-free, right? ;-)
Tools
Ruby on Rails 2.3.4      Rspec + Webrat + Cucumber

   Apache 2.2                     Paperclip

       SVN              App monitoring: RPM, Hoptoad

    MySQL 5               Service monitoring: Nagios

    Memcache
Tools: app monitoring




 Hoptoad     New Relic RPM
Architecture

• Apache 2.2 with mod_proxy
• Rails 2.3.4 running on Phusion Passenger
  2.2.5 with REE
• 4 hosts, each running 4 instances (per app)
  Appdex: 1.0, avg. response time 40ms, 130 rpm, db load 0.6 %
Servers


• Quadcore Intel Xeon E542, 32 GB Ram
• Fedora 8
• Other mumbojumbo
Architecture
               Front proxy    Front proxy




Application     Application    Application   Application
  server          server         server        server




                Database      memcache
Workflow

• BDD
• Shared behaviours
• Performance testing
• Staging and production environment
BDD

• RSpec
• Cucumber
• (Webrat)
3 slide intro to BDD:
           RSpec
describe Article do
  it_should_behave_like   "all   objects   with   userstamps"
  it_should_behave_like   "all   objects   than   can be published"
  it_should_behave_like   "all   objects   that   have an url"
  it_should_behave_like   "all   objects   that   can be searched"
  it_should_behave_like   "all   objects   with   related articles"

  it "should not be valid without a name" do
    @article.attributes = @valid_attributes.except(:name)
    @article.should_not be_valid
  end

  it "should not be valid without contents" do
    @article.attributes = @valid_attributes.except(:contents)
    @article.should_not be_valid
  end
end
3 slide intro to BDD:
     Cucumber features
Feature: Articles on the homepage
  As a visitor
  I want to view articles on the homepage
  So that I can see the latest content

  Scenario: 5 most recent articles
    Given there are 8 articles
    When I visit the homepage
    Then I should see the 5 last published articles
3 slide intro to BDD:
     Cucumber steps
Given "there are $num articles" do |num|
  num.to_i.times { create_article }
end

When "I visit the homepage" do
  visit root_path
end

Then "I should see the $num last published articles" do |num|
  Article.last_published(num).each do |article|
    response.should contain(article.title)
  end
end
Shared behaviours

• Tags
• User stamps (created by, updated by)
• Searching
• “Related” articles
• Publication timestamps (on/offline at)
Shared behaviours
module UserStamps
  def self.included(klass)
                                    class Article < ActiveRecord::Base
    klass.instance_eval do
                                      include Shared::UserStamps
      include InstanceMethods
    end                               include Shared::Published
  end                                 include Shared::Url
                                      include Shared::Search
  module InstanceMethods              include Shared::RelatedArticles
    def created_by
      User.find_by_id(creator_id)     # stuff
    end                             end

    def updated_by
      User.find_by_id(updater_id)
    end
  end
end
Workflow: performance testing


• ab, httperf, autobench, cURL
• NewRelic RPM
• Safari Web Inspector
• http://railslab.newrelic.com/scaling-rails
Autobench
Challenges


• Content Management System
• Loads and loads and loads of external data
CMS


• Articles, Themas, Specials, Subsites
• Multiple feeds, images, links
• Version control
• Media database
CMS: Articles
          Subsite



Thema      Page     Special



          Article



   Link    Feed     Image
CMS:Version control
Media DB

• Implemented as REST app
• To be used as REST service
• Files, folders, crops
External data
•   RSS feeds

•   EPG data

•   Zelda

•   Babel

•   News/sport/teletekst

•   Twitter

•   Lots of custom XML formats
External data: XML/RSS
• Empty feeds
• Encodings are off (Windows-1252,
  ISO-8859-1, UTF-8)
• “Custom” fields
• Incorrect fields (dates, unescaped HTML)
• Timeouts
• Everything that can go wrong, will go wrong
External data: Twitter
External data: EPG data


   Zelda




                   don’t sue us Nintendo... please? :)
External data

• Empty feeds
• Retrieving the feed while someone is
  updating it
• Required fields that are empty
• DTD?
<!ELEMENT aflevering (
            prid?, tite?, medium?,
            icon?, aankondiging?, inkl?,
            ingl?, infi?, inak?, inds?, inbb?,
            kykw?, orti?, aant?, land?, lcod?,
            psrt?, prem?, inh1?, afle?, atit?,
            inh2?, bron?, prij?, inh3?, mail?,
            webs?, inhk?, gids_tekst?,
            omroepen?, genres?, personen?,
            streams?, fragmenten?, serie?
)>
<!ELEMENT   inkl   (#PCDATA)>
<!ELEMENT   ingl   (#PCDATA)>
<!ELEMENT   intt   (#PCDATA)>
<!ELEMENT   inhh   (#PCDATA)>
<!ELEMENT   omro   (#PCDATA)>
<!ELEMENT   lcod   (#PCDATA)>
<!ELEMENT   herh   (#PCDATA)>
<!ELEMENT   inds   (#PCDATA)>
<!ELEMENT   infi   (#PCDATA)>
<!ELEMENT   inbb   (#PCDATA)>
<!ELEMENT   genr   (#PCDATA)>
<!ELEMENT   kykw   (#PCDATA)>
<!ELEMENT   afle   (#PCDATA)>
Lessons learned

• Cache the crap out of everything
• Rescue everything
• Test everything (frontend and backend)
Caching

• Cache the homepage
• Page cache → Fragment cache
• Don’t cache forms
• Cache as much as possible
Case: article views


• Article is page cached
• Update the number of views in realtime?
Use AJAX!

<% javascript_tag do %>
    <%= remote_function :url =>
update_views_article_path(@article) %>
<% end %>
Case: banner items
Case: banner items
• Fast requests (<10ms)
• ETags (304 Not Modified)
• Static resource → page cache
• Move to front proxy, frees up Rails cluster
• 1100rpm → 130rpm
• 20ms → 40ms
•   Average response time going up? Oh nooooes!
Caching external data

• Don’t expire cache (preferrably)
• Explicitly overwrite
• Update in background (feeeeeeeds)
• memcache FTW!
memcache

• Escape your keys using CGI::escape
• Max keylength is 250
• Max value size is 1MB
Rescueing
def self.get_feed_contents(url)
  content = ""
  open(url) { |s| content = s.read }
  RSS::Parser.parse(content, false).items
rescue => e
  logger.warn "Feed #{url} raised error: #{e.message}"
  []
rescue Timeout::Error => e
  logger.warn "Feed #{url} timed out: #{e.message}"
  []
end
          Timeout::Error is an interrupt...
Testing
• rcov
• Refactor your tests
• Peer reviews, external audits
• Run specs/features
  (continuously) in parallel
  (your Cucumber is too slooooow!)
Cucumber salad
num_of_processes.times do |count|
  pids << Process.fork do
    setup_database(conn, count)
    Cucumber::Cli::Main.execute(
      ["-f", "progress", "-l", "nl", "-r", "features"] +
      feature_sets[count]
    )
  end
end
                                           MacBook Pro
                               “Regular”               Mac Pro (8)
                                               (4)

                                 12:12        4:34        2:12
Conclusions

• Test
• Optimize
• Monitor
What’s next for us?

• Building a high-performance backend
• Uitzending Gemist statistics API
• 250+ reqs/s at minimum
@questions.any?

Mais conteúdo relacionado

Mais procurados

Habits of Effective Sqoop Users
Habits of Effective Sqoop UsersHabits of Effective Sqoop Users
Habits of Effective Sqoop UsersKathleen Ting
 
Benchmarking Solr Performance at Scale
Benchmarking Solr Performance at ScaleBenchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scalethelabdude
 
Building a near real time search engine & analytics for logs using solr
Building a near real time search engine & analytics for logs using solrBuilding a near real time search engine & analytics for logs using solr
Building a near real time search engine & analytics for logs using solrlucenerevolution
 
Safely Protect PostgreSQL Passwords - Tell Others to SCRAM
Safely Protect PostgreSQL Passwords - Tell Others to SCRAMSafely Protect PostgreSQL Passwords - Tell Others to SCRAM
Safely Protect PostgreSQL Passwords - Tell Others to SCRAMJonathan Katz
 
Cross Datacenter Replication in Apache Solr 6
Cross Datacenter Replication in Apache Solr 6Cross Datacenter Replication in Apache Solr 6
Cross Datacenter Replication in Apache Solr 6Shalin Shekhar Mangar
 
Empowering developers to deploy their own data stores
Empowering developers to deploy their own data storesEmpowering developers to deploy their own data stores
Empowering developers to deploy their own data storesTomas Doran
 
SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...
SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...
SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...Lucidworks
 
Search-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, EtsySearch-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, EtsyLucidworks
 
How a Small Team Scales Instagram
How a Small Team Scales InstagramHow a Small Team Scales Instagram
How a Small Team Scales InstagramC4Media
 
Let's Talk Operations! (Hadoop Summit 2014)
Let's Talk Operations! (Hadoop Summit 2014)Let's Talk Operations! (Hadoop Summit 2014)
Let's Talk Operations! (Hadoop Summit 2014)Allen Wittenauer
 
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Oleksiy Panchenko
 
Harnessing the power of Nutch with Scala
Harnessing the power of Nutch with ScalaHarnessing the power of Nutch with Scala
Harnessing the power of Nutch with ScalaKnoldus Inc.
 
How to build your query engine in spark
How to build your query engine in sparkHow to build your query engine in spark
How to build your query engine in sparkPeng Cheng
 
Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...
Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...
Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...Ontico
 
Achieving Infrastructure Portability with Chef
Achieving Infrastructure Portability with ChefAchieving Infrastructure Portability with Chef
Achieving Infrastructure Portability with ChefMatt Ray
 
Dublin Ireland Spark Meetup October 15, 2015
Dublin Ireland Spark Meetup October 15, 2015Dublin Ireland Spark Meetup October 15, 2015
Dublin Ireland Spark Meetup October 15, 2015Chris Fregly
 
Interactive Apache Spark in Your Browser
Interactive Apache Spark in Your BrowserInteractive Apache Spark in Your Browser
Interactive Apache Spark in Your BrowserCloudera, Inc.
 

Mais procurados (20)

Habits of Effective Sqoop Users
Habits of Effective Sqoop UsersHabits of Effective Sqoop Users
Habits of Effective Sqoop Users
 
Top ten-list
Top ten-listTop ten-list
Top ten-list
 
Benchmarking Solr Performance at Scale
Benchmarking Solr Performance at ScaleBenchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scale
 
Building a near real time search engine & analytics for logs using solr
Building a near real time search engine & analytics for logs using solrBuilding a near real time search engine & analytics for logs using solr
Building a near real time search engine & analytics for logs using solr
 
How Flipkart scales PHP
How Flipkart scales PHPHow Flipkart scales PHP
How Flipkart scales PHP
 
Safely Protect PostgreSQL Passwords - Tell Others to SCRAM
Safely Protect PostgreSQL Passwords - Tell Others to SCRAMSafely Protect PostgreSQL Passwords - Tell Others to SCRAM
Safely Protect PostgreSQL Passwords - Tell Others to SCRAM
 
Cross Datacenter Replication in Apache Solr 6
Cross Datacenter Replication in Apache Solr 6Cross Datacenter Replication in Apache Solr 6
Cross Datacenter Replication in Apache Solr 6
 
Empowering developers to deploy their own data stores
Empowering developers to deploy their own data storesEmpowering developers to deploy their own data stores
Empowering developers to deploy their own data stores
 
SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...
SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...
SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...
 
Search-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, EtsySearch-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, Etsy
 
How a Small Team Scales Instagram
How a Small Team Scales InstagramHow a Small Team Scales Instagram
How a Small Team Scales Instagram
 
Let's Talk Operations! (Hadoop Summit 2014)
Let's Talk Operations! (Hadoop Summit 2014)Let's Talk Operations! (Hadoop Summit 2014)
Let's Talk Operations! (Hadoop Summit 2014)
 
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
 
Giraph+Gora in ApacheCon14
Giraph+Gora in ApacheCon14Giraph+Gora in ApacheCon14
Giraph+Gora in ApacheCon14
 
Harnessing the power of Nutch with Scala
Harnessing the power of Nutch with ScalaHarnessing the power of Nutch with Scala
Harnessing the power of Nutch with Scala
 
How to build your query engine in spark
How to build your query engine in sparkHow to build your query engine in spark
How to build your query engine in spark
 
Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...
Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...
Efficient cluster resource management by using Cook and Mesos / Li Jin (Two S...
 
Achieving Infrastructure Portability with Chef
Achieving Infrastructure Portability with ChefAchieving Infrastructure Portability with Chef
Achieving Infrastructure Portability with Chef
 
Dublin Ireland Spark Meetup October 15, 2015
Dublin Ireland Spark Meetup October 15, 2015Dublin Ireland Spark Meetup October 15, 2015
Dublin Ireland Spark Meetup October 15, 2015
 
Interactive Apache Spark in Your Browser
Interactive Apache Spark in Your BrowserInteractive Apache Spark in Your Browser
Interactive Apache Spark in Your Browser
 

Destaque

Best approach to farming by sotonye anga
Best approach to farming by sotonye angaBest approach to farming by sotonye anga
Best approach to farming by sotonye angaSotonye anga
 
Trip to asia.ppx
Trip to asia.ppxTrip to asia.ppx
Trip to asia.ppxservinj
 
Millenium development goals progress chart to date
Millenium development goals  progress chart to dateMillenium development goals  progress chart to date
Millenium development goals progress chart to dateBread for the World
 
Table 6, mdg 6 combat hiv, malaria & other diseases
Table 6, mdg 6  combat hiv, malaria & other diseasesTable 6, mdg 6  combat hiv, malaria & other diseases
Table 6, mdg 6 combat hiv, malaria & other diseasesBread for the World
 
Vereshchagina English 1 Lesson 8
Vereshchagina English 1 Lesson 8Vereshchagina English 1 Lesson 8
Vereshchagina English 1 Lesson 8AnnMaryganova
 

Destaque (7)

Best approach to farming by sotonye anga
Best approach to farming by sotonye angaBest approach to farming by sotonye anga
Best approach to farming by sotonye anga
 
Trip to asia.ppx
Trip to asia.ppxTrip to asia.ppx
Trip to asia.ppx
 
test
testtest
test
 
Chocolate
ChocolateChocolate
Chocolate
 
Millenium development goals progress chart to date
Millenium development goals  progress chart to dateMillenium development goals  progress chart to date
Millenium development goals progress chart to date
 
Table 6, mdg 6 combat hiv, malaria & other diseases
Table 6, mdg 6  combat hiv, malaria & other diseasesTable 6, mdg 6  combat hiv, malaria & other diseases
Table 6, mdg 6 combat hiv, malaria & other diseases
 
Vereshchagina English 1 Lesson 8
Vereshchagina English 1 Lesson 8Vereshchagina English 1 Lesson 8
Vereshchagina English 1 Lesson 8
 

Semelhante a Lessons learned while building Omroep.nl

Web analytics at scale with Druid at naver.com
Web analytics at scale with Druid at naver.comWeb analytics at scale with Druid at naver.com
Web analytics at scale with Druid at naver.comJungsu Heo
 
Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...
Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...
Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...Databricks
 
Play Framework and Activator
Play Framework and ActivatorPlay Framework and Activator
Play Framework and ActivatorKevin Webber
 
How to Contribute to Apache Usergrid
How to Contribute to Apache UsergridHow to Contribute to Apache Usergrid
How to Contribute to Apache UsergridDavid M. Johnson
 
Druid at naver.com - part 1
Druid at naver.com - part 1Druid at naver.com - part 1
Druid at naver.com - part 1Jungsu Heo
 
Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1Henry S
 
6 tips for improving ruby performance
6 tips for improving ruby performance6 tips for improving ruby performance
6 tips for improving ruby performanceEngine Yard
 
5 Common Mistakes You are Making on your Website
 5 Common Mistakes You are Making on your Website 5 Common Mistakes You are Making on your Website
5 Common Mistakes You are Making on your WebsiteAcquia
 
Machine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossMachine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossAndrew Flatters
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudyJohn Adams
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackJakub Hajek
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek PROIDEA
 
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelSilicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelDaniel Coupal
 
ETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetupETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetupRafal Kwasny
 
MIGRATION - PAIN OR GAIN?
MIGRATION - PAIN OR GAIN?MIGRATION - PAIN OR GAIN?
MIGRATION - PAIN OR GAIN?DrupalCamp Kyiv
 
What is MariaDB Server 10.3?
What is MariaDB Server 10.3?What is MariaDB Server 10.3?
What is MariaDB Server 10.3?Colin Charles
 
What's New in Apache Spark 2.3 & Why Should You Care
What's New in Apache Spark 2.3 & Why Should You CareWhat's New in Apache Spark 2.3 & Why Should You Care
What's New in Apache Spark 2.3 & Why Should You CareDatabricks
 

Semelhante a Lessons learned while building Omroep.nl (20)

Web analytics at scale with Druid at naver.com
Web analytics at scale with Druid at naver.comWeb analytics at scale with Druid at naver.com
Web analytics at scale with Druid at naver.com
 
ITB2017 - Keynote
ITB2017 - KeynoteITB2017 - Keynote
ITB2017 - Keynote
 
Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...
Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...
Apache Spark Performance Troubleshooting at Scale, Challenges, Tools, and Met...
 
Play Framework and Activator
Play Framework and ActivatorPlay Framework and Activator
Play Framework and Activator
 
How to Contribute to Apache Usergrid
How to Contribute to Apache UsergridHow to Contribute to Apache Usergrid
How to Contribute to Apache Usergrid
 
Druid at naver.com - part 1
Druid at naver.com - part 1Druid at naver.com - part 1
Druid at naver.com - part 1
 
Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1
 
6 tips for improving ruby performance
6 tips for improving ruby performance6 tips for improving ruby performance
6 tips for improving ruby performance
 
5 Common Mistakes You are Making on your Website
 5 Common Mistakes You are Making on your Website 5 Common Mistakes You are Making on your Website
5 Common Mistakes You are Making on your Website
 
Machine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossMachine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy Cross
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudy
 
Mini-Training: Redis
Mini-Training: RedisMini-Training: Redis
Mini-Training: Redis
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
 
Remix
RemixRemix
Remix
 
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelSilicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
 
ETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetupETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetup
 
MIGRATION - PAIN OR GAIN?
MIGRATION - PAIN OR GAIN?MIGRATION - PAIN OR GAIN?
MIGRATION - PAIN OR GAIN?
 
What is MariaDB Server 10.3?
What is MariaDB Server 10.3?What is MariaDB Server 10.3?
What is MariaDB Server 10.3?
 
What's New in Apache Spark 2.3 & Why Should You Care
What's New in Apache Spark 2.3 & Why Should You CareWhat's New in Apache Spark 2.3 & Why Should You Care
What's New in Apache Spark 2.3 & Why Should You Care
 

Último

UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 

Último (20)

UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 

Lessons learned while building Omroep.nl

  • 1. Lessons learned while building omroep.nl Bart Zonneveld (@bartzon) Sjoerd Tieleman (@tieleman)
  • 2. Nederlandse Publieke Omroep Dutch Public broadcasting Company AVRO Joodse Omroep NMO Teleac BNN KRO NOS TROS BOS LLiNK NPS VARA EO MAX OHM VPRO HUMAN NCRV RKK ZvK IKON NIO RVU
  • 3. Rails sites • Beetlejuice • Radio 1 • Centrale • Z@PP navigatie • Omroep.nl • Z@ppelin • Nederland 1 • Zelda • Nederland 3 • Various tools
  • 5. Team • 2 coders • 1 designer • 1 editor • 1 project manager 6 months, CMS built from scratch
  • 6. Requirements • Handle 30.000-40.000 pageviews per day • Handle traffic spikes • Flexible, multi user CMS • Loads of external data
  • 8. Some numbers +----------------------+-------+-------+---------+---------+-----+-------+ | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+-------+-------+---------+---------+-----+-------+ | Controllers | 1864 | 1535 | 41 | 185 | 4 | 6 | | Helpers | 797 | 631 | 1 | 75 | 75 | 6 | | Models | 1303 | 1055 | 40 | 153 | 3 | 4 | | Libraries | 814 | 620 | 15 | 79 | 5 | 5 | | Integration tests | 0 | 0 | 0 | 0 | 0 | 0 | | Functional tests | 0 | 0 | 0 | 0 | 0 | 0 | | Unit tests | 0 | 0 | 0 | 0 | 0 | 0 | | Model specs | 1932 | 1573 | 0 | 9 | 0 | 172 | | View specs | 7322 | 5950 | 0 | 153 | 0 | 36 | | Controller specs | 7292 | 5846 | 0 | 175 | 0 | 31 | | Helper specs | 900 | 676 | 0 | 2 | 0 | 336 | | Library specs | 56 | 45 | 2 | 12 | 6 | 1 | +----------------------+-------+-------+---------+---------+-----+-------+ | Total | 22280 | 17931 | 99 | 843 | 8 | 19 | +----------------------+-------+-------+---------+---------+-----+-------+ Code LOC: 3841 Test LOC: 14090 Code to Test Ratio: 1:3.7
  • 9. Moar numbers • 410 Cucumber scenarios, 600 step definitions • 2235 RSpec specifications so it must be bug-free, right? ;-)
  • 10. Tools Ruby on Rails 2.3.4 Rspec + Webrat + Cucumber Apache 2.2 Paperclip SVN App monitoring: RPM, Hoptoad MySQL 5 Service monitoring: Nagios Memcache
  • 11. Tools: app monitoring Hoptoad New Relic RPM
  • 12. Architecture • Apache 2.2 with mod_proxy • Rails 2.3.4 running on Phusion Passenger 2.2.5 with REE • 4 hosts, each running 4 instances (per app) Appdex: 1.0, avg. response time 40ms, 130 rpm, db load 0.6 %
  • 13. Servers • Quadcore Intel Xeon E542, 32 GB Ram • Fedora 8 • Other mumbojumbo
  • 14. Architecture Front proxy Front proxy Application Application Application Application server server server server Database memcache
  • 15. Workflow • BDD • Shared behaviours • Performance testing • Staging and production environment
  • 17. 3 slide intro to BDD: RSpec describe Article do it_should_behave_like "all objects with userstamps" it_should_behave_like "all objects than can be published" it_should_behave_like "all objects that have an url" it_should_behave_like "all objects that can be searched" it_should_behave_like "all objects with related articles" it "should not be valid without a name" do @article.attributes = @valid_attributes.except(:name) @article.should_not be_valid end it "should not be valid without contents" do @article.attributes = @valid_attributes.except(:contents) @article.should_not be_valid end end
  • 18. 3 slide intro to BDD: Cucumber features Feature: Articles on the homepage As a visitor I want to view articles on the homepage So that I can see the latest content Scenario: 5 most recent articles Given there are 8 articles When I visit the homepage Then I should see the 5 last published articles
  • 19. 3 slide intro to BDD: Cucumber steps Given "there are $num articles" do |num| num.to_i.times { create_article } end When "I visit the homepage" do visit root_path end Then "I should see the $num last published articles" do |num| Article.last_published(num).each do |article| response.should contain(article.title) end end
  • 20. Shared behaviours • Tags • User stamps (created by, updated by) • Searching • “Related” articles • Publication timestamps (on/offline at)
  • 21. Shared behaviours module UserStamps def self.included(klass) class Article < ActiveRecord::Base klass.instance_eval do include Shared::UserStamps include InstanceMethods end include Shared::Published end include Shared::Url include Shared::Search module InstanceMethods include Shared::RelatedArticles def created_by User.find_by_id(creator_id) # stuff end end def updated_by User.find_by_id(updater_id) end end end
  • 22. Workflow: performance testing • ab, httperf, autobench, cURL • NewRelic RPM • Safari Web Inspector • http://railslab.newrelic.com/scaling-rails
  • 24. Challenges • Content Management System • Loads and loads and loads of external data
  • 25. CMS • Articles, Themas, Specials, Subsites • Multiple feeds, images, links • Version control • Media database
  • 26. CMS: Articles Subsite Thema Page Special Article Link Feed Image
  • 28. Media DB • Implemented as REST app • To be used as REST service • Files, folders, crops
  • 29. External data • RSS feeds • EPG data • Zelda • Babel • News/sport/teletekst • Twitter • Lots of custom XML formats
  • 30. External data: XML/RSS • Empty feeds • Encodings are off (Windows-1252, ISO-8859-1, UTF-8) • “Custom” fields • Incorrect fields (dates, unescaped HTML) • Timeouts • Everything that can go wrong, will go wrong
  • 32. External data: EPG data Zelda don’t sue us Nintendo... please? :)
  • 33. External data • Empty feeds • Retrieving the feed while someone is updating it • Required fields that are empty • DTD?
  • 34. <!ELEMENT aflevering ( prid?, tite?, medium?, icon?, aankondiging?, inkl?, ingl?, infi?, inak?, inds?, inbb?, kykw?, orti?, aant?, land?, lcod?, psrt?, prem?, inh1?, afle?, atit?, inh2?, bron?, prij?, inh3?, mail?, webs?, inhk?, gids_tekst?, omroepen?, genres?, personen?, streams?, fragmenten?, serie? )>
  • 35. <!ELEMENT inkl (#PCDATA)> <!ELEMENT ingl (#PCDATA)> <!ELEMENT intt (#PCDATA)> <!ELEMENT inhh (#PCDATA)> <!ELEMENT omro (#PCDATA)> <!ELEMENT lcod (#PCDATA)> <!ELEMENT herh (#PCDATA)> <!ELEMENT inds (#PCDATA)> <!ELEMENT infi (#PCDATA)> <!ELEMENT inbb (#PCDATA)> <!ELEMENT genr (#PCDATA)> <!ELEMENT kykw (#PCDATA)> <!ELEMENT afle (#PCDATA)>
  • 36. Lessons learned • Cache the crap out of everything • Rescue everything • Test everything (frontend and backend)
  • 37. Caching • Cache the homepage • Page cache → Fragment cache • Don’t cache forms • Cache as much as possible
  • 38. Case: article views • Article is page cached • Update the number of views in realtime?
  • 39. Use AJAX! <% javascript_tag do %> <%= remote_function :url => update_views_article_path(@article) %> <% end %>
  • 41. Case: banner items • Fast requests (<10ms) • ETags (304 Not Modified) • Static resource → page cache • Move to front proxy, frees up Rails cluster • 1100rpm → 130rpm • 20ms → 40ms • Average response time going up? Oh nooooes!
  • 42. Caching external data • Don’t expire cache (preferrably) • Explicitly overwrite • Update in background (feeeeeeeds) • memcache FTW!
  • 43. memcache • Escape your keys using CGI::escape • Max keylength is 250 • Max value size is 1MB
  • 44. Rescueing def self.get_feed_contents(url) content = "" open(url) { |s| content = s.read } RSS::Parser.parse(content, false).items rescue => e logger.warn "Feed #{url} raised error: #{e.message}" [] rescue Timeout::Error => e logger.warn "Feed #{url} timed out: #{e.message}" [] end Timeout::Error is an interrupt...
  • 45. Testing • rcov • Refactor your tests • Peer reviews, external audits • Run specs/features (continuously) in parallel (your Cucumber is too slooooow!)
  • 46. Cucumber salad num_of_processes.times do |count| pids << Process.fork do setup_database(conn, count) Cucumber::Cli::Main.execute( ["-f", "progress", "-l", "nl", "-r", "features"] + feature_sets[count] ) end end MacBook Pro “Regular” Mac Pro (8) (4) 12:12 4:34 2:12
  • 48. What’s next for us? • Building a high-performance backend • Uitzending Gemist statistics API • 250+ reqs/s at minimum