Globo.com’s Live
Video Platform
for FIFA World Cup 14
09/23/15
#nginx #nginxconf
an amazing teamwork
Leandro
Moreira
other teamsDaniel
Martins
Flávio
Ribeiro
Juarez
Bochi
Hugo
Roque
Robson
Soares
Rafael
...
streaming rights for Brazil only
Agenda (4+ years of work)
2010 Architecture for FIFA World Cup 10
2012 Mobile support
2013 HLS adoption
2014 DVR
2014 Wait...
architecture in 2010
#nginx #nginxconf
{2010} in the beginning
encoderh264+aacrtmptcp
h264+aacrtmptcp
flash player
firewall
1935
1º
2º3º
4º
RTMP
{2010} good part
small delay (2-5s)
{2010} issue :: instrumentation
tail -f
kill -9
ps aux
{2010} issue :: scalability
$$$$$
{2010} issue :: scalability
{2010} some data from FIFA10
● ~= 285k simult. users *
● max bitrate 800 Kbps
*peak
mobile support
#nginx #nginxconf
{2012} and then multi-devices...
hls
h264+aacrtmptcp
h264+aacrtmptcp
h264+aacrtmptcp
hls
{2012} how does HLS work?
seg1.ts
seg2.ts
seg3.ts
seg4.ts
240p.m3u8
480p.m3u8
720p.m3u8
seg1.ts
seg2.ts
seg3.ts
seg4.ts
se...
{2012} first hls solution
encoderh264+aacrtmptcp
h264+aachttptcp
flash player
2º3º
5º
4º
android
ios
1º
HTTP
{2012} generation and distribution
rtmp
video
playlists
files
location ~ /hls {
root "/vol/hls/";
}
dest = "/vol/hls";
htt...
{2012} first solution architecture
BEFE LB SELB LB
auth & cache mostly cache hls generation
upstream upstream
http
HLS adoption
#nginx #nginxconf
{2012} one protocol to rule them all
{2013} buffering hls x rtmp
6x less buffering with HLS
{2013} instrumentation
Graphite
● status code
● request t.
● cache hit
● cpu usage
● memory
● buffer rate
● bitrate
time-s...
{2013} sauron
Graphite
{2014} caching configuration
don’t forget to: proxy_cache_use_stale updating
011011100
110011101
101001011
011100111
1000
...
{2014} OS fine tuning
{2014} OS fine tuning
19Gpbs maxconn: 70k, cpu affinity
15Gpbs maxconn: 40k, cpu affinity
12Gpbs maxconn: 25k, irqbalance
...
{2014} what we’ve got here is
Scalabilityport:80 UX Instrumentation
2G
4G
Wifi
1080p
720p
320p
Caching
{2014} farm size
FE: 80 nodes
24 cores 64 GB 1TB
10 Gbps 10 Gbps
BE: 6 nodes
SE: 2 nodes
DVR
#nginx #nginxconf
FIFA 2014 World Cup Brasil
{2014} what is DVR for hls?
/720p.m3u8
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:5...
ingest
{2014} reviewing current solution
frontend
encoderh264+aacrtmptcp
h264+aachttptcp
flash player
2º3º
5º
4º
android
i...
{2014} dvr challenges :: failover
didn’t have a failover option for DVR
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
...
{2014} dvr challenges :: storage
2h * 60m * 60s / 5s * 4.5mb ~= 6.4GB
}1.6MB 1.1MB 0.8MB 0.5MB 0.3MB 0.1MB
4.5MB
per 5s
2 ...
{2014} redis as a datastore
ingest
embed-script: fetch and serve hls
frontend
daemon: mon file change and save hls
{2014} Brazil’s general election
too many simultaneous live streams
{2014} from redis to cassandra
ingest
embed-script: fetch and serve hls
frontend
daemon: mon file change and save hls
waiting room
#nginx #nginxconf
{2014} multihoming (anycast)
ISPx
ISPy
ISPk
SPBGP announcement
186.192.81.0/24
ISPx ISPy
ISPk
RJ
186.192.81.2
BGP announce...
{2014} user’s bandwidth
ISP X
ISP Y
20Gbps
5Gbps
IXP
ISP K
80Gbps
8Gbps
FIFA World Cup 2014 results
#nginx #nginxconf
FIFA 2014 World Cup results
FIFA 2014 World Cup Brazil data
● > 500k simult. users (peak 1 game)
● ~= 125.000 video req/s (peak 1 game)
● > 640 Gbps (peak 1 game)
FIFA 2014 World Cup...
FIFA 2014 World Cup Brazil data
● ~= 1600 watched years (sum all)
● > 40M video views (sum all)
● ~= avg 6 Gbps, max 19 Gb...
● ~= cpu usage max 10% (per node, FE)
● ~= avg bitrate 480 Kbps (crowded)
● ~= avg bitrate 1864 Kbps (less crowded)
FIFA 2...
Recap and next steps
#nginx #nginxconf
● video streaming:
○ delivery and caching
○ geolocation (geofencing)
○ authorization / authentication
● microservices
○ pl...
is amazing
Lua was made in Brazil :)
TDD with busted
easier to develop
performance near C (lang)
OSS from this history
m3u8 parser
cassandra
driver
clappr
evostream
api
live_thumb
nginx-audio-track-module
BemTV
iprange
summary and future
2010
FIFA
2012
BBB
2014
FIFA
Elections
● hls
● http
● flash
● rtmp
● hls
● http
● dvr
2016
Olympic
Game...
Questions?
THANK YOU!
leandromoreira.com.br
jbochi.me
nginx.conf  - globo.com's live video platform for fifa world cup 14
nginx.conf  - globo.com's live video platform for fifa world cup 14
Próximos SlideShares
Carregando em…5
×

nginx.conf - globo.com's live video platform for fifa world cup 14

3.597 visualizações

Publicada em

In this talk, we will describe globo.com's live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) .
Nginx is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp.

this was presented by leandro moreira and juarez bochi at nginx.conf in 09/23/15

Publicada em: Internet
  • Seja o primeiro a comentar

nginx.conf - globo.com's live video platform for fifa world cup 14

  1. 1. Globo.com’s Live Video Platform for FIFA World Cup 14 09/23/15 #nginx #nginxconf
  2. 2. an amazing teamwork Leandro Moreira other teamsDaniel Martins Flávio Ribeiro Juarez Bochi Hugo Roque Robson Soares Rafael Borsani Bernardo Camilo Lucas Mundim Juan Almeida more people
  3. 3. streaming rights for Brazil only
  4. 4. Agenda (4+ years of work) 2010 Architecture for FIFA World Cup 10 2012 Mobile support 2013 HLS adoption 2014 DVR 2014 Waiting room 2014 FIFA World Cup results 2015 Recap & Next steps
  5. 5. architecture in 2010 #nginx #nginxconf
  6. 6. {2010} in the beginning encoderh264+aacrtmptcp h264+aacrtmptcp flash player firewall 1935 1º 2º3º 4º RTMP
  7. 7. {2010} good part small delay (2-5s)
  8. 8. {2010} issue :: instrumentation tail -f kill -9 ps aux
  9. 9. {2010} issue :: scalability $$$$$
  10. 10. {2010} issue :: scalability
  11. 11. {2010} some data from FIFA10 ● ~= 285k simult. users * ● max bitrate 800 Kbps *peak
  12. 12. mobile support #nginx #nginxconf
  13. 13. {2012} and then multi-devices... hls h264+aacrtmptcp h264+aacrtmptcp h264+aacrtmptcp hls
  14. 14. {2012} how does HLS work? seg1.ts seg2.ts seg3.ts seg4.ts 240p.m3u8 480p.m3u8 720p.m3u8 seg1.ts seg2.ts seg3.ts seg4.ts seg1.ts seg2.ts seg3.ts seg4.ts 1 - GET /variant.m3u8 (text) 2 - GET /720p.m3u8 (text) 240p.m3u8 variant.m3u8 480p.m3u8 720p.m3u8 3 - GET /seg3.ts (video) 4 - GET /seg4.ts (video) 5 - GET /720p.m3u8 (text) 6 - GET /seg5.ts (video) 7 - GET /seg6.ts (video) 8 - ... http http seg3.ts seg4.ts seg5.ts seg6.ts seg3.ts seg4.ts seg5.ts seg6.ts seg3.ts seg4.ts seg5.ts seg6.ts
  15. 15. {2012} first hls solution encoderh264+aacrtmptcp h264+aachttptcp flash player 2º3º 5º 4º android ios 1º HTTP
  16. 16. {2012} generation and distribution rtmp video playlists files location ~ /hls { root "/vol/hls/"; } dest = "/vol/hls"; http BEFE segmenter
  17. 17. {2012} first solution architecture BEFE LB SELB LB auth & cache mostly cache hls generation upstream upstream http
  18. 18. HLS adoption #nginx #nginxconf
  19. 19. {2012} one protocol to rule them all
  20. 20. {2013} buffering hls x rtmp 6x less buffering with HLS
  21. 21. {2013} instrumentation Graphite ● status code ● request t. ● cache hit ● cpu usage ● memory ● buffer rate ● bitrate time-series data graph api http enqueue dequeuemetric x t snmp metric x t agents
  22. 22. {2013} sauron Graphite
  23. 23. {2014} caching configuration don’t forget to: proxy_cache_use_stale updating 011011100 110011101 101001011 011100111 1000 video0.ts video0.ts expired UPDATING FE BE proxy_cache_lock
  24. 24. {2014} OS fine tuning
  25. 25. {2014} OS fine tuning 19Gpbs maxconn: 70k, cpu affinity 15Gpbs maxconn: 40k, cpu affinity 12Gpbs maxconn: 25k, irqbalance packet loss 4Gbps, no fine tuning *each machine has 2 NIC bonded (10GB each one)
  26. 26. {2014} what we’ve got here is Scalabilityport:80 UX Instrumentation 2G 4G Wifi 1080p 720p 320p Caching
  27. 27. {2014} farm size FE: 80 nodes 24 cores 64 GB 1TB 10 Gbps 10 Gbps BE: 6 nodes SE: 2 nodes
  28. 28. DVR #nginx #nginxconf
  29. 29. FIFA 2014 World Cup Brasil
  30. 30. {2014} what is DVR for hls? /720p.m3u8 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5, seg1.ts #EXTINF:5, seg2.ts #EXTINF:5, seg3.ts /720p.m3u8 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5, seg1.ts #EXTINF:5, seg2.ts #EXTINF:5, seg3.ts #EXTINF:5, seg4.ts #EXTINF:5, seg5.ts #EXTINF:5, seg6.ts #EXTINF:5, seg7.ts #EXTINF:5, seg8.ts #EXTINF:5, seg9.ts #EXTINF:5, seg10.ts
  31. 31. ingest {2014} reviewing current solution frontend encoderh264+aacrtmptcp h264+aachttptcp flash player 2º3º 5º 4º android ios 1º
  32. 32. {2014} dvr challenges :: failover didn’t have a failover option for DVR #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:10 #EXTINF:5, seg10.ts #EXTINF:5, seg11.ts #EXTINF:5, seg12.ts #EXTINF:5, seg13.ts #EXTINF:5, seg14.ts #EXTINF:5, seg15.ts #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5, seg1.ts
  33. 33. {2014} dvr challenges :: storage 2h * 60m * 60s / 5s * 4.5mb ~= 6.4GB }1.6MB 1.1MB 0.8MB 0.5MB 0.3MB 0.1MB 4.5MB per 5s 2 games + 4 other streams ~= 38.4GB 02:00:0000:00:00
  34. 34. {2014} redis as a datastore ingest embed-script: fetch and serve hls frontend daemon: mon file change and save hls
  35. 35. {2014} Brazil’s general election too many simultaneous live streams
  36. 36. {2014} from redis to cassandra ingest embed-script: fetch and serve hls frontend daemon: mon file change and save hls
  37. 37. waiting room #nginx #nginxconf
  38. 38. {2014} multihoming (anycast) ISPx ISPy ISPk SPBGP announcement 186.192.81.0/24 ISPx ISPy ISPk RJ 186.192.81.2 BGP announcement 186.192.81.0/24 PoP PoP
  39. 39. {2014} user’s bandwidth ISP X ISP Y 20Gbps 5Gbps IXP ISP K 80Gbps 8Gbps
  40. 40. FIFA World Cup 2014 results #nginx #nginxconf
  41. 41. FIFA 2014 World Cup results
  42. 42. FIFA 2014 World Cup Brazil data
  43. 43. ● > 500k simult. users (peak 1 game) ● ~= 125.000 video req/s (peak 1 game) ● > 640 Gbps (peak 1 game) FIFA 2014 World Cup Brazil data
  44. 44. FIFA 2014 World Cup Brazil data ● ~= 1600 watched years (sum all) ● > 40M video views (sum all) ● ~= avg 6 Gbps, max 19 Gbps
  45. 45. ● ~= cpu usage max 10% (per node, FE) ● ~= avg bitrate 480 Kbps (crowded) ● ~= avg bitrate 1864 Kbps (less crowded) FIFA 2014 World Cup Brazil data
  46. 46. Recap and next steps #nginx #nginxconf
  47. 47. ● video streaming: ○ delivery and caching ○ geolocation (geofencing) ○ authorization / authentication ● microservices ○ playlist generation ○ waiting room ○ streams per account locking system Nginx usage
  48. 48. is amazing Lua was made in Brazil :) TDD with busted easier to develop performance near C (lang)
  49. 49. OSS from this history m3u8 parser cassandra driver clappr evostream api live_thumb nginx-audio-track-module BemTV iprange
  50. 50. summary and future 2010 FIFA 2012 BBB 2014 FIFA Elections ● hls ● http ● flash ● rtmp ● hls ● http ● dvr 2016 Olympic Games ● dash ??? ● ingest ??? ● 4k ???
  51. 51. Questions? THANK YOU! leandromoreira.com.br jbochi.me

×