O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.


3.882 visualizações

Publicada em

how we built a firehose with NodeJS and ZeroMQ

Publicada em: Tecnologia, Negócios
  • Dating direct: ❶❶❶ http://bit.ly/39pMlLF ❶❶❶
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Dating for everyone is here: ❤❤❤ http://bit.ly/39pMlLF ❤❤❤
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui


  1. 1. FirehoseBuilding a real-time data stream with NodeJS Luca Grulla@Forward London, 2/8/2011
  2. 2. IntroductionOr what we wanted to build
  3. 3. We like to have a (near) real-time view of the data stream
  4. 4. Resilient
  5. 5. Scalable
  6. 6. Accurate(but not necessarily precise)
  7. 7. Verse and Chorus Or how we built it
  8. 8. Stack• NodeJS (0.4.9)• CoffeeScript• ZeroMQ
  9. 9. Architecture
  10. 10. InterludeOr joys and pitfalls of building a tail library for node
  11. 11. Living the asynchronous dream
  12. 12. Living the asynchronous dream• Error manifests itself only at high throughput (7MB/secs)• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
  13. 13. Breathing in an asynchronous way
  14. 14. Learnings• You need to think asynchronous to write asynchronous code• Dispatch internal events if you need to keep control of the flow• “Internal queues” to scale(up to a certain limit)
  15. 15. Performance• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput
  16. 16. Questions ?
  17. 17. References• npm install tail• https://github.com/forward/node-tail