O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Pig at Linkedin
Chris Riccomini
9/29/10
Who?
What?
LinkedIn Analytics
Pig at LinkedIn
Why?
Production Quality
Streaming
Serialization
VoldemortStorage ~ Avro
views = LOAD '/data/awesome' USING VoldemortStorage();
Voldemort ♥ Pig
Partitioning
YYYY/MM/DD
Last N days?
views = LOAD '/data/etl/tracking/extracted/profile-view' USING
VoldemortStorage('date.range', 'num.days=90;days.ago=1’)
Some-file-YYYY-MM-DD
member_position = LOAD '/data/etl/replicated/member/member_position/#LATEST'
USING VoldemortStorage()
Scheduling
Azkaban
type=pig
pig.script=myscript.pig
Ad hoc?
Future at LinkedIn
Wishes
Dates
Fix Data Types
JSON
Cross Platform
Questions?
• criccomini@linkedin.com
• http://www.riccomini.name
• http://www.sna-projects.com
• http://www.project-voldem...
Pig at Linkedin
Pig at Linkedin
Pig at Linkedin
Próximos SlideShares
Carregando em…5
×

Pig at Linkedin

4.310 visualizações

Publicada em

Pig at LinkedIn by Chris Riccomini from LinkedIn
Pig is an integral part of data analytics at LinkedIn. Learn about LinkedIn’s analytic stack, and see how Pig is used to design, develop, and deliver data products at LinkedIn. We’ll explore a successful example of Pig deployment at LinkedIn, pain points, and integration with Azkaban, Voldemort, Hadoop, and the rest of LinkedIn’s ecosystem.

Publicada em: Educação, Tecnologia, Negócios
  • Entre para ver os comentários

Pig at Linkedin

  1. 1. Pig at Linkedin Chris Riccomini 9/29/10
  2. 2. Who?
  3. 3. What?
  4. 4. LinkedIn Analytics
  5. 5. Pig at LinkedIn
  6. 6. Why?
  7. 7. Production Quality
  8. 8. Streaming
  9. 9. Serialization
  10. 10. VoldemortStorage ~ Avro
  11. 11. views = LOAD '/data/awesome' USING VoldemortStorage();
  12. 12. Voldemort ♥ Pig
  13. 13. Partitioning
  14. 14. YYYY/MM/DD
  15. 15. Last N days?
  16. 16. views = LOAD '/data/etl/tracking/extracted/profile-view' USING VoldemortStorage('date.range', 'num.days=90;days.ago=1’)
  17. 17. Some-file-YYYY-MM-DD
  18. 18. member_position = LOAD '/data/etl/replicated/member/member_position/#LATEST' USING VoldemortStorage()
  19. 19. Scheduling
  20. 20. Azkaban
  21. 21. type=pig pig.script=myscript.pig
  22. 22. Ad hoc?
  23. 23. Future at LinkedIn
  24. 24. Wishes
  25. 25. Dates
  26. 26. Fix Data Types
  27. 27. JSON
  28. 28. Cross Platform
  29. 29. Questions? • criccomini@linkedin.com • http://www.riccomini.name • http://www.sna-projects.com • http://www.project-voldemort.com • @criccomini • LinkedIn is Hiring! Email me!

×