O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Map Reduce v2 and YARN - CHUG - 20120604

4.719 visualizações

Publicada em

View the accompanying video on vimeo: https://vimeo.com/43474797

Publicada em: Tecnologia
  • http://dbmanagement.info/Tutorials/MapReduce.htm
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Map Reduce v2 and YARN - CHUG - 20120604

  1. 1. An Introduction to MapReduce 2 and YARN Tom White, Cloudera @tom_e_white June 4, 2012 Chicago HUGTuesday, June 5, 2012
  2. 2. Road TripTuesday, June 5, 2012
  3. 3. About me • Apache Hadoop Committer, PMC Member, Apache Member • Engineer at Cloudera working on core Hadoop • Founder of Apache Whirr • Author of “Hadoop: The Definitive Guide” • http://hadoopbook.comTuesday, June 5, 2012
  4. 4. First, whatʼs MapReduce 1?Tuesday, June 5, 2012
  5. 5. Tuesday, June 5, 2012
  6. 6. Whatʼs wrong with MR1?Tuesday, June 5, 2012
  7. 7. Motivation 1 • Scaling >4000 nodes • Fewer, larger clustersTuesday, June 5, 2012
  8. 8. Motivation 2 • HA of Job Tracker • Large, complex stateTuesday, June 5, 2012
  9. 9. Motivation 3 • Poor resource utilization • Slots in MR1 are for either map or reduceTuesday, June 5, 2012
  10. 10. Yet Another Resource NegotiatorTuesday, June 5, 2012
  11. 11. Tuesday, June 5, 2012
  12. 12. Tuesday, June 5, 2012
  13. 13. Node Manager is a generalized Task Tracker • Task Tracker • fixed number of map or reduce slots • Node Manager • containers with variable resource limitsTuesday, June 5, 2012
  14. 14. Tuesday, June 5, 2012
  15. 15. Tuesday, June 5, 2012
  16. 16. MR is user space YARN is kernelTuesday, June 5, 2012
  17. 17. Bonus Apps • Distributed shell • MPI (MAPREDUCE-2911) • Master-worker (MAPREDUCE-3315) • Apache Giraph, HamaTuesday, June 5, 2012
  18. 18. Tuesday, June 5, 2012
  19. 19. Tuesday, June 5, 2012
  20. 20. Old API ≠ MR1 New API ≠ MR2Tuesday, June 5, 2012
  21. 21. Old API New API o.a.h.mapred o.a.h.mapreduce MR1 ✓ ✓ MR2 ✓ ✓Tuesday, June 5, 2012
  22. 22. Tuesday, June 5, 2012
  23. 23. Try out MR2 • Apache Hadoop 2.0.0-alpha • hadoop.apache.org • CDH4 and Cloudera Manager • cloudera.com • Cloud - Apache WhirrTuesday, June 5, 2012
  24. 24. MR1 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>1.0.3</version> </dependency> MR2 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.0.0-alpha</version> </dependency>Tuesday, June 5, 2012
  25. 25. TODO • Still alpha status • Performance tuning • Usability bug fixes • RM recovery • Security in MR2 not completeTuesday, June 5, 2012
  26. 26. Questions?Tuesday, June 5, 2012

×