O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Treinamento Hadoop - dia2

1.615 visualizações

Publicada em

Publicada em: Tecnologia
  • ⇒⇒⇒WRITE-MY-PAPER.net ⇐⇐⇐ has really great writers to help you get the grades you need, they are fast and do great research. Support will always contact you if there is any confusion with the requirements of your paper so they can make sure you are getting exactly what you need.
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • My friend sent me a link to to tis site. This awesome company. They wrote my entire research paper for me, and it turned out brilliantly. I highly recommend this service to anyone in my shoes. ⇒ www.HelpWriting.net ⇐.
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Treinamento Hadoop - dia2

  1. 1. Treinamento Hadoop Parte 2
  2. 2. Alexandre Uehara ● Analista Desenvolvedor (E-commerce) ● Coordenador Trilha Big Data (TDC 2013) http://www.thedevelopersconference.com. br/tdc/2013/saopaulo/trilha-bigdata#programacao ● Geek e nerd, trabalha com Python, Java, C, BigData, NoSQL, e outras coisas mais que aparecem no dia a dia
  3. 3. Nossa Agenda:
  4. 4. Dia 2!!!
  5. 5. Dia 1: Introdução Conceitos Básicos Big Data → O que é? → 4 V's → Por que agora? → Onde Usar? → Desafio → BI x Big Data →Profissional de Big Data → Ferramentas → Dicas
  6. 6. Dia 2 e 3: Map Reduce e Hadoop → História (Google) → Conceito MapReduce → Hadoop → Cases
  7. 7. Dia 4 Outras Ferramentas → Ecossistema → Ferramentas → Distribuições Hadoop → Hadoop na Nuvem → NoSQL Tipos → NoSQL Exemplos → Data Science → Machine Learning
  8. 8. Dia 5, 6 e 7 Hands-on → Pyhton → Hadoop
  9. 9. Antes disso...
  10. 10. Antes disso... lembram o quer era Big Data?
  11. 11. Antes disso... lembram o quer era Big Data? CONCEITO!!!
  12. 12. lembram o quer era Big Data? Agora que entedemos que é um CONCEITO, vamos ver a FERRAMENTA Antes disso...
  13. 13. Mas o que é?
  14. 14. Map Reduce
  15. 15. Map Reduce Hadoop é baseado no modelo de programação MapReduce
  16. 16. "MapReduce é um modelo de programação desenhado para processar grandes volumes de dados em paralelo, dividindo o trabalho em um conjunto de tarefas independentes." http://en.wikipedia.org/wiki/MapReduce
  17. 17. Map Reduce Vem da onde?
  18. 18. Map Reduce Framework introduzido pelo ...
  19. 19. Map Reduce Framework introduzido pelo Google
  20. 20. Map Reduce http://research.google.com/archive/mapreduce.html
  21. 21. Map Reduce http://research.google.com/archive/mapreduce.html
  22. 22. Map Reduce ● Não inventou, apenas implementou ● Mas, "definiu" / "padronizou" algorítimo e arquitetura
  23. 23. Map Reduce ● paradigma de programação para processar e analisar grandes conjuntos de dados
  24. 24. Map Reduce ● Processamento tolerante a falha e fácil abstração ● Facilidade de escalar ● Decompõe tudo em Pares - chave : valor ● Depois sumariza os resultados
  25. 25. Map Reduce FileSystem Map Reduce
  26. 26. Map Reduce Google FileSystem Google Map Reduce
  27. 27. Map Reduce
  28. 28. Map Reduce
  29. 29. Map Reduce Inspirado pelas funções map e reduce usadas em programação funcional.
  30. 30. Map Reduce "Programação funcional é um paradigma de programação que trata a computação como uma avaliação de funções matemáticas e que evita estados ou dados mutáveis. Ela enfatiza a aplicação de funções, em contraste da programação imperativa, que enfatiza mudanças no estado do programa" Wikipedia - http://pt.wikipedia.org/wiki/Programa%C3%A7%C3%A3o_funcional Exemplos de linguagem: ● Lisp ● Haskell ● Scala
  31. 31. Map Reduce Se encarrega de toda complexidade de processamento paralelo, e o programador cuida apenas da lógica
  32. 32. Map Reduce ● Você especifica função map() ● Você especifica função reduce() ● O framework cuida do restante
  33. 33. Map Reduce Lembra o que era Map / Mapa?
  34. 34. Map Reduce Map - associação chave-valor (key-value) k,v
  35. 35. Map Reduce ● Duas fases: Map Reduce ● Entre MapReduce : shuffle e sort
  36. 36. Map Reduce
  37. 37. No Linux seria mais ou menos: cat /my/log | grep '*.*. html' | sort | uniq -c > /output -file
  38. 38. Map Reduce Map Reduce
  39. 39. Outros Map Reduce ● Hadoop http://hadoop.apache.org/ ● Twister http://www.iterativemapreduce.org/ ● Phoenix http://mapreduce.stanford.edu/ ● FileMap http://mfisk.github.io/filemap/
  40. 40. Map Reduce ● Modelo Simples de programação ● Sistema de arquivos distribuidos com gerenciamento de tarefas
  41. 41. Map Reduce ● Gerenciamento de cluster ● Excelente em processamento em batch ● Não é feito para processos online
  42. 42. Google Filesystem
  43. 43. Google Filesystem (GFS) ● Sistema de arquivos distribuídos otimizados para atuar em dados não estruturados ● Necessário pois os arquivos pode começar a tornar grandes demais para serem armazenados em apenas uma máquina
  44. 44. Google Filesystem (GFS)
  45. 45. Google Filesystem (GFS) ● Tolerancia a falhas (se falhar um nó, automaticamente replica para outro) ● Suporte a falha parcial ● Suporte a recuperação de dados ● Suporte a recuperação individual de nós
  46. 46. Google Filesystem (GFS) ● Consistência ● Escalável (pode aumentar / diminuir sem afetar a aplicação) ● Milhares de nós ● Processamento paralelo ● Nós independentes
  47. 47. Map Reduce + GFS Ou seja: ● Escalável ● Econômico ● Eficiente ● Confiável
  48. 48. Map Reduce Exemplo "Word Count"
  49. 49. Alexandre Uehara @AleUehara linkedin.com/in/aleuehara uehara.alexandre@gmail.com slideshare.net/aleuehara

×