O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Real-Time-Analytics mit Spark und Cassandra

977 visualizações

Publicada em

Real-Time-Analytics mit Spark und Cassandra.

Eine konzeptionelle Übersicht zur Integration, den Hintergründen als auch Vorteilen einer Kombination zwischen Spark und Cassandra.

Vortrag am 6ten OSBI Workshop 05.03.2015 in Offenburg.

Publicada em: Dados e análise
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ...................................ALL FOR EBOOKS................................................. Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Real-Time-Analytics mit Spark und Cassandra

  1. 1. Real-Time Analytics mit Spark und Cassandra WidasConcepts Unternehmensberatung GmbH Ÿ Maybachstraße 2 Ÿ 71299 Wimsheim Ÿ http://www.widas.de im März 2015 OSBI – Workshop http://www.osbi-workshop.de/
  2. 2. REAL-TIME ANALYTICS?
  3. 3. 3 © WidasConcepts Real-Time Analytics?
  4. 4. 4 © WidasConcepts Real-Time Analytics mit Spark und Cassandra
  5. 5. CASSANDRA
  6. 6. 6 © WidasConcepts Cassandra Vorteile Cassandra: massiv skalierbare verteilte Datenbank CAP frei einstellbar, für Analytics: AP Shared Nothing, Peer to Peer KKV – Wide Columnar/Partitions Zeitreihen optimierte Datenmodelle In Memory Tabellen Daten-Lokalität mit Wide Partitions 1 2 3 4 5 6
  7. 7. 7 © WidasConcepts Cassandra – Ring Struktur Jeder Knoten in Cassandra ist äquivalent ansprechbar Konfigurierbare Replikation (lokal, DC-weise) 1 2 34 5 Client
  8. 8. 8 © WidasConcepts   „Can‘t Fail, Must Scale“ –System   Datenreplikation sichert Verfügbarkeit   Knotenausfall wird automatisch behandelt Verfügbarkeit 1 2 34 5 Client
  9. 9. 9 © WidasConcepts Skalierbarkeit Cassandra im Vergleich Quelle: Planet Cassandra Anzahl Operationen (Read/Write) pro Sec – Anzahl Knoten
  10. 10. SPARK
  11. 11. 11 © WidasConcepts Spark Kernelemente Verarbeitung im DAG (Directed Acyclic Graph) Resilent Distributed Datasets Scala lokale JVM Prozesse auf den Knoten parallele Transformationen/Aktionen auf RDDs Operationen: map, filter, groupBy … Aktionen: count, collect, save … 1 2 3 4 5 6
  12. 12. 12 © WidasConcepts Spark - RDD Resilent Distributed Datasets immutable partitioned logical collection of records rebuildable materialized in memory cached for future reuse 1 2 3 4 5 6
  13. 13. 13 © WidasConcepts Spark – RDD – Transformationen und Aktionen (parallel) Transformationen map filter groupByKey join … Aktionen reduce collect count lookupKey …
  14. 14. 14 © WidasConcepts Spark – RDD – Resilent Resilent Distributed Datasets speichern ihre Herkunft (Lineage) damit kann jederzeit bei Ausfall die entsprechende Partition der RDD neu aufgebaut werden HdfsRDD path: hdfs://… FilteredRDD func: contains(...) MappedRDD func: split(…) CachedRDD
  15. 15. 15 © WidasConcepts Spark Performanz im Vergleich Logistic Regression 127  s  /  iteration   first  iteration  174  s   further  iterations  6  s   Quelle: University of California, Berkeley
  16. 16. 16 © WidasConcepts Spark Stack Spark   Spark   Streaming   real-­‐time         Verarbeitung   von  Daten  in   “micro”   Batches   Spark   SQL             HiveQL   kompatibel   MLLib   machine   learning         Classification   Clustering   Regressing   col.  Filtering     GraphX         spez.  RDDs   Operationen   PageRank   SVD++  
  17. 17. Integration Spark & Cassandra
  18. 18. 18 © WidasConcepts Integration – Spark & Cassandra mit Spark-Cassandra Connector Cassandra Tabellen sind als RDDs verfügbar auf jedem Cassandra Knoten wird ein Spark Executor eingesetzt 1 2 34 5
  19. 19. 19 © WidasConcepts Vorteile der Integration zwischen Spark und Cassandra Daten Lokalität, Token-Aware Spark RDDs auf In Memory C* Tabellen SQL auf Cassandra (Joins!) Datenbank-basierte Filter in Spark Spark Streaming wird unterstützt Beide Richtungen: Read and Write 1 2 3 4 5 6
  20. 20. Spark Streaming
  21. 21. 21 © WidasConcepts Spark Streaming Spark   Spark   Streaming   Batches   Datenstrom   Ergebnisse  
  22. 22. 22 © WidasConcepts Spark Streaming Integrierbar mit Cassandra/Spark Treiber Micro Batches (1 Sek), Discretized Streams Exactly Once Semantik RDD Funktionalität 1 2 3 4 Integration diverser MQ (z.B. Kafka)5
  23. 23. Umfeld
  24. 24. 24 © WidasConcepts Weiterentwicklungen im Spark Umfeld SparkR, PySpark Spark Integration in R lapply Implementierung kann in Closures in R verwendet werden 1 2 3 4 Interaktives R mit Spark möglich5 auf Daten in Cassandra6
  25. 25. In Zukunft ?
  26. 26. 26 © WidasConcepts Trennung wirklich „noch“ notwendig ?
  27. 27. 27 © WidasConcepts Ist das nicht ein zu „großer“ Toolstack?
  28. 28. 28 © WidasConcepts Vielleicht etwas einfacher?
  29. 29. 29 © WidasConcepts Die Lösung für Analytics? (mit etwas HDFS)
  30. 30. WidasConcepts HighEnd-Technology requires HighEnd-Competence (Wir beraten Sie gerne WidasConcepts GmbH Maybachstraße 2 71299 Wimsheim www.widas.de 30 Dieses Dokument wurde von WidasConcepts erstellt. Die Verteilung, Zitierung und Vervielfältigung – auch auszugsweise – zum Zwecke der Weitergabe an Dritte ist nur mit vorheriger schriftlicher Zustimmung von WidasConcepts gestattet. This presentation was created by WidasConcepts. Distribution, citation, copying - completely or in extracts – for transfer purposes, is only permitted with prior written agreement. These abstracts and graphics were deployed by WidasConcepts within the scope of a presentation. It is no complete documentation of this event. Thomas Mann, Solution Architect Telefon: +49 (7044) 95103 – 100 Mobile: +49 162 259 56 90 Mail: thomas.mann@widas.de

×