Kafka & Hadoop - for NYC Kafka Meetup
âąTransferir como PPTX, PDFâą
18 gostaramâą11,953 visualizaçÔes
How do you integrate Kafka & Hadoop, and how will it get better soon.
Recomendados
Recomendados
Mais conteĂșdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Kafka & Hadoop - for NYC Kafka Meetup
Semelhante a Kafka & Hadoop - for NYC Kafka Meetup (20)
Mais de Gwen (Chen) Shapira
Mais de Gwen (Chen) Shapira (20)
Ăltimo
Ăltimo (20)
Kafka & Hadoop - for NYC Kafka Meetup
- 1. Kafka & Hadoop Gwen Shapira / Software Engineer
- 2. About Me âą 15 years of moving data around âą Formerly consultant âą Now Cloudera Engineer: ©2014 Cloudera, Inc. All rights reserved. 2 â Flume â Sqoop â Kafka
- 3. Thereâs a book on that! ©2014 Cloudera, Inc. All rights reserved. 3
- 4. We are also blogging ©2014 Cloudera, Inc. All rights reserved. 4
- 5. 5 Getting Data from Kafka to Hadoop There are only bad options. It's about finding the best one. ©2014 Cloudera, Inc. All rights reserved.
- 6. ©2014 Cloudera, Inc. All rights reserved. 6 Camus
- 7. ©2014 Cloudera, Inc. All rights reserved. 7 Camus Setup ZooKeeper Topic Offsets Other Systems HDFS Processes Task Task Task In process Avro Files In process Avro Files Audit Counts Clean Up Kakfa B A C D F G H I E
- 8. Missing in Action âą Kafka has no MR layer â InputFormat, OutputFormat, Utils⊠⹠Sqoop is a generic batch ingest framework ©2014 Cloudera, Inc. All rights reserved. 8 â Why no Kafka?
- 9. Flume + Kafka = Flafka ©2014 Cloudera, Inc. All rights reserved. 9
- 10. 10 How does work? Sources Interceptors Selectors Channels Sinks Flume Agent Twitter, logs, webserver, Kafka⊠Mask, re-format, validate⊠DR, critical Memory, file HDFS, Hbase, Solr, Kafka
- 11. 11 But I just want to get data from Kafka to Hbase / HDFS ©2014 Cloudera, Inc. All rights reserved.
- 12. 12 Channels Sinks Kafka Channel Flume Agent Kafka! HDFS, Hbase, Solr
- 13. SparkStreaming Single Pass ©2014 Cloudera, Inc. All rights reserved. 13 Source RawInput DStream RDD Source RawInput DStream RDD RDD Filter Count Print Source RawInput DStream RDD RDD RDD Single Pass Filter Count Print Pre-first Batch First Batch Second Batch
- 14. ©2014 Cloudera, Inc. All rights reserved. 14 Storm Spout Source Split words bolts Split words bolts Spout Split words bolts Split words bolts Count Count Count Spout Layer Fan out Layer 1 Shuffle Layer 2
- 15. Retro Thoughts ©2014 Cloudera, Inc. All rights reserved. 15
- 16. âą Data often has schema âą At least it should âą Kafka is unaware â which is good âą Need capability to figure out schema for events âą Without including it in every event ©2014 Cloudera, Inc. All rights reserved. 16 Schema
- 17. Kafka in Cloudera Manager ©2014 Cloudera, Inc. All rights reserved. 17
- 18. 18 Visit us at Booth #305 BOOK SIGNINGS THEATER SESSIONS TECHNICAL DEMOS GIVEAWAYS
Notas do Editor
- This gives me a lot of perspective regarding the use of Hadoop
- https://gist.github.com/gwenshap/9699072
- Batch MapReduce job. Exactly once semantics. Run once every X minutes.
- A - The setup stage fetches broker urls and topic information from ZooKeeper. B - The setup stage persists information about topics and offsets in HDFS for the tasks to read. C - The tasks read the persisted information from the setup stage. D - The tasks get events from Kakfa. E - The tasks write data to a temp location in HDFS in the format defined by the user defined decoder, in this case Avro formatted files. F - The tasks move the data in the temp location to a final location when the task is cleaning up. G - The task writes out audit counts on its activities. H - A clean up stage reads all the audit counts from all the tasks. I - The clean up stage reports back to Kakfa what has been persisted.
- Kafka source + sink for Flume
- Does not require programming.
- Does not require programming.
- MicroBatch stream processing framework. Basically Spark code executed in a slightly different context and some windowing functions.
- Stream processing framework. Quite popular. Can be event-based or micro-batching (with Trident). Requires low level awareness of API.