O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Queryable State for Kafka Streamsを使ってみた

1.104 visualizações

Publicada em

Apache Kafka Meetup Japan #2 https://kafka-apache-jp.connpass.com/event/45923/ 発表資料

Publicada em: Tecnologia

Queryable State for Kafka Streamsを使ってみた

  1. 1. Queryable State for Kafka Streams / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka Streams • State for Kafka Streams • Queryable State for Kafka Streams • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (DEIM 2014) • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) 3
  4. 4. Apache Storm PipelineDB Druid Apache Kafka 4
  5. 5. Apache Storm Memcached Redis Apache Kafka 5
  6. 6. Apache Kafka 6
  7. 7. Kafka 7
  8. 8. Apache Kafka Version 0.10.1.0 • New Features • [KAFKA-1464] - Add a throttling option to the Kafka replication tool • [KAFKA-3176] - Allow console consumer to consume from particular partitions when new consumer is used. • [KAFKA-3492] - support quota based on authenticated user name • [KAFKA-3776] - Unify store and downstream caching in streams • [KAFKA-3858] - Add functions to print stream topologies • [KAFKA-3909] - Queryable state for Kafka Streams • [KAFKA-4015] - Change cleanup.policy config to accept a list of valid policies • [KAFKA-4093] - Cluster id 8
  9. 9. Kafka Streams • Version 0.10.0.0 • Kafka API 9
  10. 10. Kafka Streams Stateless transformations Stateful transformations 10
  11. 11. State for Kafka Streams • Stateful transformations • 11
  12. 12. State • Kafka Streams • 2 • State Store RocksDB • Kafka StateChangelog • Kafka Streams • Key Value • Window 12
  13. 13. • Record Stream • Changelog Stream • State (Table) Record Changelog State 13
  14. 14. Queryable State for Kafka Streams • Version 0.10.1.0 • State Store API 14
  15. 15. Queryable State • ReadOnlyKeyValueStore • Window Streams State • Key • Value • ReadOnlyWindowStore • Window Streams State • Key, From (Long), To (Long) • Iterator<KeyValue<Long, Value>> 15
  16. 16. Kafka Streams • Stateful transformations count KafkaStreams streams = null; KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> textLines = builder.stream(stringSerde, stringSerde, inputStream); KStream<String, Long> wordCounts = textLines .flatMapValues(value -> ...) .groupBy((key, word) -> word) .count(storeName) .toStream(); wordCounts.to(stringSerde, longSerde, outputStream); streams = new KafkaStreams(builder, conf); 17
  17. 17. Queryable State Streams • Streams o.a.k.streams.KafkaStreams KafkaStreams streams = ...; String key = "kafka"; ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; 18
  18. 18. Queryable State Streams • API KafkaStreams streams = ...; get("/results/:key", (req, res) -> { String key = req.params(":key"); ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; }); 19
  19. 19. github:laclefyoshi/kafka_streams_example
  20. 20. Apache Kafka 21 HTTP
  21. 21. $ java -cp target/kafka_streams_example-1.0-SNAPSHOT.jar org.saekiyoshiyasu.App kafka -> null ... kafka -> 5 $ ./bin/kafka-console-producer.sh --topic input-stream --broker-list 127.0.0.1:9092 What is Kafka good for? ... 22
  22. 22. Queryable State for Kafka Streams • 
 Kafka Streams ? 23
  23. 23. Queryable State for Kafka Streams • 
 Kafka Streams ? State 24
  24. 24. • Queryable State • Kafka Streams 25

×