O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Lossless Stream Processing

유실없는 스트림 처리

Lossless Stream Processing

  1. 1. 7FWF R F W HW (). (, 0
  2. 2. http://readme.skplanet.com/?p=12465
  3. 3. https://voltdb.com/sites/default/files/real-time-data-report.pdf 2015 Real-time Data Report by VoltDB b 효과 같은 a b 능력 다른 a
  4. 4. b A FP WN b f4 r w 4 마감 시간(Deadline) ) y4 (( y4 https://www.soasta.com/blog/23-stats-mobile-web-performance-monitoring/ https://en.wikipedia.org/wiki/Real-time_computing b ? F A FP WN 4 F I N BS W
  5. 5. 4 4 BW F 6FWHM Unbound Bound Data Data X X v https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
  6. 6. >NH S GFWHM Unbound Data h e 작은 배치 처리 끊임없이
  7. 7. 5B5 d 실시간으로 l l ” 목표 응답 시간 b s 고객을 잃기 전까지 실시간 1 목표 스트림 1 방식 http://bigdatapage.com/4-really-real-meanings-of-real-time/
  8. 8. https://tsicilian.wordpress.com/2015/02/16/streaming-big-data-storm-spark-and-samza/ BTF n m A8 = A FI 8YFP NRW =SST m E5A? e BHFPF Exactly-onceb 목표 10초 이내
  9. 9. 사용 4 좋아 4 https://twitter.com/jaykreps/status/568499167686995969 https://twitter.com/miguno/status/568503505113247744 Storm vs. Spark Streaming 좋아 4 사용 4 w x
  10. 10. https://databaseline.wordpress.com/2016/03/12/an-overview-of-apache-streaming-technologies/ s o (). (
  11. 11. 분산 g w 장애 ! m t u l
  12. 12. 분산 장애 부분 전송 성공 At-most-once 1 At-least-once 1 Exactly-once 1
  13. 13. 8 FHWP SRH ) 8 FHWP SRH https://twitter.com/mathiasverraes/status/632260618599403520 8 FHWP SRH M F SRP W S MF I T SGP NR IN W NG W I W 1 8 FHWP SRH I PNY ) : F FRW I S I S FL 8 FHWP SRH I PNY
  14. 14. 8 FHWP SRH m t u l Idempotent exactly-once 1 in d 2 2 5W P F W SRH 8 FHWP SRH Transactional exactly-once 1 “ SPP GFH “ 76 s
  15. 15. YARN HDFS Rake Server Query Cache AppServer SQL thin JDBC client Apache Kafka Rake Library Rake API Logs LogAgent HTTPS Phoenix JDBC client (select) JDBC client (upsert) Kafka Consumer Kafka Producer Collector 6=8
  16. 16. 6=8 ) - g , g
  17. 17. YARN HDFS Web Server HQL Apache Kafka Kafka Consumer Kafka Producer Kafka Producer Collector YARN Hive Tez )) ⇔
  18. 18. )) ⇔ D G ) - g >SGNP / g D G ..((g >SGNP .,((g )) D3>( >3D Web Mobile
  19. 19. 수신, 처리, 저장 k 부분 8 FHWP SRH 전체 8 FHWP SRH 데이터 원천 비즈니스 행동 i t 목표 응답 시간 내 반응 실시간 시스템 구축 s c 데이터 파이프라인 배치 처리 pb l 스트림 처리 p
  20. 20. Appendix
  21. 21. https://lobste.rs/s/ecjfcm/why_is_exactly-once_messaging_not_possible_in_a_distributed_queue/ 8 FHWP SRH delivery N impossible. 8 FHWP SRH processing of messages N possible N WM T SH NRL HFR G FI idempotent. 8 FHWP SRH 8 FHWP SRH NI TSW RW 8 FHWP SRH
  22. 22. http://readme.skplanet.com/?p=10170

×