Realtime Data Analytics: system design challenges, specific technologies, and open source tools.

1. Realtime Data Analytics
Bo Yang

2. Realtime Data System Challenges
- System Design
- Data is big
- People want fresh data
- People want fast query
- Data Development
- Frequent and adhoc new requirements
- Many small data applications with slightly different data logic
- Data Operation
- Data could be delayed
- Application restart causes data loss or data duplication
- Need to do backfill

3. Data Delivery Guarantee
- Best Effort
- At Most Once
- At Least Once
- Most widely used
- Exactly Once
- Very Challenging doing it E2E
- High Cost
- Common Practice
- At Least Once - Steaming Processing
- Idempotent - Data Storage

4. SQL Everywhere
- Spark SQL
- Streaming SQL
- Siddhi CEP (Complex Event Processor) Engine
- Apache Calcite
- Schema
- Apache Avro
- Automate Data Processing
- Self-Service Data Onboarding

5. Memory: the More the Better
- Realtime data ingestion
- Realtime compute (streaming process)
- Instant query

6. Takeaways
- Think your realtime requirements
- Use Schema + SQL to increase development efficiency
- Use Memory to gain speed

7. Realtime Data Technologies
- Data Collection
- Data Computation
- Data Ingestion/Storage
- Data Query

8. Realtime Data Ingestion/Query - Challenges
- Continuous high throughput write
- Concurrent write and read (queries)
- Large table scan and aggregation
- Historical data + Recent data

9. Realtime OLAP Technologies
- Continuous high throughput write
- Concurrent write and read (queries)
- Large table scan and aggregation
- Historical data + Recent data
- Ingestion in memory
- Read-only segment
- Columnar Storage + Inverted Index
- Lambda Architecture

10. Realtime OLAP Tools
- Open Source Solutions
- Druid
- Widely used, mature
- Ingestion latency: milliseconds
- Pinot
- Mostly used by linkedin, more indexing
- Ingestion latency: milliseconds
- ELK
- Widely used, fast growing
- Ingestion latency: seconds
- Kylin (Streaming Cubing)
- Cubing technologies
- Ingestion latency: minutes

11. Takeaways
- Table Scan is Key
- Columnar Storage
- Inverted Index
- Data Ingestion Latency
- Balance of speed and cost
- Choose proper tool fitting your need