Aleksei Udatšnõi – Crunching thousands of events per second in nearly real time Imagine you have a product which generates up to 10 thousands events per second or around 1 billion events per day. This live stream of data need to be tracked, processed and presented to end-users in a visually appealing way. The solution needs to be integrated into a traditional web application. That is the real use case at Softonic. In this talk we will show how it was solved in Softonic. We use the stack of technologies around Big Data to process and store live stream of data and present the results to users in nearly real time. This real-life solution is built around Hadoop ecosystem and it includes Flume, Hive, Oozie and Impala. We will show how to store and query such volumes of data using NoSQL database and how to build a scalable end-user web application using nearly real time data feed.

1. Crunching thousands of events per second in nearly real time
Aleksei Udatšnõi
Lead Software Engineer @ Softonic
NoSQL matters Barcelona
22 November 2014
Photo by Chris Loxton/ CC BY

2. Softonic use case

3. Softonic use case
•Software guide, reviews, news, downloads
•Originally built on LAMP stack
•100M+ visitors monthly
•1000+ tracking events/sec generated

4. Big Data challenges
•Track large volume of events
•Process the stream of events in near real time
•Present insights to stakeholders

5. Softonic Developer Center

6. Softonic Developer Center
developer.softonic.com

7. Legacy RDMBS based solution
INSERT INTO tracking VALUES ..

8. Big Data architecture

9. Data ingestion with Flume
Photo by BilfingerSE/ CC BY

10. Data ingestion with Flume
•Tracking events in real time
•Accumulating into larger chunks
•Ingesting into HDFS

11. Data ingestion with Flume

12. Flume considerations
•Flume vs. alternatives (Kafka, Storm)

13. Flume considerations
•Flume vs. alternatives (Kafka, Storm)
•Prepare failover plan

14. Flume considerations
•Flume vs. alternatives (Kafka, Storm)
•Prepare failover plan
•Configuring rolling interval

15. Data processing
Photo by bonzoWiltsUK/ CC BY

16. Data processing
•Extracting KPI from unstructured data
•Aggregating events by date, app, customer

17. Data processing
From raw data
{“id”:1234, “country”: “AR”, “date:” : “20141113”} .. {}
{“id”:1234, “country”: “AR”, “date:” : “20141113”} .. {}
{“id”:1234, “country”: “ES”, “date:” : “20141113”} .. {}
To aggregated
1234
AR
112
1234
ES
56
1234
MX
40

18. Real-time querying
Pre-requisites
•Data is read from HDFS
•Advanced filtering
•Minimal latency
•Integrated into existing ecosystem

19. Real-time querying
Options considered
•Exporting aggregated data to MongoDB
•Apache Spark
•Presto
•Cloudera Impala

20. Impala
Photo by Colin J. McMechan/ CC BY

21. Impala
•Engine which enables real-time queries in Apache Hadoop
•Based on Dremelpaper of Google
•Open source
•Integrated into CDH
•Outperforms alternatives solutions (Shark, Presto)

22. Impala considerations
•Sharing resources with MapReducejobs

23. Impala considerations
•Sharing resources with MapReducejobs
•Consistent data format is required

24. Impala considerations
•Sharing resources with MapReducejobs
•Consistent data format is required
•Invalidating metadata

25. Putting all together

26. Putting all together

27. Conclusions
•Data ingested by applications in real-time
•Ingesting 1000+ events/sec
•Data delivered to HDFS every 30 minutes
•Average query response time < 1 sec

28. References
https://developer.softonic.com/
http://flume.apache.org/
http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-1-x/Impala/impala.html
http://research.google.com/pubs/pub36632.htmlhttp://blog.cloudera.com/blog/2014/05/new-sql-choices-in-the-apache-hadoop-ecosystem-why-impala-continues-to-lead/

29. Thank you!
@udachny