This document describes the evolution of AppsFlyer's architecture for processing big data analytics from real-time and batch data. The initial architecture struggled to keep up with increasing event volumes. It was redesigned with a functional programming approach using Clojure to break the system into small isolated services that communicate asynchronously through a centralized event stream. The new architecture utilizes best practices like infrastructure automation, service discovery, and monitoring to scale efficiently.

1. Architecture for
Real-Time and Batch
Big-Data Analytics

2. The World's leading Mobile
Measurement Platform
Founded in 2011 by Reshef Mann
and Oren Kaniel
Just completed Round B funding -
total of $28M
Processing 3.5B daily events
(it was 1.9B just 2 months ago and
250M at the start of 2014!)
13 people in the development team
(we were just 6 people 12 months ago!)
AppsFlyer
Who?!

3. The Way We Were
• We had no real concept of “Big Data”- we were
just occupied with making the system work
• Even though we were ignorant of the future, we
tried to adhere to a few abstractions:
- Small isolated services
- Central concept of message delivery via
- Different tech for different tasks
A message-bus

4. • CouchDB that served raw reports
via views couldn't keep up with view
generation
• Python processes that read from the
message bus (via pub sub) couldn't
keep up with the amount of data
• The split between aggregated and
raw data was good, but caused
discrepancies because each service
failed at a different time
• If the message bus failed (Redis), all
other services were also in a fail state –
single point of failure
First Creaks
In The
System

5. Part 1 of The Solution
• Migrate raw reports from CouchDB to Google's
Bigquery (easiest solution at the time)
• Rewrite some of the Python services in a new
language that:
– Deals better with strings and allocation of memory
– Can help us scale out
– Has a great ecosystem
– Functional

6. WHY FUNCTIONAL?!

7. Why Functional?

8. Why Clojure?
Sequence based processing
capabilities really fit in the
visualized data flow of
AppsFlyer (processing the
Event Stream)
Enforces use of
FP paradigm more
strictly than Scala
Repl based
development
Easy and
common Java
interop
JVM!

9. • Python's proprietary serialization
• Python custom data structure as the
base message in the system
The Hurdles
(or how 2 stupid mistakes can bite
you in the ass 2 years down the road)

10. Part 2
of The Solution

11. The Event
Stream

12. Small isolated services
Each service has a single business
responsibility
Each service encapsulates its own data
(if it has any) and he exposes it over a
well known interface
Data objects are always POCO/POJO
(simple data structures represented in
JSON or EDN)
Preference for queues and buffers
that pass isolated data for total async
processing
How We
Model

13. How We
Test
No QA team
Each new service can read from the
event stream to its heart content -
regular Kafka consumers behavior
Each new service handles real life
traffic and real life load because it's
connected to the event stream
Test DB if needed is easy to spin up on
the cloud
Once deemed ready, just throw the
switch to on

14. Service discovery
via Consul
How We
Orchestrate
MesosEach service is a
single uberjar in
a single docker
container
DBs get their
own dedicated
machines
Preference for ring
based architecture
for DBs
Marathon

15. Jenkins build server
Deployment via the in-house
Santa tool
Consul health checks
Statsd for the JVM and
application metrics
Sensu
End-to-end flows
Deployment
and
Monitoring

16. The Bird's Eye View