1.
BERLIN • NEW YORK • SAN FRANCISCO • SÃO PAULO • PARIS • LONDON • MOSCOW • ISTANBUL
SEOUL • SHANGHAI • BEIJING • TOKYO • MUMBAI • SINGAPORE
Scaling a platform for
real-time fraud detection

2.
2
Why is no one prepared for
success?

3.
3
‣ Focus on idea
‣ Focus on execution is limited in time to market
‣ Underestimating importance of scalability and profitability
Startup Zeitgeist

4.
4
‣ Director of Engineering
‣ Consulted with early design decisions
‣ Joined Adjust 2012 as Head of IT Operations
Who am I?

5.
Who is Adjust?
1B+
Daily active users tracked
400+ Billion
Data points tracked monthly
22K+
Apps tracked

6.
6
‣ Automatically reject fraudulent data before it gets paid
‣ Sends rejection reason callbacks to all parties
‣ Customers and their networks have full transparency
‣ Real-time statistical analysis of all ad engagements and app activity
Fraud Prevention Suite

7.
7
How did we do that?

8.
8
Bootstrapping a product
Current approach
‣ Using cloud services
‣ Base product on 3rd party IP
‣ Faster development

9.
9
Bootstrapping a product
Current approach Our approach
‣ Using cloud services
‣ Base product on 3rd party IP
‣ Faster development
‣ Own infrastructure
‣ Develop own IP
‣ Slower ramp up

10.
10
Bootstrapping a product
Current approach Our approach
‣ Using cloud services
‣ Base product on 3rd party IP
‣ Faster development
‣ Potential lock in
‣ More cost efficient in the beginning
‣ Shared environment
‣ Own infrastructure
‣ Develop own IP
‣ Slower ramp up

11.
11
Bootstrapping a product
Current approach Our approach
‣ Using cloud services
‣ Base product on 3rd party IP
‣ Faster development
‣ Potential lock in
‣ More cost efficient in the beginning
‣ Shared environment
‣ Own infrastructure
‣ Develop own IP
‣ Slower ramp up
‣ Keep flexibility
‣ More cost efficient in the long run
‣ Dedicated environment

12.
12
The machine room

13.
13
General Overview
Realtime Callbacks
Raw Data Upload
Realtime Fraud
Prevention
Realtime
Aggregation
SDK Traffic Tracking Server Device Data
Store

14.
14
So it begins…

15.
15
Redis Small
haproxy
Redis Leader
Redis Replica

16.
16
A little more please?

17.
17
Redis Big
haproxy
Redis Leader
Redis Replica

18.
18
We need to go further

19.
19
It’s all connected :)
Leader 1 Replica 1 Leader n Replica n
Sentinel 1 Sentinel n
twemproxy twemproxy

20.
20
And we felt like…

21.
21

22.
22
‣ Automation is tricky
Smoke in the machine room

23.
23
‣ Automation is tricky
‣ Weird latency spikes
Smoke in the machine room

24.
24
‣ Automation is tricky
‣ Weird latency spikes
‣ Increased average response time
Smoke in the machine room

25.
25
‣ Automation is tricky
‣ Weird latency spikes
‣ Increased average response time
‣ Long failover times
‣ Disruptive failovers
Smoke in the machine room

26.
26
And we felt like…

27.
27

28.
28
Sometimes you have to let go…

29.
29
‣ Only index is in RAM
‣ Data is on SSD
‣ Cost reduced by 85%
‣ Server count reduced from 40 to 6
Migrating to Aerospike

30.
30
‣ 1,1 million reads per second
‣ 500.000 writes per second
‣ 180TB of data
‣ 3 locations
Using Aerospike

31.
31
What if?

32.
32
‣ Easy to scale
‣ Redis interface
‣ Online resizing
‣ All dirty work is done by Amazon
Elasticache

33.
33
Q: Can I use Amazon ElastiCache for use cases other
than caching?
A: Yes. ElastiCache for Redis can be used as a primary
in-memory key-value data store, providing fast, sub
millisecond data performance, high availability and
scalability up to 15 nodes plus up to 5 read replicas,
each of up to 9.5 TiB of in-memory data.

34.
34
“Once we are successful, we will
take care of scalability and
profitability”

35.
35
What will happen to my
infrastructure when I will be
successful?

36.
36
Questions and answers

37.
New York
Paris
São Paulo
San Francisco
London Berlin
Istanbul
Moscow
Mumbai
Beijing
Seoul
Tokyo
Shanghai
Singapore
Robert Abraham
DIRECTOR OF ENGINEERING
robert@adjust.com
ADJUST HQ
Saarbrücker Str. 37a
10405 Berlin
Germany