O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Big Data LDN 2018: SCALING A PLATFORM FOR REAL-TIME FRAUD DETECTION WITHOUT BREAKING THE BANK

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Carregando em…3
×

Confira estes a seguir

1 de 37 Anúncio

Big Data LDN 2018: SCALING A PLATFORM FOR REAL-TIME FRAUD DETECTION WITHOUT BREAKING THE BANK

Date: 13th November 2018
Location: Fast Data Theatre
Time: 14:30 - 15:00
Speaker: Robert Abraham
Organisation: Adjust
About: Adjust's business is verifying billions of advertising impressions, policing any fraudulent activity at scale. They maintain all of the data on their clients’ target user devices and their actions - unlike their competitors which discard data after 90 days.

Setting up a global, clustering system with transcontinental fibre to manage 100TB of data was not trivial.

Data correctness and low latency were important criteria.

In this session, learn:

• Why they decided to move from NoSQL incumbent Redis to Aerospike

• Why bare metal works so much better than the cloud

• How they’ve scaled while still policing fraud at speed

• Their architectural view to manage the data volume, upgrades, backups and interruptions

Date: 13th November 2018
Location: Fast Data Theatre
Time: 14:30 - 15:00
Speaker: Robert Abraham
Organisation: Adjust
About: Adjust's business is verifying billions of advertising impressions, policing any fraudulent activity at scale. They maintain all of the data on their clients’ target user devices and their actions - unlike their competitors which discard data after 90 days.

Setting up a global, clustering system with transcontinental fibre to manage 100TB of data was not trivial.

Data correctness and low latency were important criteria.

In this session, learn:

• Why they decided to move from NoSQL incumbent Redis to Aerospike

• Why bare metal works so much better than the cloud

• How they’ve scaled while still policing fraud at speed

• Their architectural view to manage the data volume, upgrades, backups and interruptions

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (14)

Semelhante a Big Data LDN 2018: SCALING A PLATFORM FOR REAL-TIME FRAUD DETECTION WITHOUT BREAKING THE BANK (20)

Anúncio

Mais de Matt Stubbs (20)

Mais recentes (20)

Anúncio

Big Data LDN 2018: SCALING A PLATFORM FOR REAL-TIME FRAUD DETECTION WITHOUT BREAKING THE BANK

  1. 1. BERLIN • NEW YORK • SAN FRANCISCO • SÃO PAULO • PARIS • LONDON • MOSCOW • ISTANBUL SEOUL • SHANGHAI • BEIJING • TOKYO • MUMBAI • SINGAPORE Scaling a platform for real-time fraud detection
  2. 2. 2 Why is no one prepared for success?
  3. 3. 3 ‣ Focus on idea ‣ Focus on execution is limited in time to market ‣ Underestimating importance of scalability and profitability Startup Zeitgeist
  4. 4. 4 ‣ Director of Engineering ‣ Consulted with early design decisions ‣ Joined Adjust 2012 as Head of IT Operations Who am I?
  5. 5. Who is Adjust? 1B+ Daily active users tracked 400+ Billion Data points tracked monthly 22K+ Apps tracked
  6. 6. 6 ‣ Automatically reject fraudulent data before it gets paid ‣ Sends rejection reason callbacks to all parties ‣ Customers and their networks have full transparency ‣ Real-time statistical analysis of all ad engagements and app activity Fraud Prevention Suite
  7. 7. 7 How did we do that?
  8. 8. 8 Bootstrapping a product Current approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development
  9. 9. 9 Bootstrapping a product Current approach Our approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development ‣ Own infrastructure ‣ Develop own IP ‣ Slower ramp up
  10. 10. 10 Bootstrapping a product Current approach Our approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development ‣ Potential lock in ‣ More cost efficient in the beginning ‣ Shared environment ‣ Own infrastructure ‣ Develop own IP ‣ Slower ramp up
  11. 11. 11 Bootstrapping a product Current approach Our approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development ‣ Potential lock in ‣ More cost efficient in the beginning ‣ Shared environment ‣ Own infrastructure ‣ Develop own IP ‣ Slower ramp up ‣ Keep flexibility ‣ More cost efficient in the long run ‣ Dedicated environment
  12. 12. 12 The machine room
  13. 13. 13 General Overview Realtime Callbacks Raw Data Upload Realtime Fraud Prevention Realtime Aggregation SDK Traffic Tracking Server Device Data Store
  14. 14. 14 So it begins…
  15. 15. 15 Redis Small haproxy Redis Leader Redis Replica
  16. 16. 16 A little more please?
  17. 17. 17 Redis Big haproxy Redis Leader Redis Replica
  18. 18. 18 We need to go further
  19. 19. 19 It’s all connected :) Leader 1 Replica 1 Leader n Replica n Sentinel 1 Sentinel n twemproxy twemproxy
  20. 20. 20 And we felt like…
  21. 21. 21
  22. 22. 22 ‣ Automation is tricky Smoke in the machine room
  23. 23. 23 ‣ Automation is tricky ‣ Weird latency spikes Smoke in the machine room
  24. 24. 24 ‣ Automation is tricky ‣ Weird latency spikes ‣ Increased average response time Smoke in the machine room
  25. 25. 25 ‣ Automation is tricky ‣ Weird latency spikes ‣ Increased average response time ‣ Long failover times ‣ Disruptive failovers Smoke in the machine room
  26. 26. 26 And we felt like…
  27. 27. 27
  28. 28. 28 Sometimes you have to let go…
  29. 29. 29 ‣ Only index is in RAM ‣ Data is on SSD ‣ Cost reduced by 85% ‣ Server count reduced from 40 to 6 Migrating to Aerospike
  30. 30. 30 ‣ 1,1 million reads per second ‣ 500.000 writes per second ‣ 180TB of data ‣ 3 locations Using Aerospike
  31. 31. 31 What if?
  32. 32. 32 ‣ Easy to scale ‣ Redis interface ‣ Online resizing ‣ All dirty work is done by Amazon Elasticache
  33. 33. 33 Q: Can I use Amazon ElastiCache for use cases other than caching? A: Yes. ElastiCache for Redis can be used as a primary in-memory key-value data store, providing fast, sub millisecond data performance, high availability and scalability up to 15 nodes plus up to 5 read replicas, each of up to 9.5 TiB of in-memory data.
  34. 34. 34 “Once we are successful, we will take care of scalability and profitability”
  35. 35. 35 What will happen to my infrastructure when I will be successful?
  36. 36. 36 Questions and answers
  37. 37. New York Paris São Paulo San Francisco London Berlin Istanbul Moscow Mumbai Beijing Seoul Tokyo Shanghai Singapore Robert Abraham DIRECTOR OF ENGINEERING
 robert@adjust.com ADJUST HQ
 Saarbrücker Str. 37a
 10405 Berlin
 Germany

×