Large Scale Graph Processing & Machine Learning Algorithms for Payment Fraud Prevention: PayPal is at the forefront of applying large scale graph processing and machine learning algorithms to keep fraudsters at bay. In this talk, I’ll present how advanced graph processing and machine learning algorithms such as Deep Learning and Gradient Boosting are applied at PayPal for fraud prevention. I’ll elaborate on specific challenges in applying large scale graph processing & machine technique to payment fraud prevention. I’ll explain how we employ sophisticated machine learning tools – open source and in-house developed. I will also present results from experiments conducted on a very large graph data set containing millions of edges and vertices.

1. Large Scale Graph Processing & Machine
Learning for Fraud Prevention
1
Disclaimer: views expressed here are my own and do not
necessarily represent the views of PayPal, its affiliates or
subsidiaries.

2. © 2016 PayPal Inc. Confidential and proprietary.
Agenda
• Graph Processing
• Machine Learning
2

3. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing - Motivation
• (Big) Data is not flat
• Data is multi-relational, spatial, temporal,
multi-modal
• Machine learning algorithms assume i.i.d
(independent & identically distributed)
• Need Data Science for Graphs
3
Software
(Java/JVM)
Source: Lise Getoor, UCSC

4. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing - Motivation
• Much harder to get performance lift on
our flagship models
• Need to re-look at all aspects of
traditional model building
• Need out-of-the-box thinking
4
Area we are missing (AUC 0.98)

5. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing With Giraph (Current)
• Pregel (Google’2010)
• Vertex Centric
• Giraph (Apache Open Source)
5

6. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing With Giraph (Current)
• Used to compute 2-step neighbhors – IP,VID
• Data
• 5 events (login, logout, transaction, fso, signup)
• Stampy HDFS
• 1 year
• Workflow
• Series of (10) map-reduce jobs to join/pre-process
construct graph
• Neighorhood computation uses Giraph
• Output pushed out online as RADD
• Issues
• 2 days for end-to-end process
• Lots of intermediate read/write
• Hadoop cluster stability
• Unsupported by Hortonworks
6
Hadoop HDFS
(Stampy)
RADD
M/R Jobs
M/R Jobs
Giraph Jobs
Giraph Job
Online
Offline

7. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing (Future)
• Leverage graph representation to understand, analyze, visualize
relationships – their strengths, direction & influence and to
predict future relationships and interactions between entities
(consumers, merchant, IP address, device IDs, nodes in a
network…).
• Algorithmic convergence of graph theory & machine learning
(statistical relations learning & probabilistic graph modeling)
• Multi-layered full fledged specialized stack, hardware to
software.
7
GPU, FPGA, TPU., SSD
(specialized hardware)
Graph DB
(specialized software)
Graph Algorithms
(Deterministic &
Probabilistic)
Graph
Analytics &
Visualization
Graph
Modeling
(Prediction)

8. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing (Future)
• Solve new problems that traditional approaches can not solve:
• What is the likelihood that particular consumer (node in graph) colluding
(hidden relationship in graph) with a particular merchant (node in graph)?
• How fraudsters (series of nodes & edges) move money through our network
i.e., anti-money laundering?
• If a consumer (node) is buying (relation) product X (node) from merchant
(node), what else can we recommend to the consumer?
• If a server (node) in a data center goes down/unhealthy, what would would be
the impact to other critical components?
• Who (what) are the central influential people (assets) in our network?
• Graph theory & algorithms (link analysis, page rank, belief propagation, markov
random field) can help to uncover influencing nodes & hidden relationships.
• Graph databases are much more efficient in unearthing graph properties, have flexible
schema that is amenable to changing data, scale naturally to large data sets as they do
not require expensive join operations.
8

9. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing POC
• POC with new stack
• Single powerful machine in-lieu of 1200+ node
stampy cluster
• 160 core
• State-of-the-art NVidia Pascal GPU (4 P100)
• 1 TB RAM
• 2 TB SSD
• Graph DB optimized for scale-up architecture (Neo4j)
• Graph computing frameworks optimized for single
server/GPUs
9
Hadoop HDFS
(Stampy)
RADD
Graph DB
(Neo4J)
Online
Offline
Hardware
P100 GPU; SSD
Graph
Algorithms
(Neo4J)
Graph
Algorithms
(GunRock)

10. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing POC
• Graph
• 1.5 billion edges; 0.5 billion vertices
• Static graph (1 year); No time dimension
• Vertices - Customer ID, Merchant ID, IP;
Edges – payment transaction
10
Degree 2^0: 58104040 (58.10%)
Degree 2^1: 23034683 (23.03%)
Degree 2^2: 11711920 (11.71%)
Degree 2^3: 5098768 (5.10%)
Degree 2^4: 1575787 (1.58%)
Degree 2^5: 366774 (0.37%)
Degree 2^6: 79654 (0.08%)
Degree 2^7: 19796 (0.02%)
Degree 2^8: 5861 (0.01%)
Degree 2^9: 1886 (0.00%)
Degree 2^10: 581 (0.00%)
Degree 2^11: 179 (0.00%)
Degree 2^12: 52 (0.00%)
Degree 2^13: 12 (0.00%)
Degree 2^14: 4 (0.00%)
Degree 2^15: 1 (0.00%)
Degree Distribution

11. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing POC
• GraphAlgorithms
• Page Rank
• Connected Components
• find sets of connected
nodes
• Each node is reachable
from any other node in the
set
• Community Detection
• Label Propagation – Group
nodes with similar
properties
11

12. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing POC
• Results
• Neo4j
• GunRock, NvGraph, SNAP – in
progress
12
Algorithm Compute
Time (ms)
Write Time
(ms)
Page Rank 77 124
Connected
Components
156 337
Community
Detection
384 361

13. © 2016 PayPal Inc. Confidential and proprietary.
Graph Processing - Key Learnings
• Scale-up showing promise
• 1 year graph ~70GB
• Can fit entire graph in 1 machine’s memory
• Significant reduction in processing time
• ~2 days -> ~2 hours
• Compute ~77 milliseconds
• Significant value in Graph DB – sub-graph creation,
interactive exploration, visualization, savings in
computation
13
Source: Lise Getoor, UCSC

14. Machine Learning - Current Research Workflow
Hardware
-
Research,
Procure,
Provision
Modeling
Algorithm
Research
Tool -
Evaluate,
Benchmar
k
Data &
Variable
Collection
and
Selection
Model
Training -
Developm
ent,
Performan
ce Eval,
Spec
(offline)
Scoring
Engine
Developm
ent
(online)
Deployme
nt - One
Box &
Load test
Auditing -
Variable &
Score
Go Live
©2015 PayPal Inc. Confidential and proprietary. 14
• Manual Development (Spec -> Code)
• ML Tool Specific Software Stack
• Lack of research infrastructure/self-service tools
• Serial pipeline; No early involvement of relevant teams

15. © 2016 PayPal Inc. Confidential and proprietary.
Machine Learning - Future Modeling/Prediction Stack
• Algorithm
• Deep Learning
• Offline to Online Learning
• Human to Auto Feature Engineering
• Model Spec
• Language Agnostic Platform agnostic Spec
(ex: protobufs)
• Tools
• Scalable & Flexible (ex: TensorFlow)
• Software
• Language Agnostic Software Stack
• Hardware
• Accelerators (GPUs, TPUs,..)
15
SS
Hardware
CPU + GPU/TPU
Model Spec
(protobufs)
Tools
(TensorFlow)
Algorithm
(Deep Learning)
Software
(Language Agnostic)

16. © 2016 PayPal Inc. Confidential and proprietary.
Conclusions
• Graph algorithms & Deep Learning shows promising
analytical results
• Scale-up architecture shows promise for our use cases
• Probabilistic Graph Modeling + Deep Learning =>
Interpretable Models
• Scratched surface; lot left to accomplish
16
Software
(Java/JVM)
Source: Lise Getoor, UCSC