GraphX and Pregel - Apache Spark

Spark GraphX & Pregel
Challenges and Best Practices
Ashutosh Trivedi (IIIT Bangalore)
Kaushik Ranjan (IIIT Bangalore)
Sigmoid-Meetup Bangalore
https://github.com/anantasty/SparkAlgorithms

Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
Agenda
•Introduction to GraphX
– How to describe a graph
– RDDs to store Graph
– Algorithms available
•Application in graph algorithms
– Feedback Vertex Set of a Graph
– Identifying parallel parts of the solution.
•Challenges we faced
•Best practices
2

Bangalore Dec-2014
33
GraphX - Representation

Bangalore Dec-2014
Graph Representation
4
class Graph [ V, E ] {
def Graph(vertices: Table[ (Id, V) ],
edges: Table[ (Id, Id, E) ])
• The VertexRDD[A] extends RDD[(VertexID, A)] and adds the additional
constraint that each VertexID occurs only once.
• Moreover, VertexRDD[A] represents a set of vertices each with an
attribute of type A
• The EdgeRDD[ED], extends RDD[Edge[ED]]

Bangalore Dec-2014 5
GraphX - Representation

A BA
Vertex and Edges
Vertex Edge

Bangalore Dec-2014
Triplets Join Vertices and Edges
• The triplets operator joins vertices and edges:
TripletsVertices
B
A
C
D
Edges
A B
A C
B C
C D
A BA
B A C
B C
C D
7

Bangalore Dec-2014
88
Triplets elements

Subgraphs
Predicates vpred and epred

Feedback Vertex Set
• A feedback vertex set of a graph is a set of vertices
whose removal leaves a graph without cycles.
• Each feedback vertex set contains at least one vertex of
any cycle in the graph.
• The feedback vertex set problem is an NP-
complete problem in computational complexity theory
• Enumerate each simple cycle.

1 2
34
5
6
7
8
9
10
Strongly Connected Components
Each strongly connected component can be
considered in parallel since they do not share
any cycle
SC1 – (1) SC2 – (5) SC3 – (8) SC4 – (9)

FVS Algorithm
#Greedy recursive solution
FVS(G)
sccGraph = scc(G)
For each graph in sccGraph
For each vertex
remove vertex and again calculate scc,
vertex V = vertex which give max number of scc
#which means it kills maximum cycles
subGraph = subgraph(remove V )
FVS (subGraph )

1 2
4 3
2
4 3
Graph Iteration SCC count
3
1
4 3
1
1 2
4
3
1 2
4 3
1 2
4 3
Remove 2
Remove 1
Remove 3

1
5
8 9
1 5 8 9Feedback Vertex Set

FVS – Spark Implementation
sccGraph has one more property sccID on each vertices, extract it

sccGraph = scc(G)
For each graph in sccGraph

#Greedy recursive function

For each vertex
remove vertex and again calculate scc,
# Z is a list of scc count after removing each vertex

vertex V = vertex which give max number of scc
#which means it kills maximum cycles

subGraph = subgraph(remove V )
FVS (subGraph )

Pregel
• Graph DB
– Data Storage
– Data Mining
• Advantages
– Large-scale distributed computations
– Parallel-algorithms for graphs on multiple machines
– Fault tolerance and distributability

Oldest Follower
What is the age of oldest follower of each user ?
Val oldestFollowerAge = graph
.aggregateMessages(
#map word => (word.dst.id, word.src.age),
#reduce (a,b) => max(a, b)
)
.vertices
mapReduceTriplets is now aggregateMessages

In aggregateMessages :
• EdgeContext which exposes the triplet fields .
• functions to explicitly send messages to the source and
destination vertex.
• It require the user to indicate what fields in the triplet are
actually required.
New in GraphX

Bangalore Dec-2014
Theory – it’s Good
How it works – that’s awesome
24
Graph’s are recursive data-structures, where the
property of a vertex is dependent on the properties of
it’s neighbors, which in turn are dependent on the
properties of their neighbors.

Bangalore Dec-2014
Graph.Pregel ( initialMessage ) (
#message consumption
( vertexID, initialProperty, message ) → compute new property
,
#message generation
triplet → .. code ..
Iterator( vertexID, message )
Iterator.empty
,
#message aggregation
( existing message set, new message ) → NEW message set
)
25
Architecture

1 2
4 3
1030
30 20
1 2
4 3
10
30
30 20
max [30,10,20]
max [20] max [10]
1 2
4 3
100
10 10
1 2
4 3
10
0
10 10
max [10] max [10]

Example - output
1 2
4 3
100
0 0

Bangalore Dec-2014
Applications - GIS
• Algorithm – to compute all vertices in a directed graph, that can
reach out to a given vertex.
• Can be used for watershed delineation in Geographic Information
Systems
28
Vertices that can reach out to E are A and B

Bangalore Dec-2014
Algorithm
Graph.Pregel( Seq[vertexID’s] ) (
#message consumption
if vertex.state == 1
vertex.state → 2
else if vertex.state == 0
if ( vertex.adjacentVertices ∩ Seq[ vertexID’s ] ) isNotEmpty
vertex.state → 2
#message aggregator
Seq[existing vertex ID’s] U Seq[new vertex ID]
)
29

#message generation
for each triplet
if destinationVertex.state == 1
message( sourceVertexID, Seq[destinationVertexID] )
message( destinationVertexID, Seq[destinationVertexID] )
else if sourceVertex.state == 1 and destinationVertex.state == 2
message( sourceVertexID, Seq[destinationVertexID] )
else message( empty )
Algorithm

References
• Fork our repository at
• https://github.com/anantasty/SparkAlgorithms
• Follow us at
• https://github.com/codeAshu
• https://github.com/kaushikranjan
• https://spark.apache.org/docs/latest/graphx-programming-guide.html
31

GraphX and Pregel - Apache Spark

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a GraphX and Pregel - Apache Spark

Semelhante a GraphX and Pregel - Apache Spark (20)

Último

Último (20)

GraphX and Pregel - Apache Spark