Introduction for mining the social graph with Neo4j and R and Gephi.

1. Mining the Social Graph
mixi.inc
Shunya Kimura

2. Introduction
• Name: Shunya Kimura
• twitter: @kimuras
• Job:Data mining, Software engineering
• text mining, graph mining, search engine

3. Agenda
• Introduction
• The past work
• Introduction to GraphDB
• Introduction to Neo4j
• Introduction to analysis sample

4. Introduction

5. Motivation for social graph analysis
Test of millions of nodes, hundreds of millions of edges.
The diversity of graph algorithm by developing distributed processing technology.
Challenging.

6. Number of users on mixi
30000000
ID
22500000
# of member id
15000000
7500000
0
2007 2008 2009 2010 2011
year

7. What is Social Graph?

15. Feed Back

16. Feed Back

17. Feed Back

18. Feed Back

19. Feed Back

20. Approach for SG analysis
Feed Back

21. Approach for SG analysis
Feed Back

22. Approach for SG analysis
Feed Back

23. Approach for SG analysis
Feed Back

24. The past work

26. • Friend recommend

27. • Friend recommend
• Community recommend

28. Relational Databases
from_id to_id id name age
1 2 1 Kimura 18
1 3 2 kato 45
2 3 3 ito 21

29. Relational Databases
Dump &
Denormalization
from_id to_id id name age
1 2 1 Kimura 18
1 3 2 kato 45
2 3 3 ito 21

30. Relational Databases
Dump &
Denormalization
from_id to_id id name age Key value
1 2 1 Kimura 18 From:1 2,3
1 3 2 kato 45 From:2 3
2 3 3 ito 21 Prof:1 Kimura,18
Prof:2 Kato,45

31. Relational Databases
Dump &
Denormalization
from_id to_id id name age Key value
1 2 1 Kimura 18 From:1 2,3
1 3 2 kato 45 From:2 3
2 3 3 ito 21 Prof:1 Kimura,18
Prof:2 Kato,45

32. Relational Databases
Dump &
Denormalization
from_id to_id id name age Key value
1 2 1 Kimura 18 From:1 2,3
1 3 2 kato 45 From:2 3
2 3 3 ito 21 Prof:1 Kimura,18
Prof:2 Kato,45

33. Relational Databases
Dump &
Denormalization
from_id to_id id name age Key value
1 2 1 Kimura 18 From:1 2,3
1 3 2 kato 45 From:2 3
2 3 3 ito 21 Prof:1 Kimuras,18
Prof:2 Kato,45

34. Relational Databases
Dump &
reimplementation Denormalization
from_id to_id id name age Key value
1 2 1 Kimura 18 From:1 2,3
1 3 2 kato 45 From:2 3
2 3 3 ito 21 Prof:1 Kimuras,18
Prof:2 Kato,45

35. Relational Databases
Dump &
reimplementation Denormalization
from_id to_id id name age Key value
1
1
2
3
maintenance cost
1
2
Kimura
kato
18
45
From:1
From:2
2,3
3
2 3 3 ito 21 Prof:1 Kimuras,18
Prof:2 Kato,45

36. Relational Databases
Dump &
reimplementation Denormalization
from_id to_id id name age Key value
1
1
2
3
maintenance cost
1
2
Kimura
kato
18
45
From:1
From:2
2,3
3
2 3 3 ito 21 Prof:1 Kimuras,18
Prof:2 Kato,45
scalability

37. Introduction to GraphDB

38. What is graph

39. What is graph
Vertex (node)

40. What is graph
Vertex (node)
Edge

41. What is graph
Vertex (node)
Undirected graph
Edge

42. What is graph
Vertex (node)
Edge

43. What is graph
Vertex (node)
Edge

44. What is graph
Vertex (node)
Edge

45. What is graph
Vertex (node)
Directed graph
Edge

46. What is GraphDB
Vertex (node)
Edge

47. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge

48. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge
ID: 2
NAME: ITO
PROP: Female
AGE: 21

49. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge
ID: 2
NAME: ITO
PROP: Female
AGE: 21

50. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge
ID: 2
NAME: ITO
PROP: Female
AGE: 21

51. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge
ID: 2
ID: 3 NAME: ITO
LABEL: Like PROP: Female
Since: 2011/08/06 AGE: 21
OutGoing: 2

52. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge
ID: 2
ID: 3 NAME: ITO
LABEL: Like PROP: Female
Since: 2011/08/06 AGE: 21
OutGoing: 2

53. What is GraphDB
ID: 1
Vertex (node)
NAME: kimura
PROP: Male
AGE: 18
Edge
ID: 2
ID: 3 NAME: ITO
LABEL: Like PROP: Female
Since: 2011/08/06 AGE: 21
OutGoing: 2

54. The implementations
for GraphDB
http://en.wikipedia.org/wiki/GraphDB

55. Introduction to Neo4j

56. GraphDB Neo4j
• True ACID transactions
• High availability
• Scales to billions of nods and relationships
• High speed querying through traversals
Single instance(GPLv3) Multiple instance(AGPLv3)
Embedded EmbeddedGraphDatabase HighlyAvailableGraphDatabase
Standalone Neo4j Server Neo4j Server high availability mode
http://neo4j.org/

57. Other my favorite features
for Neo4j
http://www.tinkerpop.com/post/4633229547/tinkerpop-graph-stack

58. Other my favorite features
for Neo4j
• RESTful APIs
http://www.tinkerpop.com/post/4633229547/tinkerpop-graph-stack

59. Other my favorite features
for Neo4j
• RESTful APIs
• Query Language(Cypher)
http://www.tinkerpop.com/post/4633229547/tinkerpop-graph-stack

60. Other my favorite features
for Neo4j
• RESTful APIs
• Query Language(Cypher)
• Full indexing
– lucene
http://www.tinkerpop.com/post/4633229547/tinkerpop-graph-stack

61. Other my favorite features
for Neo4j
• RESTful APIs
• Query Language(Cypher)
• Full indexing
– lucene
• Implemented graph algorithm
– A*, Dijkstra
– High speed traverse
http://www.tinkerpop.com/post/4633229547/tinkerpop-graph-stack

62. Other my favorite features
for Neo4j
• RESTful APIs
• Query Language(Cypher)
• Full indexing
– lucene
• Implemented graph algorithm
– A*, Dijkstra
– High speed traverse
• Gremlin supported
– Like a query language
http://www.tinkerpop.com/post/4633229547/tinkerpop-graph-stack

63. Introduction simple Neo4j usecase
Single node Multi node
Embedded
Server

64. Introduction simple Neo4j usecase
Single node Multi node
Embedded
Analyses system
Server

65. Introduction simple Neo4j usecase
Single node Multi node
Embedded
Analyses system Analyses system
Server

66. Introduction simple Neo4j usecase
Single node Multi node
Embedded
Analyses system Analyses system
Analyses system
Server

67. Introduction simple Neo4j usecase
Single node Multi node
Embedded
Analyses system Analyses system
Analyses system Analyses system
Server

68. Introduction simple Neo4j usecase
Single node Multi node
Embedded
Analyses system Analyses system
Analyses system Analyses system
Server

69. Introduction simple Neo4j usecase
Single node Multi node
Analyses system
Embedded
Analyses system
Analyses system Analyses system
Server

70. Introduction simple Neo4j usecase
Single node Multi node
Analyses system
Embedded
Analyses system
Analyses system Analyses system
Server

71. Introduction to simple
embedded Neo4j
• Insert Vertices & make Relationships
• Single node & Embedded
• Traversal sample

72. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) {
GraphDatabaseService graphDb = new
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
firstNode.setProperty("Name", "Kimura");
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
}
}

73. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) {
GraphDatabaseService graphDb = new
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
firstNode.setProperty("Name", "Kimura");
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
}
}

74. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) { ID: 1
GraphDatabaseService graphDb = new NAME: kimura
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
firstNode.setProperty("Name", "Kimura");
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
}
}

75. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) { ID: 1
GraphDatabaseService graphDb = new NAME: kimura
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
firstNode.setProperty("Name", "Kimura");
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
}
}

76. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) { ID: 1
GraphDatabaseService graphDb = new NAME: kimura
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
firstNode.setProperty("Name", "Kimura");
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally { ID: 2
tx.finish(); NAME: Kato
}
graphDb.shutdown();
}
}

77. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) { ID: 1
GraphDatabaseService graphDb = new NAME: kimura
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
firstNode.setProperty("Name", "Kimura");
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally { ID: 2
tx.finish(); NAME: Kato
}
graphDb.shutdown();
}
}

78. Insert vertices,
make relationship
public final class InputVertex {
public static void main(final String[] args) { ID: 1
GraphDatabaseService graphDb = new NAME: kimura
EmbeddedGraphDatabase("/tmp/neo4j");
Transaction tx = graphDb.beginTx();
try {
Node firstNode = graphDb.createNode();
ID: 3
firstNode.setProperty("Name", "Kimura"); Relation: Like
Node secondNode = graphDb.createNode();
secondNode.setProperty("Name", "Kato");
firstNode.createRelationshipTo(secondNode,
DynamicRelationshipType.withName("LIKE"));
tx.success();
} finally { ID: 2
tx.finish(); NAME: Kato
}
graphDb.shutdown();
}
}

79. Batch Insert
• Non thread safe, non transaction
• But very fast!
public final class Batch {
public static void main(final String[] args) {
BatchInserter inserter = new BatchInserterImpl("/tmp/neo4j",
BatchInserterImpl.loadProperties("/tmp/neo4j.props"));
Map<String, Object> prop = new HashMap<String, Object>();
prop.put("Name", "Kimura");
prop.put("Age", 21);
long node1 = inserter.createNode(prop);
prop.put("Name", "Kato");
prop.put("Age", 21);
long node2 = inserter.createNode(prop);
inserter.createRelationship(node1, node2,
DynamicRelationshipType.withName("LIKE"), null);
inserter.shutdown();
}
}

80. Traversal sample
• You can specify the traverse criteria
public static void main(final String[] args) {
GraphDatabaseService graphDB = new EmbeddedGraphDatabase(args[0]);
Node node = graphDB.getNodeById(1);
Traverser friends = node.traverse(
Order.DEPTH_FIRST,
StopEvaluator.END_OF_GRAPH,
ReturnableEvaluator.ALL_BUT_START_NODE,
DynamicRelationshipType.withName("LIKE"),
Direction.OUTGOING);
for (Node nodeBuf : friends) {
TraversalPosition currentPosition = friends.currentPosition();
}
}

81. Traversal sample
• You can specify the traverse criteria
public static void main(final String[] args) {
GraphDatabaseService graphDB = new EmbeddedGraphDatabase(args[0]);
Node node = graphDB.getNodeById(1);
Traverser friends = node.traverse(
//how to traversal
Order.DEPTH_FIRST, BREADTH_FIRST
StopEvaluator.END_OF_GRAPH,
ReturnableEvaluator.ALL_BUT_START_NODE,
DynamicRelationshipType.withName("LIKE"),
Direction.OUTGOING);
for (Node nodeBuf : friends) {
TraversalPosition currentPosition = friends.currentPosition();
}
}

82. Traversal sample
• You can specify the traverse criteria
public static void main(final String[] args) {
GraphDatabaseService graphDB = new EmbeddedGraphDatabase(args[0]);
Node node = graphDB.getNodeById(1);
Traverser friends = node.traverse(
//how to traversal
Order.DEPTH_FIRST, BREADTH_FIRST
//traversal termination condition
StopEvaluator.END_OF_GRAPH, DEPTH_ONE
ReturnableEvaluator.ALL_BUT_START_NODE,
DynamicRelationshipType.withName("LIKE"),
Direction.OUTGOING);
for (Node nodeBuf : friends) {
TraversalPosition currentPosition = friends.currentPosition();
}
}

83. Traversal sample
• You can specify the traverse criteria
public static void main(final String[] args) {
GraphDatabaseService graphDB = new EmbeddedGraphDatabase(args[0]);
Node node = graphDB.getNodeById(1);
Traverser friends = node.traverse(
//how to traversal
Order.DEPTH_FIRST, BREADTH_FIRST
//traversal termination condition
StopEvaluator.END_OF_GRAPH, DEPTH_ONE
// to get the type of node
ReturnableEvaluator.ALL_BUT_START_NODE, ALL, isReturnableNode()
DynamicRelationshipType.withName("LIKE"),
Direction.OUTGOING);
for (Node nodeBuf : friends) {
TraversalPosition currentPosition = friends.currentPosition();
}
}

84. Traversal sample
• You can specify the traverse criteria
public static void main(final String[] args) {
GraphDatabaseService graphDB = new EmbeddedGraphDatabase(args[0]);
Node node = graphDB.getNodeById(1);
Traverser friends = node.traverse(
//how to traversal
Order.DEPTH_FIRST, BREADTH_FIRST
//traversal termination condition
StopEvaluator.END_OF_GRAPH, DEPTH_ONE
// to get the type of node
ReturnableEvaluator.ALL_BUT_START_NODE, ALL, isReturnableNode()
// type of relational for traverse
DynamicRelationshipType.withName("LIKE"),
Direction.OUTGOING);
for (Node nodeBuf : friends) {
TraversalPosition currentPosition = friends.currentPosition();
}
}

85. Traversal sample
• You can specify the traverse criteria
public static void main(final String[] args) {
GraphDatabaseService graphDB = new EmbeddedGraphDatabase(args[0]);
Node node = graphDB.getNodeById(1);
Traverser friends = node.traverse(
//how to traversal
Order.DEPTH_FIRST, BREADTH_FIRST
//traversal termination condition
StopEvaluator.END_OF_GRAPH, DEPTH_ONE
// to get the type of node
ReturnableEvaluator.ALL_BUT_START_NODE, ALL, isReturnableNode()
// type of relational for traverse
DynamicRelationshipType.withName("LIKE"),
// specify a edge type for traverse
Direction.OUTGOING); INCOMING, BOTH
for (Node nodeBuf : friends) {
TraversalPosition currentPosition = friends.currentPosition();
}
}

86. Traversal sample
Order.BREADTH_FIRST
• Breadth-first search

87. Traversal sample
Order.BREADTH_FIRST
• Breadth-first search

88. Traversal sample
Order.BREADTH_FIRST
• Breadth-first search

89. Traversal sample
Order.BREADTH_FIRST
• Breadth-first search

90. Traversal sample
Order.BREADTH_FIRST
• Breadth-first search

91. Traversal sample
Order.BREADTH_FIRST
• Breadth-first search

92. Traversal sample
Order.DEPTH_FIRST
• Depth-first search

93. Traversal sample
Order.DEPTH_FIRST
• Depth-first search

94. Traversal sample
Order.DEPTH_FIRST
• Depth-first search

95. Traversal sample
Order.DEPTH_FIRST
• Depth-first search

96. Traversal sample
Order.DEPTH_FIRST
• Depth-first search

97. Traversal sample
Order.DEPTH_FIRST
• Depth-first search

98. Neoclipse sample
http://wiki.neo4j.org/content/Neoclipse

99. experiment

100. experiment
• Store the mixi’s social graph for Neo4j
• Condition
• Machine: 24 core CPU, Memory 65GB
• Neo4j: BatchInsert, community, embedded
• Data
• # of node 15 million # of edge 600 million

101. experiment
• Store the mixi’s social graph for Neo4j
• Condition
• Machine: 24 core CPU, Memory 65GB
• Neo4j: BatchInsert, community, embedded
• Data
• # of node 15 million # of edge 600 million
process time 513m17sec (about 8.6h)

102. Network Dataset
• Stanford Large Network Dataset Collection
• SNAP has a Wide variety of graph data!
Social Networks Communication networks
Citation networks Collaboration networks
Web graphs Product co-purchasing networks
Internet peer-to-peer networks Road networks
Autonomous systems graphs Signed networks
Wikipedia networks and metadata Memetracker and Twitter
http://snap.stanford.edu/data/index.html

103. Introduction to Analysis
Sample

104. Architecture
Service
Database Analysis Visualization
(Social Graph)

105. Architecture
Service
Database Analysis Visualization
(Social Graph)

106. Introduction Analyses
Sample
• Centrality
• Clustering coefficient

107. Centrality
• Centrality
• to measure the importance of eahc nodes

108. Centrality
• Centrality
• to measure the importance of eahc nodes

109. Centrality
• Centrality
• to measure the importance of eahc nodes

110. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality

111. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank

112. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank
degree centrality

113. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank
degree centrality betweenness centrality

114. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank
degree centrality betweenness centrality
eigenvector centrality

115. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank
degree centrality betweenness centrality
eigenvector centrality centraization

116. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank
degree centralitybetweenness centrality
eigenvector centrality centraization

117. Centrality
• Centrality
• to measure the importance of eahc nodes
closeness centrality Pagerank
degree centralitybetweenness centrality
eigenvector centrality centraization

118. Degree centrality
• The simplest measuring.
• Counting the number of edge of each nodes.
• num of friends

119. Degree centrality
• The simplest measuring.
• Counting the number of edge of each nodes.
• num of friends
1 1
1

120. Degree centrality
• The simplest measuring.
• Counting the number of edge of each nodes.
• num of friends
2
1 1
2
1
2

121. Degree centrality
• The simplest measuring.
• Counting the number of edge of each nodes.
• num of friends
2
1 1
2
1
2

122. Degree centrality
• The simplest measuring.
• Counting the number of edge of each nodes.
• num of friends
2
1 1
5
2
1
2

123. Degree centrality
• The simplest measuring.
• Counting the number of edge of each nodes.
• num of friends
2
1 1
5
2
1
2

124. Degree distribution of mixi
• Random sampling the 1000 users
• the summary of degree sistribution
Min 1st Que. Median Mean 3rd Que. Max
1.00 3.00 10.00 25.69 30.00 903.00

125. Degree distribution of mixi

126. Clustering coefficient
• Network destiny around any node.
• ≒ destiny relationship

127. Clustering coefficient
• Network destiny around any node.
• ≒ destiny relationship
clustering coefficient
0 / 3 = 0 (min)

128. Clustering coefficient
• Network destiny around any node.
• ≒ destiny relationship
clustering coefficient
0 / 3 = 0 (min)
clustering coefficient
=1/3

129. Clustering coefficient
• Network destiny around any node.
• ≒ destiny relationship
clustering coefficient
0 / 3 = 0 (min)
clustering coefficient
=1/3
clustering coefficient
=2/3

130. Clustering coefficient
• Network destiny around any node.
• ≒ destiny relationship
clustering coefficient
0 / 3 = 0 (min)
clustering coefficient
=1/3
clustering coefficient
=2/3
clustering coefficient
= 3 / 3 = 1 (max)

131. Clustering coefficient
• Random sampling the 1000 users
• summary for Clustering coefficient
Min 1st Que. Median Mean 3rd Que. Max
0.00 0.00 0.1157 0.2071 0.2667 1.000

132. Clustering coefficient

133. Clustering coefficient

134. the sample of low Clustering
coefficient user
• degree 25, clustering coefficient 0.08

135. the sample of middle
Clustering coefficient user
• degree 14, clustering coefficient 0.17

136. the sample of high Clustering
coefficient user
• degree 10, clustering coefficient 0.68

137. the sample of MAX Clustering
coefficient user
• degree 4, clustering coefficient 1

138. Visualization Sample

139. • Visualize a my social graph on mixi
• Weighting the Edge
• Amount of communication(color, thickness)
• Weighting the Vertex
• cluster coefficient(color, thickness)
• visualization tool Gephi
http://gephi.org/

141. • Motivation for Social Graph mining
• Overview for GraphDB
• Introduction for Neo4j
• The samples for graph analysis with R
• Introduction Visualization tool Gephi

142. Thanks!