PyData Chicago 2016

1. Build a Recommendation
Engine with Neo4j and Python

2. ‣ Download Neo4j: neo4j.com/download
‣ Open your browser to http://localhost:7474
‣ Type the following command:
:play http://guides.neo4j.com/pydatachi
Getting Started

3. Problem

4. Generic recommendations are
low efficacy…

5. Generic recommendations are
typically low efficacy…

7. Proposed Solution

8. "It's all about
relationships
- Kevin Van Gundy
-Lebron James"

9. …Data Relationships

10. CAR
DRIVES
name: “Dan”
born: May 29, 1970
twitter: “@dan”
name: “Ann”
born: Dec 5, 1975
since:
Jan 10, 2011
brand: “Volvo”
model: “V70”
Property Graph Model Components
Nodes
• The objects in the graph
• Can have name-value
properties
• Can be labeled
Relationships
• Relate nodes by type and
direction
• Can have name-value
properties
LOVES
LIKES
LIVES WITH
OW
NS
PERSON PERSON

11. CAR
DRIVES
name: “Dan”
born: May 29, 1970
twitter: “@dan”
name: “Ann”
born: Dec 5, 1975
since:
Jan 10, 2011
brand: “Volvo”
model: “V70”
LOVES
LIKES
LIVES WITH
OW
NS
PERSON PERSON
Storage on Disk

12. Introducing our data set...

13. meetup.com’s recommendations

14. Recommendation queries
‣ Several different types
• groups to join
• topics to follow
• events to attend
‣ As a user of meetup.com trying to find groups
to join and events to attend

15. The data
meetup.com/meetup_api/

18. What data do we have?
‣ Groups
‣ Members
‣ Events
‣ Topics
‣ Time & Date
‣ Location

19. Find similar groups to Neo4j
"As a member of the Outdoorsy Entrepreneur Meetup
I want to find other similar meetup groups
So that I can join those groups"

20. What makes groups similar?

21. ‣ Download Neo4j: Neo4j.com/download
‣ Open your browser to http://localhost:7474
‣ Type the following command:
:play http://guides.neo4j.com/pydatachi
Recommend groups by topic

22. To the Browser!
Great Graphs
Batman!

23. Take Note
Indexes and Constraints

24. Unique constraints
We create unique constraints to:
‣ ensure uniqueness
‣ allow fast lookup of nodes which match these
(label,property) pairs.
CREATE CONSTRAINT ON (t:Topic)
ASSERT t.id IS UNIQUE

25. Indexes
We create indexes to:
‣ Allow fast lookup of nodes which match these
(label,property) pairs.
CREATE INDEX ON :Group(name)

26. The following are index backed:
‣ Equality
‣ STARTS WITH
‣ CONTAINS,
‣ ENDS WITH
‣ Range Searches
‣ (Non) Existence Checks
Indexes

27. How does Neo4j use indexes?
Indexes are only used to find the starting point for
queries.
Use index scans to look up
rows in tables and join them
with rows from other tables
Use indexes to find the starting
points for a query.
Relational
Graph

28. Next Guide
Group Membership

29. Watch Out
Transactions & WITH

30. Periodic Commit
Cypher keeps all transaction state in memory while
running a query which is fine most of the time.

31. Periodic Commit
Cypher keeps all transaction state in memory while
running a query which is fine most of the time…
But when using LOAD CSV, this state can get very
large and may result in an OutOfMemory exception.

32. Periodic Commit
// defaults to 1000
USING PERIODIC COMMIT
LOAD CSV
...

33. Periodic Commit
// defaults to 1000
USING PERIODIC COMMIT 10000
LOAD CSV
...

34. WITH
The WITH clause allows query parts to be chained
together, piping the results from one to be used as
starting points or criteria in the next.

35. WITH
It’s used to:
‣ Limit the number of entries that are then passed
on to other MATCH clauses
‣ Filter on aggregated values
‣ Separate reading from updating of the graph

36. Continue
Continue with the Guide

37. Exercise
Find yourself and your groups

38. Solution
Find yourself and your groups

39. Explore the graph
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/02_find_yourself_answers.html

40. Continue
Continue with the Guide

41. Find my similar groups
As a member of several meetup groups
I want to find other similar meetup groups
that I’m not already a member of
So that I can join those groups

42. Next Step
Member Interest

43. Member interests

44. Attention
Lists with split & UNWIND

45. Splitting up topic ids
The split function lets us convert a string into
a string array based on a delimiting character.

46. Splitting up topic ids
The split function lets us convert a string into a string
array based on a delimiting character.
RETURN split("1;2;3", ";") AS topicIds
[1, 2, 3]

47. We can use UNWIND to explode any array or list back
into individual rows.
Splitting up topic ids

48. We can use UNWIND to explode the resulting array back into individual rows.
UNWIND [1,2,3] AS value
RETURN value
1
2
3
Splitting up topic ids

49. UNWIND split("1;2;3", ";") AS topicId
RETURN topicId
1
2
3
Splitting up topic ids

50. Continue
Continue with the Guide

51. Exercise
My inferred interests

52. Solution
My inferred interests

53. My inferred interests
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/03_inferred_answers.html

54. Continue
Continue with the Guide

55. Next Guide
Events

56. Exercise
Event recommendations

57. Solution
Event recommendations

58. Event recommendations
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/04_events_answers.html

59. Continue
Continue with the Guide

60. Next Guide
Venues

61. Exercise
Import venues

62. Solution
Import venues

63. Import venues
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/05_venues_import_answers.html

64. Continue
Continue with the Guide

65. Next Step
Calculating Distances

66. Continue
Continue with the Guide

67. Exercise
Using venues in recommendation

68. Solution
Using venues in recommendation

69. Using venues in recommendations
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/05_venues_distance_queries_answers.html

70. Using venues in recommendations
WITH {latitude: 51.518551, longitude: -0.086114} AS here
MATCH (member:Member {name: "Mark Needham"})
-[:MEMBER_OF]->()-[:HOSTED_EVENT]->(futureEvent),
(venue)<-[:VENUE]-(futureEvent)
WHERE futureEvent.time > timestamp()
RETURN group.name,
futureEvent.name,
round((futureEvent.time - timestamp()) / (24.0*60*60*1000)) AS days,
distance(venue, here) AS distance
ORDER BY days, distance

71. Venues close to here
WITH {latitude: 51.518551, longitude: -0.086114} AS here
MATCH (member:Member {name: "Mark Needham"})
-[:MEMBER_OF]->()-[:HOSTED_EVENT]->(futureEvent),
(venue)<-[:VENUE]-(futureEvent)
WHERE futureEvent.time > timestamp()
WITH group, futureEvent, distance(venue, here) AS distance
WHERE distance < 1000
RETURN group.name,
futureEvent.name,
round((futureEvent.time - timestamp()) / (24.0*60*60*1000)) AS days,
distance
ORDER BY days, distance

72. Next Guide
RSVPs

73. Exercise
Events at my venues

74. Solution
Events at my venues

75. Events at my venues
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/06_my_venues_answers.html

76. Next Guide
Procedures

77. Exercise
Import photos metadata

78. Solution
Import photos metadata

79. Import photos metadata
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/07_photos_answers.html

80. Next Guide
Latent Social Graph

81. Watch Out
Transaction State

82. Transaction State
Cypher keeps all transaction state in memory
while running a query which is fine most of the
time.
But when refactoring the graph, this state can get
very large and may result in an OutOfMemory
exception.

83. We therefore need to take a batched approach to
large scale refactorings.
MATCH (m1:Process) WITH m1 LIMIT 1000
REMOVE m1:Process
WITH m1
// do the refactoring
Batch all the things

84. Continue
Continue with the Guide

85. Exercise
Add friends to recommendation

86. Solution
Add friends to recommendation

87. Add friends to recommendation
Type the following command into the Neo4j
browser to see the answers:
:play http://guides.neo4j.com/reco/08_latent_answers.html

88. Next Guide
Scoring

89. Next Guide
Your turn

90. JOIN NEO4J.COM/SLACK & #TRAINING-ATTENDEES