SlideShare uma empresa Scribd logo
1 de 95
Baixar para ler offline
Work Backwards to Your Graph Data Model & Queries with Amazon Neptune (DAT330) - AWS re:Invent 2018
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Work Backwards to Your Graph Data
Model & Queries with Amazon Neptune
Ian Robinson
Data Architect
AWS
D A T 3 3 0
Mike Personick
Sr. Software Dev Engineer
AWS
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction to Amazon Neptune
Overview of the data modelling process
How to apply the property graph building blocks
”Slow motion” example of developing a model and queries
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Monday, November 26
On-Ramp to Graph Databases and Amazon Neptune - DAT335
4:45 p.m. | Mirage, St. Thomas B
Tuesday, November 27
Migrating to Amazon Neptune - DAT338
11:30 a.m. | Venetian, Level 4, Lando 4305
Wednesday, November 28
Getting Started with Amazon Neptune and Amazon SageMaker Jupyter Notebooks - DAT359
2:30 p.m. | Aria West, Level 3, Starvine 10, Table 7
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune
Fast, reliable graph database
Optimized for storing and querying highly connected data
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully managed graph database
Fast Reliable Open
Query billions of
relationships with
millisecond latency
Six replicas of your data
across three AZs with full
backup and restore
Build powerful queries
easily with Gremlin and
SPARQL
Supports Apache
TinkerPop & W3C RDF
graph models
Easy
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Do I have a graph workload?
Complex domain model Variable schema Connected queries
Large dataset
Many different entities
Similar entities may have
different properties
Highly connected
Entities connected in
many different ways
Navigate connected structure
Take account of strength,
weight, or quality of
relationships
Variable structure
Social
networking
Recommendations Knowledge
graphs
Fraud detection Life sciences Network & IT
operations
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working backwards—Design for queryability
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working backwards—Design for queryability
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working backwards—Design for queryability
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working backwards—Design for queryability
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working backwards—Design for queryability
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working backwards—Design for queryability
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use property graph primitives to build your model
Vertices
Entities (things with identity, complex value types)
Properties Entity attributes + metadata
Labels Roles, grouping, tagging
Edges
Relationships between entities, variable structure
Properties Relationship strength, weight, or property + metadata
Labels Relationship semantics
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vertex and edge IDs
Neptune will create UUID IDs for new vertices and edges
Alternatively, you can supply your own string-based IDs
All vertex IDs must be unique
All edge IDs must be unique
You must supply your own IDs when bulk loading data
You can also create id properties
Avoid doing this: it creates confusion
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Labels
Tags, keyless properties
Typically used to “type” a vertex, or name a relationship
Vertices must have one or more labels
Neptune will create a “vertex” label if you don’t supply one
Edges must have exactly one label
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edges—The routes to success
Performance depends on how much of the graph a query must “touch”
Choose domain-meaningful edge labels
Discover only what is absolutely necessary
“Grey out” unnecessary portions of the graph
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How fine-grained should my edge labels be?
Is it an open or extensible set of label values?
Do you need to query across values in the set? (for example, all addresses)
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bi-directional relationships
Ignore edge direction in queries g.V().hasLabel('Person').
both('FRIENDS')
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Uni-directional relationships
Be explicit about edge direction in queries g.V().hasLabel('Person').
out('FOLLOWS’)
g.V().hasLabel('Person').
in('FOLLOWS')
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multiple edges between vertices
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
You can even have self-edges
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example scenario
Employment history application
People, companies, roles
Use cases
1. Find the companies where X has worked, and their roles at those companies
2. Find the people who have worked for a company at a specific location during a particular
time period
3. Find the people in more senior roles at the companies where X worked
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What questions would we have to ask of our data?
Find the companies where X has worked, and their roles at those
companies
Which companies has X worked for, and in what roles?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Identify entities, relationships, and attributes
Find the companies where X has worked, and their roles at those
companies
Which companies has X worked for, and in what roles?
Companies Company Entity
X Person Entity
Worked for Worked for Relationship
Roles Role Attribute?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Identify candidate vertices, labels, and properties
Company Entity Vertex Company
Person Entity Vertex Person
Worked for Relationship Edge WORKED_FOR
Role Attribute? Property? role
name
CompanyPerson
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Entity or attribute? Vertex or property?
Is role an entity or an attribute?
Does it have identity (or is it a value type)? X
Is it a complex type (with multiple fields)? X
Are there any structural relations between values? ?
Keep it simple
Prefer properties to vertices/edges until the need arises
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model 1
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traversal 1
Which companies has Li worked for, and in what roles?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create dataset
(g.
addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera').
addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe').
addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan').
addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles').
addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar').
addV('Company').property(id,'c-1').property('name','Example Corp').
addV('Company').property(id,'c-2').property('name','AnyCompany').
V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst').
V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst').
V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager').
V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst').
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create dataset
(g.
addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera').
addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe').
addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan').
addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles').
addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar').
addV('Company').property(id,'c-1').property('name','Example Corp').
addV('Company').property(id,'c-2').property('name','AnyCompany').
V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst').
V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst').
V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager').
V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst').
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create dataset
(g.
addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera').
addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe').
addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan').
addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles').
addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar').
addV('Company').property(id,'c-1').property('name','Example Corp').
addV('Company').property(id,'c-2').property('name','AnyCompany').
V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst').
V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst').
V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager').
V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst').
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create dataset
(g.
addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera').
addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe').
addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan').
addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles').
addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar').
addV('Company').property(id,'c-1').property('name','Example Corp').
addV('Company').property(id,'c-2').property('name','AnyCompany').
V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst').
V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst').
V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager').
V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst').
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create dataset
(g.
addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera').
addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe').
addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan').
addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles').
addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar').
addV('Company').property(id,'c-1').property('name','Example Corp').
addV('Company').property(id,'c-2').property('name','AnyCompany').
V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst').
V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst').
V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst').
V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager').
V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst').
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Write a test
Sample data and code
Amazon SageMaker Jupyter notebooks
AWS Cloud9 cloud IDE
Gremlin console
Unit tests
Bedrock of asserted behavior
Fast feedback as you evolve your model
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unit test
%%unittest
results = None # Which companies has Li worked for, and in what roles?
assert results == [{'company': 'Example Corp', 'role': 'Analyst'},
{'company': 'AnyCompany', 'role': 'Associate Analyst’}]
Fail
AssertionError: None != [{'company': 'Example Corp', 'role': 'Analyst'},
{'company': 'AnyCompany', 'role': 'Associate Analyst'}]
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unit test
%%unittest
results = (g.V('p-3').
outE('WORKED_FOR').as_('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList())
assert results == [{'company': 'Example Corp', 'role': 'Analyst'},
{'company': 'AnyCompany', 'role': 'Associate Analyst'}]
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What questions would we have to ask of our data?
Find the people who have worked for a company at a specific location
during a particular time period
Who worked for company X, and at which locations, between Y1-Y2?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Entities, relationships, and attributes
Find the people who have worked for a company at a specific location
during a particular time period
Who worked for company X, and at which locations, between Y1-Y2?
Locations Location Entity?
Y1-Y2 Date range or Attributes
dates (from and to)?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Entity or attribute? Vertex or property?
Is location an entity or an attribute?
Does it have identity (or is it a value type)? ✓
Is it a complex type (with multiple fields)? ✓
Are there any structural relations between values? ?
Model location as a vertex
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We can’t ”attribute” an edge with a vertex
name
CompanyPerson
Location
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introduce an intermediate vertex
name
CompanyPerson
Location
???
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised model
Represent facts/events as vertices
Connect to multiple dimensions
name
CompanyPerson
Location
Job
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model 2
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traversal 2
Who worked for Example Corp., and at which locations, between 2015-
2017?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 2
g.V('c-1').
in('COMPANY').
or(
(has('from', between(datetime(2015,1,1), datetime(2018,1,1)))),
(has('to', between(datetime(2015,1,1), datetime(2018,1,1))))
).
project('name', 'location').
by(in('JOB').values('firstName', 'lastName').fold()).
by(out('LOCATION').values('name', 'address').fold()).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We’ve broken query 1!
%%unittest
results = (g.V('p-3').
outE('WORKED_FOR').as_('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList())
assert results == [{'company': 'Example Corp', 'role': 'Analyst'},
{'company': 'AnyCompany', 'role': 'Associate Analyst’}]
Fail
AssertionError: Lists differ: [] != [{'company': 'Example Corp', 'role': 'Analyst'},
{'company': 'AnyCompany', 'role': 'Associate Analyst'}]
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised query 1
g.V('p-3').
outE('WORKED_FOR').as('e').
otherV().
project('company', 'role').
by('name').
by(select('e').values('role')).
toList()
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What questions would we have to ask of our data?
Find the people in more senior roles at the companies where X worked
Who were in senior roles at the companies where X worked?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Entities, relationships, and attributes
Find the people in more senior roles at the companies where X worked
Who were in more senior roles at the companies where X worked?
Senior Roles Role Entity?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Entity or attribute? Vertex or property?
Is role an entity or an attribute?
Does it have identity (or is it a value type)? X
Is it a complex type (with multiple fields)? X
Are there any structural relations between values? ✓
Model structural relations with edges
Promote role to being a vertex
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Role hierarchy
Role
Role Role
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised model
name
CompanyPerson
Location
Job
Role
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model 3
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traversal 3
Who were in more senior roles at the companies where Li worked?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create data to test predicates
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create data to test predicates
assert results == [{'role': 'Principal Analyst', 'name': ['Martha', 'Rivera']},
{'role': 'Manager', 'name': ['Saanvi', 'Sarkar']}]
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gremlin query 3
g.V('p-3').
out('JOB').as('j1').
out('ROLE').
repeat(out('PARENT_ROLE')).until(outE().count().is(0)).
emit().in('ROLE').as('j2').
or(
(where('j1', between('j2', 'j2')).by('from').by('from').by('to')),
(where('j1', between('j2', 'j2')).by('to').by('from').by('to')),
(where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from'))
).
project('role', 'name').
by(out('ROLE').values('name')).
by(in('JOB').values('firstName', 'lastName').fold()).
toList())
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We’ve broken query 1 (again)!
%%unittest
results = (g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList())
assert results == [{'company': 'Example Corp', 'role': 'Analyst'},
{'company': 'AnyCompany', 'role': 'Associate Analyst’}]
Fail
The property does not exist as the key has no associated value for the provided
element: v[j-3]:role
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised query 1
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList()
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by(out('ROLE').values('name')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Revised query 1
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by('role').
toList()
g.V('p-3').
out('JOB').
project('company', 'role').
by(out('COMPANY').values('name')).
by(out('ROLE').values('name')).
toList()
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What have we learned?
Work backwards from your use case to make a useful model
Start simple
Evolve as necessary
Your model should offer up the most efficient routes for your queries
The model is the superset of paths traversed by your queries
Use interactive development environments to develop your model and
queries
Write unit tests that give you fast feedback as you evolve your model
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What have we learned?
• Work backwards from
your use case to make a
useful model
• Start simple and evolve
as necessary
• Your model should offer
up the most efficient
routes for your queries
• The model is the
superset of paths
traversed by your queries
• Use an interactive
environment to develop
your model and queries
• Write unit tests that give
you fast feedback as you
evolve your model
Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ian Robinson
ianrob@amazon.com
Mike Personick
mprsnck@amazon.com
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Mais conteúdo relacionado

Mais procurados

Neptune, the Graph Database | AWS Floor28
Neptune, the Graph Database | AWS Floor28Neptune, the Graph Database | AWS Floor28
Neptune, the Graph Database | AWS Floor28Amazon Web Services
 
Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...
Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...
Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...Amazon Web Services
 
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017Amazon Web Services
 
Amazon Athena Capabilities and Use Cases Overview
Amazon Athena Capabilities and Use Cases Overview Amazon Athena Capabilities and Use Cases Overview
Amazon Athena Capabilities and Use Cases Overview Amazon Web Services
 
Building Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS GlueBuilding Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS GlueAmazon Web Services
 
Amazon Rekognition: Deep Learning-Based Image and Video Analysis
Amazon Rekognition: Deep Learning-Based Image and Video AnalysisAmazon Rekognition: Deep Learning-Based Image and Video Analysis
Amazon Rekognition: Deep Learning-Based Image and Video AnalysisAmazon Web Services
 
Microsoft Azure Active Directory
Microsoft Azure Active DirectoryMicrosoft Azure Active Directory
Microsoft Azure Active DirectoryDavid J Rosenthal
 
Graph & Amazon Neptune: Database Week SF
Graph & Amazon Neptune: Database Week SFGraph & Amazon Neptune: Database Week SF
Graph & Amazon Neptune: Database Week SFAmazon Web Services
 
Microsoft Azure - Introduction to microsoft's public cloud
Microsoft Azure - Introduction to microsoft's public cloudMicrosoft Azure - Introduction to microsoft's public cloud
Microsoft Azure - Introduction to microsoft's public cloudAtanas Gergiminov
 
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...Amazon Web Services
 
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfData & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfChris Bingham
 
Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...
Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...
Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...Amazon Web Services
 
Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018
Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018
Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018Amazon Web Services
 
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018Amazon Web Services
 
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...Amazon Web Services
 
Benefits of the Azure cloud
Benefits of the Azure cloudBenefits of the Azure cloud
Benefits of the Azure cloudJames Serra
 
AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례
AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례
AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례Amazon Web Services Korea
 
Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018
Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018
Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018Amazon Web Services
 

Mais procurados (20)

Neptune, the Graph Database | AWS Floor28
Neptune, the Graph Database | AWS Floor28Neptune, the Graph Database | AWS Floor28
Neptune, the Graph Database | AWS Floor28
 
Graph and Amazon Neptune
Graph and Amazon NeptuneGraph and Amazon Neptune
Graph and Amazon Neptune
 
Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...
Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...
Migrating Thousands of Workloads to AWS at Enterprise Scale – Chris Wegmann, ...
 
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017
 
Amazon Athena Capabilities and Use Cases Overview
Amazon Athena Capabilities and Use Cases Overview Amazon Athena Capabilities and Use Cases Overview
Amazon Athena Capabilities and Use Cases Overview
 
Building Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS GlueBuilding Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS Glue
 
Amazon Rekognition: Deep Learning-Based Image and Video Analysis
Amazon Rekognition: Deep Learning-Based Image and Video AnalysisAmazon Rekognition: Deep Learning-Based Image and Video Analysis
Amazon Rekognition: Deep Learning-Based Image and Video Analysis
 
Microsoft Azure Active Directory
Microsoft Azure Active DirectoryMicrosoft Azure Active Directory
Microsoft Azure Active Directory
 
Graph & Amazon Neptune: Database Week SF
Graph & Amazon Neptune: Database Week SFGraph & Amazon Neptune: Database Week SF
Graph & Amazon Neptune: Database Week SF
 
Microsoft Azure - Introduction to microsoft's public cloud
Microsoft Azure - Introduction to microsoft's public cloudMicrosoft Azure - Introduction to microsoft's public cloud
Microsoft Azure - Introduction to microsoft's public cloud
 
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
 
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfData & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
 
Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...
Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...
Building Your First Graph Application with Amazon Neptune - Workshop (DAT310-...
 
Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018
Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018
Building Advanced Workflows with AWS Glue (ANT333) - AWS re:Invent 2018
 
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018
 
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
 
Benefits of the Azure cloud
Benefits of the Azure cloudBenefits of the Azure cloud
Benefits of the Azure cloud
 
AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례
AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례
AWS Summit Seoul 2023 | 지역거점 중심의 산업 생태계 활성화 사례
 
Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018
Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018
Deep Dive on Amazon Neptune (DAT403) - AWS re:Invent 2018
 
Azure purview
Azure purviewAzure purview
Azure purview
 

Semelhante a Work Backwards to Your Graph Data Model & Queries with Amazon Neptune (DAT330) - AWS re:Invent 2018

Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018
Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018
Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018Amazon Web Services
 
Deep Dive on Amazon Neptune - AWS Online Tech Talks
Deep Dive on Amazon Neptune - AWS Online Tech TalksDeep Dive on Amazon Neptune - AWS Online Tech Talks
Deep Dive on Amazon Neptune - AWS Online Tech TalksAmazon Web Services
 
Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018
Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018
Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018Amazon Web Services
 
Using AI for real-life data enrichment - Tel Aviv Summit 2018
Using AI for real-life data enrichment - Tel Aviv Summit 2018Using AI for real-life data enrichment - Tel Aviv Summit 2018
Using AI for real-life data enrichment - Tel Aviv Summit 2018Amazon Web Services
 
Graph and Amazon Neptune - Bill Baldwin
Graph and Amazon Neptune - Bill BaldwinGraph and Amazon Neptune - Bill Baldwin
Graph and Amazon Neptune - Bill BaldwinAmazon Web Services
 
Best practices for integrating Amazon Rekognition into your own application
Best practices for integrating Amazon Rekognition into your own applicationBest practices for integrating Amazon Rekognition into your own application
Best practices for integrating Amazon Rekognition into your own applicationAmazon Web Services
 
Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018
Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018
Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018Amazon Web Services
 
Graph & Neptune: Database Week San Francisco
Graph & Neptune: Database Week San FranciscoGraph & Neptune: Database Week San Francisco
Graph & Neptune: Database Week San FranciscoAmazon Web Services
 
Understanding Graph Databases: AWS Developer Workshop at Web Summit
Understanding Graph Databases: AWS Developer Workshop at Web SummitUnderstanding Graph Databases: AWS Developer Workshop at Web Summit
Understanding Graph Databases: AWS Developer Workshop at Web SummitAmazon Web Services
 
Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...
Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...
Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...Amazon Web Services
 
Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...
Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...
Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...Amazon Web Services
 
Going Graph With Amazon Neptune - AWS Summit Sydney 2018
Going Graph With Amazon Neptune - AWS Summit Sydney 2018Going Graph With Amazon Neptune - AWS Summit Sydney 2018
Going Graph With Amazon Neptune - AWS Summit Sydney 2018Amazon Web Services
 
Let me graph that for you - Amazon Neptune
Let me graph that for you - Amazon NeptuneLet me graph that for you - Amazon Neptune
Let me graph that for you - Amazon NeptuneAmazon Web Services
 
Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...
Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...
Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...Amazon Web Services
 
[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...
[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...
[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...Amazon Web Services
 
Chaos Engineering: Why Breaking Things Should Be Practised.
Chaos Engineering: Why Breaking Things Should Be Practised.Chaos Engineering: Why Breaking Things Should Be Practised.
Chaos Engineering: Why Breaking Things Should Be Practised.Adrian Hornsby
 

Semelhante a Work Backwards to Your Graph Data Model & Queries with Amazon Neptune (DAT330) - AWS re:Invent 2018 (20)

Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018
Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018
Connecting the Unconnected using GraphDB - Tel Aviv Summit 2018
 
Deep Dive on Amazon Neptune - AWS Online Tech Talks
Deep Dive on Amazon Neptune - AWS Online Tech TalksDeep Dive on Amazon Neptune - AWS Online Tech Talks
Deep Dive on Amazon Neptune - AWS Online Tech Talks
 
Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018
Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018
Amazon Cloud Directory Deep Dive (DAT364) - AWS re:Invent 2018
 
Using AI for real-life data enrichment - Tel Aviv Summit 2018
Using AI for real-life data enrichment - Tel Aviv Summit 2018Using AI for real-life data enrichment - Tel Aviv Summit 2018
Using AI for real-life data enrichment - Tel Aviv Summit 2018
 
Graph and Neptune
Graph and NeptuneGraph and Neptune
Graph and Neptune
 
Graph and Amazon Neptune
Graph and Amazon NeptuneGraph and Amazon Neptune
Graph and Amazon Neptune
 
Graph and Amazon Neptune - Bill Baldwin
Graph and Amazon Neptune - Bill BaldwinGraph and Amazon Neptune - Bill Baldwin
Graph and Amazon Neptune - Bill Baldwin
 
Best practices for integrating Amazon Rekognition into your own application
Best practices for integrating Amazon Rekognition into your own applicationBest practices for integrating Amazon Rekognition into your own application
Best practices for integrating Amazon Rekognition into your own application
 
Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018
Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018
Detect Anomalies Using Amazon SageMaker (AIM420) - AWS re:Invent 2018
 
Graph & Neptune
Graph & NeptuneGraph & Neptune
Graph & Neptune
 
Graph & Neptune: Database Week San Francisco
Graph & Neptune: Database Week San FranciscoGraph & Neptune: Database Week San Francisco
Graph & Neptune: Database Week San Francisco
 
Understanding Graph Databases: AWS Developer Workshop at Web Summit
Understanding Graph Databases: AWS Developer Workshop at Web SummitUnderstanding Graph Databases: AWS Developer Workshop at Web Summit
Understanding Graph Databases: AWS Developer Workshop at Web Summit
 
Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...
Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...
Searching Your Data with Amazon Elasticsearch Service (ANT384) - AWS re:Inven...
 
Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...
Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...
Connecting the dots - How Amazon Neptune and Graph Databases can transform yo...
 
Going Graph With Amazon Neptune - AWS Summit Sydney 2018
Going Graph With Amazon Neptune - AWS Summit Sydney 2018Going Graph With Amazon Neptune - AWS Summit Sydney 2018
Going Graph With Amazon Neptune - AWS Summit Sydney 2018
 
Let me graph that for you - Amazon Neptune
Let me graph that for you - Amazon NeptuneLet me graph that for you - Amazon Neptune
Let me graph that for you - Amazon Neptune
 
Are you Well-Architected?
Are you Well-Architected?Are you Well-Architected?
Are you Well-Architected?
 
Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...
Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...
Search Your DynamoDB Data with Amazon Elasticsearch Service (ANT302) - AWS re...
 
[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...
[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...
[REPEAT] Better Analytics Through Natural Language Processing (AIM405-R) - AW...
 
Chaos Engineering: Why Breaking Things Should Be Practised.
Chaos Engineering: Why Breaking Things Should Be Practised.Chaos Engineering: Why Breaking Things Should Be Practised.
Chaos Engineering: Why Breaking Things Should Be Practised.
 

Mais de Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

Mais de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Work Backwards to Your Graph Data Model & Queries with Amazon Neptune (DAT330) - AWS re:Invent 2018

  • 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Work Backwards to Your Graph Data Model & Queries with Amazon Neptune Ian Robinson Data Architect AWS D A T 3 3 0 Mike Personick Sr. Software Dev Engineer AWS
  • 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Introduction to Amazon Neptune Overview of the data modelling process How to apply the property graph building blocks ”Slow motion” example of developing a model and queries
  • 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Related breakouts Monday, November 26 On-Ramp to Graph Databases and Amazon Neptune - DAT335 4:45 p.m. | Mirage, St. Thomas B Tuesday, November 27 Migrating to Amazon Neptune - DAT338 11:30 a.m. | Venetian, Level 4, Lando 4305 Wednesday, November 28 Getting Started with Amazon Neptune and Amazon SageMaker Jupyter Notebooks - DAT359 2:30 p.m. | Aria West, Level 3, Starvine 10, Table 7
  • 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Neptune Fast, reliable graph database Optimized for storing and querying highly connected data
  • 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fully managed graph database Fast Reliable Open Query billions of relationships with millisecond latency Six replicas of your data across three AZs with full backup and restore Build powerful queries easily with Gremlin and SPARQL Supports Apache TinkerPop & W3C RDF graph models Easy
  • 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Do I have a graph workload? Complex domain model Variable schema Connected queries Large dataset Many different entities Similar entities may have different properties Highly connected Entities connected in many different ways Navigate connected structure Take account of strength, weight, or quality of relationships Variable structure Social networking Recommendations Knowledge graphs Fraud detection Life sciences Network & IT operations
  • 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  • 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  • 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  • 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  • 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  • 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  • 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use property graph primitives to build your model Vertices Entities (things with identity, complex value types) Properties Entity attributes + metadata Labels Roles, grouping, tagging Edges Relationships between entities, variable structure Properties Relationship strength, weight, or property + metadata Labels Relationship semantics
  • 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vertex and edge IDs Neptune will create UUID IDs for new vertices and edges Alternatively, you can supply your own string-based IDs All vertex IDs must be unique All edge IDs must be unique You must supply your own IDs when bulk loading data You can also create id properties Avoid doing this: it creates confusion
  • 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Labels Tags, keyless properties Typically used to “type” a vertex, or name a relationship Vertices must have one or more labels Neptune will create a “vertex” label if you don’t supply one Edges must have exactly one label
  • 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Edges—The routes to success Performance depends on how much of the graph a query must “touch” Choose domain-meaningful edge labels Discover only what is absolutely necessary “Grey out” unnecessary portions of the graph
  • 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How fine-grained should my edge labels be? Is it an open or extensible set of label values? Do you need to query across values in the set? (for example, all addresses)
  • 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bi-directional relationships Ignore edge direction in queries g.V().hasLabel('Person'). both('FRIENDS')
  • 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Uni-directional relationships Be explicit about edge direction in queries g.V().hasLabel('Person'). out('FOLLOWS’) g.V().hasLabel('Person'). in('FOLLOWS')
  • 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multiple edges between vertices
  • 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. You can even have self-edges
  • 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example scenario Employment history application People, companies, roles Use cases 1. Find the companies where X has worked, and their roles at those companies 2. Find the people who have worked for a company at a specific location during a particular time period 3. Find the people in more senior roles at the companies where X worked
  • 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What questions would we have to ask of our data? Find the companies where X has worked, and their roles at those companies Which companies has X worked for, and in what roles?
  • 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identify entities, relationships, and attributes Find the companies where X has worked, and their roles at those companies Which companies has X worked for, and in what roles? Companies Company Entity X Person Entity Worked for Worked for Relationship Roles Role Attribute?
  • 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identify candidate vertices, labels, and properties Company Entity Vertex Company Person Entity Vertex Person Worked for Relationship Edge WORKED_FOR Role Attribute? Property? role name CompanyPerson
  • 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entity or attribute? Vertex or property? Is role an entity or an attribute? Does it have identity (or is it a value type)? X Is it a complex type (with multiple fields)? X Are there any structural relations between values? ? Keep it simple Prefer properties to vertices/edges until the need arises
  • 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model 1
  • 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traversal 1 Which companies has Li worked for, and in what roles?
  • 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  • 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  • 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  • 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  • 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  • 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Write a test Sample data and code Amazon SageMaker Jupyter notebooks AWS Cloud9 cloud IDE Gremlin console Unit tests Bedrock of asserted behavior Fast feedback as you evolve your model
  • 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unit test %%unittest results = None # Which companies has Li worked for, and in what roles? assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst’}] Fail AssertionError: None != [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst'}]
  • 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  • 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unit test %%unittest results = (g.V('p-3'). outE('WORKED_FOR').as_('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()) assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst'}]
  • 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What questions would we have to ask of our data? Find the people who have worked for a company at a specific location during a particular time period Who worked for company X, and at which locations, between Y1-Y2?
  • 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entities, relationships, and attributes Find the people who have worked for a company at a specific location during a particular time period Who worked for company X, and at which locations, between Y1-Y2? Locations Location Entity? Y1-Y2 Date range or Attributes dates (from and to)?
  • 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entity or attribute? Vertex or property? Is location an entity or an attribute? Does it have identity (or is it a value type)? ✓ Is it a complex type (with multiple fields)? ✓ Are there any structural relations between values? ? Model location as a vertex
  • 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. We can’t ”attribute” an edge with a vertex name CompanyPerson Location
  • 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introduce an intermediate vertex name CompanyPerson Location ???
  • 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised model Represent facts/events as vertices Connect to multiple dimensions name CompanyPerson Location Job
  • 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model 2
  • 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traversal 2 Who worked for Example Corp., and at which locations, between 2015- 2017?
  • 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  • 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. We’ve broken query 1! %%unittest results = (g.V('p-3'). outE('WORKED_FOR').as_('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()) assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst’}] Fail AssertionError: Lists differ: [] != [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst'}]
  • 65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  • 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  • 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  • 68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  • 69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What questions would we have to ask of our data? Find the people in more senior roles at the companies where X worked Who were in senior roles at the companies where X worked?
  • 71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entities, relationships, and attributes Find the people in more senior roles at the companies where X worked Who were in more senior roles at the companies where X worked? Senior Roles Role Entity?
  • 72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entity or attribute? Vertex or property? Is role an entity or an attribute? Does it have identity (or is it a value type)? X Is it a complex type (with multiple fields)? X Are there any structural relations between values? ✓ Model structural relations with edges Promote role to being a vertex
  • 73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Role hierarchy Role Role Role
  • 74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised model name CompanyPerson Location Job Role
  • 75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model 3
  • 76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traversal 3 Who were in more senior roles at the companies where Li worked?
  • 77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create data to test predicates
  • 78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create data to test predicates assert results == [{'role': 'Principal Analyst', 'name': ['Martha', 'Rivera']}, {'role': 'Manager', 'name': ['Saanvi', 'Sarkar']}]
  • 79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 85. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 86. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 87. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  • 88. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. We’ve broken query 1 (again)! %%unittest results = (g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()) assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst’}] Fail The property does not exist as the key has no associated value for the provided element: v[j-3]:role
  • 89. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by(out('ROLE').values('name')). toList()
  • 90. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by(out('ROLE').values('name')). toList()
  • 91. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 92. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What have we learned? Work backwards from your use case to make a useful model Start simple Evolve as necessary Your model should offer up the most efficient routes for your queries The model is the superset of paths traversed by your queries Use interactive development environments to develop your model and queries Write unit tests that give you fast feedback as you evolve your model
  • 93. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What have we learned? • Work backwards from your use case to make a useful model • Start simple and evolve as necessary • Your model should offer up the most efficient routes for your queries • The model is the superset of paths traversed by your queries • Use an interactive environment to develop your model and queries • Write unit tests that give you fast feedback as you evolve your model
  • 94. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Ian Robinson ianrob@amazon.com Mike Personick mprsnck@amazon.com
  • 95. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.