SlideShare uma empresa Scribd logo
1 de 127
Baixar para ler offline
Building Enterprise-Ready
Knowledge Graph Applications in the Cloud
Peter Haase
CEO, metaphacts GmbH
Michael Schmidt
Senior Software Development Engineer, Amazon Neptune
Welcome
3
typetype
likes
likes
works
forCEO of
builds
works
on
builds upon
builds
works
on
partner of
Knowledge
Graphs
Person
application
building on top
of
fast, reliable,
cloud-based
storage of
likes likes
A Knowledge Graph
metaphactory
4
COMPANY FACTS
• Founded in Q4 2014
• Headquartered in Walldorf, Germany
• Currently ~10 people
• Platform for knowledge graphs and knowledge graph applications
metaphacts at a Glance
5
1. Welcome
2. Introduction to Knowledge Graphs
1. Basic Concepts and Technologies
2. Storing, Querying, and Maintaining Knowledge Graphs
3. Applications of Knowledge Graphs
3. Managing Knowledge Graphs
1. Creating Knowledge Graphs
2. Querying Knowledge Graphs
4. Building End-user Facing Applications on top of Knowledge Graphs
1. Visualization and Templating
2. Semantic Search
5. Building Developer APIs on top of Knowledge Graphs
1. Exposing Queries via REST
2. Scaling up with Neptune
Workshop Agenda
Hands on!
Demo
Hands on!
Hands on!
Hands on!
Hands on!
Hands on!
Demo
6
• Build a portal to make current human proteomic
knowledge available
• Answer detailed questions about the human proteome
• I am researcher working researching heart diseases.
I want to know proteins associated with heart diseases.
Which of these proteins are located in the cell membrane?
• Enable querying via
• End user-oriented search
• APIs for easy consumption by applications
Use Case Scenario
7
• Wikidata
• Community curated knowledge graph
• Broad coverage
• Integration hub for the Life Sciences
• Nextprot
• A very detailed knowledge graph
about human proteomics, aggregated
and curated knowledge
from many sources
• Federation across data sources
Knowledge Graphs Used for the Tutorial
8
• Understanding of Knowledge Graphs
• Technology Stack
• Benefits of Knowledge Graphs
• Application Areas of Knowledge Graphs
• Setup of Enterprise-ready Knowledge Graph Architecture
• Scalable Storage and Querying with Enterprise Features (Amazon Neptune)
• Rapid Prototype & Application Building (metaphactory)
• Concepts for …
• … maintaining, querying, and searching knowledge graphs
• … building end-user facing applications on top of knowledge graphs
• … building and exposing scalable APIs on top of knowledge graphs
• End-to-end experience: throughout the tutorial, you’ll have the chance to set up a small
knowledge graph application on your own!
Learning Goals
Introduction to Knowledge Graphs
10
Knowledge Graphs - Hype Technology?
11
The Rise of Knowledge Graphs
12
Google Knowledge Graph
13
Entity Search and Summarizations
Google Knowledge Graph
14
Discovering Related Entities
Google Knowledge Graph
15
Google Knowledge Graph
Factual Answers
16
• We need a structured and formal representation of
knowledge
• We are surrounded by entities, which are connected by
relations
• Graphs are a natural way to represent entities and their
relationships
• Graphs can capture a broad spectrum of data from
structured to unstructured
• Graphs can be managed efficiently
Why (Knowledge) Graphs?
17
A (very small) Knowledge Graph
http://www.w3.org/TR/2014/NOTE-rdf11-primer-20140225/example-graph.jpg
18
• Semantic descriptions of
entities and their relationships
• Uses a knowledge
representation formalism
(RDF, RDF-Schema, OWL)
• Entities: real world objects (things, places, people) and
abstract concepts (genres, religions, professions)
• Relationships: graph-based data model where relationships
are first-class
• Semantic descriptions: types and properties with a well-
defined meaning (e.g. through an ontology)
• Possibly axiomatic knowledge (e.g. rules) to support
automated reasoning
What are Knowledge Graphs?
19
Knowledge Graphs Built on the Semantic Web Layer Cake
Unicode URIs
RDF (Resource Description Framework)
RDF-Schema
OWLSKOS
SPARQL
Query language
Entities
Relationships
Vocabularies
Ontologies
Expressive Ontology
Language
Thesauri,
classification
schemes
Graph data
model
Simple vocabulary
language
20
• Linked Data
• Set of standards, principles for publishing, sharing
and interrelating structured knowledge
• From data silos to a Web of Data
• Linked Data principles
1. Use URIs as names for things
2. Use HTTP URIs so that people can look up those names.
3. When someone looks up a URI, provide useful information, using the
standards: RDF, SPARQL
4. Include links to other URIs, so that they can discover more things.
• Linked Data as abstraction layer for virtualized data access across data spaces
• Platform independent data access
• Enables data portability and interoperability across current data silos
Linked Data
21
Linked Open Data Cloud
22
How do we integrate these resources?
Michel Dumontier, Chicago Semantic Web Meetup
23
Uniform Resource Identifier (URI) can be used as entity names
http://bio2rdf.org/uniprot:P05067
is a name for Amyloid precursor protein
http://bio2rdf.org/omim:104300
is a name for Alzheimer disease
uniprot:P05067
omim:104300
Allows one to talk about anything
Resource Description Framework (RDF)
Michel Dumontier, Chicago Semantic Web Meetup
24
Allows one to express statements
An RDF statement consists of:
• Subject: resource identified by a URI
• Predicate: resource identified by a URI
• Object: resource or literal
Resource Description Framework (RDF)
25
Protein
rdf:type
A RDF statement consists of:
– Subject: resource identified by a URI
– Predicate: resource identified by a URI
– Object: resource or literal
uniprot:P05067
Allows one to express statements
Resource Description Framework (RDF)
Michel Dumontier, Chicago Semantic Web Meetup
26
uniprot:P05067 Membrane
Protein
is a
located in
uniprot:P05067
uniprot:P05067 uniprot:P05067
interacts with
UniProt
Gene Ontology
uniprot:P05067
is a
located in
interacts with
Unified view
+
+
iRefIndex
depends on consistent naming
Multi-Source Data Integration
Membrane
Protein
uniprot:P05067
Michel Dumontier, Chicago Semantic Web Meetup
27
Knowledge Graphs Enabling Intelligent Applications
Knowledge Graph
Algorithms
Applications
Data Transformation, Integration
Natural Language Processing
Data Sources
• Inferencing
• Machine Learning
• Entity Recognition
• Disambiguation
• Text Understanding
• Recommendations
• Semantic Search
• Question Answering
• Knowledge Sharing
• Knowledge Management
• Analytics
• Entities
• Relationships
• Semantic Descriptions
• Dashboards
28
metaphactory Platform Architecture
29
Enterprise Knowledge Graphs Span Multiple Data Spaces
Advantages of Enterprise
Knowledge Graphs
• Unlock isolated data silos
• Query across data sources
• 360° view on data
Ephedra – metaphactory’s
federation engine
Virtual and materialized
integration of multiple data
sources
• Graph databases
• Relational databases
• Compute services
• REST APIs
browser
browser
browser
Relational Database
OBDA/ETL
Graph Database
SPARQL 1.1
Hybrid Services
REST APIs
• R2RML mappings
or query patterns
• Multiple graph
databases
• External and internal
data
• Machine Learning
Algorithms
• Data Feeds for
augmentation
30
Tutorial Setup
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Michael Schmidt
Senior Software Development Engineer, Amazon Neptune
Peter Haase,
CEO, metaphacts GmbH
Storing, Querying, and Maintaining
Knowledge Graphs Using Amazon Neptune
ISWC Tutorial:
Building Enterprise-Ready Knowledge Graph Applications in the Cloud
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Knowledge Graph Applications: Blueprint Architecture
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Account Login
Hands on!
1. Log into your AWS account at
https://aws.amazon.com/console
using the credentials on your hands-out.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Account Login
Hands on!
2. Select “N. Virginia” at the top right as region (it may already be pre-selected).
Using the AWS logo link on the top left you can always return to this start page.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
1. Find CloudFormation by typing and selecting “CloudFormation” in the search box.
2. Click the “Create Stack” button at the top left.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
3. Select “Specify an Amazon S3 template URL” and type
https://s3.amazonaws.com/nextprot-mini/jupyter/metaphactory.json
Then confirm using the “Next” button.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
4. Configure your stack as per above and proceed with the “Next” button.
Note: CIDR 0.0.0.0/0 is granting public access to your metaphactory instance world. The instance will still be password protected.
Nevertheless, this is NOT a recommended production setting!
metaphactory
0.0.0.0/0
db.r4.2xlarge
tutorial
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
5 Set a tag “Name” => “metaphactory“ and confirm with the “Next” button.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
6. Validate your parameters in the “Details” section.
0.0.0.0/0
db.r4.2xlarge
tutorial
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
7. Check the box
“I acknowledge that
AWS
CloudFormation
might create IAM
resources.” and
proceed with the
“Create” button.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Neptune & metaphactory
Hands on!
8. Congratulations!
Your Neptune + metaphactory stack creation is now in progress. It will be ready in about 10
minutes. You can use the refresh button at the top to track progress.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune: A Fast, Reliable Graph Database
Optimized for storing and querying highly connected data
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Highly Connected Data Use Cases
Social Networking Recommendations Knowledge Graphs
Fraud Detection Life Sciences Network & IT Operations
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Characteristics of a Graph Workload
• Complex domain model
• Variable schema (per entity)
• Optional attributes
• Variable structure (across domain)
• Highly connected data
• Queries require joining entities or navigating relationships
• Need to understand that things are connected
• Need to understand strength, weight or quality of relationships
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune: A Fully Managed Graph Database
Fast Reliable Open
Query billions of
relationships with
millisecond latency
6 replicas of your data
across 3 AZs with full
backup and restore
Build powerful queries
easily with Gremlin and
SPARQL
Supports Apache
TinkerPop & W3C RDF
graph models
Easy
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Leading Graph Models and Frameworks
Open Source Apache TinkerPop
Gremlin Traversal Language
Property Graph Resource Description Framework
(RDF)
W3C Standard
SPARQL Query Language• Vertices and edges (nodes
and relationships) with
properties
• Both record-like items
• Imperative traversal
language (Gremlin)
• Triples
• subject – predicate – object
• Declarative pattern matching
language (SPARQL 1.1)
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune High Level Architecture
Bulk load
from S3
Database
Mgmt.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Knowledge Graph Applications: Blueprint Architecture
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Knowledge Graph Applications: Blueprint Architecture
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Looking Up the Cluster Endpoint
Hands on!
1. Search for and select “Neptune” on the entry page.
2. Click “Clusters” on the left and click on the “DB cluster identifier”.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Looking Up the Cluster Endpoint
Hands on!
3. Save the “Cluster endpoint” in a text file for later use.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Looking up the metaphactory Output
Hands on!
Select your metaphactory
stack in CloudFormation
and click on the
”Outputs” tab. Note
down the output
variables for later:
• NeptuneLoadFromS3RoleArn
• MetaphactoryPassword
• MetaphactoryURL
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Jupyter via CloudFormation
Hands on!
1. Open CloudFormation by typing and selecting “CloudFormation” in the search box.
2. Click the “Create Stack” button at the top left.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Jupyter via CloudFormation
Hands on!
3. Select “Specify an Amazon S3 template URL” and type
https://s3.amazonaws.com/nextprot-mini/jupyter/sagemaker-neptune.json
Then confirm using the “Next” button.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Jupyter via CloudFormation
Hands on!
4. Fill in the information and
confirm with “Next”.
jupyter
Fill in the NeptuneClusterEndpoint
looked up previously.
For NeptuneClusterSubnetId and
NeptuneClusterVPC choose (one of) the
value(s) with “metaphactory” in its name.
Fill in the NeptuneLoadFromS3RoleArn noted
down after metaphactory setup.
Choose (only) value from the dropdown.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Jupyter via CloudFormation
Hands on!
5. Add a tag pair with Key “Name” / Value “jupyter”. Confirm with the “Next” button.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up Jupyter via CloudFormation
Hands on!
6. Validate your data in the “Details” section. Check the “I acknowledge that AWS
CloudFormation might create IAM resources” box. Confirm with “Next”.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune Enterprise Features
• Security
• Network isolation via Virtual Private Cloud & Security Groups
• Encryption at rest using AWS Key Management Service (KMS)
• IAM policies to secure resources & control database authentication
• Cloud-Native storage
• Up to 64TB of available storage (independent from head node instance type)
• Data replicated six times across 3 AZs
• Continuous backup to S3, built for 11 9s of durability
• Backup and Restore
• Automated daily and manual snapshots (can be shared with other AWS accounts)
• Restoring from DB snapshot creates a new instance
• Point-in time restore (from DB instance)
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Loading Data into Neptune
Bulk Load API
• UTF-8 encoded files in S3
• Supports gzip compression of single files
• Requires a VPC endpoint for Amazon S3 – cluster and bucket in same region
• Add IAM Role to Neptune allowing s3::Get* and s3::List* permissions for S3 bucket
• CSV formatted files for Gremlin; 4 standard formats for RDF (N-Triples, N-Quads, RDF/XML,
Turtle)
• Load, get status and cancel job via /loader HTTP endpoint
• Append-only
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Interacting with Amazon Neptune SPARQL
• HTTP Endpoint based on SPARQL 1.1 Protocol
• W3C Standard:
https://www.w3.org/TR/sparql11-protocol/
• Use HTTP clients (curl, wget)
• Dedicated SPARQL consoles (e.g. RDF4J console)
• Here: via Jupyter notebook & bash scripts
• Libraries for various query languages
• RDF4J (Java), dotNetRDF, pysparql (Python), ...
• APN and third-party tooling providing GUIs on top of
Neptune SPARQL
Example: Interacting with Neptune using the RDF4J console
(http://rdf4j.org)
Example: Using APN software to build, maintain, and explore
Knowledge Graphs stored in Amazon Neptune
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
First Interaction with Neptune using Jupyter
Hands on!
1. Find SageMaker by typing and selecting “SageMaker” in the main search box.
2. Select ”Notebook Instances” on the left and click the “Open” link for your instance.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
First Interaction with Neptune using Jupyter
Hands on!
3. Select the “Neptune” folder, then select the “nextprot” folder.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
First Interaction with Neptune using Jupyter
Hands on!
4. Select the first notebook, “01-NeptuneAPIs.ipynb”, by clicking on its link.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
First Interaction with Neptune using Jupyter
Hands on!
5. Using the “Play” button you can
execute cells.
Some cells contain text (and are not
executable).
Other cells contain code. They can
be executed and print the result in
the area right behind.
You should follow cells in order
(often they depend on each other),
but you can edit and re-execute
them.
Examples and Applications
66
Wikipedia page A query against Wikipedia
Query the Knowledge of Wikipedia like a Database
66
67
Wikidata for the Life Sciences
68
Current Uptake of Knowledge Graphs
See the panel on Wednesday, October 10. 17:00-18:00
PANEL: ENTERPRISE-SCALE KNOWLEDGE GRAPHS
See the presentation on Friday, October 12, 11:45 in the Industry Track:
AMAZON NEPTUNE: GRAPH DATA MANAGEMENT IN THE CLOUD
Knowledge Graphs for the
Life Sciences and Pharma
74
Demo Wikidata
https://wikidata.metaphacts.com/
75
The Power of Links
76
... Enabling Federated Queries
77
What is NextProt?
78
Nextprot Data Model
79
Sample Graph
80
Demo Nextprot & metaphactory
http://nextprot-neptune.metaphacts.cloud/
Data Loading
82
metaphactory Login
Hands on!
Open the MetaphactoryURL noted down earlier in your browser (recommended: Firefox or Chrome) and log
in using user “admin”, using the MetaphactoryPassword noted down earlier.
83
metaphactory Login
Hands on!
84
Data Loading with metaphactory: Upload Files
Load the Nextprot mini-ontology linked from
http://nextprot-neptune.metaphacts.cloud/
1. In the menu bar, select Quick Links / Data Import & Export
Drag and Drop your file
Hands on!
85
Data Loading with metaphactory: Load from S3
Hands on!
See output of the CloudFormation Stack
Use one of
• s3://nextprot-mini/1
• s3://nextprot-mini/2
• s3://nextprot-mini/3
• s3://nextprot-mini/4
Turtle format
1. In the menu bar, select Quick Links / Data Import & Export
2. Select tab “Neptune Load”, set parameters as shown below
3. Select “Start Load”
86
Data Load Status
Hands on!
87
Explore the Data
You find some sample resources at
http://nextprot-neptune.metaphacts.cloud/
Querying Knowledge Graphs
89
Knowledge Graphs Built on the Semantic Web Layer Cake
Unicode URIs
RDF (Resource Description Framework)
RDF-Schema
OWLSKOS
SPARQL
Query language
Entities
Relationships
Vocabularies
Ontologies
Expressive Ontology
Language
Thesauri,
classification
schemes
Graph data
model
Simple vocabulary
language
90
Our Graph – a bit more technical…
91
Main idea: Pattern matching
• Queries describe sub-graphs of the queried graph
• Graph patterns are RDF graphs specified in Turtle syntax,
which contain variables (prefixed by “?”)
• Sub-graphs that match the graph patterns yield a result
SPARQL Query
?isoformentry:
NX_P20592
:isoform
92
SPARQL Query
?isoform
entry:
NX_P20592
:isoform
entry:
NX_P20592:isoform
isoform:
NX_P20592-1
isoform:
NX_P20592-2
:isoform
Data:
Graph pattern:
Results:
”true" ”false"
:canonicalIsoform :canonicalIsoform
?isoform
isoform:NX_P20592-1
isoform:NX_P20592-2
93
PREFIX : <http://nextprot.org/rdf#>
PREFIX entry: <http://nextprot.org/rdf/entry>
SELECT ?isoform ?canonical
WHERE {
entry:NX_P20592 :isoform ?isoform .
?isoform :canonicalIsoform ?canonical
}
ORDER BY ?isoform
SPARQL Query: Components
Prologue:
• Prefix definitions
94
SPARQL Query: Components
Query form:
• ASK, SELECT, DESCRIBE or CONSTRUCT
• SELECT retrieves variables and their bindings as a table
PREFIX : <http://nextprot.org/rdf#>
PREFIX entry: <http://nextprot.org/rdf/entry>
SELECT ?isoform ?canonical
WHERE {
entry:NX_P20592 :isoform ?isoform .
?isoform :canonicalIsoform ?canonical
}
ORDER BY ?isoform
95
SPARQL Query: Components
Query pattern:
• Defines patterns to match against the data
• Generalises Turtle with variables and keywords – N.B. final
period optional
PREFIX : <http://nextprot.org/rdf#>
PREFIX entry: <http://nextprot.org/rdf/entry>
SELECT ?isoform ?canonical
WHERE {
entry:NX_P20592 :isoform ?isoform .
?isoform :canonicalIsoform ?canonical
}
ORDER BY ?isoform
96
SPARQL Query: Components
Solution modifier:
• Modify the result set
• ORDER BY, LIMIT or OFFSET re-organise rows;
• GROUP BY combines them
PREFIX : <http://nextprot.org/rdf#>
PREFIX entry: <http://nextprot.org/rdf/entry/>
SELECT ?isoform ?canonical
WHERE {
entry:NX_P20592 :isoform ?isoform .
?isoform :canonicalIsoform ?canonical
}
ORDER BY ?isoform
97
SPARQL supports different query forms:
• ASK tests whether or not a query pattern has a solution.
Returns yes/no
• SELECT returns variables and their bindings directly
• CONSTRUCT returns a single RDF graph specified by a
graph template
• DESCRIBE returns a single RDF graph containing RDF
data about resource
Query Forms
98
Sample Path in the Data
99
• The solution modifier projection nominates which
components of the matches should be returned
• “*” means all components should be returned
Query Form: SELECT
Query: Proteins that have a Transit peptide?
PREFIX : <http://nextprot.org/rdf#>
PREFIX cv: <http://nextprot.org/rdf/terminology/>
SELECT ?entry where {
?entry :isoform ?isoform .
?isoform :keyword ?keyword .
?keyword :term cv:KW-0809
}
100
• A property path is a possible route through a graph
between two graph nodes
• Property paths allow for more concise expression of some
SPARQL basic graph patterns and also add the ability to
match arbitrary length paths
Query Form: SELECT
PREFIX : <http://nextprot.org/rdf#>
PREFIX cv: <http://nextprot.org/rdf/terminology/>
SELECT ?entry WHERE{
?entry :isoform / :keyword / :term cv:KW-0809
}
Query: Proteins that have a Transit peptide?
102
Create and Execute your own Queries
• See sample queries from
http://nextprot-neptune.metaphacts.cloud/resource/Queries
103
Query Catalog with Sample Queries
• Use the query catalog from
http://nextprot-neptune.metaphacts.cloud/sparql
Hands on!
104
• Write a SPARQL query that
Returns proteins
• that are associated with heart diseases
• along with their location in the cell
• Save the queries to the query catalog
Exercise: SPARQL Query
Hands on!
105
• Write a SPARQL query that ...
1. Returns isoforms of Hyaluronan synthase 3
2. Finds proteins phosphorylated and located in the cell
membrane
3. Returns proteins with more than 2 intramembrane
regions
4. Returns proteins located in mitochondrion and are
associated with a disease
Optional: Exercises
Hands on!
Visualization of Knowledge Graphs
107
Table
Transform your queries into
durable, interactive tables
Many customization
possibilities, e.g. pagination,
filters and cell templates
Graph
Visualize and explore connections in a graph view
Custom styling of the graph
Variety of graph layouts
Carousel
Animated browsing through a list of
result items
Chart
Visualize trends and
relationships between
numbers, ratios, or
proportions
Tree Table
Tree-based
visualization,
navigation and
browsing through sub-
tree structures
Map
Displaying spatial
data on a
geographic map
Visualization using metaphactory
108
Visualization Components in metaphactory
Example: Proteins in Wikidata
109
HTML Pages with Web Components
See example at http://nextprot-neptune.metaphacts.cloud/resource/Graph
110
1. Open a new page. e.g.
http://ec2-52-71-247-162.compute-1.amazonaws.com/resource/MyPage
2. Click ”Edit page”
3. Paste the example snippet from
http://nextprot-neptune.metaphacts.cloud/resource/Graph
4. Save the page
5. Optional: Extend the query to show also children of child concepts (2 hops)
• Further documentation:
http://help.metaphacts.com/resource/Help:SemanticGraph
Exercise: Create Own Page with Visualization
Hands on!
111
• Associate a class in the knowledge graph with a template
• The template is applied to instances of the class
HTML5 Template Pages
entry:NX_P20592
:Entry
rdf:type
Searching over Knowledge Graphs
113
End-user Oriented Search Processes
Image based on: Tran, T., Herzig, D., Ladwig, G. SemSearchPro- Using semantics through the search process
Knowledge graph Query
Result visualization /
presentation
User query
(e.g. keywords, NL)
Query visualization
(Optional) User
System
Refinement
Presentation
Analysis
Presentation /
Ranking
Graph matching
Entity Extraction /
Semantic query analysis
114
• Search functionalities
• Graphical query construction
• End user friendly search
interfaces for building and
sharing complex queries
• Semantic auto suggestion
• Interactive result visualization
• Faceted search and exploration
of item collections
• Ability to invoke external full text
search indices such as Solr including
the possibility to score, rank and limit
the results for responsive
autosuggestion
Example: Structured Search Generation
115
Translation of keyword queries into SPARQL queries
Example: Keyword-based semantic search
116
Example: Template-based Search
117
Example: Chemical Structure Search
118
Example: Simple Search Component
See example at http://nextprot-neptune.metaphacts.cloud/resource/Search
119
1. Open a new page. e.g.
http://ec2-52-71-247-162.compute-1.amazonaws.com/resource/Search
2. Click “Edit Page”
3. Paste the example snippet from
http://nextprot-neptune.metaphacts.cloud/resource/Search
4. Save the page
• Further documentation:
https://help.metaphacts.com/resource/Help:SemanticSimpleSearch
Exercise: Create Own Page with Search Component
Hands on!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Michael Schmidt
Senior Software Development Engineer, Amazon Neptune
Peter Haase
CEO, metaphacts GmbH
Building APIs on Top Of Knowledge Graphs
ISWC Tutorial:
Building Enterprise-Ready Knowledge Graph Applications in the Cloud
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Motivation: APIs on top of Knowledge Graphs
• Knowledge Graphs (more generally: Semantic Technologies)
strong in interlinking data
• Unified view enables integrated querying across previously
heterogeneous sources
• Often the central value proposition
• Exposing data via SPARQL can be problematic
• Not the most end-user (and often, developer) friendly way
• Contradicts classical 3-tier design to keep applications agnostic from
data storage details
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Query as a Service: Exposing Queries via REST APIs
REST APIs in 3 minutes
Define precise queries and expose them as
REST APIs
Use access permissions to allow for granular
access to specific queries
Choose desired output format
Feed specific result data into external
applications
QaaS
Define a query spanning several data sources
and expose the result through one REST API
Deliver query results as
a service and feed them
into external apps
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Exposing Queries via REST APIs
1. Login to the metaphactory
2. Execute query NXQ_00007 in the
SPARQL servlet and validate it
returns results. Change the URI in
the query body to a variable called
?disease.
3. Save the variant of NXQ_00007 to
the query catalog.
4. Expose the query with parameter
?disease through the query
catalog.
Hands on!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Exposing Queries via REST APIs
5. Go to Administration / Query as a
(REST) Service.
6. Create a new REST Service with
REST URL “disease-protein”, and ACL
permissions “qaas:execute*”.
The query is now exposed via the
context path /rest/qaas/disease-
protein on your metaphactory
endpoint.
Hands on!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Making a Request against the REST API
1. Open the Jupyter
Notebook “03-
DeveloperAPIs”.
2. Execute the first two
cells to load the required
libraries.
3. Scroll down to the “Part
1” section.
Hands on!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Making a Request against the REST API
Hands on!
4. Set the
METAPHACTORY_ENDPOINT
variable and validate the
other variables.
Assuming your REST API
setup was successful,
executing your request
using the Play button at the
top should give you a non-
empty results.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling Throughput Using Neptune Scale-Up
The script should work without
modifications. It …
• … loads a files “params.txt” containing
a list of URIs to be used as params,
• … defines a function
runNeptuneRequest that executes our
sample query with a random
parameter substituted in, and
• … executes this function 3000 times
for warmup, and then 5000 times with
128 threads
Execute the script and note
down the throughput result.
Hands on!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling Throughput Using Neptune Scale-Up
Hands on!
1. Go back to the AWS login page and search for “Neptune” again.
2. Click on “Instances”, select your instance, click “Instance actions” and choose “Create read
replica”.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling Throughput Using Neptune Scale-Up
Hands on!
3. Choose DB Instance class “db.r4.8xlarge”, and provide a new DB instance identifier, such as
“myNewMaster”. Then confirm with “Create read replica”.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling Throughput Using Neptune Scale-Up
Hands on!
4. Wait until your new instance becomes available.
5. We fail over by deleting the old instance. Mark the old
instance and select “Delete” (top).
6. Then type “delete me” in the dialog (right).
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling Throughput Using Neptune Scale-Up
7. Once the instance has been
deleted, re-execute the cell in
your Jupyter throughput script.
What do you observe?
Hands on!
132
• Tutorial material
• http://knowledgegraph.info
• Tutorial repeatable in your own AWS account
• Amazon Neptune
https://aws.amazon.com/neptune/
• metaphactory available on Amazon Marketplace
https://metaphacts.com/amazon-neptune
Links and References

Mais conteúdo relacionado

Mais procurados

Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and OpportunitiesBig Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Srinath Srinivasa
 

Mais procurados (20)

Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...
 
The Bounties of Semantic Data Integration for the Enterprise
The Bounties of Semantic Data Integration for the Enterprise The Bounties of Semantic Data Integration for the Enterprise
The Bounties of Semantic Data Integration for the Enterprise
 
Using the Semantic Web Stack to Make Big Data Smarter
Using the Semantic Web Stack to Make  Big Data SmarterUsing the Semantic Web Stack to Make  Big Data Smarter
Using the Semantic Web Stack to Make Big Data Smarter
 
Discovering Related Data Sources in Data Portals
Discovering Related Data Sources in Data PortalsDiscovering Related Data Sources in Data Portals
Discovering Related Data Sources in Data Portals
 
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and OpportunitiesBig Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
 
Vital.AI Creating Intelligent Apps
Vital.AI Creating Intelligent AppsVital.AI Creating Intelligent Apps
Vital.AI Creating Intelligent Apps
 
Vital AI: Big Data Modeling
Vital AI: Big Data ModelingVital AI: Big Data Modeling
Vital AI: Big Data Modeling
 
Dataverse opportunities
Dataverse opportunitiesDataverse opportunities
Dataverse opportunities
 
Sebastian Hellmann
Sebastian HellmannSebastian Hellmann
Sebastian Hellmann
 
Towards an Open Research Knowledge Graph
Towards an Open Research Knowledge GraphTowards an Open Research Knowledge Graph
Towards an Open Research Knowledge Graph
 
Linked Open Data in the World of Patents
Linked Open Data in the World of Patents Linked Open Data in the World of Patents
Linked Open Data in the World of Patents
 
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and SparkVital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
 
Querying the Wikidata Knowledge Graph
Querying the Wikidata Knowledge GraphQuerying the Wikidata Knowledge Graph
Querying the Wikidata Knowledge Graph
 
An Introduction to Graph: Database, Analytics, and Cloud Services
An Introduction to Graph:  Database, Analytics, and Cloud ServicesAn Introduction to Graph:  Database, Analytics, and Cloud Services
An Introduction to Graph: Database, Analytics, and Cloud Services
 
Connected data meetup group - introduction & scope
Connected data meetup group - introduction & scopeConnected data meetup group - introduction & scope
Connected data meetup group - introduction & scope
 
Edgard Marx, Amrapali Zaveri, Diego Moussallem and Sandro Rautenberg | DBtren...
Edgard Marx, Amrapali Zaveri, Diego Moussallem and Sandro Rautenberg | DBtren...Edgard Marx, Amrapali Zaveri, Diego Moussallem and Sandro Rautenberg | DBtren...
Edgard Marx, Amrapali Zaveri, Diego Moussallem and Sandro Rautenberg | DBtren...
 
Introduction to Property Graph Features (AskTOM Office Hours part 1)
Introduction to Property Graph Features (AskTOM Office Hours part 1) Introduction to Property Graph Features (AskTOM Office Hours part 1)
Introduction to Property Graph Features (AskTOM Office Hours part 1)
 
Nicoletta Fornara and Fabio Marfia | Modeling and Enforcing Access Control Ob...
Nicoletta Fornara and Fabio Marfia | Modeling and Enforcing Access Control Ob...Nicoletta Fornara and Fabio Marfia | Modeling and Enforcing Access Control Ob...
Nicoletta Fornara and Fabio Marfia | Modeling and Enforcing Access Control Ob...
 
How Graph Databases efficiently store, manage and query connected data at s...
How Graph Databases efficiently  store, manage and query  connected data at s...How Graph Databases efficiently  store, manage and query  connected data at s...
How Graph Databases efficiently store, manage and query connected data at s...
 
Choosing the Right Graph Database to Succeed in Your Project
Choosing the Right Graph Database to Succeed in Your ProjectChoosing the Right Graph Database to Succeed in Your Project
Choosing the Right Graph Database to Succeed in Your Project
 

Semelhante a Building Enterprise-Ready Knowledge Graph Applications in the Cloud

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Timothy Spann
 
Feature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningFeature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine Learning
Provectus
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
Timothy Spann
 
Linked Data Platform as a novel approach for Enterprise Application Integra...
Linked Data Platform as a novel approach for Enterprise Application Integra...Linked Data Platform as a novel approach for Enterprise Application Integra...
Linked Data Platform as a novel approach for Enterprise Application Integra...
Nandana Mihindukulasooriya
 
Leveraging Knowledge Graphs in your Enterprise Knowledge Management System
Leveraging Knowledge Graphs in your Enterprise Knowledge Management SystemLeveraging Knowledge Graphs in your Enterprise Knowledge Management System
Leveraging Knowledge Graphs in your Enterprise Knowledge Management System
Semantic Web Company
 

Semelhante a Building Enterprise-Ready Knowledge Graph Applications in the Cloud (20)

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
Delivering a Linked Data warehouse and realising the power of graphs
Delivering a Linked Data warehouse and realising the power of graphsDelivering a Linked Data warehouse and realising the power of graphs
Delivering a Linked Data warehouse and realising the power of graphs
 
Ben Gardner | Delivering a Linked Data warehouse and integrating across the w...
Ben Gardner | Delivering a Linked Data warehouse and integrating across the w...Ben Gardner | Delivering a Linked Data warehouse and integrating across the w...
Ben Gardner | Delivering a Linked Data warehouse and integrating across the w...
 
Spsbepoelmanssharepointbigdataclean 150421080105-conversion-gate02
Spsbepoelmanssharepointbigdataclean 150421080105-conversion-gate02Spsbepoelmanssharepointbigdataclean 150421080105-conversion-gate02
Spsbepoelmanssharepointbigdataclean 150421080105-conversion-gate02
 
How to build your own Delve: combining machine learning, big data and SharePoint
How to build your own Delve: combining machine learning, big data and SharePointHow to build your own Delve: combining machine learning, big data and SharePoint
How to build your own Delve: combining machine learning, big data and SharePoint
 
Open Metadata and Governance with Apache Atlas
Open Metadata and Governance with Apache AtlasOpen Metadata and Governance with Apache Atlas
Open Metadata and Governance with Apache Atlas
 
Strata sf - Amundsen presentation
Strata sf - Amundsen presentationStrata sf - Amundsen presentation
Strata sf - Amundsen presentation
 
Apache atlas sydney 2017-v4
Apache atlas   sydney 2017-v4Apache atlas   sydney 2017-v4
Apache atlas sydney 2017-v4
 
Maruti gollapudi cv
Maruti gollapudi cvMaruti gollapudi cv
Maruti gollapudi cv
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
 
Introduction to APIs and Linked Data
Introduction to APIs and Linked DataIntroduction to APIs and Linked Data
Introduction to APIs and Linked Data
 
Feature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningFeature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine Learning
 
Denodo DataFest 2016: Comparing and Contrasting Data Virtualization With Data...
Denodo DataFest 2016: Comparing and Contrasting Data Virtualization With Data...Denodo DataFest 2016: Comparing and Contrasting Data Virtualization With Data...
Denodo DataFest 2016: Comparing and Contrasting Data Virtualization With Data...
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
 
I2 - SharePoint Hybrid Search Start to Finish - Thomas Vochten
I2 - SharePoint Hybrid Search Start to Finish - Thomas VochtenI2 - SharePoint Hybrid Search Start to Finish - Thomas Vochten
I2 - SharePoint Hybrid Search Start to Finish - Thomas Vochten
 
Linked Data Platform as a novel approach for Enterprise Application Integra...
Linked Data Platform as a novel approach for Enterprise Application Integra...Linked Data Platform as a novel approach for Enterprise Application Integra...
Linked Data Platform as a novel approach for Enterprise Application Integra...
 
Whither the Hadoop Developer Experience, June Hadoop Meetup, Nitin Motgi
Whither the Hadoop Developer Experience, June Hadoop Meetup, Nitin MotgiWhither the Hadoop Developer Experience, June Hadoop Meetup, Nitin Motgi
Whither the Hadoop Developer Experience, June Hadoop Meetup, Nitin Motgi
 
Leveraging Knowledge Graphs in your Enterprise Knowledge Management System
Leveraging Knowledge Graphs in your Enterprise Knowledge Management SystemLeveraging Knowledge Graphs in your Enterprise Knowledge Management System
Leveraging Knowledge Graphs in your Enterprise Knowledge Management System
 
Meetup SF - Amundsen
Meetup SF  -  AmundsenMeetup SF  -  Amundsen
Meetup SF - Amundsen
 
Semantics and Machine Learning
Semantics and Machine LearningSemantics and Machine Learning
Semantics and Machine Learning
 

Mais de Peter Haase

Linked Data as a Service
Linked Data as a ServiceLinked Data as a Service
Linked Data as a Service
Peter Haase
 
Fedbench - A Benchmark Suite for Federated Semantic Data Processing
Fedbench - A Benchmark Suite for Federated Semantic Data ProcessingFedbench - A Benchmark Suite for Federated Semantic Data Processing
Fedbench - A Benchmark Suite for Federated Semantic Data Processing
Peter Haase
 
Everything Self-Service:Linked Data Applications with the Information Workbench
Everything Self-Service:Linked Data Applications with the Information WorkbenchEverything Self-Service:Linked Data Applications with the Information Workbench
Everything Self-Service:Linked Data Applications with the Information Workbench
Peter Haase
 
The Information Workbench as a Self-Service Platform for Linked Data Applicat...
The Information Workbench as a Self-Service Platform for Linked Data Applicat...The Information Workbench as a Self-Service Platform for Linked Data Applicat...
The Information Workbench as a Self-Service Platform for Linked Data Applicat...
Peter Haase
 
Cloud-based Linked Data Management for Self-service Application Development
Cloud-based Linked Data Management for Self-service Application DevelopmentCloud-based Linked Data Management for Self-service Application Development
Cloud-based Linked Data Management for Self-service Application Development
Peter Haase
 
Semantic Technologies for Enterprise Cloud Management
Semantic Technologies for Enterprise Cloud ManagementSemantic Technologies for Enterprise Cloud Management
Semantic Technologies for Enterprise Cloud Management
Peter Haase
 

Mais de Peter Haase (11)

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory
Visual Ontology Modeling for Domain Experts and Business Users with metaphactoryVisual Ontology Modeling for Domain Experts and Business Users with metaphactory
Visual Ontology Modeling for Domain Experts and Business Users with metaphactory
 
Getting Started with Knowledge Graphs
Getting Started with Knowledge GraphsGetting Started with Knowledge Graphs
Getting Started with Knowledge Graphs
 
Mapping, Interlinking and Exposing MusicBrainz as Linked Data
Mapping, Interlinking and Exposing MusicBrainz as Linked DataMapping, Interlinking and Exposing MusicBrainz as Linked Data
Mapping, Interlinking and Exposing MusicBrainz as Linked Data
 
The Information Workbench - Linked Data and Semantic Wikis in the Enterprise
The Information Workbench - Linked Data and Semantic Wikis in the EnterpriseThe Information Workbench - Linked Data and Semantic Wikis in the Enterprise
The Information Workbench - Linked Data and Semantic Wikis in the Enterprise
 
On demand access to Big Data through Semantic Technologies
 On demand access to Big Data through Semantic Technologies On demand access to Big Data through Semantic Technologies
On demand access to Big Data through Semantic Technologies
 
Linked Data as a Service
Linked Data as a ServiceLinked Data as a Service
Linked Data as a Service
 
Fedbench - A Benchmark Suite for Federated Semantic Data Processing
Fedbench - A Benchmark Suite for Federated Semantic Data ProcessingFedbench - A Benchmark Suite for Federated Semantic Data Processing
Fedbench - A Benchmark Suite for Federated Semantic Data Processing
 
Everything Self-Service:Linked Data Applications with the Information Workbench
Everything Self-Service:Linked Data Applications with the Information WorkbenchEverything Self-Service:Linked Data Applications with the Information Workbench
Everything Self-Service:Linked Data Applications with the Information Workbench
 
The Information Workbench as a Self-Service Platform for Linked Data Applicat...
The Information Workbench as a Self-Service Platform for Linked Data Applicat...The Information Workbench as a Self-Service Platform for Linked Data Applicat...
The Information Workbench as a Self-Service Platform for Linked Data Applicat...
 
Cloud-based Linked Data Management for Self-service Application Development
Cloud-based Linked Data Management for Self-service Application DevelopmentCloud-based Linked Data Management for Self-service Application Development
Cloud-based Linked Data Management for Self-service Application Development
 
Semantic Technologies for Enterprise Cloud Management
Semantic Technologies for Enterprise Cloud ManagementSemantic Technologies for Enterprise Cloud Management
Semantic Technologies for Enterprise Cloud Management
 

Último

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
amitlee9823
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
amitlee9823
 
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
amitlee9823
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdf
MarinCaroMartnezBerg
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
amitlee9823
 
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAl Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
AroojKhan71
 

Último (20)

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
 
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
 
Edukaciniai dropshipping via API with DroFx
Edukaciniai dropshipping via API with DroFxEdukaciniai dropshipping via API with DroFx
Edukaciniai dropshipping via API with DroFx
 
Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night
Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 nightCheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night
Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night
 
BigBuy dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptxBigBuy dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptx
 
Discover Why Less is More in B2B Research
Discover Why Less is More in B2B ResearchDiscover Why Less is More in B2B Research
Discover Why Less is More in B2B Research
 
Week-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interactionWeek-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interaction
 
BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort ServiceBDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service
 
ALSO dropshipping via API with DroFx.pptx
ALSO dropshipping via API with DroFx.pptxALSO dropshipping via API with DroFx.pptx
ALSO dropshipping via API with DroFx.pptx
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
 
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdf
 
ELKO dropshipping via API with DroFx.pptx
ELKO dropshipping via API with DroFx.pptxELKO dropshipping via API with DroFx.pptx
ELKO dropshipping via API with DroFx.pptx
 
Midocean dropshipping via API with DroFx
Midocean dropshipping via API with DroFxMidocean dropshipping via API with DroFx
Midocean dropshipping via API with DroFx
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 
CebaBaby dropshipping via API with DroFX.pptx
CebaBaby dropshipping via API with DroFX.pptxCebaBaby dropshipping via API with DroFX.pptx
CebaBaby dropshipping via API with DroFX.pptx
 
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAl Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
 
Anomaly detection and data imputation within time series
Anomaly detection and data imputation within time seriesAnomaly detection and data imputation within time series
Anomaly detection and data imputation within time series
 

Building Enterprise-Ready Knowledge Graph Applications in the Cloud

  • 1. Building Enterprise-Ready Knowledge Graph Applications in the Cloud Peter Haase CEO, metaphacts GmbH Michael Schmidt Senior Software Development Engineer, Amazon Neptune
  • 3. 3 typetype likes likes works forCEO of builds works on builds upon builds works on partner of Knowledge Graphs Person application building on top of fast, reliable, cloud-based storage of likes likes A Knowledge Graph metaphactory
  • 4. 4 COMPANY FACTS • Founded in Q4 2014 • Headquartered in Walldorf, Germany • Currently ~10 people • Platform for knowledge graphs and knowledge graph applications metaphacts at a Glance
  • 5. 5 1. Welcome 2. Introduction to Knowledge Graphs 1. Basic Concepts and Technologies 2. Storing, Querying, and Maintaining Knowledge Graphs 3. Applications of Knowledge Graphs 3. Managing Knowledge Graphs 1. Creating Knowledge Graphs 2. Querying Knowledge Graphs 4. Building End-user Facing Applications on top of Knowledge Graphs 1. Visualization and Templating 2. Semantic Search 5. Building Developer APIs on top of Knowledge Graphs 1. Exposing Queries via REST 2. Scaling up with Neptune Workshop Agenda Hands on! Demo Hands on! Hands on! Hands on! Hands on! Hands on! Demo
  • 6. 6 • Build a portal to make current human proteomic knowledge available • Answer detailed questions about the human proteome • I am researcher working researching heart diseases. I want to know proteins associated with heart diseases. Which of these proteins are located in the cell membrane? • Enable querying via • End user-oriented search • APIs for easy consumption by applications Use Case Scenario
  • 7. 7 • Wikidata • Community curated knowledge graph • Broad coverage • Integration hub for the Life Sciences • Nextprot • A very detailed knowledge graph about human proteomics, aggregated and curated knowledge from many sources • Federation across data sources Knowledge Graphs Used for the Tutorial
  • 8. 8 • Understanding of Knowledge Graphs • Technology Stack • Benefits of Knowledge Graphs • Application Areas of Knowledge Graphs • Setup of Enterprise-ready Knowledge Graph Architecture • Scalable Storage and Querying with Enterprise Features (Amazon Neptune) • Rapid Prototype & Application Building (metaphactory) • Concepts for … • … maintaining, querying, and searching knowledge graphs • … building end-user facing applications on top of knowledge graphs • … building and exposing scalable APIs on top of knowledge graphs • End-to-end experience: throughout the tutorial, you’ll have the chance to set up a small knowledge graph application on your own! Learning Goals
  • 10. 10 Knowledge Graphs - Hype Technology?
  • 11. 11 The Rise of Knowledge Graphs
  • 13. 13 Entity Search and Summarizations Google Knowledge Graph
  • 16. 16 • We need a structured and formal representation of knowledge • We are surrounded by entities, which are connected by relations • Graphs are a natural way to represent entities and their relationships • Graphs can capture a broad spectrum of data from structured to unstructured • Graphs can be managed efficiently Why (Knowledge) Graphs?
  • 17. 17 A (very small) Knowledge Graph http://www.w3.org/TR/2014/NOTE-rdf11-primer-20140225/example-graph.jpg
  • 18. 18 • Semantic descriptions of entities and their relationships • Uses a knowledge representation formalism (RDF, RDF-Schema, OWL) • Entities: real world objects (things, places, people) and abstract concepts (genres, religions, professions) • Relationships: graph-based data model where relationships are first-class • Semantic descriptions: types and properties with a well- defined meaning (e.g. through an ontology) • Possibly axiomatic knowledge (e.g. rules) to support automated reasoning What are Knowledge Graphs?
  • 19. 19 Knowledge Graphs Built on the Semantic Web Layer Cake Unicode URIs RDF (Resource Description Framework) RDF-Schema OWLSKOS SPARQL Query language Entities Relationships Vocabularies Ontologies Expressive Ontology Language Thesauri, classification schemes Graph data model Simple vocabulary language
  • 20. 20 • Linked Data • Set of standards, principles for publishing, sharing and interrelating structured knowledge • From data silos to a Web of Data • Linked Data principles 1. Use URIs as names for things 2. Use HTTP URIs so that people can look up those names. 3. When someone looks up a URI, provide useful information, using the standards: RDF, SPARQL 4. Include links to other URIs, so that they can discover more things. • Linked Data as abstraction layer for virtualized data access across data spaces • Platform independent data access • Enables data portability and interoperability across current data silos Linked Data
  • 22. 22 How do we integrate these resources? Michel Dumontier, Chicago Semantic Web Meetup
  • 23. 23 Uniform Resource Identifier (URI) can be used as entity names http://bio2rdf.org/uniprot:P05067 is a name for Amyloid precursor protein http://bio2rdf.org/omim:104300 is a name for Alzheimer disease uniprot:P05067 omim:104300 Allows one to talk about anything Resource Description Framework (RDF) Michel Dumontier, Chicago Semantic Web Meetup
  • 24. 24 Allows one to express statements An RDF statement consists of: • Subject: resource identified by a URI • Predicate: resource identified by a URI • Object: resource or literal Resource Description Framework (RDF)
  • 25. 25 Protein rdf:type A RDF statement consists of: – Subject: resource identified by a URI – Predicate: resource identified by a URI – Object: resource or literal uniprot:P05067 Allows one to express statements Resource Description Framework (RDF) Michel Dumontier, Chicago Semantic Web Meetup
  • 26. 26 uniprot:P05067 Membrane Protein is a located in uniprot:P05067 uniprot:P05067 uniprot:P05067 interacts with UniProt Gene Ontology uniprot:P05067 is a located in interacts with Unified view + + iRefIndex depends on consistent naming Multi-Source Data Integration Membrane Protein uniprot:P05067 Michel Dumontier, Chicago Semantic Web Meetup
  • 27. 27 Knowledge Graphs Enabling Intelligent Applications Knowledge Graph Algorithms Applications Data Transformation, Integration Natural Language Processing Data Sources • Inferencing • Machine Learning • Entity Recognition • Disambiguation • Text Understanding • Recommendations • Semantic Search • Question Answering • Knowledge Sharing • Knowledge Management • Analytics • Entities • Relationships • Semantic Descriptions • Dashboards
  • 29. 29 Enterprise Knowledge Graphs Span Multiple Data Spaces Advantages of Enterprise Knowledge Graphs • Unlock isolated data silos • Query across data sources • 360° view on data Ephedra – metaphactory’s federation engine Virtual and materialized integration of multiple data sources • Graph databases • Relational databases • Compute services • REST APIs browser browser browser Relational Database OBDA/ETL Graph Database SPARQL 1.1 Hybrid Services REST APIs • R2RML mappings or query patterns • Multiple graph databases • External and internal data • Machine Learning Algorithms • Data Feeds for augmentation
  • 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Michael Schmidt Senior Software Development Engineer, Amazon Neptune Peter Haase, CEO, metaphacts GmbH Storing, Querying, and Maintaining Knowledge Graphs Using Amazon Neptune ISWC Tutorial: Building Enterprise-Ready Knowledge Graph Applications in the Cloud
  • 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Knowledge Graph Applications: Blueprint Architecture
  • 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Account Login Hands on! 1. Log into your AWS account at https://aws.amazon.com/console using the credentials on your hands-out.
  • 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Account Login Hands on! 2. Select “N. Virginia” at the top right as region (it may already be pre-selected). Using the AWS logo link on the top left you can always return to this start page.
  • 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 1. Find CloudFormation by typing and selecting “CloudFormation” in the search box. 2. Click the “Create Stack” button at the top left.
  • 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 3. Select “Specify an Amazon S3 template URL” and type https://s3.amazonaws.com/nextprot-mini/jupyter/metaphactory.json Then confirm using the “Next” button.
  • 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 4. Configure your stack as per above and proceed with the “Next” button. Note: CIDR 0.0.0.0/0 is granting public access to your metaphactory instance world. The instance will still be password protected. Nevertheless, this is NOT a recommended production setting! metaphactory 0.0.0.0/0 db.r4.2xlarge tutorial
  • 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 5 Set a tag “Name” => “metaphactory“ and confirm with the “Next” button.
  • 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 6. Validate your parameters in the “Details” section. 0.0.0.0/0 db.r4.2xlarge tutorial
  • 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 7. Check the box “I acknowledge that AWS CloudFormation might create IAM resources.” and proceed with the “Create” button.
  • 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Neptune & metaphactory Hands on! 8. Congratulations! Your Neptune + metaphactory stack creation is now in progress. It will be ready in about 10 minutes. You can use the refresh button at the top to track progress.
  • 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Neptune: A Fast, Reliable Graph Database Optimized for storing and querying highly connected data
  • 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Highly Connected Data Use Cases Social Networking Recommendations Knowledge Graphs Fraud Detection Life Sciences Network & IT Operations
  • 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Characteristics of a Graph Workload • Complex domain model • Variable schema (per entity) • Optional attributes • Variable structure (across domain) • Highly connected data • Queries require joining entities or navigating relationships • Need to understand that things are connected • Need to understand strength, weight or quality of relationships
  • 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Neptune: A Fully Managed Graph Database Fast Reliable Open Query billions of relationships with millisecond latency 6 replicas of your data across 3 AZs with full backup and restore Build powerful queries easily with Gremlin and SPARQL Supports Apache TinkerPop & W3C RDF graph models Easy
  • 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Leading Graph Models and Frameworks Open Source Apache TinkerPop Gremlin Traversal Language Property Graph Resource Description Framework (RDF) W3C Standard SPARQL Query Language• Vertices and edges (nodes and relationships) with properties • Both record-like items • Imperative traversal language (Gremlin) • Triples • subject – predicate – object • Declarative pattern matching language (SPARQL 1.1)
  • 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Neptune High Level Architecture Bulk load from S3 Database Mgmt.
  • 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Knowledge Graph Applications: Blueprint Architecture
  • 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Knowledge Graph Applications: Blueprint Architecture
  • 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Looking Up the Cluster Endpoint Hands on! 1. Search for and select “Neptune” on the entry page. 2. Click “Clusters” on the left and click on the “DB cluster identifier”.
  • 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Looking Up the Cluster Endpoint Hands on! 3. Save the “Cluster endpoint” in a text file for later use.
  • 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Looking up the metaphactory Output Hands on! Select your metaphactory stack in CloudFormation and click on the ”Outputs” tab. Note down the output variables for later: • NeptuneLoadFromS3RoleArn • MetaphactoryPassword • MetaphactoryURL
  • 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Jupyter via CloudFormation Hands on! 1. Open CloudFormation by typing and selecting “CloudFormation” in the search box. 2. Click the “Create Stack” button at the top left.
  • 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Jupyter via CloudFormation Hands on! 3. Select “Specify an Amazon S3 template URL” and type https://s3.amazonaws.com/nextprot-mini/jupyter/sagemaker-neptune.json Then confirm using the “Next” button.
  • 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Jupyter via CloudFormation Hands on! 4. Fill in the information and confirm with “Next”. jupyter Fill in the NeptuneClusterEndpoint looked up previously. For NeptuneClusterSubnetId and NeptuneClusterVPC choose (one of) the value(s) with “metaphactory” in its name. Fill in the NeptuneLoadFromS3RoleArn noted down after metaphactory setup. Choose (only) value from the dropdown.
  • 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Jupyter via CloudFormation Hands on! 5. Add a tag pair with Key “Name” / Value “jupyter”. Confirm with the “Next” button.
  • 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Setting up Jupyter via CloudFormation Hands on! 6. Validate your data in the “Details” section. Check the “I acknowledge that AWS CloudFormation might create IAM resources” box. Confirm with “Next”.
  • 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Neptune Enterprise Features • Security • Network isolation via Virtual Private Cloud & Security Groups • Encryption at rest using AWS Key Management Service (KMS) • IAM policies to secure resources & control database authentication • Cloud-Native storage • Up to 64TB of available storage (independent from head node instance type) • Data replicated six times across 3 AZs • Continuous backup to S3, built for 11 9s of durability • Backup and Restore • Automated daily and manual snapshots (can be shared with other AWS accounts) • Restoring from DB snapshot creates a new instance • Point-in time restore (from DB instance)
  • 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Loading Data into Neptune Bulk Load API • UTF-8 encoded files in S3 • Supports gzip compression of single files • Requires a VPC endpoint for Amazon S3 – cluster and bucket in same region • Add IAM Role to Neptune allowing s3::Get* and s3::List* permissions for S3 bucket • CSV formatted files for Gremlin; 4 standard formats for RDF (N-Triples, N-Quads, RDF/XML, Turtle) • Load, get status and cancel job via /loader HTTP endpoint • Append-only
  • 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Interacting with Amazon Neptune SPARQL • HTTP Endpoint based on SPARQL 1.1 Protocol • W3C Standard: https://www.w3.org/TR/sparql11-protocol/ • Use HTTP clients (curl, wget) • Dedicated SPARQL consoles (e.g. RDF4J console) • Here: via Jupyter notebook & bash scripts • Libraries for various query languages • RDF4J (Java), dotNetRDF, pysparql (Python), ... • APN and third-party tooling providing GUIs on top of Neptune SPARQL Example: Interacting with Neptune using the RDF4J console (http://rdf4j.org) Example: Using APN software to build, maintain, and explore Knowledge Graphs stored in Amazon Neptune
  • 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. First Interaction with Neptune using Jupyter Hands on! 1. Find SageMaker by typing and selecting “SageMaker” in the main search box. 2. Select ”Notebook Instances” on the left and click the “Open” link for your instance.
  • 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. First Interaction with Neptune using Jupyter Hands on! 3. Select the “Neptune” folder, then select the “nextprot” folder.
  • 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. First Interaction with Neptune using Jupyter Hands on! 4. Select the first notebook, “01-NeptuneAPIs.ipynb”, by clicking on its link.
  • 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. First Interaction with Neptune using Jupyter Hands on! 5. Using the “Play” button you can execute cells. Some cells contain text (and are not executable). Other cells contain code. They can be executed and print the result in the area right behind. You should follow cells in order (often they depend on each other), but you can edit and re-execute them.
  • 66. 66 Wikipedia page A query against Wikipedia Query the Knowledge of Wikipedia like a Database 66
  • 67. 67 Wikidata for the Life Sciences
  • 68. 68 Current Uptake of Knowledge Graphs See the panel on Wednesday, October 10. 17:00-18:00 PANEL: ENTERPRISE-SCALE KNOWLEDGE GRAPHS See the presentation on Friday, October 12, 11:45 in the Industry Track: AMAZON NEPTUNE: GRAPH DATA MANAGEMENT IN THE CLOUD
  • 69. Knowledge Graphs for the Life Sciences and Pharma
  • 76. 80 Demo Nextprot & metaphactory http://nextprot-neptune.metaphacts.cloud/
  • 78. 82 metaphactory Login Hands on! Open the MetaphactoryURL noted down earlier in your browser (recommended: Firefox or Chrome) and log in using user “admin”, using the MetaphactoryPassword noted down earlier.
  • 80. 84 Data Loading with metaphactory: Upload Files Load the Nextprot mini-ontology linked from http://nextprot-neptune.metaphacts.cloud/ 1. In the menu bar, select Quick Links / Data Import & Export Drag and Drop your file Hands on!
  • 81. 85 Data Loading with metaphactory: Load from S3 Hands on! See output of the CloudFormation Stack Use one of • s3://nextprot-mini/1 • s3://nextprot-mini/2 • s3://nextprot-mini/3 • s3://nextprot-mini/4 Turtle format 1. In the menu bar, select Quick Links / Data Import & Export 2. Select tab “Neptune Load”, set parameters as shown below 3. Select “Start Load”
  • 83. 87 Explore the Data You find some sample resources at http://nextprot-neptune.metaphacts.cloud/
  • 85. 89 Knowledge Graphs Built on the Semantic Web Layer Cake Unicode URIs RDF (Resource Description Framework) RDF-Schema OWLSKOS SPARQL Query language Entities Relationships Vocabularies Ontologies Expressive Ontology Language Thesauri, classification schemes Graph data model Simple vocabulary language
  • 86. 90 Our Graph – a bit more technical…
  • 87. 91 Main idea: Pattern matching • Queries describe sub-graphs of the queried graph • Graph patterns are RDF graphs specified in Turtle syntax, which contain variables (prefixed by “?”) • Sub-graphs that match the graph patterns yield a result SPARQL Query ?isoformentry: NX_P20592 :isoform
  • 89. 93 PREFIX : <http://nextprot.org/rdf#> PREFIX entry: <http://nextprot.org/rdf/entry> SELECT ?isoform ?canonical WHERE { entry:NX_P20592 :isoform ?isoform . ?isoform :canonicalIsoform ?canonical } ORDER BY ?isoform SPARQL Query: Components Prologue: • Prefix definitions
  • 90. 94 SPARQL Query: Components Query form: • ASK, SELECT, DESCRIBE or CONSTRUCT • SELECT retrieves variables and their bindings as a table PREFIX : <http://nextprot.org/rdf#> PREFIX entry: <http://nextprot.org/rdf/entry> SELECT ?isoform ?canonical WHERE { entry:NX_P20592 :isoform ?isoform . ?isoform :canonicalIsoform ?canonical } ORDER BY ?isoform
  • 91. 95 SPARQL Query: Components Query pattern: • Defines patterns to match against the data • Generalises Turtle with variables and keywords – N.B. final period optional PREFIX : <http://nextprot.org/rdf#> PREFIX entry: <http://nextprot.org/rdf/entry> SELECT ?isoform ?canonical WHERE { entry:NX_P20592 :isoform ?isoform . ?isoform :canonicalIsoform ?canonical } ORDER BY ?isoform
  • 92. 96 SPARQL Query: Components Solution modifier: • Modify the result set • ORDER BY, LIMIT or OFFSET re-organise rows; • GROUP BY combines them PREFIX : <http://nextprot.org/rdf#> PREFIX entry: <http://nextprot.org/rdf/entry/> SELECT ?isoform ?canonical WHERE { entry:NX_P20592 :isoform ?isoform . ?isoform :canonicalIsoform ?canonical } ORDER BY ?isoform
  • 93. 97 SPARQL supports different query forms: • ASK tests whether or not a query pattern has a solution. Returns yes/no • SELECT returns variables and their bindings directly • CONSTRUCT returns a single RDF graph specified by a graph template • DESCRIBE returns a single RDF graph containing RDF data about resource Query Forms
  • 94. 98 Sample Path in the Data
  • 95. 99 • The solution modifier projection nominates which components of the matches should be returned • “*” means all components should be returned Query Form: SELECT Query: Proteins that have a Transit peptide? PREFIX : <http://nextprot.org/rdf#> PREFIX cv: <http://nextprot.org/rdf/terminology/> SELECT ?entry where { ?entry :isoform ?isoform . ?isoform :keyword ?keyword . ?keyword :term cv:KW-0809 }
  • 96. 100 • A property path is a possible route through a graph between two graph nodes • Property paths allow for more concise expression of some SPARQL basic graph patterns and also add the ability to match arbitrary length paths Query Form: SELECT PREFIX : <http://nextprot.org/rdf#> PREFIX cv: <http://nextprot.org/rdf/terminology/> SELECT ?entry WHERE{ ?entry :isoform / :keyword / :term cv:KW-0809 } Query: Proteins that have a Transit peptide?
  • 97. 102 Create and Execute your own Queries • See sample queries from http://nextprot-neptune.metaphacts.cloud/resource/Queries
  • 98. 103 Query Catalog with Sample Queries • Use the query catalog from http://nextprot-neptune.metaphacts.cloud/sparql Hands on!
  • 99. 104 • Write a SPARQL query that Returns proteins • that are associated with heart diseases • along with their location in the cell • Save the queries to the query catalog Exercise: SPARQL Query Hands on!
  • 100. 105 • Write a SPARQL query that ... 1. Returns isoforms of Hyaluronan synthase 3 2. Finds proteins phosphorylated and located in the cell membrane 3. Returns proteins with more than 2 intramembrane regions 4. Returns proteins located in mitochondrion and are associated with a disease Optional: Exercises Hands on!
  • 102. 107 Table Transform your queries into durable, interactive tables Many customization possibilities, e.g. pagination, filters and cell templates Graph Visualize and explore connections in a graph view Custom styling of the graph Variety of graph layouts Carousel Animated browsing through a list of result items Chart Visualize trends and relationships between numbers, ratios, or proportions Tree Table Tree-based visualization, navigation and browsing through sub- tree structures Map Displaying spatial data on a geographic map Visualization using metaphactory
  • 103. 108 Visualization Components in metaphactory Example: Proteins in Wikidata
  • 104. 109 HTML Pages with Web Components See example at http://nextprot-neptune.metaphacts.cloud/resource/Graph
  • 105. 110 1. Open a new page. e.g. http://ec2-52-71-247-162.compute-1.amazonaws.com/resource/MyPage 2. Click ”Edit page” 3. Paste the example snippet from http://nextprot-neptune.metaphacts.cloud/resource/Graph 4. Save the page 5. Optional: Extend the query to show also children of child concepts (2 hops) • Further documentation: http://help.metaphacts.com/resource/Help:SemanticGraph Exercise: Create Own Page with Visualization Hands on!
  • 106. 111 • Associate a class in the knowledge graph with a template • The template is applied to instances of the class HTML5 Template Pages entry:NX_P20592 :Entry rdf:type
  • 108. 113 End-user Oriented Search Processes Image based on: Tran, T., Herzig, D., Ladwig, G. SemSearchPro- Using semantics through the search process Knowledge graph Query Result visualization / presentation User query (e.g. keywords, NL) Query visualization (Optional) User System Refinement Presentation Analysis Presentation / Ranking Graph matching Entity Extraction / Semantic query analysis
  • 109. 114 • Search functionalities • Graphical query construction • End user friendly search interfaces for building and sharing complex queries • Semantic auto suggestion • Interactive result visualization • Faceted search and exploration of item collections • Ability to invoke external full text search indices such as Solr including the possibility to score, rank and limit the results for responsive autosuggestion Example: Structured Search Generation
  • 110. 115 Translation of keyword queries into SPARQL queries Example: Keyword-based semantic search
  • 113. 118 Example: Simple Search Component See example at http://nextprot-neptune.metaphacts.cloud/resource/Search
  • 114. 119 1. Open a new page. e.g. http://ec2-52-71-247-162.compute-1.amazonaws.com/resource/Search 2. Click “Edit Page” 3. Paste the example snippet from http://nextprot-neptune.metaphacts.cloud/resource/Search 4. Save the page • Further documentation: https://help.metaphacts.com/resource/Help:SemanticSimpleSearch Exercise: Create Own Page with Search Component Hands on!
  • 115. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Michael Schmidt Senior Software Development Engineer, Amazon Neptune Peter Haase CEO, metaphacts GmbH Building APIs on Top Of Knowledge Graphs ISWC Tutorial: Building Enterprise-Ready Knowledge Graph Applications in the Cloud
  • 116. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Motivation: APIs on top of Knowledge Graphs • Knowledge Graphs (more generally: Semantic Technologies) strong in interlinking data • Unified view enables integrated querying across previously heterogeneous sources • Often the central value proposition • Exposing data via SPARQL can be problematic • Not the most end-user (and often, developer) friendly way • Contradicts classical 3-tier design to keep applications agnostic from data storage details
  • 117. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Query as a Service: Exposing Queries via REST APIs REST APIs in 3 minutes Define precise queries and expose them as REST APIs Use access permissions to allow for granular access to specific queries Choose desired output format Feed specific result data into external applications QaaS Define a query spanning several data sources and expose the result through one REST API Deliver query results as a service and feed them into external apps
  • 118. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Exposing Queries via REST APIs 1. Login to the metaphactory 2. Execute query NXQ_00007 in the SPARQL servlet and validate it returns results. Change the URI in the query body to a variable called ?disease. 3. Save the variant of NXQ_00007 to the query catalog. 4. Expose the query with parameter ?disease through the query catalog. Hands on!
  • 119. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Exposing Queries via REST APIs 5. Go to Administration / Query as a (REST) Service. 6. Create a new REST Service with REST URL “disease-protein”, and ACL permissions “qaas:execute*”. The query is now exposed via the context path /rest/qaas/disease- protein on your metaphactory endpoint. Hands on!
  • 120. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Making a Request against the REST API 1. Open the Jupyter Notebook “03- DeveloperAPIs”. 2. Execute the first two cells to load the required libraries. 3. Scroll down to the “Part 1” section. Hands on!
  • 121. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Making a Request against the REST API Hands on! 4. Set the METAPHACTORY_ENDPOINT variable and validate the other variables. Assuming your REST API setup was successful, executing your request using the Play button at the top should give you a non- empty results.
  • 122. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling Throughput Using Neptune Scale-Up The script should work without modifications. It … • … loads a files “params.txt” containing a list of URIs to be used as params, • … defines a function runNeptuneRequest that executes our sample query with a random parameter substituted in, and • … executes this function 3000 times for warmup, and then 5000 times with 128 threads Execute the script and note down the throughput result. Hands on!
  • 123. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling Throughput Using Neptune Scale-Up Hands on! 1. Go back to the AWS login page and search for “Neptune” again. 2. Click on “Instances”, select your instance, click “Instance actions” and choose “Create read replica”.
  • 124. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling Throughput Using Neptune Scale-Up Hands on! 3. Choose DB Instance class “db.r4.8xlarge”, and provide a new DB instance identifier, such as “myNewMaster”. Then confirm with “Create read replica”.
  • 125. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling Throughput Using Neptune Scale-Up Hands on! 4. Wait until your new instance becomes available. 5. We fail over by deleting the old instance. Mark the old instance and select “Delete” (top). 6. Then type “delete me” in the dialog (right).
  • 126. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling Throughput Using Neptune Scale-Up 7. Once the instance has been deleted, re-execute the cell in your Jupyter throughput script. What do you observe? Hands on!
  • 127. 132 • Tutorial material • http://knowledgegraph.info • Tutorial repeatable in your own AWS account • Amazon Neptune https://aws.amazon.com/neptune/ • metaphactory available on Amazon Marketplace https://metaphacts.com/amazon-neptune Links and References