Machine Learning on Knowledge Graphs: a quick and incomplete 15-min tour of knowledge graph embeddings

1. MachineLearningonKnowledgeGraphs
AQuickTourofKnowledgeGraphEmbeddings
LucaCostabello,AccentureLabsDublin
@lukostaz

2. KnowledgeGraph G
G = {(s, p, o)} ⊆ E × R × E
E : setofentitiesofG R : setofrelationsofG 2

3. SomeExamples
KnowledgeGraph Statements Entities
120M 10M
610M 51M
1.3B 6M
3.5B 364M
3

4. Applications
KnowledgeDiscovery
Question‑answeringsystems
Identify&disambiguateentitiesintext
Domain‑specificdecisionsupportsystems
4

5. Applications
KnowledgeDiscovery
Question‑answeringsystems
Identify&disambiguateentitiesintext
Domain‑specificdecisionsupportsystems
5

6. KnowledgeDiscovery:LinkPrediction
6

7. KnowledgeDiscovery:LinkPrediction
Predictprobabilityofexistenceofcandidatestatements
7

8. KnowledgeGraphEmbeddingModels
Automatic,supervisedlearningofembeddings,i.e.projectionsof
entitiesandrelationsintoacontinuouslow‑dimensionalspace.
8

9. [Cai17]
9

10. [Cai17]
10

11. ModelRequirements
Expressiveness
ModelKGpropertiesandregularitiesasmuchaspossible
(symmetry,antisymmetry,inversion,composition,etc.)
Space/timecomplexity
O(N ),O(N ),O(N ) intime&memory
Bothrequirementsareequallyimportant!
KnowledgeGraphEmbeddingsstrikeagoodtrade‑off.
E R G
11

12. AnatomyofaKGEmbeddingModel
KnowledgeGraph(KG)G
Negativesgenerationstrategy
Scoringfunctionforatriplef(t)
LossfunctionL
Optimizationalgorithm
12

13. Negativesgenerationstrategies
Uniformsampling:GenerateanduniformlysamplecorruptionsC:
C((s, p, o)) = {( , p, o)∣ ∈ E} ∪ {(s, p, )∣ ∈ E}
Basedongenerativeadversarialnetwork[Cai17]
Self‑adversarialnegativesampling[Anonymous19]
s^ s^ o^ o^
13

14. Example(Uniformsampling)
E = {Mike, Liverpool, AcmeInc, George, LiverpoolFC}
R = {bornIn, friendWith}
t ∈ G = (Mike bornIn Liverpool)
C =
Mike bornIn AcmeInc
Mike bornIn LiverpoolFC
George bornIn Liverpool
AcmeInc bornIn Liverpool
i
ti
14

15. Scoringfunctionf
f assignsascoretoatriple(s, p, o):
TransE:TranslatingEmbeddings[Bordes13]
f = −∣∣(e +r ) −e ∣∣TransE s p o n
15

16. [Anonymous19]
16

17. LossfunctionL
Pairwisemargin‑based:
L(Θ) = max(0, [γ + f(t ; Θ) − f(t ; Θ)])
Negativelog‑likelihood
...
t ∈G+
∑
t ∈C−
∑ − +
17

18. OptimizationProcedure
EstimatetheoptimalparametersΘ with(samevariantof)SGD:
L(Θ)
On‑the‑flynegativesgenerationwiththenegativesstrategy
min
Θ
18

19. EvaluationMetrics
LearningtoRankmetrics
Howwellarepositivetriplesrankedagainsttheircorruptions?
Hits@N = 1 if rank ≤ N
MR = rank [MeanRank]
MRR = [MeanReciprocalRank]
∣Q∣
1
∑i=1
∣Q∣
(s,p,o)i
∣Q∣
1
∑i=1
∣Q∣
(s,p,o)i
∣Q∣
1
∑i=1
∣Q∣
rank(s,p,o)i
1
19

20. Example
s p o score rank
Mike bornIn AcmeInc 0.789 1
Mike bornIn Liverpool 0.753 2 *
Mike bornIn LiverpoolFC 0.695 3
George bornIn Liverpool 0.695 3
AcmeInc bornIn Liverpool 0.234 5
s p o score rank
George worksFor AcmeInc 0.901 1 *
George worksFor Liverpool 0.345 2
George worksFor LiverpoolFC 0.293 3
George worksFor Mike 0.201 4
LiverpoolFC worksFor AcmeInc 0.156 5
Hits@3 = 1 if rank ≤ 3 = 1
Hits@1 = 1 if rank ≤ 1 = 0.5
2
1
∑i=1
2
(s,p,o)i
2
1
∑i=1
2
(s,p,o)i
20

21. LatestResults
[Anonymous19]
21

22. OpenResearchDirections
Howtodesignmoreexpressivemodels?
Howtosupporttime?
Howtocomeupwithexplainableresults?
Howtoinjectbackgroundknowledge?(i.e.text)
22

23. Questions?
LucaCostabello,AccentureLabsDublin
@lukostaz
23

24. References
[Anonymous19]Anonymous,"RotatE:KnowledgeGraphEmbeddingbyRelationalRotationin
ComplexSpace",SubmittedtoICLR2019(underreview).
[Bordes13]Bordesetal."Translatingembeddingsformodelingmulti‑relationaldata.",NIPS
2013.
[Cai17]Cai,Hongyunetal."AComprehensiveSurveyofGraphEmbedding:Problems,
TechniquesandApplications",IEEETrans.OnKnowledgeandDataEng,2017
[Dettmers18]Dettmers,etal."Convolutional2DKnowledgeGraphEmbeddings",AAAI2018
[Nickel15]Nickeletal."Areviewofrelationalmachinelearningforknowledgegraphs."Procsof
theIEEE104.1,2016
[Nickel17]Nickeletal."HolographicEmbeddingsofKnowledgeGraphs."AAAI2016
[Troullion17]Trouillon,Theo,etal."KnowledgeGraphCompletionviaComplexTensor
Factorization.",ICML2017.
[Yang15]BishanYang,etal."Embeddingentitiesandrelationsforlearningandinferencein
knowledgebases",ICLR2015 24