O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Combining semantics an deep learning for intelligent information services

1.368 visualizações

Publicada em

Inaugural Lecture at AIFB KIT Karlsruhe, Nov 29, 2017

Publicada em: Educação

Combining semantics an deep learning for intelligent information services

  1. 1. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017 Combining Semantics and Deep Learning for Intelligent Information Services Antrittsvorlesung Prof. Dr. Harald Sack AIFB, 29.11.2017
  2. 2. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20172 http://www.telegraph.co.uk/science/2017/10/ 18/alphago-zero-google-deepmind-supercomp uter-learns-3000-years/
  3. 3. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20173 The Futile Tries of “Strong” AI 20 years of “AI Winter”... https://www.flickr.com/photos/x-ray_delta_one/4128131032 "in from three to eight years we will have a machine with the general intelligence of an average human being", Marvin Minsky (1970)
  4. 4. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017 Inspired by Biology...
  5. 5. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20175 https://www.flickr.com/photos/x-ray_delta_one/4128131032 https://commons.wikimedia.org/wiki/File:Blausen_0657_MultipolarNeuron.png From Biological Neuron to the Artificial Neuron Modell - McCulloch & Pitts (1943) (Dendrites) (Soma) (Axon)
  6. 6. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20176 Cognitive Computing - The MARK 1 Perceptron (1957) http://techgenix.com/tgwordpress/wp-content/uploads/2017/01/perceptron.jpg weight update error
  7. 7. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20177 Timeline of Neural Networks https://www.slideshare.net/deview/251-implementing-deep-learning-using-cu-dnn/4
  8. 8. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20178 https://www.flickr.com/photos/x-ray_delta_one/4128131032 Deep Convolutional Neural Networks on GPU Supercomputers
  9. 9. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20179 Visual Concept Detection = the ability to learn visual categories in order to automatically identify new, unseen images of these categories only based on visual content https://pixabay.com/p-2175353/ https://cloud.google.com/vision/
  10. 10. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201710 Visual Concept Detection as Machine Learning Task Supervised Learning: ● Positive images (that depict the concept) ● Negative images (that don’t) ● Classification/Prediction: ○ Test image, if it depicts concept (or not): ??
  11. 11. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201711 Size Matters To achieve high quality results, we need sufficient training data ● Influence of training data size on classification accuracy for ○ Deep Convolutional Neural Networks (CNN) vs. ○ Aggregated local features and linear predictor (IFV) ● Results: ○ CNNs largely benefit from bigger datasets ○ IFVs are a competitive candidate esp. if only limited training data is available C. Hentschel, T. Wiradarma, H. Sack: If we did not have imagenet: Comparison of fisher encodings and convolutional neural networks on limited training data (AVC 2016)
  12. 12. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201712 Leveraging Social Media to Improve Visual Content Detection Extending MIRFLICKR-1M ● 1 Million Flickr images (selection based on interestingness score) ● Additional image metadata (authoritative & user created): text ● How to select appropriate training data? ○ Text: word2vec skip-gram model to determine related (similar) tags for a given query ○ Images: visual reranking to filter images visually similar to top ranked images C. Hentschel, H. Sack: Learning from the Uncertain -- Improving Image Classifiers with Community Training Data (i-KNOW 2015)
  13. 13. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201713 What Do Classifiers Really See? ● Heatmaps representing the influence of an image region on the classification result C. Hentschel and H. Sack, What Image Classifiers Really See – Visualizing Bag-of-Visual Words Models (MMM 2015) Aggregated local features and linear predictor (IFV) Deep Convolutional Neural Networks (CNN)
  14. 14. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201714 ● Near-human level image classification ● Near-human level speech recognition ● Near-human level handwriting transcription ● Improved machine translation ● Improved text-to-speech conversion ● Digital assistants such as Google Now or Amazon Alexa ● Near-human level autonomous driving ● Superhuman Go playing What Deep Learning has achieved so far https://media.wired.com/photos/59268c8ccfe0d93c474309b2/master/pass/GettyImages-627219854.jpg
  15. 15. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017 How to Represent Knowledge?
  16. 16. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201716 https://upload.wikimedia.org/wikipedia/commons/c/ce/Gottfried_Wilhelm_Leibniz%2C_Bernhard_Christoph_Francke.jpg The Universal Categories - Aristotle (384–322 BC)
  17. 17. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201717 https://upload.wikimedia.org/wikipedia/commons/c/ce/Gottfried_Wilhelm_Leibniz%2C_Bernhard_Christoph_Francke.jpg Calculemus! Calculus Ratiocinator - Gottfried Wilhelm Leibniz (1646-1716) „..alle menschlichen Schlussfolgerungen müssten auf irgendeine mit Zeichen arbeitende Rechnungsart zurückgeführt werden, wie es sie in der Algebra und Kombinatorik und mit den Zahlen gibt, wodurch nicht nur mit einer unzweifelhaften Kunst die menschliche Erfindungsgabe gefördert werden könnte, sondern auch viele Streitigkeiten beendet werden könnten, das Sichere vom Unsicheren unterschieden und selbst die Grade der Wahrscheinlichkeiten abgeschätzt werden könnten, da ja der eine der im Disput Streitenden zum anderen sagen könnte: Lasst uns doch nachrechnen!“ Leibniz in a letter to Ph. J. Spener, Juli 1687
  18. 18. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201718 Begriffsschrift - Gottlob Frege (1848-1925)
  19. 19. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201719 https://www.flickr.com/photos/x-ray_delta_one/4128131032 Frames for Represent Knowledge - Marvin Minsky (1974)
  20. 20. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201720 The Renaissance of “Soft” AI Carol Kaelson/Jeopardy Productions Inc., via Associated Press From Linked Data to Knowledge Graphs
  21. 21. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201721 Knowledge Graphs for Natural Language Processing rdf:type dbo:Philosopher rdfs:subClassOf dbo:Person 21 Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz. dbr:Gottfried_Willhelm_Leibniz dbr:Christian_Wolff dbo:doctoralAdvisor dbo:Philosopher rdf:type dbr:Ontology dbo:notableIdea text knowledge base foaf:name dbo:birthDate “Gottfried Wilhelm Leibniz“@de “1646-07-01”^^xsd:date “1716-11-14”^^xsd:date dbo:deathDate
  22. 22. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201722 Knowledge Graphs for Natural Language Processing 22 Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz. text Named Entities Common Entities Named Entity Linking Language Model ● Statistical Context Analysis (co-occurrence) Knowledge Graph ● Graph Analysis (connected components)
  23. 23. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201723 Knowledge Graphs for Natural Language Processing 23 Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz. 1. Create potential entity candidates 2. Filter entity candidates by NER type 3. Create induced subgraph of knowledge graph 4. Determine connected components N. Steinmetz, H. Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions (ESWC 2013) J. Waitelonis, H. Sack, Named Entity Linking in #Tweets with KEA, (Microposts 2016)
  24. 24. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201724 Knowledge Graphs for Question Answering “Where was Leibniz born?” Entity Linking wd:Q9047 (Gottfried Wilhelm Leibniz) Result Type wd:Q618123 (geographical object) Relation Extraction wdt:P19 (place of birth) Natural Language
  25. 25. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201725 Knowledge Graphs for Question Answering “Where was Leibniz born?” Entity Linking Result Type Relation Extraction wdt:P19 (place of birth) wd:Q9047 (Gottfried Wilhelm Leibniz) Natural Language wd:Q618123 (geographical object) SPARQL Query SELECT ?o WHERE { wd:Q9047 wdt:P19 ?o . ?o wdt:P31/wdt:P279* wd:Q618123 . }
  26. 26. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201726
  27. 27. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201727 What Knowledge Graphs have achieved so far ● Improved search results on the web ● Answering natural language questions ● Suggest content-based recommendations ● Enable serependitious discoveries ● Enables exploratory search ● Large scale data integration ● Still missing: common sense knowledge
  28. 28. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017 How to combine Deep Learning and Semantics?
  29. 29. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201729 ▶ Deep Learning for Knowledge Graphs ▶ ● NLP and Knowledge Extraction via Deep Learning to populate and extend Knowledge Graphs ● NLP and Knowledge Extraction via Deep Learning for Ontology Learning to extend and refine Knowledge Graphs ● NLP and Graph Analysis supported by Deep Learning for Ontology Alignment and Link Discovery to combine and integrate Knowledge Graphs ◀ Knowledge Graphs for Deep Learning ? ◀
  30. 30. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201730 Word Embeddings (word2vec, glove) Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz. 0.286 0.792 … −0.177 −0.10 0.109 −0.542 … 0.349 0.271 ● Words are represented as vectors that preserve the linguistic context ● Semantically similar words are represented in close neighborhood within the vector space ● Enable analogies via vector arithmetics T. Mikolov et al., Efficient Estimation of Word Representations in Vector Space, archivx 2013
  31. 31. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201731 Knowledge Graph Embeddings (rdf2vec) 0.286 0.792 … −0.177 −0.10 0.109 −0.542 … 0.349 0.271 ● RDF graph are represented as vectors that preserve the semantic context ● Semantically similar entities are represented in close neighborhood within the vector space ● Enable analogies via vector arithmetics dbr:Gottfried_Wilhelm_Leibniz dbr:Erhard_Weigel dbr:Gottlob_Frege dbr:Hanover dbr:Leipzig dbc:German_Mathematician dbo:academicAdvisor dbo:influenced dct:subject dct:subject dct:subject dbr:University_of_Leipzig dbo:almaMater dbo:city dbo:birthPlace dbo:deathPlace P. Ristovski et al., RDF2Vec: RDF Graph Embeddings and Their Applications, SWJ 2016 M. Cochez et al, Global RDF Vector Space Embeddings, ISWC 2017
  32. 32. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201732 Combined Feature Embeddings for a Compound Knowledge Space ● Various feature vectors ○ Word embeddings ○ Knowledge Graph embeddings ■ Instances ■ Ontologies ○ Embeddings for semantically enriched texts ○ Metadata and aggregated features 0.271 0.123 -0.24 -0.286 0.792 … −0.177 −0.10 0.109 −0.542 … 0.349 0.56 -0.132 0.113 0.91 ... 0.56 0.99 0.271 0.123 -0.24 -0.334 Text space Knowledge space Context space Compound Knowledge Space
  33. 33. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201733 Towards Neuro-Symbolic Integration Neuro-Symbolic Systems 1. Translation of symbolic (background) knowledge into the network 2. Learning of additional knowledge from examples (and generalisation) by the network 3. Executing the network (i.e. reasoning), and 4. Symbolic knowledge extraction from the network. Network ensembles Levels of abstraction Besold et al.: Neural-Symbolic Learning and Reasoning: A Survey and Interpretation (2017) specialization
  34. 34. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017 Deep Learning and Semantics for Information Services
  35. 35. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201735T. Tietz, J. Waitelonis, J. Jäger, H. Sack, refer: a Linked Data based Text Annotation and Recommender System for Wordpress, (ISWC 2016)
  36. 36. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201736 From Information Retrieval to Information Exploration T. Tietz, J. Jäger, J. Waitelonis, H. Sack, Semantic Annotation and Information Visualization for Blogposts with refer, (VOILA 2016)
  37. 37. Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017 Prof. Dr. Harald Sack FIZ Karlsruhe, Leibniz Institute for Information Infrastructure AIFB, KIT Karlsruhe Combining Semantics and Deep Learning for Intelligent Information Services

×