A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration

A Semantics-based UI Model
for Content Annotation,
Authoring & Exploration
Faculty of Mathematics and Computer Science
University of Leipzig
Ali Khalili
Supervisors:
PhD Defense
Prof. Dr. Ing. habil. Klaus-Peter Fähnrich, University of Leipzig
Prof. Dr. Sören Auer, University of Bonn
January 26, 2015

Outline
A Semantics-based UI Model for Content Annotation, Authoring & Exploration 2Ali Khalili
Introduction
Research Questions
Contributions
State-of-the-art Analysis
WYSIWYM User Interface Model
From WYSIWYG to WYSIWYM
Lightweight Text Analytics
Authoring of E-Learning Content
Authoring of Semantic Medical Prescriptions
Conclusion
Publications & Awards
PhD DefenseJanuary 26, 2015

Introduction
User Interface & Applications
A Semantics-based User Interface Model for
Content Annotation, Authoring & Exploration

Motivation - User Scenario
Introduction
Extracting the most demanded Data Science skills for IT jobs in Europe
Alice
(Journalist)
Job Posting
Websites
TextCo
Text AnalysisAnalysis Results
Job Poster
UIs for Semantic Content Exploration
UIs for Semantic Content Authoring

Difficult
Time-consuming
Error-prone
Requires knowledge
representation expertise
Challenges
Introduction
Semantic Content Authoring (SCA) is cumbersome!
There is a lack of approaches, technologies and
tools to facilitate collaboration.
There is a lack of incentives and instant gratiﬁcation
for users to adopt SCA.
There is a lack of standardization of UI technologies in
the domain of SCA.

Research Questions
Introduction
How can we enable user-friendly manual and semi-
automatic creation of rich semantic content?
What are existing approaches for user-friendly semantic content authoring?
How to classify existing approaches for semantic content authoring?
What types of user interfaces are used by each approach?
What are the features supported by the proposed user interfaces?
What types of users are targeted in each approach?
How is the user interface evaluated?
How can we bind user interface elements to semantic representation data models?
How can we integrate semantic content authoring features into the current authoring tools on the Social Web?
How can we exploit semantically-enriched content for content analysis?
Can we utilize semantic content benefits to incentivize users contributing to the process of manual semantic content authoring?
How can we apply crowdsourcing & collaborative content authoring techniques to
the process of semantic content authoring?
How can we apply semantic content authoring to a domain-specific use case for
achieving content interoperability?

Contributions
Introduction
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
Lightweight text Analytics
WYSIWYM for Authoring
of E-Learning Content
State of the Art Analysis
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
content interoperability
RDFaCE
SlideWiki
conTEXT
Pharmer
- Approaches for Semantic Content
Authoring (SCA)
- A set of UI types & properties for
creating efficient SCA systems
- Different views for content
authoring
- Combining NLP services for
automatic content annotation
- Bindings between semantic representation
data models & UI elements for content
visualization, exploration & authoring
- Intelligent & Interoperable
medical prescriptions
- Text analytics based on
Linked Data services
- Incorporating user feedback
for NLP
- WikiApp data model
- Crowdlearning

Contributions
Introduction
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
RDFaCE
SlideWiki
conTEXT
Pharmer
Authoring (SCA)
authoring
for NLP
- Crowdlearning
[Khalili and Auer, 2013a]

Approaches for Semantic Content Authoring
deliverable
Nature
deliveryDate
leading
partner
work
package
label
seeAlso
Deliverable
LOD2
Ontology
Article
Person
...
...
...
Domain
Ontology
Location
Organization
Person
label
label
label
...
...
...
...
Top-Down
Bottom-Up
OntologiesContent
Ontology Population
State of the Art
RDF
OWL
Semantic Markup
/ Annotation
RDFa
Microdata
Microformats

Quality Attributes for SCA System Design
Quality Attribute Realization
Usability
Customizability
Generalizability
Collaboration
Portability
Accessibility
Proactivity
Automation
Evolvability
Evolvability
Scalability
State of the Art

WYSIWYM UI Model
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
RDFaCE
SlideWiki
conTEXT
Pharmer
Authoring (SCA)
authoring
for NLP
- Crowdlearning
[Khalili and Auer, 2014]

What You See Is What You Get - WYSIWYG
WYSIWYM UI Model

What You See Is What You Mean - WYSIWYM
User Interface
Semantics
WYSIWYM UI Model

WYSIWYM Model
WYSIWYM UI Model

Semantic Representation Data Models
WYSIWYM UI Model
Tree-based
(e.g. Taxonomies)
Graph-based
(e.g. RDF)
Hypergraph-based
(e.g. Topic Maps)
Item
Itemtype
Item-subitem
Itempropertyvalue
RelatedItems
Instance
Class
Relationshipsbetween
entities
Literal property
values
Topic
Topictype
Topicassociations
Topicroleinassociation
Topic
Occurr
ences
Value
Languagetag
Datatype
Value
Datatype

Visualization Techniques
WYSIWYM UI Model
Structure encoded in: UI categories UI techniques
Visualization
text
Highlighting
Framing and segmentation (borders, overlays,
backgrounds)
Text formatting (color, font, size etc.)
Marking (appended icons)
Associating
Line connectors
Arrow connectors
Detail view
Callouts
(infotips, tooltips, popups)
images
Highlighting
backgrounds)
Image color effects
Associating
Line connectors
Arrow connectors
Detail view
Callouts
videos
Highlighting
backgrounds)
Image color effects
Bleeping
Speech
Associating
Line connectors
Arrow connectors
Detail view
Callouts
Subtitle

Exploration & Authoring Techniques
WYSIWYM UI Model
Exploration
text
Zooming
Faceting
On-demand highlighting
Expanding & Drilling down
images
Zooming
Faceting
videos Faceting (excerpts)
Authoring
text, images,
videos
Form editing
Inline edit
Drawing
Drag and drop
Context menu
(Floating) Ribbon editing
Voice commands
(Multi-Touch) Gestures

WYSIWYM UI Model Binding

* If value is available in the text/subtitle.
No binding Partial binding Full binding
Tree-based
(e.g. Taxonomies)
Graph-based
(e.g. RDF)
Hypergraph-based
(e.g. Topic Maps)
Item
Itemtype
Item-subitem
Itempropertyvalue
RelatedItems
Instance
Class
entities
Literal property
values
Topic
Topictype
Topicassociations
Topic
Occurre
nces
Value
Languagetag
Datatype
Value
Datatype
Structure
encoded in:
UI categories UI techniques
Visualization
text
Highlighting
Framing and
segmentation (borders,
overlays, backgrounds)
C1 C11 C13 C1 C1
Text formatting (color,
font, size etc.)
C2 C11 C14 C2 C2
Marking (appended icons) C3 C15 C3 C3
Associating
Line connectors * * *
Arrow connectors * * *
Detail view
Callouts
images
Highlighting
Framing and
C1 C11 C13 C1 C1
Image color effects C4 C11 C16 C4 C4
Associating
Line connectors
Arrow connectors
Detail view
Callouts
videos
Highlighting
Framing and
C1 C11 C13 C1 C1
Marking (appended icons) C3
C15 C3
C3
Bleeping C5 C5 C5
Speech
Associating
Detail view
Callouts
Subtitle
Exploration
text
Zooming
Faceting
On-demand highlighting C5 C12 C5 C5
images
Zooming
Faceting
Authoring
text, images,
videos
Form editing
Inline edit
Drawing C7 C7 C7 C7
Drag and drop C8 C8 C8 C8
Context menu
Voice commands C9 C9 C9 C9
(Multi-Touch) Gestures C10 C10 C10 C10

* If value is available in the text/subtitle.
No binding Partial binding Full binding
Tree-based
(e.g. Taxonomies)
Graph-based
(e.g. RDF)
Hypergraph-based
(e.g. Topic Maps)
Item
Itemtype
Item-subitem
Itempropertyvalue
RelatedItems
Instance
Class
entities
Literal property
values
Topic
Topictype
Topicassociations
Topic
Occurre
nces
Value
Languagetag
Datatype
Value
Datatype
Structure
encoded in:
UI categories UI techniques
Visualization
text
Highlighting
Framing and
C1 C11 C13 C1 C1
Text formatting (color,
font, size etc.)
C2 C11 C14 C2 C2
Associating
Detail view
Callouts
images
Highlighting
Framing and
C1 C11 C13 C1 C1
Associating
Line connectors
Arrow connectors
Detail view
Callouts
videos
Highlighting
Framing and
C1 C11 C13 C1 C1
Marking (appended icons) C3
C15 C3
C3
Bleeping C5 C5 C5
Speech
Associating
Detail view
Callouts
Subtitle
Exploration
text
Zooming
Faceting
On-demand highlighting C5 C12 C5 C5
images
Zooming
Faceting
Authoring
text, images,
videos
Form editing
Inline edit
Drawing C7 C7 C7 C7
Drag and drop C8 C8 C8 C8
Context menu
Voice commands C9 C9 C9 C9
(Multi-Touch) Gestures C10 C10 C10 C10
Best practices to facilitate, enhance &
customize the WYSIWYM model
For example:
Automation
Recommendation
Customization (Personalization)
Collaboration & Crowdsourcing
Accessibility
etc.
Helper Components

WYSIWYM Interface
WYSIWYM UI Model

from WYSIWYG to WYSIWYM
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
RDFaCE
SlideWiki
conTEXT
Pharmer
Authoring (SCA)
authoring
for NLP
- Crowdlearning
[Khalili et al., 2012a] [Khalili and Auer, 2013b]

RDFaCE - RDFa Content Editor
As a plugin for TinyMCE (http://tinymce.moxiecode.com) rich text editor
Generates RDFa/Microdata annotations in the background.
V1: framing of named entities
in the text
V9: a callout showing additional
type information
X2: faceted browsing
T6: command bar
T5: a context menu for
revising annotations
T1: form editing
H1: automatic content annotation
http://rdface.aksw.org
from WYSIWYG to WYSIWYM
WYSIWYM Instantiation

RDFaCE Views for SCA
from WYSIWYG to WYSIWYM RDFaCE

Automatic Content Annotation
Combining the results of different NLP APIs

Evaluation
from WYSIWYG to WYSIWYM RDFaCE NLP APIs Combination
31 articles were collected (and manually annotated) in the three
categories news articles, weblog posts and Wikipedia articles.
Results of the RDFaCE automatic annotation feature was evaluated
using this gold standard.

RDFaCE Variations
Semantic Blogging in
WordPress
Data Journalism using
rNews
SEO using
Schema.org

Semantic Blogging in WordPress
from WYSIWYG to WYSIWYM RDFaCE Use Cases
https://wordpress.org/plugins/rdface

Data Journalism using rNews
http://dev.iptc.org/rNews
rNews
- approved standard developed
by IPTC (a consortium of the
world's major news agencies,
news publishers and news
industry vendors).
- using semantic markup to
annotate news-specific
metadata in HTML documents.

SEO using Schema.org
Schema.org
initiated by the popular search engines Bing, Google and Yahoo!
a broad, Web-scale and shared vocabulary focusing on popular
concepts suitable for search results.

SEO using Schema.org
Schema.org
initiated by the popular search engines
a broad, Web-scale and shared vocabulary focusing on popular
concepts suitable for search results.

WYSIWYM for Lightweight Text Analytics
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
RDFaCE
SlideWiki
conTEXT
Pharmer
Authoring (SCA)
authoring
for NLP
- Crowdlearning
[Khalili et al., 2014]

Analytical Information Imbalance

Analytical Information Imbalance
Empower people to ﬁnd out what patterns can be discovered and what
conclusions can be drawn from the information they share.

Unstructured* Structured* Semi.structured*
•  A"ensity)
•  Trendminer)
•  MashMaker)
•  Thomson)Data)
Analyzer)
•  IBM$Content$Analy.cs$pla2orm$
•  GATE$
•  Apache$UIMA$
•  Zoho$Reports$
•  SAP$NetWeaver$
•  Jackbe$
•  Rapidminer$
•  Alchmey(
•  OpenCalais(
•  Excel&
•  DataWrangler&
•  Google&Docs&Spreadsheets&
•  Google&Reﬁne&
•  Facete&
•  CubeViz&
•  TweetDeck(
•  Topsy(
•  Flumes(

Unstructured* Structured* Semi.structured*
•  A"ensity)
•  Trendminer)
•  MashMaker)
•  Thomson)Data)
Analyzer)
•  IBM$Content$Analy.cs$pla2orm$
•  GATE$
•  Apache$UIMA$
•  Zoho$Reports$
•  SAP$NetWeaver$
•  Jackbe$
•  Rapidminer$
•  Alchmey(
•  OpenCalais(
•  Excel&
•  DataWrangler&
•  Google&Docs&Spreadsheets&
•  Google&Reﬁne&
•  Facete&
•  CubeViz&
•  TweetDeck(
•  Topsy(
•  Flumes(
Lack of tools dealing with unstructured content, catering non-expert
users and providing extensible analytics interfaces.

T2: inline editing UI
V1: framing of named entities in the text
V2: text margin formatting for visualizing hierarchy
V7: line connectors to show the relation between entities
V9: a callout showing additional type information
X2: faceted browsing
H3: recommendation for NLP feedback.
http://context.aksw.org
WYSIWYM

Lightweight Text Analytics WYSIWYM
Collecting
RSS, Atom, RDF Feeds REST APIs SPARQL Endpoints Web Crawlers
Processing
Enriching Mixing
Annotation Refinement Exploring & Visualizing
feedback
RDFaCE
Exhibit
D3.js
BOA
conTEXT
Architecture & Workﬂow

Data Visualization & Exploration
Lightweight Text Analytics WYSIWYM conTEXT Workflow

Annotation Reﬁnement & Feedback
Lightweight Text Analytics WYSIWYM conTEXT Workflow
Lightweight text analytics as an incentive for users to revise
semantic annotations
RDFaCE for reﬁning annotations
Feedback to NLP services NLP calibration
DBpedia Spotlight
FOX
http://spotlight.dbpedia.org/rest/feedback
http://139.18.2.164:4444/api/ner/feedback
Feedback APIs

Evaluation
Lightweight Text Analytics WYSIWYM conTEXT
Task-driven usability & usefulness study
25 Users
10 questions pertaining to knowledge
discovery in corpora of unstructured data
e.g. What are the ﬁve most mentioned countries by Bill Gates tweets?
usability evaluation platform
http://context.aksw.org/evaluation

Evaluation Results
Lightweight Text Analytics WYSIWYM conTEXT Usefulness
Measuring +me & Jaccard similarity for answers
using/without conTEXT
second
Avg. 136% more time without conTEXT

Evaluation Results
Lightweight Text Analytics WYSIWYM conTEXT Usability
System Usability Scale (SUS) 82

WYSIWYM for Authoring of E-Learning Content
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
RDFaCE
SlideWiki
conTEXT
Pharmer
Authoring (SCA)
authoring
for NLP
- Crowdlearning
[Khalili et al., 2012b] [Tarasowa et al., 2013] [Tarasowa et al., 2014]

SlideWiki - OpenCourseWare Authoring
E-Learning WYSIWYM
V2: text margin formatting for visualizing content tree
V7: line connectors to show the relation between slides and decks
X4: expanding & drilling down to explore content
T4: drag & drop to change the order of slides and decks
T6: floating ribbon editing to author slide content
H5: collaboration and crowdsourcing helper components
http://slidewiki.org

WikiApp Data Model
E-Learning WYSIWYM SlideWiki

Usage
January 25, 2015

Usage
Semantic Web lecture series created collaboratively on SlideWiki.

Usability Evaluation
22 subjects
252 new slide revisions were created by users.
Originally the whole course had 130 questions, and students
changed 13 of them, fixing the typos or adding additional options
to multiple-choice questions.
SUS 69
Evaluation

WYSIWYM for Authoring of Semantic Medical Prescriptions
WYSIWYM UI Model
from WYSIWYG to
WYSIWYM
WYSIWYM for
of Semantic Medical
Prescriptions
guides extending
existing
UIs
instant
user gratification
support of
collaboration
customization &
RDFaCE
SlideWiki
conTEXT
Pharmer
Authoring (SCA)
authoring
for NLP
- Crowdlearning
[Khalili and Sedaghati, 2013a] [Khalili and Sedaghati, 2013b]

Pharmer
Semantic Medical Prescriptions
V1: highlighting of drugs through framing
V9: additional information about a drug in a callout
T1/T2: combined form and inline editing of electronic prescriptions
H2: Real-time drug tagging
http://bitili.com/pharmer
Authoring of Semantic Medical Prescriptions

Contributions in a Glance
Conclusions
WikiApp
Community of
users Semantically-enriched
documents
Binding
Semantic representation
models
UI elements & techniques
Helper components
Anlaytics
Interoperability
WYSIWYM
use author
provide
incentivize

Publications
Ali Khalili and Sören Auer. WYSIWYM ‐‐ Integrated Visualiza5on, Explora5on and Authoring of Seman5cally
  enriched Un‐structured Content. Seman1c Web Journal, 2014.
Ali Khalili and Sören Auer. User Interfaces for Seman5c Authoring of Textual Content: A Systema5c Literature
  Review. Journal of Web Seman1cs: Science, Services and Agents on the World Wide Web, 22(1), 2013.
Ali Khalili and Bita Sedagha1. A WYSIWYM Interface for Seman1c Enrichment of E‐prescrip1ons using Linked
  Open Drug Data. Interna1onal Journal On Advances in Life Sciences, 5(3,4), 2013.
Darya Tarasowa, Sören Auer, Ali Khalili, and Jörg Unbehauen. Crowdsourcing (seman5cally) Structured
  Mul5lingual Educa5onal Content (cosmec). Open Praxis Journal, 6(2), 2014.
Ali Khalili, Sören Auer and Axel C.N. Ngomo. conTEXT ‐‐ Lightweight Text Analy5cs using Linked Data. 11th
  Extended Seman1c Web Conference (ESWC 2014).
Ali Khalili and Sören Auer. WYSIWYM Authoring of Structured Content based on Schema.org. The 14th
  Interna1onal Conference on Web Informa1on Systems Engineering (WISE 2013).
Darya Tarasowa, Ali Khalili, Sören Auer, and Jörg Unbehauen. Crowdlearn: Crowd‐sourcing the Crea5on of
  Highly‐Structured E‐learning Content. 5th Intl. Conference on Computer Supported Educa1on (CSEDU 2013).
Ali Khalili and Bita Sedagha1. Seman5c Medical Prescrip5ons ‐‐ Towards Intelligent and Interoperable Medical
  Prescrip5ons. In IEEE Seventh Interna1onal Conference on Seman1c Compu1ng (ICSC 2013).
Ali Khalili, Sören Auer, Darya Tarasowa, and Ivan Ermilov. Slidewiki: Elicita5on and Sharing of Corporate
  Knowledge using Presenta5ons. The 18th Interna1onal Conference on Knowledge Engineering and Knowledge
  Management (EKAW 2012).
Ali Khalili, Sören Auer, and Daniel Hladky. The RDFa Content Editor ‐ from WYSIWYG to WYSIWYM. In
  Computer Soware and Applica1ons Conference (COMPSAC 2012).

Awards
RDFaCE
Pharmer
Best-paper award of the 36th IEEE Signature
Conference on Computers, Software, and
Applications (COMPSAC) 2012.
Creative Innovation Project Award 2014 for
OpenCourseWare Excellence from OCW
Consortium.
Nominated for best-paper award at 5th
International Conference on Computer
Supported Education (CSEDU 2013) and 18th
International Conference on Knowledge
Engineering and Knowledge Management
(EKAW 2012).
1st Prize of the AI Mashup Challenge 2014.
Best-application prize at WoLE2013 challenge
(Doing Good by Linking Entities), WWW2013
workshops.
Best-poster prize at Leipzig Research Festival for
Life Sciences 2012.
WYSIWYM

hank yu for yur atention…
January 26, 2015

A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (9)

Semelhante a A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration

Semelhante a A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration (20)

Mais de Ali Khalili

Mais de Ali Khalili (12)

Último

Último (20)

A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration