1.
Semantic Knowledge and
Privacy in
the Physical Web
PRAJIT KUMAR DAS, ABHAY KASHYAP,
GURPREET SINGH, CYNTHIA MATUSZEK,
TIM FININ, ANUPAM JOSHI
UMBC ebiquity and IRAL Labs

2.
Motivation
Our goal is to provide contextually
aware information, using the IoT,
that is privacy preserving and
ubiquitously helpful
Image courtesy Batman Wikia
CARLTON
Slide 2 of 44

3.
IoT by Volume
Slide 3 of 44

4.
IoT by
Domain
Slide 4 of 44

5.
IoT by Privacy Concerns
Slide 5 of 44

6.
Salient features
 CARLTON: A context-aware, NL question-answer BOT
 Context derived from the Physical Web (IoT)
 Under development, prototype system
 Simple NLP using tools like Stanford CoreNLP
 Mobile app and Kiosk for front-end
 ABAC privacy model, Privacy rules using SWRL
 Hierarchical context ontology
 Optional authentication for UMBC people
Slide 6 of 44

7.
 Concretization of IoT
 Small, quick seamless
interactions with
physical objects and
locations with your
device
Physical web: What?
Slide 7 of 44

8.
Physical web: What?
 Everything is a tap
away
Slide 8 of 44

9.
Physical web: What?
 See what’s useful
around you
Slide 9 of 44

10.
Physical web: What?
 Any object or place
can broadcast
content
Slide 10 of 44

11.
Physical web: How?
Three main techniques
 Nearby Connections
Slide 11 of 44

12.
Physical web: How?
Three main techniques
 Nearby Connections
 Nearby Notifications
Slide 12 of 44

13.
Physical web: How?
Three main techniques
 Nearby Connections
 Nearby Notifications
 Nearby Messages
Slide 13 of 44

14.
System Overview
Slide 14 of 44

15.
System Overview
Slide 15 of 44

16.
System Overview
Slide 16 of 44

17.
Who is Tim
Finin?
System Overview
Slide 17 of 44

18.
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
Who is Tim
Finin?
System Overview
Slide 18 of 44

19.
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generator
Context
Who is Tim
Finin?
System Overview
Slide 19 of 44

20.
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generator
Context
Who is Tim
Finin?
Inference Engine
OntologyKnowledge base
System Overview
Slide 20 of 44

21.
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generator
Context
Who is Tim
Finin?
Inference EngineResponse: JSON
{“text”: “He’s a
Professor in the
Computer Science
department!”}
OntologyKnowledge base
System Overview
Slide 21 of 44

22.
Example query
Slide 22 of 44

23.
Is this room
booked from
2PM-3PM?
Example query
Slide 23 of 44

24.
User is a faculty and is in
front of Conf. room 1.
Is this room
booked from
2PM-3PM?
Example query
Slide 24 of 44

25.
Conf. room 1 calendar has no
events during that time.
Is this room
booked from
2PM-3PM?
Example query
Slide 25 of 44

26.
Is this room
booked from
2PM-3PM? No, would you like
me to book it from
2PM – 3PM?
Example query
Slide 26 of 44

27.
Is this room
booked from
2PM-3PM? No, would you like
me to book it from
2PM – 3PM?
Yes, please!
Example query
Slide 27 of 44

28.
Okay, the room
has been booked
in your name from
2PM – 3PM
Is this room
booked from
2PM-3PM? No, would you like
me to book it from
2PM – 3PM?
Yes, please!
Example query
Slide 28 of 44

29.
Example query
Slide 29 of 44

30.
Is Dr. Joshi here?
Example query
Slide 30 of 44

31.
Is Dr. Joshi here?
User could mean Dr. A.
Joshi or Dr. K. Joshi.
Example query
Slide 31 of 44

32.
Is Dr. Joshi here?
But user is in front of Dr. A.
Joshi’s office.
Example query
Slide 32 of 44

33.
Is Dr. Joshi here?
User is an advisee of Dr. A.
Joshi
Example query
Slide 33 of 44

34.
Is Dr. Joshi here?
Dr. Joshi is in a
meeting till
3PM
Example query
Slide 34 of 44

35.
Example query
Slide 35 of 44

36.
Example query
Where is Dr.
Finin’s office?
Slide 36 of 44

37.
Example query
User is in CSEE building
Where is Dr.
Finin’s office?
Slide 37 of 44

38.
Example query User is unknown to system
Where is Dr.
Finin’s office?
Slide 38 of 44

39.
Example query
Where is Dr.
Finin’s office?
Please see CSEE
front desk for
required
information
Slide 39 of 44

40.
Example1.
@prefix crltn:<https://www.ebiquity.org/ontologies/carlton/0.1>.
@prefix swrlb:<http://www.w3.org/2003/11/swrlb>.
crltn: student(?requester)∧
(
crltn: supervises(“Xavier”,?requester)∨
(crltn: affiliatedWith(?requester,?labName)∧crltn: leads(“Xavier”,?labName))
)∧
crltn: hasCurrentLocation(?requester,?aBldgLocation)∧
crltn: room(?aBldgLocation)∧crltn: sitsIn(“Xavier”,?aBldgLocation)∧
crltn: currentTime(?currTime)∧swrlb: Exists(?anEvent)∧crltn: speakingAt(“Xavier”,?anEvent)∧
(
(crltn: startTime(?anEvent,?eventStartTime)∧swrlb:
greaterThan(?eventStartTime,?currTime))∨
(crltn: endTime(?anEvent,?eventEndTime)∧swrlb: greaterThan(?currTime,?eventEndTime))
)∧crltn: hasCurrentLocation(“Xavier”,?aLocation)∧crltn: Location(?aLocation)∧
crltn: requestLocation(“Xavier”)
⇒
shareLocation(?aLocation)
Policy Example
Slide 40 of 44

41.
supervises(“Xavier”,?requester)
OR
(
affiliatedWith(?requester,?labName)
AND
leads(“Xavier”,?labName)
)
Policy Example
Slide 41 of 44

42.
hasCurrentLocation(?requester,?aBldgLocation)
AND
room(?aBldgLocation)
AND
sitsIn(“Xavier”,?aBldgLocation)
=>
shareLocation(?aLocation)
Policy Example
Slide 42 of 44

43.
Future work
 Prototype system constantly adding conversations
 Beacons on robots
 Reason over robots near you
 How robots respond to instructions?
 “Can you take me to Prof. Matuszek now?”
 “Show me the way to the ITE 346 conference
room”
Slide 43 of 44

44.
Summary
 We presented CARLTON
 A context-aware, NL question-answer BOT
 Context derived from the Physical Web (IoT)
 Semantic web technologies used to preserve data privacy
Thanks to NSF for the travel grant!
and
Thanks to Google for the gift of beacons!
Slide 44 of 44