This document discusses events in multimedia and presents an overview of event modeling. It motivates the importance of events in domains like lifelogs, experience sharing, emergency response, and news. It reviews requirements for a common event model and surveys existing event models. An event model called Event-Model-F is proposed, which defines ontology patterns for modeling events. An application for exploring social media events on mobile devices is presented. The document concludes by discussing the need for a common theory and tools for dealing with events in multimedia.

1. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 1
Events in Multimedia -
Theory, Model, Applicat
ion
Workshop on Event-based Media Integration and
Processing, ACM Multimedia, 2013
Juniorprof. Dr. habil. Ansgar Scherp
mail@ansgarscherp.net

2. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 2
Motivation
• Events are a natural abstraction
of human experience
• Events are everywhere!
• Lifelogs
• Experience sharing
• Emergency response
• Cultural heritage
• News
• News
• Sports
• Surveillance
• …

3. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 3
Theory

4. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 4
Brothers at enmity with each other …
Event
Object
Object
Event
• Earlier: object-based and entity-based systems
• Now: applications that consider events at least
as important as objects
vs.

5. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 5
What is an event?
• Perduring entities that unfold
over time
• Occurrences in which humans
participate
• Subject to discussions and
interpretations by humans
• Enduring entities that unfold
over space
Object
Event

6. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 6
Brothers at enmity with each other …
• Some philosophers consider objects as 4D
• Extend across time just as they do in space
Casati R, Varzi A (2006) Events. Stanford encyclopedia of philosophy.
http://plato.stanford.edu/entries/events
Object
Event

7. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 7
Event Object
Brothers at enmity with each other …
• Events and objects as first class entities
• Events and objects require each other!
• For example, in DOLCE
‘is participant in’
‘has participant’
• … not necessarily!

8. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 8
Model

9. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 9
Events need to be modeled …
• … and are useful in a variety of domains
• Lifelogs
• Multimedia-based experience sharing
• Emergency response
• Cultural heritage
• News
• Sports
• Surveillance
• …

10. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 10
Emergency
Control Center
Forward
Liaison
Officer
Documentary
support
Calls to report about
a power outage
Creates incident
with audio recording
Request to
report about a
flooded cellar
Reports
by taking
photos
etc.
Emergency Response
Coordination
Emergency
Hotline
Fire Department
Police Department
Coordinate and
keep up to
date
Report
and update
about the incident
Coordinate
and keep up
to date
Report and update
about the incident
Citizen
• Several emergency response entities are involved
• Using different event-based systems
• Common understanding of multimedia information
needed to efficiently communicate between ERs
Snapped pole image from:
http://www.dailymail.co.uk/
Emergency Response Scenario

11. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 11
Requirements to a Common Event
Model*
• Participative aspect
• Temporal aspect
• Spatial aspect
• Structural aspect
• Mereology (composition)
• Causality
• Correlation
• Interpretation
• Experiential aspect (documentation)
* Analysis of 21 models and systems [SM13, SSF+12, SSF+09]

12. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 12
Survey on Event Models & Systems
…
Participa-
tion

13. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 13
Survey on Event Models & Systems
…

14. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 14
Ontology Patterns of Event-Model-F
• Event-Model-F* defines six ontology patterns
• Ontology design pattern similar to SE
• Build on top of DOLCE+DnS Ultralight ontology
• Cf. theory on events and objects
• Provides Description and Situation pattern
• Specified in Web Ontology Language (OWL)
• Formalized in Description Logics
* Homage to event model E by Westermann and Jain

15. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 15
6 Patterns of the Event-Model-F
• (1) Participation pattern
• (2) Mereology pattern (composition)
• (3) Causality pattern
• (4) Correlation pattern
• (5) Documentation pattern
• (6) Interpretation pattern
• All based on Descriptions and Situations (DnS)
• Contextualization of events and objects w.r.t.
specific situations

16. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 16
Example: Descriptions ‘n’ Situations
• DomesticPowerOutage-
Description defines roles
• AffectedObjectRole
• AffectedPersonRole
• AffectedPersonRole
• …
• DomesticPowerOutage-
Situation defines objects
house-1 : Building
paul-1 : NaturalPerson
sandy-1 : NaturalPerson
…
• In a different situation, NaturalPerson
paul-1 may play a FireFighterRole
• People may have differing opinions
about the cause of the power outage
• …
Image source: Wikipedia
Classify

17. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 17
(1) Participation Pattern
• Participation of living and non-living objects in events
• Reuse of domain knowledge
Roles the
entities play
Real world
entities
EventParticipationDescription
defines exactly 1 DescribedEvent
defines min 1 Participant
defines some LocationParameter
defines some TimeParameter
defines only (DescribedEvent or Participant or
LocationParameter or TimeParameter)
isSatisfiedBy exactly 1 EventParticipationSituation
Example: Firemen and home owner are involved in an
incident of a house fire.

18. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 18
(3) Causality Pattern
• Event (cause) implies other event (effect)
• Causal relationship holds under some justification
• Causes and effects are events, and only events
Description
EventCausalityDescription
EventCausalitySituation
Situation
Cause Effect Justification
EventRole
Concept
Event Description
classifies
isRoleOf
defines
isEventIncludedIn
satisfies
isObjectIncludedIn
Role
EventCausalityDescription
defines exactly 1 Cause
defines exactly 1 Effect
defines exactly 1 Justification
defines only (Cause or Effect or Justification)
isSatisfiedBy exactly 1 EventCausalitySituation
Example: The event of a snapped power pole causes a
power outage.

19. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 19
Use Case: Emergency Response
Domain ontology by
City of Sheffield, UK
Events and
media metadata
Event details
[MTAP2012]
Web 2.0 content
and metadata

20. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 20
Application

21. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 21
• Which bakery close by is open on Sunday?
• What could I do tonight?
• Which sights are in the area?
Are they still open?
• Existing applications
– Mostly focus on location,
i.e., points-of-interests
– No support for integrated
search for events Sun: 7am-11am
Problem: Events in Social Media

22. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 22
Different Sources of Social Events …
• Opening hours of a shop like a bakery
• Concert of favorite rock star in my town
• Café serving English tea in my neighborhood
• Happy hour at a bar
• Special sales at the shopping mall
• …
…

23. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 23
Event Model for Social Media!
• Derive a specific model of social media events
• Use notions of events and objects defined in
Event-Model-F and specialize it for social media
• Focus on what is needed in the domain
–(1) Participation pattern
–(5) Documentation pattern
• Future work: (2) Mereology Pattern

24. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 24
• Exploration of POIs and
events in real-time
• Multithreaded integration
of social data sources:
KlickTel, DBpedia,
GeoNames, Eventful,
etc.
Pad
Phone
mobEx: Mobile Social Media Explorer

25. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 25
Mediator-based Architecture

26. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 26
Update Event 003Add Event 003
Incremental Data Integration on Server
mobEx
Server
Client
Café Vienna
Phone.: ???
Opening hours:
Mo – Fr. 10 – 2 Uhr
Provider 1 Provider 2
Café Vienna
Phone.: 01234
Opening hours: ???
Café
Vienna
Mo – Fr.
10 – 2
o‘clock
Café Vienna
Tel.: 01234
Mo – Fr.
10 – 2
o‘clock
Entity
Resolution
Time

27. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 27
Provider …
Entity Resolution in Detail
Provider 1
Provider 2
….
Weighted comparison of attributes:
 Location information
 Time information
 Similarity score computed from
single weights
 Above threshold  integration
Café Vienna
Tel.: 01234
Opening hours:
Mo – Fr. 10 – 2 Uhr
S1, 15
68161 Mannheim
Calculated GPS Data
www.cafevienna.de/
Description: XYZ
High Weight
Low Weight
Medium Weight
High Weight
Low Weight
High Weight
Heuristic filter (blocking):
 GPS coordinates > 500m
distance
 Except: same address
 Different entity types are not
compared (Persons, Places,
Events)
High Weight

28. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 28
Entity Resolution vs. Data Delivery
• Percentage of resources the client receives
• Percentage of resolved resolution at that time

29. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 29
Faceted Exploration of Events & Objects

30. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 30
Exploring Time as Natural as Space

31. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 31
• 18 subjects, average age of 25.5 (SD=3.03)
and six female
• Tracked detailed activities over three weeks
Evaluation in the Field
.
• More than 4000 single events in 234 sessions
• Session is defined as
• Active usage of application until it is closed
• Or: after 30 seconds of inactivity

32. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 32
Evaluation in the Field (2)
• Session lasted on average 2:42 minutes.
• 57% of the participants used the application
daily, 72% every second day or more.
• Users spent more time on the map screen than
on the screen showing the facets (ca. x1.5)
• Time slider not used
more often in the course
of the three weeks

33. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 33
Conclusion
• Events are an important concept in multimedia
• Tremendous research conducted in the past
• But, we have a lack of
–Common theory of events in multimedia
–Integrated tool chain to deal with events
–From detection over representation to use
Acknowledgements:
• R. Jain, C. Saathoff, T. Franz, S. Staab, D. Schmeiß
• B. Opitz, T. Sztyler, B. Pfister, M. Jess, C. Bikar, F. Knip

34. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 34
But there is a long interest …
• … in this topic in the multimedia community
• 1st ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2009, Beijing, China
• 2nd ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2010, Firenze, Italy
• 3rd ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2011, Scottsdale, Arizona, USA
• "Multimedia Activity and Event Understanding" Area,
ACM Multimedia 2012, Nara, Japan
• Workshop on Event-based Media Integration and
Processing, ACM Multimedia 2013, Barcelona

35. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 35
Try it out …in Barcelona!