O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
© Ramesh Jain
Ramesh Jain
and
Pinaki Sinha
Department of Computer Science
University of California, Irvine
jain@ics.uci.ed...
© Ramesh Jain
Brave New Ideas
 On a Nagging Old Problem – that won’t go
away.
 About the dead elephant in the room.
 Or...
© Ramesh Jain
Semantic Gap
Discussions
Semantic Gap
remains
strong.
Lets stop ignoring the Elephant in the room.
Our appro...
© Ramesh Jain
What is our Goal?
1. Using Intensity values to bridge the Semantic
Gap.
Or
2. Bridging the Semantic Gap in (...
© Ramesh Jain
You will be disappointed in this
presentation if you are interested in
1. Using Intensity values to bridge t...
© Ramesh Jain
Reduction: Popular Research Approach 1
 X is an important problem, let me work on it.
 X is too complex. S...
© Ramesh Jain
Data Discovery: Popular Research Approach 2
 Your algorithm fails on a real data set D.
 Select a more rel...
© Ramesh Jain
Back to Content and Context
Content
Meaning or message: The meaning or message
contained in data, as distinc...
© Ramesh Jain
Falling Tree and George Berkeley
 "If a tree falls in a forest and no one is
around to hear it, does it mak...
© Ramesh Jain
Data
Streams are
all around
us.
© Ramesh Jain
Data Streams are Omnipresent
Visual DATA
Audio DATA
Text DATA
Location DATA
EXIF DATA
Time Line
DataType
© Ramesh Jain
© Ramesh Jain
© Ramesh Jain
The Challenge
Connecting
© Ramesh Jain
Bits and Bytes
Alphanumeric Characters
Lists, Arrays, Documents, Images …
Transformations
© Ramesh Jain
Semantic Gap
The semantic gap is the lack of coincidence
between the information that one can extract
from t...
© Ramesh Jain
M ultim edia
R esearcher
Hammer
 "To a man with a
hammer, everything
looks like a nail." —
Mark Twain
Where...
© Ramesh Jain
Learning from Birds
 The birds flying at the tips and at the front
are rotated in a timely cyclical fashion...
© Ramesh Jain
Data is just that --- DATA, but
Some of us have introduced a caste system in
multimedia!!!
Video
Meta Data
A...
© Ramesh Jain
Modeling the World
 Data
 Objects
 Events
© Ramesh Jain
Events and Objects
 Exist in the real world.
 Captured using different sensory
mechanism.
 Each sensor ca...
© Ramesh Jain
Events take place in the real world.
Events result in Data and Documents
© Ramesh Jain
Events are ‘Connectors’
Events create ‘Context’
SpatialCausal
Experiential Informational
TemporalStructural
...
© Ramesh Jain
We are in the midst of the greatest
Media Explosion.
© Ramesh Jain
Dealing with Photo Explosion
 3 Billion Photos uploaded on Facebook –
every month.
 We know that most peop...
© Ramesh Jain
Context Starts Before the Photo is
Taken
 Where
 When
 Why
 Who (Photographer)
 Which device
 Paramete...
© Ramesh Jain
Types of Context
 Context in Content
 Device Parameters
 Data Acquisition Context
 Perceiver
 Interpret...
© Ramesh Jain
Context in Content
 Relationship among different objects and
even in their subparts in real world can be
ut...
© Ramesh Jain
Device Parameters
 Environmental parameters of the digital
devices at the time of photo taking may play
key...
© Ramesh Jain
Data Acquisition Context
 Knowledge about the person taking photos,
location, and environmental conditions ...
© Ramesh Jain
Where am I?
© Ramesh Jain
Perceiver
 The knowledge and personality of the
perceiver play a key role in interpretation of
data.
Knowle...
© Ramesh Jain
Interpretation Context
 Real world situation in which the data is
interpreted results in focus on different...
© Ramesh Jain
What is a Camera?
 Kodak ‘Moment’.
 EXIF data is all metadata related to the Event.
Exposure Time
Aperture...
© Ramesh Jain
Sony CyberShot DSC-T2 Touchscreen 8MP Digital
Camera with Smile Detection
Camera = Event Capture Device
© Ramesh Jain
Meta Data for photos = Event Data
© Ramesh Jain
A new problem statement for media analysis
 Given apriori knowledge about the world (concepts and
relations...
© Ramesh Jain
Annotation Sources
Calendars
Annotations
Environment
Conditions
Application
Knowledge
Geographic
Landmarks
C...
© Ramesh Jain
Annotation Process
Calendars
Annotations
Environment
Conditions
Application
Knowledge
Geographic
Landmarks
C...
© Ramesh Jain
Semantic Gap is a Tough Problem –
Lets Fight with it using all tools.
Fighting
with
a
Monster.
© Ramesh Jain
Thanks.
For more
information,
jain@ics.uci.edu
?
ACM MM
Paper
Rejection
Próximos SlideShares
Carregando em…5
×

Content without context is meaningless

6.242 visualizações

Publicada em

Publicada em: Tecnologia, Educação

Content without context is meaningless

  1. 1. © Ramesh Jain Ramesh Jain and Pinaki Sinha Department of Computer Science University of California, Irvine jain@ics.uci.edu Content Without Context is Meaningless
  2. 2. © Ramesh Jain Brave New Ideas  On a Nagging Old Problem – that won’t go away.  About the dead elephant in the room.  Or about the Emperor’s new clothes. “If you keep doing what you’ve always done, you’ll keep getting what you’ve always got.” This Presentation
  3. 3. © Ramesh Jain Semantic Gap Discussions Semantic Gap remains strong. Lets stop ignoring the Elephant in the room. Our approaches have completely failed for the last two decades.
  4. 4. © Ramesh Jain What is our Goal? 1. Using Intensity values to bridge the Semantic Gap. Or 2. Bridging the Semantic Gap in (multimedia) data. Or 3. Publishing another paper in ACM Multimedia (or CVPR).
  5. 5. © Ramesh Jain You will be disappointed in this presentation if you are interested in 1. Using Intensity values to bridge the Semantic Gap. Or 3. Publishing another paper in ACM Multimedia (or CVPR).
  6. 6. © Ramesh Jain Reduction: Popular Research Approach 1  X is an important problem, let me work on it.  X is too complex. Simplify it to X’.  X’ is too complex. Simplify it to X2 ’.  …  X99 ’ can be formally defined, rigorously explored, and clearly solved. Solve it.  Publish it in a conference with less than 16% acceptance rate.  Announce that X is a solved problem.
  7. 7. © Ramesh Jain Data Discovery: Popular Research Approach 2  Your algorithm fails on a real data set D.  Select a more relevant data set D1. Fails again.  Carefully select a subset of Flickr data. Still Fails.  …  Select a better dataset - Coral Photos, or create a set and somehow make it Work.  Run several experiments – have several graphs so reviewers don’t complain.  Publish it in a conf. with < 16% acceptance rate.
  8. 8. © Ramesh Jain Back to Content and Context Content Meaning or message: The meaning or message contained in data, as distinct from its appearance, form, or style. Context Surrounding conditions: The circumstances or events that form the environment within which something exists or takes place.
  9. 9. © Ramesh Jain Falling Tree and George Berkeley  "If a tree falls in a forest and no one is around to hear it, does it make a sound”  "No. Sound is the sensation excited in the ear when the air or other medium is set in motion.“  Observation, Reality, and Perception.
  10. 10. © Ramesh Jain Data Streams are all around us.
  11. 11. © Ramesh Jain Data Streams are Omnipresent Visual DATA Audio DATA Text DATA Location DATA EXIF DATA Time Line DataType
  12. 12. © Ramesh Jain
  13. 13. © Ramesh Jain
  14. 14. © Ramesh Jain The Challenge Connecting
  15. 15. © Ramesh Jain Bits and Bytes Alphanumeric Characters Lists, Arrays, Documents, Images … Transformations
  16. 16. © Ramesh Jain Semantic Gap The semantic gap is the lack of coincidence between the information that one can extract from the (visual) data and the interpretation that the same data have for a user in a given situation. A linguistic description is almost always contextual, whereas an (image) may live by itself. Content-Based Image Retrieval at the End of the Early Years Found in: IEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders , et. al., December 2000
  17. 17. © Ramesh Jain M ultim edia R esearcher Hammer  "To a man with a hammer, everything looks like a nail." — Mark Twain Where is the Semantic Gap? I got my Machine Learning Hammer.
  18. 18. © Ramesh Jain Learning from Birds  The birds flying at the tips and at the front are rotated in a timely cyclical fashion to spread flight fatigue equally among the flock members.  The formation makes communication easier and allows the birds to maintain visual contact with each other.
  19. 19. © Ramesh Jain Data is just that --- DATA, but Some of us have introduced a caste system in multimedia!!! Video Meta Data Audio Text Multimedia Caste SystemTraditional Caste System Priest Warrior Traders Who Cares
  20. 20. © Ramesh Jain Modeling the World  Data  Objects  Events
  21. 21. © Ramesh Jain Events and Objects  Exist in the real world.  Captured using different sensory mechanism.  Each sensor captures only a limited aspect.  Are used to understand a Situation.
  22. 22. © Ramesh Jain Events take place in the real world. Events result in Data and Documents
  23. 23. © Ramesh Jain Events are ‘Connectors’ Events create ‘Context’ SpatialCausal Experiential Informational TemporalStructural People Things Places Time Experiences Events Events Connect: Events are represented using 6 facets.
  24. 24. © Ramesh Jain We are in the midst of the greatest Media Explosion.
  25. 25. © Ramesh Jain Dealing with Photo Explosion  3 Billion Photos uploaded on Facebook – every month.  We know that most people in the world are not yet addicted to the Web, but do take photos.
  26. 26. © Ramesh Jain Context Starts Before the Photo is Taken  Where  When  Why  Who (Photographer)  Which device  Parameters of the device
  27. 27. © Ramesh Jain Types of Context  Context in Content  Device Parameters  Data Acquisition Context  Perceiver  Interpretation Context
  28. 28. © Ramesh Jain Context in Content  Relationship among different objects and even in their subparts in real world can be utilized in analysis of data.  This has been studied since early days in computer vision and is being rediscovered once again.
  29. 29. © Ramesh Jain Device Parameters  Environmental parameters of the digital devices at the time of photo taking may play key role.  What is a camera?
  30. 30. © Ramesh Jain Data Acquisition Context  Knowledge about the person taking photos, location, and environmental conditions at the time of photo acquisition (e.g., sun angle, cloudy, rainy, night, indoor, etc.) affect the content of the image.
  31. 31. © Ramesh Jain Where am I?
  32. 32. © Ramesh Jain Perceiver  The knowledge and personality of the perceiver play a key role in interpretation of data. Knowledge Personality: Rohrsach Tests
  33. 33. © Ramesh Jain Interpretation Context  Real world situation in which the data is interpreted results in focus on different aspects of the data.
  34. 34. © Ramesh Jain What is a Camera?  Kodak ‘Moment’.  EXIF data is all metadata related to the Event. Exposure Time Aperture Diameter Flash Metering Mode ISO Ratings Focal Length Time Location Face
  35. 35. © Ramesh Jain Sony CyberShot DSC-T2 Touchscreen 8MP Digital Camera with Smile Detection Camera = Event Capture Device
  36. 36. © Ramesh Jain Meta Data for photos = Event Data
  37. 37. © Ramesh Jain A new problem statement for media analysis  Given apriori knowledge about the world (concepts and relations between concepts)  Given a priori knowledge about events (lattice of events)  Given as contextual information  A set of data/ media  A possibly empty sets of tags (natural language descriptions)  The current event type to be instantiated (possibly to be first discovered)  Identify the entities occurring in the event, as described by media, data and text  with >98% precision and recall  with minimal/ acceptable user involvement Fausto Giunchiglia, DISI, University of Trento
  38. 38. © Ramesh Jain Annotation Sources Calendars Annotations Environment Conditions Application Knowledge Geographic Landmarks Camera Parameters
  39. 39. © Ramesh Jain Annotation Process Calendars Annotations Environment Conditions Application Knowledge Geographic Landmarks Camera Parameters Keynote Talk at ACM MM 2010, Speaker: Duncan WattsLocation: Palazzo dei Congressi EXIF: Indoor Scene; Face Keynote Talk at ACMMM10 by Duncan Watts on Using the Web to do Social Science
  40. 40. © Ramesh Jain Semantic Gap is a Tough Problem – Lets Fight with it using all tools. Fighting with a Monster.
  41. 41. © Ramesh Jain Thanks. For more information, jain@ics.uci.edu ? ACM MM Paper Rejection

×