Presentazione realizzata in occasione della conferenza Better Software 2009

1. Annotazione semantica dei video: lo stato
della ricerca
Thomas M. Alisi
bettersoftware.it - 06.05.09

2. who am I
• Master in Multimedia Content Design
Board of operations - www.mmm.unifi.it
• Media Integration and Communication Center
Post-doc researcher - www.micc.unifi.it
• Red Noodles S.r.l.
founder - www.rednoodles.com
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

4. Vidi Video
• Interactive semantic video search with a large thesaurus of machine-learned
audio-visual concepts
• 02.2007 - 01.2010 - Framework Programme 6
• UNIVERSITEIT VAN AMSTERDAM, NETHERLANDS
• INFORMATICS AND TELEMATICS INSTITUTE, GREECE
• INSTITUTE FOR SYSTEMS AND COMPUTER ENG., PORTUGAL
• UNIVERSITY OF SURREY, UK
• UNIVERSITA' DEGLI STUDI DI FIRENZE, ITALY
• COMPUTER VISION CENTER, SPAIN
• BEELD EN GELUID, NETHERLANDS
• FONDAZIONE RINASCIMENTO DIGITALE, ITALY
• UNIVERSITA' DI MODENA E R. EMILIA, ITALY [subcontractor]
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

5. Vidi Video WP
• 1 Management
• 2 Video processing
• 3 Audio analysis
• 4 Video analysis
• 5 Learning integrated feature detectors
• 6 Technological software development
• 7 Demonstrator and applications
• 8 Data, annotation & queries
• 9 Dissemination
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

6. Vidi Video WP7
• WP7: Demonstrator and applications
• The objective of the work package is to study
and develop methods, demonstrators and
applications that allow a natural querying and
exploration of the annotated multimedia
content in different application fields.
• TASK 7.1 ONTOLOGY OF QUERIES
• TASK 7.2 MULTIMEDIA QUERY TOOL
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

7. Ontology
• The ontology structure has been created
automatically using relations from WordNet (“is a
kind of”, “has part”, “is part of”) and annotations
ground-truth (“correlated to”)
• The ontology is able to learn rules that allow to
detect complex concepts (typically event
concepts) .
• Rules are coded in SWRL and learned from a
training set. The learning method is the First Order
Inductive Learner for Semantic Web (FOILS). The
SWRL rules are included in the OWL ontology
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

8. System
• Different modules: core (Java Tomcat w/ Jena), NLP
(C++ Stanford), Annotation (Flash/Flex + php),
Query (Flex)
• Modules are connected using HTTP POST, XML
and SOAP web service
• Support for multiple DOME (Dynamic Ontology
with Multimedia Elements) ontologies and
ontology reasoning
• Results are in RSS 2.0 format with paging (queries
treated as RSS feeds)
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

9. Specs
• Rich Internet Application (in Adobe Flex) based
system, but can run also as standalone application
• Adobe Flash Media Server video streaming
• 23 videos with approx 25.000 annotated audio-
visual concepts (with start & end timecodes)
• Ontologies dataflow: from XML to MPEG7 to
OWL
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

10. Requirements
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

12. Annotation
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

15. Query & Display
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it

19. Dissemination
Red Noodles S.r.l. - Via delle Gore 6/b, Firenze - +39 055 436 0011 - thomas@rednoodles.com
MICC Università di Firenze - L.go de Gasperi 1, Firenze - +39 055 4237 408 - alisi@micc.unifi.it