Image Tag Refinement Along the 'What' Dimension using Tag Categorization and Neighbor Voting. Paper presented at ICME 2010 in Singapore.

1. IMAGE TAG REFINEMENT ALONG THE ‘WHAT’ DIMENSION USING TAG CATEGORIZATION AND NEIGHBOR VOTING IEEE International Conference on Multimedia & Expo Singapore – July 19-23, 2010 Sihyoung Lee , Wesley De Neve, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) email: ijiat@kaist.ac.kr

5. Question Can we trust all image tags in an image folksonomy?

7. Problem Statement Need for identifying correct tags and noisy tags in order to boost the effectiveness of image retrieval

10. Neighbor Voting (2/3) input image tag relevance learning [Figure adopted from X. Li et al. ] bridge bicycle perfect MyWinners robyn fishing me bristol court 1 number Sydney bridge Australia architecture bridge tranquil bruges trees NikonE3100 Sydney bridge SuperShot clouds a5PhotosaDay ireland irlanda ingiro northireland irlandadelnord connemara Sweden bridge lake APlusPhot SuperAPlus image folksonomy retrieval of visual neighbors bridge 4 bicycle 0 perfect 0 MyWinners 0

12. Previous Tag Refinement Image folksonomy refined tags Retrieval of visual neighbors Voting Neighbor voting algorithm nikon, leaves, jw, east, clouds, coast, ground, rita, red, rain, newengland, pretty, yellow, summer, portrait, pink, macro, grass, flowers, etc …

13. Proposed Tag Refinement Tag categorization WordNet refined tags along the what dimension Visual information-based refinement GPS-based refinement where refined tags along the where dimension Time-based refinement when refined tags along the when dimension Refinement using affective content analysis how refined tags along the how dimension who refined tags along the who dimension Face recognition-based refinement what Neighbor voting algorithm Retrieval of visual neighbors Image folksonomy Scope of this paper: tag refinement along the ‘what’ dimension Voting nikon, leaves, jw, east, clouds, coast, ground, rita, red, rain, newengland, pretty, yellow, summer, portrait, pink, macro, grass, flowers, etc …

20. Objective Performance of Tag Refinement (2/2) before tag refinement after tag refinement without tag categorization after tag refinement with tag categorization

21. Subjective Performance of Tag Refinement Image Tags Before tag refinement After tag refinement without categorization with categorization bc, beach , british, cacade, canada, casio, cloud , clouds , columbia, exf1, firstquality, forest, fpg, fun, hike, hiking, island , juandefuca, lagoon , lighthouse, metchosin, mountain , mountains , ocean , picnic, play, range, sand , state, trail, vancouver, victoria, washington, water , wave , waves , witty, wittys clouds , beach , water , ocean , cloud , sand , mountains , island , wave , mountain , canada, washington, forest, trail, waves , hiking clouds , beach , water , ocean , cloud , sand , mountains , island , mountain , trail

22. Objective Performance of Image Tag Recommendation Original folksonomy Refined folksonomy without categorization with categorization [email_address] 0.285 0.505 0.680 [email_address] 0.315 0.330 0.405 the P@1 and P@5 values demonstrate that the proposed tag refinement technique improves the effectiveness of image tag recommendation for non-tagged images

23. Subjective Performance of Image Tag Recommendation Image Recommended tags Original folksonomy Refined folksonomy without categorization with categorization explore, sky , blue , water, nature, yellow , clouds , nikon, canon, 2007 sky , explore, blue , nature, yellow , water, canon, clouds , white , geotagged sky , blue , nature, yellow , water, clouds , building , architecture , beach, snow explore, nature , green , sky , blue , macro, water, nikon, flower , clouds nature , green , sky , blue , explore, macro, yellow , canon, flower , water nature , green , sky , blue , yellow , flower , water, clouds , bird, grass sky , explore, blue , clouds , nature , water, landscape, nikon, canon, geotagged sky , blue , clouds , nature , explore, water, landscape, hdr, yellow , canon sky , blue , clouds , nature , water, yellow , snow, bird , lake, beach

26. Thank you! Any questions?