O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Video, AI and News: video analysis and verification technologies for supporting journalism

Invited presentation at the JOLT ETN virtual training event, June 2020.

  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Video, AI and News: video analysis and verification technologies for supporting journalism

  1. 1. Thessaloniki, June 2020 Video, AI and News: video analysis and verification technologies for supporting journalism Vasileios Mezaris CERTH-ITI 1
  2. 2. Thessaloniki, June 2020Vasileios Mezaris Video and the News Fundamental need: - Make good use of video (either own or 3rd-party content) to convey your message! Technology-wise, this requires: - Verifying 3rd-party content; avoiding distributing falsehoods - Understanding the content / making it dicoverable (especially own content) - Adapting / re-purposing the (verified, discovered) content This is where AI can step in! 2
  3. 3. Thessaloniki, June 2020Vasileios Mezaris Verifying the content Main requirements: - Check if a supposedly new (breaking-news?) video is just “stock footage”, is used out of context; the “easy fake” - Check for manipulations / editings in the video 3
  4. 4. Thessaloniki, June 2020Vasileios Mezaris Verifying the content InVID Verification Plugin: a tool for the verification of selected newsworthy videos 4 A browser plugin to debunk fake news and to verify videos and images >20.000 users - Check prior video use: reverse video search on the Web - Check contextual information: Social- media-based contextual analysis - Keyframe/image inspection by magnifying glass - Check image (keyframe) forensics Free! Get it from: https://www.invid-project.eu/verify
  5. 5. Thessaloniki, June 2020Vasileios Mezaris Verifying the content Web application for video fragmentation and reverse web search (try it!): http://multimedia3.iti.gr/video_fragmentation/service/start.html 5
  6. 6. Thessaloniki, June 2020Vasileios Mezaris Verifying the content 6 https://www.youtube.com/watch?v=OVAxQA3gMEo “Video, the crowd in panic flees from Notre Dame in Paris after the attack of an armed man”
  7. 7. Thessaloniki, June 2020Vasileios Mezaris Verifying the content Behind the scenes: - Efficient and effective video fragmentation [1][2] - Can be extended with deep-learning based fragmentation techniques A more global view of video verification for News can be found in [3] 7 [1] E. Apostolidis, V. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, May 2014. Software available at https://mklab.iti.gr/results/video-shot-and-scene-segmentation/. [2] K. Apostolidis, E. Apostolidis, V. Mezaris, "A motion-driven approach for fine-grained temporal segmentation of user- generated videos", Proc. 24th Int. Conf. on Multimedia Modeling (MMM2018), Bangkok, Thailand, Feb. 2018. [3] "Video verification in the fake news era", V. Mezaris, L. Nixon, S. Papadopoulos, D. Teyssou (Editors), Springer, 2019.
  8. 8. Thessaloniki, June 2020Vasileios Mezaris Understanding the content / making it discoverable Content fragmentation and annotation: - Identify the different temporal fragments of a video (subshots/shots/scenes) - Annotate each fragment with text that describes it: concept labels, sentiment labels, descriptive captions - Enable the retrieval of video fragments, using any of these labels, or free-text queries 8
  9. 9. Thessaloniki, June 2020Vasileios Mezaris Understanding the content / making it discoverable 9 Shot #15 Scene #4 Scene #5 Shot #11 Shot #12 Shot #13 Shot #14 Shot #16 Subshot #58 Subshot #59 Shot #17 Shot #18 Subshot #60 … … … ……
  10. 10. Thessaloniki, June 2020Vasileios Mezaris Understanding the content / making it discoverable 10 Sample video frame Top detected concepts
  11. 11. Thessaloniki, June 2020Vasileios Mezaris Understanding the content / making it discoverable Web application [4] for video analysis and search (try it with your video!): http://multimedia2.iti.gr/onlinevideoanalysis_v5/service/start.html Demo video: https://youtu.be/mO-NRpIJ9UU 11 [4] C. Collyda, E. Apostolidis, A. Pournaras, F. Markatopoulou, V. Mezaris, I. Patras, "VideoAnalysis4ALL: An on-line tool for the automatic fragmentation and concept-based annotation, and the interactive exploration of videos", Proc. ACM ICMR 2017, Bucharest, Romania, June 2017.
  12. 12. Thessaloniki, June 2020Vasileios Mezaris Understanding the content / making it discoverable Behind the scenes: - Efficient and effective video fragmentation - Elaborate deep-neural-network architectures for concept-based annotation, e.g. [5][6] (and for video captioning, sentiment analysis, video retrieval using free-text queries [7]; not shown in this demo) [5] F. Markatopoulou, V. Mezaris, I. Patras, "Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation", IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1631-1644, June 2019. DOI:10.1109/TCSVT.2018.2848458. Software available at https://github.com/markatopoulou/fvmtl-ccelc. [6] N. Gkalelis, V. Mezaris, "Subclass deep neural networks: re-enabling neglected classes in deep network training for multimedia classification", Proc. 26th Int. Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020. [7] D. Galanopoulos, V. Mezaris, "Attention Mechanisms, Signal Encodings and Fusion Strategies for Improved Ad-hoc Video Search with Dual Encoding Networks", Proc. ACM Int. Conf. on Multimedia Retrieval (ICMR 2020), Dublin, Ireland, 2020. 12
  13. 13. Thessaloniki, June 2020Vasileios Mezaris Adapting / re-purposing the content Main requirements: - Target distribution platforms & devices have varying requirements (e.g. the optimal duration of a video differs from one platform to another) - Target audiences have different preferences / information needs Video summarization: - Create editions of the content that are adapted to different platforms and audiences 13
  14. 14. Thessaloniki, June 2020Vasileios Mezaris Adapting / re-purposing the content Example - Original video (1’38’’) - 14’’ summary 14
  15. 15. Thessaloniki, June 2020Vasileios Mezaris Adapting / re-purposing the content Web application [8] for video summarization (try it with your video!): http://multimedia2.iti.gr/videosummarization/service/start.html Demo video: https://youtu.be/LbjPLJzeNII 15 [8] C. Collyda, K. Apostolidis, E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, "A Web Service for Video Summarization", Proc. ACM Int. Conf. on Interactive Media Experiences (IMX 2020), Barcelona, Spain, June 2020.
  16. 16. Thessaloniki, June 2020Vasileios Mezaris Adapting / re-purposing the content Behind the scenes: - Elaborate generative adversarial learning architectures (GANs) for unsupervised learning [9][10] - Can be trained differently for different content, e.g. separate trained models can be used for different shows; but, creating these models does not require manually-generated training data (it’s (almost) for free) [9] E. Apostolidis, A. Metsai, E. Adamantidou, V. Mezaris, I. Patras, "A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised Video Summarization", Proc. 1st Int. Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV'19) at ACM Multimedia 2019, Nice, France, October 2019. [10] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, I. Patras, "Unsupervised Video Summarization via Attention-Driven Adversarial Learning", Proc. 26th Int. Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020. 16
  17. 17. Thessaloniki, June 2020Vasileios Mezaris Some concluding thoughts 17 - AI tools for journalists are desperately needed! - Even relatively simple approaches, such as video fragmentation for reserse Web search, can have great impact (verification is becoming more important) - The introduction of video understanding outputs (concept labels etc.) in video archives can give new life to existing / continuously archived content - Automatic generation of new content (e.g. video summaries) is a hot topic - Complete automation is sometimes not desired! (AI + human symbiosis is key) - New challenges & advances in AI (e.g. Explainable AI) will create new opportunities for even greater use of AI in journalism
  18. 18. Thessaloniki, June 2020Vasileios Mezaris Questions? 18 Contact: Dr. Vasileios Mezaris Information Technologies Institute Centre for Research and Technology Hellas Thermi-Thessaloniki, Greece Tel: +30 2311 257770 Email: bmezaris@iti.gr, web: http://www.iti.gr/~bmezaris/ This work was supported in part by the EU’s Horizon 2020 research and innovation programme under grant agreements H2020-687786 InVID, H2020-732665 EMMA, H2020-693092 MOVING, and H2020-780656 ReTV.

×