Mais conteúdo relacionado Semelhante a Artificial Intelligence (AI) in media applications and services (20) Mais de Förderverein Technische Fakultät (20) Artificial Intelligence (AI) in media applications and services1. Artificial Intelligence (AI) in media applications and services © IRT 2019
1
Artificial Intelligence (AI) in media
applications and services
Dr.-Ing. Christian Keimel
keimel@irt.de
2. Artificial Intelligence (AI) in media applications and services © IRT 2019
2
Agenda
Outline of today’s presentation
• AI – what is it?
• AI in Media
• Monitoring as a use case for AI in media
• Research topics for AI in broadcasting
• Challenges and open questions
4. Artificial Intelligence (AI) in media applications and services © IRT 2019
4
Unsere Mission
As a worldwide renowned key
research and competence centre for
audio-visual technologies we
research, evaluate, and develop new
technologies with the aim to strat-
egically adjust broadcast concepts to
new market environments and needs
Representing Broadcasters Standardisation
Maintaining Interoperability
Technology
Evaluation
Technology Scouting
Prototyping and
Pilots
Applied Research
Supporting Technology
Introduction
Knowledge TransferServices
6. Artificial Intelligence (AI) in media applications and services © IRT 2019
6
Our topics
Next
Generation
Audio
Future
Video
Artificial intelligence
Meta data
All IP/IT
IP distribution
Platforms & services
Accessibility &
Design for all
5G
7. Artificial Intelligence (AI) in media applications and services © IRT 2019
7
IRT in numbers
• Research and competence centre of the public broadcasting corporations in Germany
(ARD, ZDF, Deutschlandradio), Austria (ORF) and Switzerland (SRG/SSR)
• Location: Munich, Germany (BR TV production facility Freimann)
• Non-profit limited liability company with 14 associates
• Founded in 1956
• Approx. 130 employees
• Annual budget: ~ 25 Mio. €
8. Artificial Intelligence (AI) in media applications and services © IRT 2019
8
Deep Learning and AI
Artifical Intelligence (AI)
Machine Learning (ML)
Neural Networks (NN)
Deep Learning (DL)
Technologies
Behind the current
trend of „AI“
Recent progress in „AI“
mostly done here
9. Artificial Intelligence (AI) in media applications and services © IRT 2019
9
What is AI? – Definitions (1)
Definition 1:
„Colloquially, the term "artificial intelligence" is applied when
a machine mimics "cognitive" functions that humans associate
with other human minds, such as "learning" and "problem
solving“ “
[Russel, S., Norvig, P. (2009): Artificial Intelligence: A Modern Approach]
10. Artificial Intelligence (AI) in media applications and services © IRT 2019
10
What is AI? – Definitions (2)
Definition 2:
„Artificial intelligence (AI) refers to systems that display
intelligent behaviour by analysing their environment and
taking actions –with some degree of autonomy –to achieve
specific goals.“
[European Commission, COM/2018/237 final]
11. Artificial Intelligence (AI) in media applications and services © IRT 2019
11
What is AI? – Definitions (3)
Definition 3:
„Artificial intelligence (AI)—defined as a system’s ability to
correctly interpret external data, to learn from such data, and
to use those learnings to achieve specific goals and tasks
through flexible adaptation“
[Kaplan, A., Haenlein, M. (2019): Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations,
illustrations, and implications of artificial intelligence]
12. Artificial Intelligence (AI) in media applications and services © IRT 2019
12
What is AI? – Where are we today?
Strong/General AI Weak/Narrow AI
• Can solve any problem
• Equal to human intelligence
• Can solve specific problems
(e.g. speech recognition)
State-of-the-Art
13. Artificial Intelligence (AI) in media applications and services © IRT 2019
13
Machine Learning – Basics
System learns/trains “patterns/rules” based on representative data
Training data is often labelled (supervised learning)
Cat
Dog
Data
System
CatLabel
Data
point
Cat-Dog-Classifier
Classes
14. Artificial Intelligence (AI) in media applications and services © IRT 2019
14
After the completed training, new data from classes contained in the training
can be recognised (inference)
Data from unknown classes not contained in the training data won’t be
recognised or assigned a wrong class
New data,
known class
Machine Learning – New data
Cat
Dog
New data,
unknown class
Cat
Dog
96%
4%
96%
4%
Confidence
Wrong classification
15. Artificial Intelligence (AI) in media applications and services © IRT 2019
15
Training data – Where to get it?
Existing data
Already labelled data (for supervised learning) in (freely) available training sets
Example: ImageNet
Labelling the data yourself
Manually adding a label to each data point
Can be done inhouse (e.g. in workflow) or using online service provider
(crowdsourcing)
16. Artificial Intelligence (AI) in media applications and services © IRT 2019
16
Training data – Where to get it?
Indirect labelling by (end-)users in applications
Users perform the labelling task in the context of an
arbitrary application
Users do not explicitly know that they are
performing a labelling task
Example: security checks (Captchas)
17. Artificial Intelligence (AI) in media applications and services © IRT 2019
17
Neural Networks – Artificial Neurons
Based on the structure of neurons encountered in nature
First concepts and developments in the 1950s
Σ
summation
𝑥1
𝑥2
𝑤1
𝑤2
input weights
activation
function
output
non-linearity
𝑧 = ቊ
0, 𝑦 < 0
𝑥1 𝑤1 + 𝑥2 𝑤2, 𝑦 ≥ 0
𝑦
𝑥1 𝑤1 + 𝑥2 𝑤2
(in this example)
18. Artificial Intelligence (AI) in media applications and services © IRT 2019
18
Neural Networks – Training
Weights w are iteratively changed until z equals the values of the
labels in the training set (or rather a cost function has been optimised,
usually minimised)
Simple example:
Σ
𝑥1
𝑥2
𝑤1
𝑤2
𝑧𝑦
𝑥1 = 2, 𝑥2 = 4 → 𝑧 = 2
1
0,5
0,5
1
0,5
0,25
6 ≠ 2
3 ≠ 2
𝟐 ≡ 𝟐
𝟐
𝟒
Repeated for all data in the
training set
19. Artificial Intelligence (AI) in media applications and services © IRT 2019
19
Neural Networks – Architecture
Combination of multiple neurons in a network that consists of
multiple (hidden) layers that are connected
First application in the 1980s (MLP)
Cat
Dog
hidden layer(s)
10 weights15 weights
25 weights
per iteration,
per data point
input layer
20. Artificial Intelligence (AI) in media applications and services © IRT 2019
20
Deep Learning
Many layers, specific architectures suitable for image/video
processing (CNN) or temporal dependencies (RNN, LTSM)
Became popular since the 2010s…
…
…
…
Cat
Dog
… … …
many hidden layers (>> 2)
millions of parameters,
per iteration,
per data point
21. Artificial Intelligence (AI) in media applications and services © IRT 2019
21
Deep Learning – Why now?
The current success of deep learning has many reasons:
• Five decades of research in machine learning
• CPUs/GPUs/storage developed for other purposes
• lots of data from the Internet
• resources and efforts from large corporations
• tools and culture of collaborative and reproducible science
Resulting in frameworks e.g. Tensorflow
22. Artificial Intelligence (AI) in media applications and services © IRT 2019
22
Deep Learning – CPU/GPU power
[Francois Fleuret (2019), Deep Learning]
Flop/USD
23. Artificial Intelligence (AI) in media applications and services © IRT 2019
23
Deep Learning – Lots of Data
Example: image data sets
[Francois Fleuret (2019), Deep Learning]
24. Artificial Intelligence (AI) in media applications and services © IRT 2019
24
Deep Learning – Performance
ImageNet data set
• 1000 categories,
• > 1M images
25. Artificial Intelligence (AI) in media applications and services © IRT 2019
25
Deep Learning – Performance
Object recognition ImageNet data set
28,2
25,8
16,4
11,7
6,7
3,6
5,1
2 2
8 8
22
152
0
20
40
60
80
100
120
140
160
0
5
10
15
20
25
30
2010 2011 2012 2013 2014 2015 Human
ErrorTop-5[%]
Layers
26. Artificial Intelligence (AI) in media applications and services © IRT 2019
26
[Krizhevsky et al. (2012), He et al. (2015), Szegedy et al. (2015)]
Deep Learning - Architectures
AlexNet
2012
ResNet
2015
GoogLeNet
2015
27. Artificial Intelligence (AI) in media applications and services © IRT 2019
27
Deep Learning – Acurracy vs complexity
[Canziani et al. (2017)]
28. Artificial Intelligence (AI) in media applications and services © IRT 2019
28
Deep Learning – Performance
Example: object recognition in images with cloud services
November 2017
„a man brushing his teeth“
January 2019
„a sculpture of a man“
[Alberto Messina, RAI (2019)]
29. Artificial Intelligence (AI) in media applications and services © IRT 2019
29
Deep Learning – Comparison to nature
Number of neurons in network compared to nature
[Goodfellow et al. (2016): Deep Learning]
30. Artificial Intelligence (AI) in media applications and services © IRT 2019
30
Deep Learning as AI – Products & Services
Similarities
• Same or fairly similar architectures are often used
Differences
• Different parameters in learning process i.e. hyperparameters
• Amount and quality of training data
Training data is essential for accuracy
31. Artificial Intelligence (AI) in media applications and services © IRT 2019
31
Audio-visual content understanding
Video
sentiment
analysis
semantic
analysis
sentiment key words/
concepts
object
recognition
face
recognition
identity sentimentobjects
context free with contect
audiovisuell
sound
recognition
objects
speaker
recognition
identity
XXXXX
textual
OCR
speech-to-text
(transcription)
text
32. Artificial Intelligence (AI) in media applications and services © IRT 2019
32
AI in media
Understanding of audio-visual content
• Extracting information from audio and video assets for indexing and use in
further applications:
o Enrichment of archive content for increased „findability“
and reusability
o Enrichment during production via object recognition
o Customised views for different target groups
o Recommendations (editors and consumers)
o Verification
33. Artificial Intelligence (AI) in media applications and services © IRT 2019
33
AI in media
Automatic content generation
• “Robojournalism” & individualised content
• Teaser/trailer/highlight creation, auto-summarisation
Optimisation in distribution and production
• Streaming & encoding decisions, network routing etc.
News gathering/Monitoring
• Trend detection in/monitoring of audio-visual (news) content
34. Artificial Intelligence (AI) in media applications and services © IRT 2019
34
Should detect
topic here
Monitoring – Motivation
Life cycle of a news event
Popularity/visibility
„Trending topic“
→ Peak popularity
t
Event is
spreading
Event is
getting stale
35. Artificial Intelligence (AI) in media applications and services © IRT 2019
35
Role of Social Media
Social Media plays an increasingly important role in the news cyle
t
Original post
& dissemination starts
Dissemination gains
speed through influencers
(hash tag becomes popular)
Popularity/visibility
36. Artificial Intelligence (AI) in media applications and services © IRT 2019
36
Twitter vs Instagram
Users/Source „offical“ „private“
Demography Older Younger
Content mostly Text Images/Videos
Hash tags in posts Few Lots
Geolocation use Uncommon Very common
37. Artificial Intelligence (AI) in media applications and services © IRT 2019
37
Is considering only text enough?
• Hashtags and text are chosen by the creator of the message
• Don‘t necessarily describe the complete content of audio-visual assests
News events are potentially not visible in the trends,
especially for regional events with few „influencers“
38. Artificial Intelligence (AI) in media applications and services © IRT 2019
38
Idea
Analyse audio-visual content from (public, localised) social media posts,
recognise concepts/object with AI, and use this information for trend detection
Image/
Videos Concepts/
Objects Trends
45. Artificial Intelligence (AI) in media applications and services © IRT 2019
45
Proof of concept – Video detection results
Nach unten scrollen für Transkript & mehr…
46. Artificial Intelligence (AI) in media applications and services © IRT 2019
46
Why use AI in media and broadcasting?
Benefits of using AI in broadcasting
Content – Creator
• Cost savings
• Workload reduction
→ Assign people to more important tasks
• Staying relevant (competition by Social Media etc.)
Content – Consumer
• Increase audience
Technolgy/Distribution
• Cost savings by more efficient use of resourccess
47. Artificial Intelligence (AI) in media applications and services © IRT 2019
47
Research topics under consideration at IRT
Addressing challenges in broadcast-related applications
• Verification: how to verify audio-visual content for authenticity/source
• Speech-to-text for dialects: “of-the-shelf” models work not very well for
non-standard speech
• Clean Feed: automatic customising for multiple distribution channels
requires clean feeds from “normal” material
• Training diversity: existing large-scale training material are not
representative enough for regional content
48. Artificial Intelligence (AI) in media applications and services © IRT 2019
48
Verification
Verify content (semi-)automatically
• Automating existing concepts
Automate manual concepts for content verification that are
already used e.g. time/place verification via shadow length
https://www.suncalc.org/#/49.2548,7.0426,16/2019.04.10/15:15/2/1
49. Artificial Intelligence (AI) in media applications and services © IRT 2019
49
Speech-to-text for dialects
Extend speech-to-text beyond “standard speech”
• Transfer learning for dialects
Leverage large corpus of “standard speech” for (pretrained)
base model, then retrain model with smaller corpus of dialects
corpus model
Standard speech
model
Dialect
corpus
retraining
50. Artificial Intelligence (AI) in media applications and services © IRT 2019
50
Clean Feed
Automatic customising for multiple distribution channels requires clean feeds
• Detection of salient objects and prioritisation
“Objectification” of audio-visual content for distribution dependent
rearrangement with preservation of semantically most salient objects
„objectification“ &
semantic analysis
source material
1
2
3
prioritisation distribution
formats
51. Artificial Intelligence (AI) in media applications and services © IRT 2019
51
Training diversity
Creating more representative training material automatically
• Leveraging “hidden” meta data
Using existing information in audio-visual assets to generate
labels for training e.g. information in lower thirds banner text
Label
Data
Labels for
face recognition
52. Artificial Intelligence (AI) in media applications and services © IRT 2019
52
Training diversity
Alternative to strictly supervised learning approach
• Reinforcement learning
Using implicit user feedback contained in workflow actions as
environment variable(s)/reward e.g. selection of specific result
from search results
model
Selected result
reward
53. Artificial Intelligence (AI) in media applications and services © IRT 2019
53
AI challenges
Training data
• Diversity: regional content/languages and/or minorities are not
represented sufficiently
• Bias free: all classes are not always represented uniformly; bias can be
problematic (example „fake news“)
• Reproducibility: if the training data is changing, the models change
i.e. predictions may change over time
54. Artificial Intelligence (AI) in media applications and services © IRT 2019
54
AI Challenges
Architecture/algorithms
• Daten protection/privacy: often not a major priority, especially as the
development of AI technology is driven by companies from the USA and
China (for cloud services)
• Explainibility: Predictions can sometimes not be understood/explained
• Portability: trained models are not necessarily portable between
cloud providers
• Accuracy: even 99% accuracy is not sufficient for fully automated
systems in some use cases
55. Artificial Intelligence (AI) in media applications and services © IRT 2019
55
AI – Assistive or Autonomous?
Assistive AI
• Assists creators/journalists/operators in content creation and editing
• Final decisions und responsibility stays with a human
Autonomous AI
• Replaces creators/journalists/operators in content creation and editing
• No human in the loop – who will be responsible for the decisions?
Mandatory identification of AI generated content?
56. Artificial Intelligence (AI) in media applications and services © IRT 2019
56
Data protection & privacy
Is AI even causing any new issues?
• Many problematic applications have already been possible, but until now
the weren‘t done as it wasn‘t effective on a large scale
• Example: complete surveillance with face recognition in public places was
to expensive with human operators, but now „cheap“ with AI
• But easier implementation of such applications lower inhibitions
against using them
Is it sufficient to us existing regulations?
57. Artificial Intelligence (AI) in media applications and services © IRT 2019
57
AI – What will the future hold?
Hypothesis
• Weak AI based system for audio-
visual content recognition are
standard and will remain so
• Use of the term „AI“ will become less
popular, but the technology behind it
will continue to be used
58. Artificial Intelligence (AI) in media applications and services © IRT 2019
58
All rights reserved. All text, images, graphics and charts are protected by copyright.
Reproduction or use of the content is not permitted without the express consent of the author.
Experts in audio-visual media
Dr.-Ing. Christian Keimel
Data & Security
Floriansmuehlstraße 60
80939 Munich
Tel +49 89 323 99 – 303
FAX +49 89 323 99 – 351
www.irt.de
keimel@irt.de
Thank you for your attention!