Special Session: “New Normal” Science, Multimodality, And Inclusive Societies https://tsukuba-conference.com/sessions/s-5

1. How to Make Good Use of AI Technologies
Clarifying the Boundary between Humans and Machines
2021.09.30
Hiromu Yakura
University of Tsukuba
Tsukuba Conference

2. 2
• Hiromu Yakura
• Ph.D. student at University of Tsukuba
• Google Ph.D. Fellow (2020)
• Research area: Human-Computer Interaction
• HCI is an interdisciplinary
fi
eld that
explores new ways of using computers.
• What HCI has invented: mouse, multitouch interface, etc.
• Particularly, my research question is:
Self-Introduction
@hiromu1996
How we can make a human-centric use of AI technology
that is imperfect and often makes a mistake?

3. How to Make Good Use of AI Technologies
Clarifying the Boundary between Humans and Machines
2021.09.30
Hiromu Yakura
University of Tsukuba
Tsukuba Conference

4. How to Make Bad Use of AI Technologies
Clarifying the Boundary between Humans and Machines
2021.09.30
Hiromu Yakura
University of Tsukuba
Tsukuba Conference

5. Robust Audio Adversarial
Example for a Physical Attack
Hiromu Yakura*†, Jun Sakuma*†
* University of Tsukuba, Japan
† RIKEN Center for Arti
fi
cial Intelligence Project, Japan
IJCAI 2020

6. Background: Risk of abusing AI using adversarial example
6
• It is known that we can mislead AI models by
intentionally adding a small noise to the inputs.
• In this case, the right image is classi
fi
ed as a gibbon.
Using this adversarial example, we can abuse
AI-based systems without being noticed by humans.
Goodfellow, I. J., Shlens, J., & Szegedy, C.: Explaining and harnessing adversarial examples. In Proc. of ICLR. (2015)

7. Background: Risk of abusing AI using adversarial example
7
• The risk of such an attack is not limited to images,
but also to speech recognition.
• In particular, speech recognition is widely used
in the form such as Siri or Google Home.
However, abusing speech recognition in the real world has
been considered di
ffi
cult due to reverberation. [Carlini+, 2018]
• If we can mislead speech recognition models to transcribe
speci
fi
c words, such voice assistants can also be abused.
Carlini, N., & Wagner, D.: Audio Adversarial Examples: Targeted Attacks on Speech-to-Text. In Proc. of Deep Learning and Security Workshop. (2018)

8. • In a listening experiment involving 50 participants,
we con
fi
rmed no one could tell these hidden messages.
8
Proposal: Robust attack for speech recognition
We realized such an attack in the real world
via a new method that simulates reverberation.
Original audio
"hello world" "open the door"
Manipulated audio transcribed as

9. Generate (non-software) Bugs
to Fool Classi
fi
ers
Hiromu Yakura*†, Youhei Akimoto*†, Jun Sakuma*†
* University of Tsukuba, Japan
† RIKEN Center for Arti
fi
cial Intelligence Project, Japan
AAAI 2020

10. Proposal: Attacking self-driving cars with moth-like stickers
S. Chen, et al. Shapeshifter: Robust physical adversarial attack on faster RCNN object detector. ECML PKDD 2018.
• This mechanism is also applicable to deceive self-driving cars.
• What if the cars recognize a STOP sign as Speed 80?
• This example can cause such a mistake
but looks too suspicious not to be noticed by humans.
10
[Chen+, '18]
Proposed
We showed that these moth-like stickers can mislead
AI models without making humans feel suspicious.

11. How to Make Good Use of AI Technologies
Clarifying the Boundary between Humans and Machines
2021.09.30
Hiromu Yakura
University of Tsukuba
Tsukuba Conference

12. 12
Basic idea: How to overcome the imperfection of AI models
No matter how much technical improvement we make,
AI-based systems will make mistakes.
• AI models just perform inference based on the trends
they found in the data given in advance.
• Thus, we can't deny the possibility of their mistakes,
even if we don't intentionally attack them.
Find situations even such imperfect models are beneficial!

13. 13
Basic idea: The importance of understanding humans
• It may sound like humanities or sociology 🤔
but clarifying the area that humans are not good at is crucial.
Observe humans and gain a deeper understanding of humans!
Humans are
not good at good at
AI models are
good at
not good at
Not so bene cial
Frustrating
Sweet spots
Intractable

14. REsCUE: A framework for
REal-time feedback on behavioral CUEs
using multimodal anomaly detection
Riku Arakawa† and Hiromu Yakura‡
(equal contribution)
† The University of Tokyo, Japan
‡ University of Tsukuba, Japan
ACM CHI 2019 @ Glasgow, UK

15. Background: Limitation of current conversation analysis
15
• In executive coaching, coaches are required to observe
nonverbal cues of a client consistently while they are talking.
• AI models that can estimate the emotions of humans would
be helpful for supporting coaches.
• However, professional coaches felt
the models are useless and frustrating.
• The meaning of nonverbal behaviors
largely depends on individuals and contexts.
I. Damian, et al. Augmenting Social Interactions: Realtime Behavioural Feedback using Social Signal Processing Techniques. ACM CHI 2015.
<%BNJBO

16. Proposal: Separating observation and interpretation
16
Humans AI models
• AI models detect nonverbal signals from a constant perspective.
• Humans interpret their meaning considering the context.
Separating observation and interpretation can enable
a better collaboration of humans and AI models.
👎 understand contexts of humans
👍 understand contexts
👎 maintain consistent observation
👍 exhibit constant performance

17. Proposal: Interpretable detection of nonverbal signals
17
• AI-based system detects when the client exhibits
a sudden change in their behavior and visualizes it.
Representative states Changes in nonverbal behavior
This system is designed to help
humans show the best performance
in the area they are good at.

18. 18
Basic idea: The importance of understanding humans
Humans are
not good at good at
AI models are
good at
not good at
Not so bene cial
Frustrating
Sweet spots
Intractable

19. 19
Advanced idea: Frustrations teaches success
Humans are
not good at good at
AI models are
good at
not good at
Not so bene cial
Frustrating
Sweet spots
Intractable
Investigating why humans are frustrated against technologies
provides implications for the direction of improvement.

20. 🤝 No More Handshaking 🙅
How have COVID-19 pushed the expansion of
computer-mediated communication in Japanese idol culture?
Hiromu Yakura
University of Tsukuba, Japan
ACM CHI 2021
20

21. COVID-19 has forced Japanese idol groups
to halt their meet-and-greet events.
akushukai
(handshaking event)
The photo is taken from Wikimedia Commons under the CC-BY-SA 2.0 license: https://w.wiki/3AF4
These events have played a key role
in their marketing strategy to
cultivate fans’ loyalty
Instead, most idol groups have migrated to
computer-mediated communication (CMC).
21

22. One common approach was online one-on-one sessions.
Fans who purchased a CD are allowed to
talk to a member via a video call.
22

23. Emergent forms of online meet-and-greet events
The image is taken from a YouTube video only for presentation purpose: https://www.youtube.com/watch?v=6yOxmVoAE14
Fans were not allowed to get closer than four meters to members.
Instead, they can talk via a video call using a pair of iPads.
On-site online one-on-one event
23

24. Why did fans gather in the venue to talk via iPad?
There is something available in this event but not available
when they join a video call from their home.
This teaches that what HCI researchers should do is:
24
Aura: the unique existence of a work of art
at the place where it happens to be
Even the most perfect reproduction lacks [it]
Walter Benjamin
(1892-1940)
🙅 Improve the reproduction quality
🙆 Develop new channels that deliver the unique existence

25. How to Make Good Use of AI Technologies
Clarifying the Boundary between Humans and Machines
• Observe humans and gain a deeper understanding to
fi
nd sweet spots where AI-based systems are bene
fi
cial
• This approach simultaneously provides
a new perspective of understanding humans.
• Finding ways to make good use of AI technologies can be
a re
fl
ective process questioning our identity.