Y. Yamazaki, T. Horibe and A. Shirai, "Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience," 2022 International Conference on Cyberworlds (CW), 2022, pp. 110-113, doi: 10.1109/CW55638.2022.00024. https://ieeexplore.ieee.org/document/9937393 Best short paper award in Cyberworlds2022

1. Evaluation of time-shifted emotion through shared emoji reactions
in a video watching experience
Yusuke Yamazaki*1*2
, Takanori Horibe*1*3
and Akihiko Shirai*1*4
*1 GREE VR Studio Laboratory, REALITY Inc.
*2 Tokyo Institute of Technology *3 Meiji University *4 Digital Hollywood University
(Short paper)

2. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Many methods have been proposed to share viewers’ comments or reactions on videos.
→ Only a small percentage of viewers post comments.
Motivation
Left figure is cited from the following paper:
Qunfang Wu, Yisi Sang, and Yun Huang. 2019. Danmaku: A New Paradigm of Social Interaction via Online
Videos. Trans. Soc. Comput. 2, 2, Article 7 (June 2019), 24 pages. https://doi.org/10.1145/3329485
We want to applaud a great performance
independent of content, context, and language.
Highly dependents on the whole audience’s language ability,
language comprehension, and ideological understanding.
Verbal comment systems

3. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Motivation
● Focus:
“emoji emote”: emotive expressions using emojis.
● Hypothesis:
“time-shifted shared emotion”: Sharing the presence and emotion of other viewers
without constraint to either past or real time.
● Investigation:
How other people’s emoji emotes affect viewers’ behavior and impression.

4. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Related work:
Comparison of reactions using comments or emojis
No evaluation of the impact of emojis displayed
in videos on the viewers’ behavior and impressions.
“Onion” [Kagawa et al. 2010]
Evaluation of their system by comparing
the number of comments and emojis posted on videos.
→ Emojis can encourage more user engagement.
This figure is cited from the following paper:
Kagawa Kentaro, Ito Junko and Munemori Jun. Danmaku: Effects of
an intuitional pictograph comment function in a video sharing web
system. Int. J. Informat. Soc. vol. 2, pp. 94–99 (2010).

5. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Our proposed system: Continuous tap
emoji, button-tap type “continuous tap”
Continuous tap detection: The emoji button is tapped twice (or more) within 1 second.
● Solving practical problem such as large number of server requests.
● Expressing the magnitude of the viewers’ emotion.

6. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Verification of the effect of other viewers’ emoji.
We collected viewer’s behaviors and impressions（視聴者の行動と印象）
by the stimulus emojis under the following conditions:
“None”: No stimulus emoji “Less”: Three stimulus emojis “Many”: Nine stimulus emojis
Definition “stimulus emoji”

7. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Evaluation - “stimulus emojis”
"Back to Metaver-School" - a blooming
future of Metaverse in REALITY
https://youtu.be/1JlD6XF2HNI
[At 13 s] a boy avatar juggles a soccer ball [At 37 s] a girl avatar plays the guitar on the stage
[At 60 s] student avatars clap in the classroom [At 101 s] girl avatars play a jam session
Original Video
👏, 😲, 😀 and 😍 as stimulus emojis at these particular scenes.

8. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Evaluation - “stimulus emojis”
"Metaverse Christmas"
- UXDev by GREE VR Studio Laboratory
https://youtu.be/0eKx-mlRNd4
Original Video
Verification whether viewer behavior was influenced by the stimulus emojis or the video content.

9. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Procedure of Experiment
Participants:
● 96 volunteers who played the video and sent emojis at least once.
(24 volunteers in each condition)
● watched the video on their own devices (a computer or smartphone).
Participants who did not answer the questionnaire were regarded as dropouts who left the experiment
part-way through.
Watching the video in only one randomly
assigned condition.
Answering to the online questionnaire.
The transition button appear
at the end of the video

10. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Questionnaire
1. Please select the best method for sharing your reactions when watching a video. (Both text
comments and emojis / Only text comments / Only emojis / I don't want to share my reactions)
2. Was it easy to react with emojis? (Difficult / Easy)
3. Did not the emojis displayed on the screen be an obstacle?
(Not obstacles / Are obstacles)
4. Did you feel that there were other people watching the video besides you?
(Did not feel / felt)
5. Please select all of the following words that apply to your interpretation of 👏, 😀, 😍 and 😲:
clapping, amazing, beautiful, cute, cool, like, dislike, astonished, exciting, boring, (not in use).

11. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Purpose: Investigation of the impact of cultural differences on video content.
Experiment: With the stimulus emojis in the Many condition.
Targets: 18-47 years old men and women in Japan and the U.S.
Survey using SurveryMonkey.com
Japan , n=14
USA , n=32
Yes: 46.88% (n=15)
No: 53.13% (n=17)
Yes: 50% (n=7)
No: 50% (n=7)
Question: If there is a button to express your emotion, do you want to tap it? (Yes / no)

12. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Bottom axis: Video Playback time [s]
Result
Requests
Left axis: The cumulative count of taps or requests within 5 s.
Right axis: The single moving average (5 s window with the resolution of 1 s/red line)
Taps

13. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Result
In each condition,
● Taps/Requests tend to increase toward the end of the
video.
→ Improvement of the video viewing retention rate
by our proposed system.
● Focus on the changes in Taps/Requests,
it is possible to get which parts the viewers attend.
→ Content evaluation in a resolution in a few seconds.

14. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Taps
Requests
Result: Comparison Less/Many and None
(In 20s/106s)
Taps is a strong increase and Requests is small. This means “a continuous tap/連打”.
It is possible to detect the presence of enthusiastic viewers at the corresponding video playback time.

15. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Result: Comparison Less/Many and None
In around 20 s/106 s,
● Taps is a strong increase and Requests is small
→ “a continuous tap/連打”
It is possible to detect the presence of enthusiastic
viewers at the corresponding video playback time.
Not a characteristic scene in this video.

16. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Conclusion
Evaluation of our proposed system:
● To easily express “emoji emote” to a video by simply tapping emoji buttons.
→ (Q1-Q3) The participants favor the proposed system.
● To feel “time-shifted shared emotion”
→ (Q4) More emojis make viewers feel the presence of other viewers.
Q1. Please select the best method for sharing your reactions when watching a video.
Q2. Was it easy to react with emojis?
Q3. Did not the emojis displayed on the screen be an obstacle?
Q4. Did you feel that there were other people watching the video besides you?
Result of Questionnaire - Viewers’ behavior and impressions
Our proposed system focus on “emoji emote” enables an interactive watching experience.
None Less Many Many-B
Q1 [Prefer text] (n) 6% (1) 13% (2) 14% (3) 0% (0)
Q2 [Easy to react] (n) 82% (14) 81% (13) 86% (19) 86% (18)
Q3 [Not obstacle] (n) 88% (15) 93% (15) 73% (16) 57% (12)
Q4 [Felt others] (n) 35% (6) 69% (11) 68% (15) 81% (17)

17. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Discussion - Timeline
The peaks of the moving averages in Many-B: 49 s, 70 s
(except for the ending around 105 s)
→ all at different places from video-A.
● The timing viewers send emoji depends on the video content.
● The stimulus emojis have no effect on the timing.
[At 49 s] the avatar changes into a dress, [At 70 s] the avatar start playing piano

18. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Discussion - Emoji interpretation
Q5: participants’ interpretation of 👏, 😀, 😍 and 😲.
● The variance of interpretation for each emoji.
● A larger variance:
→ many participants’ interpretations tend to be consistent.
分散大 = emojiの解釈が似た傾向にある.
[👏: 😀: 😍: 😲]=[471:143:368:269].
The results of Q5 shows:
● 👏 and 😍 can express relatively clear emotion and are suitable for
co-occurring emotion (明確な感情表現が可能).
● 😀 is relatively interpretations differ among participants.
(視聴者間で傾向が分かれる).

19. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Conclusion
Our verification:
● To easily express “emoji emote” to a video by simply tapping emoji buttons.
→ the participants favor the proposed system.
● To feel “time-shifted shared emotion”
→ More emojis make viewers feel the presence of other viewers.
Through the evaluation,
● “Time-shifted” was understandable, and stimulus emojis was not affected directly.
● Content evaluation in a resolution in a few seconds.
（数秒の解像度でのコンテンツ評価）
● Improvement of the video viewing retention rate.
（動画視聴の継続率の向上）
Our proposed system focus on “emoji emote” enables an interactive watching experience.

20. Evaluation of time-shifted emotion through shared emoji reactions in a video watching experience
Q: What was the purpose of each emoji?
● It is impossible to fit many emoji all into on UI.
→ Focus on emoji using most often to express emotion: 👏, 😀, 😍 and 😲.
● The our system aim to contents in live entertainment.
→ No need to express negative emotions. (Even if the purpose is for e-learning.)
● Verbally comments creates a time lag.
→ It is possible to share in a very short time (about 1-2 s),
the evaluation is also in 5-10 s.