TRACK 4(3) | DAY 1 - 2 OCT 2017 Sophie Mobbs, Senior lecturer of Middlesex University (UK) Games for Health Europe 2017

1. Gaming to Understand:
How games might help carers
support loved ones in
palliative care.

2. Original Aim: To create an artefact that would allow me to dissect non-verbal communication through the lens of
a games animator, to better understand subtle non-verbal communication (body language and facial expression)
as expressed via animation in order to feed back to the animation and games practitioner and animation and
games researcher community.

3. Focus of the artefact.
1. Real (not acted) emotion.
2. Hidden, subdued or otherwise unexaggerated emotion.
3. A self-reflective, practice led research piece.
4. An analysis of live action footage of “real” emotion, expressed and interrogated via different mediums of
animation, including…

4. 3D Motion Capture
Rotoscoping
2D Traditional Animation

5. Unexpected side effects of the research
An ability to notice tiny and small indications of emotion that would previously have been overlooked.
In particular, in interpretation of tiny facial movements and body postures exhibited by my father in the final
weeks of his life.
This lead to the question: I had originally wondered if the research would trigger greater awareness of viewers
to the expressed emotion, via the heightening (but also potentially distorting) lens of animation.
If this was the case could that heightened awareness be translated to carers in end of life care?
What would the results of the research reveal?

6. How the Work was presented.
1. A documentary-style interview discussing subjects to encourage feelings of genuine happiness and sadness in
the interviewee.
2. Presented as a mix of live action, motion captured 3D Animation, 2D rotoscoped animation and traditional
style 2d hand drawn animation.
3. This artefact was made available on Vimeo and members of the public were invited to view it via social media
networks including facebook groups and blog groups and journals. Viewers were encouraged to share the link
onto their own groups and networks.
4. On viewing, viewers were given the option to fill in an anonymous, online questionnaire.

7. Taking the Research Forward – Motion Capture.
Generally the motion captured sections were the most unpopular, with respondents
complaining of the stiffness and restricted movement of the 3D dinosaur, its lack of
facial expression. Thus to encourage the viewer to look at subtle body movements,
rather than facial expressions, the next approach might be...
1. A short game or app, designed to highlight clips of a motion captured avatar, and
encourage the player to interpret the body language without other forms of animation
or live action footage to distract the viewer.
2. A redesign of the character. While non-human and stylized, the 3D avatar was still
too realistic and her lack of facial expression dropped her into the uncanny valley. I
would suggest a design of a softer, fluffier character such as a soft toy or sock puppet
transposed into motion captured movement, where viewers are not expecting or
looking for realistic facial expression.
3. Motion capturing facial expression. A more complex approach requiring more
complex software and still difficult to capture the full range of human expression
successfully without hand tweaking the motion on top of the motion capture. The
problem would remain that viewers would look to the face before studying the body.

8. Taking the Research Forward – Rotoscoping
The rotoscoped scenes proved thought provoking, generally requiring the viewers
to focus on the lines and expressions, thus as the rotoscoped sequences appeared
to work quite well in getting viewers to focus and observe facial expression, further
work might involve.
1 A short game or app, showcasing rotoscoped clips of real (not acted expressions)
and encouraging the player to interpret and respond to the expressions shown.
2. Rotoscope played without voiceover as well as with voiceover, to see how much
(if any of a different reaction this prompts, and if it encourages viewers to look
more closely) an option might be to start without sound then blend in sound, to
see if the viewers interpretation of the emotion conveyed matches up.
3. Cut with live action footage. For example, beginning with a live action close up,
blending to a rotoscoped version and then blending back to a live action close up.
This to see if the blending helps viewers continue to focus on the live action face to
trace the details previously highlighted by the rotocoped lines.
The difficulty of the rotoscoped experiment is that is does require the time and skill
of a trained artist to interpret and hand animate the emotions by picking them out
from each frame. The success of the animation is dependent on how good the
animator might be at interpreting emotion, and there is the danger that the
animator might be adding to, subtracting from or distorting the data of the facial
expressions.

9. Taking the Research Forward – Hand Drawn Animation
While the hand drawn elements were entertaining and clearly understandable for the majority of the viewers
who responded, they did not appear to trigger fresh insights into the viewer’s eye on non-verbal communication
beyond what the researcher/animator had deliberately drawn and constructed. Thus the hand drawn animation
was more valuable to me as the creator of the artefact, helping me learn and understand non-verbal
communication, and had less value to viewers.

10. Conclusion
Working on this research appeared to really help me on a self-reflective and personal level in understanding
non-verbal communication in the context of caring for a relative at the end of life.
However, carers do not have the time to undertake years of research and practical animation to gain the same
effects.
On analysing the feedback, it would appear that the different methods of animation did seem to trigger a keener
focus for viewers on viewing the emotions depicted.
These animations, in particular rotoscoped pieces and possibly carefully set up motion captured pieces, could be
reworked into a simple game or app with a quick and cost-effective delivery (such as short games delivered free
online) could allow carers, particularly relatives, potentially at very short notice, to attune themselves to
noticing smaller cues and thus help palliative care patients.
I believe this is a tantalizing glimpse of a worthwhile avenue of serious gaming research that needs to be
thoroughly explored and tested beyond the research presented here