Thesis Documentation

Contents
3 Research
4 User interviews
5 Six mindsets when we reminisce
13 Extended Reality technology
17 Memory product anlaysis
29 Design & Protyping
31 Technical investigation
34 Design process
37 Art direction
41 Technical challenges
44 User testings
23 Ideation
25 Memory aggregator
26 Emotion theory
27 User flow
50 Final Project
65 Future Potential
73 References
1
MOMENT | Research

MOMENT | Research
MOMENT
An experimental design project exploring how
extended technology could support reminiscing
behaviour. Utilizing geolocation-based AR
technology, I created this magical experience
which helps people to revisit their past
memories at the locations where those
experiences happened. The adventure-like
interaction design transforms the way we
reminisce, and allows us to generate positive
self-concept and create a strong sense
fcontextualized presence.
Demo Video: https://vimeo.com/404385691
Github Repository: https://vimeo.com/404385691
2

Research
User interviews
Literature review
Product analysis
BehaviourProductTechnology
3

User Interviews
Questions asked
Full Interview Script:
https://docs.google.com/document/d/1gD4k4mO0epJ_G0juSdfp_STsTR7A2TNTEsY9zxg7bzs/edit?usp=sharing
I conducted 6 in-depth interviews
with people in various occupations
and different life stage in order to
gain a more comprehensive
understanding of their reminiscence
behaviours.
Artist/Designer, Engineer, Accountant,
Entrepreneur, Manager, Architect
Student, mid-20, mid-30,
mid-40 and retired people.
MOMENT | Research
4

Six mindsets
that explain
why we
reminisce
5

Reminiscing is a universal behaviour that
consists of memory revisiting and
self-narrative generation. It is an important
human behaviour that helps us to develop
a strong self-concept, enhance the social
bond and improve our future thinking skills.
In the article Informing augmented memory
system design through autobiographical
memory theory, the author describes the
function and importance of memory in the
context of autobiographical memory
theory: construct our self-concept,
maintain social relationships and predict
the future. By recalling life events, people
are developing a strong self-concept that
allows them to understand and remember
who they think they are, and connect those
experiences with the current situation,
which ultimately helps them clarify the
self-identity, and create a closure that
connects their past self and the present
self. Research in autobiographical memory
theory shows the importance of
developing self-narrative. Thus, I decided
to create a feature that allows people to
write a short note about their feeling when
they revisit the memory to encourage
them to articulate abstract thought in a
writing form for revisiting in the future.
Memory as
self-making
narratives
MOMENT | Research
6

Reminiscing can be the consequence of memory sharing
which is a common behaviour in social settings. When we
meet friends or strangers, we tend to share experiences
to prove the argument or build empathy to enhance the
social bond. We exchange stories to maintain and deepen
the social relationship. In this case, memory becomes
the material of social engagement. Therefore, one of the
use cases of my project is in social settings: imagine that
you have a friend reunion at the place where you used to
hang out together, my project allows them to easily share
and revisit those moments and facilitate the reminiscing
practice.
Besides, memory can also stimulate the future, as we
constantly reflect on past experiences and try to
recognize similar patterns in current and future scenarios.
We gain affirmation and denial, and these experiences
regulate the way we behave in future scenarios.
Past Present Future
MOMENT | Research
7

We unconsciously
develop biases in
reminiscing
Adaptive memory theory shows that people
tend to look back at past experience from
an adaptive perspective when revisiting past
experience: they are more likely to think
positively and unconsciously develop biases by
embracing a more positive mindset, a rosy
retrospection, and developing fading affect
bias.
MOMENT | Research
8

We unconsciously develop biases in
reminiscence and tend to look back at
past experiences from an adaptive
perspective. According to adaptive
memory theory, people are more likely
to have a positive mindset to
understand what they did to overcome
those negative experiences happened
in the past. People choose to distance
themselves from past events and look
back at it from a third-person
perspective so that they can make
sense of it, and embrace mistakes they
have made. In my work, the idea of
discovering memory footprint and
interacting with the floating memory
bubble transforms people’s
perspective to an adventure-like
experience: although they are revisiting
their memory, the interaction
mechanism makes it like a new
adventure.
Rosy retrospection and fading affect
bias are also at play in reminiscing. In
the article Technology-mediated
memory: Is technology altering our
memories and interfering with
well-being, the author argues that all of
these unconscious biases reduce the
pain in reminiscing, and can help to
maintain well-being in the long run
instead of posing threat on mental
health.
MOMENT | Research
9

Memory and
Location
Memory is contextually affected and can be triggered
by contextual information, such as geolocation, smell,
sound, and physical state. The AR project Nevermind
created by MIT demonstrates how contextual
information such as location could help people
improve memorize skills. This project proves the power
and the effectiveness of utilizing location as a memory
trigger. The strong association between location and
memory is the major reason I chose to develop a
location-based AR memory project, and also the
biggest differentiator between my project and other
memory products
Image Credit: NeverMind, Oscar Rosello, MIT Media Lab
https://www.media.mit.edu/projects/nevermind/overview/
MOMENT | Research
10

A place is a
contextualized space
Another location-related feature in my project is a memory
footprint map that allows people to see their memory
located on the city map, and I use different colours to
represent the emotion attached to the memory pieces.
This feature generates a psycho-geographical map to let
people understand how their emotion changes around the
city and provide a new way of appreciating life and the
city. It can also be seen as a tool to help people learn
about themselves, and even uncover their true desires in
everyday life. This idea responses to situationist: build a
personal understanding of the city via lived experience
rather than being limited by the function and physical
existence of urban planning.
MOMENT | Research
11

Technology-Mediated
Memory
We constantly post information on social media, and later on, this
content becomes the memory material to support our reminiscing
behaviour. Our life is also recorded and mediated by technology, and a
huge amount of metadata is generated alongside the organic content.
For example, when we take photos, our phone automatically captures
the location tag, date, and even recognize the people, which makes
contextual understanding possible.
Technology mediated memory is more accessible, selective, and can be
displayed in a rich media format. If human memory is the raw material
that has been collected and stored in our mind, technology facilitates
the process of memory encoding and retrieval by providing a huge
amount of records and accessible opportunities for self-reflection, and
enable content curation on past experience.
MOMENT | Research
12

XR Industry Report
Biggest Barrier to Mass
Consumer Adoption of VR
46.2%
Price of HMDs
of respondents said
45.6%
Lack of content
of respondents said
38.1%
Current HMD
size/design
of respondents said
37.6%
Lowconsumer
awareness
of respondents said
36.6%
Usability of
HMDs
of respondents said
Source: XR Industry Survey Report | VR Intelligence, 2019,
www.vr-intelligence.com/xr-industry-survey-report.
MOMENT | Research
13

In 2-3 years 44%
(2022-2023)
In the next year 5%
(2021)
Never 8%In 4+ years 43%
(2024 later)
The majority of respondents believe that
“ it will take 4 or more
years for AR to move
from predominantly
occurring on people’s
smartphones to become
headset-based.
Source: XR Industry Survey Report | VR Intelligence, 2019,
www.vr-intelligence.com/xr-industry-survey-report.
“ Consumer XR may not have quite the same level
of profitability yet but it is showing signs of major
potential and the next 18 months should see key
developments. From the likely dropping of the next
generation of consoles into the consumer market,
to falling headset costs, to the explosion of
open-source development software, to the rash of
AR integration on major platforms, the space is
likely to evolve extremely rapidly and for the better.
MOMENT | Research
14

8i, a volumetric video production studio,
is building its repository for hologram
projects. In 2017, they taped a volumetric
video of a mom and her child and
created a hologram project that
allows people to watch, walk around
and even put that digital content in their
hands with a head-mounted device.
Although Mixed reality and volumetric
video capture are still at a very early
stage, and not yet commercially
accessible, these projects show the
possibility of utilizing XR technology in
reminiscing practice and demonstrate
the advantages of immersive
technology in building emotional
connections.
The emergence of
immersive technology
provides a new way of
reminiscing.
Image Credit: 8i studio, The Verge
https://www.theverge.com/2017/11/7/16613234/next-level-ar-vr-
memories-holograms-8i-actress-shoah-foundation
Volumetric video
MOMENT | Research
15

Similar research is also being conducted at
Microsoft. Holoportation visualizes how MR helps
people to communicate and interact with the
digital avatar of people who stay at another
place, which is called virtual 3d teleportation.
This technology allows people in different
places to be present in the same environment
and provide a new way for people to interact
with each other. As demonstrated in the video,
the researcher in MR office can interact with her
daughter who is in another place. The rendering
can be a bit glitch due to bandwidth issues, but
it is quite realistic as you can see the girl walking
around in the space.
Later on, both the researcher and his daughter
are captured in a video and the researcher can
re-enter this scene to watch how he and his
daughter interact with each other.
Although due to technical limitations we have
nowadays, this project is still at the exploration
phase, it shows the promising potential of MR
technology. The research also explains
technological challenges and how Microsoft is
solving those problems in order to improve the
experience in MR, such as reducing bandwidth
using innovative compression algorithms as
well as challenges in rendering, lighting, and
vibration.
“it’s almost like
walking into a living
memory that I can see
for another pair of eyes
from any perspectives
Image Credit: Holoportation, Miscrosoft Research
https://www.microsoft.com/en-us/
research/vproject/holoportation-v1-images/
Miscrosoft
Holoportation
Case Study
MOMENT | Research
16

Tech giants are embracing the memory economy by creating
products and features that leverage existing social media
content and allows users to easily revisit past experiences. As
primary social media platforms, the huge amount of existing
data puts those companies in advantageous positions.
Memory
Product
Facebook Memories
Instagram Stories
Snapchat Memories
Product Analysis
MOMENT | Research
17

Since 2016, organic content sharing has started to decline on
Facebook: people no longer share content on Facebook.
Facebook has been struggling to reverse a 21% decline in
original sharing or personal updates form its 1.6n billion
monthly active users. To improve this situation, Mark
Zuckerberg develops the “time well-spent” strategy to focus
on the quality engagement on Facebook even if the time on
site suffers as a result. Facebook decides to change the
focus to social interaction and meaningful community
building. Their algorithm will prioritize posts that promote
natural interaction between people over public content such
as businesses or publishers.
Help users automatically generate
lookback video and say thanks videos,
and use existing content to encourage
interaction between users.
Lookback and say thanks video
Decline in organic
content sharing &
Time well-spent
strategy
Facebook Memory
“ Time
well-spent
strategy
A standalone feature is created under
more tab. An expanded version of “on
this day” and friendship anniversary
also published to help users engage
with their Facebook friends and
encourage community interaction.
Standalone memory feature
2018
2016
2014
MOMENT | Research
18

“ Let people
generate new
things to share
even if they’re
laying in bed
or stuck
somewhere.
MOMENT | Research
19

Instagram Story
“On This Day” option shows a random feed
post you shared on the same calendar
date in the past. Tap the dice button to
view a different On This Day post, and
once you find one you prefer, you can
share it to Stories as an embedded post
people can open.
People can choose stickers such as tag
memory, hashtag, location, or emojis to
easily convert their old impermanent
content into fresh content. That could be
especially helpful because not everyone
does something Stories-worthy every day.
And given how many #TBT throwbacks get
shared already, there’s a demand for
sharing nostalgia with new commentary.
Instagram stories are only visible for 24
hours, after that, the content will be saved
in the archive category under the
hamburger button. The idea of temporary
posts reduces the self-representational
concern and encourages the sharing of
mundane experience. it emphasizes on
the idea of real-timeish sharing, and
temporality, and may contribute to
increased emotional rewards. The
live-story feature released later can be
seen as an expanded version.
Users can choose to make it a permanent
content by adding to the highlight, a
collection of Instagram stories. These
collections allow the user and other
people to enjoy the glimpses into friends’
lives. It also enables users to curate the
content based on their preference,
whether it is about time, or travel, or
mood.
Not everyone does
something
Stories-worthy
every day
Playfulness of
temporality
MOMENT | Research
20

Launch My Story Feature
Add Geofilter
for quick image
location tags
Launch Snap Map 2017
Memories Feature &
My Eyes only2016
2014
2013
Ephemeral social
interaction
Post saved content
Content recognition
Private Content
Snap Map
Snap
Memories
Snapchat Memories—introduced in
2016—shifted Snapchat away from
being the platform of
self-destructing photos and videos,
to be a direct competition for
Facebook. Memories are now a key
component of Snapchat.
MOMENT | Research
21

All saved photos and videos can be
searched by keywords. Snapchat will
identify the content based on captions,
emoji, dates, locations and even object
to categorize memories. Snapchat will
show users keyword suggestions, such
as this date in the past, taken nearby.
Users can use a distance radius setting
to change the distance to home or a
whole city.
Memory is now
searchable
Snapchat wants users to feel
comfortable passing their phone to
other people when sharing their
memories. Snaps and Stories can be
set to My Eyes Only —no awkward
peeking, Only share things you want.
Share with comfort
Snap Map
Snap Map lets users share their current
location by creating their virtual
avatars on Snap map. It lets users
discover stories in a new way, and
even dives into a place to see what
people are doing on the other side of
the earth. They also can see “heat”
colors on the Snap Map to see where
lots of Snaps are being uploaded and
where are people up to. Their posts will
only appear 24 hours for non-friends.
MOMENT | Research
22

Ideation
Memory aggregator
Emotion theory
User flow
23

Technology is recording and
mediating our memories,
and we are contributing to
our digital identity.
24

Our digital footprint
becomes the new
memory material
We constantly post information on social media which generates new memory
material and supports our reminiscing behaviour. Our life is mediated by
technology with a huge amount of metadata generated alongside. What if we
can create a memory aggregator that collects, analyzes, and categorizes our
memory pieces based on location, date of created, tones, and object?
MOMENT | Ideation
25

Emotion Theory
What if we can create a
filter that lets people
selectively revisit their past
experiences and curate
their memory content?
Emotion is another important aspect of memory,
and it can trigger events that are emotionally
congruent and incongruent to our current feeling.
Plutchik’s emotion theory describes that there are
eight major types of emotions (Joy, Acceptance
Surprise, Anticipation, Fear, Ease and Anger), and
the combination these emotional ingredients
generate new emotion concepts, such as love,
contempt, guilt. Therefore, in my project, I use
emotion as the attributes to categorize different
emotions and created a filter that allows people to
selectively revisit their experience and even curate
memory content.
Plutchik's Wheel of Emotions
MOMENT | Ideation
26

User Flow
This is the user flow llustrating the whole user journey:
1. Get users' consent to connect and analyze their social media accounts.
2. Enter the AR portal and reveal the memory.
“Much of the world
is mundane, but
the memories of
what happened
there are not..
MOMENT | Ideation
27

Storyboard
Draft
Two storyboard drafts I created to help
articulate the use case of MOMENT.
Story 1:
Use Moment at a friend reunion.
Story 2:
Feel lonely and take random work, being
notified to revisit a memory happened
at this place in the pat.
Story 1
Story 2
MOMENT | Ideation
28

Design &
Prototyping
Technical investigation
Design process
Art direction
Technical challenges
User testings
29

Process Overview
MOMENT | Design & Prototyping
30

Choosing the right
AR prototyping
tools is critical for
rapid iterations
Being able to identify when to use what tools for rapid
prototyping is critical as it can help to save numerous
amount of time. I spend some time testing a few
markerless AR software, such as Spark AR, Reality
Composer, Metaverse Studio, and compare the pros
and cons to find the one that can effectively visualize
and test the interaction design. Once I made the
decision about the placement and the size of those
virtual objects, I start mid-fidelity prototyping in Unity.
Technical Investigation
31

1
3 4
2
1 Use Placenote SDK to scan indoor
environment and generate point
clouds to place objects.
Placenote SDK
2 Test unity AR+GPS plugin using
placeholder object
Unity AR+GPS
3 Very stable rendering in AR using
built-in assets
Reality Composer
4 Create a world-effect filter using
the front camera
Spark AR
Tools testing
https://vimeo.com/
393174918
https://vimeo.com/434893905https://vimeo.com/363191454
https://vimeo.com/363191356
32

5 I ended up using metaverse studio for
rapid prototyping and created the first
version interactive mockups for
feedback by the end of the Fall term. In
the app, I created each screen
separately and connected them based
on the interaction flow. The file size
reduction takes some iterations and
loading speed needs to be improved.
Metaverse Studio
5https://vimeo.com/363191454
33

Design
Process
Body Storming is an efficient way of designing
interaction mechanisms in immersive experience,
and it saves a lot of time needed in unity
prototyping. I printed a few photos to
simulate the physical interaction from the user's
perspective and quickly generate ideas.
Body Storming
34

Design Process
Quick sketch helps me
brainstorm the interface design
before work on digital artboards
35

Wireframes V1
These are the first version wireframes I created for
in-class pitch and feedback collection. It illustrates
the rough flow from notification to emotion filter. The
heat map shows on the left is a memory footprint
mockup where I use different colours to represent
various types of emotions.
Mockup shows on the bottom left are the rough
design for views in augmented reality: a colourful
floating bubble can be found on top of the real
world layer and once users click on it, it will be
transformed to a couple of pictures that are
captured at this place in the past. An input field is at
the bottom of the screen where users could type in a
message or their reflection when they revisit the
memory. A folded envelop records this message and
reminds users that they have a message to look at.
36

Art
direction
Memory is such a poetic and beautiful object. To
find the style and visualize the feeling memories
bring to us, I collect images of textures, text,
lines and colours to help me decide the overall
aesthetics style and visual elements.
37

I rendered these orb animations in After Effects to
represent different types of memories. Different colours
represent different types of emotions.
Memory Bubble
Fluid
Abstract
Emotional
Distorted
Poetic
Organic
38

User interface
design V1
The design showing on the left is the first
version user interface design created in
Adobe XD, including memory footprint,
account setting, homepage, emotion filter
and delete the page. I used this prototype to
gather feedback at the end of the Fall term.
The original thought was to make the
interface as an embedded part of the
memory piece, therefore the tone of the
background is changing with the content:
when revealing a sad memory, it changes to
blue shade, when revisiting a happy
moment, it changes to red/pink shade.
Clickthrough Video
https://youtu.be/vA2bbPfDqVY
40

Technical
Challenges
Develop a fully functional AR app is quite challenging for
me. It involves multiple aspects, from VFX to animation, from
interaction logic to input and content management, no
need to say weird bugs come from nowhere.
Thanks to online communities and forums, I solved most of
the problems and completed the project as planned.
41

Visual effect and
animation in Unity
My goal is to create a stunning visual effect and a compelling
AR experience. The visual effect component in Unity is quite
limited and cannot achieve my expectation, so I rendered the
orb animations in After Effects and import the animation
sequence to Unity as animation sprite to improve the visual
effect quality.
Save posted content and
render it on 2d UI sprite
The funcation where people write a message to themselves
is a bit complex. By creating a public variable, the written
message can be saved as a string and rendered as text on a
note image.
42

Outdoor location-based
AR experience
Creating an outdoor location-based AR experience is
challenging because it is not a popular functionality
offered by AR packages such as vuforia AR or AR
foundation, and additional scripting is needed. I
purchased the AR+GPS plugin to create the prototype
and the GPS is pretty decent and accurate.
At the time of building the app using Xcode, I encounter issues where apps
on iOS (13.3.1) device crash on startup, even though no build errors found,
the only solution is to downgrade my ios version to 13.3.
Xcode app crashed on iOS 13.3.1
I purchased the Playmaker plugin because I was not very confident about
my scripting skill, but later on, I found that playmaker does not support
some customized effects and requires extra time to achieve the desired
outcome. I chose to do scripting by searching tutorials on google and
adapt existing code available online, which turns out to be more efficient
since my app does not require complex inventory or score system.
Playmaker cannot
achieve desired outcome
I used High Definition Render Pipeline in Unity to create the visual effect
(particle orb) and found that HDRP is not supported on mobile devices but only
on high-end hardware. Since the object has already been created, it cannot be
transformed into another render pipeline. I had to rebuild it from scratch.
HDRP not supported on mobile devices
Image Credit: AR +GPS Location Unity Plugin, Unity Assets store
https://assetstore.unity.com/packages/tools/integration/ar-gps-location-134882
43

User
testings
I did several in-class user testings
sessions to gather feedback and observe
peple’s reaction to my app. I also asked
about their feelings of using my app,
including if the copywriting is straightford,
and likeability of the visual style. Later on, I
analyze the testing result and refined the
design of the mobile app and AR
experience.
44

Problem Found
Image Credit: Comfortable range of motion zone, Mike Alger
https://www.youtube.com/watch?v=id86HeV-Vb8&t=931s
Image Credit: AR Design guidelines, Apple AR
https://designguidelines.withgoogle.com/ar-design/user/safety-comfort.html
Size and placement
of content in AR
When the imagery is too big, users need to step back to view the full size which can
cause dangers in an outdoor environment. When the imagery is too small, it is hard
to find it as the content can be blended in the background.
Where the content is placed in AR is also critical. When placing the object behind
users, it can cause frustration if the content cannot be found easily. If the content is
placed too high or low in the air and not within their reach, users won’t be able to
keep holding their AR device and their attention can be distracted.
Reading notes in AR is
not accessible
Reading a long paragraph in AR is not accessible
due to the lighting situation and users can only read
content that appears on the screen. They need to
move a lot to read the whole paragraph.
45

The object is anchored in the world space to
enhance the immersion and create an
adventure-like experience, but when it comes to
message reading, it is not accessible enough
especially when users are reading a long
paragraph. Therefore, I decided to change the
design to a screen-based user interface to
improve the readability: it appears as regular
text on the screen regardless of the device
position or rotation.
World Anchor vs
Screen-based UI
46

UX writing is not
straightforward enough
Some users find the UX writing is not clear
enough and hard to understand what to do.
I made several iterations to improve the visual
style and UX writing. The final version clearly
communicates the instruction and keeps the
style consistent throughout the whole journey.
Adding a vignette overlay underneath
improves the readability without distracting
attention from the memory bubble.
Although immersion is important, adding a
glowing effect makes the virtual orb more
noticeable in the background and easier to be
found. The floating animation also encourages
users to interact with it.
47

The concept of revisiting memory can be quite
abstract. A lot of people don’t know what to
expect in the app. In the user testing sessions,
people constantly asked me to explain what
they are looking at and what to do, which
means the interaction flow needs to be
improved. Although this can create a sense of
surprise and satisfaction when the AR object
suddenly appears, it can make users feel
confused. Therefore, I added an onboarding flow
before users enter the AR world, which helps to
set up users’ expectations and articulate the
concept in a clear way.
Set clear user
expectation
The concept of memory
revisiting can be hard to explain,
an onboarding flow is needed to
set users’ expectation.
Prototyping Iterations
48

Refined wireframes
Onboarding Flow
AR Flow
Message Flow
49

AR Onboarding
Experience
People tend to distance themselves from past
events and look back at them from a
third-person perspective so that they can
make sense of it, and embrace mistakes they
have made.
Find the AR Portal
The floating memory bubble flies out of the
screen and an arrow appears to ask users to
follow the direction. This design helps users to
find the object in AR and encourages them to
explore the environment.
Onboarding 1 Onboarding 2
MOMENT | Final Project
51

Onboarding 3: Animation sequence
52

The memory bubble will fade in as a glowing
orb with particle effects, then the core will
show up. The bubble is flying up and down so
that it is not blended in the background and
can be found easily. After users find the
bubble itself, the animation can trigger a
sense of satisfaction and encourages users to
tap on it.
The idea of interacting with the memory
bubble transforms people’s perspective to an
adventure-like experience: although people
are revisiting their memory, the interaction
mechanism makes it like a new adventure, a
new moment.
Revisit the Moment
Bubble fades in Bubble appears
53

Tap to animate the photo More actions
Revivify the memory
Once the user taps on the memory bubble, it
will transform into a photo or a video clip
captured and shared by the users in the past.
Tapping on it will animate the video
sequence from a static image and bring the
memory alive.
This interaction design helps to create a
sense of magic and immediate satisfaction, it
can also be seen as a metaphor: old memory
is alive and all the feelings come back. Taping
on the corner of the screen will hide the
action buttons so that the AR view will not be
blocked. Users can tap on the photo to bring
the buttons back and choose other actions.
54

Bring the memory back: animation sequence
55

Write and send a note Type in the note
By recalling life events, people are
developing a strong self-concept that
allows them to understand and clarify their
self-identity.
Writing a short note about feeling when
revisit the memory encourages people to
articulate the abstract thought and
generate positive self-concept. This note is
saved alongside the memory piece and an
envelop will show up next to the photo
content for future revisit.
Write a short note to
your future self
56

Note saved alongside the memory
Envelop animation
Envelop opening animation is a metaphor
unfolding a story that happened in the past.
It also smooths the transition to a screen-based
coordinate and bring the user to the message
reading scene.
57

Open the note: animation sequence
58

Memory examples
59

Memory examples
61

Homepage design
62

Notification Screens: Various memories
63

Emotion Filter 2 SettingEmotion Filter 1
64

When 3D capture
becomes as easy
as taking a picture,
what will our photo
albums look like?
The invention of photography disrupted
traditional painting and changed the way
we capture daily life objects. Nowadays,
Taking photos is like second nature when
we want to capture a moment for future
reminiscing.
With the emergence of 3D capture and
volumetric videos, we now have a new
way of capturing objects and physical
environments. Although volumetric video
capture still requires advanced
equipment in professional digital
production studios, 3D scan has become
more accessible in the past few years:
free mobile apps such as Scandy Pro and
Bellus3D let users quickly scan their head
and generate a full colour mesh which
can be exported to 3d software for editing
and sharing purpose. The result is quite
accurate and efficient. When 3D capture
becomes as easy as taking a picture, our
photo album will be full of objects
captured in 3D space, and we can replay
a moment in a more immersive way.
ScandyPro
Bellus3D
Display.land
Fyuse
3D mesh of my head using Bellus3D app
MOMENT | Future Potential
66

3D human
digitization
Image Credit: High-Resolution 3D Human Digitization
https://arxiv.org/pdf/2004.00452.pdf
Even objects captured in 2D in the first place can
also be converted to 3D format by utilizing
advanced algorithms in machine learning and
computer vision. Research conducted in Facebook
Reality Lab has successfully achieved
high-resolution 3d reconstruction of humans using
simple input such as a photo. Although the
performance is lower than professional 3D capture
techniques, it is still significant in enabling 3d
human digitization. This research demonstrates the
possibility of converting 2d content to spatial
objects and can unlock numerous possibilities in
storytelling, experience rebuilding, and environment
augmentation.
67

Physical
environment scan
and rebuild
The introduction of iPad Pro 2020 with a LIDAR camera
makes physical environment scanning accessible and
brings new possibilities to Augmented Reality. Its
depth-sensing capabilities enable more accurate object
and environment recognition.
In WWDC 2020, Apple demonstrates point cloud
visualization using sense depth technique and allows users
to easily digitize physical surroundings and place points in
the real-world environment. The advancement in
environment scanning and recognition can significantly
improve accuracy in raycasting and make the
world-anchored object effect more stable.
Image Credit: Explore ARKit 4, Apple
https://developer.apple.com/videos/play/wwdc2020/10611/
68

Contextualized presence
“A place is more
than just a space.
A place is redolent
with meaning...
Image Credit: Explore ARKit 4, Apple
https://developer.apple.com/videos/play/wwdc2020/10611/
WWDC2020 also introduced the built-in
location anchor feature in ARKit 4 that
allows users to easily create AR
experience in relation to the globe and
place objects as an annotation of their
surroundings. Combining high-resolution
map data and user’s geo data, the
complexity of creating a location-aware
project is remarkably reduced.
Contextualized presence is the feeling of
being in a place rather than a location.
Space becomes a place because it is
meaningful to us. If we say human
invented clocks to measure time, we
could also say human created maps to
make space meaningful. A place is
unique and memorable because each of
us can have a unique experience and
feeling about a space. Utilizing
augmented reality, my project lets
people experience a place in a way that
is unique to them, and create a
personal storytelling experience that is
related to their life. Augmented Reality
technology allows us to “customize” the
physical environment by placing
personalized mixed-media content on
top of the real world layer.
69

Re-enter our
memories in
3D space
This mockup demonstrates the look of the ultimate version I am visioning when
immersive technology and volumetric capture technology becomes fully accessible.
Imagine that we can re-enter those moments and reunion with people in the memory.
Although there will be ethical issues that needed to be discussed, it can bring lights to
industries such as healthcare, education, and marketing.
State-of-the-art Future version
70

Revivify our feelings
in a multi-model
experience
My thesis project shows the potential of applying XR in
experience rebuilding, and the notion of time travel. When
multi-sensory capture becomes accessible, creating a hybrid
experience where people could re-enter their past memory
with an MR headset will not be a dream anymore. the project
could be expanded to a multimodal experience, including
visual, audio, olfactory, and haptic. The mixed media format
will add a new dimension to reminiscence and memory
revisiting. Although this whole project might look a bit
conceptual and experimental at this point, I believe that
something similar will be invented in the near future.
71

Concept teaser video: https://vimeo.com/417008435
72

Cosley, Dan, et al. "Experiences with designing tools for everyday reminiscing." Human–Computer Interaction 27.1-2 (2012): 175-198. Accessed 23 September 2019
This article provides a holistic view of designing experiences for memory products and introduces some key design principles from a human-computer interaction perspective. The author
mentions that there is a tendency to designing technologies for experiences and human-centered content. To achieve that, research has been conducted in two areas, one is to under-
stand people’s reminiscing behavior, another one is to explore how technology could support it.
Cutler, Ben, et al. “Holoportation.” Microsoft Research, 2019, www.microsoft.com/enus/research/project/holoportation-3/ . Accessed 3 October 2019.
This is a Mixed reality prototype created by Microsoft. In the video, it visualizes how MR helps people to communicate and interact with the digital avatar of people who stay at another
place, which is called virtual 3d teleportation. This technology allows people in different place to present in the same environment and purposes a new way for people to interact with
each other. The research also explains technological challenges and how Microsoft is solving those problems in order to improve the experience in MR, as well as challenges in rendering,
lighting and vibration. These are the technical barriers XR tech is facing before achieve mainstream adoption point.
Exposito, Marc, and Oscar Rosello. “Project Overview ' NeverMind: Using AR for Memorization.” MIT Media Lab, 2016, www.media.mit.edu/projects/nevermind/overview/.
Nevermind is an AR project aiming to explore how to help people memorize more effectively. It combines location-based AR technology and memory palace, a memorize method first
developed in ancient Greece and suggests that this type of memory retrieval makes the accuracy even tripled to conventional paper-based memorizing techniques.
Goode, Lauren. “Are Holograms the Future of How We Capture Memories?” The Verge, The Verge, 7 Nov. 2017, www.theverge.com/2017/11/7/16613234/next-level-ar-vr-memo-
ries-holograms-8i-actress-shoah-foundation.
8i is a volumetric video production studio building its repository for hologram projects. Their project makes more impact on the way people experience memory and historical events. It
also poses a question on the way people interacting with each other: the ethics question around the content itself, trauma, negative experiences… There are also security issues that need
to be discussed: the governing of these digital content, and the misusage for inappropriate activities.
Hoven, Elise, and Berry Eggen. "Informing augmented memory system design through autobiographical memory theory." Personal and Ubiquitous Computing 12.6 (2008): 433-443.
Accessed 23 September 2019
Bibliography
MOMENT | References
74

This article provides an in-depth explanation of autobiographical memory theory and gives an overview of design augmented memory systems. The author first argues the
six functions of memory and why it matters, which provides the context of my work: why we need an application to help people obtain their memory. The three major functions of memory
are the construction of self-concept, maintain social relationships, and predict the future.
Isaacs, Ellen etal. "Technology-mediated memory: Is technology altering our memories and interfering with well-being?." ACM Transactions on Computer-Human Interaction (TOCHI)
23.4 (2016): 23. Accessed 25 September 2019
The authors argue that nowadays technology provides a rich record of our memory and makes memory retrieval easier and more accessible. This article also compares the differences
between organic human memory and technology-mediated memory and evaluates whether the latter can have a negative impact on well-being. Authors believe that when people
revisit and reflect on past experiences, they tend to embrace a more positive mindset and help to maintain well-being in the long run.
Konrad, Artie, et al. "Technology and reflection: Mood and memory mechanisms for well-being." Psychology of Well-being 6.1 (2016): 5. Accessed 1 October 2019
In this article, the author argues that mood is an important aspect of memory, and it can trig events that are emotionally congruent and incongruent to our current feeling. The authors
identify and summarize some common patterns in memory retrieval and self-reflection: understanding, redemption, and distancing.
Konrad, Artie. “Facebook Memories: the Research behind the Products That Connect You with Your Past.” Facebook Research, 21 Sept. 2017, research.fb.com/blog/2017/09/face-
book-memories-the-research-behind-the-products-that-connect-you-with-your-past/.
This article shows the product design rationale behind Facebook Memories from a content strategy to a standalone product. Facebook memory feature shows the foundation of utilizing
social media content to generate new content, and allow people revisit past experiences,
Donalek, Julie G. “Phenomenology as a Qualitative Research Method.” Urologic Nursing, vol. 24, no. 6, Dec. 2005, doi:10.5539/ass.v10n21p298.
Donalek provides an overview of phenomenology and introduces its origins and development process. It describes the basic principle of phenomenology and its impact in understanding
people’s behavior as a qualitative research method.
Holm, Tine, and Dorthe Kirkegaard Thomsen. “Self-Event Connections in Life Stories, Self-Concept Clarity, and Dissociation: Examining Their Relations with Symptoms of Psychopathol-
ogy.” Imagination, Cognition and Personality, vol. 37, no. 3, 2017, pp. 293–317., doi:10.1177/0276236617733839.
This article argues that the ability to connect life events and formulate stories is important for mental health, as these narratives help to connect the past self and the current self, and
therefore clarify the self-identity and maintain self-unity. It also introduces theories and practices such as self-report.
MOMENT | References
75

Neubauer, Brian E., Catherine T. Witkop, and Lara Varpio. "How phenomenology can help us learn from the experiences of others." Perspectives on Medical Education 8.2 (2019): 90-97.
Accessed 2 October 2019.
This article provides in-depth descriptions on phenomenology and research methods such as user interviews, focus group. It argues that phenomenology compared to
other research methodologies such as participatory, focuses on lived experience and can help researchers understand complex phenomena by learning from individual behaviors.
Papagiannis, Helen. Augmented human: How technology is shaping the new reality. " O'Reilly Media, Inc.", 2017.
In this book, Papagiannis envisions how AR technology could change people’s life in all aspects, from transportation to film production. It analyze the unique aspect of AR technology, such
as interaction model, haptic, relation with 3D space, and argues that XR could be seen as a mixed media format which can provide new opportunities and challenges for storytelling.
Van Den Hoven, Elise, Corina Sas, and Steve Whittaker. "Introduction to this special issue on designing for personal memories: past, present, and future." Human–Computer Interaction
27.1-2 (2012): 1-12. Accessed 23 September 2019
This article argues that digital technology give people access to numerous memory material and people can effortlessly access those content. It also allows people to organize and
curate the memory content based on their preference. The author also introduces the taxonomy of memory and provides a list of the most desirable memory content.
VR Intelligence. “ XR Industry Survey Report.” XR Industry Survey Report | VR Intelligence, 2019, www.vr-intelligence.com/xr-industry-survey-report. Accessed 14 September 2019.
This report provides an overview of the development status of XR technology in 2019, and combined both qualitative and quantitative approach to discover the current roadblock and
articulate the problem XR technology is facing in order to the point of mainstream adoption.
West, David, Aaron Quigley, and Judy Kay. "MEMENTO: a digital-physical scrapbook for memory sharing." Personal and Ubiquitous Computing 11.4 (2007): 313-328. Accessed 30 Septem-
ber 2019
This article evaluates a memory system called MEMENTO, a multimodal digital scrapbook that allows people to collaboratively collect and share memory materials, and examine its effec-
tiveness in reminiscence for both young and senior people. This article summarizes some common practices in reminiscence, and indicates that digital technology has positive impact on
reminiscence.
MOMENT | References
76

Credits
MOMENT | References
Image
Holoportation by Microsoft Research, https://www.microsoft.com/en-us/research/project/holoportation-3/
8i studio, The Verge, https://www.theverge.com/2017/11/7/16613234/next-level-ar-vr-memories-holograms-8i-actress-shoah-foundation
Comfortable range of motion zone, Mike Alger, https://www.youtube.com/watch?v=id86HeV-Vb8&t=931s
AR +GPS Location Unity Plugin, Unity Assets store, https://assetstore.unity.com/packages/tools/integration/ar-gps-location-134882
AR Design guidelines, Apple AR, https://designguidelines.withgoogle.com/ar-design/user/safety-comfort.html
NeverMind, Oscar Rosello, MIT Media Lab, https://www.media.mit.edu/projects/nevermind/overview/
High-Resolution 3D Human Digitization, https://arxiv.org/pdf/2004.00452.pdf
Explore ARKit 4, Apple, https://developer.apple.com/videos/play/wwdc2020/10611/
Footage
2046 by Wong Kar-wai, Portrait of a Lady on Fire by Céline Sciamma, Vertigo by Alfred Hitchcock, Room in Rome by Julio Medem, Paprika by Tinto Brass, Blade Runner 2049 by
Denis Villeneuve
Free videos provided by creators on Pexels
Taryn Elliott, Life On Super 8, Athena, Pixabay, cottonbro, Stef, Pressmaster, Mapo Dessin, Desmond Etsebeth, Carlos Arribas, Fauxels, Free Videos, Distill, Pixly Videos, Charlie
Mounsey, Kelly Lacy, Joseph Redfield, Ricardo Esquivel
Music
Blasphemy, Otik
Beautiful, AShamaluev
77

Afterword
Although thesis was supposed to be
completed in April 2020, I have been
working on it until July, Now I think it is time
to end this project and document
everything I designed and created in the
past 11 months.
"Put users first " is my mentality as a
designer, while for my thesis, I decided to
take this opportunity to create something
meaningful for myself. As a person who
likes reminiscing a lot, I think there is
nothing more fascinating than memory: It
can be both blurry and vivid. It is distorted
but also accurate. it cheers me up when I
feel in blue and inspires me to keep
creating. Meanwhile, as a big fan of
Augmented Reality, the power of AR can
always give me magical and mind-
blowing experience, and even add an
additional layer to everyday objects. The
idea of combining these two things
makes MOMENT come to real life.
Throughout the whole journey, I
encountered a lot of challenges. From a
conceptual level, memory is such a
common topic and everyone I talked to
has their own understanding. As the
designer of this project, I need to hold my
opinion strongly and figure out the right
path. From a technical standpoint,
creating outdoor AR experience in Unity
can be frustrating and takes a lot effort. In
the meantime, I also need to strike a
balance between thesis, work and job
hunting. I am happy about what I have
achieved and can’t wait to share it with
more people.
I also wanted to say thank you to people
who have helped me, inspired me and
encouraged me while I was working on
this project:
Vivian Wong, Alexander Manu,
Job Rutgers, Emma D’Souza,
Setayesh Babaei, Enna Kim,
Vicky Zheng, Fandongge Wu
Anran Zhou
July 15, 2020
78

Thesis Documentation

Recommended

Recommended

More Related Content

Similar to Thesis Documentation

Similar to Thesis Documentation (20)

Recently uploaded

Recently uploaded (20)

Thesis Documentation