Augmented Reality for eCommerce is everywhere. Major retailers and Shopify have mainstreamed 3D. But so far, nearly all product shoppers do is simply “see this in their room.” For complex, configurable, personalized purchases, this isn’t enough. This session focuses on effective AR uses that increase user success with planning and decision-making. Think of projects such as a kitchen redesign — design aesthetics, myriad features/options, physical characteristics, and lack of buyer knowledge all stand in the way. I’ll discuss wide-ranging aspects of AR’s potential and provide a framework for planning product-focused applications. I’ll share lots of examples and insights from recent projects, plus others I’ve found along the way, including UX principles for image-based visualizers and configurators refined over 2 decades. This knowledge with help spur ideas for your own projects. Going beyond, I’ll align user expectations with present and future capabilities of 3D platforms/engines/hardware, giving you a working knowledge for the next generation of 3D: Mixed- and eXtended-Reality.

1. OptimalARUX
forComplex
Purchases
How immersive augmented reality boosts shopper
success for personalized, configured products
Note: This session as presented contained
many video demonstrations, which may
appear in the associated session recording

2. Hello!
2

3. Mike Osswald
VP, Experience Innovation
3
@mikeosswald
@HansonInc hansoninc.com
Session Hashtag #UXAR19

4. Augmented Reality can help people
make confident purchase decisions
for complex products
When the experience is designed
in conjunction with what people really expect,
AR has applicability across the full dreaming,
planning, selecting, and buying journey
4

5. Explore and discuss ways Augmented Reality can be used to
create more engaging, immersive experiences specifically for
complex commerce
1. Quick AR groundwork
2. How people think about complex purchases
3. Foundational UX — and how AR fits in the journey
4. The UX of AR for complex purchases
5. eXtending Reality: what’s coming soon and next
6. Answer questions
Here’s the Flow
5

6. Augmented Reality
Groundwork
How we got to now
1

7. The “virtuality” continuum
1. Groundwork 7
Augmented Reality: when the products aren't real
and you’re holding your device with one or both hands Virtual Reality: when the whole world isn't real
Smartglasses/Headset:
you’re wearing the
transparent screen in
front of your eyes
Today’s focus (specifically apps)

8. Early AR
1. Groundwork 8
The earliest AR
from a decade
ago used “image
recognition” to
decide where and
how big to show
a 3D overlay
The system could
match a known
image and display
content over the top —Lego in 2011 (FDS nucklehead, YouTube)
—Layar in 2010

9. Today: ARs most basic capabilities
1. Groundwork 9
World alignment
Happens at startup and continuously
by tracking interesting points,
plus math of where the camera moves
Managing a virtual, in-scale map
of the space around you

10. ARs most basic capabilities
1. Groundwork 10
Finding horizontal and vertical surfaces
Placing objects: they appear in scale,
and stay where they are placed

11. About faces
1. Groundwork 11
Significant work has been done with face
detection long before AR
– Example use: autofocus photos
It’s CUTE! It’s CREEPY
It’s the key reason AR is so widespread
and great for try-it-on products:
glasses and makeup
But the thing to know here: this is
built-in recognition and tracking of a
specific dimensional object in the real world

12. AR for shopping today
1. Groundwork 12
—Sephora
—Wayfair

13. • AR is deceptively simple, yet very
complicated (capable)
• AR is continuously evolving — there are
gaps in what can be done easily (or custom
workarounds); but today’s R&D should
quickly become standardized
• Major players are focused on enabling cross-
platform applications (OpenXR standard)
• Some capabilities (and experiences) will
require newer and faster hardware
• 3D content is a requirement, but the
ecosystem is working on that too
Quick Tech Notes
13
OpenXR Working Group
1. Groundwork

14. What people experience with
complex products and purchases
What we’re trying to solve
2

15. Somethingsarehardtopurchase
2. Complex products & services

16. Checklist for a hard purchase:
Aestheticsareinvolved
Many stylistic/design choices
I might not know what I like
I want something that looks a particular way
I want something that goes well with other things
(that I also might be buying)
2. Complex products & services

17. Checklist for a hard purchase:
Manyphysicalconfigurationoptions
What size is this?
Will it fit?
What size is it when set up this or that way?
2. Complex products & services

18. Checklist for a hard purchase:
Thecostis“significant”
Is it worth the money?
What about delivery? Installation? And disposal?
What if I want to return it? Can I return it?
2. Complex products & services

19. Checklist for a hard purchase:
Haven’tboughtthisrecently(orever)
Where do I begin?
What am I getting into?
Will I like this in 2 years?
2. Complex products & services

20. high-involvementproducts
big-ticketitems
complexprojects
2. Complex products & services

21. Harddecisionscanbe
physicallyexhausting—
whichwecall
DecisionFatigue
2. Complex products & services

22. PeoplelackConfidence
inmakinga“good”decision
theywon’tregret
(oh,manydecisionsaremadewithothers)
2. Complex products & services

23. PeoplelackTrustinthesalesperson
(becausetheydon’tfeelincontrol)
(andbecausesalespeople
aren’tknownasimpartial)
2. Complex products & services

24. Aconfusedminddoesn’tbuy
TheymightjustAvoidordeferthedecision,
andspendonsomethingelse
2. Complex products & services

25. HomeImprovementproducts
areespeciallytroublesome

26. HomeImprovementproducts
areespeciallytroublesome
–youlivewiththem,seethem,usethem
–somepurchasesseemsmallorsimple
–youmightbecrazyaboutdesign,orhavenoideawhatlooksgood
–butyoulikelydon’tknowhowtoshopformost
–youmightevenstartoutoverconfident
2. Complex products & services

27. Furniture,largeandsmallappliances,
doorsandwindows,cabinets,wallpaper,
woodflooring,carpeting,rugs,tile,faucets,
sinks,toilets,showers/tubs,countertops,
storage,homedecorandaccessories…
2. Complex products & services
The complex products include…

28. First time buyers (with low experience)
own homes over 20 years old
Home improvers have many projects
“in the works” – 60% doing some
form of continuous planning
Research with building products tells the story
(2 years prior or in the future)
Painting
Landscaping
Kitchen Remodel
Bath Remodel
Flooring
Major Appliances
Replacing Doors/Windows
Closets/Storage
Living Room Redecorating
Bedroom Redecorating
Outdoor Areas
Replace Siding
Adding Insulation
Reroofing
0% 20% 40% 60%
Number of projects
completed or planned
>2000
1980-2000
1960-1980
1940-1960
<1940
Year home built for
first time buyers
(across all age groups)
6+
(37%)
2-5
(49%)
1
(14%)
Number of projects
completed or planned
(2 years prior or
in the future)
282. Complex products & services Hanson Research, 2018

29. Othercomplexpurchases

30. automotive
add-onsand
accessories
boats
ATVs
RVs
clothing
(outfits)and
accessories
electronics
tools
2. Complex products & services

31. The UX of complex purchase journeys,
visual feedback and composition
AR will be relevant when it supports the complete journey
3

32. Buying
(and owning)
People, products, experiences—making decisions to purchase
32
decide to pursue a project
gather information on what to buy
identify options and make a list
evaluate the short list (prioritize)
pick the winner
purchase
Unlimited/unknown
items that
might be OK
5-10
1
Stage of decision-making
Potential products
considered
People want to
compare
(ranking)
People want to
categorize
(decision set)
People want to
decide
Planning
specifics
detailss
Dreaming with
similar homes
(or body styles)
Dreaming with
my home
(or my body)
Selecting
and last steps
What they’re
doing
How they’re
doing it
3. UX of complex purchase journeys

33. 33
🙂 Inspiration often
leads to indecision 😫
• People are inspired with seemingly infinite
design images and videos they find online
• But they people lack the ability to really see
how things would look in their home
• Not everyone has a design professional or
project manager on their side
“I want to enjoy my
home more” —
78%
63%use social
platforms and video
but 66%struggle to picture how
a product would look in their home
3. UX of complex purchase journeys Hanson Research, 2018

34. For complex products, Amazon and the
majority of retail sites are designed for failure
And when people get serious, the process gets hard
34
Comparisons are just
side-by-side specs
on 3 or 4 products!)
Product data is
lacking,
filtering is
spec-based
Too many
places to buy
Too many
options
People want to
compare
(ranking)
People want to
categorize
(decision set)
💣
3. UX of complex purchase journeys

35. Complex products
need immersive,
engaging experiences
35
lots of customization
many stylistic/aesthetic choices
concerns about installation
high cost to purchase
low knowledge
Dreaming Planning Selecting Purchasing Owning

36. AR’s future will build on current UX solutions
36
Reducing the decision set
• Human-based filtering, not just technical specs
• Step-by-step planners, wizards, quizzes
• Meaningful comparisons
• Recommended “Outfits” and mood boards
• Using favorites or a wishlist
Visually seeing the solution, specifying the solution
• Photorealistic visualization tools
• Product configurators / calculators
Ways to translate project needs and lifestyle desires into “the perfect purchase”
3. UX of complex purchase journeys

37. Current UX: Human-based filtering
37
A step in the right direction,
could provide more
meaningful criteria, but this
data might not be universally
understood
(branded features
might be unclear)
(styles might be arbitrary)
3. UX of complex purchase journeys

38. Current UX: Step-by-step planners / Wizards / Quizzes
38
Filter the catalog by asking people
human-based questions they can
answer
Help define their style and make
recommendations
Help capture their required
anchors (like specific sizes)
3. UX of complex purchase journeys

39. 393. UX of complex purchase journeys
Current UX: Step-by-step planners / Wizards / Quizzes
Filter the catalog by asking people
human-based questions they can
answer
Help define their style and make
recommendations
Help capture their required
anchors (like specific sizes)

40. Current UX: Meaningful comparisons
40
(One of these is
important for
pet owners)
3. UX of complex purchase journeys
And a smaller set of
recommendations

41. Current UX: “Outfits” and moodboards
41
Easy ways to visually
shop through
recommendation
of coordinated items
3. UX of complex purchase journeys

42. Current UX for visualization:
Let’s look at existing experiences
for seeing products come to life
42

43. As projects progress, inspirational and emotional experiences precede practical and rational
Visualization/Configuration along the journey
433. UX of complex purchase journeys
VisualRealism(aestheticemotional)
Level of Specificity (getting a feel, putting together, on to purchase)
surfaces/textures
photorealcartoon
Following the typical purchase Journey
SKUs / BOMmaterials estimation… plus dimensioned objects

44. As projects progress, inspirational and emotional experiences precede practical and rational
Visualization/Configuration along the journey
443. UX of complex purchase journeys
VisualRealism(aestheticemotional)
Level of Specificity (getting a feel, putting together, on to purchase)
surfaces/textures
photorealcartoon
Planning
specific
details
Selecting –
for actual
plans and
budgets
Dreaming with
similar homes
Following the typical purchase Journey
SEEING
SCALING
SHOPPING
Dreaming with
my home
SKUs / BOMmaterials estimation… plus dimensioned objects

45. Visualization/Configuration along the journey
45
Different types of tools help these activities goVisualRealism(aestheticemotional)
Level of Specificity (getting a feel, putting together, on to purchase)
surfaces/textures
photorealcartoon
Planning
specific
details
Selecting –
for actual
plans and
budgets
Dreaming with
similar homes
SEEING
SCALING
SHOPPING
Dreaming with
my home
SKUs / BOMmaterials estimation… plus dimensioned objects
photographs
photo-real
visualizers
photo-real
visualizers
configurators
space planners
3. UX of complex purchase journeys

46. Visualization/Configuration along the journey
46
3D and augmented experiences can be used in many ways and support each stage’s activitesVisualRealism(aestheticemotional)
Level of Specificity (getting a feel, putting together, on to purchase)
surfaces/textures
photorealcartoon
Planning
specific
details
Selecting –
for actual
plans and
budgets
Dreaming with
similar homes
SEEING
SCALING
SHOPPING
Dreaming with
my home
SKUs / BOMmaterials estimation… plus dimensioned objects
photographs
photo-real
visualizers
photo-real
visualizers
configurators
space planners
3D/AR
3D/AR
3D/AR
3D/AR
AR Today
(mainly 1 item
at a time)

47. Current UX: Image-based Visualizers
47
• Focus is on realistic design/aesthetics;
product SKUs and amounts to purchase
may be secondary
• Work best for flat “surfaces” because what’s
in the photo is physically “fixed” in scale
Simplest approach is just
like putting a sticker to a
“straight on” photo
3. UX of complex purchase journeys Hanson project for Therma-Tru Doors

48. 48
circa 2000 today
Specific products but also suggested combinations
3. UX of complex purchase journeys
Current UX: Image-based Visualizers
Hanson projects for Owens Corning

49. and/or using seamless tiled
patterns for every surface
How visualizers work:
But first – photos need to be prepared
Masking area boundaries, and then
indicating the scale and perspective
curved
shapes are
possible,
but with
desktop
software
Recoloring parts of
product images
493. UX of complex purchase journeys

50. How visualizers work:
50
Can layer to create product variations Can layer to create whole rooms
3. UX of complex purchase journeys

51. transparency
curved surfaces
reflections
and shiny
surfaces
custom
patterns
and layouts
randomized
installations
see what’s
inside
shadows
51
Image-based visualizers have limits to realism…
3. UX of complex purchase journeys

52. 52
• When you make choices to
put products together or
specify measurements
• Might have several steps,
each informed by the
previous
• Might have a calculator –
and lead to quantity and
model number and exact
price
• Realism is secondary
3. UX of complex purchase journeys
—Smith & Noble
—Tylko
—MYCS
Current UX: Configurators

53. So,tomoveAugmentedRealitytothe
nextstage,weneedtointegrateitmore
completelyintothepurchasejourney
Dreaming Planning Selecting Purchasing Owning
Saidanotherway,beginARwhentheuserhassomeideaofwhattheywant

54. The UX of AR
Basics and Beyond
4

55. Stages
1. Initialization / Surface Detection (setup)
2. Main interface (doing stuff)
• Defining areas
• Placing/manipulating objects
Behind the scenes: the application continuously
evaluates the virtual/real world maps
Key Concepts: Structure/Flow for AR Application
4. The UX of AR 55
Be aware of interruptions!
• ARKit can’t keep track of position
when AR is not active
• “Relocalization” is a state where the
application attempts to reorient to the
world
• “Persistence” features are ways to
save the 3D world map, and try to
realign in a future session

56. Screen interface layers
“screen space” –
UI layer (panels,
text, buttons) that
stay aligned to the
phone screen
“virtual
environment” –
camera, 3D
objects, other UI
elements (incl.
text) “inside” the
virtual world
Key Concepts: Structure/Flow for AR Application
4. The UX of AR 56

57. Design for users
Visibility / Affordances
Consistency
Feedback
Non-destructive operations (undo/restart)
Human Interface Guidelines
Apple ARKit
Google ARCore
Microsoft (Hololens)
Core principles still apply
574. The UX of AR
https://developer.apple.com/design/human-interface-guidelines/ios/system-capabilities/augmented-reality/
https://designguidelines.withgoogle.com/ar-design/augmented-reality-design-guidelines/introduction.html
https://docs.microsoft.com/en-us/windows/mixed-reality/design

58. Ethan Allen inHome
IKEA Place
Pottery Barn 3D Room View
Burrow
Housecraft
Homestory
Houzz
Graham & Brown Wallpaper
Warby Parker
Interior Define
markilux Awnings
Sotheby’s Curate
Wayfair
Major applications referenced
584. The UX of AR

59. Examples follow for:
Onboarding / Initialization & Surface Detection (Startup)
Placing/Manipulating Objects
Defining Areas (for Surfaces ~ floors/walls)
Connecting with the Catalog
Getting product Details/Specs
Changing Variations (surface/color/material choices)
Changing Physical makeup (size, configuration)
Interactions with Product features/functions
UX/UI Experiences within the AR flow
59
Theexactsequencewill
vary–somefunctionsvary
withthedesiredexperience
Asanapplication,
sometasksarenecessary
4. The UX of AR

60. Onboarding: For
now, it’s a good
idea for apps to
show users what
they will encounter
At startup, or when
first encountering
the feature
Onboarding / Initialization&Surface Detection
604. The UX of AR

61. Initialization & Surface Detection
Indicates when the phone should move
around and then when the surface is found
Onboarding / Initialization&Surface Detection
614. The UX of AR

62. Placing/Manipulating Objects
624. The UX of AR
Items enter the virtual environment from a
catalog/list screen space
Placement / Interaction
• (A) Favor direct manipulation over separate
onscreen controls
• (A) Let people directly interact with virtual objects
using standard, familiar gestures
• (A) In general, keep interactions simple
• Consider when proximity might snap an item to
another (or to a drawn floor/wall edge)
Feedback
• (A) Respond appropriately (quickly) when the user
places an object
• (A) Create convincing illusions when placing
realistic objects – plop down
• (A) Use audio and haptic feedback to enhance the
immersive experience (throughout)
movement on 2 axes at a time
(across surface or up/down)
rotate on 1 axis
single-finger drag two-finger rotation
—Primarily Apple AR guidelines

63. Use Case:
Putting things
right where
you want them
button to drop
one finger
move
two to rotate
slider to
raise/lower
Placing/Manipulating Objects
63
movement on 2 axes at a time
(across surface or up/down)
rotate on 1 axis
4. The UX of AR

64. For products that are “flat,”
like tile or wallpaper, this step
confines where on the flat
planes the products will go
Related: a full planner might
have the user draw out a
complete room with all walls
and more
Surfaces: Masking/Defining Boundaries
644. The UX of AR

65. Users can also be assisted in
gathering other info about
their space — in this case,
measuring the size of a door to
automatically filter for a
replacement
Objects: Measuring Space
654. The UX of AR
—Hanson
Internal Demo

66. Best: Begin AR when the user
has some idea of what they
want – with large catalogs,
this is especially true
Consider the complexity of
the filtering while the user is
holding their phone the
whole time
I recommend helping the
user find items first, and try
on items from a short list
• Most solutions I’ve seen do not
work this way, instead letting
the user initialize and then use
the catalog!
Connecting with the catalog
664. The UX of AR

67. Assuming the user came from a catalog/list,
they will know some things about the item
But while viewing the item, some data might be
helpful – like actual size, or amount of product
Providing Information and Details
674. The UX of AR

68. Use Case: Since AR is actual scale, encourage users to understand
size and relationship to other real and virtual products
Details…
684. The UX of AR

69. Use Case: Since AR can be very high quality,
encourage users to get up close and
evaluate the design from any angle
It probably won’t completely
replace images*, but AR
does not have to look fake
Details…
69*But 3D modeling already has in lots of cases4. The UX of AR

70. Use Case: Since AR can be very high quality, encourage
users to get up close and evaluate the design from any angle
Details…
704. The UX of AR

71. Use Case: With detailed
models, AR can go beyond
scale and aesthetics — users
can be encouraged to check
out construction details
Details…
714. The UX of AR

72. Variations are
presented in
screen space in an
appropriate way
Changing Surfaces/Textures
724. The UX of AR

73. Variations should
be presented in
screen space in an
appropriate way
Changing Surfaces/Textures&Sizes
734. The UX of AR

74. Variations should
be presented in
screen space in an
appropriate way
Changing Size Variations
744. The UX of AR

75. Some products are custom-sized as well as custom configured.
Here user indicates specific measures and other options. Also has some interactions.
Changing Physical Makeup (Parametric Objects)
754. The UX of AR
—markilux with ViewAR platform

76. Changing Physical Makeup (Parametric Objects)
764. The UX of AR
—markilux with ViewAR platform
Some products are custom-sized as well as custom configured.
Here user indicates specific measures and other options. Also has some interactions.

77. Changing Physical Makeup (Parametric Objects)
774. The UX of AR
—markilux with ViewAR platform
Some products are custom-sized as well as custom configured.
Here user indicates specific measures and other options. Also has some interactions.

78. Changing Physical Makeup (Parametric Objects)
784. The UX of AR
—markilux with ViewAR platform
Some products are custom-sized as well as custom configured.
Here user indicates specific measures and other options. Also has some interactions.

79. Use Case: Seeing how
something is put together –
interacting and learning with
animations helps people get
comfortable with purchase
In this demo, it’s just showing
a few steps with animations
Also great for
customer support
Interactions
79
—Hanson Mobile Vision
(download in app store)
4. The UX of AR
https://apple.co/2sz49zl

80. Amazon anti-patterns teach people AR is “basic”
804. The UX of AR
AR accessed at bottom of search (camera)
Catalog has no information except picture
(top picks by who?) (related how?)
Getting details = going to product page
(no going back)
Tray hides screenshot/camera button
♥ adds to wishlist (but can’t add from list)
One item at a time
No history

81. People deserve more out of these experiences!
Where are these features?
• Help make recommendations, whether one
product or recommended combinations
• Help people easily compare a smaller set of items
in their room to decide which they like better
• Use human filters like style, and setting fixed
anchors (like size) to limit options
• What about swapping a set of products for
another –maybe a little less expensive?
• What about recommending items that are
popular with each other?
• Or suggest items that are similar in smart ways
Summary thoughts on the UX of AR up to now…
814. The UX of AR

82. What’s coming next
A bunch of really cool features are in the works
5

83. AsAugmentedRealitycombineswithother
intelligencetechnologies,youcancallit
~XRforeXtendedReality
~orMRforMixedReality
Themainpoint:
You’llbeabletomakeevenbetterexperiencesastimegoeson
Physical
Reality
Digital
Reality
Extended or Mixed Reality

84. What comes next?
5 – What’s coming next 84
For more realistic experiences
• Diminished Reality
• Occlusion with real world objects
• Collision and Physics with real world objects
For more helpful and engaging experiences
• Better space recognition and auto-masking
• Object and context identification/awareness
• Co-creation and shared experiences
• Big products and large-scale surfaces, especially outside
Expect change!
Here are some features that will make
AR even more immersive when
shopping for complex products
Some are coming soon, some are big
problems to solve

85. Where AR giveth,
“diminished reality”
taketh away.
Affects things like rugs,
window treatments,
furniture, faucets
Future: Diminished Reality
855 – What’s coming next

86. You might know of this as
“content aware fill” in
Photoshop
Future: Diminished Reality
865 – What’s coming next
—yokoyalab on YouTube (2013)

87. AR looks great, until
things get in the way
The big thing to
solve: occluding
virtual objects with
real world
Future: Handling Occlusion
87
AR always sits on top
5 – What’s coming next

88. AR looks great, until
things get in the way
Future: Handling Occlusion
885 – What’s coming next

89. Future: Collision, Physics, Interaction
895 – What’s coming next
—Demo from 6D.ai ~ 6D Reality platform

90. People Occlusion– ARKit3 + latest hardware
905 – What’s coming next

91. The ability to detect and
recognize the 3D pattern
of physical objects and
align virtual elements
Allows us to “see inside”
and opens the door to
more immersive pre-sale
and support experiences
Future: Object recognition and alignment
91
—Demo from PTC Vuforia platform
https://www.ptc.com/en/products/augmented-reality5 – What’s coming next

92. With machine learning and intelligent
features we can build on object
recognition and make preparation
easier and add guidance
Ideas:
Placing objects smarter
• Place AR chairs next to a real couch
• Setting AR table items at each real seat
• Applying AR drapes to real windows
Automatically, accurately masking entire
floors and walls, cutting out openings
Future: Intelligence to help with decisions
925 – What’s coming next
—Stanford Computational Vision & Geometry Lab
cvgl.stanford.edu/projects/objectnet3d/

93. Apple, Google, and others are
working on persistence —
retaining the 3D map at close
resolution
This can one day allow:
• Accurately positioning objects for
multiple people with their own
devices
• The ability to pick up where you
left off, or have others peer into your
world and make suggestions
• Take your project to the store or
vice versa — have experts guide and
make changes — see in a higher-
resolution “holodeck”
Future: Co-creation and shared experiences
935 – What’s coming next

94. Discussion6
@mikeosswald
Please connect and follow :)

95. AR’s future will build on current UX solutions
for complex purchases
Help people get to a smaller decision set,
then use AR to help them evaluate
• Use for up close details, actual scale, assembly
• Recommendations: help people choose what fits their needs
• Comparisons: good, better, or however people want to make a choice
• Calculations of materials, suites of products
• Smart ways to connect with and filter the catalog
Think about future features that will make the
experience even more immersive and helpful
Final Summary, Q&A
6 – Discussion 95
@mikeosswald
Session
Hashtag
#UXAR19

96. Thank you!
96
@mikeosswald
Please connect and follow :)
Session Hashtag #UXAR19