1. Augmented Reality
in Multi-Dimensionality:
Design for Space, Motion, Multiple Perceptions, and Interactions
Shalin Hai-Jew
Kansas State University

2. Presentation Overview
• Augmented reality (AR)—the use of digital overlays over physical
space—manifests in a wide range of spaces (indoor, outdoor; virtual)
and ways (in real space (with unaided human vision); in head gear; in
smart glasses; on mobile devices, and others). There are various
authoring technologies that enable the making of AR experiences for
various users. This work uses a particular tool (Adobe Aero®) to
explore ways to build AR for multiple dimensions, including the fourth
dimension (motion, changes over time).
2

3. Presentation Overview(cont.)
• Based on the respective purposes of the AR experience, some basic
heuristics are captured for
1. space design (1),
2. motion design (2),
3. multiple perception design (sight, smell, taste, sound, touch) (3),
4. and virtual- and tangible- interactivity (4).
3

4. 4

5. Some Purposes of AR Designs
5

6. Some Purposes of AR Designs
• Entertainment: short-form storytelling, e-books, e- and print posters,
mobile games, scripted chatbots
• Cultural conversations: artful expression, social commentary, political
statements
• Teaching and learning: simulations, data visualizations, analysis,
trainings, lab experiences; repeatability; versioning
• Imagination: exploration (imaginary worlds, actual worlds)
• Creativity and innovation: prototyping, design, looks-and-feels;
product testing with potential users
• Social engagement: intercommunication, co-design, research
6

7. Some Purposes of AR Designs(cont.)
• Introduction of space: navigation, exploration
• Sales and marketing: commercial applications for products and
services
• Industrial applications: worker performance (in manufacturing, for
example), [some of the earliest applications of AR in the world in the
1990s]
• And mixed purposes
• And other purposes
7

8. Appeals to…
• The AR designs may, foremost, appeal to one of the following:
• Sensory experiences (sight, smell, taste, touch, hearing; proprioception;
enteroception); embodied sensory experiences
• Cognitive experiences (emotion, thinking)
• Emotional experiences
• Social experiences (interactions with humans, via real-space or telepresence)
• Mixed experiences
8

9. Some Basics of AR Designs
9

10. Objectives
• What is the AR supposed to achieve? How will it be successful?
• What will the AR experience entail?
• What will be communicated during the AR experience, and how (with what
tools)? What story / stories will be told?
• What will the evoked mood(s) be?
• What sorts of motion will this include?
• What sorts of interactivity will this include?
• How many people can be accommodated?
• How long will this experience be?
• Who will be using it? What are their ranges of needs? (Will they have
the necessary technologies to experience the AR?)
10

11. Objectives(cont.)
• How will the teaching and learning be achieved with the AR
experience?
• What are the learning theories behind the AR experience? Why?
• How long will learners spend in the AR experience?
• How will the AR part integrate with the other parts of the learning lesson?
The assignment?
• What will the role(s) of the teacher / instructor be? The role(s) of
other learners?
• How will the AR experience function for self-learners? Heutagogical learning?
• How wide will the distribution of the AR resources be? Will they be
culturally palatable in a number of different people group contexts?
11

12. 12

13. Tools and Processes
• What technology tools will be used for the creating of the digital
assets? What are the limits and constraints? What are the learning
curves of each of the tools?
• How sustainable are the technology tools?
• The file formats? (Proprietary or more open- and universal-?)
• How will the desired aesthetics and forms be created? What are the
desired atmospherics and moods, and how may these be created?
• Are there free repositories of assets that may be used for the AR
experiences? Are these available for the long-term (and without
strings)?
• What are the most effective and efficient processes and procedures?
13

14. Technological Baselines
Makers
• Small files (under 50 mb)
• Starter files
• Open-access visuals (that can be
transcoded)
• Simple behaviors (processing
speed)
• Need for smart phones (digital
divide)
• Need for cell connection,
Bluetooth, and others
Experiencers
• Need for smart phones (digital
divide)
• Need for Adobe Aero Player (beta)
app (for Android) and Adobe Aero
(for iOS)
14

15. About Adobe Aero
15

16. 16

17. 17

18. 18

19. Light Details about Adobe Aero
• Adobe Aero is an end-to-end tool that enables the creating and
deploying of mobile augmented reality (mobile AR).
• Generally, 3d resources may be created outside of Aero first. These
include objects, characters, and backgrounds.
• The digital characters need to be “rigged” for motion.
• There are starter resources inside Adobe Aero. These resources include
“directable characters,” which are rigged puppets…that scripted motions may
be applied to.
• There is no need to code to program, but there is visual programming.
19

20. Light Details about Adobe Aero (cont.)
• The finalized AR may be experienced on mobile devices with a camera that
can read QR code…or go to a web link…and that can enable camera view
visuals.
• The visuals are placed on a horizontal surface, a vertical surface, or on a
particular hand-held surface (like a card with a marker).
• The respective objects and characters and backgrounds exist with an alpha
channel around which the camera view of the local environment of the
user may be seen.
• Hand jitter will be an issue for the AR experience as it is consumed through
a hand-held device.
• The AR experience may be recorded as a 2d video (.mp4) with Adobe Aero.
The video may be taken off the iCloud and sent on to other locales.
20

21. Technical Constraints
• The mobile screen size is a small one, so the space has to be used
effectively.
• People can experience cybersickness, so the scheduled time has to be
short. People also have sparse attention spans, so that limitation has
to be taken into consideration as well.
• AR technologies and the limits of wifi / cell service means that the
visuals have to be lightweight (few polygons), sparse and highly
selectively identified
• Some AR are video-recorded, so the visuals are consumed less
directly than in a fully embodied experience.
21

22. How to Brainstorm Some
Learning AR Scenes
22

23. 23

24. Early Days Yet
• There is little in the academic literature of applications of AR to
teaching and learning.
• There seem to be few roadmaps for design, development, and
deployment of AR for learning.
• The learning needs to fit the modality (3d, 4d)…
• Builds have to be lightweight and sparse.
• Builds have to be relevant to target learners and the learning.
24

25. Some Tips
• Identify a finite lesson. Delimit the topic. Make the lesson fairly
specific.
• Define target learners.
• Determine whether the AR experience is core or peripheral to the
learning sequence.
• Explore to see if the lesson fits the AR modality (the 3d, the 4d, the
interactivity, and so on).
• Conduct an analysis of the business case. Are there sufficient staff?
Digital resources? Budget? How much time will the digital AR builds
take?
25

26. Some Tips(cont.)
• Is the learning sufficiently value-added to go with an AR version
(given costs)?
• How will the learning be objectively / subjectively assessed?
• What sort of alternate learning sequence can others go through for
the same learning if they are averse to AR? Or experience serious
cybersickness?
• Is there a splashy approach to the AR design that may engender
positive WOM while offering some learning value?
• What modalities will be used for the build-out? Visuals? Sound?
Sequencing? Texture? Touch?
26

27. Idea: How to Do a Pitch for a Script to Movie
Executives
• The homework is to write up a script treatment and then ready a
pitch.
• The learner goes into an AR space with co-learners who are acting as
judges. The person does the pitch…based on particular standards
instantiated in a rubric.
• The judges confer and decide.
• The decision is communicated verbally but also in a 3d object with
particular behaviors…and even sound effects.
27

28. Idea: Identifying Plant Diseases
• Various visuals of 3d plants with symptoms of malaise are shown one
by one.
• There are notes taken about what is observed.
• There are notes also about weather patterns.
• There are notes about other potentially relevant and irrelevant data
(distractors).
• A group of learners (physically co-present or virtually tele-present) do
their best to diagnose the plant disease.
28

29. Idea: Collecting Digital Mushrooms
• There is a discovery learning AR experience.
• Learners are to find particular digital mushrooms of a particular type
and collect these (by clicking on the mushrooms).
• This is an exploratory and cyber-foraging activity.
29

30. Idea: Using a Machine
• Learners will use a graphical user interface of a food vending
machine.
• They will use the machine functions to order food.
30

31. Idea: Going through an Exercise Sequence
• Learners follow various aerobics exercises exercised by 3d characters.
31

32. Idea: Exploring a University Campus
• Learners use their mobile devices with the app loaded and trigger
various QR codes at a university campus.
• They start narrated videos and 3d animations that provide
information about the university.
32

33. Idea: Presenting a Hard Problem
• The rising global temperatures are a global challenge. Various fact-
based visualizations may be created to showcase the problem.
• These may be experienced directly.
• These may be experienced indirectly through narrated video.
• This can be a setup for a class discussion about possible mitigations.
33

34. Idea: AR Flashcards for Rote Memorization
• For foreign language learning (or the acquisition of new
terminologies), a series of AR flashcards may be made that may
encourage practice.
• Perhaps some light visualizations may be made to evoke the term’s
meaning or its context for usage.
34

35. Idea: Weather System
• Learners view the sequence in the building up of a weather system,
its progress, and then its dissipation.
35

36. Space (1)
36

37. 37

38. A 180-degree surround
38

39. Center Core, Outer Rim
• The region for the focal vision is where the main AR content should
be for mobile AR.
• The outer rim may add additional visuals and experiences…to define a
context and to drive attention.
• The edges are only sometimes defined.
39

40. Entry, Proceeding, and Exit
Entry
• Where is the camera on entry to
the AR?
• Is the sightline level to objects on
a flat plane?
• Are the viewers hovering above?
Hovering below?
• What is their designed point-of-
view (POV)?
• Is it clear how a user is to
proceed?
Exit
• Does the user maintain control
on when they leave the AR
experience?
• Is it clear when the AR
experience completes?
40

41. User Role
• It is important to consider who the user is when they enter the AR
context.
• They should also have a sense of how to navigate the virtual
space…and the physical one (if the physical one is part of the AR).
• What is the graphical user interface (GUI)? Are there labels? Buttons?
Arrows?
• Are there sequences of experiences?
• How much control do users have? What choices are they given to make?
41

42. User Experience
• An AR experience is not only visual-physical , but also psychological.
• A person comes into an AR experience with empowerments of
different types.
• A person comes in with different relationships (with automated
scripted agents…or with the live people around them or with
telepresent instantiations of people).
• In learning contexts, a learner will have a “script” to follow.
42

43. What the User Brings
• The AR experience may exact requirements of the user.
• There may be cognitive demands of the user.
• The user may have to engage in an embodied way in real space, for
example.
• The user will be expected to sense-make from what they experience,
to identify what is meaningful and relevant.
• The user may be expected to observe and learn…
• The user may be expected to socialize with those around them.
43

44. An AR Experiential
Wrap
A concave half-sphere of visuals and visuals
with polygons (180-degrees)
The space is concave
The polygons may be convex (as in this
scene) (or can be concave)
Or the experience can be a full-surround or
360-degrees
44

45. Near-Space, Middle-Space, Far-Space
45

46. Human Attention and its Management
• Human attention—in an AR experience—is directed in various modalities:
• Visually:
• by object size (large vs. small, visual prominence, right-sizing vs. exaggerated sizing),
• by color (bright vs. dark / contrast, forward vs. receding colors),
• by outlining,
• by posterizing (for a collage effect),
• by attraction (symmetry) vs. unattraction (asymmetry),
• by centrality of placement vs. peripheral placement, by foreground placement vs.
background, by visual hierarchy in compositing, by filled vs. empty / negative space
• by motion vs. stillness,
• by facial depictions (such as a character),
• by framing,
• by focus vs. blur
46

47. Human Attention and its Management(cont.)
• Sound:
• by place of origin,
• by alignment of sound with characters or scenes,
• by dissonance,
• by loudness
• by acuteness vs. ambiance
• Directions: arrows, signs
47

48. Human Attention and its Management(cont.)
• Conventions of visual storytelling and the setting of expectations
• The mind carries visual ideas forward into a space (off-screen)
• Creation of visual and auditory atmospherics
• Navigation track, lit paths, arrows
48

49. Compositing
• Scene design, scenario design, implied backstory
• Sizing of AR objects and characters in relation to each other
• Coming at scene from above for most tabletop visuals
• Fewer approaching at the same level of the horizon line of the AR visual
• Rare from bottom looking up at an AR visual (would need the visual to be
above the general sightline, so gyroscope in the smart phone can enable
proper positioning to see from below)
• Fit or unfit of objects in given digital and physical space (if physical
background is relevant)
• Room for AR objects to breathe, not crowding the composition, sufficient
white space (and avoidance of sensory overload)
49

50. Compositing(cont.)
• Expansive use of space, constricting use of space
• Use of space (for AR experience, for interaction, for tactual – material
engagements, for reflectance, and others)
• Reconfigurability of the design? The flexibility of the implementation
of composited scene / scenario?
50

51. Motion (2)
51

52. 52

53. 53

54. Built-in Triggers (for Motions) in Adobe Aero
• Start
• Tap
• Proximity Enter
• Proximity Exit
54

55. Digital Object Motion (in Aero)
• Play animation (if built in)
• Play images
• Play audio
• Spin
• Bounce
• Aim
• Show
• Hide
• Move
• Move to
• Scale
• Rotate
• Orbit
• Follow
• Wait
• Open URL
55

56. Pre-scripted “Starter” Asset Characters
• Locomotion, walk cycle, run cycle
• Flight, hovering
• Dance
• Deictic gesture
• Many others
56

57. Directable Actors (Rigged Digital Puppets)
• Digital puppets as humanoid, animal, and other types
• Rigged with adjustment pins (~ to those used in warp transform) in
the areas of motion or rotation (delimited based on puppet
musculature)
57

58. Thinking about Motion in a Scene
• Simultaneity (and gestalt effects, such as objects moving together as
considered related)
• Sequentiality, often through different start times (appearing and
disappearing objects), episodic changes
• Expression of cause-and-effect
• Expression of process
• Interrelationships between objects, between characters
• Speed, slowness, pacing (and the communication of energy)
58

59. Thinking about Motion in a Scene(cont.)
• Progress
• Traversing spaces
• Transitions (such as to a new scene, a time later, showing passage of
time)
• Energy transfer
• Physics in the created “universe”
• Such as collision effects for rigid bodies
• Tremor, jitter, erratic motions
59

60. Thinking about Motion in a Scene(cont.)
• Expression of character, mood shift (sometimes with incredible
subtlety)
• Expression of ecosystem, environment
• Mix of apparent influences on the depicted motions
• Effects on human user attention and focus
60

61. Motion
• Motion on-screen (sound on-screen)
• Motion off-screen (sound off-screen)
61

62. Multiple Perceptions (3)
62

63. 63

64. Going Multi-sensory for AR Experiences
• An AR experience is most commonly comprised of visuals with
motion
• A few offer the addition of sound
• A rare few offer tactual elements through devices like gloves and
wearables
• Perceptually, there are five senses: sight, smell, taste, touch, and
hearing
• Those who build to dedicated physical spaces may add smells, tastes,
and touches (tactual elements)
64

65. Multiple Perception Design
• Multiple perception design requires planning for coherence in the AR
experience.
• The various elements have to work of a piece.
• Elements should contribute to the overall learning experience.
Perhaps the elements should make the experience more engaging,
more immersive.
• The various elements should not be extraneous or distracting. They
should be purpose-driven.
• Designers suggest that it is helpful to think of the mixes of the sensory
cues together, such as for a “visual-tactual” AR experience, a “visual-
auditory” one, or even a full synaesthetic one.
65

66. About Taste
• For example, perhaps the AR experience is food-related or flavor-
related. Perhaps there is a test kitchen scene.
• Smells and tastes may add authenticity. (Such foods should be vetted for
potential allergens.)
66

67. Interactions (4)
67

68. 68

69. Interactions for Active Engagement
• There are various forms of interactions that may exist in an AR scene,
such as the basic digital object to digital object.
• In a learning context, the human interactions are central:
• Human to digital object or character
• Human to tactual object, to analog object (mobile device and a software
program)
• Human to AR and physical environment
• Human to other people (instructor, co-learners, and others)
69

70. Some Conceptualized AR Scenes
for Learning
70

71. Visual Only, Non-Interactive
• A village scene with humans and animals
• A spinning galaxy
• A fish tank with flora and fauna
71

72. Visual Only, Digital Interactive
• One creature
• Several creatures
• A machine like an ATM machine
• A lab experiment sequence
• Actual feedback loop with results (for more complexity)
72

73. Visual-Physical, Non-interactive
• A hand that writes a message
• Sequence of an insect infestation of a type on a particular plant in a
particular season in a particular locale
73

74. Virtual-Physical, Interactive
• Foreign language café with live fellow learners (and / or scripted
agents)
• An AR dart game with tangibles thrown at a virtual / physical board
• A whole bio ecological system with experiential interactive elements
74

75. AR for Teaching and Learning
75

76. A Workflow of
Designing AR for Teaching and Learning
76

77. Conventions
• AR content may be informed by the experiences of the learners with
gameplay, movies, and other cultural artifacts.
• It is important to know what the cultural conventions are…as well as
the multimedia conventions.
• It is important not to look low-budget.
77

78. Alpha and Beta Testing for Quality
Alpha
• Intellectual property
• Publishing law, media law,
privacy protections
• Accessibility
• Style
• Technical functionality
• Teaching and learning
Beta
• User experience with target
learners, usability audiences,
accessibility audiences, etc.
• Salience to users
• Clear learning objectives and
sequences
78

79. Accessibility
• “Augmented Reality for Learning and Accessibility”
• https://www.slideshare.net/ShalinHaiJew/augmented-reality-for-learning-
and-accessibility
• (Please download slideshow off of SlideShare…to avoid all the very
annoying ads every few slides.)
• If using olfactory cues in real space, also adapt to sensitivity to smells.
(Perhaps offer smells on paper in envelopes that are opt-in.)
• If using tactual approaches, avoid particular fabrics. Try not to trigger
anything.
• Avoid the cybersickness that comes from limited visual information and the
pacing of the motion. Limit the amount of time in-scene.
79

80. Live Briefing, Student Participatory Design
• Some AR experiences for learning are led by an instructor.
• The instructor may assign tasks, direct focus, and control for various
AR sequences.
• Some parts may be pre-recorded, and other parts conducted live, or a
combination.
• There can be a pre-brief, during brief, and a post-brief.
• There can be student participatory learning design.
80

81. Return on Investment (ROI)?
Inputs
• Design, development, and
deployment
• Human resources, expertise, time,
learning
• Technologies, coding / transcoding
• Vetting, test design, alpha testing,
beta testing
• Deployment on various
technologies
Outputs (value-added?)
• Teaching
• Learning
• AR resources
• Raw files
• In-house expertise (gain),
collaborative work relationships
w/ outside teams
81

82. Return on Investment (ROI)? (cont.)
Inputs
• Research, data analysis
• Documentation
• Storage
• Outreach, publicity
• Learner support
Outputs (value-added?)
• Repository, open-access sharing?
(extending life cycle of the
resources)
• Future proofing, heritability (and
README documentation), usage in
other contexts
• Word-of-mouth, electronic
word-of-mouth, publicity
82

83. Return on Investment (ROI)? (cont.)
Inputs Outputs (value-added?)
• Versioning for beyond-Aero
usages
83

84. Conclusion and Contact
84

85. 85

86. Conclusion and Contact
• Dr. Shalin Hai-Jew
• Instructional Design
• ITS
• Kansas State University
• 785-532-5262
• shalin@ksu.edu
• The visuals in this slideshow are from both Deep Dream Generator
and screenshots from the Adobe Aero. The diagrams were made in
Visio.
86

87. 87

88. 88