This document provides an overview of augmented reality (AR) and virtual reality (VR) technologies along the immersive experience continuum. It begins with a brief definition of VR and AR and discusses moving from basic VR environments towards more advanced immersive AR applications. Examples are given of educational uses of VR and emerging AR applications. Key challenges of AR development are outlined, including technical challenges of recognition and tracking and social challenges of adoption. Methods of accessing and creating basic AR content are described, from 360 cameras to target-based applications using tools like Vuforia and Unity. The document stresses the evolving nature of these technologies and considering appropriate educational applications.

1. Augmented Reality (AR)
And Related Virtual Reality (VR) Technologies
along the Immersive Experience Continuum
Moving towards new definitions . . . and understandings

2. Agenda for VR/AR portion
•Launch: A visual exploration along the
continuum of immersion and augmented
environments – emerging definitions
•Highlights: VR  AR;360 cameras, mapping,
location based AR, target based AR
•Examples – from basic to advanced; possible ways
for entry considered
•The challenge going into new dimensions

3. Definition, as per Wikipedia – but we will look
at the continuum from VR to mapping to AR

4. But where is reality? Is it
Pokémon Go?
Overlaying / augmenting
the world

5. Ready Player One? Integrating real and virtual; past &
present – within a story line

6. What about
MIT’s Game
Labs – from VR
to location
based AR?
https://education.mit.edu/projects/games/

7. Or, Wisconsin’s story, location-based
AR - ARIS?
https://fielddaylab.org/

8. Universities
are studying
these
environments
over time . . .
And, Empire State
College has used VR
environments since
Second Life since 2005

9. Starting from VR - immersive 2d/3d - VR brings you into an
alternative “universe”
● Open source -
affordable for
educators
● Pre-developed
environments
(islands) - use or
repurpose
● Many free and
Creative Commons
resources
https://www.frontiersin.org/articles/10.3389/frobt.2016.00074/full#h10 an EXCELLENT open-access
articles on the growing activities in and importance of VR environments

10. VR - immersive 2d/3d
Open source -
affordable for
educators
- Such as this
Japanese historical
setting for cultural
immersion within a
high school classroom
https://www.frontiersin.org/articles/10.3389/frobt.2016.00074/full#h10 an EXCELLENT open-access
articles on the growing activities in and importance of VR environments

11. VR - immersive 2d/3d
Open source -
affordable for
educators
Used for social
worker training –
this setting mirror
the home-base of
leaders in this field
https://www.frontiersin.org/articles/10.3389/frobt.2016.00074/full#h10 an EXCELLENT open-access
articles on the growing activities in and importance of VR environments

12. Low barrier
entry 2d/3d
• http://youtu.be/nz5
bf9x5BYs?hd=1
• Low resolution /
web-based 2d/3d
interface
• Set up groups &
interactivity –
segue to more
robust (that need
viewer download)

13. VR - advanced immersive; on the other end of the spectrum
● High resolution graphics
● For-fee games
● 3d headsets - there already or are
coming
● Costly & time consuming to
develop
● Unity / Unreal Engine - key
vendor; low cost developer entry

14. Specialized degrees & certificates – to really become expert
● Opening the way for
AR / VR
● Note the money spent
on gaming
● “With more than $60
billion in annual
revenue, the game
industry is bigger than
Hollywood . . . “

15. VR - advanced immersive
Better graphics; Military
preparation
Much already being done
with simulations for military
- “live,” interactive,
operations preparation –
M.O.S.E.S.
Aviation has long used
simulated training

16. VR - advanced immersive
Simulations
High quality / costly
simulations - but can reduce
the overall cost for clinical
practice training
A MALET student is developing
training materials for this app

17. 360 cameras
in medicine /
healthcare
Used in pre-clinical training for
operating room procedures – a
MALET students is creating videos
for operating room tech training
There is a need to optimize the
time on the actual hospital floor

18. Moving into Augment Reality . . .
For instance, updating
the costly manikins -
healthcare apps
App under development
- to use iPads over
manikin to show
interactive simulated
patient responses, by a
MALET student

19. Where/how can “you” gain entry
into these augmented worlds?
What are the viewer and developer requirements?

20. 360 cameras uploaded to 3d video players – an
example here from YouTube
Lets you experience
• Events
• Locations – operating
rooms; factories;
museums; churches
• Geography / topography
• Professional or amateur
• Growing databases of
user uploads
https://www.engadget.com/2016/08/25/google-offers-360-degree-tours-of-us-
national-parks/ - info on using Google Expeditions and US National Parks
https://youtu.be/Sn_g80FXA-0 - video of using Google Expeditions

21. Smartphones have the intelligence to
interpret and display the video footage
• Features in the
advanced phones
can allow for 360
camera viewing,
provide that you
then have . . .

22. A 3d viewer, such as this inexpensive Google
Cardboard viewer
• Two cameras shoot
the footage,
stereoscopic lenses &
an app integrate the
images for human
viewing
• Here Google
cardboard is holding
a smartphone that is
displaying a 360
camera view – two
lenses “trick” the eye

23. Get a more
sophisticated
viewer
• And, start
viewing more
educational
apps too

24. Simple entry – by creating
360 AR or panoramic view
• https://sites.google.com/view/stemste
am/tools/360-cameras
• Determine area / type of location to
tape
• Turn on camera directly or through
phone app; tripod adds stability
• Download to computer, simple editing,
upload to 3d video (YouTube)
• https://youtu.be/cmJxg29f7Cs - basic
camera review

25. Enhance a 360 or panoramic experience by:
• — Placing the camera in the center of the desired action (oft
times a challenge in action oriented scenarios); using remote
cameras start from phone app to move camera person out of the
picture
• — Determining appropriate “action” scenes — is there a process,
multifaceted environment, a nature phenomenon, a classroom
setting, or the like that you could share effectively with a 360
perspective?
• — Developing the additional contexts, supports or instructional
resources needed to embed the video within a learning or
communication scenario.

26. Advanced editing of
360 footage
Advanced editing platform
& footage from on a vessel
– a MALET student on
vessel navigation

27. Using what exists: Mapping & overlays
https://www.google.com/earth/

28. Mapping &
overlays
https://www.google.com/earth/

29. Reminder: 360 cameras
in medicine /
healthcare
Many apps for 360 besides
geographic; panoramic
perspective on work and
processes

30. Headsets – VR / AR – immersion in the experience
Moving beyond
Google Cardboard

31. However, what can we
expect from AR today?
● “Augmented Reality will not replace the
older interfaces any time soon however,
ever current computing interface still in
use provide a valuable functionality and
while in some cases it may be limiting, in
others it is ideal” 2012
● Therefore, we need the integration of
technologies for their unique and for their
blended purposes Augmented Reality : An Emerging
Technologies Guide to AR
Greg Kipper and Joseph Rampolla

32. How do you access AR?
Viewing
AR
Smartphones
/ YouTube –
3d
Headsets
• Google
Cardboard
• Advanced
Good
desktop

33. How do you develop
AR, today?
Developing
AR
App - Unity / Unreal
• Vuforia (target / pattern)
• GPS – MapBox; AR
Core (Google)
• Other – fast growing
Hovering items
• Create – Blender/ Maja
• Acquire
• Advanced Unity,
Turbosquid, Sketchup
• Basic - Google Poly
GPS

34. What are key AR
challenges?
● Technical – the interface
itself
● Social
○ Bringing in the users
● The vast complexity of the
“system” where multiple
components must work
correctly Augmented Reality : An Emerging
Technologies Guide to AR
Greg Kipper and Joseph Rampolla
“In this section we’ll explore some of the challenges with
Augmented Reality and they are broken out into two categories:
technical and social. Technical challenges examine things like
the recognition problem and sensory accuracy as well as authoring
limitations that stem from different hardware and software
platforms and usage issues such as location and position. Social
challenges explore the issues that don’t tie directly to the
technology supporting AR, but rather some of the potentially
negative side effects of using AR. Technical Challenges
of Augmented Reality: As with any complex
system, one requiring multiple components to
work properly in order for the system to
function, there will always be problems. AR
systems are no different.”

35. Categorization of AR
● From -
Augmented Reality :
An Emerging
Technologies Guide
to AR
Greg Kipper and
Joseph Rampolla
E-book – Chapter 2;
available through
www.esc.edu/library
AR Functions:
-- the augmented perception of reality
-- the creation of an artificial environment
Recognition & Tracking Challenges – “part of
the reason the recognition problem is one of
AR’s biggest challenges: the imperfection of
the real world in contrast to a lab or test
environment”

36. Categorization of AR
Augmented Reality :
An Emerging
Technologies Guide
to AR
Greg Kipper and
Joseph Rampolla
E-book – Chapter 2;
available through
www.esc.edu/library
AR Methods, based on recognition types:
• pattern -- performs recognition of basic
shape or marker which it replaces with a
static or moving digital element (3d
model, audio or video, or website/info)
• outline – recognizes a body feature &
has an interactive virtual embodiment
• location – interacts with smartphone or
device within real world (GPS based)
• surface – augments surfaces in physical
spaces (Google ARCore)

37. BUT WHERE HAVE YOU SEEN THIS ALREADY?
QR (quick response) codes – a
simple form of AR pattern
recognition
● Get QR apps developer
(many free)
● Create text term that
generates the item (a website
address, perhaps; forms &
other uses are possible (for
fee))
● Generate the pattern (press a
button)

38. Testing the QR code
● Test the pattern on your own computer
– move from download on system tray
to a desktop icon and use your cell
phone
● Place QR image on websites, posters,
documents, e-sharing
○ Someone with a smartphone with a
QR reader (free app) can hover
over QR code, click, and see your
app

39. Moving to more sophisticated VR app
development, at least for today
• Need more powerful means of generating the application
• Location based – apps / software that overlays the game engine
developers
• Unity / Unreal Engine
• Add ons and plug in’s being developed
• Google entering this game through Google AR Core
• You need a crystal ball to know how this field will define itself
• True of the entire emerging / immersive tech areas

40. AR – location based: Using MapBox (open source) to
bring in 3d locations into Unity for development
Associated with AR
application, although
more strictly a mapping
application
Using AR over GPS
locations is upcoming
with Google’s large scale
entry

41. More mapping apps – integrating maps & AR
Integrating features from
Google maps into the AR
development software
Eventual education,
textbook publishing,
marketing, tourism, and
commercial

42. More mapping applications – topography & elevation
3D of satellite
imagery too

43. Bringing in GPS coordinates

44. AR Core - Google
• https://developers
.google.com/ar/
• Watch for more to
come from Google
• GPS built –
outdoors and in
home / office
settings too

45. Reminder on ways to view: From basic Google
Cardboard to haptic (sensory) devices too

46. Pattern recognition AR: Developing apps (animations,
websites, etc.) to appear over these targets via shared apps &
targets
An example – a 3d receipe
book shown by bringing
smartphone over the ketchup
label and taking its picture
You would need an app
reader on phone; or, headset
Logistics & mechanical issues
– but the vision
https://youtu.be/GbplSdh0lGU

47. How do you make AR apps?
• Locating courses
• Not yet onto
basic education
servers like
Lynda.com
• Udemy.com –
good pricing;
leading edge;
fewer
instructional
features, though

48. Help from
Unity
• Unity has courses
too, may have AR
some day but
most through the
vendors

49. Creating AR

50. Vuforia -

51. Marketing / Product info
• https://youtu.b
e/6-3nEIX_VRY
• YouTube of
Heinz ketchup
bottle & phone
app
• Need a
common target
for this type

52. AR targets that can trigger 3d hovering items & more; must have
a shared app & common target
Trigger must be available to client
- good for product manufactures -
AR Core video clip
Choose a target sufficiently
complex (the number of turning
points in the image) and also
available to your view

53. Demo of hovering AR objects
https://youtu.be/KMyi-LQYXJU
An animated samurai
warrior hovering over a
9 of clubs

54. AR targets can trigger 3d hovering
when you share the app
● How to share beyond fun -
can gather some of the codes
for websites and the like (put
these on your websites too)
● Growing repository of apps –
contributions from public –
attributions
● http://poly.google.com

55. http://vr.google.com/blocks

56. More from
Google Poly

57. AR targets that can trigger 3d hovering with shared apps &
triggers
Creating within Unity &
Vuforia & publishing to APK
for mobile or computer
The intended audience must
share the common trigger &
have the app
Share via app store or
providing APK direct transfer

58. AR development
platform, example
Work with Unity (game
engine) & Vuforia for the
target connection
Interface showing working
space; player button;
Assets & folders; Inspector
panel
You develop “scenes” with
different objects / items
being shown – trigger item
on the bottom here

59. AR - designing
buttons, scene
changing

60. AR – setting
hovering
animations

61. AR –
integrating
buttons &
background
app images

62. 2d & 3d
overlays

63. Microsoft 3d – with standard Microsoft Office Suite

64. How can we
think, plan and
develop for
these new ways
of
communicating?
Tools
Ideas
Communities

65. So . . . what are we creating for their future?