The slides from my talk, Building The Matrix: Your First VR App at Silicon Valley Code Camp, Oct. 2016. Development, design, and sample projects for virtual reality applications.

1. BUILDING THE MATRIX: YOUR FIRST VR APP
@MISSLIVIROSE
1

2. ABOUT ME
Livi Erickson
Livi.Erickson@Microsoft.com
 Virtual & Augmented Reality Developer /
Evangelist
 VR Blog: The Matrix is my Office
 @MissLiviRose on Twitter
2

3. BUILDING VIRTUAL REALITY APPLICATIONS
 Why Build for Virtual Reality?
 Understanding the VR Device Ecosystem
 Breakdown of a VR application
 Building a VR app: Development Options
 Design Considerations & Concerns for VR today
 Q & A
3

4. WHY BUILD FOR VR?
4

5. 5
It’s Really, Really Fun.

6. 6
We’re Used to 3D Interactions

7. 7
The Freedom to Redefine Possibilities

8.  Storytelling is significantly more immersive in an
environment that surrounds the user
 Virtual experiences improve empathy in the viewer
 Visualizations of complex data
 New and improved ways of interacting with the
technical ecosystem that we’ve been building for
decades

9. VR DEVICE ECOSYSTEM
DESKTOP // MOBILE
9

10. PLATFORM GROWTH
 2012:
 Oculus Developer Kit 1
 2014:
 Oculus Developer Kit 2
 Cardboard
 GearVR Innovator Edition 1
 2015:
 GearVR Innovator Edition 2
 Gear VR
 HTC Vive Developer Kit 1
 OSVR
 2016:
 Oculus Rift
 HoloLens Developer Kit
 HTC Vive Pre
 HTC Vive
 PlayStation VR
 FOVE Developer Kit
 Gear VR

11. TODAY’S TECHNOLOGY
Virtual
Reality
Fully immersive
experience that
replaces your
physical world
Example: Oculus Rift
Augmented
Reality
Overlays digital
information about
the physical world
around you
Example: Google
Glass
Mixed
Reality
Combines virtual
objects and the
physical world
Example: Microsoft
HoloLens

12. HEAD MOUNTED DISPLAYS
Head Mounted Displays (HMDs) can be desktop-
powered devices or mobile-powered devices
 Desktop: Separate display device self-containing
screen, content provided externally from a
separate PC
 Mobile: Content provided and displayed by a
smart phone inserted into the casing
 Standalone: Entirely self-contained headsets
12

13. DESKTOP VIRTUAL REALITY
Powered from external computers
Hardware includes separate screens
Treated as an external display at the hardware level
Software runs on the desktop and is passed to the display
Examples: Oculus Rift, HTC Vive, FOVE
13

14. MOBILE VIRTUAL REALITY
 Headsets are containers that support stereoscopic rendering for mobile phones
 Applications are typical smartphone apps
 The display is the phone screen
 Usually, almost all computational aspects are done on the phone rather than the headset
 Examples: GearVR, Homido, FreeFly, Cardboard
14

15. INPUT - CONTROLLERS
 Traditional controllers don’t solve the “I can’t see
my hands” issues with VR
 Navigation is less natural for movement with
joysticks due to the sense of being in one place
 Limited number of inputs / actions
 Some apps may rely on specific types of input
controllers and have compatibility problems
15

16. INPUT - WEARABLES
 Devices that are lower profile and can be worn
to control various inputs without specific
controller input
 Example: Nod Ring
 Example: Myo Bracelet
 Example: Emotiv Headset
16

17. HAPTICS
 Components to a VR system that provide tactile
feedback based on what is going on in the
virtual environment
 Add to realism of a VR system by allowing the
user to physically touch and experience things
going on in virtual reality
 Example: GloveOne (pictured) has multiple
feedback points that mimic the sense of touch
based on collisions in a VR game
17

18. MORE OPTIONS
 Gestures via cameras: example: Leap Motion Controller
 Omnidirectional treadmills
 Voice commands
18

19. OVERVIEW OF A VR APPLICATION
19

20. DESIGNING & PLANNING
 Consider the platforms that you’re interested in targeting
 Understand the input and design considerations for the given platform as you plan out your experience
 Outline key scenes and emotions that you’re looking to capture in your app
 Choose a development option that aligns with your platform of choice
20

21. 21
Render Loop
Hardware /
Input Update
Game Logic
Lifecycle of a VR App

22. Game Logic
• Defines the behavior of your application
components
• Models and textures are defined for your scene
• Scripts control behavior of the scene objects
• Generally doesn’t rely on the status of the
hardware, but will likely contain checks for
changes
Render Loop
Hardware /
Input Update
Game Logic
22

23. Hardware / Input
• Check on the orientation of the device to
calculate how the camera should render objects in
the scene
• Evaluate if there are actions triggering different
behaviors in your scene
• Update scene variables or components based on
input checks
Render Loop
Hardware /
Input Update
Game Logic
23

24. Render Loop
• Evaluate the scene appearance based on the
orientation of the headset
• Apply effects based on changing scene (shadows,
lighting, etc.) dynamically to match the user’s
expectation
• Draw to the headset / display
• Example: If player has moved under a tree, update
the shadows cast by the player accordingly
Playback Required Refresh Rate
Film / TV 24 – 48 FPS
Console Gaming 30 FPS
PC 50-60 Hz
VR 90-120 Hz
Render Loop
Hardware /
Input Update
Game Logic
24

25. DESIGNING FOR VR
25

26. INTERACTION & DISCOVERY
 There are no standards yet in VR for expected behaviors and interfaces
 Capturing and guiding a user through a process is a new challenge for the medium
 Agency of a player is critical
 The amount of control or interaction a user can have in an environment will vary dependent on
experience and platform
26

27. PRESENCE IN VR
 Consider how your user will want to interact with the environment
 Environment scale will impact how realistic a scene feels to a player
 Non-autonomous character actions will break the sense of self
27

28. Eyes: We’re Moving!
Body: No, we aren’t!
Body: We’re moving!
Eyes: No, we aren’t!
28

29. DISPLAY PERFORMANCE & CONSIDERATIONS
 Low Framerates = Motion Sickness
 “Tearing” around the edges of the display
 Shadowing (duplication) of scene objects at high framerates
 Motion blur
29

30. TOOLS OF THE TRADE
UNITY, UNREAL ENGINE, WEBVR (OH MY!)
30

31. DEVELOPMENT OPTIONS
 Native development with Platform SDKs
 Game Engines:
 Unity
 Unreal
 CryEngine
 Web:
 WebGL VR support through WebVR
 Three.JS
 A-Frame
 Vizor
31

32. EXAMPLE NATIVE TOOLS: OCULUS SDK
 Add stereoscopic, distortion, and update effects
within the graphics pipeline of your application
 Render to a texture specifically output to the
Oculus device as well as to a non-stereoscopic
screen view
 Great for custom game engines adding in VR
support with the Rift
 More customization and control granted
 Versions available for desktop (Rift) and mobile
(GearVR)
32
Source: developer.oculus.com

33. UNITY
 Game Engine from Unity Technologies
 Visual editor for environment building & scripting
 Languages: C#, UnityScript (a JS derivative), Boo
33

34. 34

35. UNITY: VR MODE
 Added in 5.1 (current is 5.4)
 Requires runtime for headsets installed
 Supports a number of devices built-in, many others have packaged support add-ins
 Implementing it:
1. Edit -> Project Settings -> Player
2. Under “Other Settings” check the ‘Virtual Reality Supported’ box
3. Platform dependent on device type
35

36. UNREAL ENGINE
 Game Engine from Epic Games
 Visual editor for environment building &
scripting
 Language: C++
36

37. 37

38. BENEFITS OF GAME ENGINES
 Handles many of the hardware-level interactions with devices
 Templates for different app types are available
 Implements physics and lighting engines
 Robust communities for asset creation
 Additional services for common application needs (networking, multiplayer, awards)
 Central tooling for cross-platform development
38

39. DOWNSIDES OF GAME ENGINES
 Less granular control over application behaviors
 Reliant on implementation of SDKs and devices
 Black box code for engine functionality
39

40. WEB VR
 Web applications with VR support
 Front-end Rendering:
 WebGL
 Three.JS, Babylon.JS
 Desktop & mobile support with one codebase
 www.webvr.info
40

41. BENEFITS OF WEBVR
 No installation of application is needed – run in the browser on any device
 Automatically support mobile and desktop devices
 Utilize the current set of tools and libraries for JavaScript
 Easily switch between VR and non-VR mode
 Offload interfacing with hardware to the browser
41

42. DOWNSIDES OF WEBVR
 Performance hits for non-native graphics
 Non-standard support for some browsers
 Less insight and control into bugs with hardware
 Relies on browser detection of devices
42

43. PLANNING YOUR EXPERIENCE
43

44. THINK ABOUT THE FOLLOWING CONSIDERATIONS:
 3D Development
 Scripting
 Animation & 3D Modeling
 Audio
 Testing
44

45. PLANNING A VR APPLICATION
 Evaluate how the medium lends itself to your game or application
 Minimize the amount of locomotion
 Choose one platform to target at a time
 Read up on best practices, but don't lock things in
 Prepare to iterate - a lot!
45

46. IMPLEMENTATION OF A VR APP - MY STRATEGY
 Build out the basics in your environment so that you have a rough understanding of what your
application components will do
 Script the behaviors for gameplay elements & the rest of the application components
 Finalize the environment with lighting, textures, effects
46

47. TESTING YOUR VR APP
 One person QA is not quite enough!
 Variations in height, gender, life experiences, comfort with technology, etc. will all impact your user to a
much greater extent in VR
 Performance is hugely important to prevent motion sickness
47

48. SAMPLE PROJECTS
SOURCE CODE & TUTORIALS AVAILABLE!
48

49. SPACE PHOTOS IN SPACE
49
Space Photography
Viewer
Built using WebVR
https://github.com/misslivirose/webvrspace

50. BUBBLE BLASTER
50
A mini-game for the
HTC Vive
Built with Unity and
the SteamVR Toolkit
https://github.com/misslivirose/vive_bubbleblaster

51. STAR WARS VR TEXT EFFECT
51https://github.com/misslivirose/SWScrollText
Sample Experience built with
World Canvas for VR Text
Made in Unity

52. VISUAL DATA
52https://github.com/misslivirose/visualdata
WebVR App for Visualizing Excel
Charts
ASP.NET with Three.JS WebVR Site

53. FINAL CONSIDERATIONS
53

54. TODAY'S CHALLENGES
 Input devices are non-standardized across platforms
 Constant evolution in tools
 Market Size
54

55. RESOURCES
 Blog: http://livierickson.com/blog for short-form VR development tutorials, reviews, and news
 GitHub: http://github.com/misslivirose for all the code in my VR experiments (including Kitten VR!)
 ARVR Academy Introductory course materials: http://github.com/arvracademy/intro-academy/
 Just A/VR Show: http://justavrshow.com/
 Learning Unity: http://livi.link/learnunity
55

56. Andrew Moll
Technical Evangelist
Doris Chen
Technical Evangelist
John Alioto
Principal Technical Evangelist
Tobiah Zarlez
Technical Evangelist
Daniel Egan
Technical Evangelist
Livi Erickson
Technical Evangelist
Tim Reilly
Technical Evangelist
Jeremy Foster
Technical Evangelist
Tierney Wixted
Community Evangelist
Annie Bubinski
Community Evangelist
Shaina Houston
Director of Community Evangelism
Yesenia Alvarez
Community Evangelist
Joe Shirey
Director of Technical Evangelism
Beth Massi
Program Manager
Jennelle Crothers
Technical Evangelist
Robin Shahan
Content Developer
Jessica Deen
Technical Evangelist
Justin Chizer
Microsoft Student Partner
MEET THE TEAM

57. Paul Hacker
Visual Studio and Dev Tech
Mathias Brandewinder
Visual Studio and Dev Tech
Deborah Kurata
Visual Studio and Dev Tech
Jeremy Clark
Visual Studio and Dev Tech
David McCarter
Visual Studio and Dev Tech
Ward Bel
Windows Development
Vishal Saxena
Microsoft Azure
Craig Berntson
Visual Studio and Dev Tech
Chander Dhall
Visual Studio and Dev Tech
Steve Evans
Visual Studio and Dev Tech
MEET OUR MVP/RDS

58. • Complete the survey for a chance to win Beats by Dre Headphones
• Microsoft Azure Essentials book giveaway for survey completion
while supplies last
Survey  aka.ms/SVCC
• Connect with our speakers
Visit our booth
• Fill out a session card to enter a raffle for a Ricoh Theta 360 camera
Submit Your Session Cards
MAKE THE MOST OF YOUR EXPERIENCE

59. 59
@misslivirose