Slides from my session at the Global XR Conference. Dec 2021.

1. Getting started with Unity and
AR/VR for the .NET developer…
Reloaded!
Davide Zordan
B: https://davide.dev
E: mail@davide.dev
T: @DavideZordan

2. Davide Zordan
Senior software engineer
MCSD, MCSA, MS, MCP
Microsoft MVP (2009-2014)

3. Agenda
Introduction to Mixed Reality
VR interactions and locomotion using Unity and SteamVR
Enabling Oculus Quest passthrough
Telepresence using points cloud
Hand-tracking

4. What is Mixed Reality?

5. The state of things as they “actually
exist” through our human senses
without any technology.
Artificially created sensory experiences of
people, environments and objects, which can
include sight, touch, hearing, and smell.
PHYSICAL REALITY VIRTUAL REALITY

6. MIXED REALITY
VIRTUAL REALITY
PHYSICAL REALITY

7. PHYSICAL
REALITY
DIGITAL
REALITY
MIXED REALITY (MR)
Windows Mixed Reality Holographic Devices (see-through) / Magic Leap / Nreal Light
/ RealWare HMT-1 / LYNX-R / Meta Project Aria, Project Cambria / Varjo
Oculus Quest 2 / HTC VIVE - Cosmos / Valve Index / Windows Mixed
Reality Immersive Devices (Opaque) / Varjo
AR
TODAY
VR
TODAY
Physical - Digital Reality Spectrum
*The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, "A Taxonomy of Mixed Reality Visual Displays."

8. How can we design for AR/VR?

9. Designing great VR experiences
Place illusion
Plausibility illusion
Embodiment illusion
Source: https://www.coursera.org/lecture/introduction-virtual-reality/introduction-to-plausibility-illusion-psi-K5PGj

11. A new way to interact

12. Gaze
Gesture
Voice
Motion control
World coordinates
Spatial audio
Spatial mapping
Hand-tracking
Eye-tracking

13. VR interactions
Pointing
Selecting
Grabbing

14. VR Locomotion
Eye gazing
Teleportation
Continuous walk

15. DEMO
VR locomotion and custom interactions

17. 1 – Using the SteamVR controllers bindings

18. Rotate the player

19. Targeting multiple headsets

20. 1 – Creating a VR headset manager

21. 2 – Initialising device-specific settings

22. Adding vocal commands:
• Unity KeywordRecognizer
• Natural Language Understanding
… available online at https://davide.dev/blog

23. DEMO
Basic selection and interactions

25. Performance tips

26. 1 - Mark not
moving
objects as
static

27. 2 – Reduce
the number
of draw calls -
baked lights

28. 3 – Reduce
the number
of draw calls
• Combine meshes and materials
• Some Unity plugins available in the asset store
can help

29. 4 – Avoid not
performant
code at
runtime

30. 5 – Reduce
max polys
count in
scenes
• Mobile devices: ~200,000
• Desktop VR: ~1,000,000
• Do not draw polys that cannot be seen by the
user

31. 6 – Optimise
garbage
collection
• Try to avoid the GC
while a scene is
running
• Unity now allows
incremental
garbage collection

32. 7 – Use object
pooling
• Try to not
instantiate objects
at runtime but
reuse instanced
using object pooling
https://github.com/davidezordan/PerformanceBounceback/blob/master/PerformanceBounceback

33. Using Oculus Quest passthrough APIs

34. Image source: www.oculus.com/quest

35. What are
passthrough
APIs?
https://developer.oculus.com/experimental/passthrough-api/
• Passthrough APIs must be enabled using experimental features using
adb:
adb shell setprop
debug.oculus.experimentalEnabled 1
• Must be enabled in the OVR Manager script of the OVRCameraRig:
• Require additional passthrough script:
• Enable
development of
pass-through
experiences using
Quest
• New SDKs recently
announced (Voice,
Hands
manipulations)

37. Collaboration in AR/VR
Telepresence using point clouds

38. Collaboration
• Facebook Horizon Worlds
Social experience
https://www.oculus.com/facebook-horizon/
• Facebook Horizon Workrooms
Collaboration experience for meetings
https://about.fb.com/news/2021/08/introducing-horizon-
workrooms-remote-collaboration-reimagined/
• Microsoft Mesh
Virtual collaboration / Holoportation
https://www.microsoft.com/en-us/mesh
Source: https://techcommunity.microsoft.com/t5/mixed-reality-blog/microsoft-mesh-app-august-2021-update-new-features/ba-p/2746856

39. Telepresence
• Requires a sensor like Kinect for capturing colors and depth data from
cameras.
• Captured data (e.g., arrays of Vector3[] / Color[]) can be transmitted
over the network via sockets.
• Requires a PC to act as a server.
• A full implementation is available here:
Kowalski, M.; Naruniec, J.; Daniluk, M.: "LiveScan3D: A Fast and
Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors". in
3D Vision (3DV), 2015 International Conference on, Lyon, France, 2015
https://github.com/MarekKowalski/LiveScan3D
• Further improvements could include audio streaming, usage of other
protocols (e.g. WebRTC)
• Streaming point
clouds to AR/VR
devices

40. LiveScan 3D project: https://github.com/MarekKowalski/LiveScan3D

41. Adapted from LiveScan 3D HoloLens project: https://github.com/MarekKowalski/LiveScan3D-Hololens

42. Adapted from LiveScan 3D HoloLens project:
https://github.com/MarekKowalski/LiveScan3D-Hololens

44. Implementing Hand tracking

45. Mixed Reality Toolkit for Unity
• https://aka.ms/MRTK
MRTK extensions for Oculus Quest
• https://github.com/HoloLabInc/MRTKExtensionForOculu
sQuest
• https://github.com/provencher/MRTK-Quest

46. Interactions SDK for Oculus Quest (available 2022)
Source: Facebook Connect 2021 - https://www.facebookconnect.com/en-gb/

47. DEMO
Hand tracking using Oculus Quest and
Mixed Reality Toolkit

49. Enhancing the experience
using Mixed Reality services

50. HoloLens 2
Source: https://news.microsoft.com/microsoft-at-mwc19/

56. Microsoft Azure mixed reality services:
• Remote rendering
• Spatial anchors
• Object anchors

57. Summary
What is Mixed Reality
Why Mixed Reality
How to develop for AR/VR
VR Interactions / Locomotion
Performance considerations
Using Oculus passthrough mode
Telepresence
Developing for AR
Mixed Reality Services

58. Key takeaways:
Check performance early and often to avoid motion sickness
Design properly the experience for maximising the user immersion
Use cloud services to enhance the user experience (speech, vision, anchors)
Collaboration using AR/VR is an important topic which requires research

59. “Any sufficiently advanced technology is
indistinguishable from magic”
Arthur C. Clark, Profiles of the future, 1961

60. References
AR / VR / Mixed Reality samples
https://github.com/davidezordan
SteamVR Unity plugin
https://github.com/ValveSoftware/steamvr_unity_plugin
Mixed Reality Toolkit
https://aka.ms/MRTK
Unity XR Interaction Toolkit
https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@0.9/manual/index.html
Unity XR Interaction Toolkit examples
https://github.com/Unity-Technologies/XR-Interaction-Toolkit-Examples
Goldman Sachs VR/AR report
https://www.goldmansachs.com/insights/pages/virtual-and-augmented-reality-report.html

61. References
Coursera “Introduction to Virtual Reality”
https://www.coursera.org/lecture/introduction-virtual-reality/introduction-to-plausibility-illusion-psi-K5PGj
Cognitive Services
https://azure.microsoft.com/en-us/services/cognitive-services
Oculus VR performance optimization guide
https://developer.oculus.com/documentation/pcsdk/latest/concepts/dg-performance-opt-guide
Mixed Reality Academy and Azure Services tutorial
https://docs.microsoft.com/en-us/windows/mixed-reality/tutorials
Unity: the most common mistakes to avoid
https://unity3d.com/how-to/unity-common-mistakes-to-avoid
Microsoft Ignite
https://8gportalvhdsf9v440s15hrt.blob.core.windows.net/ignite2017/session-presentations/BRK2404.PPTX

62. References
Oculus passthrough APIs
https://developer.oculus.com/experimental/passthrough-api/
Kowalski, M.; Naruniec, J.; Daniluk, M.: "LiveScan3D: A Fast and Inexpensive 3D Data Acquisition System
for Multiple Kinect v2 Sensors". in 3D Vision (3DV), 2015 International Conference on, Lyon, France, 2015
https://github.com/MarekKowalski/LiveScan3D
Displaying a Point Cloud Using Scene Depth
https://developer.apple.com/documentation/arkit/environmental_analysis/displaying_a_point_cloud_usin
g_scene_depth
iPad LiDAR Depth Sample for Unity
https://github.com/TakashiYoshinaga/iPad-LiDAR-Depth-Sample-for-Unity
Prof. Mark Billinghurst - COMP 4010: Virtual Reality lectures – Introduction to XR
https://www.slideshare.net/marknb00/comp-4010-2010-lecture1introduction-to-xr

63. References
Prof. Mark Billinghurst - COMP 4010: Virtual Reality lectures – Research directions
https://www.slideshare.net/marknb00/comp4010-lecture12-research-directions

64. Thanks
Davide Zordan
B: https://davide.dev
E: mail@davide.dev
T: @DavideZordan