It's high time for augmented reality to be brought to a wider audience. In ABB, we know that it is not just a gimmick any more. However, with every innovative technology comes new challenges. In these slides, we show how to overcome them and deliver valuable products with Hololens and Unity. Speakers: Maciej Włodarczyk - ABB Rafał Kielar - ABB Watch the session on YouTube: https://youtu.be/QFsj8Pi_3Ho

1. GenerativeArt—MadewithUnity
How ABB shapes the
future of industry with
Microsoft and Unity
Nick Davis
Senior Content Marketing
Manager
Unity
Nick Klingensmith
Mixed Reality Academy
Lead Engineer
Microsoft
Maciej Włodarczyk
IS Technology Expert
Chief Specialist
ABB
Rafał Kielar
AR/VR Developer
ABB

2. Agenda
— Microsoft: The future of AR with HoloLens 2
— ABB: Best practices for AR field application development
— Q&A

3. 3
Unity is the
leading
real-time 3D
development platform
Unity

4. Gaming Gaming
Film
Architecture
Engineering
Construction
Utilities
Energy
Education
Advertising
Yesterday Today
Automotive
Aerospace
Government
Consumer Products
Retail
Industrial Machinery
Healthcare
Life Science
Technology

5. Gaming Gaming
Film
Architecture
Engineering
Construction
Utilities
Energy
Education
Advertising
Yesterday Today
Automotive
Aerospace
Government
Consumer Products
Retail
Industrial Machinery
Healthcare
Life Science
Technology

6. Design
● Experience-based design
● Immersive collaboration for rapid iteration
● Human Machine Interface development
Simulation & engineering
● Autonomous system development
● Virtual commissioning
● System-level simulation
Production &
deployment
● Digital factory
● HMI deployment
● Training & guidance
Marketing & sales
● Interactive product
configurators
● Cross platform immersive
consumer experience: Dealer
showroom, event XR
engagement
Service & usage
● IoT-based experience
● AR-assisted maintenance
● Interactive training
● Connected customer experience
Real-time 3D across the product lifecycle

7. Design
● Experience-based design
● Immersive collaboration for rapid iteration
● Human Machine Interface development
Simulation & engineering
● Autonomous system development
● Virtual commissioning
● System-level simulation
Production &
deployment
● Digital factory
● HMI deployment
● Training & guidance
Marketing & sales
● Interactive product
configurators
● Cross platform immersive
consumer experience: Dealer
showroom, event XR
engagement
Service & usage
● IoT-based experience
● AR-assisted maintenance
● Interactive training
● Connected customer experience

8. Gaming Gaming
Film
Architecture
Engineering
Construction
Utilities
Energy
Education
Advertising
Yesterday Today
Automotive
Aerospace
Government
Consumer Products
Retail
Industrial Machinery
Healthcare
Life Science
Technology

9. of all AR/VR content
is powered by Unity
60%

10. AR Foundation
Multi-platform AR development framework

11. Build once in Unity,
deploy across AR devices

12. Nick
Klingensmith
Mixed Reality Academy Lead
Engineer

13. — Nick Klingensmith — @koujaku

15. FULLY ARTICULATED HAND TRACKING

19. DYNAMICS 365 REMOTE ASSIST
Empower technicians to solve
problems efficiently by working
together from different
locations

20. MixedReality-WebRTC
aka.ms/MRWebRTC

21. Improved skilling
4x improvement to service time among
employees who receive mixed reality training.
80% of companies improve collaboration across cross-
functional teams using mixed reality.
MIXED REALITY DRIVES TRANSFORMATIONAL BENEFITS
Efficiency and quality gains
25% boost in productivity, plus errors reduced to
effectively zero, with directions and data given in
mixed reality

22. DYNAMICS 365 GUIDES
Help employees learn new
skills faster with Dynamics 365
Guides

24. — Thank you!
— aka.ms/IWantMR
— aka.ms/MRTK
— Nick Klingensmith @koujaku

25. Maciej
Włodarczyk
IS Technology Expert Chief
Specialist
Rafał Kielar
AR/VR Developer

26. GenerativeArt—MadewithUnity
Agenda:
About us
Augmented reality
How to get started
Case study
HoloLens 1 to 2 migration

27. ABB IS Innovation & Digital Scouting
— Technical CoE
— Digital Scouting
— IS Innovation
– Founded in 2014
– 8 AR projects
– Minimum Awesome Product

28. Interaction between realities:
— Ability to see hidden
/inaccessible elements
— Adapting holograms to
real life
Everyone is an expert:
— Remote support
— Perform on-site
actions without
additional preparation
Real life HUD:
— Instant access to real
life data
— Augmented guidelines
to help in real world
— Hands-free navigation
Why does AR stand out?

29. GenerativeArt—MadewithUnity
Case study:
ABB Ability™ Augmented
Field Procedures

30. Problems to solve
Time-consuming trainings for field
operators
Tasks dangerous for operators if
done improperly
Procedures written on paper
Difficult communication between
on-site and control room operators
Task assigning and tracking
Difficult access to device
parameters

31. On-site digitalized proceduresMulti-platform (mobile + AR) application
with common backend
Our solution

32. — Integration with
live device data
— Documents panel
(animation, video,
PDF)
— Remote assist panel
Our solution

33. Challenges we faced
— Location/device recognition
— UI design (safety concerns)
— Lack of internet connection on industrial sites
— Ergonomy of interactions
— Storing hologram positions

34. Designing Augmented Reality UI
Disappearing windows
Menu obscuring user's view
Auto docking/scaling
Clear way of navigation
Easily visible windows
Context menu on request
Adjustable windows

35. Designing Augmented Reality UI
Too small interactables for gesture/gaze
No accounting for FOV limitations
Voice commands as the only way of
interaction
Bigger elements if far interaction
Buttons close to context
Eye tracking for scrolling
Multiple interaction possibilities

36. Ergonomy of interactions
— Voice commands
— "One click experience"
— Window spawning
— Automate as much as
you can
— Instinctive design
— Testing involves your
whole body

37. • QR scan provides its
position and rotation in
Unity coordinate
system
• Not using web camera
• QR codes allow for
easy data gathering
• ARUCO codes allow to
precisely determine
location
• Many similar devices
on-site make it difficult
to identify them
individually using
image/model
recognition
• Can’t download specific
data
Device & Location recognition
Currently
Improvments

38. Storing hologram
positions and data
Storing
— World anchors
— Azure spatial anchors
Data
— Connecting anchors with
backend data
— Storing coordinates on backend

39. GenerativeArt—MadewithUnity
How to get started:
Unity & Mixed Reality
Toolkit

40. MRTK building blocks
Examples

41. MRTK building blocks
January 29, 2020Slide 41
Examples
Interactive elements
Sliders
Buttons
Manipulation & gestures

42. GenerativeArt—MadewithUnity
HoloLens 2
migration process

43. — Cannot run HoloLens 1 apps on
HoloLens 2
• UWP on x86 platform vs UWP on ARM
platform
• Breaking changes in HoloToolkit →
MixedRealityToolkit
— Some effort is required to prepare app
to run on HoloLens 2
Migrating projects
January 29, 2020Slide 43
Analysis and basic questions

44. Migrating projects
January 29, 2020Slide 44
Analysis and basic questions
Question Consideration
Which Unity version is used for project development? Migration between Unity versions can be big task (ex.
TextMeshPro asset) (latest editor version recommended)
Which scripting backend is used? .Net → deprecated in Unity 2018.2
IL2CPP → the only option in latest Unity version
What native platform plugins are used in project? Which platform given plugin supports?
x86? x64? ARM?
How are user interactions designed? On HoloLens 2 device the user interface should be closer to
the user to be able to use hand (finger) manipulation.

45. — Step 1: Download tools
• Install the most recent version of Windows 10
• Visual Studio 2019 (16.1 or higher)
• Windows SDK, build 18362 or later
• (optional) HoloLens Emulator
— Step 2: Update Unity Project
• Migrate project to latest version of
Unity and Update scene/project settings in Unity
Migrating projects
January 29, 2020Slide 45
Steps
è
è

46. — Step 3: Migrate to MRTKv2
• New input model
• Replace scene prefabs
• Replace interfaces in scripts
• Replace components
Migrating projects
January 29, 2020Slide 46
— Step 4: Compile for ARM
• Dependencies/plugins for ARM procesor
• Developing for ARM64 applications is generally
preferred as there is a material difference in
performance.

47. GenerativeArt—MadewithUnity
Thank you
for listening.

48. XR Interaction Toolkit
AR/VR app interactivity without coding
Preview in 2019.3 release

49. Mixed and
Augmented Reality
Studio (MARS)
Work environment for intelligent AR authoring
Coming 2020