SlideShare uma empresa Scribd logo
1 de 50
Baixar para ler offline
Reality As Your Next Build Target
Mobile AR and the Future of Authoring
Authoring Tools Group, Unity Labs
Timoni West
Matt Schoen
Amy DiGiovanni
Stella Cannefax
Jono Forbes
Unity Labs
Animation
Augmented and Virtual Reality
Graphics Research
Future of Game Creation
Machine Learning
Today we are focusing on the
Authoring Tools Group, which has
been investigating how Unity will
both be used to make the future of
spatial computing, and what Unity
will look like in that future.
Authoring Tools Group
AnimateVR
ARView
Carte Blanche
EditorXR
MARS
XR Foundational Toolkit
Reality As Your Next Build Target, Mobile AR, and the Future of Authoring
Project MARS
All the world’s a stage, but you don’t know
what’s on it
• The way you author for augmented reality is uncharted territory
• In completely digitals worlds, you know everything that will
happen
• In partially digital worlds, you can only control the digital
• Your experience must be as robust, flexible, and responsive as
possible
SUPER robust, flexible, responsive
• You need to be able to test, test, test
• Test against unusual, inaccessible, or varied
environments
• Must have all the information about the world to edit
directly
• Machine learning can help, but is not widely available in a
way to do what we need—yet
The challenge with world data
• Usually only available in apps on the device after shipping
• We need to flip this
• Computer vision providers need to have their tech work
on many kinds of devices
• ML is often tied to specific hardware now—needs to
become more ubiquitous and consistent
Jono Forbes
UX Lead - MARS
Unity Labs
Designing for an
unpredictable real world
Use cases
• Sick table jump
• Essentially the demo you saw
• Zeldify the world
• Floor -> Water // Tables -> Grass // Walls -> Cliffs
• Character enters a room
• Semantically understanding a door and a chair
• How do I food?
• Great semantics & object tracking
Resonai
Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
• Analytics will be a big deal for AR devs
Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
• Analytics will be a big deal for AR devs
Markers
3D Markers / Geofences
Fallbacks
Procedural content
Procedural content
Amy DiGiovanni
Software Engineer
Unity Labs
Conditions
• Check against real world data
• Flexible
• Adaptable
• Author around the real world
Conditions specify what a MARSEntity
requires to perform some kind of function
Spatial conditions in the scene view
Miniature worlds
Scale the camera parent
“XR Cameras” GDC talk by Matt Schoen
World scale in AR
World scale in the scene view
It’s all relative
• MR authoring =/= traditional 3D authoring
• The scene is an abstract setup of conditions about the real world
• World scaling is necessary to support certain use cases - it must be
clear what scale your content is relative to real objects
• Positioning of entities is not relevant at runtime, but in editor is
meant to convey how the content is spatially related
Stella Cannefax
Software Engineer
Unity Labs
Contextual Authoring
Instead of explicitly designing scenes, think about
what context you need in the real world.
Data-Driven Authoring
Data-Driven Authoring
Each entity defines only what it depends on.
Data-Driven Authoring
Two more big advantages:
1.Easier multi-platform support
1.Simulation & testing
Data-Driven Authoring
Queries (editor)
Performance
Modern mobile devices experience performance drops due to heat
and processor throttling.
Graph is from our Mobile Performance Handbook:
http://on.unity.com/2Di8Hl7
Performance
MARS strives to be efficient in several ways:
• The behind-the-scenes work is distributed across time
• Built-in module to run processing tasks on an interval
• Managed memory is allocated only when absolutely necessary
Matt Schoen
Integrations Lead - MARS
Unity Labs
Hardware
Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Object recognition Light Estimation
ARKit devices X X X (X) X X X X
ARCore devices X X X (X) X (X)
Tango (defunct) X X X X
Hololens X X X
Magic Leap X X X X ? X X ? ? X
Vive Pro X X X X X
Windows MR X
Mirage Solo X
Santa Cruz X ? ? ? ? ? ? ? ? ?
Vive X
Rift X
Oculus Go (X)
GearVR (X)
Software
PC Mobile Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Obj rec Light Est Body Tracking Hand Tracking
Vuforia X X X X X X X X X
6d.ai X X
Placenote X X
Selerio X X X
ULsee X X X
Visage X X
Google Mobile
Vision
X (X) X
Apple Vision X X (X) X
Wrnch.ai X X X
Leap Motion* X X X
OpenCV X X (X) (X) X X X X
dlib X X
somewhere? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
• Faces
• Landmarks from (ARKit) face mesh
• Expressions from blendshapes
• 2D -> 3D landmark poses
• Expressions from landmark positions
• Surfaces
• “Meta-surface”
• Elevation / floor
• Rotation / alignment
• Overlap test
• Pause button
Room for improvement
Functionality Injection
Functionality Injection
Functionality Injection
MARS Provider Types (so far)
• CameraImage
• CameraIntrinsics
• CameraOffset
• CameraPose
• CameraPreview
• CompassHeading
• FaceTracking
• FacialExpressions
• FunctionalityInjection*
• LightEstimation
• MarkerTracking
• PlaneFinding
• PointCloud
• ReferencePoints
• WorldLocation
Editor Providers
• Must run in edit mode
• PC / mobile parity
• 3D face pose
• Markerless tracking
• Surface detection
• Remoting and Recording
• Local testing / debug
• Field recording
• Multi-user recording
• Generated Rooms
• ISimulatable and runInEditMode
Reasoning APIs
Fill in the missing pieces
• Which surface is the floor?
• Markers for relocalization
• Data correlation
• Which face is which?
• Which object is which?
• More to come
Thank you!
Hacked

Mais conteúdo relacionado

Semelhante a Reality As Your Next Build Target, Mobile AR, and the Future of Authoring

HoloLens.pdf
HoloLens.pdfHoloLens.pdf
HoloLens.pdfVishwas N
 
Learning The Rules to Break Them: Designing for the Future of VR
Learning The Rules to Break Them: Designing for the Future of VRLearning The Rules to Break Them: Designing for the Future of VR
Learning The Rules to Break Them: Designing for the Future of VRMichael Harris
 
Context Rising : Wearable Interfaces
Context Rising : Wearable InterfacesContext Rising : Wearable Interfaces
Context Rising : Wearable InterfacesKharis O'Connell
 
SEARIS 2014 Keynote - MiddleVR - Philosophy and architecture
SEARIS 2014 Keynote - MiddleVR - Philosophy and architectureSEARIS 2014 Keynote - MiddleVR - Philosophy and architecture
SEARIS 2014 Keynote - MiddleVR - Philosophy and architectureSebastien Kuntz
 
Building a game engine with jQuery
Building a game engine with jQueryBuilding a game engine with jQuery
Building a game engine with jQueryPaul Bakaus
 
The future of User Interface
The future of User InterfaceThe future of User Interface
The future of User InterfaceJerome Lacote
 
Comp4010 Lecture7 Designing AR Systems
Comp4010 Lecture7 Designing AR SystemsComp4010 Lecture7 Designing AR Systems
Comp4010 Lecture7 Designing AR SystemsMark Billinghurst
 
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with UnitySAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with UnitySebastien Kuntz
 
T3con10_html5_kosack_zinner
T3con10_html5_kosack_zinnerT3con10_html5_kosack_zinner
T3con10_html5_kosack_zinnerRobert Zinner
 
Surface computing
Surface computingSurface computing
Surface computingAkash R
 
Introduction to three.js & Leap Motion
Introduction to three.js & Leap MotionIntroduction to three.js & Leap Motion
Introduction to three.js & Leap MotionLee Trout
 
COMP 4010 - Lecture11 - AR Applications
COMP 4010 - Lecture11 - AR ApplicationsCOMP 4010 - Lecture11 - AR Applications
COMP 4010 - Lecture11 - AR ApplicationsMark Billinghurst
 
UX Unicorns - legend or fable?
UX Unicorns - legend or fable?UX Unicorns - legend or fable?
UX Unicorns - legend or fable?Vanessa Kirby
 
Introduction to mobile programming with Androids.
Introduction to mobile programming with Androids. Introduction to mobile programming with Androids.
Introduction to mobile programming with Androids. Maksim Golivkin
 
VR and MR in games: today, tomorrow and day afte
VR and MR in games: today, tomorrow and day afteVR and MR in games: today, tomorrow and day afte
VR and MR in games: today, tomorrow and day afteDevGAMM Conference
 
The convergence of all things (wdu keynote)
The convergence of all things (wdu keynote)The convergence of all things (wdu keynote)
The convergence of all things (wdu keynote)Chris Wilson
 
2022 COMP4010 Lecture 6: Designing AR Systems
2022 COMP4010 Lecture 6: Designing AR Systems2022 COMP4010 Lecture 6: Designing AR Systems
2022 COMP4010 Lecture 6: Designing AR SystemsMark Billinghurst
 
Microsoft surface by NIRAV RANA
Microsoft surface by NIRAV RANAMicrosoft surface by NIRAV RANA
Microsoft surface by NIRAV RANANirav Rana
 

Semelhante a Reality As Your Next Build Target, Mobile AR, and the Future of Authoring (20)

HoloLens.pdf
HoloLens.pdfHoloLens.pdf
HoloLens.pdf
 
Learning The Rules to Break Them: Designing for the Future of VR
Learning The Rules to Break Them: Designing for the Future of VRLearning The Rules to Break Them: Designing for the Future of VR
Learning The Rules to Break Them: Designing for the Future of VR
 
Context Rising : Wearable Interfaces
Context Rising : Wearable InterfacesContext Rising : Wearable Interfaces
Context Rising : Wearable Interfaces
 
SEARIS 2014 Keynote - MiddleVR - Philosophy and architecture
SEARIS 2014 Keynote - MiddleVR - Philosophy and architectureSEARIS 2014 Keynote - MiddleVR - Philosophy and architecture
SEARIS 2014 Keynote - MiddleVR - Philosophy and architecture
 
Designing Usable Interface
Designing Usable InterfaceDesigning Usable Interface
Designing Usable Interface
 
Building a game engine with jQuery
Building a game engine with jQueryBuilding a game engine with jQuery
Building a game engine with jQuery
 
The future of User Interface
The future of User InterfaceThe future of User Interface
The future of User Interface
 
Comp4010 Lecture7 Designing AR Systems
Comp4010 Lecture7 Designing AR SystemsComp4010 Lecture7 Designing AR Systems
Comp4010 Lecture7 Designing AR Systems
 
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with UnitySAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
 
T3con10_html5_kosack_zinner
T3con10_html5_kosack_zinnerT3con10_html5_kosack_zinner
T3con10_html5_kosack_zinner
 
Surface computing
Surface computingSurface computing
Surface computing
 
Introduction to three.js & Leap Motion
Introduction to three.js & Leap MotionIntroduction to three.js & Leap Motion
Introduction to three.js & Leap Motion
 
COMP 4010 - Lecture11 - AR Applications
COMP 4010 - Lecture11 - AR ApplicationsCOMP 4010 - Lecture11 - AR Applications
COMP 4010 - Lecture11 - AR Applications
 
UX Unicorns - legend or fable?
UX Unicorns - legend or fable?UX Unicorns - legend or fable?
UX Unicorns - legend or fable?
 
Introduction to mobile programming with Androids.
Introduction to mobile programming with Androids. Introduction to mobile programming with Androids.
Introduction to mobile programming with Androids.
 
VR and MR in games: today, tomorrow and day afte
VR and MR in games: today, tomorrow and day afteVR and MR in games: today, tomorrow and day afte
VR and MR in games: today, tomorrow and day afte
 
20101025 aiai2010
20101025 aiai201020101025 aiai2010
20101025 aiai2010
 
The convergence of all things (wdu keynote)
The convergence of all things (wdu keynote)The convergence of all things (wdu keynote)
The convergence of all things (wdu keynote)
 
2022 COMP4010 Lecture 6: Designing AR Systems
2022 COMP4010 Lecture 6: Designing AR Systems2022 COMP4010 Lecture 6: Designing AR Systems
2022 COMP4010 Lecture 6: Designing AR Systems
 
Microsoft surface by NIRAV RANA
Microsoft surface by NIRAV RANAMicrosoft surface by NIRAV RANA
Microsoft surface by NIRAV RANA
 

Mais de Unity Technologies

Build Immersive Worlds in Virtual Reality
Build Immersive Worlds  in Virtual RealityBuild Immersive Worlds  in Virtual Reality
Build Immersive Worlds in Virtual RealityUnity Technologies
 
Augmenting reality: Bring digital objects into the real world
Augmenting reality: Bring digital objects into the real worldAugmenting reality: Bring digital objects into the real world
Augmenting reality: Bring digital objects into the real worldUnity Technologies
 
Let’s get real: An introduction to AR, VR, MR, XR and more
Let’s get real: An introduction to AR, VR, MR, XR and moreLet’s get real: An introduction to AR, VR, MR, XR and more
Let’s get real: An introduction to AR, VR, MR, XR and moreUnity Technologies
 
Using synthetic data for computer vision model training
Using synthetic data for computer vision model trainingUsing synthetic data for computer vision model training
Using synthetic data for computer vision model trainingUnity Technologies
 
The Tipping Point: How Virtual Experiences Are Transforming Global Industries
The Tipping Point: How Virtual Experiences Are Transforming Global IndustriesThe Tipping Point: How Virtual Experiences Are Transforming Global Industries
The Tipping Point: How Virtual Experiences Are Transforming Global IndustriesUnity Technologies
 
Unity Roadmap 2020: Live games
Unity Roadmap 2020: Live games Unity Roadmap 2020: Live games
Unity Roadmap 2020: Live games Unity Technologies
 
Unity Roadmap 2020: Core Engine & Creator Tools
Unity Roadmap 2020: Core Engine & Creator ToolsUnity Roadmap 2020: Core Engine & Creator Tools
Unity Roadmap 2020: Core Engine & Creator ToolsUnity Technologies
 
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...Unity Technologies
 
Unity XR platform has a new architecture – Unite Copenhagen 2019
Unity XR platform has a new architecture – Unite Copenhagen 2019Unity XR platform has a new architecture – Unite Copenhagen 2019
Unity XR platform has a new architecture – Unite Copenhagen 2019Unity Technologies
 
Turn Revit Models into real-time 3D experiences
Turn Revit Models into real-time 3D experiencesTurn Revit Models into real-time 3D experiences
Turn Revit Models into real-time 3D experiencesUnity Technologies
 
How Daimler uses mobile mixed realities for training and sales - Unite Copenh...
How Daimler uses mobile mixed realities for training and sales - Unite Copenh...How Daimler uses mobile mixed realities for training and sales - Unite Copenh...
How Daimler uses mobile mixed realities for training and sales - Unite Copenh...Unity Technologies
 
How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...
How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...
How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...Unity Technologies
 
QA your code: The new Unity Test Framework – Unite Copenhagen 2019
QA your code: The new Unity Test Framework – Unite Copenhagen 2019QA your code: The new Unity Test Framework – Unite Copenhagen 2019
QA your code: The new Unity Test Framework – Unite Copenhagen 2019Unity Technologies
 
Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...
Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...
Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...Unity Technologies
 
Supplying scalable VR training applications with Innoactive - Unite Copenhage...
Supplying scalable VR training applications with Innoactive - Unite Copenhage...Supplying scalable VR training applications with Innoactive - Unite Copenhage...
Supplying scalable VR training applications with Innoactive - Unite Copenhage...Unity Technologies
 
XR and real-time 3D in automotive digital marketing strategies | Visionaries ...
XR and real-time 3D in automotive digital marketing strategies | Visionaries ...XR and real-time 3D in automotive digital marketing strategies | Visionaries ...
XR and real-time 3D in automotive digital marketing strategies | Visionaries ...Unity Technologies
 
Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...
Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...
Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...Unity Technologies
 
Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...
Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...
Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...Unity Technologies
 
What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019
What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019
What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019Unity Technologies
 
How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019
How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019
How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019Unity Technologies
 

Mais de Unity Technologies (20)

Build Immersive Worlds in Virtual Reality
Build Immersive Worlds  in Virtual RealityBuild Immersive Worlds  in Virtual Reality
Build Immersive Worlds in Virtual Reality
 
Augmenting reality: Bring digital objects into the real world
Augmenting reality: Bring digital objects into the real worldAugmenting reality: Bring digital objects into the real world
Augmenting reality: Bring digital objects into the real world
 
Let’s get real: An introduction to AR, VR, MR, XR and more
Let’s get real: An introduction to AR, VR, MR, XR and moreLet’s get real: An introduction to AR, VR, MR, XR and more
Let’s get real: An introduction to AR, VR, MR, XR and more
 
Using synthetic data for computer vision model training
Using synthetic data for computer vision model trainingUsing synthetic data for computer vision model training
Using synthetic data for computer vision model training
 
The Tipping Point: How Virtual Experiences Are Transforming Global Industries
The Tipping Point: How Virtual Experiences Are Transforming Global IndustriesThe Tipping Point: How Virtual Experiences Are Transforming Global Industries
The Tipping Point: How Virtual Experiences Are Transforming Global Industries
 
Unity Roadmap 2020: Live games
Unity Roadmap 2020: Live games Unity Roadmap 2020: Live games
Unity Roadmap 2020: Live games
 
Unity Roadmap 2020: Core Engine & Creator Tools
Unity Roadmap 2020: Core Engine & Creator ToolsUnity Roadmap 2020: Core Engine & Creator Tools
Unity Roadmap 2020: Core Engine & Creator Tools
 
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...
 
Unity XR platform has a new architecture – Unite Copenhagen 2019
Unity XR platform has a new architecture – Unite Copenhagen 2019Unity XR platform has a new architecture – Unite Copenhagen 2019
Unity XR platform has a new architecture – Unite Copenhagen 2019
 
Turn Revit Models into real-time 3D experiences
Turn Revit Models into real-time 3D experiencesTurn Revit Models into real-time 3D experiences
Turn Revit Models into real-time 3D experiences
 
How Daimler uses mobile mixed realities for training and sales - Unite Copenh...
How Daimler uses mobile mixed realities for training and sales - Unite Copenh...How Daimler uses mobile mixed realities for training and sales - Unite Copenh...
How Daimler uses mobile mixed realities for training and sales - Unite Copenh...
 
How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...
How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...
How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...
 
QA your code: The new Unity Test Framework – Unite Copenhagen 2019
QA your code: The new Unity Test Framework – Unite Copenhagen 2019QA your code: The new Unity Test Framework – Unite Copenhagen 2019
QA your code: The new Unity Test Framework – Unite Copenhagen 2019
 
Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...
Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...
Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...
 
Supplying scalable VR training applications with Innoactive - Unite Copenhage...
Supplying scalable VR training applications with Innoactive - Unite Copenhage...Supplying scalable VR training applications with Innoactive - Unite Copenhage...
Supplying scalable VR training applications with Innoactive - Unite Copenhage...
 
XR and real-time 3D in automotive digital marketing strategies | Visionaries ...
XR and real-time 3D in automotive digital marketing strategies | Visionaries ...XR and real-time 3D in automotive digital marketing strategies | Visionaries ...
XR and real-time 3D in automotive digital marketing strategies | Visionaries ...
 
Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...
Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...
Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...
 
Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...
Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...
Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...
 
What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019
What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019
What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019
 
How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019
How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019
How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019
 

Último

Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 

Último (20)

Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 

Reality As Your Next Build Target, Mobile AR, and the Future of Authoring

  • 1. Reality As Your Next Build Target Mobile AR and the Future of Authoring Authoring Tools Group, Unity Labs Timoni West Matt Schoen Amy DiGiovanni Stella Cannefax Jono Forbes
  • 2. Unity Labs Animation Augmented and Virtual Reality Graphics Research Future of Game Creation Machine Learning Today we are focusing on the Authoring Tools Group, which has been investigating how Unity will both be used to make the future of spatial computing, and what Unity will look like in that future.
  • 3. Authoring Tools Group AnimateVR ARView Carte Blanche EditorXR MARS XR Foundational Toolkit
  • 6. All the world’s a stage, but you don’t know what’s on it • The way you author for augmented reality is uncharted territory • In completely digitals worlds, you know everything that will happen • In partially digital worlds, you can only control the digital • Your experience must be as robust, flexible, and responsive as possible
  • 7. SUPER robust, flexible, responsive • You need to be able to test, test, test • Test against unusual, inaccessible, or varied environments • Must have all the information about the world to edit directly • Machine learning can help, but is not widely available in a way to do what we need—yet
  • 8. The challenge with world data • Usually only available in apps on the device after shipping • We need to flip this • Computer vision providers need to have their tech work on many kinds of devices • ML is often tied to specific hardware now—needs to become more ubiquitous and consistent
  • 9. Jono Forbes UX Lead - MARS Unity Labs
  • 11. Use cases • Sick table jump • Essentially the demo you saw • Zeldify the world • Floor -> Water // Tables -> Grass // Walls -> Cliffs • Character enters a room • Semantically understanding a door and a chair • How do I food? • Great semantics & object tracking
  • 13. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..)
  • 14. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..) • Finally, very context specific / trait-based • Analytics will be a big deal for AR devs
  • 15. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..) • Finally, very context specific / trait-based
  • 16. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..) • Finally, very context specific / trait-based • Analytics will be a big deal for AR devs
  • 18. 3D Markers / Geofences
  • 23. Conditions • Check against real world data • Flexible • Adaptable • Author around the real world Conditions specify what a MARSEntity requires to perform some kind of function
  • 24. Spatial conditions in the scene view
  • 26. Scale the camera parent “XR Cameras” GDC talk by Matt Schoen
  • 28. World scale in the scene view
  • 29. It’s all relative • MR authoring =/= traditional 3D authoring • The scene is an abstract setup of conditions about the real world • World scaling is necessary to support certain use cases - it must be clear what scale your content is relative to real objects • Positioning of entities is not relevant at runtime, but in editor is meant to convey how the content is spatially related
  • 31. Contextual Authoring Instead of explicitly designing scenes, think about what context you need in the real world.
  • 33. Data-Driven Authoring Each entity defines only what it depends on.
  • 34. Data-Driven Authoring Two more big advantages: 1.Easier multi-platform support 1.Simulation & testing
  • 37. Performance Modern mobile devices experience performance drops due to heat and processor throttling. Graph is from our Mobile Performance Handbook: http://on.unity.com/2Di8Hl7
  • 38. Performance MARS strives to be efficient in several ways: • The behind-the-scenes work is distributed across time • Built-in module to run processing tasks on an interval • Managed memory is allocated only when absolutely necessary
  • 39. Matt Schoen Integrations Lead - MARS Unity Labs
  • 40. Hardware Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Object recognition Light Estimation ARKit devices X X X (X) X X X X ARCore devices X X X (X) X (X) Tango (defunct) X X X X Hololens X X X Magic Leap X X X X ? X X ? ? X Vive Pro X X X X X Windows MR X Mirage Solo X Santa Cruz X ? ? ? ? ? ? ? ? ? Vive X Rift X Oculus Go (X) GearVR (X)
  • 41. Software PC Mobile Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Obj rec Light Est Body Tracking Hand Tracking Vuforia X X X X X X X X X 6d.ai X X Placenote X X Selerio X X X ULsee X X X Visage X X Google Mobile Vision X (X) X Apple Vision X X (X) X Wrnch.ai X X X Leap Motion* X X X OpenCV X X (X) (X) X X X X dlib X X somewhere? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  • 42. • Faces • Landmarks from (ARKit) face mesh • Expressions from blendshapes • 2D -> 3D landmark poses • Expressions from landmark positions • Surfaces • “Meta-surface” • Elevation / floor • Rotation / alignment • Overlap test • Pause button Room for improvement
  • 46. MARS Provider Types (so far) • CameraImage • CameraIntrinsics • CameraOffset • CameraPose • CameraPreview • CompassHeading • FaceTracking • FacialExpressions • FunctionalityInjection* • LightEstimation • MarkerTracking • PlaneFinding • PointCloud • ReferencePoints • WorldLocation
  • 47. Editor Providers • Must run in edit mode • PC / mobile parity • 3D face pose • Markerless tracking • Surface detection • Remoting and Recording • Local testing / debug • Field recording • Multi-user recording • Generated Rooms • ISimulatable and runInEditMode
  • 48. Reasoning APIs Fill in the missing pieces • Which surface is the floor? • Markers for relocalization • Data correlation • Which face is which? • Which object is which? • More to come