https://youtu.be/YQlcwrTvDOU
Learn how any creator with an iPhone X will soon be able to leverage the built-in camera to animate their characters' faces for use in any project on any platform. Also, get insight into authoring philosophy and the fun, nitty gritty details of developing digital experiences using real-world data from mobile devices, APIs, and even new wearable AR devices.
Speakers:
Stella Cannefax (Unity Technologies)
Amy DiGiovanni (Unity Technologies)
Jono Forbes (Unity Technologies)
Matthew Schoen (Unity Technologies)
Timoni West (Unity Technologies)
https://unite.unity.com/2018/berlin/
Watermarking in Source Code: Applications and Security Challenges
Reality As Your Next Build Target, Mobile AR, and the Future of Authoring
1. Reality As Your Next Build Target
Mobile AR and the Future of Authoring
Authoring Tools Group, Unity Labs
Timoni West
Matt Schoen
Amy DiGiovanni
Stella Cannefax
Jono Forbes
2. Unity Labs
Animation
Augmented and Virtual Reality
Graphics Research
Future of Game Creation
Machine Learning
Today we are focusing on the
Authoring Tools Group, which has
been investigating how Unity will
both be used to make the future of
spatial computing, and what Unity
will look like in that future.
6. All the world’s a stage, but you don’t know
what’s on it
• The way you author for augmented reality is uncharted territory
• In completely digitals worlds, you know everything that will
happen
• In partially digital worlds, you can only control the digital
• Your experience must be as robust, flexible, and responsive as
possible
7. SUPER robust, flexible, responsive
• You need to be able to test, test, test
• Test against unusual, inaccessible, or varied
environments
• Must have all the information about the world to edit
directly
• Machine learning can help, but is not widely available in a
way to do what we need—yet
8. The challenge with world data
• Usually only available in apps on the device after shipping
• We need to flip this
• Computer vision providers need to have their tech work
on many kinds of devices
• ML is often tied to specific hardware now—needs to
become more ubiquitous and consistent
11. Use cases
• Sick table jump
• Essentially the demo you saw
• Zeldify the world
• Floor -> Water // Tables -> Grass // Walls -> Cliffs
• Character enters a room
• Semantically understanding a door and a chair
• How do I food?
• Great semantics & object tracking
13. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
14. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
• Analytics will be a big deal for AR devs
15. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
16. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
• Analytics will be a big deal for AR devs
23. Conditions
• Check against real world data
• Flexible
• Adaptable
• Author around the real world
Conditions specify what a MARSEntity
requires to perform some kind of function
29. It’s all relative
• MR authoring =/= traditional 3D authoring
• The scene is an abstract setup of conditions about the real world
• World scaling is necessary to support certain use cases - it must be
clear what scale your content is relative to real objects
• Positioning of entities is not relevant at runtime, but in editor is
meant to convey how the content is spatially related
37. Performance
Modern mobile devices experience performance drops due to heat
and processor throttling.
Graph is from our Mobile Performance Handbook:
http://on.unity.com/2Di8Hl7
38. Performance
MARS strives to be efficient in several ways:
• The behind-the-scenes work is distributed across time
• Built-in module to run processing tasks on an interval
• Managed memory is allocated only when absolutely necessary
40. Hardware
Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Object recognition Light Estimation
ARKit devices X X X (X) X X X X
ARCore devices X X X (X) X (X)
Tango (defunct) X X X X
Hololens X X X
Magic Leap X X X X ? X X ? ? X
Vive Pro X X X X X
Windows MR X
Mirage Solo X
Santa Cruz X ? ? ? ? ? ? ? ? ?
Vive X
Rift X
Oculus Go (X)
GearVR (X)
41. Software
PC Mobile Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Obj rec Light Est Body Tracking Hand Tracking
Vuforia X X X X X X X X X
6d.ai X X
Placenote X X
Selerio X X X
ULsee X X X
Visage X X
Google Mobile
Vision
X (X) X
Apple Vision X X (X) X
Wrnch.ai X X X
Leap Motion* X X X
OpenCV X X (X) (X) X X X X
dlib X X
somewhere? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
42. • Faces
• Landmarks from (ARKit) face mesh
• Expressions from blendshapes
• 2D -> 3D landmark poses
• Expressions from landmark positions
• Surfaces
• “Meta-surface”
• Elevation / floor
• Rotation / alignment
• Overlap test
• Pause button
Room for improvement
47. Editor Providers
• Must run in edit mode
• PC / mobile parity
• 3D face pose
• Markerless tracking
• Surface detection
• Remoting and Recording
• Local testing / debug
• Field recording
• Multi-user recording
• Generated Rooms
• ISimulatable and runInEditMode
48. Reasoning APIs
Fill in the missing pieces
• Which surface is the floor?
• Markers for relocalization
• Data correlation
• Which face is which?
• Which object is which?
• More to come