The Metaverse will require more content than we can realistically build manually, and it will need to be dynamically generated, personalised and completely interactive. This talk focuses on two groundbreaking projects: Creating an entirely AI-generated and interactive TV show and our inevitable future of 3D streamed media, specifically interactive streamed volumetric sports broadcasts. Adam will show how both projects work under the hood with live demos of the technology and breakdowns of the key points. Components include procedural cinematography, shot evaluation, ML pose estimation, dialogue and narrative synthesis, touch interactivity, intuitive UX and the challenges of streaming huge amounts of content to mobile devices. You’ll see what will hopefully be the very first, fully AI-generated TV show running 24 hours a day and never before presented versions of the latest Metacast technology, transforming how media is broadcast.
From AI-Generated Stories to Interactive Volumetric Content by Adam Myhill, Unity | SIC 2022
1. Building for the Metaverse
From AI-generated stories to
interactive volumetric content
Adam Myhill, Creative Director @ Unity Technologies
Date: Wednesday October 26th
Time: 11:00 AM - 11:40 AM
Location: Room 1 - Upstairs
@adamjcmyhill
3. Adam Myhill
20 years in video games
5 years in film as a DP
Created Cinemachine, sold to Unity
Created Cinecast, Metacast
7 patents in camera, UX, streaming, volumetric file formats
Technical Emmy
Creative Director of Metacast / Sports @ Unity Technologies
Investor and Advisor to Transitional Forms
Owner of 3 alpacas
10. Adam’s hot take on the Metaverse
A persistent shared environment
On a decentralized open standard
Proof of Position + Spatial Contracts
Dynamic subscription to content layers - modified by time, position,
biometrics
While purely creative sub-worlds will be cool (pick the blue or red door)
The data-overlaid layers of content over IRL will be where it really takes
hold
18. 18
WE’VE BUILT A REAL-TIME SHOW
PLATFORM WITH MASSIVE
POTENTIAL.
Our Transforms TV prototype is already proving that
Artificial Intelligence is a viable creative partner for real-
time content through:
★ Live human input
★ Autonomous content generation
★ Dynamic scene assembly
★ Real-time effects
Prototype
Reel (1min)
28. Learnings
Subtle changes in input can have such huge changes on output
The differences between GPT-2 and GPT-3 are astounding and each one has its
quirks and character
It can be totally dynamic - the show is rendered in real-time - so many
possibilities of how people can interact and modify
Building a completely procedural ‘show’ is a possibility
31. Metacast
Vision / Mission: Revolutionize how fans
engage with sports
3D volumetric capture of the event
Streamed in 3D to your mobile
Full agency over time and POV
Super hard
We have it working, it’s real
34. Build your own capture studio
Solve volumetric data (partnership)
Run ML pose estimation
Make your own ML pose estimation
Make your own volumetric file format
Make your own compression
Create custom AI camera system
Make custom volumetric NLE
Figure out how to stream giant data
Design app to render 3D quickly
Design 3D interaction UX
35. Build your own capture studio
Solve volumetric data (partnership)
Run ML pose estimation
Make your own ML pose estimation
Make your own volumetric file format
Make your own compression
Create custom AI camera system
Make custom volumetric NLE
Figure out how to stream giant data
Design app to render 3D quickly
Design 3D interaction UX
Difficult
36. Volumetric Pipeline & Platform
End to end delivery of Volumetric data to fans and broadcasters
Raw
Volumetric
Data
First Citizen
Unity Format
Storage &
Streaming
Unity CCD
Unity
Volumetric Editor
Unity App
Active
Distribution
Networks
37.
38.
39.
40. Estimate Depth
Generate Point
Sets
Depth/PC Fusion
(Registration)
Color
Reprojection
Meshing &
Reduction
Algorithm Overview
Type #1 - Calibrated Environment (Metastage etc)
46. Nobody can be there
to ‘film’ a variable
scenario in real-time
47.
48.
49.
50.
51. New ways to monetize
Virtually try-on and purchase gear from within the app, personalized in-world ads
Drive and measure engagement through integrations
with e-commerce and social media platforms.
53. “Metatok” gives a
compressed TikTok-
like UX to sports
broadcasts
Swipe left/right for
highlights
Swipe up/down for
different fights or
games
At any time you can
still touch-orbit
70. THE ENORMOUS SCALE OF DYNAMIC, REAL-TIME
CONTENT IS LITERALLY BEYOND HUMAN.
There simply aren’t enough creators on earth to fulfill the promise of endless,
immersive, hyper-personalized content.
In short, we’re bringing entertainment to life, by giving it a mind of its own.
In our Transforms TV prototype, we’ve proven that dynamic content is possible. With the right support, our plan is to design and scale this tech to empower studios and creators alike to co-create with cutting edge machine intelligence.
The sheer volume needed to scale endless, hyper-personalized content, is quite literally beyond human.