Standard proposal: Metadata representation for outdoor Mixed and Augmented Reality Services

1. Woontack Woo, Ph.D.
KAIST UVR Lab.
Daejeon, Korea
ISO SC24 WG9 Meeting @ Seoul, Korea, 01/19/2017]

2. Based on Two Korean Standards
TTAK.KO-10.0434 (Dec. 2010): Representation for AR: POI, Content, Link
TTAK.KO-10.0851 (Dec. 2015): Metadata for AR content visualization & sharing
Proposal on “Metadata: data about data”
Metadata representation for context-aware MAR services
what::Status, (PoI ID, physical/virtual object ID)
who::Role, (author profile)
when::Time, (created, modified)
where::Location, (geographical/relative position)
how::Action, (link/interaction/response method)
why::Goal, reason
Meta-tagging: the process of adding metadata with 5W1H to various
physical/virtual assets/entities

3. Metaverse for MAR Services
Modeling
Interaction
Networking
Sensing
Metaverse
Measured
Space
Physical
Space
Virtual
Space
SensingModeling
Anchoring
(Augmentation)

4. UVR, Context-aware MAR for DigiLog World (Woo)
3D Link with IoE (inc. IoT) between dual (real & virtual) spaces with
additional information
Augmentation of Context-of-Interest, not only sight but also
Everything (sound, haptics, smell, taste, etc.)!
Bidirectional Interaction/collaboration in dual spaces
SPACES
Private (3rd skin)
Social
General/Public
Real space
IoT/IoE, SNS
Seamless
Augmentation btw
dual spaces
How
to
Link
Seam
lessly? CoI
Ancho
r
MAR-
Content
Virtual space

5. Indoor MAR Applications [UVR Lab 2008-10]

6. Outdoor MAR Applications

7. Holistic Layers : In addition to 3D CG
.
.
.
.
.
.
IoT/IoE
Real/Physical
Virtual/Cyber
Information/
Knowledge
SPACE
TIME
Layers Social wisdom

8. Increased Interest and Demands on MAR
Popularized since the advent of “smartphone”
Increasing # of MAR Apps
However, Incompatible MAR Contents
Most AR contents are not re-usable in other applications
Due to different “processes, data structure/format, etc.”
In addition, differences in describing PoI & linking approach
So, The key of Context-aware MAR services is
Metadata representation for “Virtual, Real & Anchoring”
However, no standard “metadata” Format for MAR content

9. Now, What do you think …
What should we do for useful MAR services?
MAR
Services
…
DB B
DB CDB D
DB E

10. Metadata: Structured Data Fields with 5W1H
(Who, When, Where, What, How) + Why
Representation of Metadata Should be
Interoperable: platform independent
Ca: Semantic searching, filtering, and sharing of content
ASAP: be concise, compact, yet flexible, extensible to use
Scope: Metadata for refering
MAR content (image, text, video, audio, 3D model)
PoI: real entity (person/object/space/location) or content
MAR Anchor: Link between RC & VC:– contains the
instructions for MAR visualization

11. ISO/IEC CD 18039

12. External
Services
User profile
(Experience,
Knowledge,
Behavior)
Simulator
(Event, Space)
Renderer
MAR Execution Engine
Contextualized PoIs
Personal Device Display
User
Interface
MAR Scene
(Events, Targets, …)
Environment (or
wearable) Sensors
for Real World
Capturer
Internet of Thing
(IoT)
Sensors
Sensors
GPS
Compass
Gyro sensor
Personal Device
Embedded
Camera
Media
Assets
MAR Content
Virtual
Object
Physical
World
Open API
Physical World
User
User
Input
User
Command
Enhanced
Experience
MAR
Visualization
Recognition Data
Context Analyzer
PoI Analyzer
User Localization
View Point Setting
PoI Analyzing
Context Processor
Context Integration
Context Inference
Context Management
Recognized
PoIs
User Context
Sensor Data
PoI Recognition
Data
MAR
Anchor
Prope
rties
MAR Ontology
MAR
Entity
MAR
Entity Info.

13. Metadata Type Definition
Who.Creator String Person or group that have rights or custody on the object
Who.Owner String Person or group that create the object
When.Created Time DateTime The time when the object was created
When.Modified Time DateTime The time when the object was modified
Where.PoI Location GPS GPS coordinate of the PoI
Where.PoI Direction Float
Direction of front of the PoI measured by Compass. If the PoI i
s view-point invariant(i.e. symmetric) then the value is NULL
Where.PoI 3D
Coordinate
Float (x, y
(or, z))
Position of MAR Contents in MAR scene (Can be 2D or 3D)
Where.GPS Location
of Recognition Data
GPS
GPS coordinate of the Recognition Data when the data are
acquired
Where.Contents 3D
Coordinate
Float (x, y
(or, z))
Position of MAR Contents in MAR scene
Where.Contents 3D
Direction
Float (x, y, z)
3D normal vector of the Contents in MAR scene. Absolute
value of normal vector is 1.
Where.User Location GPS GPS coordinate of the user
Where.User Direction Float (x, y, z) 3D value of device pose measured by gyroscope of user device

14. Metadata Type Definition
What.PoI ID String Identification of the PoI
What.Recognition Data ID String Identification of the Recognition Data
What.Recognition Data FileFile Local data file path or URL for Recognition Data
What.Contents ID String Identification of the Contents
What.Contents type String Type of the Contents
What.Contents Reference String References of the Contents
What.Contents Data File File Local data file path or URL for Contents Data
How.Lens Intrinsic
Parameter of Acquisition
Device
Float (fx,fy,
cx,cy,k1,k2,k3,r)
Intrinsic parameter of the embedded lens of capturing devic
e. fx, fy are focal length of the lens. cx, cy are principal point
of the lens. (c.f. Skewness of the cell array is 0.) k1,k2,k3,r ar
e distortion parameters of lens acquired by camera calibratio
n. Those parameters can be converted with respect to camer
a model. (In this case, Pin-hole model)
How.Lens Intrinsic
Parameter of User Device
Float (fx,fy,
cx,cy,k1,k2,k3,r)
Intrinsic parameter of the embedded lens of user device
How.Contents Scale
Float (x, y
(or , z))
Size of the MAR contents in MAR scene. (dimension of the
scale depends on the content)

15. MAR Ontology
Devised to cover the
Relation-based data
model
Consists of correlated
MAR entities and
properties to describe
and Literals Value for
the primitive value
All object in MAR
Ontology has the URI
(Unified Resource ID)
instead of ID
Through the PoI, user
can start the exploring
the MAR ontology
MAR
Entity
Literals
Value
Literals
Value
MAR
Entity
MAR
Entity
Property
Property
MAR Ontology
Property
PoI
MAR
Contents
MAR
Entity Info.
Property
MAR
Contents
MAR
Entity Info.
MAR
Entity Info.
MAR
Anchor
MAR
Anchor
User’s start point

17. Korean Palace Augmentation
MAR Content
PoI
http:.//Geovid.org

18. Meta-tagging
WHO WHEN WHERE WHAT HOW
+Virtual
Palace.{Own
er, Creator}
+Virtual Palace.{Created Time,
Modified Time, Accessed Time,
Duration}
+Virtual Palace Info.{Begin
Time, End Time}
+Virtual
Palace
Information.{L
ocation}
+Virtual Palace
Information.{Type,
Description}
+Virtual Palace VO.{Type,
String, Data File}
+Virtual
Palace
Info.{Condi
tion state}
WHO WHEN WHERE WHAT HOW
+Palace
Anchor.{Ow
ner, Author}
+Palace
Anchor.{Authored
time, Modified Time}
+Palace Anchor.{PoI
Coord., Contents
Coord., Coord. Type}
+Palace Anchor.{ID,
Tag, PoI.ID, MAR
Contents.ID}
+Palace Anchor.{Scale,
Interaction, Animation,
Accessibility}
WHO WHEN WHERE WHAT HOW
+Palace
Recognition.
{Owner,
Creator}
+Palace
Recognition.{Create
d Time, Modified
Time}
+Palace PoI.{Location,
Front Direction}
+Palace
Recognition.{Personal
Device Location, PoI 3D
Position}
+Palace PoI.{ID, Name,
Tag, Rec. Data, MAR
Anchor.ID}
+Palace Recognition.{ID,
Data Set ID, Candidate Sub
Data}
+Palace
Recognition.{Sc
ale, Gyroscope,
Compass,
Intrinsic
Parameter}
2
1
3
http:.//Geovid.org

19. Implementation in KCTM
 As implementation of the MAR reference model with 360 video, we are developing the
MAR service in K-Culture Time Machine project that provide personalized MAR storytelling
of the historical site in Korea

20. Implementation in KCTM
 As implementation of the MAR reference model with 3D 360 video, we are developing the
MAR service in K-Culture Time Machine project that provide personalized MAR storytelling
of the historical site in Korea

21. Implementation in KCTM
 As implementation of the MAR reference model, we are developing the phone-based MAR
service in K-Culture Time Machine project that provide personalized MAR storytelling of
the historical site in Korea

22. 5W1H Metadata Schema
who MAR Contents.Essential.Who
when MAR Contents.Essential.When
where MAR Contents.Essential.Where
what
MAR Contents.Essential.What, MAR Contents.{Type, Reference, MAR
Anchor.ID}, VO.{Type, String, Data File}, Media Asset.{Type, Data File}
how MAR Contents.Essential.How
Anchor
MAR Entity:
Metadata
1
3 MAR Conte
nt: Metadat
a
5W1H Metadata Schema
who MAR Anchor.Essential.Who
when MAR Anchor.Essential.When
where
MAR Anchor.{Essential.Where, PoI Coordinate, Contents Coordinate,
Coordinate Type}
what MAR Anchor.{Essential.What, MAR Entity.URI, MAR Contents.ID}
how
MAR Anchor.{Essential.How, Scale, Interaction, Animation,
Accessibility}
MAR Anchor:
Metadata
2
Anchor
MAR
Entity
Virtual
Contents
Anchor
5W1H Metadata Schema
who {PoI, Recognition Data, MAR Entity Information}.Essential.Who
when
{PoI, Recognition Data, MAR Entity Information}.Essential.When, MAR Entity
Information.{Begin Time, End Time}
where
{PoI, Recognition Data, MAR Entity Information}.Essential.Where,
PoI.{Location, Front Direction}, Recognition Data.{PoI 3D Position, GPS
Location}, MAR Entity Information.{Location}
what
{PoI, Recognition Data, MAR Entity Information}.Essential.What,
PoI.{Recognition Data.ID, MAR Entity.URI}, Recognition Data.{Data Set ID,
Sub‐Data Set ID, Current Sub Data, Candidate Sub Data}, MAR Entity.{URI,
Label, MAR Entity Information.ID, PoI.ID, MAR Anchor.ID, Property.URI},
MAR Entity Information.{Type, Description}, Property.{URI, Label}, Literal
Value.{Type, Value}
how
{PoI, Recognition Data, MAR Entity Information}.Essential.How, Recognition
Data.{Scale}, MAR Entity Information.{Condition state}
PoI
MAR Entity
Information
Related
MAR
Entity
Property

23. 5W1H Metadata Schema
who ‐
when ‐
where GPS, Gyroscope, Compass
what ID
how Intrinsic Parameter
Essential:
Metadata
1 2 User:
Metadata
5W1H Metadata Schema
who Creator, Owner
when Created Time, Modified Time
where ‐
what ID, Name, Tag
how ‐
Essential
Metadata
Anchor
PoI
MAR
Contents
MAR
Anchor
Recognition
Data
MAR Entity
Information

24. MAR execution engine
Contextualized PoIs
Context Analyzer
PoI Analyzer
User Localization
View Point Setting
PoI Analyzing
Context Processor
Context Integration
Context Inference
Context Management
Recognized
PoIs
User profile
User
Context
Sensors
Sensor
Data
Recognition
Data
PoI Recognition
Data Contextualized
PoIs
Contextualized
PoIsPoIs

25. 
Simulator
(event, space)
Renderer
MAR Execution Engine
Contextualized PoIs
Context Analyzer
External
Services
Open API
Personal
Device
Display
MAR
Visualization
User
Command
MAR Contents
MAR
Anchor
MAR
Ontology
MAR
Entity
MAR
Entity
Info.팔작
지붕
인정전_PoI1
Contents Retrieval
Click
MAR
Entity
Info
MAR
Contents

26. 
Simulator
(event, space)
Renderer
MAR Execution Engine
Contextualized PoIs
Context Analyzer
External
Services
Open API
Personal
Device
Display
MAR
Visualization
User
Command
MAR Contents
MAR
Anchor
MAR
Ontology
MAR
Entity
MAR
Entity
Info.이화
문양
인정전_PoI2
Contents Retrieval
Click
MAR
Entity
Info
MAR
Contents

27. Contents Retrieval

Simulator
(event, space)
Renderer
MAR Execution Engine
Contextualized PoIs
Context Analyzer
External
Services
Open API
Personal
Device
Display
MAR
Visualization
User
Command
MAR Contents
MAR
Anchor
MAR
Ontology
MAR
Entity
MAR
Entity
Info.
희정당
희정당_PoI1
Click
MAR
Entity
Info
MAR
Contents

28. Now, Metadata for MAR Services…
Is it useful or worthy of standard?
Finally, I prepared for NWIP…
MAR
Services
…
DB B
DB CDB D
DB E

29. “The future is already here. It is just not
uniformly distributed” -William Gibson (SF writer)
More Information
Woontack Woo, Ph.D.
Mail: wwoo@kaist.ac.kr
Web: http://uvr.kaist.ac.kr