The Convergence of Robotics, the Web, and the IoT

The Convergence of Robotics, the Web, and the
Internet of Things
Michael McCool, Principal Engineer, Intel
IoT Dev Fest, June 4, 2017

Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Abstract
What do the web, the IoT, and robotics have in common? For one, they can all benefit from the
network effect, where the whole is greater than the sum of the parts. For the Web, the network allows
distribution of information between agents anywhere in the world. For the IoT, the focus is on the
remote monitoring and coordination of physical sensors and actuators, and the analysis of data to
provide actionable insight. Robots can be defined as autonomous cyberphysical systems that can
sense and manipulate their environment. Although not essential to the definition, robots are often
connected to the internet as well, and can use it to enhance or enable their function in various ways:
supporting, for example, telepresence on the web, remote sensing integrated with the IoT, or
coordination between multiple robots and IoT devices, or just to offload computation. In the past
these three ecosystems had their own software frameworks and standards but there are indications
they are converging: IoT systems often integrate with cloud services that have web APIs. Web API
standards are being used to define the data models for IoT devices. Machine learning models can be
trained with data from the Web and used for real-time perception in robots. IoT gateways can be used
to command robots by voice. Robots can serve their own graphical user interfaces over the Web.
Robots are adopting service-oriented distributed software architectures and programming models
originally developed for web servers. IoT devices and radio networks can be used a navigation
beacons and even raw sensory data for robots. In this talk, I will talk about a number of these
convergences and what the implications are.
2

Outline and Agenda
• Ecosystem Overview
• Web, IoT, and Robotics
• Requirements and technologies
• Commonalities and differences
• Trends and Technologies
• Embedded HPC and AI
• Remote compute offload
• Real-time (sub)systems
• Semantic data modelling
• Emerging Standards
• OSRF*/ROS2* and OMG*/DDS*
• OCF* and W3C* Web of Things
• OpenFog*
3
Web
IoT Robotics

The World-Wide Web
• “Distributed information
space with resources
identified by URLs”
• Hyperlinked documents
• Delivered by Internet
• Displayed in browsers
• Many, many extensions…
• Search engines
• Web service RESTful APIs
• Applications as content
• Semantic markup
4
https://en.wikipedia.org/wiki/World_Wide_Web
This Wikipedia and Wikimedia Commons image is from the user Chris 73 and is freely available at
//commons.wikimedia.org/wiki/File:WorldWideWebAroundWikipedia.png under the
creative commons cc-by-sa 3.0 license.

Technologies Engendered by the Web
• Web Services
• RESTful: Set of properties that allows an online service to scale gracefully
• Stateless interface, small set of verbs, structured data
• Metadata: RAML, Swagger/OpenAPI
• Cloud Services: X-aaS, SD-X, virtualization…
• “Software applications as content”
• Containers and Microservices
• Docker, Snaps, etc.
• Compose services out of loose collections of other services
• Flexible structured data formats (and now efficient binary encodings…)
• JSON  CBOR; metadata: JSON-Schema
• XML  EXI; metadata: XSI
5

Semantic Web
To enable Web 3.0:
 Allow the meaning (semantics) of data to be
understood
Technologies to facilitate data
interchange:
 RDF: Resource Description Framework
 OWL: Web Ontology Language
 SPARQL: Protocol and RDF Query Language
 Syntaxes: XML, N3, JSON-LD, etc.
 Ontologies (Vocabularies): Haystack,
Schema.org, etc.
6/37

RDF: Resource Description Framework
7/37
 General approach for describing
data as sets of “triples”
 Subject-Predicate-Object
 RDF triples are stored in a
distributed fashion
 Federated database
 Subsumes relational databases
 Standard terms and relations are
defined in ontologies
(“vocabularies”), including RDF and
OWL
Socrate
s
Man
Is-a
Mortal
IsIs

Internet of Things
Network of devices, each capable
of communication and associated
with a set of sensors and/or
actuators.
• Enable numerous applications
• Multiple verticals: home,
industrial, city, building, vehicle,
etc.
• Typically rely on backend
systems for data analytics and
services (eg web services)
• Edge + “infrastructure” nodes…
8
https://commons.wikimedia.org/wiki/File:Nest_Learning_Thermostat_(cropped).JPG

Optimization Notice
Internet of Things (IoT) Systems
Gateway
Gateway
Analysis
(Server)
Human
Interface
Wi-Fi*
LTE
Wired
Endpoint
Endpoint

Robots
• Cyberphysical systems with the capability
to autonomously interact with their
environments
• Ideally capable of autonomous
• Perception
• Planning
• Mobility
• Manipulation
• Human-Robot Communication
• Note: typical “Industrial Robots” may only
be capable of a subset of these
10

Typical Robot Software Stack
Servos
Wheel
Motors
RGB
Camera
Depth
Camera
Motion Touch Mic Array
Discrete SensorsCamera Drivers Motor Drivers
SensorFusion
Point Cloud
2DMap
Body Tracking
Face Tracking
Object Recognition
RGBD+IMU / SLAM
Speech Recognition
Behavior
Control
Reflex
Goals
Interact
Speech
Output
Dialog
Control
User/ World
Knowledge
Gesture /
Action
World
View
Robot
State
Command
Inputs
Behavior
Arbitrator
Path
Planning
Movement
External
Device
Control
Voice
Command
External
Controller
Gesture Input
Environment
Stimulus
Behavior
Behavior
Behaviors
Applications
Acoustics
OptionalInputs
Cloud
offload &
control
IR
Ultra-
Sonic
Sense Plan Act
Sensor Hardware and Drivers Actuators

• Sense a potentially dynamic
environment
• Determine current location
• Plan path and then navigate
from one location to another
• Avoid obstacles
• Interact with humans as
needed (avoid, follow,
respond to commands, etc.)
Mobile Robotics
12

• Speech recognition
• Face recognition
• Body pose understanding
• Object recognition
• Dialogue management
• Context awareness
• Emotional state recognition
• Social relationship
understanding
Social Robotics
13

14

Robotics and the IoT
Robots are…
Cyberphysical systems with the capability to autonomously
interact with their environments
 Not just passive sensors
If connected to the internet, robots may also participate in the
IoT ecosystem
 Share experiences via the network
 Use sensor data from other IoT devices
 Offload work to external systems
 Access large amounts of data for training data
 Coordinate motion across multiple robots/devices
Next-generation connected robots are explicitly
leveraging IoT technologies and middleware
But they also need to be autonomous
15

Comparative Requirements
16
Web IoT Robots
Scalability
Connectivity
Scalability
Low cost
Low power
Functional safety
Connectivity
Functional safety
High performance
Real time
Reliability
Availability
Flexibility
Security
Reliability
Availability
Flexibility
Security
Reliability
Availability
Flexibility
Security

Emerging Trends and Technologies
Artificial Intelligence
• Machine Learning
• Perception
• Semantic Processing
Embedded HPC (especially for Robotics)
• Computer Vision and Speech
• Localization and Mapping
Distributed computing
• Remote (cloud, fog) compute offload
• Service-oriented computing
Real-time (sub)systems
• Control systems
• Functional safety
17

Case Studies
ROS*: Robot Operating System
• Microservices architecture, Web integration, ROS2, OMG’s DDS,
embedded perception
OCF*: IoT standardization effort
• Microservices architecture using CoAP/CBOR, IoTivity
implementation of Smart Home example system
W3C* Web of Things: IoT standardization effort
• Standardized metadata for microservices architecture
OpenFog*: Edge computing standardization effort
• Infrastructure support for microservices architecture
18

ROS*: Robot Operating System
• Supports distributed system
based on a graph of
communicating nodes
• Both pub-sub and request-
response patterns
• Large community with many
existing nodes for sensors,
planning, navigation, etc.
• Various visualization tools
and simulators are also
provided

Autonomous Navigation ROS* Node Graph

Rviz: Robot Visualization

Robot Web Tools* and ROSWeb*
http://robotwebtools.org/ and https://github.com/EESC-LabRoM/rosweb
22
ROSWeb Robot Manager
• Manage/control ROS
robot via web console
• Uses rosbridge, which
bridges ROS to a JSON-
over-Websocket
protocol
• Javascript library can
then access ROS
functionality either from
Node.js or a browser
• Can also use WebGL,
TLS, D3S, Google maps,
etc.

ROS2*
Designed to resolve a number of issues with current ROS architecture:
• Real-time support
• Quality of service control in communications
• Distributed node management
• Lifecycle management
• Determinism
 Built on top of OMG’s DDS standards, a middleware framework for
industrial IoT systems, that provides real-time pub-sub
https://github.com/ros2/ros2/wiki and http://www.omg.org/spec/DDS/

SAWR Project
Simple Autonomous Wheeled Robot
• Demonstration system for autonomous
robotics
• Can be built using either
• Intel® RealSense™ Robotic Development Kit,
which combines Aaeon’s UP* Board and an
Intel® RealSense R200 camera
• Intel® Joule™ compute module and an Intel®
RealSense ZR300 camera
• OSH design: laser-cut frame/BOM
https://github.com/01org/sawr

Robotis* Turtlebot 3
25
Modular Robotics Research
and Training Platform
• Intel® Joule™ compute
module
• Intel® RealSense R200
camera
• Laser triangulation 360
degree rangefinder
• Real-time Arduino-based
subsystem
http://turtlebot3.robotis.com/en/latest/hardware.html

Intel Confidential
Intel® RealSense™ SDK: Perceptual Capabilities
Objects
Database
Library of pre-
trained
or customer
specific objects
Object
Tracking
Track objects of
interest, even when
robot and/or object
are moving
Object
Localization
Determine
where objects
are in the scene
Object
Segmentation
Pixels that are part of
the object for
manipulation or
removal

Person Interaction

OCF* & IoTivity*
Specification
Open Source Coordination
Business (Marketing, Strategy, Membership)
Open Source Project
Reference Implementation of OCF
(and Non-Spec Related Code)
Sponsored (funded) by OCF
Separate Governance
Coordination
April
2017
Open
Connectivit
y
Foundation
Public
29
Certification
Data Modelling
Specs & Open Source ready simultaneously

OCF – Conceptual Framework
30 April
2017
Core
Framework
Profiles
Consumer Enterprise Industrial Auto Education Health
Security, Identity & Permissions
Discovery
Data
Transmission
Data
Management
Device
Management
Transports
(Smart)
Remote
Access Cloud
Resource Model
Interaction / Data Model
Open
Connectivit
y
Foundation
Public

OCF – Resource Model
31 April
2017
Resource URI
rt: Resource Type
if: Resource Interface
prop: Policy
n: Resource Name
…
…
Common
Properties
Resource
Specific
Identifies the type of resource
List of interfaces associated with
the resource
Policy associated with resource:
discoverable, observable, secure,
etc
Friendly name
§ The resource model, coupled with the common data model, provides the base interoperability of OCF
§ Any physical or software artifact on a device that needs to be manipulated or made visible across the
network can be described via the resource model
§ A resource has a URI and a collection of Properties
links: Other Resource URI
Resources can include links to
other resources
Open
Connectivit
y
Foundation
Public

Resource Model - examples
32 April
2017
14
Resource URI :/a/light1
rt: oic.ex.light
if: oic.if.rw
prop: discoverable,
observable
n: myHallWayLight
State: 0 (OFF)
Dim Level: 0
Resource URI :/a/fan1
rt: oic.ex.fan
if: oic.if.rw
prop: discoverable
n: myKitchenFan
State: 1 (ON)
Speed: 10
Open
Connectivit
y
Foundation
Public

CREATE
 Create a new Resource on the Server
RETRIEVE
 Get the current State or representation of a Resource from a Server
UPDATE
 Request a partial or complete update to the information stored in a Resource
DELETE
 Remove a Resource from the Server
NOTIFY
 Request asynchronous notification of state changes in a Resource
 Special-cased to support OBSERVE for publish/subscribe support
Common Interaction Model – Transport Agnostic
April 201733
Open
Conne
ctivity
Found
ation
Public

Let’s look deeper at a Resource …
34
Server - DevA
/dimming
/oic/d
/oic/res
Main Living Room Light
/switch
/oic/d
/oic/p
oic.r.switch.binary
oic.wk.d
oic.wk.p
false100
/switch true
/dimming oic.r.light.brightness
/oic/p LED-Light-Co, 40W Equivalent Bulb

There can be more Properties …
35
/oic/p
mnml
mnmn
pi
http://ledlight.com/gg3-40
Led-Light-Co
GoGlow III - 40W
mnmo GG3-40

Resource Type - Data Modeling
36
JSON Schema*
• Describes the Properties and
Representation
ReSTful API Modelling Language
(RAML)*
• Describes the Request and Response
*JSON Schema and RAML are normative and also can be used for code generation
"oic.r.switch.binary": {
"type": "object",
"properties": {
"value": {
"type": "boolean",
"description": "Status of the switch"
}
}
}
get:
responses :
200:
body:
application/json:
schema: |
{
}
Defined by:
(see OIC_Resource_Type_Specification)

OCF/IoTivity Smart Home Demo
https://github.com/01org/SmartHome-Demo
37
Prototype Smart Home
• Example architecture
for Smart Home
• Based on OSS IoTivity
implementation of OCF
• Combines cloud
services, gateways, and
sensor nodes
• Technologies: Docker
containers, Node.js,
Zephyr RTOS, HTTPS,
CoAP, WebGL, RAML,
JSON/CBOR

W3C Web of Things
https://www.w3.org/WoT/ and https://github.com/w3c/wot
38
Thing
API A
Applications
Thing
API B
Applications
Interaction Model Interaction Model
Protocol Mappings Protocol Mappings
CoAP
OneM2M
CoAP
OCF
CoAP
BACnetBridging Gateway
Thing
Description
HTTP
W3C Working Group
• Standardize
descriptions of data
and interaction
models for IoT
devices (“Things”)
• Support semantic
interoperability via
linked data, JSON-
LD, RDF, IoT
ontologies
• Bridge multiple other
standards

https://www.openfogconsortium.org/
39
Description
• Reference architecture
for distributed “fog”
computation
• Fog: cloud computing,
but supported
throughout the
network, both in data
centers and at the edge

Summary and Conclusions
• Web, IoT, and Robotics ecosystems to interoperate, but not merge
• Several key differences mean completely shared technology base is not feasible
• However, many core components can and should be shared
• Many key “IoT” services will depend on these systems interoperating
• New opportunities and challenges
• Moving cloud services to the edge:
• Embedded microservice architectures, fog computing, containerization
• Using web technologies and infrastructure in IoT and Robotics:
• RESTful APIs, cloud services, semantic interoperability
• Software development challenges:
• Data-centric, distributed computing, microservices, scaling, security
40

Legal Disclaimer & Optimization Notice
INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR
OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO
LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS
INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE,
MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
Software and workloads used in performance tests may have been optimized for performance only on Intel
microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer
systems, components, software, operations and functions. Any change to any of those factors may cause the results
to vary. You should consult other information and performance tests to assist you in fully evaluating your
contemplated purchases, including the performance of that product when combined with other products.
Copyright © 2014, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel
logo are trademarks of Intel Corporation in the U.S. and other countries.
Optimization Notice
Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel
microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the
availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent
optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture
are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the
specific instruction sets covered by this notice.
Notice revision #20110804
41

The Convergence of Robotics, the Web, and the IoT

The Convergence of Robotics, the Web, and the IoT

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to The Convergence of Robotics, the Web, and the IoT

Similar to The Convergence of Robotics, the Web, and the IoT (20)

More from Intel® Software

More from Intel® Software (20)

Recently uploaded

Recently uploaded (20)

The Convergence of Robotics, the Web, and the IoT