The document discusses an IETF hackathon focused on NUBOMEDIA APIs/SDKs and PaaS. It provides an introduction to NUBOMEDIA, describes the various NUBOMEDIA APIs including the Media, Repository, Signaling, Room, and Tree APIs. It also discusses the NUBOMEDIA SDKs and PaaS, including how to deploy applications using the NUBOMEDIA PaaS GUI. The document concludes by outlining steps to develop a sample application using NUBOMEDIA APIs and deploying it on the NUBOMEDIA PaaS.
2. Table of contents
1. Introduction
2. NUBOMEDIA APIs/SDKs
3. NUBOMEDIA PaaS
4. Let's get to work
3. Table of contents
1. Introduction
− Review
− What is NUBOMEDIA?
− References
1. NUBOMEDIA APIs/SDKs
2. NUBOMEDIA PaaS
3. Let's get to work
4. 1. Introduction
Review
−Kurento is a media server and a set of APIs
aimed to create applications with advance media
capabilities
−The building blocks for applications are named
media elements, chained into media pipelines
5. 1. Introduction
What is NUBOMEDIA?
−NUBOMEDIA is an open source PaaS
(Platform as a Service) based on Kurento
−NUBOMEDIA exposes to developers the
ability of deploying and leveraging
applications with media capabilities:
• WebRTC, media recording, group
communications, computer vision, augmented
reality…
7. 1. Introduction
What is NUBOMEDIA?
−From the developer’s perspective,
NUBOMEDIA capabilities are accessed
through a set of APIs/SDKs
−NUBOMEDIA applications can be
deployed using the NUBOMEDIA PaaS
Manager
We are going to see this features
in detail in this presentation
9. Table of contents
1. Introduction
2. NUBOMEDIA APIs/SDKs
− Development model
− NUBOMEDIA APIs
− NUBOMEDIA SDKs
1. NUBOMEDIA PaaS
2. Let's get to work
10. 2. NUBOMEDIA APIs/SDKs
Development model
−The development model in NUBOMEDIA is the
same than in Kurento
• Three-tier model (inspired in the Web)
Client
Application
Server
Media Server
NUBOMEDIA PaaS
11. 2. NUBOMEDIA APIs/SDKs
Development model
−Like every application with media capabilities, it is
important to distinguish between the media and
signaling plane
Client
Application
Server
Media Server
NUBOMEDIA PaaS
signaling
media
13. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs
API Description
Media API Enables developers consuming the Media Server capabilities among which
we can find media transport, media archiving, media processing, media
transcoding, and so on
WebRtcPeer API Abstracts the client WebRTC media capabilities, exposing the media
capture and communication capabilities of a browser in a seamless way
Repository API Makes possible to access an elastic scalable media repository for archiving
media information and meta-information
Signaling API Provides a simple signaling mechanism based on JSON-RPCs for
applications
Room API Enables application developers functionalities to create group
communication applications adapted to real social interactions
Tree API Allows developers to build video broadcasting web applications
14. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Media API
−Media API allows to Java developers consume
the media services provided by Kurento Media
Server (KMS)
−Concepts:
• Media Element
• Media Pipeline
15. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Media API
−KMS instances are provided elastically by NUBOMEDIA
• The number of available KMS instances depends on the PaaS
Manager configuration (next section)
−Each KMS has a total amount of available points to create
Media Pipelines and Media Elements
• The total points depends on the number of VCPUs of the KMS
• The type of the instance can be selected on the PaaS Manager
configuration (next section)
Instance type # VCPUs KMS points
Medium 2 200
Large 4 400
16. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Media API
−Each KMS is controlled by an instance of KurentoClient
−With each media session an instance of KurentoClient
should be created
−The number of available points per KMS decreases with
each Media Element creation (scaling in/out)
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-client</artifactId>
</dependency>
<dependency>
<groupId>de.fhg.fokus.nubomedia</groupId>
<artifactId>nubomedia-media-client</artifactId>
</dependency>
Dependencies
(Maven)
KurentoClient kurentoClient = KurentoClient.create();
17. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Media API
−Example: nubomedia-magic-mirror
// One KurentoClient instance per session
KurentoClient kurentoClient = KurentoClient.create();
// Media logic (pipeline and media elements connectivity)
MediaPipeline mediaPipeline = kurentoClient.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
FaceOverlayFilter faceOverlayFilter = new
FaceOverlayFilter.Builder(mediaPipeline).build();
faceOverlayFilter.setOverlayedImage("http://files.kurento.org/img/mario-wings.png",
-0.35F,
-1.2F, 1.6F, 1.6F);
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
18. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Repository API
−NUBOMEDIA provides access to a Kurento Repository in
order to store and recover multimedia streams (and
metadata)
−The repository is controlled by an instance of
RepositoryClient (Java)
<dependency>
<groupId>de.fhg.fokus.nubomedia</groupId>
<artifactId>nubomedia-media-client</artifactId>
</dependency>
<dependency>
<groupId>de.fhg.fokus.nubomedia</groupId>
<artifactId>nubomedia-repository-
client</artifactId>
</dependency>
Dependencies
(Maven)
RepositoryClient repositoryClient =
RepositoryClientProvider.create();
19. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Repository API
−Example: nubomedia-repository-tutorial
The first step is to carry out the
WebRTC loopback communication.
We can see the live media while it is
being recorded in the repository.
After that, we are able to reproduce
the recording, reading the media
stored in the repository.
20. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Repository API
−Example: nubomedia-repository-tutorial
// KurentoClient
kurentoClient = KurentoClient.create();
// Media pipeline
mediaPipeline = kurentoClient.createMediaPipeline();
// Repository item (recorder)
Map<String, String> metadata = Collections.emptyMap();
repositoryItemRecorder = repositoryClient.createRepositoryItem(metadata);
// Media elements and connectivity
webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
recorderEndpoint = new RecorderEndpoint.Builder(mediaPipeline,
repositoryItemRecorder.getUrl()).withMediaProfile(MediaProfileSpecType.WEBM).build();
webRtcEndpoint.connect(webRtcEndpoint);
webRtcEndpoint.connect(recorderEndpoint);
// WebRTC negotiation
String sdpAnswer = performWebRtcNegotiation(session, sdpOffer);
// Start recording
recorderEndpoint.record();
21. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Repository API
−Example: nubomedia-repository-tutorial
// KurentoClient
kurentoClient = KurentoClient.create();
// Media pipeline
mediaPipeline = kurentoClient.createMediaPipeline();
// Repository item (player)
RepositoryItemPlayer repositoryItemPlayer =
repositoryClient.getReadEndpoint(repositoryItemRecorder.getId());
// Media elements and connectivity
webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
playerEndpoint = new PlayerEndpoint.Builder(mediaPipeline,
repositoryItemPlayer.getUrl())
.build();
playerEndpoint.connect(webRtcEndpoint);
// WebRTC negotiation
String sdpAnswer = performWebRtcNegotiation(session, sdpOffer);
// Start playing
playerEndpoint.play();
22. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - WebRtcPeer API
−JavaScript API that abstracts the client RTC media (Media
Capture and PeerConnection)
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-utils-js</artifactId>
</dependency>
Maven dependency
<script src="js/kurento-utils.js"></script>
var options = {
localVideo : videoInput,
remoteVideo : videoOutput,
onicecandidate : onIceCandidate
}
webRtcPeer = new
kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
function(error) {
if (error) {
return console.error(error);
}
webRtcPeer.generateOffer(onOffer);
});
}
JavaScript library
Example of use
(send-receive)
23. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Signaling API
−NUBOMEDIA capabilities can be accessed through any
kind of signaling protocol (e.g. SIP, XMPP or REST)
−NUBOMEDIA provides a very signaling protocol based on
JSON-RPCs over WebSockets
−This API has several components
• Server signaling API
• JavaScript client signaling API
• Android client signaling API
• iOS client signaling API
http://nubomedia.readthedocs.io/en/latest/api/signaling/
24. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Room API
−The Room API is a high-level communications library that
provides capabilities for managing multi-conference
WebRTC sessions. It has the following components:
• Room Server: a container-based implementation of the server,
uses JSON-RPC over WebSockets for communications with the
clients
• Room JavaScript Client: module implementing a Room client for
Web applications
• Room Client: a client library for Java web applications or Android
clients
28. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA APIs - Tree API
−The NUBOMEDIA Tree API allows developers to build
WebRTC broadcasting applications
−This API is composed by a server and two clients:
• tree-server is a component of the API designed to be deployed
and controlled by clients
• tree-client implements a tree client designed to be used in Java
web applications or Android applications. The client contains an
implementation of JSON-RPC WebSocket protocol implemented
by Tree server
• tree-client-js implements a tree client to be used in Single Page
Applications (SPA)
29. 2. NUBOMEDIA APIs/SDKs
NUBOMEDIA SDKs
−More info on:
• http://webrtcpeer-android.readthedocs.io/
• http://jsonrpc-ws-android.readthedocs.io/
• http://kurento-room-client-android.readthedocs.io/
• http://kurento-ios.readthedocs.io/
SDK Description
Android SDK Android version of the client-side NUBOMEDIA APIs: WebRtcPeer,
Signaling, Room, Tree
iOS SDK Complete SDK to provide NUBOMEDIA client capabilities to iOS
devices
30. Table of contents
1. Introduction
2. NUBOMEDIA APIs/SDKs
3. NUBOMEDIA PaaS
− Introduction
− PaaS GUI
1. Let's get to work
31. 3. NUBOMEDIA PaaS
Introduction
−The NUBOMEDIA PaaS manager is a tool aimed to
control the way in which the NUBOMEDIA applications are
built and deployed inside the NUBOMEDIA PaaS
−The capabilities provided by the Paas Manager can be
used by developers using the PaaS GUI:
• The PaaS Manager GUI is a web application that allows to use
the NUBOMEDIA PaaS Manager
32. 3. NUBOMEDIA PaaS
Introduction
−Internally, the NUBOMEDIA PaaS uses Docker
containers to deploy applications
−Therefore it is a requirement to include a Dockerfile in
GitHub repository to be deployed on NUBOMEDIA
−Example: FROM nubomedia/apps-baseimage:src
MAINTAINER Nubomedia
ADD keystore.jks /
ADD . /home/nubomedia
RUN sudo chown -R nubomedia /home/nubomedia
RUN cd /home/nubomedia && mvn compile
ENTRYPOINT cd /home/nubomedia && mvn exec:java
https://docs.docker.com/engine/reference/builder/
33. 3. NUBOMEDIA PaaS
PaaS GUI
−Web application to manage NUBOMEDIA applications
http://paas-manager.nubomedia.eu:8081/
Credentials
needed to login
Home shows a
summary of the
apps currently
deployed
34. 3. NUBOMEDIA PaaS
PaaS GUI
−A NUBOMEDIA application can be deployed using the
PaaS GUI
−It is done providing the GitHub repository URL and a set
of configuration parameters
35. 3. NUBOMEDIA PaaS
PaaS GUI
−Most important configuration values:
−For the rest of parameters take a look to the doc:
http://nubomedia.readthedocs.io/en/latest/paas/paas-gui/
Number of
KMSs
KMS host type:
-Medium = 2 VCPUs (200 points)
-Large = 4 VCPUs (400 points)
36. Table of contents
1. Introduction
2. NUBOMEDIA APIs/SDKs
3. NUBOMEDIA PaaS
4. Let's get to work
− Introduction
− Server-side
− Client-side
− Deployment
37. 4. Let's get to work
Introduction
−Devil is in the details, and so, we are going to see a
complete NUBOMEDIA step by step
−This applications the nubomedia-magic-mirror tutorial
https://github.com/nubomedia/nubomedia-magic-mirror/
http://nubomedia.readthedocs.io/en/latest/tutorial/nubomedia-magic-mirror/
38. 4. Let's get to work
Server-side
−We recommend Spring-Boot as the base technology for
NUBOMEDIA applications
• Spring-Boot embeds a Tomcat server in a simple seamless way
for developers
• The application is packaged as a runnable JAR, and when this
JAR is executed, the Tomcat server is started and the web
application automatically deployed
−We recommend Maven as for managing the life-cycle and
managing the dependencies of our applications
39. 4. Let's get to work
Server-side
−The first step is to clone GitHub repository
−We recommend use an IDE to develop applications. For
example, Eclipse (importing app as Maven project)
git clone https://github.com/nubomedia/nubomedia-magic-mirror
40. 4. Let's get to work
Server-side
−First step is understand the pom.xml file:
<parent>
<groupId>org.kurento</groupId>
<artifactId>kurento-parent-pom</artifactId>
<version>6.5.0</version>
</parent>
<properties>
<!-- Nubomedia -->
<nubomedia-media-client.version>1.0.2</nubomedia-media-client.version>
<!-- Main class -->
<start-class>eu.nubomedia.tutorial.magicmirror.MagicMirrorApp</start-class>
</properties>
Inherit from kurento-parent-pom
makes simpler our pom.xml by
reusing versions defined in the parent
It’s important
declare the
fully qualified
name of the
main class
41. 4. Let's get to work
Server-side
<dependencies>
<!-- Spring -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-websocket</artifactId>
</dependency>
<!-- Kurento -->
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-client</artifactId>
</dependency>
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-utils-js</artifactId>
</dependency>
<!-- Nubomedia -->
<dependency>
<groupId>de.fhg.fokus.nubomedia</groupId>
<artifactId>nubomedia-media-client</artifactId>
<version>${nubomedia-media-
client.version}</version>
</dependency>
</dependencies>
…
…
Dependencies clause is one of
the most important. Notice that
our app depends on Spring,
Kurento, and NUBOMEDIA
42. 4. Let's get to work
Server-side
−We need to define also the way
in which signaling is going to
work in our application
−For the shake of simplicity, we
use a raw WebSocket between
the server and client side
−We need to define the messages
exchanged to carry out the
communication about client and
server
clientserver
start
startResponse
iceCandidate
onIceCandidate
stop
error
notEnoughResources
Starts a media
session
ICE candidates
(WebRTC
negotiation)
Stops a media
session
Error cases
43. 4. Let's get to work
Server-side
−The class diagram of our application is the following:
MagicMirrorApp
MagicMirrorHandler
UserSession
Main class
Message handler
(signaling)
User session
(session id,
media logic)
44. 4. Let's get to work
Server-side
@SpringBootApplication
@EnableWebSocket
public class MagicMirrorApp implements WebSocketConfigurer {
@Bean
public MagicMirrorHandler handler() {
return new MagicMirrorHandler();
}
@Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(handler(), "/magicmirror");
}
public static void main(String[] args) throws Exception {
new SpringApplication(MagicMirrorApp.class).run(args);
}
}
MagicMirrorApp defines the
Spring-Boot application and
the WebSocket used for
signaling ("/magicmirror",
managed by
MagicMirrorHandler)
45. 4. Let's get to work
Server-side
public class MagicMirrorHandler extends TextWebSocketHandler {
private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<>();
@Override
public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
JsonObject jsonMessage = new GsonBuilder().create().fromJson(message.getPayload(),
JsonObject.class);
switch (jsonMessage.get("id").getAsString()) {
case "start":
start(session, jsonMessage);
break;
case "stop":
release(session);
break;
case "onIceCandidate":
onIceCandidate(session, jsonMessage);
break;
default:
error(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
break;
}
}
MagicMirrorHandler
manages signaling
messages. It keeps a
collection of user sessions
(thread-safe map users)
46. 4. Let's get to work
Server-side
public class UserSession {
public String startSession(final WebSocketSession session, String sdpOffer) {
// One KurentoClient instance per session
kurentoClient = KurentoClient.create();
log.info("Created kurentoClient (session {})", sessionId);
// Media logic (pipeline and media elements connectivity)
mediaPipeline = kurentoClient.createMediaPipeline();
log.info("Created Media Pipeline {} (session {})", mediaPipeline.getId(), sessionId);
webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(mediaPipeline).build();
faceOverlayFilter.setOverlayedImage("http://files.kurento.org/img/mario-wings.png", -0.35F,
-1.2F, 1.6F, 1.6F);
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
// WebRTC negotiation
String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);
webRtcEndpoint.gatherCandidates();
return sdpAnswer;
}
UserSession
handles media logic
(media pipeline,
media elements,
WebRTC
negotiation…)
47. 4. Let's get to work
Server-side
−As of Chrome 47, access to user media can only be done
by secure apps (HTTPS), and so, it is needed a Java
keystore
server.port: 8443
server.ssl.key-store: keystore.jks
server.ssl.key-store-password: kurento
server.ssl.keyStoreType: JKS
server.ssl.keyAlias: kurento-selfsigned
application.properties
http://doc-kurento.readthedocs.io/en/stable/mastering/securing-kurento-applications.html
48. 4. Let's get to work
Client-side
−Client-side dependencies are handled by Bower
−Our bower.json file contains:
−In addition the file .bowerrc configures the target path for
the JavaScript libraries
"dependencies": {
"bootstrap": "~3.3.0",
"ekko-lightbox": "~3.1.4",
"adapter.js": "v0.2.9",
"demo-console": "1.5.1"
}
{
"directory" : "static/bower_components"
}
49. 4. Let's get to work
Client-side
−The app has been implemented as a SPA (single page
application) which contains a single HTML
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" href="bower_components/bootstrap/dist/css/bootstrap.min.css">
<link rel="stylesheet" href="bower_components/ekko-lightbox/dist/ekko-lightbox.min.css">
<link rel="stylesheet" href="bower_components/demo-console/index.css">
<link rel="stylesheet" href="css/styles.css">
<script src="bower_components/jquery/dist/jquery.min.js"></script>
<script src="bower_components/bootstrap/dist/js/bootstrap.min.js"></script>
<script src="bower_components/ekko-lightbox/dist/ekko-lightbox.min.js"></script>
<script src="bower_components/adapter.js/adapter.js"></script>
<script src="bower_components/demo-console/index.js"></script>
<script src="js/kurento-utils.js"></script>
<script src="js/index.js"></script>
<title>NUBOMEDIA Tutorial: WebRTC Magic Mirror</title>
</head>
JavaScript/CSS are
linked here. Using
Bootstrap to provide
responsive design
50. 4. Let's get to work
Client-side
−A JavaScript file contains the client-side logic (message
handling and WebRtcPeer use)
var ws = new WebSocket('wss://' + location.host + '/magicmirror');
ws.onmessage = function(message) {
var parsedMessage = JSON.parse(message.data);
switch (parsedMessage.id) {
case 'startResponse':
startResponse(parsedMessage);
break;
case 'error':
onError("Error message from server: " + parsedMessage.message);
stop(false);
break;
case 'iceCandidate':
webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
if (error) return console.error("Error adding candidate: " + error);
});
break;
// ...
}
}
51. 4. Let's get to work
Client-side
−A JavaScript file contains the client-side logic (message
handling and WebRtcPeer use)
var webRtcPeer;
function start() {
var options = {
localVideo : videoInput,
remoteVideo : videoOutput,
onicecandidate : onIceCandidate
}
webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
if (error) {
return console.error(error);
}
webRtcPeer.generateOffer(onOffer);
});
}
52. 4. Let's get to work
Deployment
−We need a Dockerfile to deploy our app
FROM nubomedia/apps-baseimage:src
MAINTAINER Nubomedia
ADD keystore.jks /
ADD . /home/nubomedia
RUN sudo chown -R nubomedia /home/nubomedia
RUN cd /home/nubomedia && mvn compile
ENTRYPOINT cd /home/nubomedia && mvn exec:java
53. 4. Let's get to work
Deployment
−Finally we can deploy our app in the NUBOMEDIA PaaS
Manager:
54. 4. Let's get to work
Deployment
−The app changes its state, from CREATED to RUNNING (it
takes a couple of minutes to finish):
55. 4. Let's get to work
Deployment
−We can trace two types of logs: build and app
I0708 12:51:02.335080 1 builder.go:41] $BUILD env
var is {"kind":"Build","apiVersion":"v1","metadata":{...}
I0708 12:51:02.423366 1 source.go:96] git ls-remote
https://github.com/nubomedia/nubomedia-magic-mirror
--heads
I0708 12:51:02.423547 1 repository.go:275]
Executing git ls-remote
https://github.com/nubomedia/nubomedia-magic-mirror
--heads
I0708 12:51:04.931566 1 source.go:189] Cloning
source from https://github.com/nubomedia/nubomedia-magic-
mirror
...
Image already exists
latest: digest:
sha256:24ac51ca448edfb1582f0bffc990c95506065f0aa26d5b295d
1cb32d35ce2dfe size: 49623
I0708 12:54:31.392578 1 docker.go:99] Push
successful
