WebRTC, and most of HTML5 tecnologies as File API, WebGL, WebWorkers, ...,answer to a common problem, i.e. to make 'in browser' apps able to do operations impossible before, as access to the video and audio stream by a javascript api.

1. WebRTC
... GWT & Browser Performance
Alberto Mancini, Francesca Tosi
JooinK.com

2. WebRTC
Plug-in free realtime communication …
WebRTC is a free, open project that enables web browsers
with Real-Time Communications (RTC) capabilities via
simple Javascript APIs.
source: webrtc.org

3. WebRTC
Just another JS API
or
WebRTC is a new front in the long war for an
open and unencumbered web
Brendan Eich (Mozilla CTO and inventor of JavaScript)
?

4. Browser Piercing
Once upon a time ...
LDP: Firewall Piercing mini-HOWTO
François-René Rideau
v0.97, 24 November 2001

5. Browser Piercing
a Javascript Application cannot
➔ access the filesystem
and then File API
➔ open full-duplex socket connection
and then WebSockets

6. Browser Piercing
a Javascript Application cannot
➔ use graphics accelerator
and then WebGL
➔ span multiple threads
and then WebWorkers

7. Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
and then WebRTC
➔ communicate P2P
and then WebRTC

8. WebRTC across platforms
●
●
●
●
Chrome (mobile too)
Firefox (mobile too)
Opera
Native Java and Obj-C

9. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

10. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

11. WebRTC JS-API
Constraints
● Controls the contents of the MediaStream
● Media type, resolution, frame rate
var constraints = {video: true};
video: {
mandatory: { ... },
optional [{... }]
}

12. WebRTC JS-API
Constraints
● Controls the contents of the MediaStream
● Media type, resolution, frame rate
var constraints = {video: true};
video: {
mandatory: { chromeMediaSource: ‘screen’ },
optional [{... }]
}

13. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

14. WebRTC JS-API
errorCallback
function errorCallback(error) {
console.log("error: ", error);
}
;-)

15. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

16. WebRTC JS-API
successCallback
function successCallback(stream) {
...
}
stream: MediaStream; a flux of audio- or video-related
data.

17. WebRTC JS-API
successCallback
function successCallback(stream) {
var video = ...
video.src =
window.URL.createObjectURL(stream);
}
(W3C File API)

18. WebRTC JS-API
full sample
var constraints = {video: true};
function successCallback(stream) {
var video = document.querySelector("video");
video.src = window.URL.createObjectURL(stream);
}
function errorCallback(error) {
console.log("navigator.getUserMedia error: ", error);
}
navigator.getUserMedia(constraints, successCallback, errorCallback);

19. WebRTC JS-API
Do not forget
● to play the video
● to polyfill getUserMedia (moz*,webkit*,ms*)

20. WebRTC JS-API
RTCPeerConnection
→
getUserMedia
+
RTCPeerConnection
←

21. WebRTC JS-API
RTCDataChannel
Bidirectional communication of arbitrary data
● low latency
● API similar to WebSockets

22. RTCPeerConnection sample
pc = new RTCPeerConnection(null);
pc.onaddstream = gotRemoteStream;
pc.addStream(localStream);
pc.createOffer(gotOffer);
function gotOffer(desc) {
pc.setLocalDescription(desc);
sendOffer(desc);
}
function gotAnswer(desc) {
pc.setRemoteDescription(desc);
}
function gotRemoteStream(e) {
attachMediaStream(remoteVideo, e.stream);
}

23. RTCPeerConnection

24. RTCPeerConnection
.

25. RTCPeerConnection
.

26. STUN
.

27. TURN
.

28. Yes BUT
VideoChat already exists and we are too late to build skype without
skype ...
●
●
Video chat:
○ SimpleWebRTC
○ easyRTC
○ webRTC.io
Peer-to-peer data:
○ PeerJS
○ Sharefest

29. Yes BUT
● getUserMedia: gives us real time stream from the cam
● we can draw the <video/> into a <canvas/>
● from the canvas we can extract pixel data getImageData()
and then elaborate.

30. Computing (GWT 4 us)
goals
- Almost native performance
- Manage complex algorithms and applications

31. Computing (GWT 4 us)
goals
- Almost native performance
- Manage complex algorithms and applications
our approach
- javascript as a target language (GWT)
- hand written optimized pieces of code

32. JooinK’s WebRTC sample
Sample
http://picshare.jooink.com
Demo …

33. PicShare
Tecnologies
●
●
●
●
WebGL for 3D graphics
WebRTC for MediaSteams
ImgUR for pictures store
AppEngine as a glue

34. FOSS
gwt-nyartoolkit.googlecode.com
gwt-webgl.googlecode.com
The strategy: compile a java-lib with GWT into javascript

35. What’s next
Challanges
● real-time on mobile apps
● real world recognition: computer vision
(BoofCV)

36. What’s next
Our approach
● use TypedArrays everywhere
● offload computation to the graphic
accelerator through WebGL
● help the JIT compiler/optimizer by asm.js

37. That’s all folks!!!
Alberto Mancini
alberto@jooink.com

38. Markers