Real-Time Text and WebRTC @ Kamailio World 2023

1. Bringing real-time text to WebRTC for NG Emergency Services Lorenzo Miniero @lminiero@fosstodon.org Kamailio World June 7th 2023,

2. Who am I? Lorenzo Miniero • Ph.D @ UniNA • Chairman @ Meetecho • Main author of Janus Contacts and info • lorenzo@meetecho.com • https://fosstodon.org/@lminiero • https://www.slideshare.net/LorenzoMiniero • https://lminiero.bandcamp.com

3. Just a few words on Meetecho • Co-founded in 2009 as an academic spin-off • University research efforts brought to the market • Completely independent from the University • Focus on real-time multimedia applications • Strong perspective on standardization and open source • Several activities • Consulting services • Commercial support and Janus licenses • Streaming of live events (IETF, ACM, etc.) • Proudly brewed in sunny Napoli(*), Italy

4. ((*) A different picture, for once!)

5. Emergency Services for the Hearing Impaired • Ad-hoc text telephones and telecommunication devices • Originally conceived and implemented in the 60’s • Commonly known as TDD / TTY / TT • “Telecommunications Device for the Deaf” • Units made of keyboard, small screen and modem • Messages converted to electric signals • Phone line used to deliver messages • Widely used in legacy Emergency Services • e.g., to allow 911 operators to engage with hard of hearing people • Uses phone line + batteries −→ still works in power failure

8. TDD/TTY/TT (PSTN)

9. TDD/TTY/TT (PSTN) • Different acronyms for mostly similar things • TDD (Telecommunications Device for the Deaf) • TTY (TeleTYpe) • TT (Text Telephone) • Some important limitations • Half-duplex (GA = GO AHEAD) • Can’t be interrupted (garbled output) • Limited set of “characters” (e.g., Baudot) • No specific handshake, and no error correction • Requires specific device

10. TDD/TTY/TT (PSTN) • Different acronyms for mostly similar things • TDD (Telecommunications Device for the Deaf) • TTY (TeleTYpe) • TT (Text Telephone) • Some important limitations • Half-duplex (GA = GO AHEAD) • Can’t be interrupted (garbled output) • Limited set of “characters” (e.g., Baudot) • No specific handshake, and no error correction • Requires specific device

11. Next Generation Emergency Services • Multiple efforts in different areas • NG9-1-1 (https://www.911.gov/issues/ng911/) • NG112 (https://eena.org/next-generation-112/) • Attempt to switch to modern ways of communication • Existing NG services based on PSTN • Internet-based communication allows for richer media, though • Location, , photos, videos, etc.

12. Next Generation Emergency Services • Multiple efforts in different areas • NG9-1-1 (https://www.911.gov/issues/ng911/) • NG112 (https://eena.org/next-generation-112/) • Attempt to switch to modern ways of communication • Existing NG services based on PSTN • Internet-based communication allows for richer media, though • Location, , photos, videos, etc.

13. Next Generation Emergency Services • Multiple efforts in different areas • NG9-1-1 (https://www.911.gov/issues/ng911/) • NG112 (https://eena.org/next-generation-112/) • Attempt to switch to modern ways of communication • Existing NG services based on PSTN • Internet-based communication allows for richer media, though • Location, real-time text, photos, videos, etc.

14. Next Generation Emergency Services • Multiple efforts in different areas • NG9-1-1 (https://www.911.gov/issues/ng911/) • NG112 (https://eena.org/next-generation-112/) • Attempt to switch to modern ways of communication • Existing NG services based on PSTN • Internet-based communication allows for richer media, though • Location, real-time text, photos, videos, etc.

15. Real-Time Text (RTT) • Text transmitted instantly, as it is typed or created • TDD/TTY work that way too • How to do it over IP, though? • Text over IP (ToIP) • ITU-T T.140 (Protocol for multimedia application text conversation) • Allows for real-time editing (e.g., backspace, rewriting) • T.140 messages transported over RTP • RFC 4103 (RTP Payload for Text Conversation) • Redundancy implemented via RED (RFC 2198)

18. T.140 • ITU-T T.140 (Protocol for multimedia application text conversation) • https://www.itu.int/rec/T-REC-T.140/ • Protocol based on the concept of T140blocks • Set of characters or special codes that one party is delivering to one or more others • Participant typing −→ typed characters bundled together in a T.140 block • Buffering size up to sender (overhead vs. latency) • UTF-8 used for characters, including special codes (actions) • e.g., Byte Order Mark (BOM), backspace, etc.

21. T.140 over RTP 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC=0 |M| T140 PT | sequence number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp (1000Hz) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | T.140 encoded data | + +---------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

22. Dealing with packet loss using RED

27. T.140 over RTP (with redundancy) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC=0 |M| "RED" PT | sequence number of primary | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp of primary encoding "P" | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| T140 PT | timestamp offset of "R" | "R" block length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| T140 PT | "R" T.140 encoded redundant data | +-+-+-+-+-+-+-+-+ +---------------+ + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+ | "P" T.140 encoded primary data | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

28. RTT and SIP/SDP • As an RTP-based stream, easy to negotiate via SDP • Needs text as media in m-line • t140 used to advertise codec in rtpmap • Sampling rate always 1000 (also for RED) SDP example m=text 11000 RTP/AVP 98 100 a=rtpmap:98 t140/1000 a=rtpmap:100 red/1000 a=fmtp:100 98/98/98

35. What about WebRTC? • WebRTC heavily based on existing VoIP technologies • SDP (on “steroids”) is mandatory • RTP used for media (encrypted via DTLS-SRTP) • That said, not all RTP media is supported • Only m=audio and m=video supported out of the box • No support at all for m=text, so RTT won’t work • Only m=application media that can be used is UDP/DTLS/SCTP • WebRTC data channels • Generic data sent via SCTP and encapsulated in DTLS

38. T.140 RTT over WebRTC Data Channels • Why not use data channels for RTT, then? • T.140 Real-Time Text Conversation over WebRTC Data Channels (RFC 8865) • https://www.rfc-editor.org/rfc/rfc8865.html • Sending T.140 frames directly in data channels, not on RTP • Data channels can be configured to be ordered and reliable • Helps avoiding hoops needed for RTP (e.g., RED) • A specific label can be associated with a specific RTT session • Of course, needs gatewaying to be backwards compatible • WebRTC endpoints can’t do RTP/RTT, and legacy ones don’t do data channels • Something needs to “translate” (media and SDP)

41. T.140/data channels SDP usage m=application 911 UDP/DTLS/SCTP webrtc-datachannel c=IN IP6 2001:db8::3 a=max-message-size:1000 a=sctp-port 5000 a=setup:actpass a=dcmap:2 label="ACME customer service";subprotocol="t140" a=dcsa:2 fmtp:t140 cps=20 a=dcsa:2 hlang-send:es eo a=dcsa:2 hlang-recv:es eo

42. Prototyping T.140/WebRTC in Janus • Created a prototype in Janus a few years ago • Old PR: https://github.com/meetecho/janus-gateway/pull/1898 (closed) • Blog post: https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/ • New PR: https://github.com/meetecho/janus-gateway/pull/3231 • Used the SIP plugin in Janus as the necessary gateway • Negotiates text m-lines and the text/t140 format on the SDP side • Negotiates data channels on the WebRTC side (binary) • T140blocks are then bridged (no translation) as they are • T140blocks decapsulated from RTP packets −→ data channels • Data channels −→ crafting RTP packets with T140blocks • RED optionally supported in both directions on the RTP side

45. SIP plugin in Janus https://janus.conf.meetecho.com/docs/sip

46. SIP plugin in Janus + RTT

47. Example: SDP offer from SIP/RTT endpoint v=0 o=Lorenzo_Miniero 1 1 IN IP4 192.168.1.74 s=Omnitor_SDP_v1.1 c=IN IP4 192.168.1.74 t=0 0 m=text 1024 RTP/AVP 99 98 a=rtpmap:99 red/1000 a=fmtp:99 98/98/98 a=rtpmap:98 t140/1000

48. Example: SDP offer to WebRTC endpoint v=0 o=Lorenzo_Miniero 1 1 IN IP4 192.168.1.74 s=Omnitor_SDP_v1.1 t=0 0 a=group:BUNDLE data a=msid-semantic: WMS janus m=application 9 UDP/DTLS/SCTP webrtc-datachannel c=IN IP4 192.168.1.74 a=sendrecv a=sctp-port:5000 a=mid:0 [.. ICE/DTLS details follow..]

49. Testing: TIPcon1 (SIP/RTT) https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

50. Testing: Janus SIP demo (WebRTC) https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

51. T.140 conversation on the wire https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

52. What’s next? • First of all, revive the effort • The original PR was closed a few months ago • Little to no feedback, and core had changed a lot in the meanwhile • New PR: https://github.com/meetecho/janus-gateway/pull/3231 • New revision can take advantage of some new features • e.g., RED supported in the Janus core, now • Maybe write a prettier and more functional UI too... • Possibly test with something more complex than TIPcon1 • TIPcon1 good for basic functional testing, but very limited • RTT endpoints very hard to find, especially in FOSS • Possibly integration tests with actual RTT deployments?

57. Thanks! Questions? Comments? Get in touch! • https://fosstodon.org/@lminiero • https://twitter.com/elminiero • https://twitter.com/meetecho • https://www.meetecho.com/blog/

Real-Time Text and WebRTC @ Kamailio World 2023

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Real-Time Text and WebRTC @ Kamailio World 2023

Semelhante a Real-Time Text and WebRTC @ Kamailio World 2023 (20)

Mais de Lorenzo Miniero

Mais de Lorenzo Miniero (20)

Último

Último (20)

Real-Time Text and WebRTC @ Kamailio World 2023