This document summarizes a presentation about data stream networks and how they help the IoT ecosystem. It defines a data stream network as a secure, global network for two-way communication and real-time data push. It discusses how data stream networks provide services like publish/subscribe messaging, global replication and reliability, presence detection, storage and playback, and access control. Finally, it outlines how data stream networks help solve IoT pain points like presence detection, scalability, bandwidth usage, and security by enabling real-time communication, analytics, message routing and more between mobile/IoT/web applications.
13. Publish/Subscribe Messaging
• Protocol-independent pub/sub core messaging service
• Messages up to 32kb w/ TLS encryption
• Lightweight channel handling: no limit on number of
channels & no cost impact
• Publish 1-to-1, 1-to-many, many-to-many
// publish
pubnub.publish({
channel : “hw_test”,
message : “Hello World!”
});
// subscribe
pubnub.subscribe({
channel : "hw_test",
message : receiver
})
14. Global Replication & Reliability
• 14 data centers located on 4 continents
• 99.999% uptime guaranteed
• All messages replicated to all 14 data centers by default
• Automatically manages dropped connections & reconnects
• Message caching catches devices up on missed data
15. PubNub Presence
• Built into API for all channels
• Automatically detect join/leave or device status change events
• Trigger events/behaviors based on presence status changes
• API method for retrieving list of device in a given
state & device location
• Set custom status changes, i.e. GPS location, or any custom
name/value pairs
// presence
pubnub.subscribe({
channel : “hw_test",
presence : pres_reciever,
message : receiver
});
16. Storage & Playback
• Default 24 hour storage of all messages;
unlimited retention available
• Catch users up on message history when they come online
• Load a configuration onto a device when it is booted up
• Audit past messages for compliance
• Play back messages at same rate they were received
// storage and playback
pubnub.history({
channel : “hw_test",
callback : sp_reciever
})
17. Access Manager
• Token-based access control scheme (similar to OAuth)
• Grant access to individual channels or groups of channels
• Read / write controlled at token level
• Tokens can auto-expire based on time (TTL)
• Grant/revoke tokens manually
// Key Authorization
var pubnub = PUBNUB.init({
publish_key : "pam",
subscribe_key : "pam",
auth_key : "3y8uiaj"
})
21. The different
components of IoT
• Data stream network does the
heavy lifting
• Smart network that manages
“things” and the data that flows
between them.
Things
Data Stream
Network
Mobile
applications
22. IoT pain points
Presence
detection
Decreased time to
market
Storing offline
messages
Minimum bandwidth
usage
Cross platform
support
security
Read/write
access
Bidirectional data
flow
Power drain
24. IoT pain points
• Detect machine connections
• Monitor user and machine
presence
• Global scale
Presence Detection
• Storage of realtime data
• History to retrieve messages
• Replay messages in order
received
Message History
25. • Sensors
• Always on devices
• Protocol independent approach
Bandwidth/power usage
• Millions of devices
• Plug and play configuration
• Decreased time to market
Scalability
27. Enterprise-Grade Security
Cross-platform AES & TLS
Message envelope meta-
data for smart routing &
analytics
Fine-grain token-based
access control
Auto-expiration of tokens
& manual revocation
No inbound open ports
removes attack vector
Can route to different data
centers to thwart regional
attacks.
Safe Harbor Compliant
Safe for HIPAA and PII
Intelligent Data Routing
Access Control Attack Prevention LegislativeEncryption
28. Why a Data Stream Network matters for
IoT?
• Realtime communication
• Analytics
• Message Routing
• Scalability
Mobile/IoT/web
29. IoT Use cases
• Control and monitor devices
• Stream vehicle data
• Trigger automatic actions
• Monitor device metadata
• http://www.pubnub.com/solutions/
Home Automation Connected/Smart cars
Wearables/ Connected devices
Coffe pot monitor : https://vimeo.com/134363392
PubNub security camera : https://vimeo.com/131005832
PubNub IoT house : https://vimeo.com/114900127
Motion controlled servos with leap motion and the Pi :
https://vimeo.com/136779399
Hi everybody, thank you for coming. My name is Bhavana srinivas and I work as a developer evangelist for PubNub.
PubNub is a global DSN that provides easy to use API for real time , reliable signaling between sensors, mobile devices, embedded chips, or software applications, anywhere around the globe in a secure manner. We have 70+ SDKs that we support, to let you build and scale real time applications for mobile, web and IoT.
As I go through my presentation, you will see what a data stream network means and how it will benefit IoT.
My twitter handle is @bhavana1110 and you can reach me at bhavana@pubnub.com. I have some cards and data sheets and frisbees with me as well in case you are interested.
I will walk you through what a data stream network is, what you can do with it and why it is criticial for IoT.
Lets start of by understanding what this even means and then move on to how we can use it for IoT.
BUILD SLIDE
More and more companies/software's needed the ability to control and work with real time data. Whether you needed stock updates, multi player game movements, screen sharing, voice over IP, telephony etc, you had this new data in motion, as compared to the traditional data in rest, which can be pulled out of a database every now and then.
So data is being constantly emitted be it social streams or financial streams, and there was no way to route it, or manage it or any kind of infrastructure to get the data to the devices that needed it in real time.
more and more companies were spending all their time and effort to build this kind of infrastructure than focussing on their application logic.
So just to make this really clear where PubNub sits, this is the evolution of the Internet. MAKE NOTES FROM TODDS PRESENTATION.
Lets go back 25 years or so. People were building the core infrastructure for the internet. So you had companies like Verizon, Comcast and level3 that were offering peering, routing or PoPs that speak TCP/UDP. Basically, connecting you to the internet.
Fast forward 10/15 years, you had all these webisites with tonnes of informtaion , all trying to scale. So you had CDNS like Akamai, cogent etc, that made copies of the information on these sites, and put them all over the world. If you were trying to access yahoo weather info from Japan, you would probably be connected to a data center that’s diff from someone accessing that information from California. These guys were speaking difffernet protocols like HTTP to do caching services etc.
A data stream network is similar to this, but meamt for two way communication to and from these data centers. U still have data centers all over the world, but you are publishing and subscribing data from them, and also speak other protocols like websockets,, MQTT, which are constantly evolving.
The current infrastructure for data at rest is pretty mature and well understood..They have been used for the past so many years, and they make it very easy to build mobile and web apps.
For data in motion on the other hand, its first of all not compatible with the infrastructure available for data at rest applications. The software or technologies themselves are relatively new. There are several open source technologies that can be used, but when you have to scale it becomes a problem. They are not secure, since they are new as well, and hasn’t been documented well.
So what we’ve just seen is that pretty much the whole internet is architected to handle data at rest. We’re interested in data in motion – streaming data. Similar to how you wouldn’t build your own ISP or CDN, what we’re suggesting is you shouldn’t try to architect your own DSN either.
You can roll your own, but…
Time to build is costly and increases time to market.
Hidden costs of scaling up are disproportionately expensive.
Ongoing maintenance and orchestration requires a dedicated team of experts.
Flexibility and expansion potential constrained by limited feature sets.
Security is an ongoing risk.
When you are devloping an application on a small scale , it is easy.
So does it make sense for everyone building apps, build hteir own dsn? No. why? But in the real world, you have a lot of problems like
unrealiable connectiviity’
different networks, - devices are always moving from one network to another. Within tunnels or losing powerr anf hence connectivity.
diff devices to be supported -
Latency
scaling
security
and many more.
Add notes from Todds presentation on what a global DSN is.
This is why using a Data stream network helps,, since it provides a way to connect, scale and secure real time data streams for rhe web, mobile and IoT.
Massive numbers of connections across any device
Extremely low latency
Unmatched reliability / uptime
End to end security over our network
Connect
Support several platforms
Powered by publish/subscribe messaging
Storage & Playback functionality
Secure
End to end security: TLS/SSL & AES included by default
PubNub Access Manager for token-based access control
No open ports to the web
EU SafeHarbor, HIPAA compliant
Scale
Global data replication to 14 data centers
Intelligent routing automatically connects users to closest PoP
Message & bandwidth efficiency with PubNub Stream Controller
Add scale easily, automatically, and economically
Audience simulation service for smooth rollouts & QA
Manage
Developer console & debug tools
Stream analytics shows data stream activity in real time
Steam syndication design pattern makes it easy to monetize data streams
Business model scales with your project’s success
No year-long contracts
PubNub excels because we deliver:
If you’ve had a chance to align on required capabilities before the call, list them here. If not, think about taking the time to fill this in with the prospect during the presentation.
Real-time data transfer speeds
Real-time location tracking
Extremely high reliability (works in the real world)
Easy, efficient scalability
Robust security built in
Developer tools to build in preferred environment
Business model that scales with your success
Add-on features to enable your use case
Explain the reasoning behind each of these features as per Todd’s video.
A data stream network has the ability to manage bi-directional communication between the devuces connected to them. This can be 1-1, 1-many or many-to-many, basucally the ability to publish and subscribe to messages.
For several applicattions, you want to be able to know who is part of the conversastion, or who has joined that channel of communicaationEssentially you wajnt to know the presence on that channel. ThisThis is similar to ,the green dot you see in a chat app like skype, when a person is online. Or a grey when they leave.
This is being used a lot in IoT, where you want to know when your remote controlled thermostat comes online, or also know when that connection goes off.
Now that being said, say you have several users on a chat channel, or several devices that are sending and receiving information using the data stream network.
. You are able to receive messages when you are connected to the network. But say your devuce went offline for a day, month or year. You want to be able to see the history of the streams that you missed on that channel and played back at the same rate at which it was received.
Also, in order to secutre the deivices that send/receive messages through the network, you want to be able to give
fine grained read/ write access on a channel and token level.
This way, only devices that have access can publish or subscribe.
What is the current state of IoT.
What are the different verticals
What are some of the common problems – no bi directional comm/ no uniformity ex all devices cant talk to each other/ memory/bandwidth/security/
How does a data stream network/pubnub solve it
So, as is often described, there are many machine to machine communications within the category of IoT. You have smart home appliances to wearables that may or may not be plugged into medical diagnostics to connected cars and transportation.
These, along with many other categories are the “Things” in IoT.
But when describing the “Internet” part of IoT, most presentations simply had some sort of amorphous cloud that connected everything magically.
The point is that everyone seems to talk about the big data and the devices that make up the IoT, but not the lines to the “cloud” that connects it all.
People are working on different verticals. And a very common application is when a sensor or a device is sending information back to the user. But you want to be able to do more than just receiving info from your thermostat. You want to be able to take some actions based on that value. You want to be able to control the devices you own, remotely, irrrespective of where you are in the world. You want to be able to do this in real time, cos noone wants their garage door to open five min after you press a button. You want it to happen instantly.
And that’s where a data stream network like pubnub fits in. It is the communication layer that connects the IoT.
What is the best way to build an IoT application that handles everything from controlling home appliances remotely to inform an aero-engine manufacturer that one of its engines need servicing in the opposite side of the world to gathering information from sensors destributed in the wild?
The starting point for Internet of Things applications are the things themselves. These edge devices typically have no screen (although that's not always the case), a low-power processor, some sort of embedded operating system and a way of communicating (usually wirelessly) using one or more communication protocols. The things may connect directly to the Internet, to neighboring things or to an Internet gateway device – typically a plastic box with blinking lights.
The next tier of the system, is a software and infrastructure that runs in a corporate data center or in the cloud and receives and organizes the streams of data coming from the things. Software running in the ingestion tier is usually also responsible for managing things and updating their firmware when necessary. After this comes the analytics tier; this takes the organized data and processes it. Finally, there's the end-user tier, the application that the end user actually sees and interacts with. This may be an enterprise application, a Web app or, perhaps, a mobile app.
So from the previous image, you can see how there are so many problems to solve startting from the “tjings in the internet of things, the network or infrastructure trat will support them, and the devices that run the application.
Lets start with the things – there are so many of them, from MCUs to MPUs, all runninf diff Oses, diff platforms etc. A samsung device doesn’t talk to an LG one which doesn’t talk to Apple devices., Whehn they are so varied, you need cross platform support.
Let me go through some of the obstacles in building an IoT application.
power drain : PubNub’s publish / subscribe paradigm reduces “chattiness” to keep power drain to a minimum
bandwidth : Protocol-independent approach uses the minimum bandwidth necessary to monitor and control IoT devices
Firmware: Use secure, private channels to remotely upgrade firmware with no user action needed
time to market : Don’t have to waste months of your team’s time architecting and maintaining a realtime infrastructure in-house
device provisioning: Make your IoT device plug-and-play, even through home and business firewalls
The way we do security is that the device opens a connection to Pn, and we keep that connection opne, so that now data canonly flow back through the pn nextwork to the device.
Remote Firmwarae upgrades – we don’t provide the FTP server that people usually need for this. We have a design practise to do this.
The server first lets its devices know that a new firmware update is available by broadcasting an alert message on a channel that all devices can read.
Then the devices that are online (and we know which ones they are through the use of Presence) download the update immediately.
For those devices that are not online, they can download the update from a cache on the channel when they are booted back up and available.
For some applications, its all about collecting data from the device like a sensor . For those, uni directional communication is more that enough
Data collection is:
* Easy
* Everyone is focusing on it today
[CLICK FOR ANIMATION]
Device Control is:
* Harder
* With smart home, connected cars, industrial valve controls, robots, etc. we don’t just want to read their status, we also want to control the devices
The promise of IoT is with device control, it’s not just about receiving the data from the devices
Built into API for all channels
Automatically detect join/leave or device status change events
Trigger events/behaviors based on presence status changesstate & device location
Set custom status changes, i.e. GPS location, or any custom name/value pairs
We have spotty connections everywhrre, when a handoff happens. In remote areas and when you go under tunnerls. This is when devices go offline. There is a meed for the device to automatically try to reconnect, and come back online as soon as possible. When it does, it needs to access all the information that might be lost, at the same rate that it was sent out,.
Default 24 hour storage of all messages; unlimited retention available
Catch users up on message history when they come online
Load a configuration onto a device when it is booted up
Audit past messages for compliance
Play back messages at same rate they were received
These sensors and hardare devices are becoming very small and cheap., but the constraint is power and memory. They have low program memory, and also cant last too long on, unless they conserve battery. The device must be able to send and receive messahes , by not spending too much energy .
When building an application, it is usually targeted to a large number of users. Once these devuces are deployed, it is necessary to ensure that every one of them can communicate with the DSN. When bought, these devices will prove ineffective if they need to be configured to work with the house/office firewalls. People expect it to be plug and play. You bring a new speaker at hime, that connects to the internet, you expect to be able to set it up really easily.
Already, too many manufacturers worry more about getting their IoT product to market quickly than about securing it. In some cases, the devices are so small that it’s hard to build in the right protection.
how do you go about managing the username/passwords for your ever increasing number of connected devices and appliances? What about the privacy of your information? Take as an example various Internet connected video cameras with easily defeated security controls or baby monitors.
So offloading security to the network is the right answer for many developers.
Jeep Hack (August 2015)
KQED.org (August 2015): http://ww2.kqed.org/futureofyou/2015/08/03/millions-of-americans-use-medical-devices-that-are-vulnerable-to-hacking/
Deep brain neurotransmitters
Gastric stimulators
Foot drop implants
Cochlear implants
Cardiac defibrillators / pacemakers
Insulin pumps
Security has to be a forethought more than anything else. The race to market for new devices means that security is not included in the design, or it is severely limited or poorly implemented. ’
You need to be able to encrypt the message using maybe TLS to secure the message from the device to the network, and trhen use something like AES that does the encryption even when it goes through the networlk.
OPEN PORTS: most devices on the internet, are also listening to any messages that come to them throufh the network. They always have open ports. This is asking for trouble since anyone can hack into that and sned them the wrong messages. You want to be able to open a connection to your device tham the other way rounf to be more secure.
TLS + AES:
ACCESS CONTROL – giving different users diff read/write permissions.
That's why it usually makes more sense to build an application on top of a ready-made "Internet of Things platform," Gillett adds. These platforms usually include an ingestion tier that carries out time-series archiving for incoming data, as well as an analytics tier, thin provisioning, activation and management capabilities, a real-time message bus, and an API to allow communication between the platform and applications built on top of it.
Build vs buy argument. How scale also helps.
A data stream network lets you manage, connect, scale your realitme applications for mobile web and IoT. At pubnub as well, we have seen several customers , over 2000, in a lot of verticals like chat, social apps, screen share, collaborative apps and IoT. With IoT, we have seen people using us for home automatrion, devcice contril, connected cars and connected devuces,
We have seen and built a lot of demos from hobbysists like raspberry pi and arduino community. They involve bidirectional messaging to and from the device.