The document discusses using machine learning and on-device intelligence to analyze sensor data. It notes that currently 99% of sensor data is discarded due to constraints, but drawing conclusions directly on sensors can vastly increase their usefulness. On-device ML is needed to detect patterns in real-time sensor streams and identify anomalies. The speaker presents an example of classifying sheep activity with a neural network model deployed on an edge device. Edge Impulse is introduced as a service that helps collect sensor data, extract features, train models, and deploy them to edge devices for efficient real-time analysis using limited resources.

1. Jan Jongboom
Open Hardware Event
1 April 2020
Tiny intelligent computers
and sensors

2. Jan Jongboom
CTO and co-founder, Edge Impulse
jan@edgeimpulse.com

3. 3
Typical industrial sensor in 2019
Vibration sensor (up to 1,000 times per second)
Temperature sensor
Water & explosion proof
Can send data >10km using 25 mW power
Processor capable of running >20 million
instructions per second

4. 4
But... what does it actually do?
Once an hour:
• Average motion (RMS)
• Peak motion
• Current temperature

5. 5
99% of sensor data is discarded due to
cost, bandwidth or power constraints.
https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/
The%20Internet%20of%20Things%20The%20value%20of%20digitizing%20the%20physical%20world/The-
Internet-of-things-Mapping-the-value-beyond-the-hype.ashx

6. 6
Lots of interesting events get lost
Peak

7. 7
Single numbers can be misleading
updown
circle
avg. RMS
3.3650
3.3515

8. 8
On-device intelligence is the only solution
🚫

9. 9
On-device intelligence is the only solution
Vibra&on pa+ern
heard that lead to fault
state in a weekTemperature
varies in a way that
I've never seen
before
Machine
oscillates different
than all other
machines in the
factory

10. 10
On-device intelligence is the only solution
Temperature varies in a way
that I've never seen before
(0x1)

11. 11
Drawing conclusions directly on the sensor
will drastically increase usefulness
(and allow us to move to higher value use cases)

12. 12
Interesting questions...
Classification
What's happening right now?
Anomaly detection
Is this behavior out of the ordinary?
Forecasting
What will happen in the future?

14. https://cdn2.i-scmp.com/sites/default/files/styles/980x551/public/images/methode/2017/05/23/6660b96e-3f9d-11e7-8c27-b06d81bc1bba_1280x720_183924.JPG?itok=ZmONr2a_

15. 15
ML is everywhere
Customer segmentation
Finding fraudulent transactions
Recommendation systems
Virtual assistants (Siri, Google Home)
Spam classification

16. 16
Machine learning

17. 17
Downsides
I'm not rich enough to develop 5,000
custom processors
Centralized
Costs lots of power and bandwidth
Privacy
Not me

18. 18
Learning tic-tac-toe
X
X
O
O
Content of lucifer box
Every box
resembles a state
of the board

19. 19
Learning tic-tac-toe
X
X
X
O
O
Content of lucifer box
Rules
Lose: Remove marble
Draw: Place 1 marble back
Win: Place 3 marbles back

20. 20
X
X
X
O
O
O
Learning tic-tac-toe
Content of lucifer box
You lose

21. 21
Learning tic-tac-toe
X
X
X
O
O
Content of lucifer box
You win

22. 22
Learning tic-tac-toe
X
X
O
O
Content of lucifer box

23. 23
Training vs. classification
Hundreds of different states
Need to encounter states many times
Training takes long!
Classification is however simple
Play the game, and you have to open up max. 4 drawers!

24. 24
Machine learning on the edge
Typically only inferencing, no training
Typically more efficient than sending data over the
network
Signal processing is still key

25. 25
Enabling new use cases
Sensor fusion
http://www.gierad.com/projects/supersensor/

26. 26
Anomaly detection
Enabling new use cases

27. 27
Enabling new use cases
Livestock monitoring
https://os.mbed.com/blog/entry/streaming-data-cows-dsa2017/

28. Machine learning is great at finding
patterns in messy data
(anything you can't reason about in Excel)

29. 29
In the industry
Large push from Google, Arm in creating better software ecosystem
(TensorFlow Lite, uTensor)
Lots of new chips coming out (ETA Compute, Arm Cortex-M55) with hardware
acceleration
Making smaller neural networks: quantization, pruning, lottery tickets
But... collecting and organizing high-quality data is hard!

30. 30
Signal processing is still key

31. 31
Edge Impulse - TinyML as a service
Embedded or edge
compute deployment
options
Test
Edge Device Impulse
Dataset
Acquire valuable
training data securely
Test impulse with
real-time device
data flows
Enrich data and
generate ML process
Real sensors in real time
Open source SDK
Free for developers: edgeimpulse.com

32. 32
v
In practice
https://www.flickr.com/photos/120586634@N05/14491303478

33. 33
Sheep activity tracker
h+ps://pixabay.com/photos/sheep-curious-look-farm-animal-1822137/

34. 34
Capturing raw data
11 minutes of raw data over 4 classes. Collected on device, synced over WiFi

35. 35
Extracting features

36. 36
Training two models
Neural network classifier Anomaly detec7on

37. 37
From model to device
Signal processing, neural network and
anomaly detec&on

38. 38
Conclusions back to cloud
♻
Sample for four seconds
Classify
Result differs? Message.
Sheep is walking
h+ps://pixabay.com/photos/sheep-curious-look-farm-animal-1822137/

39. 39
Device
80MHz Cortex-M4F processor
128 KiB RAM
Time to analyze 1 second of ac&vity data
(DSP + classifica&on + anomaly):
0.008 seconds

40. Demo 🚀

41. 41
How to get started?
edgeimpulse.com tinymlbook.com

42. Recap
The ML hype is real
ML + sensors = perfect fit
Start using the remaining 99% of sensor data
edgeimpulse.com

43. 43
Thank you!
Slides: janjongboom.com