O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
ENABLING IOT AT
PROXIMUS
HI, MY NAME IS DAVE.
Software Engineer
Ordina Belgium
dave.degroote@ordina.be
IN THIS SESSION
How to implement mongoDB in a IoT M2M use case.
Data modelling is key when storing time series data.
How t...
INITIAL QUESTIONS
How to store data while keeping storage amount minimal?
What storage engine to choose?
How to store my d...
ESSENTIAL PROBLEMS
How do we model our data?
De ning storage engine
Setting up your mongo Instances
How do we insert our d...
SETTING THE SCENE
Mythings Project at Proximus
What is IoT?
LoRa Technology
2 Front-end
3 Back-end
MYTHINGS DEVELOPMENT TEAM
Small team:
Mongo experience: 1,5 year
MYTHINGS AT PROXIMUS
MYTHINGS AT PROXIMUS
INTERNET OF THINGS
Very broad, not contained to a single technology.
Not what but HOW !
Internet is the keyword
Collecting...
LORA TECHNOLOGY
LORA TECHNOLOGY VS ...
LORA ARCHITECTURE
SENSORS
Any type is possible as long as it is connected with a LoRa chip
CO2, PIR, Humidity, Luminance, Temperature, Press...
SENSORS
multiple containers per sensor possible
LORA ALLIANCE
https://www.lora-alliance.org
ESSENTIAL PROBLEMS
How do we model our data?
De ning storage engine
Setting up your mongo Instances
How do we insert our d...
DATA MODELING
SENSOR HISTORY
company info
sensor info
container info
noti cation value
noti cation timestamp
SENSORHISTORY COLLECTION
{
"_id": ObjectID("5613d33401e101c27f8e668d"),
"company": {
"name": "Proximus",
"companyId": 2
},...
Daily document size/sensor =
SENSORHISTORY COLLECTION SIZING
Document size = 496 Bytes
Sensor will send 48 time a day
On a...
Document with 3 additions:
BUCKETING TECHNIQUE
Data array (data values)
Count metadata (Array size)
Date eld (day timestam...
SENSORHISTORY COLLECTION (BUCKETING)
{
"_id": ObjectID("5613d33401e101c27f8e668d"),
"company": {
"name": "Proximus",
"comp...
daily document size/sensor =
SENSORHISTORY COLLECTION SIZING
(BUCKETING)
Document size = 3.98 KB
Sensor will send 48 time ...
Use if possible (depending on use-case)
vs
DATA MODELING
BUCKETING
11GB/day 68GB/day
Concurrent updates on the bucket document may lead to queue
operations
Do not overdue it!
DATA MODELING
BUCKETING
Advantag...
DEFINING STORAGE ENGINE
APPLICATION REQUIREMENTS:
Write intensive
Keep storage volume to a minimum
Read and write simultan...
DEFINING STORAGE ENGINE
WIREDTIGER 3.2
Document level lock instead of collection level
Data is stored in a compressed form...
SETTING UP MONGODB INSTANCES
CLOUD MANAGER
Once automation agent is installed, smooth sailing!
SETTING UP YOUR MONGODB
INSTANCES
CLOUD MANAGER
Easy setup
Easy manage...
INSERTING DATA
MYTHINGS ARCHITECTURE
INSERTING DATA
REST INTERFACE
Sleepy Mongoose (Python)
https://github.com/mongodb-labs/sleepy.mongoose
Rest Heart (Java)
l...
Latest feature support
Uses PATCH method for updates (Upsert)
=> Not supported current release
Not mature enough: no bucke...
Back to square one!
INSERTING DATA
SLEEPY MONGOOSE
Interchange Rest heart <> Sleepy Mongoose
Easy transition (1-2 days development)!
INSERTING DATA
SLEEPY MONGOOSE
Simple insert:
Advanced upsert:
INSERTING DATA
$ curl --data 'docs=[{"x":2},{"x":3}]' 'http://localhost:27080/foo/bar/_ins...
QUERING DATA
Query framework
Aggregation framework
Retrieve sensorhistory for a week for all companies:
AGGREGATION FRAMEWORK
"op" : "command",
"ns" : "sensoringlabs.sensorH...
Retrieve sensorhistory for a week for all companies:
QUERY FRAMEWORK
"op" : "query",
"ns" : "sensoringlabs.sensorHistory",...
Same search:
Query framework :
Aggregation framework :
QUERING DATA
1ms
360 ms
-> fast for retrieving data
-> data aggregations that take more time
Read concern:
read from secondary
QUERING DATA
Query ...
Query example:
QUERING DATA
SLEEPY MONGOOSE
$ curl -X GET 'http://localhost:27080/foo/bar/_find
?criteria={"_id":{"$oid":"...
INDEX STRATEGY
COMPOUND INDEXES
Multitenant system: company id
Time based data: date
Data retention policy:
-> Deterministic way to drop documents once expired
-> Fixed size collections
CC Not suitable becau...
PUTTING IT ALL
TOGETHER
Bucketing
DATA MODELING
Wired tiger 3.2
STORAGE ENGINE
Sleepy Mongoose
RestHeart (future)
Java mongo Driver
DATA INSERTION
query framework
aggregation framework
DATA EXTRACTION
compound indexes
TTL index
INDEXES
SHARDING
WHEN
There is more data than one machine can hold on its drives
The application is write heavy and you could expe...
CONCLUSION
WHAT WE LEARNED
Take enough time to model new collections (bucketing)
=> Can decrease the storage volume when d...
THANKS FOR LISTENING!
Questions?
Próximos SlideShares
Carregando em…5
×

MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company

849 visualizações

Publicada em

Proximus is one of the biggest Telecom companies in the Belgian market. This year the company began developing a new IoT network using LoRaWan technology. The talk will detail our development team’s search for a database suited to meet the needs of our IoT project, the selection and implementation of MongoDB as a database, as well as well as how we built a system for storing a variety of sensor data with high throughput by leveraging sleepy.mongoose. The talk will also discuss how different decisions around data storage impact applications in regards to both performance and total cost.

Publicada em: Dados e análise
  • Login to see the comments

MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company

  1. 1. ENABLING IOT AT PROXIMUS
  2. 2. HI, MY NAME IS DAVE. Software Engineer Ordina Belgium dave.degroote@ordina.be
  3. 3. IN THIS SESSION How to implement mongoDB in a IoT M2M use case. Data modelling is key when storing time series data. How to insert data into mongoDB using Rest interface.
  4. 4. INITIAL QUESTIONS How to store data while keeping storage amount minimal? What storage engine to choose? How to store my data using a third party system? How can I easily deploy my mongoDB solution?
  5. 5. ESSENTIAL PROBLEMS How do we model our data? De ning storage engine Setting up your mongo Instances How do we insert our data in mongoDB? How do we query our data?
  6. 6. SETTING THE SCENE Mythings Project at Proximus What is IoT? LoRa Technology
  7. 7. 2 Front-end 3 Back-end MYTHINGS DEVELOPMENT TEAM Small team: Mongo experience: 1,5 year
  8. 8. MYTHINGS AT PROXIMUS
  9. 9. MYTHINGS AT PROXIMUS
  10. 10. INTERNET OF THINGS Very broad, not contained to a single technology. Not what but HOW ! Internet is the keyword Collecting, analyzing and above all communicating data.
  11. 11. LORA TECHNOLOGY
  12. 12. LORA TECHNOLOGY VS ...
  13. 13. LORA ARCHITECTURE
  14. 14. SENSORS Any type is possible as long as it is connected with a LoRa chip CO2, PIR, Humidity, Luminance, Temperature, Pressure, …
  15. 15. SENSORS multiple containers per sensor possible
  16. 16. LORA ALLIANCE https://www.lora-alliance.org
  17. 17. ESSENTIAL PROBLEMS How do we model our data? De ning storage engine Setting up your mongo Instances How do we insert our data in mongoDB? How do we query our data?
  18. 18. DATA MODELING SENSOR HISTORY company info sensor info container info noti cation value noti cation timestamp
  19. 19. SENSORHISTORY COLLECTION { "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "data": { "type": "bool", "ts": 1444139821000, "value": "1" } }
  20. 20. Daily document size/sensor = SENSORHISTORY COLLECTION SIZING Document size = 496 Bytes Sensor will send 48 time a day On average 3 containers/sensor 496 B * 48 documents * 3 = 71,4 KB 71,4 KB 1 M sensors => 68.09 GB/day
  21. 21. Document with 3 additions: BUCKETING TECHNIQUE Data array (data values) Count metadata (Array size) Date eld (day timestamp) Use in combination with Upsert mechanism!
  22. 22. SENSORHISTORY COLLECTION (BUCKETING) { "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "date": 1444086000000, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "count": 48, "data": [ { "type": "bool", "ts": 1444139821000, "value": "1" }, { "type": "bool", "ts": 1444152754000, "value": "1" },... ] }
  23. 23. daily document size/sensor = SENSORHISTORY COLLECTION SIZING (BUCKETING) Document size = 3.98 KB Sensor will send 48 time a day On average 3 containers/sensor 3.98 KB * 1 document * 3 = 11,9 KB 11,9 KB 1 M sensors => 11.34 GB/day
  24. 24. Use if possible (depending on use-case) vs DATA MODELING BUCKETING 11GB/day 68GB/day
  25. 25. Concurrent updates on the bucket document may lead to queue operations Do not overdue it! DATA MODELING BUCKETING Advantages: Collection size will be smaller Indexes size will be considerable reduced Reads will be faster Remarks: ! Maximum document size 16MB !
  26. 26. DEFINING STORAGE ENGINE APPLICATION REQUIREMENTS: Write intensive Keep storage volume to a minimum Read and write simultaniously to collection
  27. 27. DEFINING STORAGE ENGINE WIREDTIGER 3.2 Document level lock instead of collection level Data is stored in a compressed format Index size is smaller Matched the application requirements WiredTiger was recommended by MongoDB
  28. 28. SETTING UP MONGODB INSTANCES CLOUD MANAGER
  29. 29. Once automation agent is installed, smooth sailing! SETTING UP YOUR MONGODB INSTANCES CLOUD MANAGER Easy setup Easy management Metrics
  30. 30. INSERTING DATA MYTHINGS ARCHITECTURE
  31. 31. INSERTING DATA REST INTERFACE Sleepy Mongoose (Python) https://github.com/mongodb-labs/sleepy.mongoose Rest Heart (Java) latest feature support http://restheart.org/ Advised to change to Rest Heart by MongoDB
  32. 32. Latest feature support Uses PATCH method for updates (Upsert) => Not supported current release Not mature enough: no bucketing for upsert! INSERTING DATA REST HEART
  33. 33. Back to square one! INSERTING DATA SLEEPY MONGOOSE
  34. 34. Interchange Rest heart <> Sleepy Mongoose Easy transition (1-2 days development)! INSERTING DATA SLEEPY MONGOOSE
  35. 35. Simple insert: Advanced upsert: INSERTING DATA $ curl --data 'docs=[{"x":2},{"x":3}]' 'http://localhost:27080/foo/bar/_insert' $ curl --data 'criteria= { "sensor":{"sensorId":1533,"name":"Binary mesurement","mac":"020000FFFF00B151"}, "company":{"companyId":134,"name":"Opinum"}, "container":{"containerId":1223,"name":"Door counter"}, "date":1456354800000 }, "count":{"$lt":100} &newobj={ "$push":{"data":{"ts":1456403435000,"value":"456","type":"int"}}, "$inc":{"count":1}, "$setOnInsert":{"location":{"latitude":"-1","longitude":"-1"}}} &upsert=true' 'http://localhost:27080/foo/bar/_insert'
  36. 36. QUERING DATA Query framework Aggregation framework
  37. 37. Retrieve sensorhistory for a week for all companies: AGGREGATION FRAMEWORK "op" : "command", "ns" : "sensoringlabs.sensorHistory", "command" : { "aggregate" : "sensorHistory", "pipeline" : [{ "$match" : { "company.companyId" : { $in" : [,54,61,76,77,79,80,81,82,83,86,87,90,122,123 ,124,125,126,127,128,129,130,131,132,133,134,135,136, 137,138,146,147,148,149,150,151,152,153,154] } } }, { "$match" : { "date" : {"$gte" : NumberLong("1457049600000")}, "$and" : [{"date" : {"$lte" : NumberLong("1457568000000")}}] } }, {"$skip" : NumberLong(0)}, {"$limit" : NumberLong(100)} ], "cursor" : {} } "responseLength" : 278888, "protocol" : "op_command", "millis" : 366
  38. 38. Retrieve sensorhistory for a week for all companies: QUERY FRAMEWORK "op" : "query", "ns" : "sensoringlabs.sensorHistory", "query" : { "find" : "sensorHistory", "filter" : { "company.companyId" : { "$in" : [2,54,61,76,77,79,80,81,82,83,86,87,90,122,123,124, 125,126,127,128,129,130,131,132,133,134,135,136,137, 138,146,147,148,149,150,151,152,153,154] }, "date" : { "$gte" : NumberLong("1457049600000"), "$lte" : NumberLong("1457568000000") } }, "limit" : 100, "singleBatch" : false } "nreturned" : 100, "responseLength" : 278888, "protocol" : "op_command", "millis" : 1
  39. 39. Same search: Query framework : Aggregation framework : QUERING DATA 1ms 360 ms
  40. 40. -> fast for retrieving data -> data aggregations that take more time Read concern: read from secondary QUERING DATA Query framework Aggregation framework
  41. 41. Query example: QUERING DATA SLEEPY MONGOOSE $ curl -X GET 'http://localhost:27080/foo/bar/_find ?criteria={"_id":{"$oid":"4f8c6f05db61e2a72600001d"}} &sort={"date”:-1} &limit=1'
  42. 42. INDEX STRATEGY COMPOUND INDEXES Multitenant system: company id Time based data: date
  43. 43. Data retention policy: -> Deterministic way to drop documents once expired -> Fixed size collections CC Not suitable because time range of documents will depend on throughput! (high throughput will decrease the time range stored). CAPPED COLLECTION VS TTL INDEXES GETTING RID OF THE EXPIRED DATA 3 months TTL index Capped collection
  44. 44. PUTTING IT ALL TOGETHER
  45. 45. Bucketing DATA MODELING
  46. 46. Wired tiger 3.2 STORAGE ENGINE
  47. 47. Sleepy Mongoose RestHeart (future) Java mongo Driver DATA INSERTION
  48. 48. query framework aggregation framework DATA EXTRACTION
  49. 49. compound indexes TTL index INDEXES
  50. 50. SHARDING WHEN There is more data than one machine can hold on its drives The application is write heavy and you could experience too much latency The working set outgrows the memory can be allocated to a single machine
  51. 51. CONCLUSION WHAT WE LEARNED Take enough time to model new collections (bucketing) => Can decrease the storage volume when done right The use of cloud manager drastically decreases deployment and maintenance time => More time to think about your application There are endless ways of getting your data into mongoDB => Mongo supplies and support many di erent solutions
  52. 52. THANKS FOR LISTENING! Questions?

×