Gartner predicts that there will be over 21 billion connected devices in the world by 2020 [https://www.gartner.com/doc/3299317]. More than anything else, this number drives all 3 vs of Big Data: Velocity, Volume, Variety. The Internet of Things (IoT) causes some challenges for any database system: security, resource consumption, ingestion rate, replication as well as coping with various access patterns from transactional systems to complex analytics. This talk will discuss how PostgreSQL's current feature set, the new features in PostgreSQL 11 and Postgres-BDR will provide the required feature set for IoT.

1. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Boriss Mejías
Consultant - 2ndQuadrant
Air Guitar Player
Internet of Things
with PostgreSQL
Performance & Security

2. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Boriss Mejías
Consultant - 2ndQuadrant
Air Guitar Player
Internet of Things
with PostgreSQL
Performance & Security
@postgreswomen

3. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Internet (of people)
Internet of Things

4. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Internet of Things
● Many Data Collectors
● Widely distributed
● Via public networks
● Multiple measures
● Massive volumes
● Low value data
● Insert-mostly workload

5. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Internet of Things
● Many Data Collectors
● Widely distributed
● Via public networks SSL mode on
● Multiple measures
● Massive volumes
● Low value data
● Insert-mostly workload “append-only”

6. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Which day is it?
● Recent and active data for immediate analysis
and immediate actions
● Older data for analytics to detect trends and
patterns

7. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Why PostgreSQL
● SQL with post-relational features

8. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Why PostgreSQL
● SQL with post-relational features
● DBEngines Database of the Year 2017
● Stack Overflow #1 Best Loved RDBMS 2018
● Cloud Adoption (AWS, Google, Microsoft, IBM)
● Other databases following PostgreSQL features
● Many forks, but open source version is strong

9. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Logical Database Design
CREATE TABLE iot
(device_id TEXT NOT NULL
,logtime TIMESTAMP WITHOUT TIME ZONE
,datum JSONB);

10. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Logical Database Design
CREATE TABLE iot
(device_id TEXT NOT NULL
,logtime TIMESTAMP WITHOUT TIME ZONE
,datum JSONB);

11. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Logical Database Design
CREATE TABLE iot
(device_id TEXT NOT NULL
,logtime TIMESTAMP WITHOUT TIME ZONE
,datum JSONB);
● Encrypted communication
● Authenticate each device

12. © 2ndQuadrant 2014-8
IoT with PostgreSQL
JSON Features
● JSON
● JSONB
● Search functions
● Manipulation functions

13. © 2ndQuadrant 2014-8
IoT with PostgreSQL
JSON Features
● JSON
● JSONB
● Search functions
● Manipulation functions
● Partition by JSON expression
● Index on JSON field
● HOT updates on JSON fields

14. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Declarative Partitioning
CREATE TABLE iot
(device_id TEXT NOT NULL
,logtime TIMESTAMP WITHOUT TIME ZONE
,datum JSONB)
PARTITION BY RANGE (logtime);

15. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Partition over Time

16. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Partition over Device
CREATE TABLE iot
(device_id TEXT NOT NULL
,logtime TIMESTAMP WITHOUT TIME ZONE
,datum JSONB)
PARTITION BY LIST (device_id);

17. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Partition over Device and Time
CREATE TABLE iot
(device_id TEXT NOT NULL
,logtime TIMESTAMP WITHOUT TIME ZONE
,datum JSONB)
PARTITION BY LIST (device_id);
CREATE TABLE device_a
PARTITION OF iot FOR VALUES IN ('a')
PARTITION BY RANGE (logtime);

18. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Declarative Partitioning

19. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Workload - extending partitions
● ATTACH PARTITION
● DETACH PARTITION

20. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Workload - extending partitions
● ATTACH PARTITION
● DETACH PARTITION
● Local protection laws:
– Easier to delete → Compliance
– Detached data → Retention Policy

21. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Partitioning Solutions
● Use fewer partitions, each with BRIN indexes
● Add partitions in groups

22. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Aggregate Data
● From ARM nodes
● To VLDB
● Using logical
replication
(pglogical)

23. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Aggregate Data
● From ARM nodes
● To VLDB
● Using logical
replication
(pglogical)
● Do NOT use trust
on pg_hba

24. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Bi-Directional Replication (BDR)

25. © 2ndQuadrant 2014-8
IoT with PostgreSQL
Conclusions
● IoT produces a lot of data
● Scale insert with partitions and BRIN
● Use JSON
● Scale with many nodes and pglogical
● PostgreSQL is really cool for IoT

26. © 2ndQuadrant 2014-8
IoT with PostgreSQL

27. © 2ndQuadrant 2014-8
IoT with PostgreSQL
● Sustainable Open Source Development
● 24/7 Bug Fix Support
● On-premise, Cloud, Hybrid with RemoteDBA
● Consulting & Training