The presentation introduces a Google Cloud native architecture for collecting, processing, analyzing and archiving events from IoT devices, vehicles as well as upstream software systems. InfluxDB and its connection to global native Google Cloud services like BigQuery or Cloud Machine Learning Engine as well as Kubernetes is at the center of the architecture. The architecture demonstrates how access to global scaling cloud services address use cases from the Energy Sector.

1. Christoph Bussler
June 23rd, 2020
IoT Event Processing and
Analytics with InfluxDB in
Google Cloud

2. Agenda
Energy
Sector
Energy sector
use case
analysis
Life cycle
of an
IoT event
Google Cloud
overview
1 2 3 4 5 6
InfluxDB TICK
architecture
Google Cloud
native
IoT event
processing
architecture

3. ● Production
○ Energy production systems monitoring and anomaly detection
■ Oil, gas, wind, hydro, solar and others
● Distribution
○ Smart grid: maintaining an equilibrium across energy supply and demand
■ Renewable and non-renewable energy production/demand forecasting
■ https://towardsdatascience.com/how-machine-learning-can-transform-the-ener
gy-industry-caaa965e282a (devices: synchrophasers)
● Consumption
○ Fleet performance optimization (cars, trucks, planes, etc.)
○ Commercial manufacturing sites, office buildings
○ Public infrastructure
○ Private households
Use Cases: Energy Sector

4. ● Event collection
○ From production systems
○ From consumption locations
○ Absence check: requires static
equipment inventory data
● Event monitoring
○ Outages
○ Trends
○ Anomalies
■ Was there actually an outage?
Use Cases: Analysis
● Forecasting / prediction
○ Combination of
■ Current events
■ Historic events
■ Non-event data like models,
weather data, road conditions,
etc.
● Off-line analysis
○ Combination with non-event data
● Archiving
○ Long duration analysis
○ Regulatory requirements

5. Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

6. Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

7. Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

8. Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

9. Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

10. Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

11. Google Cloud: Regions and Zones
https://cloud.google.com/about/locations

12. Google Cloud: Global Network
https://cloud.google.com/about/locations#network

13. Google Cloud: Cloud Services
https://cloud.google.com/-select“Products”

14. Google Cloud: Cloud Services
https://github.com/gregsramblings/google-cloud-4-words/bl
ob/master/DarkPoster-lowres.png

15. Google Cloud: Select Services
BigQuery (https://cloud.google.com/bigquery/)
● Cloud native analytics database
● Columnar - SQL - large scale (PB) - multi-regional service
Cloud Spanner (https://cloud.google.com/spanner/)
● Cloud native global relational database
● Relational - SQL - consistent - linear scaling - multi-regional and intercontinental service
Coldline Storage (https://cloud.google.com/storage/docs/storage-classes)
● Cloud native storage - long term storage
Cloud Machine Learning Engine (https://cloud.google.com/ai-platform)
● Cloud native ML engine - online and batch predictions

16. InfluxDB - TICK Architecture
https://www.influxdata.com/time-series-platform/

17. Cloud Native IoT Event Processing Architecture
Gateway

18. Cloud Native IoT Event Processing Architecture
Gateway
Cloud IoT
Core

19. Cloud Native IoT Event Processing Architecture
Gateway
Cloud IoT
Core
TICK stack

20. Cloud Native IoT Event Processing Architecture
Gateway
Offline Analytics
Cloud IoT
Core
TICK stack
Cloud
Pub/Sub
Cloud
Dataflow
Big
Query
Cloud
SQL
Cloud
Dataflow

21. Cloud Native IoT Event Processing Architecture
Gateway
Offline Analytics
Application Systems
Cloud IoT
Core
TICK stack
Cloud
Pub/Sub
Cloud
Dataflow
Big
Query
Cloud
SQL
Cloud
Dataflow
Cloud
Pub/Sub
Cloud
Dataflow
Cloud
Spanner
Kubernet
es
Engine

22. Cloud Native IoT Event Processing Architecture
Gateway
Offline Analytics
Application Systems
Machine Learning
Cloud IoT
Core
TICK stack
Cloud
Pub/Sub
Cloud
Dataflow
Big
Query
Cloud
SQL
Cloud
Dataflow
Cloud
Pub/Sub
Cloud
Dataflow
Cloud
Spanner
Kubernet
es
Engine
Cloud
Pub/Sub
Cloud
Run
Cloud
Storage
AI
Platform

23. Cloud Native IoT Event Processing Architecture
Gateway
Offline Analytics
Application Systems
Machine Learning
Archiving
Cloud IoT
Core
TICK stack
Cloud
Pub/Sub
Cloud
Dataflow
Big
Query
Cloud
SQL
Cloud
Dataflow
Cloud
Pub/Sub
Cloud
Dataflow
Cloud
Spanner
Kubernet
es
Engine
Cloud
Pub/Sub
Cloud
Run
Cloud
Storage
AI
Platform
Cloud
Run
Cloud
Storage
Cloud
Schedul
er

24. Summary
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Offline
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

25. Thank You!