Downloaded 12 times
Uploaded byInfluxData
IoT Event Processing and Analytics with InfluxDB in Google Cloud | Christoph Bussler | Google, Inc
The document discusses IoT event processing and analytics specifically within the energy sector, utilizing InfluxDB on Google Cloud for various applications such as energy production monitoring and anomaly detection. It outlines the life cycle of an IoT event, covering aspects from event collection to forecasting and archiving, while highlighting the necessary cloud services like BigQuery and Cloud Spanner. The focus is on the tick architecture for real-time data handling and the integration of dynamic and static data sources.
IoT Event Processing and Analytics with InfluxDB in Google Cloud | Christoph Bussler | Google, Inc
- 1. Christoph Bussler October 3rd,2019 IoT Event Processing and Analytics with InfluxDB in Google Cloud
- 2. Agenda Energy Sector Energy sector use case analysis Lifecycle of an IoT event Google Cloud 1 2 3 4 5 6 TICK architecture Cloud native IoT event processing architecture
- 3. ● Production ○ Energyproduction systems monitoring and anomaly detection ■ Oil, gas, wind, hydro, solar and others ● Distribution ○ Smart grid: maintaining an equilibrium across energy supply and demand ■ Renewable and non-renewable energy production/demand forecasting ■ https://towardsdatascience.com/how-machine-learning-can-transform-the-ener gy-industry-caaa965e282a (devices: synchrophasers) ● Consumption ○ Fleet performance optimization (cars, trucks, planes, etc.) ○ Commercial manufacturing sites, office buildings ○ Public infrastructure ○ Private households Use Cases: Energy Sector
- 4. ● Event collection ○From production systems ○ From consumption locations ○ Absence check: requires static equipment inventory data ● Event monitoring ○ Outages ○ Trends ○ Anomalies ■ Was there actually an outage? Use Cases: Analysis ● Forecasting / prediction ○ Combination of ■ Current events ■ Historic events ■ Non-event data like models, weather data, road conditions, etc. ● Off-line analysis ○ Combination with non-event data ● Archiving ○ Long duration analysis
- 5. Life Cycle ofan IoT Event Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamic data sources Application Systems Short / Long term archive
- 6. Life Cycle ofan IoT Event Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamic data sources Application Systems Short / Long term archive
- 7. Life Cycle ofan IoT Event Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamic data sources Application Systems Short / Long term archive
- 8. Life Cycle ofan IoT Event Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamic data sources Application Systems Short / Long term archive
- 9. Life Cycle ofan IoT Event Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamic data sources Application Systems Short / Long term archive
- 10. Life Cycle ofan IoT Event Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamic data sources Application Systems Short / Long term archive
- 11. Google Cloud: Regionsand Zones https://cloud.google.com/about/locations/#regions-tab
- 12. Google Cloud: GlobalNetwork https://cloud.google.com/about/locations/#network-tab
- 13. Google Cloud: CloudServices https://cloud.google.com/-select“Products”
- 14. Google Cloud: SelectServices BigQuery (https://cloud.google.com/bigquery/) ● Cloud native analytics database ● Columnar - SQL - large scale (PB) - multi-regional service Cloud Spanner (https://cloud.google.com/spanner/) ● Cloud native global relational database ● Relational - SQL - consistent - linear scaling - multi-regional and intercontinental service Coldline Storage (https://cloud.google.com/storage/docs/storage-classes) ● Cloud native storage - long term storage Cloud Machine Learning Engine (https://cloud.google.com/ml-engine/) ● Cloud native ML engine - online and batch predictions
- 15. InfluxDB - TICKArchitecture https://www.influxdata.com/time-series-platform/
- 16. Cloud Native IoTEvent Processing Architecture Gateway
- 17. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core
- 18. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core TICK Stack
- 19. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core TICK Stack Cloud Pub/Sub BigQuery Cloud Dataflow Cloud SQL Cloud Dataflow
- 20. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core TICK Stack Cloud Pub/Sub BigQuery Cloud Dataflow Cloud SQL Cloud Dataflow Cloud Pub/Sub Cloud Spanner Kubernetes Engine Cloud Dataflow
- 21. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core TICK Stack Cloud Pub/Sub BigQuery Cloud Dataflow Cloud SQL Cloud Dataflow Cloud Pub/Sub Cloud Spanner Kubernetes Engine Cloud Dataflow
- 22. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core TICK Stack Cloud Pub/Sub BigQuery Cloud Dataflow Cloud SQL Cloud Pub/Sub Cloud Dataflow Cloud Pub/Sub Cloud Spanner Kubernetes Engine Cloud Run Cloud Storage AI Platform Cloud Dataflow
- 23. Cloud Native IoTEvent Processing Architecture Gateway Cloud IoT Core TICK Stack Cloud Pub/Sub BigQuery Cloud Dataflow Cloud SQL Cloud Run Cloud Storage Cloud Pub/Sub Cloud Dataflow Cloud Pub/Sub Cloud Spanner Kubernetes Engine Cloud Scheduler Cloud Run Cloud Storage AI Platform Cloud Dataflow
- 24. Summary Gateway Ingest Time Series Database Real-time Analytics Offline Analytics Static data sources Dynamicdata sources Application Systems Short / Long term archive
- 25.
- 26.