Downloaded 36 times
![Stream Analytics Query • Perform processing on data stream • Stream Analytics Query Language • SQL-like language SELECT * INTO [YourOutput] FROM [YourInput]](https://image.slidesharecdn.com/implementing-real-time-iot-stream-processing-in-azure-191121195423/75/Implementing-Real-Time-IoT-Stream-Processing-in-Azure-11-2048.jpg)
![Functions • Integrate Azure Machine Learning WITH sentiment AS ( SELECT text, sentiment1(text) as result FROM datainput ) SELECT text, result.[Score] INTO datamloutput FROM sentiment](https://image.slidesharecdn.com/implementing-real-time-iot-stream-processing-in-azure-191121195423/75/Implementing-Real-Time-IoT-Stream-Processing-in-Azure-19-2048.jpg)
Uploaded byChris Pietschmann (Microsoft MVP)
Implementing Real-Time IoT Stream Processing in Azure
This document discusses implementing real-time IoT stream processing using Azure Stream Analytics. It introduces the Lambda architecture pattern for processing real-time and batch data streams. Azure Stream Analytics is presented as a tool for real-time stream processing that can ingest data from sources like IoT Hub and Event Hubs and output to databases, services, and functions. The document demonstrates Stream Analytics queries, windowing functions, and integrating with Azure Machine Learning models. It also discusses running Stream Analytics on IoT Edge devices to analyze and filter data locally.
Implementing Real-Time IoT Stream Processing in Azure
- 1. Implementing Real-Time IoTStream Processing in Azure Chris Pietschmann cpietschmann@solliance.net
- 2. Lambda Architecture Azure StreamAnalytics Agenda
- 3. Lambda Architecture • Dataaggregation design patter • Real-Time Processing / Analytics • Fast / Hot path • Batch Processing • Slow / Cold Path
- 4.
- 5. Lambda Architecture Broker IoT Hub EventHub Stream Processor Stream Analytics HDInsight Spark Streaming Storage Cosmos DB SQL Database Service Bus Azure Data Lake Action Azure Functions
- 6. Azure Stream Analytics •Real-time stream processing • Stream millions of events per second • Multiple Input and Output Streams • Familiar SQL-like language • Serverless
- 7. Stream Analytics DataFlow Processing Data Output(s) Data Stream(s)
- 8. Azure Stream Analyticsin the Cloud DeliverIngest Continuous Intelligence/ Real-time analyticsLogs, Files, Media Customer data, Financial Transactions Weather data Business Applications Analyze Alerts and actions Dynamic Dashboarding Data Warehousing Storage / Archival Event Hubs, Service Bus, Azure Functions etc. Power BI SQL Data Warehouse SQL DB, Azure Data Lake Gen1 and Gen 2, Cosmos DB, Blob Storage etc. Kafka Reference Data (SQL DB, Blob store) Real-time scoring (Azure ML service) IoT Devices
- 9. Stream Analytics Inputs DataStream •Azure IoT Hub •Azure Event Hub Reference Data •Azure Blob Storage •Azure SQL Database
- 10. Stream Analytics Outputs •CosmosDB •SQL Database •Azure Table Storage •Azure Service Bus •Power BI •Azure Data Lake
- 11. Stream Analytics Query •Perform processing on data stream • Stream Analytics Query Language • SQL-like language SELECT * INTO [YourOutput] FROM [YourInput]
- 12. Stream Analytics Query Aggregate AVG,COUNT, Collect, CollectTOP, MAX, MIN, Percentile_Cont, Percentile_Disc, SUM, TopOne, VAR Analytic ISFIRST, LAG, LAST Array GetArrayLength, GetArrayElement, GetArrayElements Conversion CAST, GetType, TRY_CAST Geospatial CreateLineString, CreatePoint, CreatePolygon Date and Time DATEADD, DATEDIFF, DATENAME, DATEPART, DAY, MONTH, YEAR Mathematical ABS, CEILING, EXP, FLOOR, POWER, SIGN, SQUARE, SQRT Record GetRecordProperties, GetRecordPropertyValue String CONCAT, LEN, LOWER, UPPER, SUBSTRING, REGEXMATCH
- 13. Window Functions Temporal WindowFunctions •TumblingWindow •HoppingWindow •SlidingWindow
- 14.
- 15.
- 16.
- 17.
- 18. Functions • JavaScript UDF(user defined functions) // Convert Hex value to integer.function hex2Int(hexValue) { return parseInt(hexValue, 16); } SELECT time, UDF.hex2Int(offset) AS IntOffset INTO output FROM InputStream
- 19. Functions • Integrate AzureMachine Learning WITH sentiment AS ( SELECT text, sentiment1(text) as result FROM datainput ) SELECT text, result.[Score] INTO datamloutput FROM sentiment
- 20. Azure Stream Analyticson IoT Edge
- 21. Azure Stream Analyticson IoT Edge • Industrial IoT • Too much data to upload to cloud • Send aggregate, average, or only “significant” events where values changed • Examples: • Jet Engines – single flight can produce 1TB of data • Manufacturing – sensors can produce 1MB/s to 10MB/s of event data
- 22. Demo Setup Azure StreamAnalytics Input from Azure IoT Hub Output to Cosmos DB and Azure Functions Setup Lambda Architecture
- 23. © Microsoft Azure+ AI Conference All rights reserved. Thank You! Chris Pietschmann Microsoft MVP – Azure Solution Architect / Developer, Solliance Blog: Build5Nines.com Email: cpietschmann@solliance.net
- 24. © Microsoft Azure+ AI Conference All rights reserved. Please use EventsXD to fill out a session evaluation. Thank you!
- 25. Build5Nines Cloud & EnterpriseTechnology https://Build5Nines.com
Editor's Notes
- #2 Good morning! {Intro myself} {ask questions}
- #3 Look at Azure Sphere from security perspective We’re developers, building IoT solutions. Need to be sure to protect our company, our clients, and our solutions In this session, we’ll take a look at…
- #18 https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions#session-window
- #21 https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-edge