Download to read offline
Uploaded byWillem Meints
Big data streaming with Apache Spark on Azure
The document discusses the integration of big data streaming with microservices and analytics using an event bus architecture, emphasizing the independence and scalability of microservices. It presents scenarios for generating annual sales reports and detecting anomalies using machine learning and data lakes, along with tools like Apache Spark for data processing. Additionally, it provides insights into deploying these solutions with Azure services, including Event Hubs and HDInsight.
Big data streaming with Apache Spark on Azure
- 1.
- 2.
- 3. Event bus Micro services •Multiple smaller services that scale independedely • Each service his own data store • Data flows between services through the event bus
- 4.
- 5. Data analytics challengeswith microservices • A complete picture is there, but spread over a vast landscape • Most data doesn’t come in a database • Data changes rapidly
- 6.
- 7. Scenario 1: Geta annual sales report • The goal is to get a complete picture of the situation • Data based on business events
- 8.
- 9.
- 10. Scenario 2: Detectanomalies • The goal is to detect anomalies on the website and prevent abuse • Machine learning needed to detect the anomalies • Data based on the data lake
- 11. Click stream collector Event bus Dataanalytics Data Lake
- 12. Click stream collector Event bus Dataanalytics Data Lake Model
- 13.
- 14.
- 15.
- 16. Event bus Data processing tool Distributed database Alerting Dashboarding Flowcontrol logic Cluster Manager
- 17. The Azure basedsolution
- 18. Azure Event HubHDInsight Azure Data Lake Alerting Dashboarding Azure App Services Cluster Manager
- 19. Demo A short introductioninto Apache Spark
- 20. Spark SQL SparkStreaming Machine Learning GraphX Apache Spark Core
- 22. Resilient Distributed DataSets Resilient Distributed Dataset Partition Record Record Partition Record Record
- 23. Stream Batches Processeddata Streams with Spark
- 24. Stream Batches Processeddata Streams with Spark Lists of RDDs
- 25. Demo Deploying Spark toAzure using HDInsight
- 26. Azure Event Hubs •Capable of streaming large volumes of data • SDK available in many languages • Ruby • Python • Java/Scala • C# • Apache Spark
- 27. Hoe werkt eenAzure Event Hub? Partition Partition Partition Consumer group Consumer group
- 28. Demo Using Azure EventHub with Spark
- 29. Tips for goingin production • When using streams, always have n+1 worker nodes • More partitions = more speed • Longer intervals is slower, but sometimes better
- 30. Thanks! Willem Meints Technical Evangelist/MicrosoftMVP @willem_meints