Download to read offline
![Necessità: complex messages/entities, no mapping { "UserId": "123456", "UserName": "Mario Rossi", "EMail": "mario.rossi@somewhere.it", "StoreInfo": { "PreferredStore": { "PreferredStoreId": "12876", "PreferredStoreName": «Dovunque di Corso Vittorio" } } } { "UserId": "123456", "Reservations": [ { "LastUpdate": "", "CodiceArticolo": "", "StoreId": "", "AvailableParts": 4, "ProductWarehouseHash": "638AB6D83H" } ] } { "CodiceArticolo": "AK2457", "Descrizione": "Prodotto bello", "GiacenzaAttuale": 100, "Negozi": { "CodiceNegozio": { "GiacenzaAttuale": 20, "LastUpdate": "2021-04-26T10:20:30" } }, "LastUpdate": "2021-04-26T10:20:30" } { "CodiceArticolo": "AK2457", "Descrizione": «Prodotto", "Codici": { "B3812": { "Descrizione": "Prodotto bello", "Usage": "blablabla" } }, "DistintaBase": [ { "CodiceParte": "AK2457-P1" }, { "CodiceParte": "AK2457-P2" } ], "LastUpdate": "2021-04-26T10:20:30" }](https://image.slidesharecdn.com/2021-211121222823/75/Deploy-Microsoft-Azure-Data-Solutions-24-2048.jpg)
![Modeling challenge #1: To embed or reference? { "menuID": 1, "menuName": "Lunch menu", "items": [ {"ID": 1, "ItemName": "hamburger", "ItemDescription":...} {"ID": 2, "ItemName": "cheeseburger", "ItemDescription":...} ] } { "menuID": 1, "menuName": "Lunch menu", "items": [ {"ID": 1} {"ID": 2} ] } {"ID": 1, "ItemName": “hamburger", "ItemDescription":...} {"ID": 2, "ItemName": “cheeseburger", "ItemDescription":...}](https://image.slidesharecdn.com/2021-211121222823/75/Deploy-Microsoft-Azure-Data-Solutions-26-2048.jpg)
![Storage Cost Avg Record Size (KB) 1 Number of Records 100,000,000 Total Storage (GB) 100 Monthly Cost per GB $0.25 Expected Monthly Cost for Storage $25.00 Throughput Cost Operation Type Number of Requests per sec Avg RU's per Request RU's Needed Create 100 5 500 Read 400 1 400 Total RU/sec 900 Hourly Cost per 100 RU/sec $0.008 Monthly Cost per 100 RU/sec $6.00 Expected Monthly Cost for Throughput $54.00 Total Monthly Cost [Total Monthly Cost] = [Monthly Cost for Storage] + [Monthly Cost for Throughput] = $25 + $54 = $79 per month * pricing may vary by region; for up-to-date pricing, see: https://azure.microsoft.com/pricing/details/cosmos-db/ Pricing Example](https://image.slidesharecdn.com/2021-211121222823/75/Deploy-Microsoft-Azure-Data-Solutions-32-2048.jpg)
Uploaded byMarco Parenzan
Deploy Microsoft Azure Data Solutions
The document provides insights from Marco Parenzan, a Senior Solution Architect, on leveraging Azure for data management, including both relational and non-relational databases, IoT data handling, and Azure's cognitive services. It discusses the architectural frameworks for microservices, event processing, and Cosmos DB's capabilities, along with pricing and operational efficiencies. The presentation also touches upon the importance of managing costs and effective data governance within Azure environments.
Deploy Microsoft Azure Data Solutions
- 1.
- 2. Senior Solution Architect@ beanTech Microsoft Azure MVP Community Lead 1nn0va // Pordenone 1nn0va After Hour https://bit.ly/1nn0va-video Linkedin: https://www.linkedin.com/in/marcoparenzan/ Marco Parenzan
- 4. Da SQL Servera Azure SQL e oltre: le opportunità per i dati in Azure Dati relazionali in Azure Dati non relazionali Dati blob in Azure Architetture a Messaggi, Comandi ed Eventi in Azure Big Data in Azure Gestire i dati dell'IoT in Azure Servizi Cognitivi per i dati Sicurezza Agenda
- 6. • Polyglot Persistence •…non «spendere per spendere»… • No more SQL...only! • Microservizi • Ogni servizio porta con sé uno storage • «Next,Next,Next,Next,...» • Operations & DevOps • Quanto è «dato» e quanto è «servizio»? • Un confine sempre più labile • Questo è tutto ciò che è il cloud Alcuni temi
- 9.
- 11. Pricing Azure StorageBlobs Pricing | Microsoft Azure Files and API Events External Tables Azure Blobs
- 12. Storage Account IoT Hub Devices Events Ingest The batchpoint of view...
- 13. With no anyspecific request... what is IoT all about? Efficiency Anomalies Batch Streaming
- 14.
- 15.
- 16.
- 17.
- 18. • Azure StreamAnalytics • Azure Container Apps & co • Apache Spark (and again Synapse) Event Processing
- 19.
- 20. • ML.NET • Spark •ADX (and again Synapse) • Notebooks • Azure Functions • Cognitive Services • Anomaly Detector • Custom Vision Anomaly Detection
- 21.
- 22. • Azure loveBlobs • Developers love Blobs • Developers love services that handles Blobs as first class citizens • No relational • No file system Lesson learned
- 24. Necessità: complex messages/entities,no mapping { "UserId": "123456", "UserName": "Mario Rossi", "EMail": "mario.rossi@somewhere.it", "StoreInfo": { "PreferredStore": { "PreferredStoreId": "12876", "PreferredStoreName": «Dovunque di Corso Vittorio" } } } { "UserId": "123456", "Reservations": [ { "LastUpdate": "", "CodiceArticolo": "", "StoreId": "", "AvailableParts": 4, "ProductWarehouseHash": "638AB6D83H" } ] } { "CodiceArticolo": "AK2457", "Descrizione": "Prodotto bello", "GiacenzaAttuale": 100, "Negozi": { "CodiceNegozio": { "GiacenzaAttuale": 20, "LastUpdate": "2021-04-26T10:20:30" } }, "LastUpdate": "2021-04-26T10:20:30" } { "CodiceArticolo": "AK2457", "Descrizione": «Prodotto", "Codici": { "B3812": { "Descrizione": "Prodotto bello", "Usage": "blablabla" } }, "DistintaBase": [ { "CodiceParte": "AK2457-P1" }, { "CodiceParte": "AK2457-P2" } ], "LastUpdate": "2021-04-26T10:20:30" }
- 25. Column-family Document Graph Turnkey global distribution Elasticscale out of storage & throughput Guaranteed low latency at the 99th percentile Comprehensive SLAs Five well-defined consistency models Table API Key-value Cosmos DB’s API for MongoDB Azure Cosmos Db
- 26. Modeling challenge #1:To embed or reference? { "menuID": 1, "menuName": "Lunch menu", "items": [ {"ID": 1, "ItemName": "hamburger", "ItemDescription":...} {"ID": 2, "ItemName": "cheeseburger", "ItemDescription":...} ] } { "menuID": 1, "menuName": "Lunch menu", "items": [ {"ID": 1} {"ID": 2} ] } {"ID": 1, "ItemName": “hamburger", "ItemDescription":...} {"ID": 2, "ItemName": “cheeseburger", "ItemDescription":...}
- 27. hash(City ID) Beijing Seattle Paris Redmon d Shanghai Tianjin Berlin Austi n Range 1Range 2 Range 5 RU’s divided among Partitions
- 28. Multi-Master – Read/Writein any region Benefits • Write scalability around the world • Low latency (<10ms P99 for 1kb document) writes around the world • 99.999% High Availability around the world • Well-defined consistency models • Automatic conflict management
- 29. Common Change FeedScenarios
- 30. Request Units Each requestconsumes # of RU Approx. 1 RU = 1 read of 1 KB document Approx. 5 RU = 1 write of a 1KB document Query: Depends on query & documents involved GET POST PUT Query … = = = =
- 31. Operation Type #Requests per sec # RU's per Request RU's Needed Write Single Document 10,000 10 100,000 Top Query #1 700 100 70,000 Top Query #2 200 100 20,000 Top Query #3 100 100 10,000 Total RU/s 200,000 RU/s Estimating Required RU’s
- 32. Storage Cost Avg RecordSize (KB) 1 Number of Records 100,000,000 Total Storage (GB) 100 Monthly Cost per GB $0.25 Expected Monthly Cost for Storage $25.00 Throughput Cost Operation Type Number of Requests per sec Avg RU's per Request RU's Needed Create 100 5 500 Read 400 1 400 Total RU/sec 900 Hourly Cost per 100 RU/sec $0.008 Monthly Cost per 100 RU/sec $6.00 Expected Monthly Cost for Throughput $54.00 Total Monthly Cost [Total Monthly Cost] = [Monthly Cost for Storage] + [Monthly Cost for Throughput] = $25 + $54 = $79 per month * pricing may vary by region; for up-to-date pricing, see: https://azure.microsoft.com/pricing/details/cosmos-db/ Pricing Example
- 33. Service Bus Architettura genericaper User Story • Microservizio responsabile del dominio applicativo e della sua consistenza e persistenza • API di frontend per gestire le richieste delle singole user stories specifiche verso il microservizio • Gestione in memory con Redis • In caso di fail lettura da Cosmos • In caso di update • Aggiornamento di Redis • Accodamento in serviceBus • Batch aggiorna Cosmos per abbassare RU API #1 Micro Service Cosmos DB REDIS Caching Update Batch API #n Read Write Command Read Invoke Invalidate Cache Write Invoke Read Command
- 34.
- 35. • Azure loveNo SQL and Cosmos DB • Developers love No SQL and Json • Developer hates(?) Cosmos Db pricing? • Developers loves (fuffa style ) microservices • Everything has solution... Lesson learned
- 37. • È tuttoAzure! • Le dashboard, se si può, si fanno in Power BI (8€/utente/mese, ricordatevi, anche solo per visualizzare) • Governance dello sviluppo • Premium Capacity • Developers • Deployment BI? Power BI? Cosa c’entra con Azure?
- 38. • Developer maleducati •Power BI developer MOLTO maleducati Scenario
- 39. • PowerShell • AzureDevOps • Git non è pbix-friendly • Ma è comunque un repo versionato • E le pipeline sono un ottimo strumento di automation Ops, Deployments
- 40.
- 41. • DR ofa DataWarehouse: big problem • BIOps is not a practice Lesson learned
- 43. SQL virtual machinesManaged instances Azure SQL Databases • SQL Server surface area (vast majority) • Native virtual network support • Fully managed service • SQL Server and OS server access • Expansive SQL and OS version support • Automated manageability features for SQL Server • Hyperscale storage (up to 100TB) • Serverless compute • Fully managed service • Resource sharing between multiple databases to price optimize • Simplified performance management for multiple databases • Fully managed service
- 44. Azure SQL Database— Everything built-in •Business continuity High availability Automated backups Long term backup retention Geo-replication •Scale Advanced security Automatic tuning Built-in monitoring Built-in intelligence
- 45. Azure SQL: multi-facetedservice
- 46. On-demand flexible scale Operateat the true rhythm of your business Fully managed & intelligent Focus on your applications, not your infrastructure Cost-effective Pay for performance. Period. Adapts compute resources to the workload without sacrificing performance Automatically pauses and resumes Fully-managed and intelligent database service Built-in 99.99% availability Pay only for compute resources you consume, on a per-second basis Further optimize costs with configurable compute thresholds Best for unpredictable and intermittent workloads on single databases, such as: Dev/test E-commerce Line of Business
- 48.
- 49.
Editor's Notes
- #26 Azure Cosmos DB offers the first globally distributed, multi-model database service for building planet scale apps. It’s been powering Microsoft’s internet-scale services for years, and now it’s ready to launch yours. You can add Azure locations to your database anywhere across the world, at any time, with a single click. Cosmos DB will seamlessly replicate your data and make it highly available. Cosmos DB allows you to scale throughput and storage elastically, and globally! You only pay for the throughput and storage you need – anywhere in the world, at any time.
- #28 27
- #31 The number of RU’s each operation consumes depends on many factors which include: Document size Number of indexed fields Type of indexes Consistency model choice Not all queries will consume equal numbers of RU’s. Some operations are more computationally complex or require scans through more documents and therefore use more RU’s.
- #45 Business continuity enables your business to continue operating in the face of disruption, particularly to its computing infrastructure. High availability of Azure SQL Database guarantees your databases are up and running 99.99% of the time, no need to worry about maintenance/downtimes. 2 read replicas - GP 3 replicas, 1 read-scale replica, zone-redundant HA - BC Primary read/write replica + up to 4 read replicas - HS Automated backups are created and use Azure read-access geo-redundant storage (RA-GRS) to provide geo-redundancy. Also Point in Time restore Long term backup retention enables you to store specific full databases for up to 10 years. Geo-replication by creating readable replicas of your database in the same or different data center (region). You can manually failover to readable replicas Also Auto-failover groups allows an application to recover in case of a data center outage Scale by easily adding more resources (CPU, memory, storage) without long provisioning. Advanced security detects threats and vulnerabilities in your databases and enables you to secure your data. Automatic tuning analyzes your workload and provides you the recommendations that can optimize performance of your applications by adding indexes, removing unused indexes, and automatically fixing the query plan issues. Built-in monitoring capabilities enable you to get the insights into performance of your databases and workload, and troubleshoot the performance issues. Built-in intelligence automatically identifies the potential issues in your workload and provides you the recommendations that can help you to fix the problems.
- #47 Azure SQL Database serverless is our answer to cost-effectively resourcing these types of unpredictable and intermittent workloads. It is a dynamically scaling, on-demand version of Azure SQL Database that eliminates the complexity of resourcing and managing unpredictable workloads. Simply create a serverless database and connect your application, with no upfront resource configuration required. Serverless SQL databases automatically pause, resume and scale compute based upon your app’s requirements, making them particularly cost effective for variable or unpredictable workloads – or when you’re simply unsure of your requirements. With resources available on-demand, serverless SQL databases optimize costs with pay per-second billing that aligns with the app’s performance, so you only pay for the compute resources you use . Built upon the SQL Server architecture, serverless SQL databases are fully-managed, always up to date and highly available with a 99.99% uptime guarantee. Combined with built-in intelligence to optimize database performance and security, serverless SQL Databases help you be more productive so you can focus more on what you do best, building great apps faster and more cost effectively.