Research Report: Event-Driven Data
Processing & Automation
Use Case 1: Netflix on AWS
🔗 Link: [Link]
📌 Overview: Netflix uses an event-driven architecture on AWS to process real-time data
streams for personalized recommendations and playback optimization.
🧩 Key Components / Technologies: AWS Lambda, Amazon Kinesis, Amazon S3, Amazon EMR
🔔 Triggering Mechanism: Playback events or user actions are sent to Amazon Kinesis Data
Streams.
🔄 Processing Flow: Kinesis receives the stream → Triggers Lambda → Data transformed →
Stored in S3 → Processed via EMR.
🤖 Automation & Deployment: AWS CloudFormation (IaC), CI/CD using AWS CodePipeline
and CodeDeploy.
Use Case 2: NetApp on Azure
🔗 Link: [Link]
📌 Overview: NetApp uses Azure Event Grid to automate infrastructure monitoring and
reporting.
🧩 Key Components / Technologies: Azure Event Grid, Azure Functions, Azure Blob Storage,
Power BI
🔔 Triggering Mechanism: Storage events (like file uploads) trigger Event Grid.
🔄 Processing Flow: Event Grid → Azure Functions → Data cleaning → Storage → Power BI for
reports.
🤖 Automation & Deployment: ARM templates (IaC), Azure DevOps CI/CD pipelines.
Use Case 3: AWS Blog - Real-Time Pipeline
🔗 Link: [Link]
pipeline-with-aws-lambda-amazon-kinesis-and-amazon-s3/
📌 Overview: Shows a real-time ingestion pipeline using Lambda and Kinesis to collect,
transform, and store data.
🧩 Key Components / Technologies: AWS Lambda, Amazon Kinesis, Amazon S3, Amazon
CloudWatch
🔔 Triggering Mechanism: Kinesis triggers Lambda whenever new data arrives.
🔄 Processing Flow: Raw data → Kinesis → Lambda processes → S3 storage → CloudWatch
monitoring.
🤖 Automation & Deployment: CloudFormation (IaC), CI/CD using CodePipeline +
CodeCommit.
Use Case 4: Azure IoT Event Streaming
🔗 Link: [Link]
azure-functions-and-event-hubs/
📌 Overview: Uses Event Hubs and Functions for streaming telemetry data from IoT devices,
with real-time monitoring.
🧩 Key Components / Technologies: Azure Event Hubs, Azure Functions, Azure Blob Storage,
Azure Monitor
🔔 Triggering Mechanism: Event Hubs receives messages → triggers Functions.
🔄 Processing Flow: Devices → Event Hub → Azure Functions → Process → Store → Monitor.
🤖 Automation & Deployment: IaC using Bicep/ARM templates, Azure DevOps pipelines.
Use Case 5: AWS Step Functions + QuickSight
🔗 Link: [Link]
aws-step-functions-and-amazon-quicksight/
📌 Overview: Automated daily business reporting using Step Functions and QuickSight
dashboards.
🧩 Key Components / Technologies: AWS Step Functions, AWS Lambda, Amazon QuickSight,
Amazon S3
🔔 Triggering Mechanism: CloudWatch Event triggers Step Functions at scheduled time.
🔄 Processing Flow: Step Functions → Lambdas → Data fetch/process → S3 → QuickSight.
🤖 Automation & Deployment: AWS CDK (IaC), CI/CD with GitHub Actions.
Use Case 6: Event-Driven Architecture on Azure
🔗 Link: [Link]
event-driven-architecture-on-azure/ba-p/3843858
📌 Overview: Describes a full event-driven microservices system on Azure.
🧩 Key Components / Technologies: Azure Event Grid, Azure Logic Apps, Azure Functions,
Azure Data Lake
🔔 Triggering Mechanism: REST APIs and user actions trigger Event Grid → logic flows.
🔄 Processing Flow: Event Grid → Logic Apps → Functions → Data Lake → Analytics.
🤖 Automation & Deployment: Terraform (IaC), CI/CD using Azure DevOps.
Architecture & Justification
High-Level Architecture Diagram
The following diagram represents the high-level architecture of the event-driven data
processing pipeline implemented on AWS:
[Insert Diagram Here — Can be drawn using tools like [Link] or Lucidchart and exported
to the final submission PDF]
Justification of Design Choices
The architecture is designed to be modular, serverless, and scalable. AWS was chosen due to
its mature ecosystem and support for event-driven services.
Each component of the pipeline has been selected based on its capability to support
automation, low-latency event processing, and seamless integration.
Automation, Deployment, and Reporting Flow
Infrastructure is provisioned using Terraform (Infrastructure as Code), which allows
repeatable and version-controlled deployments. The application
code and infrastructure are stored in a GitHub repository and integrated with GitHub
Actions for CI/CD. When new data files are uploaded to an S3 bucket,
they trigger an AWS Lambda function. The function processes the data, stores the
cleaned/aggregated output in another S3 bucket, and optionally sends
notifications or logs to CloudWatch. Daily summary reports are generated using scheduled
triggers (CloudWatch Events) that invoke reporting Lambdas, which
aggregate data and store reports.
Fault-Tolerance and Scalability Considerations
- **Fault Tolerance**: The use of AWS Lambda ensures built-in retry mechanisms. Data is
stored in S3 which is durable and redundant. CloudWatch monitors
failures and sends alerts.
- **Scalability**: AWS services like Lambda and S3 scale automatically with load. Kinesis (if
used for streaming) handles large-scale ingestion seamlessly.
