1. Docker Compose Containers
   
   This shows Docker running our containers for the weather pipeline, Prometheus, and Grafana.

2. Grafana Dashboard
   
   Displays the pipeline metrics (data volumes, durations, API performance) once properly configured.

3. Airflow Login Page
   
   Airflow prompts for username and password to access its web UI.

4. Airflow DAGs Interface
   
   A list of DAGs (pipelines) that can be scheduled and monitored via Airflow.

The OpenWeatherMap Data Pipeline Engineering Project is a comprehensive data engineering solution to collect, process, and analyze weather data from the OpenWeatherMap API. It demonstrates a complete ETL pipeline with integrated monitoring, visualization, and multiple deployment options.

flowchart LR User([User]) API[OpenWeatherMap API] ETL[ETL Pipeline] DB[(Storage)] Monitor[Monitoring] Insight[Analytics] User --> API API --> ETL ETL --> DB DB --> Insight ETL <--> Monitor Insight --> User style API fill:#93c5fd,stroke:#2563eb,stroke-width:2px style ETL fill:#fde68a,stroke:#d97706,stroke-width:2px style Monitor fill:#d1fae5,stroke:#059669,stroke-width:2px style Insight fill:#fbcfe8,stroke:#db2777,stroke-width:2px 

1. Key Features
2. Technology Stack
3. Project Structure
4. Installing & Running (The Story)
5. Processing Pipeline
6. Data Analysis
7. Deployment Options
8. Monitoring
9. References
10. License

• Automated Weather Data Collection

  • Multi-city weather data extraction
  • Configurable sampling frequency
  • Resilient retry logic for API calls

• Robust Data Processing

  • Data cleaning, outlier handling
  • Derived metric computation

• Comprehensive Analytics

  • City-to-city comparisons
  • Temperature trend analysis
  • Weather pattern visualizations

• Enterprise-Grade Infrastructure

  • Docker containerization
  • Kubernetes orchestration
  • Optional Airflow scheduling

• Python 3.12+
• Docker / Kubernetes
• Prometheus / Grafana
• Apache Airflow (for advanced scheduling)

flowchart TD Python[Python 3.12] --> Pandas[Pandas] & Matplotlib[Matplotlib] Python --> Docker[Docker] Docker --> K8s[Kubernetes] Python --> Airflow[Airflow] Python --> Prometheus[Prometheus] Prometheus --> Grafana[Grafana] 

weather_data_pipeline/ ├── README.md ├── images/ │ ├── Grafana.png │ ├── apache_airflow_interface.png │ ├── apache_airflow_login.png │ └── docker-compose up.png ├── config/ │ └── config.yaml ├── data/ │ ├── raw/ │ ├── processed/ │ └── output/ ├── logs/ ├── requirements.txt ├── src/ │ ├── extract.py │ ├── transform.py │ ├── load.py │ ├── analyze.py │ └── utils.py ├── main.py ├── Dockerfile ├── docker-compose.yml ├── airflow/ │ └── weather_pipeline_dag.py ├── kubernetes/ │ └── deployment.yaml └── monitoring/ ├── prometheus.yml └── grafana-dashboard.json 

Below is a step-by-step flow illustrating how to install Docker, confirm everything is up and running, then transition to Kubernetes and Airflow.

1. On macOS, ensure Docker is not stuck:

   sudo launchctl remove com.docker.vmnetd

2. Verify Docker commands:

   docker pull hello-world docker run hello-world docker ps docker ps -a docker --version

3. Check existing images & remove any (optional):

   docker images docker rmi <IMAGE_ID>

4. Open Docker Desktop GUI (macOS):

   open -a Docker

1. Build the Docker image:

   docker build -t weather-pipeline .

2. Run the container with your API key:

   docker run --env-file .env weather-pipeline

3. Spin up services via Docker Compose:

   docker compose up

   If you get a port conflict (e.g., for port 9090), try:

   lsof -i :9090 kill -9 <PID> pkill -f prometheus docker compose up

4. Check Docker containers:

   docker ps

   Screenshot:
   

   You should see 3 containers:

   • Weather Pipeline (port 8000)
   • Prometheus (port 9090)
   • Grafana (port 3000)

Once Docker is up, open Grafana at http://localhost:3000.

• Username: admin
• Password: admin (by default, if unchanged)

Screenshot:

You’ll see panels for:

• Pipeline Duration
• Data Volumes (Records Processed & Data Points Extracted)
• API Performance

If you see “No data,” check your prometheus.yml or the pipeline’s main logs to ensure metrics are being scraped properly.

Screenshot (Login):

1. Install & Initialize Airflow:

   pip install apache-airflow airflow db init

2. Create Admin User:

   airflow users create \ --username admin \ --password admin \ --firstname Admin \ --lastname User \ --role Admin \ --email admin@example.com

3. Add the DAG:

   mkdir -p ~/airflow/dags cp airflow/weather_pipeline_dag.py ~/airflow/dags/

4. Start Airflow:

   airflow webserver --port 8080 airflow scheduler

5. Screenshot (DAGs):
   
   You should now see a list of DAGs (pipelines). Enable or trigger the relevant ones.

1. Start Minikube:

   minikube start

2. Apply the Weather Pipeline Deployment:

   kubectl apply -f kubernetes/deployment.yaml

3. (Optional) Create a secret for your API key:

   kubectl create secret generic weather-pipeline-secrets \ --from-literal=API_KEY=your_openweathermap_api_key

4. Check pods:

   kubectl get pods

Now your pipeline can run in a Kubernetes environment!

graph TD A[Input Config] --> B[Data Extraction] B --> C[Data Transformation] C --> D[Data Loading] C --> E[Data Analysis] E --> F[Visualization] F --> G[Results] 

1. Extract: Grab weather data from OpenWeatherMap
2. Transform: Clean, normalize, handle outliers
3. Load: Store processed data in local files or DB
4. Analyze: Generate city comparisons, identify trends
5. Visualize: Plot charts & graphs (Matplotlib, etc.)

mindmap root((Weather Analysis)) City Comparisons Temperature Humidity Wind Speed Temporal Analysis Daily Variation Long-term Trend Weather Conditions Condition Distribution Alerts Correlation Temperature-Humidity Wind-Temperature 

The pipeline can generate:

• Time-series plots (temperature trends)
• Comparison charts across multiple cities
• Correlation analyses (humidity vs. temperature)

• Local Docker: docker-compose up --build
• Kubernetes (Minikube): minikube start && kubectl apply -f deployment.yaml
• Airflow: Local scheduler and UI (port 8080)
• EC2: GitHub Actions CI/CD for continuous deployment

1. Prometheus collects pipeline metrics (port 9090).
2. Grafana visualizes metrics (port 3000).
   • Import monitoring/grafana-dashboard.json for a pre-built dashboard.
3. Alerts can be configured in Prometheus/Grafana to notify on pipeline failures or anomalies.

• OpenWeatherMap API Docs
• Docker Documentation
• Kubernetes Docs
• Prometheus Docs
• Grafana Docs
• Apache Airflow Docs

© 2025 Fahmi Zainal All rights reserved. This project and its contents are proprietary and confidential. Unauthorized copying, distribution, or modification of this software, via any medium, is strictly prohibited. For licensing inquiries, please contact the project maintainer.