This ETL (Extract, Transform, Load) project employs several Python libraries, including Airflow, Soda, Polars, YData Profiling, DuckDB, Requests, Loguru, and Google Cloud to streamline the extraction, transformation, and loading of CSV datasets from the U.S. government's data repository at https://catalog.data.gov.
python docker bigquery airflow continuous-integration data-visualization astro soda data-quality-checks extract-transform-load github-actions duckdb polars looker-studio astro-python-sdk ydata-profiling
- Updated
Dec 10, 2023 - HTML