Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

The Goal of this project is to provide documentation for the Lakehouse Engine framework.

Example applications of spark-trend-calculus

🏂 A machine learning model that performs topic classification of news articles for media bias analysis. Final project for UC Berkeley MIDS 266 (Natural Language Processing)

Example applications of GDELT mass media intelligence data

This is a code sample repository for demonstrating how to perform Databricks Delta Table operations.

Project MEP: Meme Evolution programme. A terraformed multi-language library to do statistical experiments in Twitter.

End-to-end water data platform built with PySpark, a Medallion Lakehouse, and DataOps principles (CI/CD, Testing). A local-first, containerised data platform (Docker). A governed Medallion Lakehouse with (Data Quality), and DataHub (Governance). Features Medallion architecture, automated data quality, and CI/CD.

Distributed Computing with Spark

Pipeline de Engenharia de Dados (Databricks Free Edition) para o SCANIA Component X Dataset: ingestão via Volumes, Delta Lake e arquitetura Medalhão (Bronze→Silver→Gold), modelagem em Esquema Estrela e dashboards/SQL para manutenção preditiva

A framework that eliminates the dependency on Apache Spark by leveraging delta-rs for the creation and management of Delta Lake tables. This framework follows Medallion architecture.

TPC-H ETL pipeline on Databricks with PySpark and Delta Lake for ingestion, transformation, analysis, and a BI-ready denormalized warehouse.