Overview

A production-style streaming data platform that moves live e-commerce orders from Kafka into an analytics-ready warehouse using the medallion (bronze/silver/gold) architecture. Orders published to Kafka land in a Parquet bronze layer on S3-compatible storage via PySpark Structured Streaming. Batch stages clean and standardize the data into silver, then model a gold star schema, customer, product, and date dimensions around a central sales fact table, ready for BI and ML workloads.

Highlights

• Real-time ingestion with Kafka + PySpark Structured Streaming
• Medallion architecture on S3-compatible object storage (Parquet)
• Gold star schema modeled for analytics (facts + dimensions)
• Batch stages orchestrated with an Airflow DAG, all running on Docker Compose
• Postgres as the serving warehouse

Stack: Kafka · PySpark · Airflow · Postgres · Docker Compose

Links

• Code: github.com/saaim12/etl_pipeline
• Write-up: Building a real-time e-commerce ETL pipeline with Kafka, Spark and Airflow