scd-type-2

End-to-end lakehouse pipeline for retail data: Postgres to Databricks (query-based incremental ingestion), dbt (incremental models, SCD2, metadata-driven OBT), Airflow orchestration, S3 secondary ingestion, CI-gated on dbt tests.

Build a data warehouse from scratch, including full load, daily incremental load, design schema, SCD Type 1 and 2.

This project simulates a real-world enterprise data migration and modernization strategy. It extracts transactional data from a simulated "On-Premise" environment (hosted on AWS EC2), performs heavy distributed processing using a Hadoop/Spark cluster, and ultimately serves the data via a Cloud-Native, serverless architecture to optimize costs .

Enterprise-grade Microsoft Fabric Lakehouse project implementing Medallion architecture (Bronze, Silver, Gold) for financial transactio

A Data Warehousing project for retail sales using dimension modelling best practices with SCD type 2 on AWS Redshift. Utilizing AWS Lambda, Glue Workflows and Python Shell jobs to create and automate an ELT pipeline where batch data coming into S3 is loaded onto Redshift and necessary transformations are performed to meet requirements.

End-to-end data lakehouse on Azure Databricks — Medallion Architecture, Star Schema, SCD Type 2, CI/CD, Data Quality Framework

End-to-end data engineering project using AWS S3, Snowflake, and dbt to implement Medallion Architecture with SCD Type 1 & Type 2 logic on Walmart sales data, followed by analytical visualizations using Seaborn and Plotly.

A focused dbt Core project demonstrating proficiency in the dbt workflow: from raw source transformation to a final analytics-ready table. Features implementation of snapshots, tests, macros, and models..

End-to-end Ride Sharing Data Engineering project using PySpark, Delta Lake, Databricks Structured Streaming, dbt, SCD Type 2 Snapshots, and Dimensional Modeling.

Developed a metadata-driven ETL framework that automates configurable data ingestion, SCD processing, and data quality using generic pipelines.

End-to-end Azure stock market analytics pipeline - ADF ForEach fetches daily OHLCV via Alpha Vantage API for MSFT/AAPL/GOOGL, implements SCD Type 2 with Delta MERGE, Delta time travel, incremental loading and PySpark broadcast variables.

End-to-end AWS data engineering pipeline using SQL Server, AWS Glue, S3 Delta Lake, Streamlit, CloudWatch observability, and GitHub Actions CI/CD.

End-to-end ETL Data Pipeline for Olist E-Commerce. Spark ingestion, Parquet transformation, Data Warehousing (Galaxy Schema) with SCD Type 2, incremental PostgreSQL loading, daily scheduling, Analytical SQL queries, and persistent Audit Logging.

dbt + Snowflake analytics warehouse using medallion architecture (Bronze/Silver/Gold) with SCD Type 2 snapshots

Built a metadata-driven Azure lakehouse that incrementally ingests Spotify-style SQL data with ADF, processes new Parquet files with Databricks Auto Loader, and publishes SCD-managed Delta facts and dimensions.

Real-time AWS ecommerce data pipeline using MSK Serverless, MSK Connect, S3, AWS Glue, Delta Lake, SCD Type 2, and Redshift Serverless.

E-commerce analytics pipeline using snowflake and dbt - staging, intermediate, marts, scd2 snapshots

Databricks Lakehouse | CRM + ERP Integration | Medallion Architecture | SCD Type 2 | Star Schema | Power BI Direct Lake

Kimball-style star schema with SCD Type 2 history tracking, built with dbt + DuckDB, tested and CI'd via GitHub Actions. 100% free stack.