This repository is my data engineering journal to record and keep track of my learning process and insights.
- E-commerce Dashboard - v0.1.0 placeholder for v0.2.0 roadmap
- Github Analytics Data Engineering - A small scale end-to-end data engineering project with near real-time data update
- Awesome Data Engineering
- Data Engineer Handbook
- Awesome Public Dataset - Repository of public dataset for learning
- Data Engineering Zoomcamp
| No. | Module | Content |
|---|---|---|
| 1 | Modern Data Stack & SQL Foundations | - Analytics engineering in the data lifecycle - ELT vs. ETL in modern pipelines - SQL querying essentials - Analytical and window functions - OLTP vs. OLAP and warehouse basics - Normalization vs. denormalization trade-offs - Dimension and fact tables in PostgreSQL - Data warehousing principles |
| 2 | Data Modeling and ELT Concepts | - What is dbt, profiles, and project structure - Initialize and run dbt projects - Facts, dimensions, grain, and keys - Star vs. snowflake schema design - Design a retail star schema - Query KPIs using dimensional models - ELT workflows and data movement - Raw, staging, and mart layers - Staging tables and basic cleaning with SQL - Populate mart tables from staging |
| 3 | Building and Testing dbt Models | - Configuring sources and refs - Folder structure and naming conventions - Build staging and dimension models from sources - Visualize lineage and DAG in dbt Docs - Table, view, and incremental materializations - Seeds for static data and lookup tables - Build an incremental fact orders model - dbt tests (unique, not_null, relationships) - Custom SQL tests and documentation strategy - Run tests and analyze failures |
| No. | Module | Content |
|---|---|---|
| 1 | Containerization and Infrastructure as Code | - Introduction to GCP - Docker and Docker Compose - Running PostgreSQL with Docker - Infrastructure setup with Terraform |