From 05603382a5ef50beedd6428251dd12d6d733515c Mon Sep 17 00:00:00 2001 From: Tobias Wietelmann Date: Tue, 28 Jul 2026 22:27:19 +0200 Subject: [PATCH 1/2] Finalize portfolio README --- README.md | 370 +++++++++++++++++++++++------------------------------- 1 file changed, 154 insertions(+), 216 deletions(-) diff --git a/README.md b/README.md index d82d7b4..72d2587 100644 --- a/README.md +++ b/README.md @@ -2,35 +2,39 @@ [![Python quality](https://github.com/DataTideHH/python-data-basics/actions/workflows/python-quality.yml/badge.svg)](https://github.com/DataTideHH/python-data-basics/actions/workflows/python-quality.yml) -**Python 3.12 · pandas · data quality · reporting · matplotlib · pytest · Ruff · Jupyter · GitHub Actions** +**Python 3.12 · pandas · data quality · KPI reconciliation · reporting · Jupyter · pytest · Ruff · GitHub Actions** -This repository is a compact, tested foundation for reproducible Python Data/BI workflows. It combines environment setup, deterministic pandas transformations, an auditable data-quality workflow, reconciled reporting outputs, clean notebooks and cross-platform CI. +A compact, tested Data/BI workflow that turns an intentionally imperfect CSV into validated records, auditable rejections, reconciled KPIs, reporting summaries, charts and a clean verification notebook. -It is part of my DataTideHH portfolio during the IHK retraining program in Data and Process Analysis. The scope remains deliberately bounded: this is a reusable learning and verification baseline, not a production ETL platform, predictive-model showcase or enterprise reporting solution. +This repository is part of the DataTideHH portfolio for my IHK retraining in Data and Process Analysis. Its purpose is not to imitate an enterprise platform. It demonstrates a small workflow that can be inspected, executed, tested and explained end to end. + +> **Portfolio status:** Core scope complete. Future changes are limited mainly to dependency compatibility, defect correction and documentation clarity. See [`docs/project-status.md`](docs/project-status.md). --- -## Project at a Glance +## What This Demonstrates -| Area | Current implementation | +| Capability | Evidence in this repository | |---|---| -| Python baseline | Python 3.12-compatible environment and deterministic pandas sanity check | -| Dependency model | Runtime, notebook, optional ML and development groups in `pyproject.toml` | -| Data-quality workflow | CSV input, schema checks, controlled conversion, rejection reasons, cleaning, KPIs and export | -| Reporting workflow | KPI reconciliation, rejection-reason summary, control totals and Matplotlib SVG charts | -| Notebook verification | Clean reporting notebook with tables, charts and explicit assertions | -| Testing | pytest coverage for baseline logic, quality rules, reporting reconciliation and notebook hygiene | -| Code quality | Ruff linting and formatting plus Python bytecode compilation | -| Continuous integration | Complete Ubuntu 24.04 and Windows 2025 matrix with Python 3.12 | -| Credential safety | Synthetic/public-safe inputs; local environments, secrets and generated outputs excluded | - -## End-to-End Flow +| Reproducible Python setup | Python 3.12 project metadata, explicit dependency groups and platform-specific setup documentation | +| Data-quality controls | Required columns, controlled type conversion, missing values, ranges, duplicate handling and row-level rejection reasons | +| Data lineage | Original CSV row retained as `source_row` in validated and rejected outputs | +| KPI logic | Module-level result counts, learner counts, average scores, pass/fail counts and pass rates | +| Reconciliation | Persisted KPI output is recalculated from cleaned records and rejected when values differ | +| Reporting | Deterministic summaries plus Matplotlib SVG charts | +| Notebook discipline | Clean notebooks without committed outputs, execution counts, local paths or IDE timestamps | +| Automated verification | pytest, Ruff, bytecode compilation and end-to-end execution on Ubuntu and Windows | +| Scope discipline | Synthetic data, explicit limitations and no production-scale claims | + +--- + +## End-to-End Workflow ```text data/raw/training_results.csv │ ▼ -data-quality validation and cleaning +validation and controlled cleaning │ ├── cleaned_results.csv ├── rejected_results.csv @@ -38,7 +42,7 @@ data-quality validation and cleaning └── quality_report.json │ ▼ -reporting reconciliation and presentation +KPI recalculation and reconciliation │ ├── rejection_reason_summary.csv ├── reporting_summary.json @@ -46,50 +50,42 @@ reporting reconciliation and presentation └── pass_rate_by_module.svg │ ▼ -clean reporting notebook with repeated control assertions +clean reporting notebook with repeated assertions ``` -The reporting layer does not trust a persisted KPI file blindly. It recalculates module KPIs from the cleaned records and fails when values differ. +The reporting layer does not trust `module_kpis.csv` blindly. It derives the expected KPIs again from `cleaned_results.csv` and stops if the persisted and recalculated values differ. --- -## Quick Start +## Five-Minute Portfolio Review -Detailed platform instructions are in [`docs/setup.md`](docs/setup.md). +A technical reviewer can verify the main workflow without reading every example module. -### Windows PowerShell +### 1. Install the development environment + +#### Windows PowerShell ```powershell py -3.12 -m venv .venv & ".\.venv\Scripts\Activate.ps1" python -m pip install --upgrade pip python -m pip install -r requirements-dev.txt -python main.py ``` -### macOS and Linux +#### macOS or Linux ```bash python3.12 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -r requirements-dev.txt -python main.py ``` -On the Intel iMac used for local portfolio work, Python 3.12 is available at `/usr/local/bin/python3.12`. - ---- - -## Data-Quality Workflow +Detailed setup and troubleshooting are documented in [`docs/setup.md`](docs/setup.md). -The tested workflow in [`data_quality/`](data_quality/) processes: +### 2. Run the quality workflow -```text -data/raw/training_results.csv -``` - -### Run on Windows PowerShell +#### Windows PowerShell ```powershell python -m data_quality ` @@ -97,7 +93,7 @@ python -m data_quality ` --output ".ci-output/data-quality" ``` -### Run on macOS or Linux +#### macOS or Linux ```bash python -m data_quality \ @@ -105,55 +101,9 @@ python -m data_quality \ --output .ci-output/data-quality ``` -### Generated files - -```text -.ci-output/data-quality/ -├── cleaned_results.csv -├── rejected_results.csv -├── module_kpis.csv -└── quality_report.json -``` - -### Implemented validation rules - -- required-column validation -- missing-value detection -- strict ISO date parsing -- controlled numeric conversion -- positive `max_score` -- non-negative score and pass threshold -- score not above maximum -- pass threshold not above maximum -- exact duplicate removal -- conflicting duplicate rejection -- identifier and whitespace normalisation - -Invalid rows are not silently discarded. They remain auditable in `rejected_results.csv` with explicit pipe-separated reason codes. +### 3. Run reconciled reporting -Accepted records receive: - -- `source_row` for lineage to the raw CSV -- `score_percentage` -- non-null Boolean `passed` - -The committed fixture contains 15 raw rows and produces: - -```text -accepted rows: 8 -rejected rows: 7 -exact duplicate rows removed: 1 -``` - -Detailed rules and boundaries are documented in [`docs/data-quality-workflow.md`](docs/data-quality-workflow.md). - ---- - -## Reporting Workflow - -Run reporting after the data-quality output exists. - -### Windows PowerShell +#### Windows PowerShell ```powershell python -m reporting ` @@ -161,7 +111,7 @@ python -m reporting ` --output ".ci-output/reporting" ``` -### macOS and Linux +#### macOS or Linux ```bash python -m reporting \ @@ -169,157 +119,126 @@ python -m reporting \ --output .ci-output/reporting ``` -### Generated files +### 4. Run the complete automated checks -```text -.ci-output/reporting/ -├── average_score_by_module.svg -├── pass_rate_by_module.svg -├── rejection_reason_summary.csv -└── reporting_summary.json +```bash +python -m pytest +python -m ruff check main.py data_quality reporting examples tests +python -m ruff format --check main.py data_quality reporting examples tests ``` -### Verified control totals +--- + +## Verified Control Totals + +The committed synthetic fixture contains 15 source rows with deliberately introduced quality problems. -| Control | Expected value | +| Control | Verified value | |---|---:| -| Modules | 4 | -| Accepted results | 8 | +| Input rows | 15 | +| Accepted rows | 8 | | Rejected rows | 7 | +| Exact duplicate rows removed | 1 | +| Modules | 4 | | Overall average score | 70.00% | | Overall pass rate | 62.50% | | Distinct rejection reasons | 7 | | KPI reconciliation | passed | -The reporting workflow stops with a non-zero exit code when an expected input file is missing, a required column is absent or persisted module KPIs differ from the fresh calculation. - -Detailed behaviour is documented in [`docs/reporting-notebook.md`](docs/reporting-notebook.md). +These numbers are automated control values for the fixture, not claims about real learners or business operations. --- -## Verified Reference Charts +## Data-Quality Rules -The repository includes compact SVG snapshots for the committed synthetic fixture. The CI workflow generates fresh Matplotlib SVG files from the current outputs on every run. +The package in [`data_quality/`](data_quality/) implements: -### Average score +- required-column validation +- missing-value detection +- strict ISO date parsing +- controlled numeric conversion +- positive `max_score` +- non-negative scores and pass thresholds +- scores not exceeding their maximum +- pass thresholds not exceeding their maximum +- exact duplicate removal +- rejection of conflicting records sharing one result ID +- identifier and whitespace normalisation -![Average score by module](docs/assets/average-score-by-module.svg) +Invalid rows are not silently dropped. They remain visible in `rejected_results.csv` with explicit pipe-separated reason codes. -### Pass rate +Accepted records include: -![Pass rate by module](docs/assets/pass-rate-by-module.svg) +- `source_row` for lineage to the raw CSV +- normalised identifiers and module names +- `score_percentage` +- a non-null Boolean `passed` -These charts are descriptive controls for a small synthetic dataset, not statistical evidence about real learners or business operations. +The complete contract is documented in [`docs/data-quality-workflow.md`](docs/data-quality-workflow.md). --- -## Reporting Notebook - -[`notebooks/reporting_verification.ipynb`](notebooks/reporting_verification.ipynb) reads the generated data-quality outputs and performs the same reporting checks through the reusable package. - -It contains: +## Reporting and Visual Evidence -1. verified module KPI table -2. rejection-reason summary -3. average-score chart -4. pass-rate chart -5. explicit assertions for the expected control totals -6. final `Reporting notebook verification passed.` marker +The package in [`reporting/`](reporting/) validates the generated files, recalculates KPIs, summarises rejection reasons and writes deterministic reporting outputs. -The committed notebook contains no outputs, execution counts, local paths or IDE timestamps. GitHub Actions executes a temporary copy only after the data-quality and reporting command-line workflows succeed. +### Average score by module -[`dataspell_test.ipynb`](dataspell_test.ipynb) remains the smaller environment and import check. - ---- - -## Baseline Entry Point - -[`main.py`](main.py) is a separate deterministic environment and pandas sanity check. It reports the active interpreter and direct runtime package versions, validates a tiny DataFrame and calculates a stable category summary. +![Average score by module](docs/assets/average-score-by-module.svg) -This separates environment verification from the row-level data-quality and reporting workflows. +### Pass rate by module ---- +![Pass rate by module](docs/assets/pass-rate-by-module.svg) -## Additional Example Modules +Fresh SVG files are generated by the workflow during every CI run. The committed reference charts show the expected result for the synthetic fixture. -- [`examples/01_csv_pandas_basics.py`](examples/01_csv_pandas_basics.py) — CSV and grouped pandas operations -- [`examples/02_json_basics.py`](examples/02_json_basics.py) — nested JSON normalisation -- [`examples/03_api_request_basics.py`](examples/03_api_request_basics.py) — public Open-Meteo request without credentials -- [`examples/04_ollama_local_api_basics.py`](examples/04_ollama_local_api_basics.py) — optional localhost-only JSON request -- [`examples/optional/logistic_regression_basics.py`](examples/optional/logistic_regression_basics.py) — bounded scikit-learn API example without a model-quality claim +Detailed reporting behaviour is documented in [`docs/reporting-notebook.md`](docs/reporting-notebook.md). --- -## Dependency Model - -`pyproject.toml` is the source of truth. - -| Installation | Included scope | -|---|---| -| `python -m pip install -e .` | Runtime baseline, data quality and reporting | -| `python -m pip install -e ".[notebook]"` | Runtime plus Jupyter | -| `python -m pip install -e ".[ml]"` | Runtime plus optional scikit-learn example | -| `python -m pip install -e ".[dev]"` | Runtime plus pytest and Ruff | -| `python -m pip install -r requirements-dev.txt` | Complete CI-equivalent environment | - -Installed command-line entry points: +## Notebook Verification -```text -python-data-quality -python-data-reporting -``` +[`notebooks/reporting_verification.ipynb`](notebooks/reporting_verification.ipynb) reads generated workflow outputs rather than embedding a second copy of the source data. -The requirement files remain small wrappers rather than machine-specific freezes of every transitive package. +It displays and verifies: ---- - -## Local Quality Checks - -```bash -python -m compileall -q main.py data_quality reporting examples tests -python -m ruff check main.py data_quality reporting examples tests -python -m ruff format --check main.py data_quality reporting examples tests -python -m pytest -python main.py -python -m data_quality --input data/raw/training_results.csv --output .ci-output/data-quality -python -m reporting --input .ci-output/data-quality --output .ci-output/reporting -python examples/optional/logistic_regression_basics.py -``` +1. reconciled module KPIs +2. rejection-reason counts +3. average-score chart +4. pass-rate chart +5. expected reporting control totals +6. a final verification marker -Execute the notebooks separately: +[`dataspell_test.ipynb`](dataspell_test.ipynb) remains a smaller interpreter and package-import check. -```bash -jupyter nbconvert --to notebook --execute dataspell_test.ipynb --output environment-check.executed.ipynb --output-dir .ci-output --ExecutePreprocessor.timeout=120 -jupyter nbconvert --to notebook --execute notebooks/reporting_verification.ipynb --output reporting-verification.executed.ipynb --output-dir .ci-output --ExecutePreprocessor.timeout=120 -``` +Both committed notebooks are kept free of outputs, execution counts, absolute local paths and IDE-specific timestamps. CI executes temporary copies only. --- ## Continuous Integration -The workflow in [`.github/workflows/python-quality.yml`](.github/workflows/python-quality.yml) runs on: +The workflow in [`.github/workflows/python-quality.yml`](.github/workflows/python-quality.yml) runs with Python 3.12 on: - Ubuntu 24.04 - Windows 2025 -- Python 3.12 - -Each matrix job: - -1. installs the project and optional quality groups -2. compiles Python sources -3. runs Ruff lint and format checks -4. runs the complete pytest suite -5. executes `main.py` -6. executes the full data-quality workflow -7. verifies the 15/8/7 source control totals -8. executes the reporting workflow -9. verifies KPI reconciliation and reporting totals -10. executes the optional ML example -11. executes both clean notebooks -12. uploads short-lived verified data, reporting, chart and notebook artefacts -13. enforces the final quality gate - -The workflow is quality assurance, not deployment or release automation. + +Each matrix job performs: + +1. dependency installation +2. bytecode compilation +3. Ruff lint and format checks +4. the complete pytest suite +5. the deterministic baseline entry point +6. the full data-quality workflow +7. source control-total verification +8. reporting and KPI reconciliation +9. reporting control-total and SVG verification +10. the bounded optional ML example +11. both notebook executions +12. short-lived artifact upload +13. a final quality gate + +This is a quality-assurance workflow, not a deployment or release pipeline. --- @@ -330,29 +249,16 @@ python-data-basics/ ├── .github/workflows/python-quality.yml ├── data/raw/training_results.csv ├── data_quality/ -│ ├── __init__.py -│ ├── __main__.py -│ └── workflow.py ├── reporting/ -│ ├── __init__.py -│ ├── __main__.py -│ └── workflow.py -├── notebooks/ -│ └── reporting_verification.ipynb +├── notebooks/reporting_verification.ipynb ├── docs/ │ ├── assets/ -│ │ ├── average-score-by-module.svg -│ │ └── pass-rate-by-module.svg │ ├── data-quality-workflow.md +│ ├── project-status.md │ ├── reporting-notebook.md │ └── setup.md ├── examples/ ├── tests/ -│ ├── test_data_quality_workflow.py -│ ├── test_main.py -│ ├── test_notebook_hygiene.py -│ ├── test_optional_ml.py -│ └── test_reporting_workflow.py ├── dataspell_test.ipynb ├── main.py ├── pyproject.toml @@ -363,25 +269,57 @@ python-data-basics/ --- +## Supporting Examples + +The repository retains a few bounded learning examples without presenting them as the main portfolio result: + +- [`examples/01_csv_pandas_basics.py`](examples/01_csv_pandas_basics.py) — CSV and grouped pandas operations +- [`examples/02_json_basics.py`](examples/02_json_basics.py) — nested JSON normalisation +- [`examples/03_api_request_basics.py`](examples/03_api_request_basics.py) — public API request without credentials +- [`examples/04_ollama_local_api_basics.py`](examples/04_ollama_local_api_basics.py) — optional localhost-only request +- [`examples/optional/logistic_regression_basics.py`](examples/optional/logistic_regression_basics.py) — scikit-learn API demonstration without a model-quality claim + +[`main.py`](main.py) is a deterministic environment and pandas sanity check, separate from the substantive data-quality and reporting workflow. + +--- + ## Data and Credential Safety -Only synthetic learning data and public endpoints belong in this repository. The committed training-results file contains no real learners or personal information. +Only synthetic learning data and public endpoints belong in this repository. The committed fixture contains no real learner, customer or company data. + +Excluded content includes: -Excluded content includes local environments, `.env` files, API keys, OAuth tokens, credential downloads, personal/customer data, IDE metadata, caches, executed notebooks and generated workflow outputs. +- virtual environments and caches +- `.env` files and credentials +- API keys and OAuth tokens +- personal or customer data +- IDE metadata +- executed notebook copies +- generated local workflow outputs --- -## Current Boundaries +## Scope Boundaries This repository does not claim: -- a production Python package, ETL platform or semantic model +- a production package, ETL platform or semantic model - streaming or distributed processing -- production orchestration, observability or publication -- regulatory data validation -- a validated predictive model +- production orchestration, observability or deployment +- regulatory validation - an interactive dashboard or Power BI report +- a validated predictive model - statistical inference from the synthetic fixture -- deployment or cloud infrastructure -The complete workflow remains small enough to inspect, run, test and explain in an interview or technical review. +The value is the complete, testable chain from imperfect input through validation, auditable rejection, KPI reconciliation and documented reporting. + +--- + +## Related DataTideHH Projects + +- [`sql-server-docker-basics`](https://github.com/DataTideHH/sql-server-docker-basics) — reproducible SQL Server analytics lab, relational integrity and star schema +- [`hamburg-district-data-basics`](https://github.com/DataTideHH/hamburg-district-data-basics) — public Hamburg data and Power BI preparation +- [`open-meteo-germany-weather-ranking`](https://github.com/DataTideHH/open-meteo-germany-weather-ranking) — public API-to-CSV scoring workflow +- [`flask-country-data-api`](https://github.com/DataTideHH/flask-country-data-api) — validated ingestion, persistence and API delivery + +Portfolio overview: [datatidehh.github.io/DataTideHH](https://datatidehh.github.io/DataTideHH/) · Website: [datatidehh.de](https://datatidehh.de/) · LinkedIn: [linkedin.com/in/datatidehh](https://www.linkedin.com/in/datatidehh/) From 2e73b1519270ff6f091870574465b6f9bc44c521 Mon Sep 17 00:00:00 2001 From: Tobias Wietelmann Date: Tue, 28 Jul 2026 22:27:40 +0200 Subject: [PATCH 2/2] Document portfolio completion status --- docs/project-status.md | 68 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/project-status.md diff --git a/docs/project-status.md b/docs/project-status.md new file mode 100644 index 0000000..d6b787e --- /dev/null +++ b/docs/project-status.md @@ -0,0 +1,68 @@ +# Project Status + +## Status + +**Portfolio-complete / maintenance mode** + +The core scope of `python-data-basics` is complete as of July 2026. The repository now provides a small but coherent Python Data/BI workflow from imperfect CSV input through validation, auditable rejection, KPI reconciliation, reporting and notebook verification. + +The repository remains active and public, but it is no longer intended to grow through unrelated examples or speculative platform features. + +## Completed scope + +The completed portfolio scope includes: + +- reproducible Python 3.12 project setup +- explicit runtime, notebook, ML and development dependency groups +- deterministic pandas baseline check +- synthetic CSV fixture with documented quality defects +- required-column, type, missing-value, range and duplicate controls +- cleaned and rejected row exports +- row-level rejection reasons and source-row lineage +- module KPI aggregation +- independent KPI recalculation and reconciliation +- deterministic reporting summaries and SVG charts +- clean environment and reporting notebooks +- pytest, Ruff and bytecode checks +- end-to-end GitHub Actions on Ubuntu and Windows +- explicit data-safety and scope boundaries + +## Maintenance policy + +Further changes should normally be limited to: + +- compatibility updates for supported dependencies or GitHub Actions +- defect corrections +- clearer documentation +- security or credential-safety improvements +- small test additions required to protect existing behaviour + +Changes should preserve the existing control totals unless the fixture or business rules are deliberately revised and documented. + +## Out of scope + +The following additions are intentionally not planned for this repository: + +- larger artificial datasets added only to create scale +- machine learning claims based on the small synthetic fixture +- Streamlit or another dashboard layer +- Power BI files unrelated to the repository's current purpose +- cloud deployment or orchestration +- streaming or distributed processing +- database infrastructure already demonstrated in other DataTideHH repositories +- unrelated Python language exercises + +Those subjects belong in dedicated projects when there is a clear business question and sufficient evidence. + +## Portfolio role + +This repository serves as the tested Python foundation within the wider DataTideHH portfolio. It demonstrates that a bounded Data/BI workflow can be: + +1. understood from its input contract +2. executed locally +3. tested automatically +4. reconciled before reporting +5. reviewed through code, outputs, charts and documentation +6. explained without overstating its scale or maturity + +Related projects provide deeper evidence for SQL Server, APIs, public data, Power BI preparation and infrastructure-oriented data workflows.