[cemililik]

Summary

• v0.5.0 release — consolidates Phase 11 (data ingestion), Phase 11.5 (PII review fixes), Phase 12 (data curation maturity), and Phase 12.5 (data-curation polish) into a single comprehensive release.
• Phase 12.5 polish — four data-curation improvements: silent-failure guards, Presidio canonicalization, Croissant card cleanup, and review-feedback application.
• Site + user manuals — documentation refresh to reflect new data curation features (PII ML, all-mask, Croissant cards, ingestion improvements) in both EN and TR mirrors.

Changes at a glance

• forgelm/cli.py, forgelm/data_audit.py, forgelm/wizard.py — Phase 12.5 feature work + silent-failure guards
• tests/test_phase12_5.py (+528 lines) — comprehensive coverage for new behavior
• docs/usermanuals/{en,tr}/data/ — new bilingual user manuals for PII ML, all-mask, Croissant cards
• docs/roadmap/, CHANGELOG.md, README.md — release docs synchronized with merged state
• site/ — landing pages reflect v0.5.0 messaging
• pyproject.toml — version bump + extras adjustments

Test plan

• ruff format . && ruff check . clean
• pytest tests/ green (including new test_phase12_5.py)
• forgelm --config config_template.yaml --dry-run succeeds
• CI passes on the PR
• Bilingual doc pairs (EN/TR) verified structurally aligned
• Manual smoke: wizard BYOD flow + data audit pipeline

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added --all-mask (combined secrets+PII masking), --pii-ml (optional ML-based PII detection), --croissant (optional Croissant 1.0 dataset card), and a wizard “audit first” prompt after ingestion.
• Documentation
  • Consolidated phase notes into a single 0.5.0 release and added user guidance for the new flags, CLI examples, and wizard flow.
• Tests
  • Added comprehensive Phase 12.5 test coverage validating masking, audit wizard, Croissant output, and ML-PII behavior.

[qodo-code-review]

ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one.

[sourcery-ai]

Sorry @cemililik, your pull request is larger than the review limit of 150000 diff characters

[coderabbitai]

Warning

Rate limit exceeded

@cemililik has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 38 minutes and 57 seconds before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 049c566e-b57b-4899-9310-2d8805852117

📥 Commits

Reviewing files that changed from the base of the PR and between 502ff88 and a4b2bb2.

📒 Files selected for processing (3)

• CHANGELOG.md
• docs/usermanuals/en/data/pii-ml.md
• forgelm/data_audit.py

📝 Walkthrough

Walkthrough

Consolidates Phases 11–12.5 into a single v0.5.0 release and adds Phase 12.5 features: --all-mask (combined secrets+PII masking), optional --pii-ml (Presidio ML‑NER), --croissant dataset‑card emission, a --quality-filter hook, and a wizard “audit first” interactive post‑ingest flow.

Changes

Cohort / File(s)	Summary
Release & Roadmap CHANGELOG.md, docs/roadmap.md, docs/roadmap-tr.md, docs/roadmap/releases.md, docs/roadmap/*	Consolidates Phases 11/11.5/12/12.5 under v0.5.0, removes duplicated 0.5.x entries, updates roadmap tables, Mermaid graphs, and release-note structure.
Guides & Reference README.md, docs/guides/..., docs/reference/usage.md, docs/reference/usage-tr.md, docs/guides/*.md	Reframes version text to v0.5.0; documents --all-mask, --pii-ml (+language/model preflight), --croissant, and --quality-filter; updates examples and CLI refs.
User Manuals & i18n docs/usermanuals/_meta.yaml, docs/usermanuals/en/data/*, docs/usermanuals/tr/data/*	Adds pages for Combined Masking, ML‑PII, and Croissant card; updates ingestion/audit manuals, cross‑refs, localized nav entries and examples.
Architecture & Notebook docs/standards/architecture.md, notebooks/data_curation.ipynb	Adds ingestion-pii-ml optional extra to architecture; updates notebook to v0.5.0 and documents --all-mask, Croissant, and Presidio+spaCy install snippet.
CLI forgelm/cli.py	Adds --all-mask to ingest; adds --croissant, --pii-ml, and --pii-ml-language to audit; threads flags into audit invocation and surfaces report.croissant in JSON output.
Audit Core forgelm/data_audit.py	Implements Presidio ML‑NER path (detect_pii_ml, severity mapping, _require_presidio preflight), Croissant emitter (_build_croissant_metadata / report.croissant), quality_summary wiring, and merges ML findings with regex PII counts.
Wizard forgelm/wizard.py	Adds post‑ingest “audit first” hook prompting to run audit on produced JSONL (size warning >100MB), lazy imports, error recovery, and BYOD prompt adjustments to avoid double prompts.
Project Metadata pyproject.toml	Bumps project version to 0.5.0 and adds optional extra ingestion-pii-ml = ["presidio-analyzer>=2.2.0,<3.0.0"].
Website & Quickstart site/index.html, site/compliance.html, site/quickstart.html, site/js/translations.js	Updates public version label to v0.5.0, expands compliance text with optional Presidio/Croissant mentions, updates quickstart to use --all-mask and document new audit flags (i18n strings updated).
Tests tests/test_phase12_5.py, tests/test_wizard_byod.py	Adds comprehensive Phase 12.5 tests covering --all-mask semantics, wizard audit hook behavior and error handling, Croissant emission mapping and path handling, and Presidio ML preflight/failure semantics; updates BYOD tests to decline audit prompt.

Sequence Diagram

sequenceDiagram
    actor User
    participant Wizard as "Wizard (CLI)"
    participant Ingest as "Ingestion Module"
    participant Audit as "Audit Module"
    participant Report as "AuditReport / Croissant"

    User->>Wizard: run `forgelm ingest --wizard ...`
    Wizard->>Ingest: ingest_path(...) -> produce JSONL
    Ingest-->>Wizard: return JSONL path

    Wizard->>Wizard: check JSONL size (warn if >100MB)
    Wizard->>User: "Audit this dataset first? (y/n)"

    alt User accepts
        User-->>Wizard: y
        Wizard->>Audit: audit_dataset(jsonl_path, enable_pii_ml?, emit_croissant?, quality_filter?)
        Audit->>Audit: preflight (presidio/spaCy checks), dedup, PII regex + ML
        Audit-->>Report: AuditReport (pii_summary, quality_summary, croissant)
        Wizard->>User: summarize_report(report) -> display verdict
    else User declines or audit error
        User-->>Wizard: n / error
        Wizard->>User: continue without audit
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat: Phase 11 + post-v0.4.5 sync (v0.5.0) #11 — foundational ingestion/audit work touching the same modules (cli.py, data_audit.py, wizard.py).
• Phase 12 — Data Curation Maturity (v0.5.2) #13 — implements follow‑ups and Phase 12.5 wiring (--all-mask, Presidio ML, Croissant) on the same code paths.
• feat(phase-11.5): Ingestion / audit polish (v0.5.1) #12 — related phase work that overlaps CLI/audit/wizard surfaces extended here.

Poem

🐰 A tidy hop, a consolidation cheer,

Four phases tucked into one release this year.
Croissant cards and Presidio eyes,
The wizard asks first, then shows the prize.
--all-mask hops in — short, sweet, and clear.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 31.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly summarizes the main change: consolidating four phases into a v0.5.0 release.
Description check	✅ Passed	The PR description is well-structured with summary, changes overview, and test plan, covering the main objectives of consolidating phases and Phase 12.5 features.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch development

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 38 minutes and 57 seconds.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codacy-production]

Not up to standards ⛔

🔴 Issues 42 medium

Alerts:
⚠ 42 issues (≤ 0 issues of at least minor severity)

Results:
42 new issues

Category	Results
ErrorProne	42 medium

View in Codacy

🟢 Metrics 101 complexity · 15 duplication

Metric	Results
Complexity	101
Duplication	15

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request consolidates Phases 11, 11.5, 12, and 12.5 into a single v0.5.0 release, introducing several data curation and governance features including a Presidio-based ML-NER PII detector, Google Croissant 1.0 dataset card emission, an '--all-mask' ingestion shorthand, and a wizard-integrated audit entry point. My feedback focuses on clarifying documentation phrasing, ensuring explicit links for environment setup in air-gapped scenarios, and improving error logging for the new audit hooks.

[gemini-code-assist]

The phrasing '(no unreleased changes — next minor will pick up Phase 14...)' is slightly ambiguous. It is better to state clearly that the release is finalized and future work is tracked elsewhere.

[cemililik]

✅ Addressed in commit 7fae240 — reworded to make the Unreleased section state explicit ("Empty — v0.5.0 is finalized. The next minor (v0.5.1) covers Phase 14...").

[gemini-code-assist]

The instruction to download the spaCy model is critical. Ensure that the installation instructions for the spaCy model are explicitly linked to the environment setup section to avoid runtime errors in air-gapped or restricted environments.

[cemililik]

Status: declined — already inline.

The fail-loud install-recipe behaviour is documented immediately below the install block (this same section, lines 375-381):

Without the model, forgelm audit --pii-ml raises an ImportError with the same install hint before any rows are scanned (the pre-flight check in forgelm.data_audit._require_presidio catches spaCy's OSError and re-raises it as the typed install error). This fail-loud behaviour exists because earlier prototypes silently scored zero ML findings on a missing model — a critical compliance blind spot for an opt-in detector that operators expect to actually run.

Operators in air-gapped environments hit this fail-loud path before any rows are scanned, with the same install recipe. The pre-flight + the install hint live in _require_presidio at the top of the module so the recipe is in one canonical place. No separate "environment setup" section exists to link to — the docs are intentionally flat for this guide.

[gemini-code-assist]

While catching Exception is necessary to prevent wizard crashes, ensure that specific expected errors (like FileNotFoundError) are logged with enough context for the user to troubleshoot the audit failure.

[cemililik]

Status: declined — current behaviour already surfaces the exception type and message.

The wizard renders interactive output via print() (project convention for the Step-N flow), and print(f" Audit could not run: {exc}") already includes the exception type and message in the rendered string — e.g. Audit could not run: [Errno 2] No such file or directory: '/path/to/foo.jsonl' for FileNotFoundError.

Adding except FileNotFoundError as exc: would emit the same string with no additional context. The catch-all except Exception is intentional (with the inline docstring rationale at lines 547-554) so a malformed JSONL row, a datasketch version mismatch, or any other downstream audit-module failure doesn't crash the wizard mid-flow.

KeyboardInterrupt and SystemExit propagate by virtue of being BaseException subclasses, so Ctrl-C still aborts cleanly. Leaving as-is.

[coderabbitai]

Actionable comments posted: 13

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@CHANGELOG.md`:
- Around line 96-99: Update the fenced code block that contains the two commands
("pip install 'forgelm[ingestion-pii-ml]'" and "python -m spacy download
en_core_web_lg'") by adding a language tag (e.g., ```bash) immediately after the
opening backticks so the block becomes a bash code block and satisfies
markdownlint MD040.

In `@docs/guides/data_audit-tr.md`:
- Around line 415-416: Update the Croissant field list in the docs snippet that
currently reads "dataset seviyesinde kimlik (`name`, `description`, `version`,
`datePublished`, `url`)" by removing the `version` token so it matches the
generator output; edit the string in docs/guides/data_audit-tr.md where the
dataset identity list is defined (the line containing the backticked list with
name, description, version, datePublished, url) and remove `version` from the
comma-separated backticked items, keeping punctuation and localization style
consistent with surrounding sentences.

In `@docs/guides/data_audit.md`:
- Around line 409-423: The docs list includes a `version` Croissant field that
the code does not emit; update the documentation to remove `version` from the
documented Croissant field list (or explicitly note it's omitted) to match the
implementation in forgelm/data_audit.py::_build_croissant_metadata(); ensure the
docs reflect that dataset-level identity contains name, description,
datePublished, and url only (or state that `version` will appear only if
_build_croissant_metadata() is changed to populate it).
- Around line 322-425: The CLI synopsis block that begins with "forgelm audit
PATH \" is missing the new flags; update that copy-pasteable CLI example to
include --pii-ml, --pii-ml-language <lang>, and --croissant (e.g. append them to
the options list or add a separate example showing their usage) so the
documented invocation matches the newly described features (pii-ml,
pii-ml-language, croissant) in the preceding sections.

In `@docs/guides/ingestion.md`:
- Around line 143-159: The CLI reference block for the "forgelm ingest" command
is missing the new shorthand flag --all-mask; update the ingest
synopsis/reference block that currently lists --pii-mask and --secrets-mask to
also include --all-mask so the guide is consistent with the new "All-in-one
masking" section and clarifies that --all-mask composes with --pii-mask and
--secrets-mask.

In `@docs/roadmap-tr.md`:
- Around line 12-31: The roadmap text is internally inconsistent: the top claims
Faz 11 + 11.5 + 12 + 12.5 were consolidated into a single release `v0.5.0`, but
later map entries still reference separate tags `v0.5.1` / `v0.5.2` / `v0.5.3`;
update the later roadmap map and the "Sonraki" section so all references reflect
the consolidation (replace mentions of `v0.5.1`/`v0.5.2`/`v0.5.3` with `v0.5.0`
where appropriate, adjust the entries for Faz 14 and Faz 13 to show they follow
`v0.5.0`, and ensure the "PyPI'deki son sürüm" / "Başlangıçta dört ardışık PyPI
tag'i" wording matches the single `v0.5.0` consolidation described near `Faz
11`/`Faz 11.5`/`Faz 12`/`Faz 12.5`).

In `@docs/roadmap.md`:
- Around line 12-31: The roadmap text contradicts itself by saying Phases
11/11.5/12/12.5 were consolidated into v0.5.0 while later still mapping them to
v0.5.1/v0.5.2/v0.5.3; update the document map so all references to Phase 11,
11.5, 12, and 12.5 point to v0.5.0 (remove the old split v0.5.1/v0.5.2/v0.5.3
mapping), keep Phase 14 listed as v0.5.1 and Phase 13 as v0.6.0-pro, and ensure
the table rows/summary blocks (the lines mentioning "Phase 11 + 11.5 + 12 + 12.5
— Document Ingestion & Data Curation Pipeline", "v0.5.0", "Phase 14 —
Multi-Stage Pipeline Chains", and "Phase 13 — Pro CLI & Observability
Dashboard") are consistent so the page no longer contradicts itself.

In `@docs/usermanuals/en/data/croissant-card.md`:
- Around line 130-132: Replace the broken fragment links '#/data/audit',
'#/compliance/annex-iv', and '#/compliance/gdpr' in croissant-card.md with the
project's resolvable internal documentation links (use the docs tree's canonical
relative/absolute doc URLs used elsewhere), e.g., update the three list items
for "Dataset Audit", "Annex IV", and "GDPR / KVKK" to point to their actual
markdown targets per docs/standards (use the same link style as other docs files
so they pass MD051 validation).

In `@docs/usermanuals/en/data/pii-ml.md`:
- Around line 79-87: The docs are wrong about Presidio errors being "surfaced
loudly" because detect_pii_ml() currently catches ValueError/RuntimeError
(around the handler at line ~518) and returns {} per row; update the code to
perform a single pre-flight validation of the requested language instead of
swallowing per-row errors: add a language-model check in _require_presidio() (or
at the start of detect_pii_ml()) that verifies the requested --pii-ml-language
is registered/available and if not either raise a clear exception or log an
error and abort the audit before scanning rows; ensure the same unique symbols
detect_pii_ml() and _require_presidio() are modified so failures for custom
languages are not silently converted to empty findings.

In `@docs/usermanuals/tr/data/audit.md`:
- Line 103: Satırdaki `--croissant` açıklamasında kullanılan dış bağlantı
`http://mlcommons.org/croissant/` güvenli değil; düzeltmek için metindeki URL'yi
`https://mlcommons.org/croissant/` olarak güncelleyin (aynı metin içinde
Croissant 1.0 referansını içeren `--croissant` açıklamasını bulun ve yalnızca
protokolü http → https olarak değiştirin).

In `@docs/usermanuals/tr/data/pii-ml.md`:
- Around line 79-86: The docs claim language-registration failures are raised,
but detect_pii_ml() currently swallows ValueError/RuntimeError per row; change
detect_pii_ml so it fails loudly by doing a pre-flight language registration
check (validate the requested PII-ML engine/language once at the start) and
raise an explicit exception if the engine or spaCy model is not available
instead of catching and returning {} per-row; alternatively, if you prefer
per-row handling, update the catch to re-raise registration-related exceptions
so callers observe the failure (refer to detect_pii_ml and its current
ValueError/RuntimeError except block).

In `@forgelm/data_audit.py`:
- Around line 2473-2490: The code builds file_id from the split label (file_id =
f"{split_name}.jsonl") and sets contentUrl/url from source_input, which causes
wrong filenames for single-file/alias audits and leaks absolute paths; instead
derive the file_id and contentUrl from the real source filename supplied in the
split metadata (e.g. use info.get("source_filename") or info.get("source") /
source_input when present), sanitize it with os.path.basename (or a normalized
published name) to strip local filesystem prefixes, and fall back to the split
label only if no source filename exists; update the distribution.append block
(the file_id, "name", "contentUrl" and any url variable) to use this sanitized
source-derived name so cards reference the real source file names and do not
leak absolute paths.
- Around line 480-526: The current detect_pii_ml() swallows ValueError from
Presidio (causing unsupported --pii-ml-language to silently return {} per row);
change it to call _require_presidio() up front (or validate in audit_dataset())
and then obtain the analyzer via _get_presidio_analyzer() and explicitly verify
the requested language is registered on the AnalyzerEngine (raise ValueError if
not) so misconfiguration fails fast; after that, keep the per-row RuntimeError
resilience but do not catch ValueError from analyzer.analyze() so
unsupported-language errors surface immediately.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4b319bd3-f126-45ea-a6a4-519b415f5a3b

📥 Commits

Reviewing files that changed from the base of the PR and between 4f0457f and 7d0f150.

📒 Files selected for processing (40)

• CHANGELOG.md
• README.md
• docs/guides/data_audit-tr.md
• docs/guides/data_audit.md
• docs/guides/ingestion-tr.md
• docs/guides/ingestion.md
• docs/reference/usage-tr.md
• docs/reference/usage.md
• docs/roadmap-tr.md
• docs/roadmap.md
• docs/roadmap/phase-11-5-backlog.md
• docs/roadmap/phase-12-5-backlog.md
• docs/roadmap/phase-12-data-curation-maturity.md
• docs/roadmap/phase-14-pipeline-chains.md
• docs/roadmap/releases.md
• docs/standards/architecture.md
• docs/usermanuals/_meta.yaml
• docs/usermanuals/en/data/all-mask.md
• docs/usermanuals/en/data/audit.md
• docs/usermanuals/en/data/croissant-card.md
• docs/usermanuals/en/data/ingestion.md
• docs/usermanuals/en/data/pii-masking.md
• docs/usermanuals/en/data/pii-ml.md
• docs/usermanuals/tr/data/all-mask.md
• docs/usermanuals/tr/data/audit.md
• docs/usermanuals/tr/data/croissant-card.md
• docs/usermanuals/tr/data/ingestion.md
• docs/usermanuals/tr/data/pii-masking.md
• docs/usermanuals/tr/data/pii-ml.md
• forgelm/cli.py
• forgelm/data_audit.py
• forgelm/wizard.py
• notebooks/data_curation.ipynb
• pyproject.toml
• site/compliance.html
• site/index.html
• site/js/translations.js
• site/quickstart.html
• tests/test_phase12_5.py
• tests/test_wizard_byod.py

[cemililik]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 5

♻️ Duplicate comments (1)

docs/usermanuals/en/data/croissant-card.md (1)

130-132: ⚠️ Potential issue | 🟡 Minor

Fix unresolved internal doc links in “See also”.

Line 130–132 use #/... fragments that fail MD051 in this repo layout. Switch to resolvable relative markdown links.

Suggested patch

-- [Dataset Audit](`#/data/audit`) — where `--croissant` is invoked.
-- [Annex IV](`#/compliance/annex-iv`) — the EU AI Act Article 11 artefact that the audit feeds into.
-- [GDPR / KVKK](`#/compliance/gdpr`) — broader regulatory context.
+- [Dataset Audit](./audit.md) — where `--croissant` is invoked.
+- [Annex IV](../compliance/annex-iv.md) — the EU AI Act Article 11 artefact that the audit feeds into.
+- [GDPR / KVKK](../compliance/gdpr.md) — broader regulatory context.

As per coding guidelines docs/**/*.md: Documentation must follow docs/standards/documentation.md and docs/standards/localization.md standards.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/usermanuals/en/data/croissant-card.md` around lines 130 - 132, The "See
also" links using `#/...` fragments (the entries "Dataset Audit", "Annex IV",
and "GDPR / KVKK") are unresolved and trigger MD051; update each link to a
resolvable relative markdown link to the actual documentation pages (replace the
`#/data/audit`, `#/compliance/annex-iv`, and `#/compliance/gdpr` fragments with
correct relative .md targets in the docs tree) so the links resolve in this repo
layout and conform to docs/standards/documentation.md and
docs/standards/localization.md.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/usermanuals/en/data/pii-ml.md`:
- Line 8: Replace the fragment-style internal links like "#/data/pii-masking" in
docs/usermanuals/en/data/pii-ml.md (occurring at the text around the line that
mentions "PII Masking" and also at the other referenced locations) with
resolvable internal doc links per the docs standards: use a relative or absolute
docs path (e.g., ../ or /docs/... style) that points to the target markdown file
instead of the "#/..." fragment, update all occurrences noted (the fragment at
"PII Masking" on line ~8, the one around line ~24, and the group at ~137–140),
and ensure the new links conform to docs/standards/documentation.md and
docs/standards/localization.md to satisfy MD051.

In `@docs/usermanuals/tr/data/croissant-card.md`:
- Around line 130-132: Replace unresolved fragment links in croissant-card.md —
specifically the link targets `#/data/audit`, `#/compliance/annex-iv`, and
`#/compliance/gdpr` — with correct relative .md paths to their corresponding
docs (e.g., the Audit, Annex IV, and GDPR/KVKK markdown files). Update the three
link targets in the paragraph containing "[Veri Seti Denetimi]", "[Annex IV]"
and "[GDPR / KVKK]" so they point to the actual relative file locations instead
of `#/...` fragments to satisfy docs/standards and fix MD051.

In `@docs/usermanuals/tr/data/pii-ml.md`:
- Line 8: Replace internal fragment-style links like [PII
Maskeleme](`#/data/pii-masking`) with relative .md links that resolve (e.g. [PII
Maskeleme](../data/pii-masking.md) or the correct relative path from the current
doc); update every occurrence of the "#/..." fragment format (including the
instance in the string "Varsayılan [PII Maskeleme](`#/data/pii-masking`)" and
other similar fragments) to use proper relative .md targets so MD051 warnings
are resolved and localization/documentation standards are followed.

In `@forgelm/data_audit.py`:
- Around line 405-418: _get_presidio_analyzer currently always returns a default
_PresidioAnalyzer (which constructs a presidio AnalyzerEngine with no args and
therefore only "en"), so the CLI flag --pii-ml-language / operator-config cannot
influence supported_languages; change _get_presidio_analyzer and
_PresidioAnalyzer to accept and propagate a language/model parameter (e.g.,
language or nlp_engine_provider) so callers can pass the configured
--pii-ml-language value, ensure the AnalyzerEngine is instantiated using a
custom NlpEngineProvider when a non-"en" language is requested, and update any
call sites (preflight checks and detect_pii_ml) to call
_get_presidio_analyzer(language) so supported_languages reflects the actual
analyzer configuration.
- Around line 537-545: The except block catching (ValueError, RuntimeError)
should log the recoverable Presidio ML‑NER per-row errors before returning an
empty dict: change the handler to capture the exception (e.g., except
(ValueError, RuntimeError) as exc:) and call logger.warning with a concise
message and the exception (include exc_info=True or pass exc) plus any available
per-row context (row id/record/index from the surrounding scope) so it's clear
this was a per-row ML‑NER failure; leave the return {} behavior intact and keep
_require_presidio semantics unchanged.

---

Duplicate comments:
In `@docs/usermanuals/en/data/croissant-card.md`:
- Around line 130-132: The "See also" links using `#/...` fragments (the entries
"Dataset Audit", "Annex IV", and "GDPR / KVKK") are unresolved and trigger
MD051; update each link to a resolvable relative markdown link to the actual
documentation pages (replace the `#/data/audit`, `#/compliance/annex-iv`, and
`#/compliance/gdpr` fragments with correct relative .md targets in the docs
tree) so the links resolve in this repo layout and conform to
docs/standards/documentation.md and docs/standards/localization.md.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 76cb804e-d7e2-4b22-a600-610931a45f35

📥 Commits

Reviewing files that changed from the base of the PR and between 7d0f150 and 42676af.

📒 Files selected for processing (15)

• CHANGELOG.md
• docs/guides/data_audit-tr.md
• docs/guides/data_audit.md
• docs/guides/ingestion-tr.md
• docs/guides/ingestion.md
• docs/roadmap-tr.md
• docs/roadmap.md
• docs/usermanuals/en/data/audit.md
• docs/usermanuals/en/data/croissant-card.md
• docs/usermanuals/en/data/pii-ml.md
• docs/usermanuals/tr/data/audit.md
• docs/usermanuals/tr/data/croissant-card.md
• docs/usermanuals/tr/data/pii-ml.md
• forgelm/data_audit.py
• tests/test_phase12_5.py

[cemililik]

Status: declined — false positive against the project's SPA convention.

#/<section>/<page> is the user-manual SPA hash-router scheme parsed at runtime by site/js/guide.js:69-71 (parseHash). These are route identifiers, not in-page anchors — markdownlint's MD051 can't distinguish the two and produces 42 false positives across 488 link sites.

Switching to relative .md paths (../data/audit.md) would break SPA navigation: tools/build_usermanuals.py does not rewrite link targets, so <a href="../data/audit.md"> would render as a literal href that the browser navigates to a non-existent static URL.

The proper resolution is on the Codacy side: either disable rule MD051 at project level in the Codacy UI (one-time admin action), or mark these specific findings as False Positive. SonarCloud's Quality Gate is already passing.

This thread covers the same issue across 4 files (en/tr × pii-ml/croissant-card); leaving as-is.

[coderabbitai]

@cemililik, understood — thanks for the detailed explanation. The #/<section>/<page> scheme being a runtime hash-router contract (not in-page anchors) is a meaningful distinction, and the reasoning around tools/build_usermanuals.py not rewriting link targets makes the relative-path suggestion actively harmful here. I'll record this for future reviews.

---

✏️ Learnings added

Learnt from: cemililik
Repo: cemililik/ForgeLM PR: 18
File: docs/usermanuals/en/data/pii-ml.md:8-8
Timestamp: 2026-04-29T20:40:44.227Z
Learning: In the ForgeLM repository (cemililik/ForgeLM), links in the format `#/<section>/<page>` (e.g., `#/data/pii-masking`, `#/data/audit`, `#/operations/air-gap`) within `docs/usermanuals/**/*.md` files are intentional SPA hash-router route identifiers, NOT in-page anchors. They are parsed at runtime by `site/js/guide.js` (the `parseHash` function around lines 69-71). markdownlint rule MD051 ("link-fragments") produces false positives for these links (42 false positives across 488 link sites). Do NOT suggest replacing these with relative `.md` paths — `tools/build_usermanuals.py` does not rewrite link targets, so relative paths like `./pii-masking.md` would render as broken static URLs in the SPA and break navigation. The correct resolution is to suppress MD051 at the Codacy project level as a false positive.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

“Bakınız” linkleri çözülmeyen fragment formatında.

Line 130–132’deki #/... bağlantıları bu docs ağacında çözümlenmiyor (MD051). Relative .md hedeflerine geçin.

Önerilen yama

-- [Veri Seti Denetimi](`#/data/audit`) — `--croissant`'ın çağrıldığı yer.
-- [Annex IV](`#/compliance/annex-iv`) — audit'in beslediği EU AI Act Madde 11 artefaktı.
-- [GDPR / KVKK](`#/compliance/gdpr`) — daha geniş düzenleyici bağlam.
+- [Veri Seti Denetimi](./audit.md) — `--croissant`'ın çağrıldığı yer.
+- [Annex IV](../compliance/annex-iv.md) — audit'in beslediği EU AI Act Madde 11 artefaktı.
+- [GDPR / KVKK](../compliance/gdpr.md) — daha geniş düzenleyici bağlam.

As per coding guidelines docs/**/*.md: Documentation must follow docs/standards/documentation.md and docs/standards/localization.md standards.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	- [Veri Seti Denetimi](#/data/audit) — `--croissant`'ın çağrıldığı yer.
	- [Annex IV](#/compliance/annex-iv) — audit'in beslediği EU AI Act Madde 11 artefaktı.
	- [GDPR / KVKK](#/compliance/gdpr) — daha geniş düzenleyici bağlam.
	- [Veri Seti Denetimi](./audit.md) — `--croissant`'ın çağrıldığı yer.
	- [Annex IV](../compliance/annex-iv.md) — audit'in beslediği EU AI Act Madde 11 artefaktı.
	- [GDPR / KVKK](../compliance/gdpr.md) — daha geniş düzenleyici bağlam.

🧰 Tools

🪛 markdownlint-cli2 (0.22.1)

[warning] 130-130: Link fragments should be valid

(MD051, link-fragments)

---

[warning] 131-131: Link fragments should be valid

(MD051, link-fragments)

---

[warning] 132-132: Link fragments should be valid

(MD051, link-fragments)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/usermanuals/tr/data/croissant-card.md` around lines 130 - 132, Replace
unresolved fragment links in croissant-card.md — specifically the link targets
`#/data/audit`, `#/compliance/annex-iv`, and `#/compliance/gdpr` — with correct
relative .md paths to their corresponding docs (e.g., the Audit, Annex IV, and
GDPR/KVKK markdown files). Update the three link targets in the paragraph
containing "[Veri Seti Denetimi]", "[Annex IV]" and "[GDPR / KVKK]" so they
point to the actual relative file locations instead of `#/...` fragments to
satisfy docs/standards and fix MD051.

[cemililik]

Status: declined — false positive against the project's SPA convention.

#/<section>/<page> is the user-manual SPA hash-router scheme parsed at runtime by site/js/guide.js:69-71 (parseHash). These are route identifiers, not in-page anchors — markdownlint's MD051 can't distinguish the two and produces 42 false positives across 488 link sites.

Switching to relative .md paths (../data/audit.md) would break SPA navigation: tools/build_usermanuals.py does not rewrite link targets, so <a href="../data/audit.md"> would render as a literal href that the browser navigates to a non-existent static URL.

The proper resolution is on the Codacy side: either disable rule MD051 at project level in the Codacy UI (one-time admin action), or mark these specific findings as False Positive. SonarCloud's Quality Gate is already passing.

This thread covers the same issue across 4 files (en/tr × pii-ml/croissant-card); leaving as-is.

[coderabbitai]

@cemililik, understood — I apologize for the false positive. The #/<section>/<page> format is a SPA hash-router convention, not a standard Markdown in-page anchor, and MD051 cannot distinguish between the two. Switching to relative .md paths would indeed break runtime navigation since tools/build_usermanuals.py does not rewrite link targets. I'll keep this in mind for future reviews.

---

✏️ Learnings added

Learnt from: cemililik
Repo: cemililik/ForgeLM PR: 18
File: docs/usermanuals/tr/data/croissant-card.md:130-132
Timestamp: 2026-04-29T20:40:46.196Z
Learning: In the ForgeLM repository (cemililik/ForgeLM), `#/<section>/<page>` link targets in docs/usermanuals/**/*.md files are SPA hash-router route identifiers parsed at runtime by site/js/guide.js (parseHash, lines 69-71). They are NOT in-page anchors. markdownlint MD051 produces false positives for all 42 such occurrences across 488 link sites. Do NOT flag these as broken fragment links. Switching to relative `.md` paths would break SPA navigation because tools/build_usermanuals.py does not rewrite link targets.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

TR sayfasındaki dahili linkler çözümlenemiyor (#/...).

Line 8, Line 24 ve Line 137–140 için fragment formatı MD051 uyarısı üretiyor; relative .md linklere geçilmeli.

Önerilen yama

-Varsayılan [PII Maskeleme](`#/data/pii-masking`) katmanı regex-anchored'dır ...
+Varsayılan [PII Maskeleme](./pii-masking.md) katmanı regex-anchored'dır ...

-- Pre-staged spaCy NER modeli olmadan air-gap ortamda çalışıyorsunuz — [Air-Gap Operasyonu](`#/operations/air-gap`)'na bakın.
+- Pre-staged spaCy NER modeli olmadan air-gap ortamda çalışıyorsunuz — [Air-Gap Operasyonu](../operations/air-gap.md)'na bakın.

-- [PII Maskeleme](`#/data/pii-masking`) — her zaman açık olan regex katmanı.
-- [Veri Seti Denetimi](`#/data/audit`) — `--pii-ml`'in çağrıldığı yer.
-- [GDPR / KVKK](`#/compliance/gdpr`) — düzenleyici bağlam.
-- [Air-Gap Operasyonu](`#/operations/air-gap`) — spaCy modelini ön-stage etme.
+- [PII Maskeleme](./pii-masking.md) — her zaman açık olan regex katmanı.
+- [Veri Seti Denetimi](./audit.md) — `--pii-ml`'in çağrıldığı yer.
+- [GDPR / KVKK](../compliance/gdpr.md) — düzenleyici bağlam.
+- [Air-Gap Operasyonu](../operations/air-gap.md) — spaCy modelini ön-stage etme.

As per coding guidelines docs/**/*.md: Documentation must follow docs/standards/documentation.md and docs/standards/localization.md standards.

Also applies to: 24-24, 137-140

🧰 Tools

🪛 markdownlint-cli2 (0.22.1)

[warning] 8-8: Link fragments should be valid

(MD051, link-fragments)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/usermanuals/tr/data/pii-ml.md` at line 8, Replace internal
fragment-style links like [PII Maskeleme](`#/data/pii-masking`) with relative .md
links that resolve (e.g. [PII Maskeleme](../data/pii-masking.md) or the correct
relative path from the current doc); update every occurrence of the "#/..."
fragment format (including the instance in the string "Varsayılan [PII
Maskeleme](`#/data/pii-masking`)" and other similar fragments) to use proper
relative .md targets so MD051 warnings are resolved and
localization/documentation standards are followed.

[cemililik]

Status: declined — false positive against the project's SPA convention.

#/<section>/<page> is the user-manual SPA hash-router scheme parsed at runtime by site/js/guide.js:69-71 (parseHash). These are route identifiers, not in-page anchors — markdownlint's MD051 can't distinguish the two and produces 42 false positives across 488 link sites.

Switching to relative .md paths (../data/audit.md) would break SPA navigation: tools/build_usermanuals.py does not rewrite link targets, so <a href="../data/audit.md"> would render as a literal href that the browser navigates to a non-existent static URL.

The proper resolution is on the Codacy side: either disable rule MD051 at project level in the Codacy UI (one-time admin action), or mark these specific findings as False Positive. SonarCloud's Quality Gate is already passing.

This thread covers the same issue across 4 files (en/tr × pii-ml/croissant-card); leaving as-is.

[coderabbitai]

@cemililik, understood — thanks for the thorough explanation. The #/<section>/<page> links are SPA route identifiers consumed by parseHash at runtime, not in-page anchors, so MD051 is indeed a false positive here and relative .md paths would break navigation. I'll record this for future reviews.

(ᴗ_ ᴗ。)

---

✏️ Learnings added

Learnt from: cemililik
Repo: cemililik/ForgeLM PR: 18
File: docs/usermanuals/tr/data/pii-ml.md:8-8
Timestamp: 2026-04-29T20:40:46.342Z
Learning: In the ForgeLM repository (`docs/usermanuals/**/*.md`), links in the format `#/<section>/<page>` (e.g., `#/data/pii-masking`, `#/data/audit`, `#/compliance/gdpr`, `#/operations/air-gap`) are intentional SPA hash-router route identifiers parsed at runtime by `site/js/guide.js` (`parseHash` function). They are NOT standard in-page markdown anchors. markdownlint MD051 produces false positives for these links (42 false positives across 488 link sites). Do NOT suggest replacing them with relative `.md` paths — `tools/build_usermanuals.py` does not rewrite link targets, so relative `.md` hrefs would break SPA navigation by pointing to non-existent static URLs.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cd /tmp/repo && head -c 1000 forgelm/data_audit.py | head -20

Repository: cemililik/ForgeLM

Length of output: 121

---

🏁 Script executed:

cd /tmp/repo && wc -l forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 121

---

🏁 Script executed:

cd /tmp/repo && sed -n '520,560p' forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 121

---

🏁 Script executed:

cd /tmp/repo && sed -n '480,550p' forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 121

---

🏁 Script executed:

wc -l forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 88

---

🏁 Script executed:

sed -n '520,560p' forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 1952

---

🏁 Script executed:

sed -n '470,550p' forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 4161

---

🏁 Script executed:

cat -n docs/standards/error-handling.md

Repository: cemililik/ForgeLM

Length of output: 8231

---

🏁 Script executed:

grep -n "logger\|logging\|warn" forgelm/data_audit.py | head -30

Repository: cemililik/ForgeLM

Length of output: 710

---

🏁 Script executed:

sed -n '1,50p' forgelm/data_audit.py

Repository: cemililik/ForgeLM

Length of output: 1954

---

Add logging for Presidio ML-NER per-row errors.

Returning {} without logging hides whether detection succeeded (no PII) or failed (error). Per error-handling standards, "log and swallow" patterns require logging unless the path is explicitly non-fatal (webhooks, cleanup). Use logger.warning() to surface recoverable errors while continuing per-row resilience:

Fix

-    except (ValueError, RuntimeError):  # pragma: no cover — Presidio edge cases
+    except (ValueError, RuntimeError) as exc:  # pragma: no cover — Presidio edge cases
         # Per-row resilience for the narrow class of failures Presidio
         # raises on bad input or transient engine state. ``OSError`` is
         # deliberately NOT caught here — that's the missing-spaCy-model
         # signal and ``_require_presidio``'s pre-flight should have
         # converted it to ImportError before any row was scanned. If
         # one slips through (e.g. lazy model load triggered later),
         # surfacing it loudly is the correct behaviour.
+        logger.warning(
+            "Presidio ML-NER skipped a row due to %s (language=%s): %s",
+            type(exc).__name__,
+            language,
+            exc,
+        )
         return {}

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@forgelm/data_audit.py` around lines 537 - 545, The except block catching
(ValueError, RuntimeError) should log the recoverable Presidio ML‑NER per-row
errors before returning an empty dict: change the handler to capture the
exception (e.g., except (ValueError, RuntimeError) as exc:) and call
logger.warning with a concise message and the exception (include exc_info=True
or pass exc) plus any available per-row context (row id/record/index from the
surrounding scope) so it's clear this was a per-row ML‑NER failure; leave the
return {} behavior intact and keep _require_presidio semantics unchanged.

[cemililik]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[cemililik]

Status: declined — false positive against the project's SPA convention.

#/<section>/<page> is the user-manual SPA hash-router scheme parsed at runtime by site/js/guide.js:69-71 (parseHash). These are route identifiers, not in-page anchors — markdownlint's MD051 can't distinguish the two and produces 42 false positives across 488 link sites.

Switching to relative .md paths (../data/audit.md) would break SPA navigation: tools/build_usermanuals.py does not rewrite link targets, so <a href="../data/audit.md"> would render as a literal href that the browser navigates to a non-existent static URL.

The proper resolution is on the Codacy side: either disable rule MD051 at project level in the Codacy UI (one-time admin action), or mark these specific findings as False Positive. SonarCloud's Quality Gate is already passing.

This thread covers the same issue across 4 files (en/tr × pii-ml/croissant-card); leaving as-is.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

forgelm/data_audit.py (1)

2506-2546: Consider extracting "@type" to a constant to satisfy SonarCloud.

SonarCloud flags the literal "@type" appearing 6 times across the Croissant metadata construction. While this is a JSON-LD vocabulary key and the duplication is semantically meaningful, extracting it to a module-level constant would silence the linter and marginally improve maintainability.

♻️ Optional refactor

+_JSONLD_TYPE_KEY: str = "@type"
+
 _CROISSANT_CONTEXT: Dict[str, Any] = {
     "@language": "en",
     ...

Then use _JSONLD_TYPE_KEY in the distribution/recordSet builders. This is a low-priority nit — the current form is perfectly readable.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@forgelm/data_audit.py` around lines 2506 - 2546, SonarCloud complaints come
from repeating the literal "@type" in the Croissant JSON-LD build; introduce a
module-level constant (e.g. _JSONLD_TYPE_KEY = "@type") and replace all
occurrences of the string literal in _CROISSANT_CONTEXT and the related
distribution/recordSet builders with that constant (search for uses in the
Croissant metadata construction code and update keys like "data": {"@id":
"cr:data", "@type": "@json"} to use _JSONLD_TYPE_KEY) so the linter is satisfied
while preserving identical runtime behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/usermanuals/en/data/pii-ml.md`:
- Line 8: Replace the British spelling "organisation names" in the sentence that
starts "The default PII Masking layer..." with the American spelling
"organization names" so it matches the rest of the document and the code
category name "organization"; ensure any other occurrences of "organisation" in
this file are changed to "organization" for consistency.

---

Nitpick comments:
In `@forgelm/data_audit.py`:
- Around line 2506-2546: SonarCloud complaints come from repeating the literal
"@type" in the Croissant JSON-LD build; introduce a module-level constant (e.g.
_JSONLD_TYPE_KEY = "@type") and replace all occurrences of the string literal in
_CROISSANT_CONTEXT and the related distribution/recordSet builders with that
constant (search for uses in the Croissant metadata construction code and update
keys like "data": {"@id": "cr:data", "@type": "@json"} to use _JSONLD_TYPE_KEY)
so the linter is satisfied while preserving identical runtime behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e717c18f-c8e8-4b07-8eb6-b7ab64dab2ac

📥 Commits

Reviewing files that changed from the base of the PR and between 42676af and 502ff88.

📒 Files selected for processing (4)

• docs/usermanuals/en/data/pii-ml.md
• docs/usermanuals/tr/data/pii-ml.md
• forgelm/data_audit.py
• tests/test_phase12_5.py

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud