Repository navigation
ART M7: sovereignty, classification, filtered audit reports and performance (#111) - #228
Merged
Merged
Conversation
…ance (M7, #111; BR-35, BR-37; decisions 7 and 8) - Audit reports: GET /api/audit/events filters by actor, action, file, table and time with a formula-safe CSV download; GET /api/csv/uploads takes status, uploader, file and time filters and gives each failed attempt's stored reason. A rejected upload's message is now stored with it, so a report can say why. - Classification and retention (ADR-0003: per deployment): DATA_CLASSIFICATION (default OFFICIAL: Sensitive) is marked on every response in X-Data-Classification and exposed to browsers; AUDIT_RETENTION_YEARS (default 7). GET /api/policy reads them. PROTECTED is refused at startup. - Sovereignty: pytest --block-egress refuses any in-process connection that isn't local and fails the run if one is attempted, even if swallowed (make test-egress-blocked, and a CI step). Static scans of the API and frontend source, and scripts/check_bundle_egress.py for a built frontend. - Performance: scripts/perf_import.py times the whole approval flow, with a smoke test in tests/performance. The first preview page parsed every row of a large file (1.66 s for 100,000 rows); it now parses only the head and counts the rest with the C csv reader (0.26 s). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
The Windows unit job failed three guard tests with AttributeError on socket.AF_UNIX. Look it up with getattr, and skip the Unix-socket check where it doesn't exist. Verified against an interpreter with the attribute removed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
Base automatically changed from
claude/issue-110-approval-staged-import
to
claude/issue-109-dedup-versioned-datasets
November 10, 2026 03:15
It imported the Unix-only resource module, so it crashed at startup on Windows. Peak and total memory now come from the Windows API there, and degrade to "unknown" instead of failing anywhere they can't be read. A missing httpx2 (which starlette's TestClient needs, and api/requirements.txt doesn't list) now gives a one-line fix instead of a traceback. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
…e database (M7, #111) The API's own start-up retries a failed connection 30 times, built for a database that is still starting. A wrong password or port therefore looked like a hang of about five minutes. The script now tries once first and says what is wrong, including how to set the variables in PowerShell. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
…111) On a real Windows run the memory readouts came back "unknown": the current-process pseudo-handle (-1) was passed to GetProcessMemoryInfo without argument types, which overflows on 64-bit Windows. Declare them, and read total and peak memory separately so one failing lookup can't hide the other. Still degrades to unknown rather than failing the run. Verified on Linux and with a simulated no-resource interpreter; the Windows API path itself can only be exercised on Windows. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
amar-python
marked this pull request as ready for review
November 10, 2026 05:08
amar-python
changed the base branch from
claude/issue-109-dedup-versioned-datasets
to
main
November 10, 2026 05:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
M7 of the ART plan: sovereignty and reports (BR-35, BR-37; decisions 7 and 8). Tracker: #111.
Stacked on #227 (M6), which is stacked on #226 (M5). The base is the M6 branch, so this diff shows only M7. Merge #226, then #227, and GitHub retargets each in turn.
GET /api/audit/eventsfilters by actor, action, file, table and time (frominclusive,toexclusive), pages, or downloadsformat=csvwith spreadsheet formulas neutralised exactly as the frontend export does.GET /api/csv/uploadstakesstatus,uploader,file,fromandto, and each failed, rejected, duplicate or partly imported attempt now says why (failure_reason). A rejected upload's message used to be returned to the caller and never stored; it is now.DATA_CLASSIFICATION(defaultOFFICIAL: Sensitive) is marked on every API response inX-Data-Classificationand exposed to browsers;AUDIT_RETENTION_YEARS(default 7).GET /api/policyreads both.PROTECTEDis refused at startup, with the IRAP reason.pytest --block-egress, which refuses any in-process connection that isn't to this machine and fails the run if one is attempted even when the code swallows the error (make test-egress-blocked, and a new CI step); andscripts/check_bundle_egress.pyfor a built frontend.scripts/perf_import.pytimes the whole approval flow, with a smoke test intests/performance. It also found a problem: the first preview page parsed every row of a large file in Python. It now parses only the head of a file over 200,000 characters and counts the rest with the Ccsvreader.Results
100,000 rows (9.8 MB, 8 typed columns), measured here:
Targets: under 2 s and under 60 s. The built frontend (client and server bundles) has no load paths and no analytics or CDN hosts; it names only XML namespaces and documentation links. The API tests pass with egress blocked, with zero attempts.
Decisions to check
python scripts/perf_import.py --rows 100000 --enforce.tests/performanceasserts the targets only withPERF_ENFORCE=1, because a time on a laptop or CI runner says nothing about that instance.totalRowsApproxon large files is now an approximation (as its name says): it differs from a full parse only by trailing blank rows. Small files and files whose head ends in blank lines still get the full parse, so their results are identical. The frontend only displays it as "~N data rows".Also found
A CORS bug in M6: browsers couldn't send
Idempotency-Keyto the import endpoint, so a preflight got 400. Fixed on the M6 branch (#227) with a test, not here.Verification
Full suite (everything except the browser job) against PostgreSQL 16: all 584 tests pass, and
tests/systemalso passes under--block-egress.scripts/lint.shclean, bandit shows only the existingapi/db.pynotes,docs/TEST_CASES.mdregenerated.58 new tests:
test_reports_and_policy.py(23),test_sovereignty.py(13),test_egress_guard.py(10),test_preview_fast_path.py(10),test_import_scale.py(2). Breaking the formula neutralisation, dropping the marking from one router, or breaking the fast-path row count makes the relevant tests fail, and a negative control confirms a swallowed egress attempt fails the run only when the guard is on.#111 stays open until this merges.
Generated by Claude Code