Skip to content

test: guard the closed and open issues, with an issue-to-test matrix - #224

Merged
amar-python merged 22 commits into
mainfrom
claude/tests-issue-coverage
Nov 10, 2026
Merged

amar-python merged 22 commits into
mainfrom
claude/tests-issue-coverage

Conversation

@amar-python

@amar-python amar-python commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Tests for the closed and open issues, plus a matrix of which test guards which issue.

What's here

docs/ISSUE_TESTS.md: one row for each of the 69 issues: the test that fails if it comes back, or why there isn't one (decisions, docs, work needing a person or another system). State as of 10 Nov, after M5, M6, M7 and #229 merged.

New tests

File Guards
tests/system/test_issue_regressions.py #175 an uploader can't destroy a dataset by uploading (asserts the outcome: the original table and rows survive; the T&E path still needs delete permission) · #178 distinct rows kept (all 31 control characters) · #183 rate-limit identity · #189 T&E error row numbers
tests/unit/test_tooling_issues.py #130 · #169 · #171 setup.sh keeps the old config · #180 · #181 (wrapper) · #182 bundle stops on a failed download · #194 · #207. The shell tests run the real block cut out of each script
tests/unit/test_oidc_https_jwks.py #113: https key fetch against a local TLS server, key rotation, the refresh cooldown, untrusted certificate refused. Overlaps tests/system/test_oidc_https.py from #229
tests/unit/test_open_issues.py #57 SECURITY.md · #60 offline scripts
fe: tenant-isolation.test.tsx, fe: log-step-contract.test.ts #176 · the response schema must accept every log step the API can emit, and list each once
e2e/ (Playwright, production build) #187, #188, #176, and the #154 Streamlit screen

I re-introduced the original bug for #171, #175, #178, #181, #182, #183, #189, the T&E delete gate, and the OIDC certificate check, and each test failed.

What the tests found

Running

  • pytest tests, npx vitest run: as before. Verified here against PostgreSQL: strict free-tier report 282 passed, 0 skipped; unit and system categories all pass; vitest 176 passed + 1 expected fail.
  • New tooling marker: tests that run the repo's real shell scripts need bash, so the Windows job leaves them out; the Linux jobs and local runs include them.
  • pytest e2e: not part of the default run (needs Playwright, a build and three local servers; see e2e/README.md). 7 passed, 1 expected fail, stable across repeated runs. Not wired into CI.

https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY


Generated by Claude Code

These closed issues had no behavioural test. Each test fails when the
original bug is put back (checked by hand for all four): key-prefix rate
limit identity, U+001F delimiter hashing, uploader overwrite and mode
switch over an existing dataset, and survivor-position T&E row numbers.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…207

Shell blocks are cut out of the real scripts and run (setup.sh config
write, run_tests.sh failure parse, build_bundle.sh download step), not
re-implemented. Re-introducing each original bug fails its test.

#181's second half (Tier S psql calls without ON_ERROR_STOP) is recorded
as a strict xfail: it is not on main.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
… scripts (#60)

#113: a local TLS server with a throwaway certificate exercises the real
PyJWKClient https fetch, key rotation past the refresh cooldown, no
re-fetch inside it, and refusal of an untrusted certificate. A sign-in
against the real IdP is still the manual step.

#57: SECURITY.md is on main with a reporting channel, which is the issue's
done-when. #60: the build and install scripts parse and stop on error and
the guide names them; the dry run itself stays manual.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…leak test

e2e/ drives the real API, the production build of the React app and the
Streamlit spike in Chromium. #187: a failed upload is listed and opens its
row errors. #188: no Stop once the request is on the server. #154: Streamlit
role gating, edit with a required reason, exclude, history, and sign-out
leaving nothing for the next tenant.

#176 is a strict expected failure, in vitest and in the browser: the leak
is live in the shipped app (a second tenant's history page lists the
first tenant's file). The browser tests are not part of the default run.

Found by these tests: every dynamic import showed as Failed (fixed in the
separate header_mapping PR, merged here so the suite runs on it).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
docs/ISSUE_TESTS.md has a row for each of the 69 issues (46 closed, 23 open):
the test that guards it, or why none can. Notes the two strict expected
failures (#176, #181 second half) and what the end-to-end run found.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
scripts/test_report.py --strict counts an expected failure as a skip and
fails the build ("unmet prerequisites must FAIL, not skip"). Remove the
strict xfail for the second half of #181 and record the gap in
docs/ISSUE_TESTS.md instead. #176's expected failures are vitest and
browser tests, which that step does not run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
@amar-python
amar-python marked this pull request as ready for review November 9, 2026 14:37
claude and others added 7 commits November 9, 2026 14:41
The Windows job runs every unit-marked test, and `bash` there is not a
POSIX shell, so the tests that run the repo's real shell blocks, plus
`bash -n` and the catalogue check, failed. Add a `tooling` marker for
them (Linux jobs and local runs) and leave the pure-Python tests as unit.

The coverage-gate test also failed on Windows: the script prints
box-drawing characters and a Windows console's default encoding can't.
The test now runs it with UTF-8 output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…verage

# Conflicts:
#	docs/TEST_CASES.md
#	pytest.ini
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
The conflict resolution in the browser kept both count lines (901 tests across 44
files and 1013 across 45). The file is generated, so regenerate it:
python3 scripts/test_catalogue.py gives 1105 tests across 49 files. Only the header
changes; the catalogue check now passes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
…sts for versioned datasets

Main now carries M5 (versioned datasets): overwrite is ignored and a repeated
name becomes a new version, so nothing is dropped and an uploader's repeat
upload correctly returns 200. The two #175 tests that expected 403 pinned the
pre-M5 behaviour. They now assert the new semantics: version 2,
overwritten false, and the earlier dataset left registered. The delete
permission itself stays covered by the delete-endpoint tests.

docs/TEST_CASES.md regenerated after the merge.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
…o claude/tests-issue-coverage

# Conflicts:
#	docs/TEST_CASES.md
#	tests/system/test_issue_regressions.py
…ma fix, #232)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
The Streamlit exclude test waited for "\d+ excluded", which "0 excluded"
already matches, so it clicked the grid mid-redraw. It now waits for a
non-zero count and for the dialog to close, the grid click retries, and
the suite's expect timeout is 10 s for cold starts.

docs/ISSUE_TESTS.md: #109, #110, #111, #112, #113, #172, #184 and #195 now
point at the tests that merged; #175 is guarded by outcome since M5.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
#231 and #232 both added the three M5 log steps, so main lists version,
schema_check and cross_upload_duplicates twice. zod collapses duplicate
enum values, so it is harmless at runtime, but the duplicates were an
accident. Remove them and add a test that reads the source (the parsed
schema can't show them); it fails with a duplicate present.

docs/ISSUE_TESTS.md: #174 is done on main (#204).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
@amar-python
amar-python merged commit 253dfa2 into main Nov 10, 2026
4 checks passed
amar-python pushed a commit that referenced this pull request Nov 10, 2026
main merged #224, #233 and others after the previous merge and regenerated
docs/TEST_CASES.md, which conflicts with this branch's regeneration. The file is
generated and CI gates on it (`test_catalogue.py --check`), so it is
regenerated, not merged by hand. It is the only conflict.

Verified on the merged tree: this branch's 16 runtime-role tests pass; free
tier 290 passed, 0 failed, 0 skipped; lint, health and the catalogue check
clean. The rest of main's tests are covered by main's own CI.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqjTEDRRQGLEYPPFVXx
amar-python pushed a commit that referenced this pull request Nov 10, 2026
main (#224) regenerated docs/TEST_CASES.md after #204 merged, which cleared the
red free-tier job this branch was opened to fix. What remains is the file move.
The catalogue conflict is the only one; it is generated and gated by
`test_catalogue.py --check`, so it is regenerated, not merged by hand.

Free tier on the merged tree: 282 passed, 0 failed, 0 skipped; lint, health and
the catalogue check clean.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6WqjTEDRRQGLEYPPFVXx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants