Repository navigation
test: guard the closed and open issues, with an issue-to-test matrix - #224
Merged
Merged
Conversation
These closed issues had no behavioural test. Each test fails when the original bug is put back (checked by hand for all four): key-prefix rate limit identity, U+001F delimiter hashing, uploader overwrite and mode switch over an existing dataset, and survivor-position T&E row numbers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…207 Shell blocks are cut out of the real scripts and run (setup.sh config write, run_tests.sh failure parse, build_bundle.sh download step), not re-implemented. Re-introducing each original bug fails its test. #181's second half (Tier S psql calls without ON_ERROR_STOP) is recorded as a strict xfail: it is not on main. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
… scripts (#60) #113: a local TLS server with a throwaway certificate exercises the real PyJWKClient https fetch, key rotation past the refresh cooldown, no re-fetch inside it, and refusal of an untrusted certificate. A sign-in against the real IdP is still the manual step. #57: SECURITY.md is on main with a reporting channel, which is the issue's done-when. #60: the build and install scripts parse and stop on error and the guide names them; the dry run itself stays manual. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…leak test e2e/ drives the real API, the production build of the React app and the Streamlit spike in Chromium. #187: a failed upload is listed and opens its row errors. #188: no Stop once the request is on the server. #154: Streamlit role gating, edit with a required reason, exclude, history, and sign-out leaving nothing for the next tenant. #176 is a strict expected failure, in vitest and in the browser: the leak is live in the shipped app (a second tenant's history page lists the first tenant's file). The browser tests are not part of the default run. Found by these tests: every dynamic import showed as Failed (fixed in the separate header_mapping PR, merged here so the suite runs on it). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
docs/ISSUE_TESTS.md has a row for each of the 69 issues (46 closed, 23 open): the test that guards it, or why none can. Notes the two strict expected failures (#176, #181 second half) and what the end-to-end run found. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
scripts/test_report.py --strict counts an expected failure as a skip and
fails the build ("unmet prerequisites must FAIL, not skip"). Remove the
strict xfail for the second half of #181 and record the gap in
docs/ISSUE_TESTS.md instead. #176's expected failures are vitest and
browser tests, which that step does not run.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
amar-python
marked this pull request as ready for review
November 9, 2026 14:37
The Windows job runs every unit-marked test, and `bash` there is not a POSIX shell, so the tests that run the repo's real shell blocks, plus `bash -n` and the catalogue check, failed. Add a `tooling` marker for them (Linux jobs and local runs) and leave the pure-Python tests as unit. The coverage-gate test also failed on Windows: the script prints box-drawing characters and a Windows console's default encoding can't. The test now runs it with UTF-8 output. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…verage # Conflicts: # docs/TEST_CASES.md # pytest.ini
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…verage # Conflicts: # docs/TEST_CASES.md
The conflict resolution in the browser kept both count lines (901 tests across 44 files and 1013 across 45). The file is generated, so regenerate it: python3 scripts/test_catalogue.py gives 1105 tests across 49 files. Only the header changes; the catalogue check now passes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
…verage # Conflicts: # docs/TEST_CASES.md
2 tasks done
…sts for versioned datasets Main now carries M5 (versioned datasets): overwrite is ignored and a repeated name becomes a new version, so nothing is dropped and an uploader's repeat upload correctly returns 200. The two #175 tests that expected 403 pinned the pre-M5 behaviour. They now assert the new semantics: version 2, overwritten false, and the earlier dataset left registered. The delete permission itself stays covered by the delete-endpoint tests. docs/TEST_CASES.md regenerated after the merge. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
…o claude/tests-issue-coverage # Conflicts: # docs/TEST_CASES.md # tests/system/test_issue_regressions.py
…ma fix, #232) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqhUfEMZv2Dxerqs2Eq
The Streamlit exclude test waited for "\d+ excluded", which "0 excluded" already matches, so it clicked the grid mid-redraw. It now waits for a non-zero count and for the dialog to close, the grid click retries, and the suite's expect timeout is 10 s for cold starts. docs/ISSUE_TESTS.md: #109, #110, #111, #112, #113, #172, #184 and #195 now point at the tests that merged; #175 is guarded by outcome since M5. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
…o claude/tests-issue-coverage
#231 and #232 both added the three M5 log steps, so main lists version, schema_check and cross_upload_duplicates twice. zod collapses duplicate enum values, so it is harmless at runtime, but the duplicates were an accident. Remove them and add a test that reads the source (the parsed schema can't show them); it fails with a duplicate present. docs/ISSUE_TESTS.md: #174 is done on main (#204). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
amar-python
pushed a commit
that referenced
this pull request
Nov 10, 2026
main merged #224, #233 and others after the previous merge and regenerated docs/TEST_CASES.md, which conflicts with this branch's regeneration. The file is generated and CI gates on it (`test_catalogue.py --check`), so it is regenerated, not merged by hand. It is the only conflict. Verified on the merged tree: this branch's 16 runtime-role tests pass; free tier 290 passed, 0 failed, 0 skipped; lint, health and the catalogue check clean. The rest of main's tests are covered by main's own CI. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqjTEDRRQGLEYPPFVXx
amar-python
pushed a commit
that referenced
this pull request
Nov 10, 2026
main (#224) regenerated docs/TEST_CASES.md after #204 merged, which cleared the red free-tier job this branch was opened to fix. What remains is the file move. The catalogue conflict is the only one; it is generated and gated by `test_catalogue.py --check`, so it is regenerated, not merged by hand. Free tier on the merged tree: 282 passed, 0 failed, 0 skipped; lint, health and the catalogue check clean. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T6WqjTEDRRQGLEYPPFVXx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tests for the closed and open issues, plus a matrix of which test guards which issue.
What's here
docs/ISSUE_TESTS.md: one row for each of the 69 issues: the test that fails if it comes back, or why there isn't one (decisions, docs, work needing a person or another system). State as of 10 Nov, after M5, M6, M7 and #229 merged.New tests
tests/system/test_issue_regressions.pytests/unit/test_tooling_issues.pysetup.shkeeps the old config · #180 · #181 (wrapper) · #182 bundle stops on a failed download · #194 · #207. The shell tests run the real block cut out of each scripttests/unit/test_oidc_https_jwks.pytests/system/test_oidc_https.pyfrom #229tests/unit/test_open_issues.pyfe: tenant-isolation.test.tsx,fe: log-step-contract.test.tse2e/(Playwright, production build)I re-introduced the original bug for #171, #175, #178, #181, #182, #183, #189, the T&E delete gate, and the OIDC certificate check, and each test failed.
What the tests found
header_mappinglog step, fix(ui): accept the header_mapping log step, so imports no longer show as Failed #223) and again after M5 (version,schema_check,cross_upload_duplicates, fix(ui): accept the log steps M5 added, so dynamic imports stop showing as Failed #231/fix(ui): accept the M5 log steps so versioned uploads don't show as Failed #232). The schema rejected log steps the API had started to send.log-step-contract.test.tsnow checks the schema against every step the API's loaders can emit; it failed onmainboth times. Frontend tests don't run in CI, which is howmainstayed red unnoticed.evals/runner.pyhave noON_ERROR_STOP. Not tested: CI's--strictreport fails any skipped or expected-fail pytest test.overwriteis ignored, so nothing is dropped. The tests were rewritten to guard the outcome.Running
pytest tests,npx vitest run: as before. Verified here against PostgreSQL: strict free-tier report 282 passed, 0 skipped; unit and system categories all pass; vitest 176 passed + 1 expected fail.toolingmarker: tests that run the repo's real shell scripts need bash, so the Windows job leaves them out; the Linux jobs and local runs include them.pytest e2e: not part of the default run (needs Playwright, a build and three local servers; seee2e/README.md). 7 passed, 1 expected fail, stable across repeated runs. Not wired into CI.https://claude.ai/code/session_011T6Wqi8yaiaYWUVH8C4QAY
Generated by Claude Code