fix: preserve malformed LLM responses as incomplete analysis

[yashrajp22]

The batch compatibility parser converted malformed JSON or an invalid response schema into an empty successful result. Those failures now use the core structured-response retry and inspection-ledger path. Confidence validation catches null and non-numeric values; meta soft fields are repaired before validation while findings remain strict. Optional prose assessments do not discard otherwise valid finding verdicts.

The compatibility guard checks both keyword-only constructor parameters and forwards numeric or callable timeouts. Pool waits and key retries share a monotonic deadline; cancellation and constructor errors release slots, and shorter connection budgets remain intact.

Validation: 608 tests passed against a freshly installed wheel core, including 29 pool tests; the preceding source suite passed 579 tests. Ten added regressions reproduce failures against the original source and wheel core. Contributor tools are not packaged in the wheel, so wheel checks use the contributor checkout with the installed core. Tests use deterministic fake providers; these focused tests made no live provider calls.

Combined verification across the updated PRs: 6,243 regression tests passed against source and again against the freshly installed wheel, with seven conditional skips and four expected failures per run. All 19 source/wheel sample pairs matched. The 12-skill corpus retained its findings and risk ratings; four former hangs now finish with explicit partial-analysis results. The 93 extension tests passed. Two synthetic live NVIDIA Build checks passed on the final wheel: benign-note was complete/SAFE, and the exfiltration sample retained SSD-3 with complete semantic and meta analysis and a DO_NOT_INSTALL recommendation. All seven recorded LLM analyses succeeded. Live checks used the configured model/reasoning defaults through a test-only proxy that kept the real credential outside the scanner.

[rng1995]

[SkillSpector Review]

Hi @yashrajp22, thank you for fixing the compatibility constructor and both raw-JSON parsers together, so the batch scanner's LLM analyzers now fail visibly instead of returning empty results! The new compat test runs the real run_batches_detailed/arun_batches_detailed loop with a fake model and checks the ledger outcome, which is exactly where this bug was hidden.

Value and readiness: The problem is real on main.

• Core meta path. Main's MetaAnalyzerResult turns {"findings": "invalid"} into []. The result is one provider call, a successful empty batch, and meta reported as completed.
• Batch-scan compat mode. Here the same bug is currently hidden by another one: _patched_base_init rejects timeout=, so all four LLM analyzers are recorded as unavailable before any call is made. If only the constructor fix is applied to main's parsers, a not JSON response gives is_complete=true, every analyzer completed and llm_call_log ok=True. Shipping the constructor fix and the parser fix together is therefore the right shape.

With this PR's logic transcribed onto main, every malformed or schema-invalid response I tried takes exactly 4 calls. The inputs were invalid JSON, {"findings": "invalid"}, a top-level list or null, prose around JSON, and empty-string findings. Each one ends as llm_structured_response_invalid, with ledger skipped, the analyzer degraded and completeness partial. That holds for sync and async, and for discovery and meta. Fenced valid JSON and {"findings": []} still take one call and complete, and a graph run's risk score is unchanged.

The core change is sound, and the results above verify it. Two required follow-through fixes remain, so I am requesting changes. The existing contrib test for _patched_base_init now fails against the timeout forwarding (finding 1). The contrib docs still say malformed responses return [], which is the opposite of the new behaviour (finding 2). Both are small. Once they are in, this is ready for final maintainer review; CI also still has to run. Findings 3 to 8 are non-blocking.

Material findings

1. [Blocker] contrib/batch_scan/runner.py:141 (test at contrib/batch_scan/tests/test_monkeypatch_fragility.py:275): with the timeout forwarding, the existing contrib test for this function now fails. The fix is small, but it is required because this PR changes the function under test.
   • _patched_base_init now always calls _original_base_init(..., node=node, timeout=timeout).
   • test_patched_init_forwards_keyword_only_node mocks _original_base_init and asserts assert_called_once_with(instance, "prompt", "model", node="semantic_security_discovery"), which compares kwargs exactly.
   • On main the file passes 28/28. Replaying this test against a transcription of the head's function fails with Actual: ... node='semantic_security_discovery', timeout=None.
   • CI does not collect the file: pyproject.toml:117 sets testpaths = ["tests"], and make test-ci runs tests/.
   • The wider contrib suite is already red on main: test_pool_wiring.py errors at collection, and the rest gives 16 failed and 29 errors, mostly from missing API keys.
   • The production change is correct; only the test expectation is stale. The only check that the timeout is forwarded is tests/test_batch_scan_security.py:68 (analyzer._timeout == 7).
   • Expected fix:
     • Add timeout=None to the assertion.
     • Add a sibling case that passes timeout=7 and asserts it is forwarded unchanged.
     • Add a case that passes a callable deadline and asserts call_args.kwargs["timeout"] is deadline.
     • All three assertions pass against the transcription.
2. [Blocker] contrib/batch_scan/docs/DESIGN.md:286-297: the contrib docs still say that malformed responses return []. After this PR they state the opposite of the actual behaviour.
   • Stale text. The PR changes no docs, and these places still describe the old behaviour:
     • DESIGN.md says invalid JSON and schema violations are "returned as []". Its "Error propagation" paragraph says the analyzer "returns [] (no findings for that file)".
     • contrib/batch_scan/docs/README.md:344-347 (Known Limitation 5) says the user "won't know which findings were lost".
     • contrib/batch_scan/tests/docs/TEST_DESIGN.md:98 describes the same [] behaviour.
     • The runner.py:331 comment says "silent degradation if broken".
   • What the head does.
     • runner.py:156-158 and :184-185 raise _StructuredResponseValidationError. The batch is retried and then reported as incomplete.
     • The WARNING lines now come from core: "LLM structured response validation failed for ... retrying" and "... after 4 attempts".
     • A missing model_validate would now raise AttributeError past the narrowed except. It would be recorded as llm_batch_failed, not a silent [].
   • Effect. Operators who read these docs will expect silent drops. They will not understand the new "incomplete skill(s)" counts in the batch reports.
   • Expected fix.
     • Rewrite the DESIGN.md Patch 2/3 bullets and the "Error propagation" paragraph. Say that the batch is retried up to 4 attempts, then recorded as ledger skipped / llm_structured_response_invalid, with the analyzer degraded and the skill counted as incomplete. Say that the scan continues, and give the new WARNING text.
     • Narrow README Limitation 5 rather than deleting it. GapFillAnalyzer.parse_response (contrib/batch_scan/gap_fill.py) still returns [] on malformed output until #797 lands.
     • Update the TEST_DESIGN.md row and the runner.py:331 comment.
     • BUGS_FOUND.md and REVIEW_RESPONSE.md are history and can stay as they are.
3. [Non-blocking] src/skillspector/nodes/meta_analyzer.py:141: a non-JSON overall_assessment string now fails the whole meta batch, even though nothing reads that field.
   • The change. _parse_stringified_assessment now returns json.loads(v) for any string, where main fell back to None for unparseable strings.
   • Nothing reads the field. A grep of src/ and contrib/ finds no reader: LLMMetaAnalyzer.parse_response (meta_analyzer.py:436-447) and _patched_meta_parse read only .findings.
   • Measured. I compared main's model with the transcribed validator, using one valid verdict plus an overall_assessment of "HIGH risk: exfiltrates credentials", "LOW" or "".
     • main makes 1 call and applies the verdict.
     • The PR makes 4 calls and ends llm_structured_response_invalid. The finding keeps review outcome failed.
   • Scope. main was already strict for strings that parse but have the wrong shape, such as '"LOW"' or '42'. The PR extends that strictness to unparseable strings.
   • Effect. Providers without strict json_schema (tool/function calling, CLI providers, compat mode) can put prose here. A model that does this habitually costs 3 extra calls per batch and loses valid meta verdicts. It fails closed, but it costs coverage.
   • Expected fix.
     • Keep findings strict.
     • For overall_assessment, restore the None fallback (optionally also returning None for non-dict values), or drop the unused field.
     • Limit the invalid-string cases in tests/nodes/test_meta_analyzer.py:1098-1104 to findings.
     • Add a case where a prose overall_assessment validates and the verdict is kept.
4. [Non-blocking] contrib/batch_scan/runner.py:156: a null or non-numeric confidence raises TypeError, so the batch is not retried.
   • Cause.
     • Both _normalize_confidence validators call float(v) in mode="before" (src/skillspector/llm_analyzer_base.py:613, src/skillspector/nodes/meta_analyzer.py:100).
     • Pydantic converts only ValueError and AssertionError into ValidationError, so "confidence": null raises TypeError on main's models.
     • runner.py:156 and :184 catch only (json.JSONDecodeError, ValidationError).
     • _invoke_batch_with_retries does not retry a TypeError, because it is not a provider error.
   • Measured (transcription).
     • confidence null or [1]: 1 call, llm_batch_failed, ledger failed. This holds for discovery and meta, sync and async.
     • "abc": 4 calls and llm_structured_response_invalid.
     • On main the null input is a silent empty success (ledger completed), so the PR still improves this case.
   • Effect. The compat prompt's "never use null" rule (runner.py:223) suggests these models do emit nulls. Such a batch is lost without a retry and is labelled a generic failure, which contradicts the claim that malformed responses follow retry.
   • Expected fix.
     • In both validators, wrap float(v) and re-raise TypeError/ValueError as ValueError, so both the core and compat paths retry. Alternatively, add TypeError to the compat except tuples.
     • Add a "confidence": null case to test_compat_parse_failures_retry_and_remain_visible.
5. [Non-blocking] contrib/batch_scan/runner.py:183: _sanitize_meta_finding runs after validation, so it can never repair the quirks it was written for.
   • Why it never fires.
     • MetaAnalyzerFinding.impact is a Literal, and explanation/remediation are str (meta_analyzer.py:108-112).
     • model_validate at :183 therefore rejects these inputs before the sanitizer at :188 sees them: an impact of "none", null, "catastrophic" or "High", and a null remediation or explanation.
     • On valid input the sanitizer is a no-op. DESIGN.md:299-302 calls these quirks recoverable soft errors.
   • Effect.
     • On main these inputs became a silent [].
     • With the PR they cost 4 calls and leave the meta batch incomplete. One bad item fails every finding in the batch.
     • The ordering bug predates the PR; the PR changes its consequence.
   • Expected fix. Since this function is being rewritten anyway:
     • Sanitize the raw dicts in data["findings"] before model_validate, and casefold impact first so that "High" is not downgraded to "low".
     • Drop the post-validation call.
     • A follow-up PR is also fine.
6. [Non-blocking] contrib/batch_scan/runner.py:94 (pre-existing, outside the diff): in multi-key pool mode, _pooled_get_chat_model(model=None) still rejects timeout=.
   • How it fails.
     • Core always calls get_chat_model(model=model, timeout=...) (llm_analyzer_base.py:938, :1021).
     • When create_api_key_pool_from_env() finds two or more keys, batch_scan.py:195-196 calls set_api_pool, which installs this factory.
     • With the PR's constructor transcribed, LLMAnalyzerBase, LLMMetaAnalyzer and GapFillAnalyzer all raise TypeError: _pooled_get_chat_model() got an unexpected keyword argument 'timeout'.
     • The nodes record the analyzer as unavailable (semantic_security_discovery.py:327).
   • Effect. The PR moves this failure from _patched_base_init to the factory. In pool mode, neither the restored compat analysis nor the new malformed-response handling runs. It fails closed.
   • Not fixed elsewhere. No other open PR in this batch, and not #763, changes this factory.
   • Expected fix.
     • Change the signature to def _pooled_get_chat_model(model=None, *, timeout=None).
     • Pass timeout to PooledChatModel, which already accepts it, and to the fallback.
     • Add a pooled-mode constructor test.
     • Alternatively, state in the PR body that pool mode is out of scope and track it separately.
7. [Non-blocking] contrib/batch_scan/runner.py:311: the Patch 1 guard in _verify_patch_targets does not check the newly forwarded timeout parameter (added to the patched signature at runner.py:133).
   • What the guard checks. The guard (runner.py:304-322) checks only the positional signature, a keyword-only node and response_schema. It is byte-identical to main.
   • What it misses.
     • On main the guard passes even though Patch 1 is already broken by timeout, which is the drift this PR fixes.
     • I tested a transcribed init against hypothetical core inits that remove or rename timeout. The guard passes, and construction then fails separately for each analyzer.
   • Mitigation. The new CI test (tests/test_batch_scan_security.py:68) would catch drift within this repo. The guard matters when contrib runs against a different installed skillspector, which is its documented purpose (contrib/batch_scan/CONTRIBUTING.md:132).
   • Expected fix. Require a keyword-only timeout in the guard, as it already does for node, and add a matching TestGuardPatch1Init case.
8. [Non-blocking] tests/nodes/test_meta_analyzer.py:1099: two of the six new parametrized cases already pass on main.
   • The two cases. overall_assessment = "42" and '"wrong type"' parse on main and already fail OverallAssessment's model_type check there, so they do not test this change.
   • The rest. The other four cases do test the change, and a revert of either validator is still caught. The six new cases plus the two updated tests at tests/nodes/test_llm_analyzer_base.py:995-1001 make eight:
     • Reverting only the findings validator makes 5 of the 8 fail.
     • Reverting only the assessment validator makes 1 of the 8 fail.
   • Expected fix (optional).
     • Use unparseable values such as "" or "{not json" for overall_assessment, or drop those cases if finding 3 restores the fallback.
     • Optionally add findings="null".
     • Optionally move the ValidationError import to module level.

PIC tradeoffs:

• Retry cost. A persistently malformed compat batch now costs up to 4 sequential provider calls instead of 1, with 0.5/1/2 s backoff. The gain is visible incompleteness. Semantic analyzers in batch scan run under a 90 s per-skill wall clock and a 30 s per-request ceiling. A provider that habitually wraps JSON in prose could therefore turn a partial result into a whole-skill timeout. I did not measure this with a real provider.
• Strict compat parser. The compat parser only strips code fences. Core CLI providers instead use the prose-tolerant _extract_json_object (src/skillspector/llm_utils.py:207). Providers that wrap JSON in prose now get incomplete scans instead of empty ones. Reusing the extractor would recover more of these responses.
• Private core symbol. contrib now imports _StructuredResponseValidationError (runner.py:46), and #797 does the same. It is the only exception a custom parse_response can raise to get retry and ledger handling: a ValidationError raised directly is a ValueError, and is re-raised as misconfiguration. A public alias would be cleaner. If the symbol is renamed, the import fails loudly at load time.
• Logs. Core's generic structured-response warning replaces the separate "invalid JSON" and "schema validation failed" warnings. Operators lose that distinction, but raw LLM output (pydantic input_value) no longer reaches the logs.
• TP4. _TP4Analyzer has no compat parse patch. On main it failed at construction. Now it spends one provider call and then fails with NotImplementedError. The final ledger outcome is the same.

Verification and gaps:

• Baselines on main, using main's own code.
  • The compat constructor raises TypeError on timeout=. In a graph run, all four LLM analyzers are unavailable and no LLM calls are made.
  • On the core meta path, {"findings": "invalid"} gives 1 call, a success with [], and meta completed.
  • With only the constructor fix applied, not JSON responses are reported complete.
• PR behaviour, from transcriptions. I transcribed runner.py:133-191 and meta_analyzer.py:126-142 onto main's code and ran them in the main venv.
  • The 4-call results described above.
  • In a graph run: completeness partial, static findings kept with llm_review_outcome failed, and a risk score of 83/CRITICAL, the same as main.
  • Neither the ledger events nor the logs contain the raw response.
  • ValidationError subclasses ValueError, but it is wrapped before reaching the except (ValueError, NotImplementedError): raise. Both loops catch the wrapper (llm_analyzer_base.py:1352, :1470), so no new crash path appears.
• Tests: which fail on main.
  • The 16-case test_compat_parse_failures_retry_and_remain_visible fails on main in every case, because of the constructor TypeError. All 16 pass under transcription, including the recover-on-retry case (2 calls).
  • The two updated tests at tests/nodes/test_llm_analyzer_base.py:995-1001 fail on main.
  • Four of the six new meta cases fail on main; see finding 8.
  • test_valid_stringified_meta_fields_are_supported passes on both and serves as a guard.
  • Two tests-pro tests that currently fail on main with the timeout TypeError should pass with this PR. That is inferred; I did not run them.
• Lint. ruff check and ruff format --check pass on the changed src/ and tests/ files.
• Commits. All seven author commits carry a sign-off. 7ed4487 is the automated merge of main.
• CI: No check runs exist on the head. The CI workflow run is action_required and needs a maintainer to approve it, and it must finish green before merge.
• Conflicts: The PR is mergeable with main (current with a0d489e). It conflicts with #797 in contrib/batch_scan/runner.py, where both rewrite the skillspector.llm_analyzer_base import, and both add tests to tests/test_batch_scan_security.py. The conflict is textual and the two PRs agree: #797 raises the same exception from gap-fill's parse_response, which covers the gap-fill [] path this PR leaves alone. Whichever PR lands second needs a rebase.
• Head update: I reviewed 7ed4487e6b7e3363be1ddf18381c83952000c8c3. The current head d1bcf2c4e02a831b3b47b54daad2636397411694 only adds automated merges of main (#691, #577, #608) from update-pr-branches.yml; those commits touch none of this PR's files (#691 only routes structured-output binding through an overridable method in llm_analyzer_base.py), so line references to llm_analyzer_base.py and llm_utils.py below are to the current head. The PR's own diff is unchanged (same patch-id), so this review applies to the current head.
• I did not run the PR's tests or code, per policy. Every PR-behaviour result above comes from my transcriptions run on main's code (Python 3.13, pydantic 2.13.4). CI uses Python 3.12.
• Gaps.
  • No live provider was exercised. I inferred from the code, without checking, that real LangChain structured-output parsers surface the stricter meta errors as ValidationError.
  • How often real models emit a prose overall_assessment, impact: "none", a null remediation or a null confidence is unknown.
  • I did not run the contrib tests-pro suite or mutation_max.py against the head.
  • I did not verify the PR body's claims: 536 targeted tests, 12 pinned sample skills, and source/wheel parity.

---

Decision: Changes Requested (reviewed head 7ed4487e6b7e3363be1ddf18381c83952000c8c3; current head d1bcf2c4e02a831b3b47b54daad2636397411694 only adds merges of main)

[yashrajp22]

Thanks for the pool timeout feedback! Waiting for a slot and retrying another key now use the same remaining deadline. Cancellation and constructor errors release the slot, and shorter connection limits stay intact. The sync and async regression tests pass against the installed wheel core.

[rng1995]

[SkillSpector Review]

Hi @yashrajp22, thank you for this round of fixes. Moving the meta soft-field repair ahead of validation, with case-folding so "High" stays high, means one quirky item no longer costs a whole meta batch. The pool factory now forwards timeout, so the compat analyzers can be constructed in multi-key pool mode. On main that construction fails for every analyzer.

Value and readiness:

• The bug is real. On main, the batch-scan compat parser turns malformed JSON or a schema-invalid response into an empty successful result. Main's core meta path does the same for {"findings": "invalid"}. With this PR such a response is retried up to 4 attempts and then recorded as llm_structured_response_invalid / ledger skipped, and the skill is counted as incomplete.

• New commits since my last review. That review was at d1bcf2c4e. Since then there are four new commits, each signed off:

  • 12fde02a: the review fixes.
  • 88fd9912: Patch 6 keeps shorter request deadlines.
  • 54cc2789: pooled requests share one deadline.
  • 5c309f96: formatting.

  The other new commits merge main or the author's own branch, and git log --remerge-diff shows no extra changes in any of them.

• Status. All 8 earlier findings are resolved, including both blockers. I found nothing new that blocks merging. Two non-blocking notes follow:

  • a test that would hang, rather than fail, if the deadline fix regresses;
  • the remaining textual conflict with #797.

  This is ready to merge once CI has run green. A maintainer has to approve the CI run first.

Previous findings:

The author's replies all cite 8219589f. That is the merge commit that brought the fixes into the branch. The fixes themselves are in 12fde02a. Each reply's description matches the code.

1. Resolved. F1 [Blocker], stale contrib test for _patched_base_init (contrib/batch_scan/runner.py:136-144).
   • Fix. test_patched_init_forwards_keyword_only_node now loops over a timeout of None, 7 and a lambda (contrib/batch_scan/tests/test_monkeypatch_fragility.py:265-279). For each one it asserts both assert_called_once_with(..., timeout=timeout) and call_args.kwargs["timeout"] is timeout.
   • Check. I replayed the updated test against my transcription of runner.py:136-144, with _original_base_init mocked. All 3 subcases pass.
   • Reply. Accurate.
   • Nit. The default case passes timeout=None explicitly instead of leaving it out. The two behave the same.
2. Resolved. F2 [Blocker], contrib docs said malformed responses return [].

   • Docs. contrib/batch_scan/docs/DESIGN.md:287-302 now describes:

     • up to 4 attempts, then ledger skipped / llm_structured_response_invalid, with the skill counted as incomplete;
     • the core WARNING text;
     • that Patch 3 sanitizes before validation.

     contrib/batch_scan/docs/README.md:344-348 narrows Limitation 5 to gap-fill. contrib/batch_scan/tests/docs/TEST_DESIGN.md:98 is updated, and the comment at runner.py:346 now reads "Validate dependencies before patching".

   • Check. My transcription agrees with the new text. Not-JSON input and a string findings value that is not valid JSON each take 4 calls and end as llm_structured_response_invalid. Valid JSON and fenced JSON each take 1 call and complete.

   • Reply. Accurate.

   • Optional. The _verify_patch_targets docstring at runner.py:310-311 still says the deep dependencies are "called inside try/except that could silently degrade". That is no longer true for Patches 2 and 3. I grepped contrib/batch_scan, leaving out the history docs, and found no other stale [] claims.

3. Resolved. F3 [Non-blocking], a prose overall_assessment failed the whole meta batch (src/skillspector/nodes/meta_analyzer.py).
   • Fix. _parse_stringified_assessment returns None on JSONDecodeError again (meta_analyzer.py:139-148). findings stays strict (:136). The new test test_optional_prose_assessment_keeps_finding_verdict covers 3 prose values (tests/nodes/test_meta_analyzer.py:1123).
   • Check. One valid verdict plus an overall_assessment of "HIGH risk: exfiltrates credentials", "LOW", "" or "null" takes 1 call and keeps the verdict, on both the compat path and the core path.
   • Reply. Accurate for unparseable prose.
   • Optional, not adopted. Strings that parse to something other than an object, such as "42" or '"LOW"', still fail the batch after 4 calls. main does the same, so this is not a regression.
4. Resolved. F4 [Non-blocking], a null or non-numeric confidence raised TypeError without a retry.
   • Fix. Both _normalize_confidence validators now wrap float(v) and re-raise TypeError and ValueError as ValueError("confidence must be numeric") (src/skillspector/llm_analyzer_base.py:613-616, meta_analyzer.py:100-103). Null and list cases were added to test_compat_parse_failures_retry_and_remain_visible, covering both recovery and exhaustion.
   • Check (transcription). A confidence of null, [1] or "abc" takes 4 calls and ends as llm_structured_response_invalid / ledger skipped. That holds for discovery and meta, sync and async. On the core structured meta path, null now takes 4 calls and ends invalid. Main's own code makes 1 call there, raises TypeError and records llm_batch_failed.
   • Reply. Accurate.
5. Resolved. F5 [Non-blocking], _sanitize_meta_finding ran after validation.

   • Fix. _patched_meta_parse now sanitizes the raw dicts in data["findings"] before model_validate, and the post-validation call is gone (runner.py:181-201). _sanitize_meta_finding case-folds impact, and maps non-strings and unknown labels to low (:169-178). New test: test_compat_meta_repairs_soft_fields_before_validation (tests/test_batch_scan_security.py:632).

   • Check (transcription). Compat meta with null explanation and remediation takes 1 call in each case:

     • High becomes high, and CRITICAL becomes critical.
     • none, None and catastrophic become low.
     • Null text fields become "".

     Hard-invalid items still take 4 calls and end invalid: a missing is_vulnerability, intent=None, or an item that is not a dict.

   • Reply. Accurate.

   • Nit. An impact of " high" becomes low, because surrounding whitespace is not stripped. Nothing in src/ or contrib reads impact, so this has no effect.

6. Resolved. F6 [Non-blocking], _pooled_get_chat_model rejected timeout= in multi-key pool mode.
   • Fix. The signature is now _pooled_get_chat_model(model=None, *, timeout=None). It builds PooledChatModel with _DEFAULT_REQUEST_TIMEOUT when timeout is None and with timeout otherwise, and the fallback forwards timeout (runner.py:94-103). New test: test_compat_pooled_constructor_forwards_timeout (tests/test_batch_scan_security.py:660).
   • Check. With an ApiKeyPool installed, every analyzer construction on main raises TypeError. With the head transcribed:
     • LLMAnalyzerBase with a timeout of None, 7 or a lambda builds a pooled model with a timeout of 30.0, 7.0 and 7.0.
     • LLMMetaAnalyzer and GapFillAnalyzer build one with 30.0.
     • End to end through the pool, valid input takes 1 call and succeeds. Not-JSON input and null confidence each take 4 calls and end invalid.
     • 0 slots are active afterwards.
   • Reply. This finding had no inline thread. The PR body describes the fix.
7. Resolved. F7 [Non-blocking], the Patch 1 guard did not check timeout.
   • Fix. The guard now requires both node and timeout to be keyword-only (runner.py:325-331). New test: test_guard_catches_missing_timeout_param (test_monkeypatch_fragility.py:281-287).
   • Check (transcription). The guard passes on main's current LLMAnalyzerBase.__init__. It raises for an init where timeout is missing, renamed or positional. Main's guard has no such check.
   • Reply. Accurate.
8. Resolved. F8 [Non-blocking], two of the new meta test cases already passed on main.
   • Fix. test_invalid_stringified_meta_findings_fail_validation now covers only findings (tests/nodes/test_meta_analyzer.py:1098), with the values "invalid JSON", "", "{not json", "null", "42" and '"wrong type"'. The prose-assessment cases moved to a separate acceptance test.
   • Check. Main's MetaAnalyzerResult accepts all six values as findings=[], so all six cases fail on main and test the change.
   • Reply. Accurate.
   • Optional, not done. The ValidationError import is still local to the function (:1100).
9. Still open, non-blocking. The conflict with #797. It now spans 3 files instead of 1. See finding 2.

Material findings

1. [Non-blocking] tests/test_batch_scan_security.py:698 and :700 (test defined at :688): test_pooled_request_slot_wait_obeys_deadline would hang, rather than fail, if the deadline fix regresses.
   • How the test works. It holds the only slot (:692) and then calls model.ainvoke (:698) or model.invoke (:700). The only pool.release(occupied) is in the finally at :704, which runs only after that call returns.
   • Pre-fix behaviour. Main's PooledChatModel shows what a regression would look like:
     • The sync path calls self._pool.acquire() with no timeout (main contrib/batch_scan/api_pool.py:488).
     • The async path awaits asyncio.to_thread(self._pool.acquire) (:531).
     • With no rate-limited keys, ApiKeyPool.acquire waits on its condition with no timeout (:234-235).
   • Measured. I ran a transcription of the test body against main's real PooledChatModel, each variant in a daemon thread. Both were still blocked after 3 s, with no result.
   • Nothing else bounds the wait.
     • pyproject.toml:64-66 declares only pytest, pytest-asyncio and pytest-cov, and uv.lock has no pytest-timeout.
     • [tool.pytest.ini_options] sets no timeout.
     • CI runs uv run make test-ci (ci.yml:92), and the test job has no timeout-minutes, so GitHub's 6-hour default applies.
   • Consequence. If the deadline logic were later reverted, this test would hold the CI job for up to 6 hours instead of reporting a failure. On this head the test passes.
   • Expected fix (optional, can be a follow-up).
     • Async variant. Use await asyncio.wait_for(model.ainvoke("unused"), 2). On a regression this fails cleanly: the task is cancelled, the worker thread gets the slot once finally releases it, and the final active_requests == 0 assertion fails.
     • Sync variant. Run invoke in a daemon thread, join(timeout=2), and assert the thread has finished before releasing occupied.
     • Stub the client builder. Also monkeypatch model._build_llm to a stub that raises. Otherwise, on a regression, the unblocked thread would build a real ChatOpenAI with key "synthetic" against the default OpenAI endpoint.
     • Precedents in the repo. tests/nodes/test_llm_analyzer_base.py:3846 uses asyncio.wait_for, and tests/unit/test_tp4_openai_http.py:138 uses join(timeout=...).
     • Why not pytest-timeout. Bounding the call inside the test is better than adding the plugin. Adding it would also switch on the currently inert pytest.mark.timeout marks in test_mp2_regex_backtracking.py and test_chmod_regex_backtracking.py, which is a wider change.
2. [Non-blocking] contrib/batch_scan/runner.py:43-47: the PR still conflicts with #797 (head 6fdb5733), now in 3 files.
   • The conflicts. git merge-tree --write-tree --name-only refs/remotes/prlive/796 refs/remotes/prlive/797 reports CONFLICT in 3 files. The merge base is current main.
     • contrib/batch_scan/runner.py. One hunk, the skillspector.llm_analyzer_base import. This PR adds _StructuredResponseValidationError (:43-47). #797 adds ledger_events_for_batches there, plus an inspection_ledger import. Line 40 (from pydantic import ValidationError) merges cleanly. Both PRs make byte-identical edits to _pooled_get_chat_model, _patched_base_init and _patched_chatopenai_init, so those merge cleanly too.
     • contrib/batch_scan/tests/test_monkeypatch_fragility.py. This PR replaces test_patched_init_forwards_keyword_only_node with the subTest loop and adds test_guard_catches_missing_timeout_param. #797 only appends timeout=None to the old assertion.
     • tests/test_batch_scan_security.py. Two hunks. In each, one side is empty, because each PR appends its own tests.
   • What merges cleanly. The api_pool.py diffs against main have the same patch-id (301ff7a7) in both PRs, so that file auto-merges. The five tests both PRs add identically merge without duplicates.
   • Resolving. I resolved the hunks myself on a scratch copy and parsed the results with ast. A union of both sides parses for the runner.py imports and for tests/test_batch_scan_security.py. In the fragility file, a union also parses but is wrong: #797's leftover assertion lands inside test_guard_catches_missing_timeout_param, where instance and original_init are unbound, so the test raises NameError. Taking this PR's side there parses and is correct.
   • Consequence. Whichever PR lands second needs a rebase. A careless resolution could drop this PR's guard test or its three-value forwarding test, or break the guard test.
   • Expected fix (when the second PR is rebased).
     • Union the two import lists.
     • Keep each PR's appended tests in tests/test_batch_scan_security.py, once each.
     • Take this PR's version of the fragility test file. #797 does not change the Patch 1 guard, so this PR's guard and its test must stay.

PIC tradeoffs:

• Pooled requests now share one deadline (54cc2789; #797 has the identical change). PooledChatModel's timeout is now one budget covering slot waiting, the request and key retries (api_pool.py:490-494, :519-529).
  • Core semantic and meta analyzers pass the remaining workflow budget. resolve_input.py:56 always installs the 600 s workflow budget (state.py:46), and semantic_security_discovery.py:207-208 passes it on as a callable. Pooled requests there therefore get about 600 s minus elapsed time, not the 30 s fallback.
  • Measured. I ran main's graph with my transcription of the head's pool logic, with time scaled 1:30 and both keys returning 429 first. The request waited out the key backoff and recovered: all four LLM analyzers were ok and completeness was complete.
  • Gap-fill passes no timeout (contrib/batch_scan/gap_fill.py:187), so it uses the 30 s fallback (runner.py:99, :113). That equals the minimum key backoff (api_pool.py:71). During a pool-wide rate limit, gap-fill can therefore stall for about 30 s and then return [] with no ledger entry. The scaled transcription made 2 calls and returned [].
  • Compared with main. Pooled gap-fill on main returns [] immediately, from the constructor TypeError. Dropped gap-fill batches are the documented Limitation 5. The outcome is the same and the wait is longer.
  • Optional. The PooledChatModel docstring at api_pool.py:438-439 still calls timeout the "Request timeout in seconds passed to each ChatOpenAI instance". It should say timeout is the total budget.
• Patch 6. It now takes the per-field minimum of the requested httpx.Timeout and the 30 s / 8 s caps (runner.py:258-265), so shorter budgets passed through _build_llm survive. _build_llm sets max_retries=0, and core treats PooledChatModel as having no native retries. That removes the SDK-level double retry while core's own retries still apply.
• Unchanged from the previous review, and acceptable:
  • A persistently malformed batch costs up to 4 calls, with 0.5/1/2 s backoff, inside the 90 s per-skill limit (contrib/batch_scan/batch_scan.py:204).
  • The compat parser strips only code fences. Core's _extract_json_object also tolerates surrounding prose.
  • contrib imports the private _StructuredResponseValidationError (runner.py:46).
  • Core's generic structured-response warning replaces the two contrib warnings, so raw LLM output no longer reaches the logs.
  • _TP4Analyzer spends one call and then fails with NotImplementedError. The final ledger outcome is the same as on main.

Verification and gaps:

• CI. The CI run on 581ded07 is action_required, and the head has 0 check runs. A maintainer must approve the run, and every check must finish green before merge. mergeStateStatus is BLOCKED, because reviewDecision is still CHANGES_REQUESTED from my earlier review; this approval replaces it. All 7 review threads are resolved.
• Conflicts.
  • With main (5edc347c): the PR merges cleanly (git merge-tree --write-tree exits 0), and GitHub reports MERGEABLE.
  • With #797: see finding 2.
  • With #798 (102bec18): it conflicts with this PR only in the tests both append to tests/test_batch_scan_security.py. #798's cli.py conflict is with main itself.
  • With #790: it conflicts with main itself, not because of this PR.
  • With #741, #793, #794 and #795: each merges cleanly with this PR.
• Lint. ruff check and ruff format --check pass on the changed src/ and tests/ files. The contrib files have the same ruff errors and formatting deviations as on main, and CI does not lint contrib.
• How I tested the PR's behaviour. All results above come from my own transcription of the head's logic. I ran it on main 5edc347c with main's venv (Python 3.13). No PR code was imported. Further transcription checks:
  • Cancelling during ainvoke or during a slot wait leaves 0 active slots.
  • A 429 on both keys recovers when the backoff is shorter than the budget.
  • No raw response text reaches the ledger events.
• Tests that fail on main. All six findings cases in tests/nodes/test_meta_analyzer.py:1098 fail on main. The compat retry tests fail on main because of the constructor TypeError. The prose-assessment cases pass on main and act as guards.
• I did not run the PR's tests or code, per policy.
• Gaps.
  • CI uses Python 3.12; my transcriptions ran on 3.13.
  • I did not exercise the multi-process path, where the pool is a BaseManager proxy. The transcription used an in-process ApiKeyPool.
  • I did not use a live provider. Real ChatOpenAI/httpx timeout behaviour and real 429 classification are inferred from the code.
  • The pool storm results come from scaled timings. I did not measure how often a real pool sees every key rate-limited at once.
  • I did not run the contrib tests-pro suite or mutation_max.py.
  • I did not verify the PR body's numbers: 608 / 579 tests, 6,243 regression tests, 19 sample pairs, the 12-skill corpus, and the two live NVIDIA Build checks.

---

Decision: Approved (reviewed head 581ded0704be8a2708352c7d664697441ffff5d1)