Skip to content

fix(report): keep missing references as non-blocking caveats - #805

Open
efegokdemir wants to merge 1 commit into
NVIDIA:mainfrom
efegokdemir:codex/804-cli-reference-caveats
Open

efegokdemir wants to merge 1 commit into
NVIDIA:mainfrom
efegokdemir:codex/804-cli-reference-caveats

Conversation

@efegokdemir

@efegokdemir efegokdemir commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What

Treat scans with only missing-reference ledger caveats as complete when every discovered file was inspected and no other limitations exist. Preserve the reference and exception details in reports, and keep --fail-on-incomplete from rejecting these scans.

Why

The MCP safety gate already allowed a missing-reference-only result, but CLI and SARIF completeness still reported it as partial and raised the recommendation to CAUTION. This made the same fully inspected workspace appear incomplete depending on the output path. Ambiguous references and actual coverage gaps still need to block completeness.

How

  • Centralize the missing-reference-only predicate and use it when finalizing the shared completeness ledger.
  • Represent that state as complete_with_caveats, with is_complete=true, only when there are no limitations or uninspected files.
  • Preserve the caveat status in SARIF and keep the MCP gate aligned with the shared predicate.

Testing

  • Reproduced on pristine main: a no-findings scan with only missing workspace references returned partial, CAUTION, and exit code 1 with --fail-on-incomplete.
  • Added CLI JSON, terminal, and SARIF regression coverage; verified missing-reference details remain visible and the strict gate succeeds. Ambiguous-reference and opaque/incomplete coverage cases remain blocking.
  • pytest -q tests/test_reference_destinations.py tests/test_reference_destination_kinds.py tests/nodes/test_finalize_inspection_ledger.py tests/nodes/test_analysis_completeness.py tests/nodes/test_security_remediation.py — 755 passed.
  • uv run --frozen pytest -m 'not integration and not provider' -q tests/ — 10,540 passed, 14 skipped, 133 deselected, 4 xfailed.
  • uv run --frozen make lint — passed.
  • uv run --frozen make format-check — passed.
  • uv build --out-dir /tmp/skillspector-804-dist — passed.

Compatibility

Single-key and normal complete-scan behavior is unchanged. Missing references remain visible; only their effect on completeness/recommendation changes when there is no actual inspection gap. Ambiguous references and incomplete coverage remain blocking.

Checklist

  • Added regression tests for the reported behavior.
  • Ran targeted and full non-integration/non-provider tests.
  • Ran lint, formatting, and package build.
  • Prepared with AI assistance.

Fixes #804

  • uv run --frozen make test-ci — passed (10,540 passed, 14 skipped, 133 deselected, 4 xfailed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI still reports zero-finding scans as partial/CAUTION when the only exceptions are reference_missing (#510 fixed MCP only)

1 participant