Skip to content

fix(ledger): exclude scope boundaries from analyzer work - #802

Merged
rng1995 merged 1 commit into
NVIDIA:mainfrom
efegokdemir:codex/skill-800-scope-exclusion-accounting
Oct 14, 2026
Merged

rng1995 merged 1 commit into
NVIDIA:mainfrom
efegokdemir:codex/skill-800-scope-exclusion-accounting

Conversation

@efegokdemir

Copy link
Copy Markdown
Contributor

What

Why

Binary members that are intentionally out of scope were included in each analyzer's planned work. Finalization then counted the unrecognized out_of_scope outcome as failed work even though the analyzer status was completed, producing an inconsistent report that SkillEvaluator rejects.

How

analyzer_status_for_events now excludes scope_boundary records from the analyzer work plan. If an analyzer has only scope boundaries, its status is not_applicable; the exclusions are still reported separately.

Testing

  • The new finalization regression failed before the fix: it counted 2 planned items instead of 1 and treated the scope boundary as failed.
  • After the fix, the regression and ledger tests pass.
  • make test-unit: 10,539 passed, 14 skipped, 133 deselected, 4 xfailed.
  • make lint, make format-check, and python -m build --no-isolation passed.
  • Mypy on the changed module reports an existing call-overload error at inspection_ledger.py:1081; the same error is present on the base commit. The changed module passes when that existing diagnostic is suppressed and imports are silenced.

AI assistance

ChatGPT assisted with the implementation and regression tests. The commit is DCO signed off.

Signed-off-by: Efe <efe@rexcode.co.uk>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

@efegokdemir thank you for tracing #800 to the one summarizer every analyzer shares, and for a regression that fails on the old accounting.

Value and readiness

Real report-integrity defect. Static analyzers emit out_of_scope SCOPE_BOUNDARY rows for unreferenced binary members. analyzer_status_for_events counted those rows as planned work, and finalize_ledger maps the unknown out_of_scope outcome to failed. Reports therefore said completed with failed: 8, which SkillEvaluator rejects (102 of 105 skills in #800). Excluding scope boundaries at the summarizer (inspection_ledger.py:529-532) fixes this at the source, and the exclusions still appear in scope_exclusions. Ready for final maintainer review.

Material findings

None blocking.

  1. Optional: a CLI-level regression using #800's shape (a DOCX member with embedded fonts) would pin the end-to-end JSON. The new finalize_ledger test already covers the accounting.

PIC tradeoffs

Output shift, not a defect. planned_work for static analyzers drops by the number of unreferenced binary members. An analyzer whose only rows are scope boundaries now reports not_applicable (no_applicable_files) instead of completed. Consumers that compare counts across versions will see the change.

Verification and gaps

  • The only analyzer-tagged SCOPE_BOUNDARY producer is static_runner.py:2530-2540, for unreferenced BINARY content. Referenced binaries stay WORK_ITEM PARTIAL, so real gaps still count as planned work. build_context's discovery scope rows carry no analyzer and are not summarized.
  • I traced finalize_ledger. Path coverage already ignored OUT_OF_SCOPE outcomes, and out-of-scope inventory is excluded from relevant components unless referenced, so removing these targets from primary_targets does not change the fully/partially/entirely-inspected counts. If every analyzer's rows were scope boundaries, the fallback branch only touches components whose inventory is already out of scope, so no gap can become complete.
  • static_yara and semantic_security_discovery build planned_work by hand but never emit scope boundaries. guard_analyzer_node is unaffected.
  • Tests: the new finalize test fails on the old code (planned 2, one counted failed), and the unit test pins not_applicable.
  • Main (b07f181d) is 49 commits ahead; main's new RULE_SET rows sit nearby but do not overlap, and git merge-tree is clean.
  • CI: all 6 checks are green on 28ecaf3. I did not run PR code, per policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

analyzer_statuses reports completed while counting out-of-scope binary members as failed, so SkillEvaluator rejects the report

2 participants