Skip to content

insufficient_evidence headline claims the tool surface "could not be fully enumerated" when coverage reports 12/12 reachable #396

Description

@pengfei-threemoonslab

Summary

The insufficient_evidence headline says the tool surface could not be fully enumerated, directly above coverage numbers stating that all 12 catalog tools were reached with zero unbound tools, zero identity gaps and zero source warnings. The real condition is low extraction confidence, not incomplete enumeration, and the message picks the wrong half of its own disjunction.

Evidence

Scan output, adk-samples#1745 / python/agents/smart_closer, CLI 0.16.0b7:

Decision: insufficient_evidence
Summary: Insufficient evidence: the tool surface could not be fully enumerated
         (create_salesforce_quote [adk_agent]).
Evidence coverage: mixed (12 low-confidence tool(s); 12/12 catalog tools reachable;
                   36 semantic evidence gap(s); 0/12 actions pass-eligible)
"binding_coverage":  {"total_catalog_tools": 12, "reachable_tools": 12,
                      "unbound_tools": 0, "possible_tools": 0,
                      "pass_eligible": true, "gap_count": 0},
"identity_coverage": {"canonical_tools": 12, "pass_eligible_tools": 12,
                      "ambiguous_name_count": 0, "gap_count": 0},
"source_warning_count": 0

The incomplete_surface gap body states the condition accurately as a disjunction:

"why": "tool surface enumeration is incomplete or not high confidence"

Only the second disjunct is true here. The headline reports the first.

Why this matters

  • It contradicts the numbers printed on the following line, which is the fastest way to make a deterministic tool look broken.
  • It sends the user to fix enumeration when nothing is missing.
  • It obscures what is actually being asked. The gap's own fields say so plainly: accepted_values: ["reviewed_explicit_inventory"], requires_human_review: true. This is a human-attestation gate, not an evidence gap.

Naming it honestly — "no reviewed inventory has attested this tool surface" — is both truer and more actionable, and it is the only framing under which the transcription work the remedy asks for is obviously worth doing.

Suggested direction

Split the two conditions in the reason text and select on which one fired:

  • enumeration genuinely incomplete (unbound/possible tools, source warnings, unresolved names) → "the tool surface could not be fully enumerated"
  • surface complete but not attested → "no reviewed tool inventory attests this surface (extraction_confidence=medium)"

Related: #393 — why medium is unavoidable for ADK in the first place.


Found during a first-time-user adoption walk of adk-samples#1745 on 2026-08-20.

Activity

  1. added
    bugSomething isn't working
    P2Queued; valuable but not blocking
    area:verifyverify / preview orchestration and SHIP-VERIFY-* checks
    on Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Queued; valuable but not blockingarea:verifyverify / preview orchestration and SHIP-VERIFY-* checksbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions