Skip to content

fix(mcp): advertise that track_container needs an explicit user request - #445

Open
dodeja wants to merge 7 commits into
mainfrom
docs/mcp-track-container-only-when-asked
Open

dodeja wants to merge 7 commits into
mainfrom
docs/mcp-track-container-only-when-asked

Conversation

@dodeja

@dodeja dodeja commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Why

In #400's baseline eval, the agent called track_container on "Where is it?" without being asked. Models see the initialize instructions and the tools/list descriptions, not docs/mcp/home.mdx. The "call only when asked" rule lived only in the MDX, and the instructions applied it to the two set_ tools but not to track_container. #400 merged into mcp-preview, not main, so the guidance on when to call track_container is still missing from main. #440 has since merged most of #400's other content.

What

This PR changes advertised text: the track_container description in tools/list and the initialize instructions. It is not docs-only, so the earlier "no tool description changes" claim no longer holds.

  • packages/mcp/src/instructions.ts: the full variant now says "Call track_container only when the user asks to track or add a number, and the two set_ tools only when the user asks to change those records." The reduced variant (no set_ tools registered) says "Call track_container only when the user asks to track or add a number." "They can detach a party" became "The two set_ tools can detach a party" so it cannot read as including track_container.
  • packages/mcp/src/server.ts: the track_container description now says it adds a tracking request to the account, so call it only when the user asks to track or add a number, and to use search_container for a lookup, which only reads the account. It makes no claim about already-tracked numbers, because passing scac can still create a second request for a number that has an in-progress request.
  • docs/mcp/home.mdx: a short tip in the track_container section with the same rule.
  • docs/mcp/setup/agent-plugins.mdx: the "Where is container CAIU1234567?" example no longer lists track_container. It says the agent should look the number up with search_container and get_container, and should call track_container only when asked to track or add a number.
  • Snapshots (tools-list.json, instructions.txt) regenerated with vp test run packages/mcp/src/tools-list.snapshot.test.ts -u. The merge with main took main's tools-list.json (feat(mcp): publish full output schemas for get_container, get_shipment_details, get_container_transport_events #456's closed output schemas) and regenerated it. The only difference from main is the track_container description.
  • No change to annotations, tiers, toolsets, defaults, auth, or write-tool placement.

Tests

New tests assert what the real server returns, not the MDX:

  • protocol-compat.test.ts, "advertises the explicit-request rule for track_container in instructions and tools/list": checks client.getInstructions() and the tools/list description. An earlier test also keeps the MDX tip aligned with the advertised annotations.
  • instructions.test.ts, "keeps the explicit-request rule wherever track_container is described": every toolset combination that registers track_container carries the rule.

Fail-without-fix: with the tests added and the source unchanged, both new tests failed (the core toolset instructions and the tools/list description lacked the rule). With the fix they pass.

Validation on the head: vp run check, vp run test (@terminal49/mcp: 50 files, 744 tests), vp run build, and the protocol smoke at 2024-11-05, 2025-06-18, 2025-11-25, and 2026-07-28 (30 tools listed each) all pass.

Open item for Akshay

The MDX tip also says "If your client doesn't ask, confirm with the user first" and "offer to track" when a number isn't found. Those clauses exist only in the tip. The server text does not tell the model to confirm first or offer to track. Whether the server should say so is a policy choice and is left as is.

Out of scope and pre-existing, left unchanged here:

  • packages/mcp/MCP_FLOW.md (an unpublished package doc that nothing links to) prescribes the behavior this PR bans. At lines 7, 38-50 and 197-199, a user asks "Get container information for CAIU2885402" and Claude calls track_container. It needs its own rewrite.
  • Stale "only write tool" text in packages/mcp/README.md:70, DIRECTORY_PREFLIGHT.md:10, and eval/README.md:65.

Refs #400, #440.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

RetriggerView in GreptileConfidence Score: 5/5

This PR appears safe to merge.

Summary

Adds guidance to call track_container only when the user asks to track or add a number.

  • Agents use tracking only when users ask.

No actionable issues found. Tests were not run during this review.

Reviews (1) · Last reviewed commit: "test(mcp): tie the track_container guida..." · Reviewed by Greptile

dodeja and others added 2 commits October 17, 2026 00:41
track_container adds a tracking request to the account and is the tool
advertised with readOnlyHint: false. Tell agents to call it only when
the user asks to track a number, to confirm first when the client does
not ask for approval, and to look numbers up with search_container.

Carries the guidance from #400 without its code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assert that tools/list still marks track_container as not read-only and
search_container as read-only, and that the MCP guide's track_container
section keeps the call-only-when-asked guidance that depends on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
terminal49 🟢 Ready View Preview Oct 19, 2026, 6:06 AM

@vercel

vercel Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
api Ready Ready Preview Oct 19, 2026 6:06am UTC

Request Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T06:10:32.128413Z ca4cb49 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the risk-shadow:medium Shadow-only risk signal: medium label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Shadow risk calibration

Signal Result Authority
Path classifier risk:high Deterministic advisory policy
heuristic fallback risk-shadow:high Shadow only

Calibration result: agreement.

Shadow risk never blocks or enables merge. Keep it in observation mode until roughly 50 pull requests have human-reviewed labels, then calibrate before considering any policy use.

@github-actions github-actions Bot added the risk:unknown Advisory evidence unavailable or changed; no merge permission label Oct 19, 2026
@github-actions github-actions Bot added risk:high Advisory path risk; current-head human review needed and removed risk:unknown Advisory evidence unavailable or changed; no merge permission labels Oct 19, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3b9368614

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +178 to +180
expect(trackSection).toMatch(
/call `track_container` only when the user asks to track/,
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Advertise the consent rule through the MCP server

This assertion only verifies text in the independently published MDX file, so it does not prevent the reported behavior in a normal MCP session: models receive the initialization instructions and tools/list, not docs/mcp/home.mdx. The current instructions even restrict “only when the user asks” to the two set_ tools (instructions.ts:52), while the track_container description (server.ts:1316-1324) contains no consent rule; consequently, a model can still call it for “Where is it?” and create a tracking request. Add this rule to an advertised instruction or tool description and assert that returned text here.

AGENTS.md reference: AGENTS.md:L9-L14

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid finding, fixed in 46a4415.

Before: the 'only when the user asks' rule for track_container lived only in docs/mcp/home.mdx. The initialize instructions limited it to the two set_ tools, and the track_container tool description said nothing about it, so a model could call it for a lookup like 'Where is it?'.

Now the server advertises it, using the PR's own wording:

  • Initialize instructions: 'Call track_container only when the user asks to track or add a number, and the two set_ tools only when the user asks to change those records.' The reduced variant (no set_ tools) says 'Call track_container only when the user asks to track or add a number.'
  • tools/list track_container description: 'Adds a tracking request to the account, so call this only when the user asks to track or add a number; to look a number up, use search_container, which only reads the account.'

No change to annotations, tiers, toolsets, defaults, auth or write-tool placement.

Tests assert what the real server returns, not the MDX:

  • protocol-compat.test.ts 'advertises the explicit-request rule for track_container in instructions and tools/list' (client.getInstructions() and the tools/list description).
  • instructions.test.ts 'keeps the explicit-request rule wherever track_container is described' (every toolset combination whose instructions mention track_container).

Fail-without-fix: with the tests added and the source unchanged, both failed (core toolset instructions and the tools/list description lacked the rule). With the fix both pass. The tools-list.json and instructions.txt snapshots were regenerated with 'vp test run packages/mcp/src/tools-list.snapshot.test.ts -u'. vp run check, test, build and protocol smoke (2024-11-05, 2025-06-18, 2025-11-25, 2026-07-28) pass locally.

The rule lived only in docs/mcp/home.mdx. Put it in the initialize instructions and the track_container tool description, and assert both through the real server. Snapshots regenerated with the repo's update path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added risk-shadow:high Shadow-only risk signal: high risk:unknown Advisory evidence unavailable or changed; no merge permission and removed risk-shadow:medium Shadow-only risk signal: medium risk:high Advisory path risk; current-head human review needed labels Oct 19, 2026
@github-actions github-actions Bot added risk:high Advisory path risk; current-head human review needed and removed risk:unknown Advisory evidence unavailable or changed; no merge permission labels Oct 19, 2026
dodeja and others added 2 commits October 18, 2026 22:50
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Correct the agent-plugins example to the explicit-request contract, say 'The two set_ tools' where 'They' could read as including track_container, make the track_container description accurate for already-tracked numbers, and guard the instructions test on registration instead of generated text. Snapshots regenerated with the repo's update path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added risk:unknown Advisory evidence unavailable or changed; no merge permission and removed risk:high Advisory path risk; current-head human review needed labels Oct 19, 2026
@dodeja dodeja changed the title docs(mcp): say to call track_container only when the user asks fix(mcp): advertise that track_container needs an explicit user request Oct 19, 2026
@github-actions github-actions Bot added risk:high Advisory path risk; current-head human review needed and removed risk:unknown Advisory evidence unavailable or changed; no merge permission labels Oct 19, 2026
A number with an in-progress request under another SCAC can still get a second request when scac is passed, so drop 'only for a number that is not already tracked' from the description. Soften the agent-plugins sentence to 'should'. Snapshot regenerated with the repo's update path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added risk:unknown Advisory evidence unavailable or changed; no merge permission and removed risk:high Advisory path risk; current-head human review needed labels Oct 19, 2026
@github-actions github-actions Bot added risk:high Advisory path risk; current-head human review needed risk:unknown Advisory evidence unavailable or changed; no merge permission and removed risk:unknown Advisory evidence unavailable or changed; no merge permission risk:high Advisory path risk; current-head human review needed labels Oct 19, 2026

This branch was successfully deployed

2 active deployments
Preview — ca4cb495 Deployed Oct 19, 2026 by vercel[bot]
staging - docs — ca4cb495 Deployed Oct 19, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Advisory path risk; current-head human review needed risk-shadow:high Shadow-only risk signal: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant