[dodeja]

Summary

Follow-up to #397 from live testing on mcp.t49.dev and an audit of the eval suite.

• Evals run on the PR preview. The live eval only ran against production, so a PR's tool changes were never evaluated before deploy. The preview job now runs npm run eval against the PR's Vercel preview (once per run, not per protocol version). The production run stays as a deployed-state check.
• New eval cases for list_shipments and list_containers: has_more/next_page when meta.total exceeds the page, no ID overlap between pages 1 and 2, a SCAC filter round-trip, a pickup_lfd round-trip derived from returned data, and pre-request rejection of duplicate filters and impossible dates.
• Error punctuation. SDK filter messages are fragments, so the appended guidance ran on (...comparison expressions Use the filter values...). The unit test now asserts the exact text.

Expected eval results until the backend fix deploys

list_containers reports a next page and list_containers pickup_lfd filter fail against production today. Both are fixed in https://github.com/Terminal49/t49/pull/4200. These cases are the regression gate for it.

Testing

• Error-message test fails without the fix and passes with it.
• @terminal49/mcp: 379 tests pass; eval:check and lint pass.
• The new eval cases have not run locally (no long-lived token here). The preview job in this PR is their first run.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Confidence Score: 5/5

The PR appears safe to merge, with non-blocking improvements needed in the live checks and generated status type.

Findings

1.  Ignored date filters pass ▶
2.  Valid status arrays need casts ▶
3.  Broken list calls skip ▶
4.  Invalid second pages pass ▶

Fix with agent prompt

### Issue 1
packages/mcp/eval/tools.eval.ts:537-543
The new `pickup_lfd` check also filters by the sampled container’s exact `number`, then only checks that its ID comes back. If the server ignores `pickup_lfd`, the number filter still returns that container and CI passes this check. Add a second call with the same number and a different date, and require that container to be absent.

### Issue 2
sdks/typescript-sdk/scripts/generate-filter-types.mjs:31
The generated `ContainerStatusFilter` only allows plain status names inside arrays. The MCP tests and SDK checks support expressions such as `['@exists']`, `['=available']`, and `['available,on_ship']`, but TypeScript callers cannot use them without a cast. Generate the array member type from the same supported expressions as the scalar member.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 3
packages/mcp/eval/tools.eval.ts:undefined-451
The new next-page check skips before asserting `score.contractPass`. If the tool returns an error or drops `meta.total`, `listTotal` returns `undefined`, which becomes zero and skips the failed check. This can hide a broken response in CI. Assert that the call succeeded and returned a numeric total before skipping accounts with two or fewer records.

### Issue 4
packages/mcp/eval/tools.eval.ts:undefined-467
The page-overlap check can pass without receiving a valid second page. `listItems` turns malformed `items` into an empty array, and `.every()` then returns true. `requiredKeys` only checks that `items` exists, so a response such as `items: null` passes. Assert the array shape and valid IDs; when the first page's total exceeds five, also require a non-empty second page.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds preview evaluations and live checks for list filtering and pagination. The wider diff also adds typed filters across the API, SDK, and MCP, optional custom-field reads, documentation search, and automation updates.

• The pickup_lfd check needs a nonmatching-date case to catch ignored filters.
• The generated status type excludes supported array expressions.
• dodeja acknowledges that production container pagination and LFD checks fail until the backend fix deploys; those expected failures are not reported as new issues.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Spec[OpenAPI filter catalog] --> SDK[SDK filter types and checks]
  Spec --> MCP[MCP input schemas]
  MCP --> SDK
  SDK --> API[Terminal49 API]
  CI[Preview CI job] --> Eval[Live list evaluations]
  Eval --> Preview[Vercel MCP preview]
  Preview --> MCP

Loading

_{Reviews (2) · Last reviewed commit: "test(mcp): eval list filtering and pagin..." · Reviewed by Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
api	Ready	Preview	Oct 8, 2026 6:07pm UTC

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-08T18:12:52.614810Z	b976808	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5c2edc30d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fail when pagination totals are missing

When meta.total is absent or changes to a non-number, listTotal() returns undefined, this expression coerces it to zero, and line 451 skips the test. Consequently, a response that loses its total and also reports incorrect has_more/next_page values passes this new regression gate. Require a numeric total before conditionally skipping, or fail explicitly when the pagination metadata is malformed.

Useful? React with 👍 / 👎.

[greptile-apps]

Broken list calls skip

The new next-page check skips before asserting score.contractPass. If the tool returns an error or drops meta.total, listTotal returns undefined, which becomes zero and skips the failed check. This can hide a broken response in CI. Assert that the call succeeded and returned a numeric total before skipping accounts with two or fewer records.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/mcp/eval/tools.eval.ts
Line: 451

Comment:
**Broken list calls skip**

The new next-page check skips before asserting `score.contractPass`. If the tool returns an error or drops `meta.total`, `listTotal` returns `undefined`, which becomes zero and skips the failed check. This can hide a broken response in CI. Assert that the call succeeded and returned a numeric total before skipping accounts with two or fewer records.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Invalid second pages pass

The page-overlap check can pass without receiving a valid second page. listItems turns malformed items into an empty array, and .every() then returns true. requiredKeys only checks that items exists, so a response such as items: null passes. Assert the array shape and valid IDs; when the first page's total exceeds five, also require a non-empty second page.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/mcp/eval/tools.eval.ts
Line: 467

Comment:
**Invalid second pages pass**

The page-overlap check can pass without receiving a valid second page. `listItems` turns malformed `items` into an empty array, and `.every()` then returns true. `requiredKeys` only checks that `items` exists, so a response such as `items: null` passes. Assert the array shape and valid IDs; when the first page's total exceeds five, also require a non-empty second page.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
terminal49	🟢 Ready	View Preview	Oct 7, 2026, 9:31 PM

[greptile-apps]

Ignored date filters pass

The new pickup_lfd check also filters by the sampled container’s exact number, then only checks that its ID comes back. If the server ignores pickup_lfd, the number filter still returns that container and CI passes this check. Add a second call with the same number and a different date, and require that container to be absent.

Knowledge Base Used: MCP observability and quality assurance

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/mcp/eval/tools.eval.ts
Line: 537-543

Comment:
**Ignored date filters pass**

The new `pickup_lfd` check also filters by the sampled container’s exact `number`, then only checks that its ID comes back. If the server ignores `pickup_lfd`, the number filter still returns that container and CI passes this check. Add a second call with the same number and a different date, and require that container to be absent.

**Knowledge Base Used:** [MCP observability and quality assurance](https://app.greptile.com/terminal49/-/custom-context/knowledge-base/terminal49/api/-/docs/mcp-observability-and-quality.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Valid status arrays need casts

The generated ContainerStatusFilter only allows plain status names inside arrays. The MCP tests and SDK checks support expressions such as ['@exists'], ['=available'], and ['available,on_ship'], but TypeScript callers cannot use them without a cast. Generate the array member type from the same supported expressions as the scalar member.

Prompt To Fix With AI

This is a comment left during a code review.
Path: sdks/typescript-sdk/scripts/generate-filter-types.mjs
Line: 31

Comment:
**Valid status arrays need casts**

The generated `ContainerStatusFilter` only allows plain status names inside arrays. The MCP tests and SDK checks support expressions such as `['@exists']`, `['=available']`, and `['available,on_ship']`, but TypeScript callers cannot use them without a cast. Generate the array member type from the same supported expressions as the scalar member.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b976808351

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Verify pickup LFD without the exact-number fallback

When the backend ignores pickup_lfd but still applies the exact number filter, this call returns the sampled container and the new regression gate passes. Since the test is intended to detect ineffective LFD filtering, also query the same number with a deliberately nonmatching date, or otherwise validate results without a second filter that independently guarantees the expected row.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Require a valid second page before passing overlap checks

When page 1 has five rows but page 2 is broken, listItems() converts a malformed or missing items value into [], and .every() then passes vacuously; an unexpectedly empty second page also passes even when the first response advertises more rows. Assert the second-page array shape and, when page 1 indicates more results, require at least one valid ID before checking overlap.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Include supported expressions in status-array types

The runtime and MCP schema accept status-array elements such as '=available', '@exists', and 'available,on_ship', but this generated public type restricts arrays to plain ContainerStatus values. TypeScript consumers therefore need casts for inputs the SDK validates and forwards successfully; generate the array member from the same expression union as the scalar member rather than editing the generated output directly.

AGENTS.md reference: AGENTS.md:L42-L43

Useful? React with 👍 / 👎.

[dodeja]

Superseded by #411, which carries this change rebased onto main.