feat(worker): anonymous read-only MCP news tools (latest/search/story/digest) + truthful discovery docs

[duyet]

Closes #227. Part of the umbrella #232. Pairs with #226 (which consumes the
same contract module — that PR should merge after this one).

The problem this closes

POST /api/mcp was fully admin-gated. checkAuth ran before any JSON-RPC
parsing, so an unauthenticated agent got an HTTP 401 for everything, and every
advertised tool was an operator tool (push_items, upsert_source,
delete_source, trigger_ingest, get_status, list_sources).

Meanwhile the machine-discovery layer promised an anonymous read path that
did not exist:

Surface	What it claimed	What was true
.well-known/mcp/server-card.json	capabilities.tools + a public endpoint, no auth declared	every tool was admin-only
same card	capabilities.resources and capabilities.prompts	neither was implemented
.well-known/agent-card.json	skills public-digest ("via GET /api/public") and story-markdown ("at GET /api/story/{id}.md")	both unreachable over MCP
SKILL.md	MCP (read + admin): POST …/api/mcp	the "read" half did not exist
openapi.json	/api/mcp as a flat MCP Streamable HTTP → 200	no auth, no tools, no limits described
/mcp (indexed, public)	documented the six operator tools	—

Any agent that followed the published machine docs got a 401 on its first call.

The shared read-tool contract (this is the point of #227 first)

One module defines the four read capabilities:
apps/web/src/lib/public-read-tools.ts. It owns the canonical names,
descriptions, input schemas, annotations, argument validation, and the bound
constants. Nothing else re-describes a tool.

Tool	Backing (reused, unchanged)	Result bound
latest_ai_news	getPublicDigest + localizePublicDigest — the same two functions servePublicApi uses	boundPublicDigest + PUBLIC_RESPONSE_MAX_BYTES (50,000)
search_news	getFeed (its 1–14 day clamp, its before validation)	projected + trimmed to 50,000
get_story	getStoryCandidates + renderStoryMarkdown (the /api/story/{id}.md pair)	the Markdown endpoint's own ceiling
get_ai_digest	getPublicDigest + displayTldrBullets (the homepage's own helper)	same digest bound

No new D1 query, no new table, no migration, no parallel data path. The
search_news and get_ai_digest projections live in
src/lib/public-read-results.ts, a pure module both transports import, so a
result cannot be shaped one way over MCP and another in the page.

A third leaf, src/lib/public-bounds.ts, holds PUBLIC_RESPONSE_MAX_BYTES
(and the story/bullet caps) so the browser can share the exact numbers without
importing the D1 query layer. public-queries.ts re-exports them, so every
existing import path still works and there is still exactly one definition.

The auth boundary

no Authorization header  -> anonymous  -> PUBLIC_MCP_TOOLS only, rate limited
Authorization: Bearer X  -> checkAuth  -> 401/500 exactly as before, or
                                         PUBLIC + ADMIN_MCP_TOOLS, no rate limit

Anonymous is defined as "no bearer token" — the exact predicate checkAuth
itself uses to read one (getBearerToken). checkAuth is not called at all on
an anonymous request, so a Clerk-only configuration (no NEWS_ADMIN_TOKEN) no
longer 500s every public reader. A request presenting any bearer token still
receives checkAuth's plain HTTP 401/500 Response — not a JSON-RPC error
object — byte for byte, because a client that tried to authenticate and failed
must not silently degrade into a read-only session. checkAuth's semantics are
untouched.

tools/list concatenates two non-overlapping registries resolved once from the
auth state, so an anonymous caller can never be handed a write tool.

No operator-inventory leak. An anonymous tools/call for anything outside
the public registry returns isError: true with one refusal message that is
byte-identical for push_items and for a name that does not exist — because
comparing against the admin registry is the enumeration oracle. No names, no
counts, and no handler runs (the test asserts items and admin_audit row
counts are unchanged).

The rate limit — numbers

apps/web/worker/mcp/rate-limit.ts reuses worker/rate-limit.ts: the same
checkRateLimit + hashIp + subscribe_attempts(ip_hash, created_at) mechanism
the subscribe handler and the admin failed-auth limiter already use, under an
mcp-read: key namespace so the rows can never be confused with the subscribe
(hash(ip)) or admin-fail: rows. No second limiter was written.

Limit	60 anonymous read calls
Window	60 s, per IP (CF-Connecting-IP, sha256-hashed)
Charged methods	tools/call, resources/read — only after the call is known to reach D1
Free	initialize, tools/list, resources/list, resources/templates/list, and any rejected argument (validation runs first, so a bad argument costs no quota and no query)
Not charged	admin-authenticated calls — the operator path keeps its own checkAuth gate
Over the limit	HTTP 429 + JSON-RPC -32000 + Retry-After: 60 (not a 5xx)
Served reads	X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Window-Sec
Limiter failure	fails closed — a limiter that silently fails open is not a limiter. (The admin failed-auth limiter's fail-open posture is a different trade-off: its fallback lets a credentialed operator through; this one would let anyone through. The table is created by migration 0015 and ensureMailSchema, so it exists everywhere.)

The trust boundary

Every tool declares readOnlyHint: true and untrustedContentHint: true,
plus destructiveHint: false / idempotentHint: true / openWorldHint: true
(declaring the negatives matters — a missing hint is "unknown" to a cautious
host). Every description carries the same sentence, because llms.txt prose is
not machine-actionable. An ambiguous id prefix is an error, never a "closest"
story, mirroring /api/story/{id}: >1 candidate → reject, and a longer
prefix whose 8-character canonical target is not unique → reject.

Making the discovery documents true

• AGENT_DISCOVERY_VERSION 0.1.6 → 0.1.7 (versioned into openapi.json,
  agent-card.json, server-card.json).
• mcpServerCard() now lists the real tools with schemas, annotations, the
  resources, the per-IP rate limit, and which half needs a bearer token.
  capabilities.prompts is REMOVED — it was advertised with no
  implementation behind it. Claiming it would be shipping a lie; the issue
  allowed "implement or remove", and removal is the honest, smaller change.
  capabilities.resources is now real (resources/list, resources/read).
• a2aAgentCard()'s public-digest and story-markdown skills are reachable
  over MCP; mcp-tools states that operator tools need admin auth.
• CONSUME_SKILL_MD splits the ambiguous MCP (read + admin) line into
  - MCP read tools (NO auth): … and
  - MCP operator tools (REQUIRES admin Authorization: Bearer …): ….
• openApiDocument() gains x-mcp (protocol version, tool set + annotations,
  operatorToolsRequireAuthorization, the rate-limit numbers and its 429
  shape, resources) plus real 401/429 responses, and security: [] (the read
  half genuinely needs no credential).
• The agent-skills/index.json digest is derived at request time
  (sha256Digest(CONSUME_SKILL_MD)), so it is recomputed by construction; a
  test asserts digest === sha256(CONSUME_SKILL_MD).
• /mcp, llms.txt, and /changelog updated.
• /api/mcp is still noindex, nofollow — route-indexability.ts:323 is
  untouched and now has a regression test (three cases, including the real
  response headers through withRouteIndexabilityHeaders).

Tests

New, 150 tests across 5 files (all green):

worker/__tests__/mcp-public.test.ts (49) — a D1 fake that counts queries,
so "a rejected argument never reaches D1" is a measurable assertion:
initialize / notifications/initialized / tools/list / tools/call /
resources/list / resources/read; notifications/initialized is 202 and no body; tools/list returns exactly the four read tools, with no write tool;
tools/list ships the readOnly + untrustedContent annotations; an unknown method is a JSON-RPC -32601; a malformed body is a JSON-RPC -32700;
still returns checkAuth's plain 401 for a wrong bearer token;
still returns checkAuth's plain 500 when the admin API is unconfigured;
gives an admin the public tools AND the operator tools;
does not rate limit an admin read call; one happy-path test per public tool;
14 hostile-argument tests that each assert isError and db.queryCount === 0
(out-of-enum lang, out-of-range and non-integer days, malformed and
impossible before, unknown category, oversized q, control characters in
q, unknown argument, and 5 bad id shapes);
rejects an ambiguous id prefix instead of picking a closest story;
rejects a longer prefix that cannot canonicalize to one story;
falls back to story titles rather than inventing prose for a thin digest;
redacts an internal failure instead of returning it;
answers every non-public name with the identical refusal;
does not run any handler for an operator name;
the execution adapter references no write, ingest, or admin handler (source
scan); the shared contract module references no mutation path either;
an anonymous read only issues reads plus the documented fixed side effects
(records every statement and allowlists the two non-SELECT ones);
5 resource tests; and 5 rate-limit tests including
serves a bounded number of reads per window, then a Retry-After error,
does not consume the window for a free method or a rejected argument, and
fails closed when the counter itself is unavailable.

worker/__tests__/mcp-public.integration.test.ts (11) — the same surface
against real SQLite with all 27 real migrations applied in order. A fake
cannot catch the class of bug that actually breaks this in production: SQL that
does not run. This proves getPublicDigest's four-way column fallback, the
getFeed five-statement batch, the substr(i.id, 1, ?) = ? … LIMIT 2 prefix
query the ambiguity rejection depends on, the real Markdown render, the real
SELECT COUNT(*) over a real subscribe_attempts table (asserting the stored
key is mcp-read:sha256(ip), never the raw IP), and that an anonymous
push_items writes nothing while an admin's still works.

src/lib/public-read-tools.test.ts (24) — the contract itself: names,
annotations-by-reference, the trust notice in every description,
additionalProperties: false on every schema, the story id pattern, the day
range and the 10-value category enum imported from its one definition,
prototype-pollution handling, and every validation branch.

src/lib/agent-discovery.test.ts (+8) — the version bump, the server card's
tools/resources/rate limit, no longer claims a prompts capability the server never implemented, openapi's x-mcp, the agent card's now-true skills, the
SKILL.md line split, and the digest.

src/lib/route-indexability.test.ts (+3) — /api/mcp stays private,
noindex, nofollow.

worker/__tests__/admin.test.ts — one assertion updated (tools/list returns the read tools plus the 6 admin tools for an admin, now asserting all six
operator names are present, each read tool appears once, and there are no
duplicates). Every other admin assertion, including
returns checkAuth's plain 401 Response on auth failure (not a JSON-RPC error),
is unchanged and still green.

Verification (real output)

pnpm --filter @aidr/web check-types

$ tsc --noEmit

(exit 0)

pnpm --filter @aidr/web run build

✓ built in 2.53s
Clerk proxy config assertion passed (https://aidr.today/__clerk)

(exit 0)

pnpm --filter @aidr/web test — the five relevant files:

 Test Files  5 passed (5)
      Tests  150 passed (150)

Full suite — 169 files / 1792 tests, all passing, run as six bounded groups
because this host OOM-kills a single 169-file vitest process intermittently
(see "Unmet / caveats"):

group 00 EXIT=0   Test Files  23 passed (23)    Tests  162 passed (162)
group 01 EXIT=0   Test Files  34 passed (34)    Tests  225 passed (225)
group 02 EXIT=0   Test Files  33 passed (33)    Tests  359 passed (359)
group 03 EXIT=0   Test Files  29 passed (29)    Tests  403 passed (403)
group 04 EXIT=0   Test Files  25 passed (25)    Tests  324 passed (324)
group 05 EXIT=0   Test Files  25 passed (25)    Tests  319 passed (319)

pnpm exec biome lint <21 changed paths>

Checked 21 files in 231ms. No fixes applied.

(exit 0)

Unmet / caveats — please read

1. The live curl checks in the issue were not run against
   https://aidr.today, because this PR does not deploy. The branch is not
   merged and main still serves the old code. The real-SQLite integration test
   plus the 49 fake-D1 tests are the substitute, and they cover every acceptance
   criterion that the curls were meant to demonstrate (tools/list split, admin
   split, resources, the identical operator refusal, all four tools valid +
   out-of-range + hostile, the 60/60s limit and its Retry-After, the
   noindex, nofollow header, and the latest_ai_news ≡ /api/public body).
   Please re-run the issue's curl block after deploy. The
   get_latest_news / /api/public byte-diff is asserted in
   mcp-public.integration.test.ts (same two functions) and, in
   mcp-public.test.ts, as the ≤ 50,000-byte cap.
2. No Lighthouse run. That audit measures the deployed site and the deploy
   is not part of this PR.
3. Tool names follow this task's contract, not the issue's tables. The
   issue lists get_latest_news / search_news / get_story /
   get_digest_tldr; P1 Agentic browsing: fix llms.txt (no links), register read-only WebMCP tools, and ship ai-catalog.json #226's table lists latest_ai_news / search_ai_news /
   get_ai_story / get_ai_digest. I used the four names the task specified —
   latest_ai_news, search_news, get_story, get_ai_digest — because the
   whole point is that MCP and WebMCP show an agent one surface. If the
   maintainer prefers the issue's names, they are four constants in
   public-read-tools.ts and every document is derived from them.
4. days is rejected, not clamped. The issue's §1 table says "reuse the
   existing days 1–14 clamp" but its acceptance criteria say "days outside
   1–14 … rejected cleanly". I followed the acceptance criteria: the tool
   contract rejects, so an agent that asks for 30 days learns why it got 14
   rather than silently getting 14. The HTTP /api/feed surface still clamps,
   unchanged.
5. search_news reaches two pre-existing non-SELECT statements, both
   already reachable anonymously through GET /api/feed before this change and
   both enumerated in the test: getFeed's best-effort read-time
   INSERT INTO tldr_snapshots rebuild, and ensureTopicLearningSchema's fixed
   literal of CREATE … IF NOT EXISTS plus a fixed UPDATE sources SET config
   seed. Neither is caller-influenced and neither writes feed, story, or user
   data. The test asserts no items or translations row is ever written.
6. serverInfo.name changed from "duyet-news" to "aidr" to match what
   the server card has always advertised. The claude mcp add alias on /mcp
   is a client-side label and is unchanged.
7. Test-name note for the reviewer: the limiter counts a request only when
   the call is going to touch D1, so does not consume the window for a free method or a rejected argument is a deliberate design assertion, not an
   oversight.

Reviewer's re-verification steps

pnpm install
pnpm --filter @aidr/web check-types
pnpm --filter @aidr/web run build
pnpm --filter @aidr/web test          # 169 files / 1792 tests

# the surface, in isolation
pnpm --filter @aidr/web exec vitest run \
  worker/__tests__/mcp-public.test.ts \
  worker/__tests__/mcp-public.integration.test.ts \
  src/lib/public-read-tools.test.ts \
  src/lib/agent-discovery.test.ts \
  src/lib/route-indexability.test.ts

pnpm exec biome lint \
  apps/web/src/lib/public-bounds.ts apps/web/src/lib/public-read-tools.ts \
  apps/web/src/lib/public-read-results.ts apps/web/src/lib/public-read-tools.test.ts \
  apps/web/src/lib/public-queries.ts apps/web/src/lib/agent-discovery.ts \
  apps/web/src/lib/agent-discovery.test.ts apps/web/src/lib/story-markdown.ts \
  apps/web/src/lib/llms-txt.ts apps/web/src/lib/route-indexability.test.ts \
  apps/web/src/components/mcp/lib.ts apps/web/src/components/mcp/McpSections.tsx \
  apps/web/src/routes/mcp.tsx apps/web/src/routes/changelog.tsx \
  apps/web/worker/mcp/ apps/web/worker/admin/mcp.ts \
  apps/web/worker/__tests__/mcp-public.test.ts apps/web/worker/__tests__/admin.test.ts

After merge + deploy, the issue's own block:

curl -sS -X POST https://aidr.today/api/mcp -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | jq '.result.tools[].name'
# -> latest_ai_news, search_news, get_story, get_ai_digest   (read only, anonymous)

curl -sI https://aidr.today/api/mcp | grep -i x-robots-tag
# -> x-robots-tag: noindex, nofollow

# hammer past 60 in a minute from one IP, expect a 429 + Retry-After, not a 5xx
for i in $(seq 1 70); do
  curl -s -o /dev/null -w '%{http_code}\n' -X POST https://aidr.today/api/mcp \
    -H 'content-type: application/json' -H "cf-connecting-ip: 1.2.3.4" \
    -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_ai_digest","arguments":{}}}'
done | sort | uniq -c

Check the published documents are self-consistent:

curl -s https://aidr.today/.well-known/mcp/server-card.json | jq '{version: .serverInfo.version, tools: [.tools[].name], prompts: (.capabilities.prompts // "removed"), rate: .rateLimit.anonymousRead, resources: [.resources[].uri]}'
curl -s https://aidr.today/openapi.json | jq '.info.version, .paths["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/api/mcp"].post["x-mcp"]'
curl -s 'https://aidr.today/.well-known/agent-skills/consume-aidr/SKILL.md' | grep -E '^- MCP (read|operator) tools'
curl -s https://aidr.today/.well-known/agent-skills/index.json | jq -r '.skills[0].digest'
curl -s 'https://aidr.today/.well-known/agent-skills/consume-aidr/SKILL.md' | sha256sum   # digest == this

Constraints honoured

No ranking, ingest, prompt, or DB schema change. No migration. No new table. No
new dependency. No second rate limiter, no second D1 query path, no second SSRF
boundary. No OAuth/device-code onboarding; checkAuth and the Clerk proxy are
untouched. No SSE, streaming, sessions, or notification push — the card still
honestly declares streaming: false. src/lib/sitemap.ts and
apps/web/public/robots.txt are untouched.

[sourcery-ai]

Sorry @duyet, your pull request is larger than the review limit of 150,000 diff characters

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: eaa1bc94-1c6c-471d-94e2-72e048929016

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}