feat(web): valid llms.txt links, read-only WebMCP tools, ai-catalog.json

[duyet]

Closes #226. Part of the umbrella #232.

Base: rebased onto master after #237 (this PR's shared read-tool
contract) and #236 (RSS + news sitemap) both merged. The branch is one
commit containing only this PR's 14 files — no worker/ file is touched, so
the two PRs are independently reviewable.

The problem this closes

Lighthouse "Agentic Browsing" measured 2/3 on https://aidr.today/:

llms.txt does not follow recommendations — Error: File does not appear
to contain any links.

plus three unscored-but-empty agent surfaces:

WebMCP form coverage — Consider adding WebMCP annotations to the forms
listed below.
WebMCP tools registered — (no data)
WebMCP schemas are valid — (no data)
ai-catalog.json schema is valid — (no data)

Cloudflare's WebMCP bridge was being injected and loaded — Lighthouse's
network tree shows /.webmcp/bridge.js(aidr.today) - 1,534 ms, 14.42 KiB on
the critical path — but the app registered zero tools and zero form
annotations, so the 14.42 KiB was pure cost.

The llms.txt body itself was already good content. It was just emitted as one
prose blob with bare GET https://… text in - list items instead of as
markdown links.

1. llms.txt is now spec-conformant markdown

Every endpoint is a real [label](url) link. The missing machine surfaces are
now linked too: /openapi.json, /.well-known/agent-card.json,
/.well-known/mcp/server-card.json, /.well-known/ai-catalog.json, the
.well-known/api-catalog linkset, the agent skill path, and /auth.md.
Exactly one H1: # AI News (aidr.today).

No information was lost. The locale/cache contract, the story-text trust
boundary, the ranking formula, the submit flow, and the suggest flow are all
still in the file, and a test asserts each of them by content. An llms.txt
that "passes" by dropping the ranking formula is not an improvement. Two
assertions also pin the absence of the old shape:

expect(llmsTxt()).not.toMatch(/^- .*GET \$\{SITE_URL\}/m);
expect(llmsTxt()).not.toMatch(/: GET https:\/\/aidr\.today\//);

New tests in llms-txt.test.ts: has exactly one H1, and it names the resolved brand, contains at least one real markdown link, links the surfaces the audit expected and that did not exist before, every listed absolute URL resolves to a reachable aidr path, and keeps the ranking formula, the trust boundary, and the submit flow.

"Every absolute URL resolves" is asserted against the shared site constants,
not by a network call — the test walks every markdown link, and each
aidr.today path must be in a reachable set; a non-aidr host must be an
intentional external project URL (github.com, duyet.net) rather than a
typo'd internal path. Story examples must be a real 8-character hex id, not a
template.

2. WebMCP tools — the SAME four capabilities, from the same module

src/lib/webmcp.ts registers the four read capabilities through
document.modelContext.registerTool({name, description, inputSchema, annotations, execute}). It does not re-describe them:

export const WEBMCP_TOOLS = PUBLIC_READ_TOOLS;   // identity, not a copy

and each registration's inputSchema is the contract's own object. The
browser has no D1 binding, so execute reads the same public endpoints over
same-origin fetch and applies the same validators and the same
projection + bound as the MCP transport. Validation runs first, so a hostile
argument costs one bounded pass and no request — the browser-side twin of
"rejected arguments touch no D1".

Tool	WebMCP backing	Annotations
latest_ai_news	GET /api/public?lang=	readOnlyHint, untrustedContentHint, destructiveHint: false, idempotentHint: true, openWorldHint: true
search_news	GET /api/feed?q=&days=&category=&before=	same
get_story	GET /api/story/{id}.md?lang=	same
get_ai_digest	TL;DR bullets from /api/public	same

untrustedContentHint: true is load-bearing and every description repeats the
boundary, because llms.txt prose is not machine-actionable.

Guards

• Client-side only. A <WebMcpTools /> component calls
  registerWebmcpTools() in an effect and renders nothing. Registration is
  never at module scope and never during render, so nothing WebMCP-related is
  in the SSR HTML, the hydration payload, or the LCP path.
• No-op, not a throw. document.modelContext undefined → return 0. Also
  covered for modelContext present but lacking registerTool, and for no
  document at all (the SSR path). One tool being rejected by the bridge does
  not take the other three down.
• The bridge is never ours. /.webmcp/bridge.js is Cloudflare-injected.
  webmcp-ssr.test.ts scans the source tree for an import, a <script src>,
  and a direct reference. ai-catalog.json may name the URL so an agent
  knows where it comes from; it may never load it.
• Queries are built with URLSearchParams, never string interpolation, so
  a hostile q stays a value: the test asserts
  ?lang=en&q=a%26b%3Dc%23frag&days=7&category=Models and that the search
  params are exactly ["lang","q","days","category"] — the # did not create a
  fragment.
• Results are byte-bounded by PUBLIC_RESPONSE_MAX_BYTES — the identical
  50,000-byte cap GET /api/public publishes, imported from the same leaf
  module. An over-bound response is refused, never truncated.
• Ambiguity is an error. The endpoint's 409 is surfaced as
  "ambiguous story id … use a longer prefix", never a closest match.

3. /.well-known/ai-catalog.json

Served from handleAgentDiscovery, derived from the same module:

tools: WEBMCP_TOOLS.map((tool) => ({ name, description, parameters: tool.inputSchema, annotations, restPath, auth: "none" }))

parameters is named for the catalog convention; the object is byte-identical
to the MCP inputSchema. The document also carries the MCP cross-reference,
the Cloudflare-injected bridge URL, the document-level trust.untrustedContent
statement, and the audited forms array.

4. WebMCP form annotations — the audit, and the decision

I audited the real forms. Two are agent-callable:

Form	Callable	Annotations	Why
/submit	yes	readOnlyHint: false, consequentialHint: true, untrustedContentHint: true	publishing to a feed thousands of people read is a real consequence, and url/title/note are third-party text
email subscribe	yes	readOnlyHint: false, consequentialHint: true	a subscription commits the user to recurring mail
/sign-in, /sign-up	no	—	a tool whose schema is { password: string } is a credential-harvesting primitive advertised to every agent on the page, and no host model can distinguish "fill the sign-in form" from "exfiltrate the user's password". Clerk owns those forms and they are not in this codebase, so there is nothing to annotate either.
header SearchBox	no	—	it is a client-side filter over an already-loaded feed; the registered search_news tool covers it with real semantics instead of a form-filling approximation
story suggest	no	—	requires a signed-in Clerk session, so it shares the sign-in credential boundary

The three refusals are stated in ai-catalog.json as agentCallable: false
WITH a notCallableReason, so a reader sees the decision rather than
inferring an oversight. A test asserts no form annotation ever declares a
password / token / secret / otp / code property, that non-callable
forms carry an empty schema plus a reason, and that the two callable forms'
DOM attributes are inert data-* only.

A real bug the tests caught

boundSearchResult re-serialized the whole payload after every single item
removal. A hostile 4,000-item feed therefore cost 4,000 serializations of a
~16 MB string — the bounding step had become the denial of service it exists
to prevent, and the MCP transport shared the code. It is now a binary search
over a newest-first prefix: O(log n) measurements, the same answer (the
largest newest-first prefix that fits), and a regression test with a time
bound.

Tests

New and changed, all green:

src/lib/webmcp.test.ts (27, new) — module identity (3, the load-bearing
ones: the WebMCP registration is the contract module, not a copy asserts
WEBMCP_TOOLS[i] === PUBLIC_READ_TOOLS[i]; the registered registrations carry the contract's own schema object asserts registration.inputSchema === source.inputSchema; the catalog is derived from the same module asserts
entry.parameters === source.inputSchema — by identity, not by comparing
rendered JSON, because a re-implementation that happened to produce the same
bytes would otherwise pass); client-side-only registration (6, including the
three no-op paths and one-bridge-rejection resilience); tool execution validates before it fetches (4, each asserting fetch was not called);
tool execution hits the public endpoints (8); ai-catalog.json (3);
form annotations (3).

src/lib/webmcp-ssr.test.ts (6, new) — the root route registers inside an effect, not during render, no SSR-reachable module calls registerTool outside an effect (a directory scan), never imports, vendors, or references /.webmcp/bridge.js (a recursive source scan), ai-catalog names the bridge as Cloudflare-injected, not as ours, and two registration-shape tests.

src/lib/llms-txt.test.ts (7, up from 2) — the original content assertions
are unchanged and still pass, plus the five new conformance tests.

src/lib/agent-discovery.test.ts (+1) — serves ai-catalog.json with the registered read tools (#226): status, content type, the four tool names, every
annotation pair, additionalProperties: false, and the form decisions.

src/lib/public-read-results.ts — the boundSearchResult fix described
above (shared by both transports).

Verification (real output)

pnpm --filter @aidr/web check-types

$ tsc --noEmit

(exit 0)

pnpm --filter @aidr/web run build

✓ built in 1.97s
Clerk proxy config assertion passed (https://aidr.today/__clerk)

(exit 0)

pnpm --filter @aidr/web exec vitest run src/lib/webmcp.test.ts src/lib/webmcp-ssr.test.ts src/lib/llms-txt.test.ts src/lib/agent-discovery.test.ts --maxWorkers=1

 Test Files  4 passed (4)
      Tests  62 passed (62)

Every test name in the three new/changed files, verbatim:

 ✓ src/lib/webmcp.test.ts > module identity: one definition, four consumers > the WebMCP registration is the contract module, not a copy
 ✓ src/lib/webmcp.test.ts > module identity: one definition, four consumers > the registered registrations carry the contract's own schema object
 ✓ src/lib/webmcp.test.ts > module identity: one definition, four consumers > the catalog is derived from the same module
 ✓ src/lib/webmcp.test.ts > client-side-only registration > is a no-op, not a throw, when modelContext is undefined
 ✓ src/lib/webmcp.test.ts > client-side-only registration > is a no-op when modelContext exists but has no registerTool
 ✓ src/lib/webmcp.test.ts > client-side-only registration > is a no-op with no document at all (the SSR path)
 ✓ src/lib/webmcp.test.ts > client-side-only registration > registers all four tools with the read-only annotations
 ✓ src/lib/webmcp.test.ts > client-side-only registration > survives one tool being rejected by the bridge
 ✓ src/lib/webmcp.test.ts > client-side-only registration > never references the Cloudflare bridge script
 ✓ src/lib/webmcp.test.ts > tool execution validates before it fetches > rejects an out-of-enum lang with no network call
 ✓ src/lib/webmcp.test.ts > tool execution validates before it fetches > rejects out-of-range days, an unknown category, and a bad before
 ✓ src/lib/webmcp.test.ts > tool execution validates before it fetches > rejects a non-hex story id with no network call
 ✓ src/lib/webmcp.test.ts > tool execution validates before it fetches > rejects non-object arguments
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > latest_ai_news reads /api/public with the explicit lang
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > get_ai_digest projects the same bullets the MCP tool returns
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > search_news builds the query with URLSearchParams, never interpolation
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > get_story reads the bounded Markdown endpoint
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > surfaces the endpoint's 409 ambiguity as an error, never a guess
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > refuses a payload that exceeds the public read bound
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > bounds a hostile search payload at the shared layer, newest first
 ✓ src/lib/webmcp.test.ts > tool execution hits the public endpoints > does not leak internals when the endpoint errors
 ✓ src/lib/webmcp.test.ts > ai-catalog.json > describes the registered tools with schemas and annotations
 ✓ src/lib/webmcp.test.ts > ai-catalog.json > is a valid JSON-serializable document
 ✓ src/lib/webmcp.test.ts > ai-catalog.json > exposes the submit and subscribe forms and refuses the credential ones
 ✓ src/lib/webmcp.test.ts > form annotations > no form annotation ever declares a write-capable credential schema
 ✓ src/lib/webmcp.test.ts > form annotations > annotates submit and subscribe, and states why sign-in is not
 ✓ src/lib/webmcp.test.ts > form annotations > emits inert data attributes, never behaviour
 ✓ src/lib/webmcp-ssr.test.ts > WebMCP is client-side only > the root route registers inside an effect, not during render
 ✓ src/lib/webmcp-ssr.test.ts > WebMCP is client-side only > no SSR-reachable module calls registerTool outside an effect
 ✓ src/lib/webmcp-ssr.test.ts > WebMCP is client-side only > never imports, vendors, or references /.webmcp/bridge.js
 ✓ src/lib/webmcp-ssr.test.ts > WebMCP is client-side only > ai-catalog names the bridge as Cloudflare-injected, not as ours
 ✓ src/lib/webmcp-ssr.test.ts > registration shape > registers only the four read tools, none of them a write
 ✓ src/lib/webmcp-ssr.test.ts > registration shape > every execute is a function, so a host can call any of them
 ✓ src/lib/llms-txt.test.ts > llmsTxt is spec-conformant markdown (#226) > has exactly one H1, and it names the resolved brand
 ✓ src/lib/llms-txt.test.ts > llmsTxt is spec-conformant markdown (#226) > contains at least one real markdown link
 ✓ src/lib/llms-txt.test.ts > llmsTxt is spec-conformant markdown (#226) > links the surfaces the audit expected and that did not exist before
 ✓ src/lib/llms-txt.test.ts > llmsTxt is spec-conformant markdown (#226) > every listed absolute URL resolves to a reachable aidr path
 ✓ src/lib/llms-txt.test.ts > llmsTxt is spec-conformant markdown (#226) > keeps the ranking formula, the trust boundary, and the submit flow
 ✓ src/lib/agent-discovery.test.ts > handleAgentDiscovery > serves ai-catalog.json with the registered read tools (#226)

Full suite on the rebased branch — 178 files / 1995 tests, all passing, run
as bounded groups
because this host OOM-kills a single 171-file vitest process intermittently
(this reproduces on an unmodified checkout too — it is the machine, not the
change; see "Unmet / caveats"):

group 00    EXIT=0   Test Files  24 passed (24)    Tests  184 passed (184)
group 01    EXIT=0   Test Files  36 passed (36)    Tests  248 passed (248)
group 02    EXIT=0   Test Files  35 passed (35)    Tests  395 passed (395)
group 03    EXIT=0   Test Files  30 passed (30)    Tests  453 passed (453)
group 04/00 EXIT=0   Test Files  15 passed (15)    Tests  257 passed (257)
group 04/01 EXIT=0   Test Files  12 passed (12)    Tests   87 passed  (87)
group 05/00 EXIT=0   Test Files  14 passed (14)    Tests  233 passed (233)
group 05/01 EXIT=0   Test Files  12 passed (12)    Tests  138 passed (138)

(178 files, 1995 tests. Two groups needed splitting further — the host is
memory-starved; see caveat 2.)

pnpm exec biome lint <14 changed paths>

Checked 14 files in 38ms. No fixes applied.

(exit 0)

Unmet / caveats — please read

1. The live curl checks and the Lighthouse before/after were not run,
   because this PR does not deploy. main still serves the old llms.txt, so
   a curl https://aidr.today/llms.txt | grep -c '^# ' would report the
   pre-change value. The same properties are asserted in
   llms-txt.test.ts (exactly one H1, ≥ 1 markdown link, every URL reachable),
   and I did not want to paste a Lighthouse screenshot of the old site as
   "after". Please re-run the issue's verification block after merge +
   deploy, including the browser-console check
   await document.modelContext.getTools().
2. I could not build a Miniflare/workerd smoke for this PR. The installed
   miniflare@5.20260918.0-alpha rejects d1Databases and bindings in every
   option position (I probed workers[].config, workers[].dev,
   workers[], and the flat form; each returns unrecognized_keys), so the
   two test:workerd:* scripts in this repo can only test pure functions. I
   therefore wrote the SSR/bridge guards as source-scan tests instead
   (webmcp-ssr.test.ts), which is a weaker guarantee than a runtime one. If
   you want the runtime version, it needs a Miniflare that accepts D1
   bindings.
3. The RSS feed and news sitemap ARE in llms.txt, as links. When I
   started, issue P0 Discovery: ship public RSS at /feed.xml, a Google News sitemap, a sharded sitemap index, and feed autodiscovery #225 was still open, so linking them would have advertised
   two 404s. P0 Discovery: ship public RSS at /feed.xml, a Google News sitemap, a sharded sitemap index, and feed autodiscovery #225 merged as feat(web): public RSS at /feed.xml, Google News sitemap, sharded sitemap index, feed autodiscovery #236 while I was working; I rebased and kept
   /feed.xml, the /rss.xml alias, and /news.xml in the ## Consume list
   as real markdown links, and added them to the test's reachable-path
   allowlist. The rebased llms.txt has exactly one H1, no duplicate
   "This file" entry, and no bare-URL list items — the merge was the only
   genuine conflict, and the ## Consume section needed hand-merging because
   feat(web): public RSS at /feed.xml, Google News sitemap, sharded sitemap index, feed autodiscovery #236's lines were bare-URL text, which is the exact shape this PR removes.
4. The H1 changed from # aidr.today to # AI News (aidr.today). The
   issue asked for "exactly one H1 that names the site and matches the
   resolved brand"; SITE_NAME is AI News, so the H1 now matches the
   brand constant rather than only the domain. grep -c '^# ' is 1 either way.
5. I did not touch src/lib/sitemap.ts or apps/web/public/robots.txt.
   Both are owned by the merged feat(web): public RSS at /feed.xml, Google News sitemap, sharded sitemap index, feed autodiscovery #236/fix(web): stop noindex on bare localized URLs, 308 legacy story redirects, valid robots.txt #233 work; my only interaction with the
   sitemap is the llms.txt sentence describing the index, plus the test
   allowlist.
6. story suggest is not annotated as a form. It is listed in
   ai-catalog.json as agentCallable: false with the reason, which is the
   honest answer given the Clerk session requirement, but the form element
   itself carries no data-webmcp-* attributes. If you would rather it be
   annotated-but-not-callable in the DOM too, that is a one-line change.

Reviewer's re-verification steps

pnpm install
pnpm --filter @aidr/web check-types
pnpm --filter @aidr/web run build
pnpm --filter @aidr/web test

# the surfaces this PR is about
pnpm --filter @aidr/web exec vitest run \
  src/lib/webmcp.test.ts src/lib/webmcp-ssr.test.ts \
  src/lib/llms-txt.test.ts src/lib/agent-discovery.test.ts

pnpm exec biome lint \
  apps/web/src/lib/webmcp.ts apps/web/src/lib/webmcp.test.ts \
  apps/web/src/lib/webmcp-ssr.test.ts apps/web/src/lib/ai-catalog.ts \
  apps/web/src/lib/llms-txt.ts apps/web/src/lib/llms-txt.test.ts \
  apps/web/src/lib/agent-discovery.ts apps/web/src/lib/agent-discovery.test.ts \
  apps/web/src/lib/public-read-results.ts \
  apps/web/src/components/WebMcpTools.tsx \
  apps/web/src/components/submit/SubmitForm.tsx \
  apps/web/src/components/EmailSubscribeForm.tsx \
  apps/web/src/routes/__root.tsx apps/web/src/routes/changelog.tsx

After merge + deploy:

curl -s https://aidr.today/llms.txt | grep -c '^# '            # must be 1
curl -s https://aidr.today/llms.txt | grep -cE '\[[^]]+\]\(https://'   # >= 1
curl -s https://aidr.today/.well-known/ai-catalog.json | jq '{version, tools: [.tools[].name], forms: [.forms[] | {id, agentCallable}]}'
curl -s https://aidr.today/0031a3a8?lang=en | grep -c modelContext    # must be 0

In a WebMCP-capable browser console:

await document.modelContext.getTools()
// -> the four tools, each with annotations.readOnlyHint === true
//    and annotations.untrustedContentHint === true

Constraints honoured

No new dependencies. No ranking, ingest, prompt, or DB schema change. No
migration. No second rate limiter, no second D1 query path, no second SSRF
boundary. /.webmcp/bridge.js is not vendored or bundled. No OAuth or Clerk
changes. src/lib/sitemap.ts and apps/web/public/robots.txt untouched.
Registration is client-side only and off the LCP path.

[sourcery-ai]

Sorry @duyet, your pull request is larger than the review limit of 150,000 diff characters

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: f581f7e1-f13c-41fe-9328-aee1bfb611f9

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}