fix: stop encoded probes from poisoning the robots.txt cache - #15
Merged
Merged
Conversation
A request for /robots%2Etxt missed the static robots route, fell through to the docs catch-all, and had its slug decoded before the page was cached. The resulting 404 render was written under the /robots.txt key, so every later robots.txt request failed with "app-route received invalid cache entry APP_PAGE" and returned 500 to Google. Two guards close this. The catch-all now sets dynamicParams = false, so unknown slugs 404 without rendering or writing anything to disk, which also ends the unbounded growth of 404 pages in the server cache. The proxy answers any percent-encoded path with a 404 before routing, because on its own the segment config still left encoded aliases of route handlers returning 500. A new smoke:standalone script boots the standalone server, replays the probe cold and warm, and checks robots.txt, sitemap, docs pages, Markdown negotiation and the cache directory over HTTP and on disk.
Verify built preview mode once and never made a request, so a broken robots route in the public build that production ships could pass CI behind a healthy build step. The job now builds public mode and runs the standalone smoke check against it, then does the same for preview mode. That checks both what crawlers may do and what they may not on every pull request. The timeout grows to 20 minutes to cover the second build.
The Fly health check only probes /, so the deploy stayed green while robots.txt returned 500 and Google stopped crawling the site. A new check-live-robots.sh fetches /robots.txt as Googlebot, retrying while a stopped machine starts, and fails unless it gets a 200 text/plain response that allows Googlebot and advertises the canonical sitemap. The deploy workflow runs it against docs.prose.md after flyctl deploy, so a broken robots route turns the deploy run red instead of going unnoticed. Run against production before this fix, it fails with 500 on every attempt.
josemontesdeoca
added a commit
that referenced
this pull request
Sep 18, 2026
Both handlers looked a page up with the last slug segment dropped, so any file name under a real page (/og/setup/anything.png, /llms.mdx/setup/other.md) rendered that page and saved a copy under the requested name. Next never deletes those files, so scanner traffic could grow the disk without limit. Every image and markdown file is generated at build time, so both routes now export dynamicParams = false, the guard the page catch-all got in #15. Anything the build did not generate, including bare /llms.mdx, now 404s before the handler runs and nothing is written. The standalone smoke run now fails if a request adds a file to the cache directory, or if any dynamic route in the prerender manifest can render params on demand, so a new route cannot reopen this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the change
docs.prose.md/robots.txtreturns 500 because a request for/robots%2Etxtoverwrote its cache entry, and a 5xx on robots.txt stops Google from crawling the site; this blocks that write and checks robots.txt over HTTP in CI and after every deploy.Special things to note
dynamicParams = falsealone,/robots.txtstayed healthy, but the encoded probe itself still returned 500.Change outline
The root catch-all page decodes its slug before using it as a cache key, and route handlers share that cache: