Repository navigation
fix(http): retry transient public reads and type Cloudflare IP blocks - #17
Merged
Merged
Conversation
Some session machines get an egress IP that Substack's Cloudflare edge challenges (403 + cf-mitigated: challenge) on every request. Public GETs now retry network errors, 5xx, and challenges with exponential backoff, and a challenge that outlasts the retries surfaces as the retry-safe CLOUDFLARE_CHALLENGE error instead of a generic HTTP_ERROR 403.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The 3-hourly @theahura Notes → X syndication trigger intermittently dies at its first Substack call: some trigger machines get an egress IP that Substack's Cloudflare edge challenges on every request (
HTTP 403,cf-mitigated: challenge; a plaincurlgets the same). Transcript history shows ~6 blocked runs out of ~250 in the last month, each clearing by the next run. Today the CLI fails on the first 403 with a genericHTTP_ERROR, so agents can't tell an IP block from a real auth/endpoint failure.What
publicGetinsrc/core/http.ts: public GETs retry network errors, 5xx, and Cloudflare challenges with exponential backoff (NORI_SUBSTACK_HTTP_RETRIES, default 2;NORI_SUBSTACK_HTTP_RETRY_DELAY_MS, default 5000 → 5s, 10s).CLOUDFLARE_CHALLENGEerror (exit 8) withstatus,cfMitigated,attempts.PublicClientgo through the same path.RATE_LIMITEDenvelope, and authenticated/mutating requests are untouched.Not included
No browser fallback. The block is on the machine's IP, so a browser on the same machine gets the same challenge, and getting past it would mean solving it, which we don't do.
Testing
tests/public-retry.test.ts: challenge then success, persistent challenge →CLOUDFLARE_CHALLENGEafter 3 attempts, 5xx retry, plain 403 not retried.dist/recompiled.note export --user-id 9744387 --lookback-hours 0andpost list --publication-url https://12gramsofcarbon.compass against real Substack.🤖 Generated with Nori
Co-Authored-By: Nori contact@tilework.tech