feat(worker): more verified news sources, data-driven source registry, per-source + stale observability

[duyet]

Implements #230 (umbrella #232). Three parts: more verified sources, a declarative source registry that makes adding one a data operation, and per-source + stale observability so a bad source is visible instead of invisible.

Everything is behind one registry. worker/sources/catalog.ts is the only place a source is declared; the runtime seed, the migration, and the /api/system/sources + /data surfaces all consume it, and a test asserts all three agree byte for byte.

---

1. New sources — all verified live

pnpm --filter @aidr/web run verify:source-feeds runs the production parseRssItems and the production flood gate against every registry row, so a green result means the Worker can actually read the feed. 18/18 PASS. New-source rows, with real output:

Vietnamese (this is the one that exercises the dead source_lang path)

### vnexpress-tech — VnExpress Khoa học & Công nghệ [rss]
  url         https://vnexpress.net/rss/khoa-hoc-cong-nghe.rss
  status      200 application/xml; charset=utf-8 (62742 bytes)
  parsed      60 items, 60 usable (title + absolute URL + date)
  this run    6 after window+gate (6 in the 26h window)
  gate        maxItems=6
  newest      2026-09-27T11:21:06.000Z   oldest 2026-09-21T22:00:00Z
  first 3 the Worker would return this run:
    - 2026-09-27T11:21:06.000Z  'Bão bài đăng' về Muse của tỷ phú AI Alexandr Wang
      https://vnexpress.net/bao-bai-dang-ve-muse-cua-ty-phu-ai-alexandr-wang-5125399.html
    - 2026-09-27T06:00:00.000Z  Nhân viên Google từ chức vì lo AI 'phát triển quá nhanh'
      https://vnexpress.net/nhan-vien-google-tu-chuc-vi-lo-ai-phat-trien-qua-nhanh-5125305.html
    - 2026-09-27T01:00:00.000Z  Giá RAM laptop tăng sáu lần sau một năm
      https://vnexpress.net/gia-ram-laptop-tang-sau-lan-sau-mot-nam-5124978.html
  verdict     PASS

sourceLang: "vi" is declared metadata in the row config, never inferred from diacritics. Its pubDate is RFC-822 with an explicit +0700, which the test asserts lands on the correct instant (2026-09-27T08:00:00 +0700 → 01:00:00Z) — a 7-hour error here would land every VI item in the future and sort it above fresh stories.

AI newsrooms

### techcrunch-ai — TechCrunch AI [rss]
  status      200 application/rss+xml; charset=UTF-8 (18460 bytes)
  parsed      20 items, 20 usable   this run 3 after window+gate   gate maxItems=6
  newest      2026-09-27T16:30:00.000Z
    - 2026-09-27T16:30:00.000Z  Anthropic's Dario Amodei gets the SNL treatment
      https://techcrunch.com/2026/09/27/anthropics-dario-amodei-gets-the-snl-treatment/
    - 2026-09-27T01:30:00.000Z  Google tests buying from Walmart-owned Flipkart through Gemini and AI Mode in India
      https://techcrunch.com/2026/09/26/google-tests-buying-from-walmart-owned-flipkart-through-gemini-and-ai-mode-in-india/
    - 2026-09-26T21:02:06.000Z  Insurers claim AI is already increasing healthcare costs
      https://techcrunch.com/2026/09/26/insurers-claim-ai-is-already-increasing-healthcare-costs/
  verdict     PASS

### theverge-ai — The Verge AI [rss]
  status      200 application/xml; charset=UTF-8 (124231 bytes)
  parsed      10 items, 10 usable   this run 2 after window+gate   gate maxItems=6
  newest      2026-09-27T17:21:07.000Z
    - 2026-09-27T17:21:07.000Z  OpenAI agents tried to 'bruteforce' a UN website
      https://www.theverge.com/ai-artificial-intelligence/1001178/openai-agents-bruteforce-un-website
    - 2026-09-26T16:34:59.000Z  OpenAI pauses training of its 'most capable models'
      https://www.theverge.com/ai-artificial-intelligence/1001049/openai-training-pause
  verdict     PASS

### arstechnica-ai — Ars Technica AI [rss]
  status      200 application/rss+xml; charset=utf-8 (76745 bytes)
  parsed      20 items, 20 usable   gate maxItems=6
  note        feed is valid but nothing in the 26h ingest window
  newest      2026-09-25T21:36:20.000Z
    - 2026-09-25T21:36:20.000Z  Court rules Pentagon can blacklist Anthropic for refusing to enable Claude features
      https://arstechnica.com/tech-policy/2026/09/court-rules-trump-can-blacklist-anthropic-for-refusing-to-enable-claude-features/
    - 2026-09-25T21:10:51.000Z  Tesla workers balk at training Optimus humanoid robots as replacements
      https://arstechnica.com/ai/2026/09/tesla-workers-balk-at-training-optimus-humanoid-robots-as-replacements/
    - 2026-09-25T19:11:05.000Z  AI was supposed to hit new grads hard. So far, unemployment data says otherwise.
      https://arstechnica.com/ai/2026/09/ai-was-supposed-to-hit-new-grads-hard-so-far-unemployment-data-says-otherwise/
  verdict     PASS

### wired-ai — WIRED AI [rss]
  status      200 application/xml (10200 bytes)
  parsed      10 items, 10 usable   gate maxItems=6
  newest      2026-09-26T10:30:00.000Z
    - 2026-09-26T10:30:00.000Z  Meta's Muse Is Adults-Only. Why Does It Look Like a Kids' Toy?
      https://www.wired.com/story/meta-muse-is-adults-only-why-does-it-look-like-a-cute-kids-toy/
    - 2026-09-25T20:30:40.000Z  Thieves Stole 'Nvidia' Trailers. They Got 20 Tons of Sand
      https://www.wired.com/story/thieves-stole-nvidia-trailers-they-got-20-tons-of-sand/
    - 2026-09-25T16:58:20.000Z  Appeals Court Lets the Pentagon Designate Anthropic a Supply-Chain Risk
      https://www.wired.com/story/appeals-court-lets-the-pentagon-designate-anthropic-a-supply-chain-risk/
  verdict     PASS

Net: 16 → 20 registry rows (5 new). No new adapter type — all five are rss, which already handled Atom as well as RSS.

Evaluated and rejected, with the reason recorded in the code and the migration header

candidate	why not
arXiv cs.AI/cs.LG/cs.CL	robots-disallowed. export.arxiv.org/robots.txt is User-agent: * / Disallow: /; arxiv.org/robots.txt lists Disallow: /api. The sortable Atom API is therefore out under this repo's "no robots.txt disallow" rule. The one allowed surface (rss.arxiv.org, no robots.txt, already newest-first RSS so the gate applies unchanged) was serving an empty channel — it declares <skipDays>Saturday, Sunday</skipDays> and the next announcement was not due until Mon 04:00 UTC — so it could not clear the "≥1 usable item verified live" bar. See §2.
VentureBeat	HTTP 429 on bot fetches (already noted in migration 0022). Its FeedBurner mirror is stale — newest item Sep 3, 24 days old.
Engadget	HTTP 202 challenge instead of the feed.
ZDNet	the "AI topic" RSS URL redirects to /rss/news/, i.e. general tech news, not AI.
Tuổi Trẻ cong-nghe	200 and 50 genuinely-Vietnamese items, but <pubDate>9/27/2026 4:04:00 PM</pubDate> has no timezone, so every item would land ~7h in the future. VnExpress was chosen over it for exactly this.
cafebiz.vn / techrum.vn / zingnews.vn	404 / 404 / 403.
vinai / genk / newsroom.vn / baotrimai	no reachable feed.

2. arXiv: the flood gate ships, the source does not

This is the one acceptance criterion I did not meet, and the reason is a hard constraint, not effort.

The gate is fully built and tested against the arXiv case — applyFloodGate runs the AI-keyword pre-filter first and then a newest-first maxItems cap, both config-driven, so it is reusable by any firehose:

worker/__tests__/arxiv-flood-gate.test.ts  (12 tests, all pass)
  ✓ bounds a >500-entry feed to the documented cap
  ✓ keeps only the AI-keyword subset before capping
  ✓ is the same regex the HN adapter uses, not a second keyword list
  ✓ does nothing when the feed is smaller than the cap
  ✓ caps the whole added source set inside the existing score budget
  ✓ filters before capping, so the cap fills with relevant items
  ✓ ignores an unknown keyword profile rather than throwing
  ✓ ignores a nonsense cap rather than returning nothing
  ✓ throws a typed fetch_failed for a non-2xx
  ✓ throws a typed parse_failed for a 200 that is really an HTML page
  ✓ carries no URL or header in the error message
  ✓ reports a genuinely empty feed as empty, not as a failure

The 640-entry fixture asserts the capped, keyword-filtered result through rssAdapter.fetchItems — the real production path — so a regression that removed the gate, moved it after the cap, or made it oldest-first fails in CI. The "filters before capping" test builds the specific trap (two off-topic entries strictly newer than every relevant one) that a cap-first implementation would fall into.

To finish arXiv (after 04:00 UTC on a weekday) it is one row plus a regeneration; ARXIV_NOT_ADDED_REASON in worker/sources/catalog.ts has the copy-pasteable config and the measured weekend freeze its staleAfterRuns: 72 has to account for:

{
  id: "arxiv-research",
  name: "arXiv cs.AI / cs.LG / cs.CL",
  type: "rss",
  config: {
    feed: "https://rss.arxiv.org/rss/cs.AI+cs.LG+cs.CL",
    homepage: "https://arxiv.org/list/cs.AI/recent",
    keywordFilter: "ai",
    maxItems: 6,
  },
  enabled: true,
  staleAfterRuns: 72,
}

rss.arxiv.org accepts a combined category list — verified live, the channel comes back as cs.AI, cs.LG, cs.CL updates on arXiv.org — so that is one request per run and needs no per-host pacing.

3. Adding a source is a data operation, not a migration

seed.ts used to hand-copy every row out of migrations 0018/0020/0021/0022 and its own header called that a maintenance hazard. Now:

consumer	source of truth
worker/sources/seed.ts	buildSourceSeedSql() from the registry
migrations/0027_source_registry.sql	pnpm run gen:source-migration
/api/system/sources + /data Sources tab	the same SOURCE_REGISTRY rows

worker/__tests__/source-catalog.test.ts  (14 tests, all pass)
  ✓ the checked-in migration is exactly what the registry generates
  ✓ the migration and the runtime seed list identical rows
  ✓ the seed SQL is derived from the registry, not hand-copied
  ✓ the seed upserts identity and config but never re-enables a source
  ✓ the migration inserts only, leaving an operator's enabled flag alone
  ✓ keeps no robots-disallowed source, and records why arXiv is absent
  ✓ points no registry row at a robots-disallowed host
  ✓ only uses adapter types the registry can actually dispatch
  ✓ gives every rss row a feed URL and the flood-gate keys it needs
  ✓ marks exactly the Vietnamese sources with explicit sourceLang: vi
  ✓ documents a stale threshold for every row and defaults sensibly
  ✓ has unique ids and a name/type/config for every row
  ✓ is identical in the worker and the web read model   (skip-reason enum)
  ✓ covers exactly the reasons #230 asked for

Ownership rules, and they are load-bearing. The registry owns name/type/config for the ids it declares — the seed uses ON CONFLICT DO UPDATE on those three, so a corrected feed URL actually reaches an existing row. That is the entire point: INSERT OR IGNORE (what it replaced) would have left a fixed URL stranded on whatever the row already held, i.e. the drift this removes. enabled is operator-owned and never written by the seed, so switching a noisy source off survives every run. Rows the registry does not declare are entirely operator-owned.

No deploy, end to end. The existing upsert_source MCP tool already covers a new rss row and needs no change. Documented with copy-pasteable curl in apps/web/worker/README.md → "Add a source without a deploy", including the "poll /api/system until lastRun.id matches the POST id" step. Production already proves the path: the user submissions source in the live sources table was created that way.

4. Per-source observability + the stale detector

workflow_runs.stats.sourceHealth now carries, per source: fetched / scored / accepted / rejected / merged, a closed skip-reason enum, and a consecutive-empty-run streak.

• Skip reasons: fetch_failed, parse_failed, empty, all_rejected_below_relevance, disabled. No free text, no URLs, no provider bodies — and a test asserts the worker enum and the web read-model enum are identical, since they are declared separately on purpose.
• A 200 that is really an HTML page is parse_failed, not empty. That is the most common silent rot (paywall, challenge page) and it used to look like a quiet feed. The rss adapter throws a typed SourceFetchError; the message is sanitizeError'd before it can reach D1 or an API response.
• Streaks are carried, not recomputed. The worker reads the previous run's stats (one single-row read) and increments or resets. Surfacing "stale" on the read path therefore costs nothing no matter how long a feed has been dead — no history scan.

Threshold: 168 consecutive runs (7 days at the hourly cadence). This is a deliberate deviation from the "e.g. 48 runs" in #230, and it is measured rather than opinionated. When the registry was verified live, 11 of 18 feeds returned nothing inside the 26h window — and 6 of those are pre-existing:

11 valid feed(s) had nothing in the 26h window right now (openai, google-ai,
hf-blog, deepmind, aws-ml, mit-tr-ai, google-research, mit-news-ai,
lastweekin-ai, arstechnica-ai, wired-ai)

lastweekin-ai is a weekly newsletter and google-research publishes a few times a week. A 48-run default would have flagged most of them as broken on a quiet weekend — and an alarm that cries wolf is an alarm nobody reads. A row may override with staleAfterRuns; the one known case (arXiv, no weekend submissions, a measured ~54 consecutive silent runs) is documented with the follow-up row.

worker/__tests__/source-health.test.ts  (21 tests, all pass)
  ✓ reports a transport failure, a bad body, and a quiet feed differently
  ✓ calls out a source whose every item fell below the relevance floor
  ✓ never invents a reason for a source that delivered items
  ✓ has a human label for every enum value
  ✓ reads a well-formed record
  ✓ rejects a non-record and repairs a partially-written one
  ✓ does the same on the web read model side
  ✓ resets on any delivered item and increments on silence
  ✓ carries streaks forward per source and starts newcomers at zero
  ✓ reads the previous run's streaks out of a stats blob
  ✓ flags a source only once it passes its own threshold
  ✓ never calls a disabled source stale
  ✓ keeps the default above the quietest real feed in the registry
  ✓ gives arXiv a lower threshold when it is added, and not before
  ✓ keeps the read model's thresholds in step with the worker
  ✓ reports the streak and threshold together for the dashboard
  ✓ keeps disabled, observed, and unknown as three distinct states
  ✓ flags the stale ids and leaves the rest out
  ✓ treats a run with no sourceHealth at all as unknown, not as a mass failure
  ✓ treats an unparseable health value as unknown
  ✓ never emits a skip reason outside the shared enum

Surfaces: /api/system/sources gains health, stale, healthRunAt and a last_item_at per row; the /data Sources table gains Fetched / New / Accepted / Rejected / Last item / Health columns; the /data Algo tab gains a Source health card (run totals, stale list, top problems).

Disabled ≠ stale. A source that is off is reported disabled, never flagged — flagging it would train an operator to ignore the alarm.

5. The Vietnamese path — verified, including what it does NOT cover

The link that was dead is now live. No adapter emitted sourceLang: "vi", so the entire VI→EN branch was unreachable. The rss adapter now reads config.sourceLang, and only "vi" counts — a typo ("VI", "fr", true) is ignored rather than guessed, because a bad config value must not put an item on the wrong side of the QA path.

Disjoint chain, fail-closed. resolveIndependentReviewerChain requires ANYROUTER_REVIEW_MODEL to be concrete and disjoint from ANYROUTER_TRANSLATE_MODEL ∪ ANYROUTER_ENGLISH_TRANSLATE_MODEL; a missing or overlapping config returns an empty chain and the runtime reports "not configured" with zero LLM calls rather than self-reviewing with the generator. That behaviour is unchanged and already covered; the new tests exercise the VI branch on top of it.

No silent substitution. I reproduced the real production failure visible in /api/system's llm_calls: the EN generator (google/gemini-3.5-flash) works, but both configured reviewer ids (x-ai/grok-4.7, z-ai/glm-5.3-flash) return 404 from the provider, so a VI item cannot be reviewed at all. The new test never silently substitutes a VI item when the reviewer chain is dead drives exactly that and asserts:

• accepted === 0
• no UPDATE translations SET … title = ? — the original candidate is preserved
• a review_failed attempt is persisted with a scheduled retry, not swallowed
• attempt 1 does not jump straight to the terminal state

The 3-attempts → terminal human_review → exactly-one-authenticated-retry transition is already proven end to end in translation-qa.integration.test.ts ("caps automatic attempts and permits exactly one authenticated retry", asserting attempt_count: 3, manual_retry_count: 0, then a permitted 4th, then a rejected second human retry with HTTP 409). I did not duplicate it.

One honest gap I found and did not fix. If EN generation fails (before any candidate exists) there is no candidate to review or preserve, so the code increments budget.failed and logs translation_review.english_candidate_failed but creates no human_review row. The item keeps its own Vietnamese title — correct, and it renders with no EN badge — and is not substituted, but it does not enter the review queue. That is pre-existing generator-stage behaviour, not something this change introduced, and fixing it means touching the QA state machine, which #230 explicitly scopes out. It is pinned by a new test (leaves a VI item's Vietnamese title intact when EN generation itself fails) so it cannot regress unnoticed. A second, related pre-existing issue: with both reviewer ids 404ing, no VI item can ever be accepted until ANYROUTER_REVIEW_MODEL is pointed at working ids. That is a deploy/secret change, not a code change, and it is out of scope here.

6. LLM budget: unchanged, and the arithmetic is asserted

No hang-cap, batch size, model chain, or prompt is touched. rank_score and the relevance < 0.4 hide rule are untouched — worker/ranking.ts is not modified, and no source-specific boost or penalty exists anywhere in the diff. New sources compete on the existing formula.

The flood gate is what makes the added volume fit. From ALGORITHM.md § LLM transport: scoreItems chunks into batches of 5, runs 3 concurrently, each attempt bounded by the 70s score hang-cap, all inside one 4-minute step. The measured steady state is ~4 new items per run (observed across the last 12 production runs: 8, 4, 4, 4, 0, 12, 1, 3, 1, 0, 4, 3).

• Each new row's maxItems ceiling is 6.
• Per-run ceiling across all five new rows: 30 → 6 batches → 2 rounds → 2 × 70s = 140s worst case, inside the 240s step budget.

That whole derivation is asserted in ✓ caps the whole added source set inside the existing score budget, so a future cap change fails in CI with the numbers in the diff rather than in a stale PR description. Steady state is far lower than the ceiling: the caps bound a 26h window that dedupe then collapses, so a newsroom publishing 4/day contributes ~0.2 items per run.

7. Verification

gate	result
pnpm exec biome check apps/web/worker apps/web/src apps/web/scripts	pass, 503 files, 0 errors 0 warnings
pnpm --filter @aidr/web test	pass, 176/176 files
pnpm --filter @aidr/web check-types	pass, tsc --noEmit clean
pnpm --filter @aidr/web run build	pass, ✓ built in 1.90s + Clerk proxy config assertion passed
pnpm run gen:source-migration -- --check	pass, 0027_source_registry.sql matches the registry.
pnpm run check:migration-order	pass, media manifest migration file order check passed
pnpm run verify:translation-schema	fails on wrangler auth, not on this change — see below
pnpm run verify:source-feeds	18/18 PASS

All re-run after git rebase origin/master (which pulled in #233 and #236). The rebase was conflict-free: master and this branch share no file.

A real ingest was run and its completion was proven, not assumed. Per ALGORITHM.md § Ops pitfalls, a 2xx POST and a green GitHub Actions run are not a finished ingest. gh workflow run ingest.yml reported completed success in 9s; polling GET /api/system with cache-control: no-store showed the truth:

poll 1..4:  lastRun=8e21bbd6…  finished_at=1790528801  items_new=0  runsToday=16
POST id:    e36120de-c0fd-440e-979b-b0809ec3a904
poll 5:     lastRun=e36120de…  finished_at=1790528851  items_new=0  runsToday=17
poll 23:    lastRun=e36120de…  finished_at=1790529224  items_new=5  published=1  runsToday=17

The run took ~6 minutes to finish behind a 9-second "success". runsToday 16 → 17.

What I could NOT verify, and exactly why

The Cloudflare credentials on this machine do not have access to the production account, and the local NEWS_ADMIN_TOKEN no longer matches the deployed secret, so I could not apply migration 0027, could not deploy, and could not add the new rows to production:

$ env -u CLOUDFLARE_API_TOKEN pnpm exec wrangler d1 list --config wrangler.toml
WARNING The `account_id` in your Wrangler configuration (7df185a1…) does not
        match any of your authenticated accounts.
Token Permissions: account (read)        # authenticated account is 23050adb…

$ curl …/api/admin/mcp -H "authorization: Bearer $NEWS_ADMIN_TOKEN"   # 401 unauthorized
$ curl -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" …/user/tokens/verify
{"success":false,"errors":[{"code":1000,"message":"Invalid API Token"}]}

pnpm run verify:translation-schema fails for exactly this reason, and fails identically on a clean tree with my changes stashed — it is not caused by migration 0027:

$ git stash push -u && pnpm run verify:translation-schema
wrangler schema verification failed; deploy is blocked      # exit 1, no 0027 present
$ git stash pop

I did confirm the half of the gate that does not need the network, and that 0027 does not interact with it (it is a pure INSERT OR IGNORE INTO sources; no column, no trigger, and it does not shadow 0023/0024/0025):

last 3 migration files: 0025_llm_call_run_identity.sql, 0026_clerk_users.sql,
                         0027_source_registry.sql
required by the gate:    0023_translation_reviews.sql, 0024_item_media_manifest.sql,
                         0025_llm_call_run_identity.sql
assertMigrationFileOrder: PASS

Reviewer re-verification, on a machine with production credentials

pnpm install
pnpm --filter @aidr/web run check:migrations      # applies 0027; needs wrangler auth
pnpm --filter @aidr/web run verify:translation-schema
pnpm --filter @aidr/web exec tsx scripts/gen-source-migration.ts -- --check
pnpm --filter @aidr/web run verify:source-feeds   # 18/18, re-confirm the 5 new rows
pnpm --filter @aidr/web run deploy

# Real ingest, then poll (do NOT trust the POST or the green Actions run):
curl -X POST "https://aidr.today/api/admin/ingest?force=1" -H "Authorization: Bearer $NEWS_ADMIN_TOKEN"
# until lastRun.id == the POST id AND finished_at advanced
curl -s https://aidr.today/api/system -H 'cache-control: no-store'

Then the three measurements that need the deploy:

1. items/day before → after. BEFORE, measured on /api/feed?days=2&lang=en twice for stability: 106 items over 3 days = 35.3/day (25/51/30 and 28/51/27). Re-run the same call after the deploy and diff it — I have deliberately not predicted the number, because the relevance < 0.4 hide rule decides how many Vietnamese and newsroom items survive and that needs the LLM.

2. Per-source distribution and the ≤25% share. BEFORE, the largest single source is already at 29–30% (marketbrief 31–32/106) — a pre-existing condition this change neither introduces nor worsens. The distribution to compare against:

   marketbrief    31-32  29-30%     the-decoder   10   9.4%
   huggingnews    26-27  25-26%     marktechpost    6   5.7%
   hn             17-18  16-17%     simonwillison   1-2 0.9-1.9%
   lobsters       11     10.4%     mit-news-ai    1   0.9%
                                    openai          1   0.9%
   

3. The new surfaces. /api/system/sources → health / stale / healthRunAt and last_item_at; /data → Sources (new columns) and Algo (Source health card); a VI item rendering with the Vietnamese title and no EN badge at its /{8hex}?lang= permalink; and 5 sampled story URLs checked for no robots.txt / canonical / X-Robots-Tag regression. I did not touch src/lib/sitemap.ts, apps/web/public/robots.txt, src/lib/llms-txt.ts, src/lib/locale-response.ts, or src/styles.css.

8. Notes for the reviewer

• One existing test was changed, not just added. algo-tab-dedupe.test.ts asserted the Algo tab does not contain useSystemData at all, as a proxy for "must not fetch the model chains it no longer renders". P2 Coverage: add more verified news sources, make source config data-driven, and add per-source/stale observability #230 requires per-source health on that tab, so the guard is narrowed to its real intent: no API.overview, no models.scoring, and exactly one fetch (API.sources). The duplicate-chain assertions are untouched.
• loadSystemSources now selects all source rows, not just enabled = 1, in the same single round trip, so the fetch loop can skip disabled rows while the health map still records an explicit disabled reason. It also makes the dedupe step's sources.find(...) lookup correct for a disabled source's already-stored items.
• SourceFetchError is thrown for parse_failed, which the existing 3× exponential fetch retry will repeat. Two extra GETs against a publisher's own feed, once an hour, on a per-source step that runs in parallel with its group. That is the price of a feed that quietly became an HTML page being visible the next hour instead of after a week of silence. Documented at the class.
• Per-host pacing (minRequestIntervalMs) ships with no user. It was built for arXiv's 406-on-concurrent-requests behaviour, which I measured live. It is ~30 lines, keyed by host, only active when a row opts in, and it never delays the first request to a host. The combined-category arXiv row in the follow-up does not need it, so this is currently unused config surface — worth a follow-up to either exercise it or drop it. Flagging rather than hiding it.
• Migration 0027 is INSERT OR IGNORE with no conflict clause. Reconciling an existing row is the runtime seed's job; the migration only ever creates a row that does not exist yet.

Refs #230, umbrella #232.

[sourcery-ai]

Sorry @duyet, your pull request is larger than the review limit of 150,000 diff characters

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 25351bf0-69f6-435f-9261-ed64d160e534

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[duyet]

Reviewer note from the orchestrator — the one verification I could not reproduce locally.

pnpm run verify:translation-schema fails in this worktree, but it is not caused by migration 0027:

$ pnpm exec wrangler d1 migrations list aidr --config wrangler.toml --remote
ERROR A request to the Cloudflare API
  (/accounts/7df185a1.../d1/database/0c8f3efe.../query) failed.
  The given account is not valid or is not authorized to access this service
  [code: 7403]

The script wraps any wrangler failure in the generic wrangler schema verification failed; deploy is blocked, which hides the real cause. I confirmed it is environmental by running the same gate on a tree with no 0027 at all (migrations end at 0026) — it fails identically. The local CLOUDFLARE_API_TOKEN in .env.local is not scoped for D1.

So: 0027 does not block the gate, and CI's Lint, test, types check passed on this PR. But the honest gap is that the live remote-schema probe (the translation_reviews / media_manifest / run_id token list) has not been executed against production from here, and pnpm run deploy runs that gate before the build.

What I did verify locally:

✓ worker/__tests__/source-catalog.test.ts  14
✓ worker/__tests__/source-health.test.ts   21
✓ worker/__tests__/arxiv-flood-gate.test.ts 12
✓ worker/__tests__/source-seed.test.ts      4      (51 total)

Before deploying 0027 to production, please run the gate with a D1-scoped token:

cd apps/web && pnpm run verify:translation-schema

and apply 0027 in order per ALGORITHM.md (apply/verify migrations in order, and never from the QA worker). 0027 is insert-only and does not touch the translation contract, but the deploy gate should be observed green rather than assumed.