feat(sources): add arXiv via rss.arxiv.org, flood-gated

[duyet]

Refs #230

Most of #230 shipped in #241. The arXiv row was the piece deferred because rss.arxiv.org was serving an empty weekend channel. It serves items now, so this adds it through the existing registry.

Changes

• arxiv-research row in worker/sources/catalog.ts (ARXIV_SOURCE, outside the frozen REGISTRY_0027 list) plus new idempotent migration 0030_arxiv_source.sql (upsert on name/type/config, never enabled). 0027 is untouched because prod already applied it.
• Read model sourceStaleThreshold mirrors the 72-run override.
• Tests updated: 0027 + 0030 must equal the registry; registry guard now asserts the arXiv row is robots-clean and gated; score-budget ceiling is 36 items / 3 rounds / 210 s, under the 240 s step budget; stale threshold test.

Verification

check	result
live feed (verify:source-feeds arxiv-research)	200 application/rss+xml, 1314 parsed, 1314 usable (title, URL, date), PASS
flood gate on live feed	1314 in window -> 6 after gate
biome lint	clean
@aidr/web test	211 files, 2498 tests pass
check-types	pass
build / deploy / prod D1	not run

Still open for #230

• Real ingest run, per-source share (25% cap) distribution, before/after /api/feed volume (needs deploy).
• Additional Lobsters tags / HN ranges.
• Migration 0027 needs applying in prod (D1 token).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LUvnDNBM63UC4pk8VMMz72

Summary by Sourcery

Add the robots-compliant arXiv RSS feed to the source registry with flood protection and weekend-aware health monitoring.

New Features:

• Add arXiv cs.AI, cs.LG, and cs.CL content as an enabled RSS source.

Enhancements:

• Apply AI keyword and item-count flood gates to limit arXiv feed volume.
• Use a weekend-safe 72-run stale threshold for arXiv while retaining the default threshold for other sources.

Tests:

• Update source registry, flood-gate, and stale-detector tests for the arXiv source and its budget constraints.

Chores:

• Regenerate the source registry migration with the arXiv entry.

[sourcery-ai]

Sorry @duyet, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 hours and 37 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 97631fea-c356-464b-b10d-96ee7c19ff2d

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

This PR enables arXiv through the existing source registry via the robots-clean rss.arxiv.org feed, limits its firehose volume with keyword and item-count gates, and applies a weekend-safe 72-run stale threshold. Registry, health, and score-budget tests are updated to enforce the new source’s compliance and runtime limits.

Flow diagram for arXiv RSS flood-gated ingestion

flowchart LR
    Feed[rss.arxiv.org combined feed] --> Keyword[ai keyword filter]
    Keyword --> Cap[Newest-first maxItems 6]
    Cap --> Registry[Source registry]
    Registry --> Ingest[Ingestion pipeline]

Loading

File-Level Changes

Change	Details	Files
Adds arXiv as a flood-gated RSS source using the permitted syndication host.	Registers the combined cs.AI, cs.LG, and cs.CL feed with an AI keyword filter and six-item cap. Regenerates the source registry migration with the enabled arXiv row. Updates registry tests to enforce the robots-clean host and both flood gates.	apps/web/worker/sources/catalog.ts apps/web/migrations/0027_source_registry.sql apps/web/worker/__tests__/source-catalog.test.ts
Introduces a source-specific stale threshold for arXiv’s weekend publication gap.	Maps arXiv to a 72-run threshold while retaining 168 runs for all other sources. Adds coverage for the arXiv override and default behavior.	apps/web/src/lib/system-queries.ts apps/web/worker/__tests__/source-health.test.ts
Revises the score-budget expectations for the additional source.	Raises the tested workload to 36 items across three rounds and eight batches while keeping the 240-second step budget ceiling.	apps/web/worker/__tests__/arxiv-flood-gate.test.ts

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.