perf: idempotent atomic parallel PMC downloads

[SkyeAv]

Makes PMC article file downloads idempotent, atomic, and bounded-parallel, so reruns cost nothing for already-fetched files and a kill never leaves a torn file. Previously every file downloaded sequentially, single-attempt, on every rerun.

Download contract

• Idempotency: a non-empty existing file is skipped without any request; a zero-length file (torn prior attempt) re-downloads.
• Atomicity: downloads write a .part sibling and os.replace it into place on completion; the .part is removed even on KeyboardInterrupt, so no half-written file survives.
• Parallelism: fetch_pmc_article fetches up to net.DEFAULT_FETCH_CONCURRENCY (8) files concurrently via a ThreadPoolExecutor, collecting results in submission order.
• Fail-fast validation: concurrency < 1 raises ValueError at the top of fetch_pmc_article, before any network call (kept in _download_files too, as defense in depth).

Design

• Worst-case wall clock (documented in fetch_pmc_article's docstring): 3 serial listing/metadata calls + ceil(N/8) waves, each paying ≤ 60 s of sleeps — ≤ 300 s at the observed N ≈ 15, about 6% of the 90-minute per-article budget; a healthy run pays zero added time.

Testing

• uv run pytest tests/test_agent_net.py -q --no-cov -n 0 → 12 passed in 1.28s: skip-without-request, torn-file re-download, atomic replace + .part cleanup, deterministic submission-order under reverse completion (threading events, no sleeps), in-flight bound via simultaneous-worker counters, concurrency < 1, first-failure propagation (two scenarios separated by monkeypatch.undo())
• All tests < 0.03 s — no wall-clock assertions anywhere
• Full gate on the stack tip: uv run pytest -q → 1565 passed, 15 skipped; ruff/format/pyright clean; Tier-2 review passed all 11 criteria

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5fec2659-092c-4549-bbc8-e7348682301b

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}