dft_stac_fetch() reads one STAC page — a wide AOI can build a raster from a truncated item set #51

Problem

dft_stac_fetch() calls rstac::get_request() with no items_fetch()
(R/dft_stac_fetch.R:127-134), so the item collection handed to
gdalcubes::stac_image_collection() is one page. If the API paginates, the cube is built from
a partial item set and the returned raster is silently wrong over the missing tiles — no error, no
warning, plausible output.

Planetary Computer returns no numberMatched (measured: the response carries
type, links, features, numberReturned only), so nothing in the response advertises the
total. The only signal is a rel="next" link.

Corrected during the work (2026-09-01): a next link is not a truncation signal.
rstac:::items_fetch.doc_items mutates only items$features and never items$links, so a
fully paged collection still carries page 1's next. Measured on io-lulc-annual-v02 over the
packaged AOI (14 items ground truth): at limit=1 and limit=3 the fetch returns the complete
14 items and still reports a next link. A surviving next means paging happened, not
paging is incomplete. See the "Suggested" correction below.

Measured 2026-09-01 on a small AOI: 7 items, links rel: root, self — no next, so this has
never fired in testing. A whole-watershed-group floodplain over 7 years is a much larger item set,
which is exactly the case that has not been exercised. Same shape as
"a fixture set that cannot reach the failure mode is not validation".

What changes if we do it

dft_stac_fetch() pages to exhaustion, so a wide AOI returns the raster it should. Consumers stop
having to detect truncation themselves — floodplains currently checks for a next link and warns
(NewGraphEnvironment/floodplains#33), which is a workaround for something only drift can fix.

What happens if we never do

A large-AOI fetch can silently produce a raster with missing tiles. Every downstream number is
computed from it and looks fine. Because it depends on AOI size, it will first appear on the
biggest, most published area rather than in a test.

Suggested

rstac::items_fetch() after get_request(). or a documented page limit with a hard error when a
next link survives. Worth a test with a deliberately tiny page size so the paging path is
exercised at all.

Corrected during the work: the second option would have been a bug. A hard error on a
surviving next link aborts every correctly-paged fetch (see the correction above) — a guard
that fails toward abort. What shipped instead:

• page with items_fetch(), then sign (order matters — signing first leaves page 2+ unsigned);
• strip the stale next link rather than erroring on it, so a caller re-running
  items_fetch() on attr(, "stac_items") cannot silently duplicate pages 2..N;
• match next case-sensitively, as rstac does, and deliberately keep a case-variant with
  a warning — links(items, rel == "next") does not match NEXT, so such a link means rstac
  could not follow it and stopped after page one, i.e. the truncation this issue is about. It is
  the last local evidence of that, so stripping it would destroy the only signal;
• completeness checks: duplicate item ids (never skipped, and the only one that works on PC), and
  items_matched() vs the count, asymmetric — abort on fewer, warn on more, since pgstac can
  report an estimated total.

The tiny-page-size test was built as suggested and is the two-answer test for the fix.

Downstream

floodplains' next-link check (NewGraphEnvironment/floodplains#33) becomes inert rather than
wrong: drift now strips the link it was watching for, so the workaround stops firing without
needing removal. Worth confirming there before closing that issue.

Also shipped

A deliberate cache-format break. No stac_cache_key() parameter changes with this fix, so a
raster already written from a truncated item set would keep being served under force = FALSE —
the wide-AOI users the bug hit hardest would get no fix at all. The key gains a salt and moves
79f67b7b9dae -> 2264b5dbef6e; fetch caches rebuild once. dft_stac_cube() caches are
untouched.

Secondary, much smaller

stac_cache_key() (R/dft_stac_fetch.R:210) is computed and embedded in the cache filename but
never returned. attr(result, "stac_items") is already attached three lines later; attaching
attr(result, "cache_key") <- cache_key alongside would let a caller record which cache entry
served a fetch. floodplains has a slot for it and writes null today.