Repository navigation
gdalcubes read failures after open, mask-open failures and worker crashes are still silent #99
Description
Problem
#87 made drift abort on a gdalcubes chunk whose band image fails to open (an expired or refused signed URL), by reading the per-chunk chunk_status gdalcubes writes into its NetCDF. That is the only failure gdalcubes records. It sets chunk_status in six places, all in image_collection_cube::read_chunk(), and only band GDALOpen returning NULL can fire in practice. Every other I/O step between GDALOpen and the final nc_close drops its return code, or warns and continues, leaving the status OK: ChunkAndWarpImage, every netCDF call between worker and main process, CPLMoveFile, sqlite3_step.
A code-check enumeration of the write pipeline (#87 round 3: 38 failure paths, 25 detected by drift, 13 not; table in the #87 PR's planning/archive/2026-10-issue-87-silent-chunk-read-failures/review-round3.md) and offline probes on gdalcubes 0.7.5 (appelmar/gdalcubes@ed68331, parallel 1 and 4):
| case | chunk_status |
cells |
|---|---|---|
| mask (SCL) image opens, then a tile read fails | OK | scene used UNMASKED: the mask buffer stays NaN, which never matches a mask value, so cloud values reach the median (all-cloud scene: 1600 of 1600 cells set) |
| scene with no mask asset | OK | scene used unmasked |
worker writes no chunk file (nc_create / CPLMoveFile fails; worker exits 0) |
fill, no R warning | those cells NA |
worker chunk file partial (missing dims/vars, or nc_get_var_double fails unchecked) |
OK | empty, or an uninitialised buffer merged as data (read from source) |
sqlite3_step error in find_range_st |
OK | chunk's image list silently truncated (read from source) |
| band image opens, then a tile read fails (corrupted tile here; a throttled or refused range request on the network) | OK | every cell set, from the scenes that read |
| mask (SCL) image fails to open | OK | that scene silently dropped |
| a worker chunk file the main process cannot open (e.g. truncated when the temp dir fills) | OK, no R warning | those cells NA (measured in the #87 round-2 review: 1536 of 6400 cells, a C++ stderr line only; chunk_data::read_ncdf() returns on nc_open failure without setting a status) |
| a gdalcubes worker killed by a signal (SIGKILL, segfault) | — | write_ncdf() hangs; never raises or returns (round-1 review of #87, measured to a 150 s timeout) |
gdalcubes_options(log_file =, debug = TRUE) writes the same 11 lines for clean and failing reads, so it is not a detector either.
Where in gdalcubes (paths under src/gdalcubes/src/, from the #87 plan review):
image_collection_cube.cpp:424-437sets INCOMPLETE only whenGDALOpenfails orwarp()returns nullptr;warp.cppwrites into a MEM dataset it creates first, does not checkChunkAndWarpImage(:256), and logs a failure only atGCBS_DEBUG(:650-654).image_collection_cube.cpp:569-579: a mask that fails to open warns andcontinues the image loop withoutset_status.
Status (2026-10-14): fixed on the fork, drift side in drift#102
Edited 2026-10-14: the "Options" section and the "NOT POSTED" draft below were replaced by
what was built. The table above is the 0.7.5 measurement and stays as measured, except the
"mask (SCL) image fails to open" row: in drift's read shape (apply_pixel(), no
select_bands()) gdalcubes also reads SCL as a band, so a deleted SCL is already recorded on
0.7.5. The row holds only with select_bands().
gdalcubes fork: first built as one branch (NewGraphEnvironment/gdalcubes#2, off ed68331, now closed), then split for upstream (below). With the fix, each row of the table ends up as follows:
| case | with the fix |
|---|---|
| mask opens, then a tile read fails | INCOMPLETE; the scene is dropped, not used unmasked |
| band opens, then a tile read fails | INCOMPLETE |
| mask fails to open | INCOMPLETE (ERROR when strict) |
| worker chunk file the main process cannot open, or partial | ERROR; the buffer is discarded |
| worker killed by a signal | raises. The hang was the log printer: while (!eof()) on a log a worker killed at startup never wrote. A worker killed mid-write already raised |
| scene with no mask asset | unchanged in gdalcubes; drift drops the item (stac_features_masked(), drift#102) |
worker writes no chunk file (nc_create / CPLMoveFile) |
not fixed: fill, no warning |
sqlite3_step error |
not fixed |
Root cause: warp_simple() ignored ChunkAndWarpImage()'s return. warp() now returns nullptr on a failed warp, which reaches read_chunk()'s existing INCOMPLETE branches.
Tests:
test_chunk_status.R: 32 assertions, 10 fail on 0.7.5, all pass on the fork.test_worker_failure.R: times out on 0.7.5, passes in about 1 s on the fork.
Upstream: posted 2026-10-14 as three PRs, no issues. The issue drafts were folded or held back, to keep the ask small:
| upstream PR | fork PR | fix |
|---|---|---|
| appelmar/gdalcubes#113 | gdalcubes#4 | hang when a worker dies before writing its log |
| appelmar/gdalcubes#114 | gdalcubes#3 | failed reads and mask failures recorded in chunk_status (the one drift needs) |
| appelmar/gdalcubes#115 | gdalcubes#5 | worker crash: overview opened after the source was closed |
Each PR carries its own reprex (fails on 0.7.5, passes with the fix) and test. They are independent and conflict only in NEWS.md; we rebase the rest after each merge. Issue A (gdalcubes#6) was folded into #114. Issues B and C (gdalcubes#7, gdalcubes#8) are reports without a fix, held back (upstream-held) until we have one or someone else hits them.
Available now: all three fixes are merged on the fork's newgraph branch (upstream 0.7.5 + fixes; full suite 165/165): pak::pak("NewGraphEnvironment/gdalcubes@newgraph").
drift: drift#102.
Remotes:staysappelmar/gdalcubes.- Tests that need the fix skip on 0.7.5 with a drift#99 reason.
- Normal runs stay unprotected against a read that fails after open until the fix is in the gdalcubes they run.
- The drift-side worker timeout (option 2) is no longer needed, since the hang is fixed at its cause.
Close when: the fix is in the gdalcubes drift runs, through an upstream merge or a decision to pin or install the fork.
Mitigation already in drift
#87 added GDAL_HTTP_MAX_RETRY = 3 and GDAL_HTTP_RETRY_DELAY = 5 to the cube/composite session and to every dft_stac_fetch(), so a transient 429/5xx is retried rather than left as a hole. Per-extent re-signing (#79) keeps tokens live.
Relates: #87, #79, drift#102, NewGraphEnvironment/gdalcubes#3, NewGraphEnvironment/gdalcubes#4, NewGraphEnvironment/gdalcubes#5, appelmar/gdalcubes#113, appelmar/gdalcubes#114, appelmar/gdalcubes#115, NewGraphEnvironment/gdalcubes#6, NewGraphEnvironment/gdalcubes#7, NewGraphEnvironment/gdalcubes#8
Activity
Merge pull request #102 from NewGraphEnvironment/99-gdalcubes-read-fa…