Skip to content

Record failed image reads and mask failures in chunk_status - #3

Merged
NewGraphEnvironment merged 1 commit into
newgraphfrom
chunk-status-read-failures
Oct 14, 2026
Merged

NewGraphEnvironment merged 1 commit into
newgraphfrom
chunk-status-read-failures

Conversation

@NewGraphEnvironment

@NewGraphEnvironment NewGraphEnvironment commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Second of three independent PRs (see ).

chunk_status records a band image that fails to open, but two other failures leave the chunk OK:

  • A pixel read that fails after the image opened (a corrupted tile, or a refused or throttled HTTP range request on /vsicurl/): warp_simple() ignores ChunkAndWarpImage()'s return, so the NaN output is used as data. For a mask image the scene is then used unmasked, since NaN never matches a mask value, and cloud reaches the composite.
  • A mask image that fails to open: the image loop continues without set_status(), so the scene is dropped with status OK.

Reproducible example (no network)

gdalcubes 0.7.5 · R 4.5.2 · macOS 15.6.1 (arm64) · GDAL 3.13.0. Two dates, a band and an SCL-style mask; the second date is all cloud, and its mask file is cut short so it opens but cannot be read.

library(gdalcubes)
d = tempfile(); dir.create(d)
tif = function(prefix, date, value) {
  v = cube_view(srs = "EPSG:32609", extent = list(left = 0, right = 400, bottom = 0, top = 400,
                t0 = date, t1 = date), dx = 10, dy = 10, dt = "P1D")
  invisible(write_tif(gdalcubes:::.raster_cube_dummy(v, 1, value), d, prefix = prefix, COG = TRUE,
                      creation_options = list(COMPRESS = "DEFLATE")))
}
tif("B04_", "2021-07-05", 500); tif("SCL_", "2021-07-05", 4)   # clear scene
tif("B04_", "2021-07-15", 900); tif("SCL_", "2021-07-15", 9)   # cloud scene
fmt = file.path(d, "format.json")
writeLines(c('{"description": "x", "tags": ["test"], "pattern": ".*\\\\.tif",',
  ' "images": {"pattern": ".*_([0-9-]{10})\\\\.tif"},',
  ' "datetime": {"pattern": ".*_([0-9-]{10})\\\\.tif", "format": "%Y-%m-%d"},',
  ' "bands": {"B04": {"pattern": ".*/B04_.*"}, "SCL": {"pattern": ".*/SCL_.*"}}}'), fmt)
col = create_image_collection(list.files(d, "\\.tif$", full.names = TRUE), format = fmt,
                              out_file = file.path(d, "col.db"))

# Damage the cloud scene's mask: drop the last bytes, which in a COG are tile data.
# The file still opens; reading its pixels fails.
f = file.path(d, "SCL_2021-07-15.tif")
b = readBin(f, "raw", file.size(f)); writeBin(b[1:(length(b) - 16)], f)

v = cube_view(srs = "EPSG:32609", extent = list(left = 0, right = 400, bottom = 0, top = 400,
              t0 = "2021-07-05", t1 = "2021-07-24"), dx = 10, dy = 10, dt = "P20D",
              aggregation = "median")
out = tempfile(fileext = ".nc")
raster_cube(col, v, mask = image_mask("SCL", values = c(8, 9))) |>
  select_bands("B04") |> write_ncdf(out)
nc = ncdf4::nc_open(out)
table(B04 = ncdf4::ncvar_get(nc, "B04"), useNA = "ifany")
ncdf4::ncvar_get(nc, "chunk_status")

On 0.7.5 the cloud scene is used unmasked (the median of 500 and 900) and the chunk is OK:

B04
 700
1600
[1] 0

With this change: 500 in all 1600 cells and chunk_status 2 (incomplete).

Change

  • warp() returns nullptr when Initialize() or ChunkAndWarpImage() fails, so read_chunk()'s existing !gdal_out branches mark the chunk INCOMPLETE (ERROR when incomplete_ok = FALSE), for band and mask images alike.
  • A mask image that fails to open sets INCOMPLETE, as the band path does.

One behaviour change to be aware of: an image for which GDAL cannot compute a source window also makes ChunkAndWarpImage() fail, so it now gives INCOMPLETE instead of silent NaN with status OK.

Adds inst/tinytest/test_chunk_status.R (band read, mask open and mask read failures, plus clean controls; 10 of its 32 assertions fail on 0.7.5). The full suite passes. R-CMD-check on master is currently failing as well, independent of this change.


Fork review copy. Everything above the line is the draft for the upstream PR to appelmar/gdalcubes; not posted. Replaces the issue drafted in #6, which is folded in here.

Relates to NewGraphEnvironment/drift#99
Relates to NewGraphEnvironment/sred-2025-2026#16

🤖 Generated with Claude Code

write_ncdf() records a per-chunk status, but only a band image that fails
to OPEN was ever recorded as non-OK. Two other failures produced a chunk
computed from partial input with status OK:

- gdalwarp_client::warp_simple() ignored the return of
  ChunkAndWarpImage(), and warp_complex() only logged it at DEBUG, so the
  INIT_DEST=nan output came back as if it were data. A band whose tile
  read fails after a successful open (corrupted tile, refused or
  throttled HTTP range request) was silently dropped from the chunk; for
  a MASK band the scene was used unmasked, since a NaN mask value never
  matches a mask class. warp() now warns and returns nullptr on a failed
  Initialize() or ChunkAndWarpImage(), after the existing cleanup, so
  read_chunk()'s existing `!gdal_out` branches mark the chunk INCOMPLETE
  (ERROR when strict).
- A mask image that fails to open `continue`d without set_status(); it
  now sets INCOMPLETE, and the strict branch returns an ERROR chunk like
  the band path does.

The test builds a two-date collection of single-band COGs (a band and an
SCL-style mask, the second date a cloud scene) from dummy cubes and
breaks the second date three ways: a band file whose first tile is
corrupted with the IFD intact, a deleted mask file, and a corrupted mask
file. At parallel 1 and 2 each must give a non-OK status and never let
the cloud value through; strict mode with a missing mask must give ERROR.
Clean controls (same CRS, another CRS, partial coverage, a coarser view
read through overviews) must stay OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRhUJsuKABLfpBGktPoiBN
@NewGraphEnvironment

Copy link
Copy Markdown
Owner Author

Posted upstream as appelmar#114 (2026-10-14).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant