Saving a solved FlowSystem spends ~80% of its time in zlib, and zstd is blocked by the netcdf4 pin #780

Prompt: "Compare how long the saving/archiving and Loading of an archived solve takes"

Summary

Saving a solved FlowSystem spends ~80 % of its wall time inside zlib. netCDF4 supports zstd, which on the same bytes writes ~8× faster and reads ~1.7× faster at the same file size — but the filter is absent from every netcdf4 release in our pinned range except 1.7.4, and the library's own capability flag reports it as present anyway. So there is nothing to change today. Filing it so that whoever next moves the netcdf4 pin knows the win is sitting there.

Measured — where the time goes

FlowSystem.to_netcdf() / from_netcdf(), 8760 hourly steps, N sink/source pairs, HiGHS, min of 3 repeats.

Linear in solution values (~190 ns/value save, ~85 ns/value load at the default). Phase split at 5.45 M: to_dataset 129 ms + netCDF write 934 ms on the way out; netCDF read 243 ms + from_dataset 215 ms on the way back. The bytes are the cost, not the Python object graph — _stack_equal_vars is already doing its job.

The codec, on identical data

5 M float64 (40 MB raw), one variable, through netCDF4 directly:

Round-trip exact in all three. Same size, ~8× the write, ~1.7× the read.

Why it cannot be adopted yet

pyproject.toml pins netcdf4 >=1.6.1, <1.7.5. Writing a variable with compression='zstd' and reading it back:

Two consequences:

1. Adopting it as the default would mean pinning >=1.7.4,<1.7.5 — a single release, and the one our own pin comment flags for missing wheels.
2. Files would stop being portable. A file written with zstd on 1.7.4 fails to open on 1.6.1 with NetCDF: Filter error: undefined filter encountered. For a format we hand between machines that is the whole cost, not a detail.

The capability flag cannot be used to feature-detect

>>> import netCDF4                      # netcdf4 1.6.1
>>> netCDF4.__has_zstandard_support__
1
>>> # ... and both writing and reading zstd then fail

It reports how libnetcdf was compiled, not whether the HDF5 filter plugin is reachable at runtime. A try zstd, fall back to zlib in save_dataset_to_netcdf would therefore have to probe with a real write rather than read the flag — and even then the on-disk format would depend on the writer's environment, which is worse for a format meant to be exchanged.

Also measured: the compression level is inert

On a real solved FlowSystem (3.7 M solution values, 29.7 MB in memory):

Levels 1 through 9 produce the same file; the whole cost is zlib being invoked at all. Compression itself earns its keep here — 2.2× for ~290 ms — so the default of 5 is not wrong and should stay. Lowering it to 1 would buy ~5 % of the write for 0.7 % more bytes, which is inside the noise. Worth recording so nobody re-measures it.

Proposed action

None now. Revisit when the netcdf4 pin moves past 1.7.4 — at that point save_dataset_to_netcdf can take compression='blosc_zstd' in its encoding dict for an ~8× write, provided we are willing to say that files written by that version need a reader of that version.

Severity

Performance only; no incorrect results. Saving is ~2 % of a solve at these sizes (1.0 s against 166 s), so this matters for workflows that save many small-to-medium solutions, not for one large one.