Repository navigation
Allow regridding for very large grids #878
Description
Activity
- addedenhancementNew feature or requestNew feature or requestregriddingRelating to regridding operationsRelating to regridding operations
on Jul 21, 2025 I attempted to regrid a tripolar grid to a Lat-lon grid using cf-python and was unable to. The tripolar grid was of size (1440x1206), is this considered a "large grid" in this case? What does a failure due to size look like in this case because I had let the command run for a few hours (in hopes of receiving some sort of error message), but decided to stop as it was taking too long and it seemed like it was not working.
Reacted by Sadie L. BartholomewHi @gchingjohnson,
Thanks for the question. I indeed hadn't really defined "large"!. Large is determined somewhat vaguely by the total number of grid points in both the source and destination grids and also the amount of memory on your machine. The main factor is the logical size of weights matrix, which has shape
MxN, whereMandNare the numbers of destination and source points respectively. Even though this matrix is calculated internally as a very sparse matrix, the temporary space required by ESMF for it's creation can be many times the matrix's size.In your case
Mis 1736640 (= 1440 x 1206), which is ~100 times smaller than the example above which prompted this. So, depending on the source grid size, maybe not large ...?Anyway, the symptom of "too-large" is RAM usage spiraling to 100%, your computer freezing, and eventually the python process killing itself (hopefully - I usually end up doing a hard re-boot because I'm impatient). No nice raised exceptions for this, I'm afraid, but it's easy enough see if it is the case.
I have a unit-tested PR ready to go on this, so if it is too large, we'll get that reviewed and in pronto (which I should do anyway :)).
However, if instead you've got a nice raised Exception, please post it here (along with a
dumpof the source and destination grids, if possible), and we can try to see where the issue is.Cheers,
DavidReacted by Sadie L. Bartholomew
Regridding to and/or from a very large grid (in terms of number of grid points) can fail due to running out of memory.
This occurs during the call to
esmpy.Regridwhich we use to calculate the regridding weights (https://github.com/NCAS-CMS/cf-python/blob/v3.18.0/cf/regrid/regrid.py#L2434). The problem isn't (so much) the size of the final weights matrix (although that can be quite large), more thatesmpy.Regriduses up a lot of memory for temporary arrays.A real use case using 'linear' regridding has destination grid size 181440000, resulting in a sparse matrix representation of the weights taking up$$8.79\ GiB \equiv 4 \times 181440000 \times 3.25 \times 4 / 2^{30}\ GiB$$ (have a poke round inside the guts of a
scipyCSR sparse array to see where that number comes from). The peak memory memory required byesmpyto create this is many times that!A work around for this is to calculate weights separately for non-overlapping partitions of the destination (not the source) grid, and then combining the weights from each of these partitions to create the final weights to partition the final weights matrix.
We can add this as an option to the regrid API:
Why not do this always? Because it is often not required, and there can be a large speed performance hit for doing this. (Therefore it could be recommended that when using
dst_grid_partitionsyou also save the resulting weights to a file for future use.)To see how this works, consider a nearest neighbour regridding from a size 6 source grid to a size 5 destination grid. The
5 x 6weights matrix for this could look like:This is clearly just a concatenation of the following
3 x 6and2 x 6weights matrices:The key is that two calls to
esmpy.Regrid(one for theJ0partition, one of theJ1partition) create the correct rows of this matrix for any subset of the destination grid, provided:conservative_2dandnearest_dtosmethods, then some weights would be incorrect, as when they are calculated they "don't know" about destination grid cells in any of the other partitions.