Skip to content

Persist data after computation #929

Description

@davidhassell

When the Dask array inside a Data object is computed the compute result is currently not cached. This means that when the data is accessed again, the computation (which may involve a reading from disk) is repeated. This is great for keeping resident memory down, but not so good for speed performance. Sometimes, speed needs to be prioritised over memory, particularly when data is being accessed from remote stores.

This behaviour should be controllable in two ways:

  1. Provide a persist keyword argument to Data.compute which leaves the Dask graph in a persisted state.
  2. Provide a configuration function cf.persist_data that controls whether or not to persist the data of every Dask graph computation.

Note that accessing the array attribute always trigger a Dask graph computation under the hood.

Currently, the only way to do this is in two stages:

>>> # Current (== 3.19.0) behaviour
>>> d.persist(inplace=True)
>>> a = d.compute()
>>> a = d.array

Proposed new options:

>>> # New ( > 3.19.0) behaviour
>>> a = d.compute(persist=True)
>>> with cf.persist_data(True):
...     a = d.compute()
...
>>> cf.persist_data(True)
>>> a = d.compute(persist=False)  # Override environment setting

See NCAS-CMS/cfdm#389

Activity

  1. added this to the NEXTVERSION milestone on Mar 9, 2026
  2. added
    enhancementNew feature or request
    performanceRelating to speed and memory performance
    daskRelating to the use of Dask
    on Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    daskRelating to the use of DaskenhancementNew feature or requestperformanceRelating to speed and memory performance

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions