Repository navigation
ENH: Caching of inputs #338
Description
Is your feature request related to a problem?
Yes and no, at least pytask and my expectations were not aligned when I ran the code snippet below.
In particular, I expected the task to be re-run whenever the result from load_model_dict() changes, which is specified in a central module.
Describe the solution you'd like
I would love to see the possibility to hash Python inputs similarly to what is being done for file contents. Usually, they will be much smaller and it allows for more granularity (e.g., above I could of course specify the central config.py as a dependency, but this will mean doing so everywhere and whenever it changes, the entire pipeline will be run. Splitting its contents across many files would also be possible, but ugly).
I would actually like to see the inputs of hash_python_inputs or whatever a decorator might be called to be true in the spirit of correctness trumps performance (typically, these objects will be small relative to files).
API breaking implications
If the default is set differently as suggested before, behavior in some cases might change. Else just an addition.
Additional context
Add any other context, code examples, or references to existing implementations about
the feature request here.
def _get_parametrization(models):
id_to_kwargs = {}
for model in models:
id_to_kwargs[model] = {
"model_dict": load_model_dict(model_name=model),
"depends_on": ORIGINAL_DATA["data.dta"],
"produces": BLD["data"][model]["result.pickle"]
}
return id_to_kwargs
for id_, kwargs in _get_parametrization(MODELS).items():
@pytask.mark.task(id=id_, kwargs=kwargs)
def task_final_data(depends_on, produces, model_dict):
pass
Activity
tobiasraabe commentedon Jan 23, 2023
I like the idea! What are possible interfaces?
Extending
pytask.mark.depends_on@pytask.mark.depends_on({"first": ..., "second": ...}, hash=<>)where
<>can beTrueto hash all."first"to hash only the first dependency. (Positional indices if not a dict?)Extra decorator
@pytask.mark.hashed_depends_on()Wrapper for dependencies
Something similar to
pytest.paramhttps://docs.pytest.org/en/7.1.x/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-testhmgaudecker commentedon Jan 23, 2023
model_dictis an input to the task function similar todepends_onand do think that the leaves ofdepends_onshould always be files.IIUC, the above is mostly about hashing files, not hashing Python objects that are inputs to a task function?
tobiasraabe commentedon Jan 23, 2023
I had two things in mind.
depends_onalready exists and should be the entry-point for all dependencies regardless of type. It would feel counterintuitive to have it only for file path dependencies and not for everything else.FilePathNodethat collects files. pytask could be extended to aPythonObjectNode.And, then, hashing can be turned on if necessary. Paths are also currently not hashed; instead, we use the last modified date.
hmgaudecker commentedon Jan 23, 2023
Sounds good, but it might be hard to distinguish between
FilePathNodesandPythonObjectNodes. E.g., a string could be both? Would be fine to explicitly requirePythonObjectNode(model_dict)in the above example?I was genuinely surprised that the above did not work -- for me as a user it seems the same whether the body of a function changes or some input does. Is there any reason to allow task functions to have arguments beyond
depends_onandproduces, then?tobiasraabe commentedon Jan 23, 2023
Maybe deprecate strings as filepaths or some logic to differentiate it. Second, can be indeed ugly.
Not all input is tracked. Only
depends_onandproducesare. Sometimes additional inputs change the signature that triggers a re-run.Everything should be a dependency except for the products. Thus, we could remove
pytask.mark.depends_onbut how do we define products? With the decorator or is there something less bulky?hmgaudecker commentedon Jan 23, 2023
Wild thought: Can't we borrow from dags logic and just use reserved keywords
for task functions instead of the
depends_onandproducesdecorators? (picked those names only for obvious differentiation) These two would only ever refer to files; everything else would be aPythonObjectNodeunless aFilePathNodeis explicitly passed.Sure, I understand that now, it just was not intuitive to me 😇
tobiasraabe commentedon Mar 24, 2023
A new feature dropped in FastAPI that uses
Annotatedto add more metadata to function arguments. This could be very interesting for the implementation.https://github.com/tiangolo/fastapi/pull/4871tobiasraabe commentedon Sep 30, 2023
The feature will be available in v0.4. It is documented here: https://pytask-dev.readthedocs.io/en/latest/how_to_guides/hashing_inputs_of_tasks.html.