ENH: a few design ideas

Some ideas I wanted to share:

Add an ignore marker.
Currently we have persist (run if output is missing) and skip (skip the task and everything downstream). An ignore marker could skip only the task itself while still allowing downstream tasks to run. This could be useful, for example, when dependencies or products were removed via DVC (or similar tools) and the user wants to ensure the task is not re-run.
Should the file location be part of the task ID?
At the moment, a task is re-run if the containing file is moved. An alternative design would ignore such changes if the task content itself did not change.

@DrorSh

Hi @DrorSh, suprt late to the party but

https://pytask-dev.readthedocs.io/en/stable/reference_guides/configuration.html#confval-ignore should help you. Maybe a decorator is more ergonomic? Let me know.
if the file location is not part of the task id, then you easily have duplicated ids. I always wanted to avoid that because I didn't want people to think about task ids and hide this detail as much as possible. I thought https://pytask-dev--847.org.readthedocs.build/en/847/how_to_guides/reconciling_lockfile_state.html#accept-the-current-state-for-selected-tasks could be a better way to move tasks and align the lockfile with the changes without running the tasks again. Let me know what you think if you have time

Tobias,

Thank you so much for your work on the lockfiles! I'm excited to transition to the new version.

Regarding the two comments above:

Thanks for pointing out the configuration option for ignoring a task. I still think a decorator can be helpful, and more in line with the persist and skip features.
I'll explore the lockfile accept feature. Looks promising. I'm wondering whether it makes sense to be a bit more paternalistic and require task id uniqueness (regardless of file name). Non-unique task ids mean that DAG representations can be confusing etc. On the other hand, I realize now that moving files around without rerunning tasks might break the chain and the user will not notice. I guess then that its better to leave as is.

Best
Dror

Navigation Menu