Skip to content

Prevent retention cleanup from deleting unrelated or active result directories #55

Closed

Description

@guijacquemet

Problem

Two retention routines delete directories based on their location/name without proving that the directory is a completed result or record owned by LabConstrictor:

  • cli._new_results_dir matches any directory in <LC_HOME>/results whose name begins with YYYYMMDDTHHMMSS_, sorts matches and removes all but 19 before creating the new result folder. The regex is only start-anchored, not an exact name match.
  • runs._prune removes all but the newest 50 subdirectories of <LC_HOME>/runs, without checking their names or whether they are still being written.

Data loss scenarios

  1. A researcher saves their own directory under <LC_HOME>/results, for example 20261008T120000_reanalysis_backup. Once enough runs accumulate, it matches _RUN_FOLDER and shutil.rmtree deletes it. The existing test renames generated results folders and verifies only the count, not preservation of foreign folders (test_regressions_run2.py).
  2. More than 20 concurrently running CLI jobs can cause the next job to remove another job's still-active output directory, causing failed or incomplete scientific results. The code has no completion marker or in-use check.
  3. A manually created subdirectory under <LC_HOME>/runs is included in the pruning set even if the recorder did not create it.

The deletion happens as part of a routine tool run, without asking for confirmation. This is a data-retention correctness issue, not just cosmetic cleanup.

Proposed approach

  • Prune only entries known to have been created by LabConstrictor; use an exact naming/marker contract, not just a timestamp prefix.
  • Keep active result folders protected until the producing task completes. Account for concurrent CLI processes and failed/crashed runs.
  • Scope run-record pruning to valid owned record directories; ignore other folders and symlinks.
  • Prefer explicit retention age/size policy or a safe cleanup API over deleting unknown directories.

Acceptance criteria

  • A user-created YYYYMMDDTHHMMSS_*backup directory survives hundreds of CLI runs.
  • An active run cannot lose its directory when more than RESULTS_KEPT jobs run concurrently.
  • Arbitrary directories in <LC_HOME>/runs are not deleted by recording a task.
  • Normal, completed LabConstrictor results and run records are pruned to the configured retention count.
  • Regression tests run against a temporary LC_HOME and verify file contents as well as counts.

Priority: P1 — potential irreversible deletion of scientific output or unrelated files.

Verified by reading the two deletion paths and their tests. No destructive reproduction was run against a real installation.

Cross-host audit linkage

The same shared-home retention ownership problem is tracked in Fiji #28 and QuPath #14. Keep these host-specific issues open because each has its own destructive code path, but design a single retention policy for the shared LC_HOME so fixing the Toolkit alone does not leave other hosts deleting active or unrelated directories.

Activity

  1. guijacquemet commented on Oct 14, 2026

    @guijacquemet
    ContributorAuthor

    Addressed on main (checked 14 Oct 2026): result and run folders carry an owner marker when made and a done marker when the run ends, prune (labconstrictor-tools prune, retention.py) removes only marked folders, and the hosts adopt files they keep. Folders without a marker are never deleted. Delivered with the retention work (Tools 0.3.0a1 and host PRs #51 Napari, #30 QuPath). Closing.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions