Skip to content

Make the mask relationship scan resumable per FOV #164

Open

Description

@guijacquemet

Problem

scan_masks() currently accumulates the whole experiment in memory and describe_masks() writes only the final mask_relationships.json.

If Colab/Jupyter is interrupted late in the mask scan, the next run starts again from the first image. This is unlike the main preprocessing run, which stages completed images and resumes them.

This matters for large experiments and for segmentation workflows where masks arrive progressively.

Suggested simple implementation

Use one image/FOV as the resume unit.

After one FOV has been scanned successfully, stage a small record containing:

  • the mask inputs/fingerprints used for that FOV;
  • the scan/report schema version and thresholds;
  • the per-mask-type evidence for that FOV;
  • the per-pair relationship counts for that FOV.

On the next scan:

  • reuse staged FOVs whose masks and scan settings are unchanged;
  • rescan only new/changed FOVs;
  • never stage a FOV whose masks changed while being read;
  • aggregate the staged/current FOV evidence into the final report.

Keep this straightforward; no distributed scheduler or complicated database is needed.

Acceptance

  • An interrupted scan resumes without rereading completed unchanged FOVs.
  • A changed mask invalidates only the affected FOV.
  • New masks/FOVs are added without rescanning unchanged completed FOVs.
  • A resumed scan produces the same final mask_relationships.json as a clean full scan.
  • CLI and notebook use the same implementation.
  • Partial/interrupted work is never mistaken for the final published report.

Relevant code: mask_scan.py::scan_masks, run/mask_step.py::describe_masks.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions