Skip to content

[Audit/Performance] Mask component and pair scans repeat expensive full-plane work #69

Open

Description

@guijacquemet

Evidence (Sad Catapella, Batch 1; main d7a538e)

  • piece_counts does a whole-plane np.unique(..., return_inverse=True), creates a dense label-index image, then calls ndimage.label separately per object's bounding box. If many labels occur in widely separated patches, their bounding boxes approach the full image and the algorithm repeatedly traverses those pixels.
  • _scan_plane runs pair_counts(child,parent) for every ordered type pair. pair_counts revalidates entire masks, reconstructs unique labels, overlap pairs and parent sizes for each direction.
  • Number of ordered comparisons for K types is K(K−1) per T/Z plane (30 comparisons for six types). This repeats scans of the same pixels; the repeated pair work scales at least quadratically in mask-type count.

Impact

Severe potential CPU and allocation amplification on large label masks, large numbers of mask types, and multi-plane datasets. This is an algorithmic bottleneck hypothesis, not a measured slowdown.

Optimization and Numba plan

  1. Profile piece_counts, pair_counts, and _scan_plane with sparse instances, high-cardinality labels, scattered grouped objects, and 2–12 mask types.
  2. Compare existing SciPy C kernels against a single-pass label-aware connected-components implementation (Numba union-find or suitable compiled library) rather than Numba-wrapping ndimage.label.
  3. Cache each mask's label-ID remapping/areas once per plane, calculate unordered type pairs once when symmetric, and derive directed evidence without rescanning where mathematically sound.
  4. For pair overlaps, benchmark a Numba njit streaming contingency accumulator against NumPy unique/searchsorted; guard int32 label identity, int64 counts, sparse IDs and bounded memory. Do not introduce type coercion or nondeterministic tie-breaking.
  5. Require identical output report JSON/evidence against an independent reference on adversarial fixtures before accepting any optimization; report wall time, allocations, peak RSS, compilation cost and scalability.

Classification: confirmed repeated-work structure, speed benefit unmeasured. No benchmarks executed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions