Skip to content

[Sad Catapella B3][Performance] Histogram run materializes every plane's bins and rewrites the entire CSV #77

Open

Description

@guijacquemet

Baseline: d7a538e56d41ac1093ba79adcad397c21b3dfd32. Classification: confirmed architecture/space complexity; specific speed and RSS impact unmeasured.

Evidence: _read_file() builds all PlaneRecord objects for a file; run_experiment() lines 582–645 retains all new and unchanged records and range checks in process-wide Python lists; lines 644–690 sorts all records, calculates intensity checks and rewrites the full table atomically. _load_previous() lines 914–952 reads the entire existing CSV into PlaneRecord objects even for no-change reruns. memory_limit is documented as per plane, not total job; this is a scalability opportunity, not a violation of the documented limit.

Scale illustration (not a measurement): 10,000 rows × 65,536 bins × 8-byte int64 counts = ~5.24 GB for one channel's count arrays alone (before records, range metadata, CSV text, channels and old/new structures). This can undermine successful blockwise image processing for large datasets.

Fix: design bounded-memory record aggregation and incremental staged CSV output, or an indexed columnar intermediate that preserves the current public CSV and atomic rollback semantics. Stream per-file rows, avoid accumulating both old and new representations, and make intensity summaries aggregate incrementally only where scientifically equivalent. Confirm row checksums/source identity and deterministic row ordering. Benchmark N planes, N channels, bins/row and repeated no-change runs, reporting RSS and wall time; no Numba benefit for orchestration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions