Skip to content

[Sad Catapella N5][Scientific integrity] CSV accepts negative and int64-overflowing histogram bin counts #45

Closed

Description

@guijacquemet

Baseline: f784c95. Confirmed source-level defect (isolated NumPy overflow probe performed; repository tests not run).

Code: PlaneHistogram.post_init has no numeric invariants. csv_io._record checks only int(counts.sum()) + tail_counts == valid_pixel_count and does not refuse negative bin counts, int64 overflow, negative totals, or valid_pixel_count > total_pixel_count. Writer accepts direct malformed histograms. Existing test changes one count but not compensating negatives or overflow.

Counterexamples: A valid row whose bins are [1,1] (valid=2) can be edited to [3,-1], still summing to 2. Counts [2**62,2**62,2**62,2**62,3] have exact sum 2**64+3 but NumPy int64 sum wraps to 3; setting valid=3 and total=3 passes the current equality check. An isolated NumPy probe verified the wrap (no package fixture run).

Impact: Scientifically impossible pixel counts accepted and potentially used for erroneous percentiles/normalization, distinct from #31 pooled float-count precision and #43 duplicate rows.

Fix/acceptance: Enforce consistent invariants for both public histogram objects and CSV read/write: 1D integral int64 counts, expected bin length, all counts nonnegative, nonnegative integral total and valid, valid<=total, nonnegative tails, overflow-safe exact aggregation and equality to valid. Refuse totals that exceed representable limits rather than wrapping. Include row/channel references in CsvFormatError. Regression: both counterexamples, negative valid/total, valid>total, float counts, empty/all-NaN controls, no partial write, and ordinary large valid histograms. Benchmark validator before considering Numba; correctness first.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions