Skip to content

[Sad Catapella N2][Correctness] PooledHistogram loses exact pixel counts through float64 weighted bincount #31

Closed

Description

@guijacquemet

Baseline and classification

Normalization Batch 2, commit f784c95e3049483f87466150ffc5764375fa631d. Confirmed precision defect from source inspection and an isolated NumPy probe. No repository pytest run.

Evidence

  • PooledHistogram.__init__, lines 107–138 combines per-plane int64 counts using np.bincount(inverse, weights=...), which returns float64, then casts rounded values back to int64. Pixel totals are consequently derived from the rounded merged counts, rather than the original exact integer totals.
  • _cumulative and percentile then operate on these rounded values; WeightedHistogram also relies on float64 weighted accumulation, which is appropriate for genuinely fractional weights but requires a stated precision policy.
  • Existing tests test_pooling.py 25–69 use small arrays and do not probe >2^53 counts. PlaneHistogram has no published 2^53 count limit.

Analytic counterexample, isolated NumPy probe confirmed

One valid logical singleton-bin histogram with counts = np.array([2**53 + 1], dtype=np.int64), and total_pixel_count = valid_pixel_count = 2**53 + 1. np.bincount(np.array([0]), weights=np.array([2**53 + 1], dtype=np.int64)).astype(np.int64)[0] is 2**53 on float64 NumPy: one pixel disappears. PooledHistogram.pixel_count and percentile rank inputs are then wrong. This does not require the histogram itself to be inexact; it is separate from #28 (high-intensity value rounding).

Impact and fix

Silent loss of exact count/provenance in very large or pre-aggregated datasets; count error might affect extreme percentile ranks and record validation. Rare for one actual FOV; important for a library advertising exact integer pooling without a documented limit.

For unweighted PooledHistogram, aggregate int64 counts with integer-safe operations (including explicit overflow detection near 2^63), e.g. ordered unique-key reduction with integer summation rather than float64 bincount weights. Distinguish the weighted API's floating tolerance and range. Assert pixel_count == sum(hist.valid_pixel_count) and counts/tail totals exactly, or explicitly refuse out-of-range counts. Add synthetic >2^53, near-int64-max, multi-histogram overlapping-bin and q=100 boundary tests, plus ordinary fixtures. Any schema-level change needs an explicit decision; no silent cap or float coercion.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions