You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Sad Catapella N1][Scientific precision] 'Exact' int64/uint64 histogram becomes inexact when representatives are stored as float64 #28
Normalization Batch 1 at f784c95e3049483f87466150ffc5764375fa631d. Confirmed numerical precision limitation conflicting with the 'exact' language, downstream fit policy requires decision. No full-repository tests executed.
Evidence
PlaneHistogram.representatives, lines 159–169 makes an np.float64 array and computes integer representatives using self.binning_min + (a+b)/2. The true singleton value cannot be represented for every int64/uint64 integer larger than 2**53.
PlaneHistogram.is_exact, lines 174–207 reports exactness based only on integer bin widths, ignoring actual representability of values in the float64 downstream model; max representative error is reported as 0.
Tests only check that full-range int64/uint64 counts sum correctly and representatives are finite, not that large integer quantiles are exactly preserved.
Counterexample
For np.array([2**53+1], dtype=np.uint64), the histogram has one occupied singleton bin and reports is_exact=True, but its representative becomes 9007199254740992.0, not 9007199254740993. An isolated NumPy float-conversion probe confirmed this loss. The counts and integer bin boundaries themselves can remain correct; the numerical downstream summary is inexact.
Impact / policy
Incorrect quantile value and false zero-error guarantee for high-bit-depth integers; grouping nearby representable and unrepresentable values may coalesce. Exact float output for arbitrary uint64 values is impossible, so either retain exact integer order-statistic values in a dtype-aware representation or explicitly qualify 'exact' as counts/partition only with accurate summary error bounds/refusal for nonrepresentable outputs.
Fix / acceptance
Choose a versioned contract policy, preserve exact 64-bit bin identities and quantile ordering where feasible, document any intentional float64 approximation for normalization parameters, and ensure error reporting never claims 0 if a represented singleton differs from the pixel. Test 2**53 ±1, near signed/unsigned extrema, adjacent integers collapsing to one float, single-bin and pooled quantiles, CSV round-trip, and exactness flags. No Numba benefit until numerical semantics are defined.
Activity
State the release impact of #28 in the changelog