Skip to content

[Sad Catapella N1][Scientific precision] 'Exact' int64/uint64 histogram becomes inexact when representatives are stored as float64 #28

Closed

Description

@guijacquemet

Scope / classification

Normalization Batch 1 at f784c95e3049483f87466150ffc5764375fa631d. Confirmed numerical precision limitation conflicting with the 'exact' language, downstream fit policy requires decision. No full-repository tests executed.

Evidence

  • PlaneHistogram.representatives, lines 159–169 makes an np.float64 array and computes integer representatives using self.binning_min + (a+b)/2. The true singleton value cannot be represented for every int64/uint64 integer larger than 2**53.
  • PlaneHistogram.is_exact, lines 174–207 reports exactness based only on integer bin widths, ignoring actual representability of values in the float64 downstream model; max representative error is reported as 0.
  • PooledHistogram.__init__, lines 107–136 takes those rounded representatives and percentile(), lines 166–170 returns the rounded float64, causing an objectively incorrect 'exact' percentile for 64-bit input.
  • Tests only check that full-range int64/uint64 counts sum correctly and representatives are finite, not that large integer quantiles are exactly preserved.

Counterexample

For np.array([2**53+1], dtype=np.uint64), the histogram has one occupied singleton bin and reports is_exact=True, but its representative becomes 9007199254740992.0, not 9007199254740993. An isolated NumPy float-conversion probe confirmed this loss. The counts and integer bin boundaries themselves can remain correct; the numerical downstream summary is inexact.

Impact / policy

Incorrect quantile value and false zero-error guarantee for high-bit-depth integers; grouping nearby representable and unrepresentable values may coalesce. Exact float output for arbitrary uint64 values is impossible, so either retain exact integer order-statistic values in a dtype-aware representation or explicitly qualify 'exact' as counts/partition only with accurate summary error bounds/refusal for nonrepresentable outputs.

Fix / acceptance

Choose a versioned contract policy, preserve exact 64-bit bin identities and quantile ordering where feasible, document any intentional float64 approximation for normalization parameters, and ensure error reporting never claims 0 if a represented singleton differs from the pixel. Test 2**53 ±1, near signed/unsigned extrema, adjacent integers collapsing to one float, single-bin and pooled quantiles, CSV round-trip, and exactness flags. No Numba benefit until numerical semantics are defined.

Activity

  1. added a commit that references this issue on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions