Skip to content

Special, Stats, Integrate, Optimize: go-numerics and go-statistics as cgo-free scipy-style modules - #819

Merged
tannevaled merged 9 commits into
mainfrom
wave70-special
Oct 18, 2026
Merged

tannevaled merged 9 commits into
mainfrom
wave70-special

Conversation

@tannevaled

@tannevaled tannevaled commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Ruby gets scipy's special functions, distributions, quadrature and optimisers with no SciPy and no cgo — four modules of two organisations, under scipy's own names:

Ruby module from surface
Special go-numerics/special all 67 exported functions, under 66 Ruby names
Stats go-statistics/stats all 32 univariate distributions, 19 methods each
Integrate go-numerics/integrate 10 quadrature routines
Optimize go-numerics/optimize 11 root finders and minimisers
Special.gamma(5.0)                                      # => 24.0
Special.zeta(2.0)                                       # => 1.6449340668482264   (pi^2/6)
Stats.norm.ppf(0.975)                                   # => 1.9599639845400538
Stats.poisson(3.0).pmf(2)                               # => 0.22404180765538775
Integrate.quad(0.0, 1.0) { |x| x * x }                  # => [0.3333333333333333, 3.7e-15]
Optimize.brentq(0.0, 2.0) { |x| x * x - 2 }             # => 1.4142135623730951

a = NDArray.from([1.0, 2.0, 3.0, 4.0], [2, 2])
Special.gamma(a)                                        # => array([[1., 1.], [2., 6.]])
Stats.t(10.0).cdf(a)                                    # element-wise, same shape
Special.xlogy(2.0, a)                                   # a Float broadcasts against an array

66 Ruby names for 67 Go functions, because scipy spells the Riemann and Hurwitz zeta as one zeta(x, q = nil).

The names are measured, not transliterated

Every module states its own specification: it compares itself against scipy in ULPs, and the pairing is a file in its repository — bench/scipy_server.py and testdata/gen/fam_*.py for special, each distribution's doc comment for stats. Those files are the bibliography, so the Ruby names come from them rather than from the Go name:

Go Ruby (= scipy) transliteration would give
Log1pExp softplus log1pexp ❌
LogGamma gammaln log_gamma ❌
GammaP / GammaQ gammainc / gammaincc gamma_p ❌
StudentT t student_t ❌
ChiSquared / NoncentralChiSquared chi2 / ncx2 chi_squared ❌
DiscreteUniform randint discrete_uniform ❌
Weibull / Gumbel weibull_min / gumbel_r weibull ❌

Two functions have no scipy counterpart and keep the Go name: log_factorial, loggamma_sign.

One Ruby name, two Go paths — which is what a ufunc is

special ships each function twice: a float64 form in go-numerics/special and an element-wise form over *ndarray.Array in go-numerics/special/nd. They cannot share a package, because Go cannot hold two functions of one name — which is why there are two import paths here. rbgo already binds go-ndarray, so the array type is shared rather than introduced (see Dependency versions below for which version, and why it moved).

  • every argument a number → the float64 function, answering a Float;
  • any argument an NDArray → every argument goes through go-ndarray, a Float becoming a 0-d array, which broadcasts against any shape.

Measured before being written: Scalar(2) × shape{3}, shape{3} × Scalar(2), ternary, 2-D shape preservation. Stats's per-point methods are ufuncs the same way, through the module's own stats.Map.

Nothing is reimplemented. The element-wise path calls the module's own nd package, which already parallelises and broadcasts; argument checking reuses NDArray's helpers (ndArgc, ndFloat, ndInt, mustArray, ndRaise, ndKw); and the seventeen methods Stats::Continuous and Stats::Discrete share are defined once, against a Go interface holding exactly the part of the module's two interfaces that is the same. The two Ruby classes differ only in the density — pdf/logpdf against pmf/logpmf — which is the only place the Go interfaces differ.

#rvs answers a Float with no size and an NDArray with one, as scipy's does; without seed: it draws from one generator per VM that advances across calls (scipy's module-level random state), and seed: asks for a fresh reproducible stream. A parameter outside its domain is refused at construction, as an ArgumentError where SciPy raises ValueError.

Verification: facts, not the code being bound

Comparing a binding against the Go function it calls proves only that the call happened. So every expected value is a mathematical fact:

  • 56 closed forms for Special — gamma(5)=24, gamma(0.5)=sqrt(pi), zeta(2)=pi^2/6, digamma(1)=-euler_gamma, polygamma(1,1)=pi^2/6, ellipk(0)=ellipe(0)=pi/2, beta(2,3)=1/12, xlogy(0,0)=0, betainc(1,1,x)=x, inverse pairs round-tripping, j0(0)=1…
  • 57 closed forms for Stats, covering all 32 distributions — each constructor is written out separately in scipy's shape-then-loc-then-scale order, so a table of known moments is exactly what catches a swapped or dropped parameter: norm.entropy=(1/2)log(2 pi e), poisson.var=mean, binom(10,.5).pmf(5)=252/1024, zipf(2).pmf(1)=1/zeta(2), randint(0,10).var=8.25, gumbel_r.mean=euler_gamma, rayleigh.mean=sqrt(pi/2), f(5,10).mean=1.25, ncx2(2,3).mean=5, hypergeom(10,5,3).mean=1.5…
  • 42 identities across four continuous and two discrete laws — sf=1-cdf, the logs are logs, ppf inverts cdf, isf(q)=ppf(1-q), std^2=var, median=ppf(0.5), interval(0.95) covering 0.95 of the mass. These hold for any distribution, so a method wired to the wrong Go function breaks one without breaking the others.
  • the five non-finite answers scipy gives outside a domain (gamma(0)=Infinity, logit(2)=NaN…), which a binding easily turns into a Ruby error or a zero;
  • jn_zeros against the tabulated zeros of J0 and against j0 vanishing at the first one — a check on the pair, not on either alone;
  • the ufunc property — scalar and element-wise paths answering the same numbers, at every arity and every method, which is the only part of this that is not the modules' own code;
  • the method set of each Stats class, as a SET, so a method lost from the shared definition fails;
  • #rvs draws from the law asked for — 20000 seeded draws from norm(10, 2) with a sample mean and sd within 6 standard errors, which is a check and not a flake;
  • the surface is complete — Special defines exactly 66 names and Stats exactly the 32 the module has univariate constructors for, so a function added upstream and forgotten here fails rather than going unnoticed;
  • argument and parameter errors at every arity, with MRI's message.

Ten negative controls on the bindings, each verified to have really changed the file: binding softplus to Logit, pointing gamma's array path at LogGamma, dropping an arity check (which makes the census name the exact 33 methods affected), swapping loc and scale, pointing Stats.t at ChiSquared, wiring #pdf to CDF, dropping a shared method, ignoring seed:, drawing from the wrong law, and skipping parameter validation — all turn the suite red.

A README that cannot drift

The README lists these names one by one, which is the claim that goes stale silently. readme_numeric_surface_test.go reads the names the four bindings register and the names the README lists and requires the sets to be equal in both directions, with each entry's non-method words named beside their reason (scipy's loc/scale, the method: values, the block-pass idiom) rather than one shared list that would also excuse a real omission.

Writing it found two defects in the README text committed minutes earlier: the Special entry claimed all 67 functions while omitting log_factorial and loggamma_sign from its list, and it backticked log1pexp and log_gamma as counter-examples — which reads as a promise. Both fixed; the counter-examples are italic now, because a name that deliberately does not exist is not code. The guard was then broken in all three of its directions (a name dropped from the list, a skellam promised that does not exist, #entropy removed from the binding) and went red for each, and again for the two new modules.

A side finding, filed separately

A census over rbgo itself found 31 methods — all 17 of FFT's and 14 of Image's — that index args[0] with no arity check, so FFT.fft() answers ArgumentError: fft: runtime error: index out of range [0] with length 0: a Go runtime panic converted into a Ruby error, where MRI says wrong number of arguments. NDArray is clean because it uses ndArgc, which is what all four modules here use. Filed as #820 with the full table; TestSpecialArityIsAlwaysChecked, TestStatsArityIsAlwaysChecked and TestNumericsArityIsAlwaysChecked keep these four modules at 0, as a census rather than a list, so a method added later is covered without being named.


Second half: Integrate and Optimize

The request names go-numerics, which is three modules. Special above is one; these are the other two, and together they are the four symbols the notebook census ranks highest inside scipy — integrate.quad, optimize.minimize, minimize_scalar, brentq.

Integrate.quad(0.0, 1.0) { |x| x * x }                  # => [0.3333333333333333, 3.7e-15]
Integrate.quad(0.0, Float::INFINITY) { |x| Math.exp(-x) }   # an infinite limit
Optimize.brentq(0.0, 2.0) { |x| x * x - 2 }             # => 1.4142135623730951
Optimize.minimize([-1.2, 1.0]) { |v| 100*(v[1]-v[0]**2)**2 + (1-v[0])**2 }

The function is a block. That is how Ruby passes a callable, and it takes &f for a Proc or lambda with no further surface — so there is no second spelling to document or test. A function of several variables receives one Array; dblquad's inner limits may be numbers or callables, as scipy allows.

Return shapes are scipy's, under one rule: one number → a Float; a tuple → an Array (quad is (value, abserr) and curve_fit is (popt, pcov), both destructuring as scipy's tuples do); an OptimizeResult or RootResults → a Hash with symbol keys, which full_output: true asks a root finder for.

Verified against integrals and roots that are known exactly

  • 16 definite integrals: x² over [0,1] = 1/3, sin over [0,π] = 2, 1/x over [1,e] = 1, e^-x over [0,∞) = 1, the Gaussian integral √π over (−∞,∞), 1/√x over [0,1] = 2 — an integrable singularity — the Gauss-Legendre exactness property (an n-point rule is exact to degree 2n−1), the two-point rule's nodes ±1/√3 with weight 1, the sampled rules on data where they are exact, and the unit square and a 1×2 rectangle.
  • five root finders × two functions: √2 and the Dottie number 0.7390851332151607, the unique solution of cos x = x. Each root is checked against the known value and by evaluating the function there, so a plausible wrong number fails twice. newton with fprime: answers 1.4142135623730951 exactly.
  • Rosenbrock through all five methods (BFGS, CG, Nelder-Mead, Powell, L-BFGS-B): the minimum is exactly (1, 1), so a method that merely descends somewhere fails.
  • curve_fit on data lying exactly on its model recovers the model's own parameters; approx_fprime on a linear objective recovers its constant gradient; bracket is checked by the property that its three points bracket a minimum, not against a fixed triple.
  • every keyword, in the "given" direction, with an observable effect: points: naming a kink, divmax: 1 producing the one reachable non-convergence (a RuntimeError naming DivMax, where SciPy warns), a callable inner limit distinguishing the triangle's 1/2 from the square's 1, bounds: binding minimize to 1 where the free minimum is 5, fprime2: (Halley) and x1: (the secant's second point), and method: as a Symbol.

What the coverage gate asked for, and what it found

The first push failed CI — not the tests, which were green, but the per-function coverage ratchet: five functions newly below 100%. Recording them in BELOW100 was the wrong fix, and working out why took three different answers:

  • untested — Truthy on both distribution classes (only a boolean context calls it) and factorial2 on an array: tests.
  • unreachable, and so deleted — an ErrFit arm when no Fit* function is bound; cycle_ier in quad's full record, which is only non-nil for the QAWF weight that is not bound; nd.LogSumExp's error, which reports a complex array when rbgo's NDArray is float64 only. Each was a branch nothing could reach, which BELOW100 would have preserved as a promise. #logsumexp now calls the root function with the slice it already holds — fewer layers and no dead arm.
  • looked unreachable and was not — #rvs's error arm. I validate negative sizes myself, so I was about to call it unreachable; measuring instead, Stats.norm.rvs(2 ** 40, 2 ** 40) answers "shape … is too big", refused by go-ndarray at validation without allocating. It has a test.

Separately, twenty-one inline if err != nil { raiseNumerics(err) } arms are now one raiseIf, which the refusals that are reachable cover — twenty-one half-covered functions became one covered branch. The rewrite that did it matched its own helper and briefly made raiseIf call raiseIf: infinite recursion that compiled and that go build accepted without a word. Caught by counting both sides of the replacement, and the helper is now shaped so the pattern cannot match it.

All four files are at 100% of statements with covblocks, which names the blocks go tool cover -func rounds away. That figure is from a plain go test -coverprofile run: the CI-equivalent measurement (-race, -coverpkg ./internal/...) was started and killed — this machine was carrying 30 concurrent agent sessions and a load average of 70, and both the test binary and the ratchet died with signal: terminated. So the -race ratchet here is CI's word, not mine; the lane result on this PR is the measurement.

Dependency versions, and what resolved them

This branch originally pinned go-numerics/{integrate,optimize} at v0.1.0 on purpose: their v0.2.0 required go-ndarray v0.12.0, and Go resolves to the maximum of all requirements, so taking it would have moved go-ndarray under NDArray, FFT, Special and Stats at once. That is a dependency decision with its own witness, not a side effect of a binding, so it was filed as #822 rather than smuggled in — with the evidence that rbgo's NDArray.zeros(0) expectation of array([]) is wrong on either version, since go-ndarray v0.12.0 ports NumPy 2.5.3's arrayprint.py and its expectations are copied verbatim from NumPy's own test suite, where np.zeros(0) is array([], dtype=float64).

#821 then landed exactly that, and #822 is closed. So the reason to hold back is gone, and this branch is rebased onto it:

go-ndarray/ndarray v0.12.0 (from main)
go-numerics/{integrate,optimize} v0.2.0
go-numerics/special, go-statistics/stats v0.1.0 (their latest)

The two v0.2.0 releases claim results unchanged to the last bit. That claim is checked here, not taken: the 113 closed forms, the 42 identities and every keyword test pass unchanged on the new pair, and nothing in the four bindings' tests depends on an array repr, so v0.12.0's printing rework touches none of them.

credscan 0 over 37 directories walked; govulncheck 0 reachable across ./.... The gitsafe and agentsync binaries and both global hooks were checked at v0.10.0/v0.5.0, matching their repositories.


What the rbgo_core lane said, and the measurement it forced

The core wasm build grew past its bound — 27,479,576 bytes against 27,000,000. The gate was right to stop it, and raising the number without saying what bought the growth would have turned a budget into a formality. So the cost is attributed, and the two obvious ways to attribute it disagree, which turned out to be the useful part:

one module alone added in turn down the branch
Special +591,000 +591,000
Stats +2,685,365 +2,438,092
Integrate +267,470 +266,797
Optimize +1,020,511 +987,989
+4,564,346 +4,283,878

Two rows of that table are the model confirming itself:

  • Special is identical in both columns — which is what "whoever goes first pays for the shared code" predicts for whoever goes first.
  • isolated overcounts the real +4,284,384 by 279,962, and that residual is the code the four share (go-ndarray, the special/nd layer, the argument helpers) — a figure neither method reports on its own. The same pair of measurements on go-ndarray v0.11.0 gave 279,063: 899 bytes apart across a dependency upgrade, so the residual is a property of the code and not of one run.
  • cumulative undercounts all but the first, and its sum lands 506 bytes from the real total.

Neither is "the" per-module cost. What both agree on is the ranking and the total, which is what the comment in ci.yml now says to read: Stats ~59% (36 distributions, 27,599 lines of Go), Optimize ~22%, Special ~13%, Integrate ~6% — against ~1,500 lines of binding.

An earlier revision of that comment, which I wrote, presented the cumulative column as "each module added to origin/main alone". It was not, and a reviewing session that measured Special in isolation could only report the two as 7% apart on an unstable baseline. They were not noisy — they were two different quantities.

One more thing that comparison turned up: the build is not byte-reproducible across worktrees. The same commit gave 23,173,500 in one and 23,172,890 in another, 610 bytes apart. Only same-worktree comparisons mean anything, so every figure above comes from one.

The bound is 29,000,000, ~1.5 MB above the measurement. It deliberately does not restore the original proportional headroom (3.8 MB on 23.1 MB): this number only has to notice growth nobody intended, and a gem coming back is caught by the module-path check in the same step, by name and exactly. That check was run against this branch and passes — 47 modules linked, none on the list, with the six numeric ones present as intended. CI's rbgo_core lane is green on the rebased head.

A concurrent session had implemented the other answer — Stats behind !rbgo_core, joining Image outside the core build, landing at 24,987,852 with no new bound — and dropped it, because it would cost the browser build the single most-used part of scipy (stats.norm leads the notebook census). That trade is recorded in the branch history rather than left implicit.

README's wasm table was re-measured in full rather than patched in one cell, because both columns moved, and it is dated:

cmd/wasm (js/wasm) default rbgo_core
unstripped 102.6 → 107.4 MB 23.7 → 28.2 MB
stripped 99.9 → 104.5 MB 23.1 → 27.5 MB
stripped, gzip -9 22.3 → 22.2 MB 5.7 → 6.4 MB
Go modules 253 → 256 44 → 47, no gem

core.go's statement of what core is names the four modules too; it listed the numeric stack as "NDArray and FFT", which was the whole of it when written.

Documentation

docs#27 adds a section per module to the site's scientific-stack page — and a tools/docexample guard that runs every Ruby example on the site and checks it against the answer written beside it, which nothing did before. It reports 11 examples / 25 claimed answers / 0 disagreements against an rbgo built from this branch, and correctly refuses the four new sections against today's main, naming each missing module. It merges after this PR, which is why.

@tannevaled tannevaled changed the title Special: bind go-numerics/special, scipy.special's functions as cgo-free ufuncs Special + Stats: bind go-numerics/special and go-statistics/stats as cgo-free scipy-style modules Oct 12, 2026
@tannevaled tannevaled changed the title Special + Stats: bind go-numerics/special and go-statistics/stats as cgo-free scipy-style modules Special, Stats, Integrate, Optimize: go-numerics and go-statistics as cgo-free scipy-style modules Oct 17, 2026
tannevaled and others added 8 commits October 18, 2026 08:47
Ruby gets scipy.special's function set with no SciPy and no cgo: 67 of the
module's 67 exported functions, under scipy's own names.

The names are taken from the correspondence the Go module MEASURES itself
against -- bench/scipy_server.py and testdata/gen/fam_*.py pair each Go
function with the scipy function it is compared to in ULPs -- rather than
transliterated from the Go name. That is why Log1pExp is #softplus and not
#log1pexp, which is what guessing would have produced, and why LogGamma is
#gammaln. The two functions with no scipy counterpart keep the Go name:
#log_factorial and #loggamma_sign.

Each binding is a ufunc, as scipy's is. The module ships both a float64 form
and an element-wise form over *ndarray.Array -- in two packages, since Go
cannot hold two functions of one name -- and rbgo already binds go-ndarray at
the same v0.11.0. So one Ruby name serves both: a Float takes the scalar path,
and as soon as one argument is an NDArray every argument goes through
go-ndarray, a Float becoming a 0-d array, which broadcasts against any shape.
Special.gamma(5.0) answers a Float, Special.gamma(arr) an NDArray of the same
shape, and Special.xlogy(2, arr) broadcasts.

Nothing here reimplements the module: the element-wise path calls its own nd
package, which parallelises and broadcasts already, and argument checking
reuses NDArray's helpers (ndArgc, ndFloat, ndInt, mustArray, ndRaise).

Verification is against values that are mathematical facts, not against the Go
functions being bound -- comparing a binding to the code it calls proves only
that the call happened. 56 closed forms (gamma(5)=24, zeta(2)=pi^2/6,
digamma(1)=-euler_gamma, ellipk(0)=pi/2, xlogy(0,0)=0, ...), the five
non-finite answers scipy gives outside a domain, jn_zeros against the
tabulated zeros of J0 AND against j0 vanishing there, and the property that
the scalar and element-wise paths agree -- the one part of this that is not the
module's own code -- at every arity. 98 subtests.

Three negative controls, each checked to have really changed the file: binding
#softplus to Logit, pointing #gamma's array path at LogGamma, and dropping an
arity check all turn the suite red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tions

Ruby gets scipy.stats's univariate distributions with no SciPy and no cgo: all
32 the Go module has Continuous or Discrete constructors for, under scipy's own
names, with scipy's argument order and scipy's method set.

    n = Stats.norm(0.0, 1.0)
    n.ppf(0.975)                  # => 1.9599639845400538
    Stats.poisson(3.0).pmf(2)     # => 0.22404180765538775
    Stats.t(10.0).cdf(arr)        # element-wise over an NDArray

As with Special, the names are the ones the module MEASURES itself against:
each distribution's doc comment states its scipy.stats equivalent and the
fixtures are generated from it. So StudentT is Stats.t, ChiSquared is
Stats.chi2, DiscreteUniform is Stats.randint, Weibull is Stats.weibull_min,
Gumbel is Stats.gumbel_r and NoncentralChiSquared is Stats.ncx2 -- none of
which transliteration would have produced.

The method set is not invented either. The module's Continuous and Discrete
interfaces name their scipy counterpart method by method, and differ only in
the density, so #pdf/#logpdf and #pmf/#logpmf are defined per class and the
other seventeen methods once, for both, through a Go interface holding exactly
what the two share. Every per-point method is a ufunc, over a Float or an
NDArray, through the module's own stats.Map.

#rvs answers a Float with no size and an NDArray with one, as scipy's does.
Without seed: it draws from one generator per VM that advances across calls --
scipy's module-level random state -- and seed: asks for a fresh reproducible
stream.

A parameter outside its domain is refused at CONSTRUCTION, with the module's
own message, as an ArgumentError where SciPy raises ValueError.

Verification is against mathematical facts, not the code being bound: 57 closed
forms covering all 32 distributions -- each constructor is written out
separately in scipy's order, so a table of known moments is what catches a
swapped parameter. norm.ppf(0.975)=1.96, norm.entropy=(1/2)log(2 pi e),
poisson.var=mean, binom(10,.5).pmf(5)=252/1024, zipf(2).pmf(1)=1/zeta(2),
randint(0,10).var=8.25, gumbel_r.mean=euler_gamma, rayleigh.mean=sqrt(pi/2),
f(5,10).mean=1.25, ncx2(2,3).mean=5, ... Plus 42 identities checked across four
continuous and two discrete laws (sf=1-cdf, ppf inverts cdf, std^2=var,
interval covers its confidence), loc/scale behaving as scipy says, the ufunc
property at every method, the method SET of each class, a 20000-draw check that
#rvs really draws from the law asked for (within 6 standard errors), and the
parameter and arity errors.

A README guard is added with them: internal/vm/{special,stats}.go register
their Ruby names in string literals and the README lists them one by one, so
the two sets must be equal in both directions. Writing it found two defects in
the README text committed minutes earlier -- the Special entry claimed all 67
functions while omitting log_factorial and loggamma_sign from its list, and it
backticked log1pexp and log_gamma as counter-examples, which reads as a
promise. Both fixed; the counter-examples are now italic, because a name that
deliberately does not exist is not code.

95 Stats subtests, 193 with Special. Seven negative controls, each checked to
have really changed the file: swapping loc and scale, pointing Stats.t at
ChiSquared, wiring #pdf to CDF, dropping a shared method, ignoring seed:,
drawing from the wrong law, and skipping parameter validation all turn the
suite red. The README guard was broken in all three of its directions too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ys checked

Two gaps in the tests just added, both found by measuring rather than reading.

A census over the live VM -- every method of every numeric module called with
zero arguments, classified by its message -- says Special defines 66 methods,
not the 67 the module exports. The two counts differ in one place and it is
scipy's: the Riemann and Hurwitz zeta are ONE scipy name with an optional
second argument, so Zeta and HurwitzZeta meet in #zeta. Nothing asserted that,
so a function added to the module and forgotten here, or a name quietly
dropped, would have gone unnoticed. TestSpecialCoversEveryFunction pins the 66
names and the reason there are 66.

The same census found a defect in the modules NEXT DOOR, filed as #820: all 17
of FFT's methods and 14 of Image's index args[0] with no arity check, so
FFT.fft() answers "ArgumentError: fft: runtime error: index out of range [0]
with length 0" -- a Go fault converted into a Ruby error, where MRI says wrong
number of arguments. NDArray is clean because it uses ndArgc, which is what
Special and Stats use.

So the census becomes the test, for these two modules. It is better than a list
because a method added later is covered without being named, and it asserts the
number of methods WALKED as well as the empty result, so a census that reads
nothing fails rather than passing -- 66 for Special, 70 for Stats (32
distributions and 19 methods on each of the two classes).

Verified to bite: removing the arity check from Special's one-argument helper
makes it name the exact 33 methods that helper registers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ll four

The request named go-numerics, which is THREE modules. Special covered one;
these are the other two, and with them the four symbols the notebook census
ranks highest inside scipy are bound: integrate.quad, optimize.minimize,
minimize_scalar and brentq.

    Integrate.quad(0.0, 1.0) { |x| x * x }               # => [0.333..., 3.7e-15]
    Integrate.quad(0.0, Float::INFINITY) { |x| Math.exp(-x) }
    Optimize.brentq(0.0, 2.0) { |x| x * x - 2 }          # => 1.4142135623730951
    Optimize.minimize([-1.2, 1.0]) { |v| 100*(v[1]-v[0]**2)**2 + (1-v[0])**2 }

The function is a BLOCK, which is how Ruby passes a callable and which takes
&f for a Proc or lambda with no further surface -- so there is no second
spelling to document or test. Return shapes are scipy's under one rule: one
number is a Float, a tuple an Array (quad is (value, abserr), curve_fit is
(popt, pcov)), and an OptimizeResult or RootResults a Hash with symbol keys,
which full_output: true asks a root finder for.

Verified against integrals and roots known exactly, not against the Go code:
16 definite integrals including the Gaussian integral over the whole line, an
integrable singularity, and the Gauss-Legendre exactness property with the
two-point rule's +/-1/sqrt(3) nodes; five root finders on sqrt(2) and the
Dottie number, each checked against the known value AND by evaluating the
function at the answer; Rosenbrock through all five minimisers, whose minimum
is exactly (1,1) so descending somewhere is not enough; curve_fit recovering
its own model's parameters from exact data. Every keyword is tested in the
"given" direction with an OBSERVABLE effect -- points: naming a kink,
divmax: 1 producing the one reachable non-convergence, a callable inner limit
distinguishing the triangle's 1/2 from the square's 1, bounds: binding
minimize to 1 where the free minimum is 5.

The first push failed CI on the per-function coverage ratchet, not on a test.
Recording the five functions in BELOW100 was the wrong fix, and the right one
needed three different answers:

  - untested: Truthy on both distribution classes (only a boolean context
    calls it) and factorial2 on an array -> tests.
  - unreachable, so DELETED: an ErrFit arm with no Fit* function bound;
    cycle_ier, which is only non-nil for the QAWF weight that is not bound;
    nd.LogSumExp's error, which reports a COMPLEX array where rbgo's NDArray
    is float64 only. Each was a branch nothing could reach, and BELOW100 would
    have preserved it as a promise. #logsumexp now calls the root function
    with the slice it already holds: fewer layers and no dead arm.
  - looked unreachable and was NOT: #rvs's error arm. I validate negative
    sizes myself, so I was about to call it unreachable; measuring instead,
    Stats.norm.rvs(2 ** 40, 2 ** 40) answers "shape is too big", refused by
    go-ndarray at validation without allocating. It has a test.

Twenty-one inline `if err != nil { raiseNumerics(err) }` arms are now one
raiseIf, which the refusals that ARE reachable cover -- twenty-one
half-covered functions became one covered branch. The rewrite that did it
matched its own helper and briefly made raiseIf call raiseIf: infinite
recursion that compiled and that go build accepted without a word. Caught by
counting both sides of the replacement; the helper is now shaped so the
pattern cannot match it.

All four files are at 100% of statements, located with covblocks, which names
the blocks go tool cover -func rounds away.

The README guard now covers all four modules, with each entry's non-method
words named beside their reason (scipy's loc/scale, the method: values, the
block-pass idiom) rather than a shared list that would also excuse a real
omission. It was broken in three directions again and went red for each.

go-numerics/integrate and /optimize stay at v0.1.0 deliberately: v0.2.0 is
performance work whose own notes say results are unchanged to the last bit,
AND a bump to go-ndarray v0.12.0, which Go would then apply to the whole
numeric stack. Filed separately with the measurement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… absorbed

The rbgo_core lane failed on its size bound: the core wasm module is 27 393 739
bytes against a 27 000 000 bound. The four scipy-style modules cost 4.2 MB
between them, and the bound had 3.8 MB of headroom.

A concurrent session reached this first and did the better experiment -- each
module added to origin/main ALONE rather than cumulatively down a branch, which
attributes the cost instead of ordering it -- and raised the bound to 29 000 000
with the table in ci.yml. Its argument is why that is right rather than
indulgent: the size bound exists to notice growth NOBODY INTENDED, while a gem
coming back is caught by the module-path check two lines below, by name and
exactly. Growth that is intended, measured and attributed is what should move a
bound; growth that is none of those is what the bound is for.

I had implemented the other answer -- Stats behind `!rbgo_core`, joining Image
outside the core build, which lands at 24 987 852 and needs no new bound. It is
dropped here. It cost the browser build the single most-used part of scipy
(stats.norm leads the notebook census), and it was the conservative choice
rather than the argued one.

What I did add is the reconciliation, because two independent measurements of
the same thing did not agree to the byte and a reader should not have to find
that out. "Special added to main alone" came out at +590 200 for me against
+631 646 there -- 7% apart -- from baselines that themselves differ by 411
bytes between two worktrees of the same commit. So ci.yml now says to read the
SHARES as measured and the exact bytes as one run's: Stats over half, Special
under a fifth, the four together ~4.2 MB.

Verified independently, every figure in the README's core table, to the digit:
28 081 523 unstripped (28.1 MB), 27 393 739 stripped (27.4 MB), 6 418 318
gzip -9 (6.4 MB), 47 modules.

Also checked, and NOT a defect: the core build links 28 go-ruby-* modules while
the README says "no gem". All 28 are standard-library bindings -- the rows the
capability table calls "preloaded by MRI" and "small standard libraries" --
except go-ruby-timecop, which vm.go imports for the VM's own clock rather than
as a registered gem; gems.go, which registers the Timecop module, is tagged out
as it should be.

Builds, vets and tests green under both tags.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…is attributed

The core build grew past its bound: 27_393_387 bytes against 27_000_000. The
gate was right to stop it, and raising the number without saying what bought
the growth would have turned a budget into a formality.

So the cost is measured, each module added to origin/main alone (js/wasm,
-s -w, 2026-10-17):

    Special      +631_646   15.0%   go-numerics/special
    Stats      +2_416_448   57.3%   go-statistics/stats
    Integrate    +238_387    5.6%   go-numerics/integrate
    Optimize     +933_406   22.1%   go-numerics/optimize
                ---------
               +4_219_887           23_173_500 -> 27_393_387

Stats is more than half of it, which is what 36 distributions and 27_599 lines
of Go cost; the four bindings themselves are ~1_500 lines. The attribution sits
in ci.yml beside the bound it moved, so the next person reads why rather than
guessing.

The new bound is 29_000_000 -- about 1.6 MB above the measurement. It does not
restore the original proportional headroom (3.8 MB on 23.1 MB) on purpose: this
number only has to notice growth nobody intended, and a gem coming back is
caught by the module-path check in the same step, by name and exactly. That
check was run against this branch and passes: 47 modules linked, none on the
list, with the six numeric ones (go-ndarray, go-fft, go-numerics/{special,
integrate,optimize}, go-statistics/stats) present as intended.

The first attribution attempt was wrong and said so plausibly: a perl
`s/.../ if !$d++` applies to the first LINE of a file, not the first match, so
no register call was inserted and four builds reported deltas of 37 KB, 6 KB
and 60 bytes while the total sat 4.2 MB below the branch. Redone with the
insertion verified at each step, the figures close to within 344 bytes of the
branch's own build.

README's wasm table is re-measured in full rather than patched in one cell --
both columns moved, and the default build grew by the same 4.2 MB:

    unstripped        102.6 -> 107.3 MB   |  23.7 -> 28.1 MB
    stripped           99.9 -> 104.5 MB   |  23.1 -> 27.4 MB
    stripped, gzip -9  22.3 ->  22.2 MB   |   5.7 ->  6.4 MB
    Go modules           253 -> 256       |    44 -> 47, no gem

and the table is now dated, with a line pointing at the attribution.

core.go's statement of what core IS names the four modules too: it listed the
numeric stack as "NDArray and FFT", which was the whole of it when written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… is the shared code

a20114f's comment, which I wrote, said the per-module figures were "each module
added to origin/main alone". They were not: the loop that produced them kept the
previous module in place, so each delta is the MARGINAL cost in that order. The
peer who reviewed it measured Special in isolation, got +590_200 against my
+631_646, and could only report the two as 7% apart from an unstable baseline.

They were not noisy. They are two different quantities, and measuring both
settles it -- isolated, one worktree, one baseline, 2026-10-17:

                     one module      added in turn
                     alone           down the branch
    Special            +590_255          +631_646
    Stats            +2_685_070        +2_416_448
    Integrate          +258_839          +238_387
    Optimize           +965_130          +933_406
                     ----------        ----------
                     +4_499_294        +4_219_887

+590_255 reproduces the peer's +590_200 to 55 bytes, so the 7% was the method,
not the measurement.

And neither column is "the" per-module cost, which is the part worth writing
down:

  - ISOLATED overcounts. The four share go-ndarray, the special/nd layer and the
    argument helpers, so each pays for the same code. The sum exceeds the real
    +4_220_231 by 279_063 -- and that residual IS the shared part, which is a
    figure neither method reports on its own.
  - CUMULATIVE undercounts all but the first: whoever goes first pays for the
    shared code and the rest ride free. It depends on the order.

What both agree on is the ranking and the total, so the comment now says to read
those: Stats ~60%, Optimize ~21%, Special ~13%, Integrate ~6%, together ~4.2 MB.

One more thing the comparison turned up: the build is NOT byte-reproducible
across worktrees. The same commit gave 23_173_500 in one and 23_172_890 in
another, 610 bytes apart, which is why the peer's baseline and mine differed and
why every figure above comes from a single worktree. A cross-worktree delta of a
few hundred bytes means nothing.

The bound stays at 29_000_000 and the module-path check is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…asure everything

#821 landed the upgrade #822 asked for: go-ndarray v0.12.0, whose Repr prints
what NumPy prints, with rbgo's `array([])` expectation corrected to
`array([], dtype=float64)`. That removes the only reason this branch pinned
go-numerics/{integrate,optimize} at v0.1.0 -- their v0.2.0 required v0.12.0 and
would have dragged the whole numeric stack with them. They are now at v0.2.0,
two releases of performance work whose own notes claim results unchanged to the
last bit.

That claim is checked here rather than taken: the 113 closed forms, 42
identities and every keyword test pass unchanged on the new pair. Nothing in the
four bindings' tests depends on an array repr, so v0.12.0's printing rework
touches none of them.

Every size figure is re-measured, because all of them were taken on v0.11.0 and
v0.1.0:

                       was (v0.11.0)   now (v0.12.0 + v0.2.0)
    baseline (main)       23_172_890            23_195_192
    this branch           27_393_739            27_479_576
    delta                 +4_220_231            +4_284_384
    headroom to bound      1_606_261             1_520_424

The attribution too, both columns, same worktree:

                     one module ALONE      added IN TURN
    Special                 +591_000           +591_000
    Stats                 +2_685_365         +2_438_092
    Integrate               +267_470           +266_797
    Optimize              +1_020_511           +987_989
                          ----------         ----------
                          +4_564_346         +4_283_878

Two things in that table are the model confirming itself. Special is IDENTICAL
in both columns, which is what "whoever goes first pays for the shared code"
predicts for whoever goes first. And the isolated sum overcounts the real
+4_284_384 by 279_962, against 279_063 for the same pair of measurements on
v0.11.0 -- 899 bytes apart across a dependency upgrade, so the shared-code
residual is a property of the code and not of one run.

The bound stays at 29_000_000: the growth moved by 64_153 bytes and the
headroom is 1.5 MB. README's wasm table is re-measured in full and re-dated
(107.4 / 28.2 unstripped, 104.5 / 27.5 stripped, 22.2 / 6.4 gzipped, 256 / 47
modules).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The module comment and the README both said the Ruby names "come from the
correspondence the Go module measures itself against". Auditing that sentence
rather than repeating it: 63 of special's 67 functions carry such a pairing in
bench/scipy_server.py or testdata/gen/fam_*.py, and FOUR do not. For those the
name is my reading of scipy, which is a weaker claim and is now made as one.

    GammaSign      -> gammasgn        scipy has it; the module pairs nothing
    Kn             -> kn              scipy has it; unpaired here
    In             -> in              scipy has NO `in`
    LogGammaSign   -> loggamma_sign   no scipy counterpart at all

LogFactorial is a fifth of the kind, and the module's own fixture says so in
words: "log n! (SciPy: gammaln(n+1))".

So Special.in is a name I INVENTED that merely looks like it came from a spec.
That is the failure worth naming: a plausible name sitting inside an otherwise
measured table inherits the table's authority. scipy cannot have an `in` --
it is a Python keyword -- and iv(0, x) is the faithful spelling.

The three relationships those names stand on are asserted now instead of
asserted in prose: in(n, x) == iv(n.to_f, x) EXACTLY, the same for kn against
kv, and gammasgn against the sign of gamma itself on the intervals where it
alternates. Exact rather than approximate, because they are the same Go call
and a tolerance would hide a wrong conversion.

The audit's first answer was wrong, by my own instrument, and it is worth
recording how: I extracted the paired fixture names with a pattern anchored on
exactly four spaces of indentation. fam_bessel.py indents its table by eight,
so fourteen Bessel functions looked unpaired and the count came out 49/18
instead of 63/4. The Go program doing the join was correct and faithfully
reported my filter's view. The answer was implausible against a grep I had
already read, and `^[[:space:]]+` would have cost nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant