Repository navigation
perf(set): one-pass Set algebra — close the ~10x MRI gap - #87
Merged
Merged
Conversation
… gap)
Set#|/&/-/^ ran 5-25x slower than MRI 4.0.5 because the binding kept a
third parallel store: the go-ruby-set library held canonical keys, while
the wrapper held a separate key->Ruby-value map plus a parallel order
slice, and every algebra op re-materialised the library result via
fromKeys with an extra Include lookup per element and rebuilt that third
store. So each op was 3-4 passes with 3 membership stores.
Now the library Set stores the *Ruby value itself* as its member (keyed
through setKey via the Hasher), so the library's own key->value map and
insertion order are the single source of truth. The wrapper is just
{ s *rbset.Set } -- no parallel vals/order, no fromKeys, no re-Include.
Algebra results come straight from the library (which fills them in one
pre-sized pass) and are wrapped without copying. each/map/select/reduce
and the external readers (Bag, the Psych/YAML emitter) iterate the
library through a single-pass EachPair/ToSlice helper.
Paired with go-ruby-set's single-pass pre-sized algebra
(v0.0.0-20260630204346-ab6a8a92a910), set algebra vs MRI 4.0.5 (best of
3, Apple M-class):
op before(x MRI) after(x MRI)
union n=100 17.1 2.88
intersection n=100 5.4 1.38
difference n=100 7.8 1.96
xor n=100 10.1 2.36
union n=10000 24.8 3.59
intersection n=10000 5.9 1.50
difference n=10000 8.2 2.18
xor n=10000 12.8 3.00
Semantics unchanged (MRI ordering preserved); the differential-vs-MRI
set oracle and the full internal/vm suite stay green. set.go is 100%
covered; gofmt/vet clean; all 6 64-bit arches build. CGO=0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Jul 1, 2026
* json: single-pass parse/generate via go-ruby-json streaming API Close the JSON.parse (~3.9x) / JSON.generate (~3.7x) gap vs MRI 4.0.5. The binding previously converted the whole document twice each way: parse built a go-ruby-json `any` tree that fromJSON then walked to build the rbgo object graph; generate built an intermediate json value tree that the library then walked. Both are now one pass over the new streaming API: - jsonParse drives json.ParseInto with an objBuilder that materialises the rbgo object graph directly (no intermediate tree), interning small integers (object.IntValue) and pre-sizing each Hash/Array to its parsed width (new object.NewHashCap). - jsonGenerate / jsonPrettyGenerate push the rbgo graph straight into the library's json.Encoder via an objSource, with no intermediate json tree. Output stays byte-identical to MRI 4.0.5 (verified end-to-end: key order, floats, escaping, unicode/surrogates, bignums, -0.0, duplicate keys, generate vs pretty_generate, symbolize_names). Re-pins go-ruby-json to the streaming-API release. Measured rbgo-vs-MRI 4.0.5 (M4 Max, best-of-8), ratio before -> after: small object parse 3.73x -> 2.67x generate 4.10x -> 2.50x 1k-array parse 6.10x -> 3.90x generate 4.85x -> 2.19x deep nested parse 4.80x -> 4.82x generate 4.24x -> 1.63x big strings parse 18.6x -> 5.0x generate 15.7x -> 4.0x 100% coverage on every file touched; gofmt/vet clean; CGO=0 builds on all six 64-bit arches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Merge main into perf/json-fastpath; re-pin go-ruby-json to merged main Brings in the set-algebra perf merge (#87) and re-pins go-ruby-json to its squash-merged main pseudo-version after lib PR #1 landed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Set#|/&/-/^ran ~10× slower than MRI 4.0.5 (5–25× across ops/sizes). Root cause: a redundant third parallel store. The go-ruby-set library held canonical keys; the rbgo wrapper held a separate key→Ruby-value map plus a parallelorderslice; and every algebra op re-materialised the library result throughfromKeyswith an extraIncludelookup per element and rebuilt that third store — 3–4 passes and 3 membership stores per op.Fix
main@ab6a8a92a910): algebra ops fill the result in a single pass into pre-sizedvals/order(noDup-then-Add, no Add membership re-probe). NewEachPairexposes ordered(key, member)in one pass.setKeyvia the Hasher), so the library's own key→value map + insertion order are the single source of truth. The wrapper drops its parallelvals/order(now just{ s *rbset.Set });fromKeysis gone; algebra results are wrapped without copying;each/map/select/reduceand the external readers (Bag, Psych/YAML emitter) iterate via a single-passEachPair/ToSlicehelper.Results — Set algebra vs MRI 4.0.5 (best of 3, Apple M-class)
The algebra-only micro gap (~7.5×) collapses to ~1.4–3.6×. Intersection/difference are at/near the 2× goal; the residual on union/xor is the inherent cost of Go
map[any]anywith boxed keys vs MRI's tuned Hash building a full-size result — the algebra itself is now single-pass and pre-sized.Constraints
internal/vmsuite green.set.go100% covered; gofmt/vet clean; all 6 64-bit arches build.v0.0.0-20260630204346-ab6a8a92a910(re-pinned viago get …@main && go mod tidy).🤖 Generated with Claude Code