Identify multimap entries by (key, value) and drop the discriminator - #80
Merged
Merged
Conversation
added 2 commits
September 1, 2026 14:12
WorkTablesIndex 0.0.9 removes RandomMultiPair, whose insert scanned every entry sharing the key because the value did not participate in the order. On a table whose non-unique index puts a whole generation under one key that is O(n) per insert: AgentCode measured a one-file update at 698 ms on beta.13 and 15.1 s on beta.14, with marginal cost per row going from 9.04 us to 330 us. MultiPair is now OrdMultiPair, identified and ordered by its (key, value) pair, so an entry is located by binary search and duplicate replacement falls out of the order. The discriminator plumbing that existed only to serve the random representation is gone: - MultiPairRecreate no longer takes a discriminator. A snapshot stores key and value, so reconstruction has nothing left to synthesise. - reconstruct_multi_index_nodes no longer invents a discriminator per entry, which also removes the node-maxima collision it could produce on a damaged file. - Pages are ordered by their minimum rather than their node id. A node id is a page's maximum, and ordering by maximum puts a page that merely ends late ahead of one that starts earlier. Two reconstruction fixtures were built around states only the random representation could produce, where one page's range sat inside another's. A (key, value) index partitions an ordered space, so a page's maximum and its minimum agree on the ordering and that state is unrepresentable. Both are rebuilt as valid partitions and keep what still has meaning: that reconstruction does not depend on the order the pages arrive in. This is a persisted format change. An index written by beta.14 or earlier orders entries within a key by discriminator, which is not (key, value) order, so such a file must be reindexed rather than loaded.
data_bucket is pinned exactly here too, so it has to move with the index it carries. The reconstruction fixture rewrite also dropped the #[test] attribute off mixed_boundary_page_with_adversarial_link_order, which left it compiling as an ordinary unused function and silently not running. Clippy caught it; the suite is 267 rather than 266 with it back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to pathscale/WorkTablesIndex#13, which removes
RandomMultiPair. Merge and publish that first; see the release cascade below.Why
RandomMultiPair::insertscanned every entry sharing the key, because the value did not participate in the order and a stored entry could not be located by it. O(n) per insert, quadratic to fill one key.A non-unique index that puts a whole generation under one key hits this directly. AgentCode measured, controlled A/B on one machine, same fixture, identical application code:
put_symbolsput_dependenciesensure_text_indexarctic-wt 0.1.6resolves under both and is not implicated.What changed
MultiPairis nowOrdMultiPair, identified and ordered by its(key, value)pair, so an entry is located by binary search and duplicate replacement falls out of the order.The discriminator plumbing existed only to serve the random representation and is gone:
MultiPairRecreateno longer takes a discriminator. A snapshot stores key and value, so reconstruction has nothing left to synthesise.reconstruct_multi_index_nodesno longer invents one per entry, which also removes the node-maxima collision it could produce on a damaged file.Two fixtures were rebuilt, and why
mixed_boundary_page_with_adversarial_link_orderandmulti_node_straddle_chainwere built around a state only the random representation could produce: one page whose range sits inside another's. A(key, value)index partitions an ordered space, so a page maximum and its minimum agree on the ordering and that state is unrepresentable.Both are rebuilt as valid partitions and keep the property that still has meaning: reconstruction does not depend on the order pages arrive in.
Format change
An index written by beta.14 or earlier must be reindexed, not loaded. Such a file orders entries within a key by discriminator, which is not
(key, value)order. The checked-in.wt.idxgoldens are regenerated in this commit, which is the visible evidence of it.Release cascade
data_bucketpinsWorkTablesIndex =0.0.8, so publishing 0.0.9 needs adata_bucketrelease before this can build against the published crate:Tests
266 lib + 586 integration + 2 doc, all passing, verified against the index fix applied at 0.0.8 so
data_bucket's pin still resolves and this change is isolated. Goldens are stable across repeated runs.Related: size and eviction asks for beta.15 in #78.