Skip to content

Identify multimap entries by (key, value) and drop the discriminator - #80

Merged
pathscale merged 2 commits into
masterfrom
ord-multipair
Sep 1, 2026
Merged

pathscale merged 2 commits into
masterfrom
ord-multipair

Conversation

@pathscale

Copy link
Copy Markdown
Owner

Companion to pathscale/WorkTablesIndex#13, which removes RandomMultiPair. Merge and publish that first; see the release cascade below.

Why

RandomMultiPair::insert scanned every entry sharing the key, because the value did not participate in the order and a stored entry could not be located by it. O(n) per insert, quadratic to fill one key.

A non-unique index that puts a whole generation under one key hits this directly. AgentCode measured, controlled A/B on one machine, same fixture, identical application code:

beta.13 beta.14
one-file update 698 ms 15.10 s
put_symbols 138.5 ms 4.76 s
put_dependencies 112.9 ms 5.17 s
ensure_text_index 263.6 ms 4.96 s
marginal cost per row 9.04 us 330 us

arctic-wt 0.1.6 resolves under both and is not implicated.

What changed

MultiPair is now OrdMultiPair, identified and ordered by its (key, value) pair, so an entry is located by binary search and duplicate replacement falls out of the order.

The discriminator plumbing existed only to serve the random representation and is gone:

  • MultiPairRecreate no longer takes a discriminator. A snapshot stores key and value, so reconstruction has nothing left to synthesise.
  • reconstruct_multi_index_nodes no longer invents one per entry, which also removes the node-maxima collision it could produce on a damaged file.
  • Pages are ordered by their minimum rather than their node id. A node id is a page maximum, and ordering by maximum puts a page that merely ends late ahead of one that starts earlier.

Two fixtures were rebuilt, and why

mixed_boundary_page_with_adversarial_link_order and multi_node_straddle_chain were built around a state only the random representation could produce: one page whose range sits inside another's. A (key, value) index partitions an ordered space, so a page maximum and its minimum agree on the ordering and that state is unrepresentable.

Both are rebuilt as valid partitions and keep the property that still has meaning: reconstruction does not depend on the order pages arrive in.

Format change

An index written by beta.14 or earlier must be reindexed, not loaded. Such a file orders entries within a key by discriminator, which is not (key, value) order. The checked-in .wt.idx goldens are regenerated in this commit, which is the visible evidence of it.

Release cascade

data_bucket pins WorkTablesIndex =0.0.8, so publishing 0.0.9 needs a data_bucket release before this can build against the published crate:

  1. WorkTablesIndex 0.0.9 (Remove RandomMultiPair and make (key, value) the only entry identity WorkTablesIndex#13)
  2. data_bucket, accepting 0.0.9
  3. WorkTable beta.15 (this)

Tests

266 lib + 586 integration + 2 doc, all passing, verified against the index fix applied at 0.0.8 so data_bucket's pin still resolves and this change is isolated. Goldens are stable across repeated runs.

Related: size and eviction asks for beta.15 in #78.

meh added 2 commits September 1, 2026 14:12
WorkTablesIndex 0.0.9 removes RandomMultiPair, whose insert scanned every entry sharing
the key because the value did not participate in the order. On a table whose non-unique
index puts a whole generation under one key that is O(n) per insert: AgentCode measured a
one-file update at 698 ms on beta.13 and 15.1 s on beta.14, with marginal cost per row
going from 9.04 us to 330 us.

MultiPair is now OrdMultiPair, identified and ordered by its (key, value) pair, so an
entry is located by binary search and duplicate replacement falls out of the order.

The discriminator plumbing that existed only to serve the random representation is gone:

- MultiPairRecreate no longer takes a discriminator. A snapshot stores key and value, so
  reconstruction has nothing left to synthesise.
- reconstruct_multi_index_nodes no longer invents a discriminator per entry, which also
  removes the node-maxima collision it could produce on a damaged file.
- Pages are ordered by their minimum rather than their node id. A node id is a page's
  maximum, and ordering by maximum puts a page that merely ends late ahead of one that
  starts earlier.

Two reconstruction fixtures were built around states only the random representation could
produce, where one page's range sat inside another's. A (key, value) index partitions an
ordered space, so a page's maximum and its minimum agree on the ordering and that state is
unrepresentable. Both are rebuilt as valid partitions and keep what still has meaning:
that reconstruction does not depend on the order the pages arrive in.

This is a persisted format change. An index written by beta.14 or earlier orders entries
within a key by discriminator, which is not (key, value) order, so such a file must be
reindexed rather than loaded.
data_bucket is pinned exactly here too, so it has to move with the index it carries.

The reconstruction fixture rewrite also dropped the #[test] attribute off
mixed_boundary_page_with_adversarial_link_order, which left it compiling as an ordinary
unused function and silently not running. Clippy caught it; the suite is 267 rather than
266 with it back.
@pathscale
pathscale merged commit e4dcfdf into master Sep 1, 2026
6 checks passed
@pathscale
pathscale deleted the ord-multipair branch September 1, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant