feat(table): decode the BucketVectorSearchSplit byte form - #746
Merged
Conversation
JunRuiLee
marked this pull request as ready for review
August 26, 2026 04:19
JunRuiLee
marked this pull request as draft
August 26, 2026 04:20
JunRuiLee
force-pushed
the
pkvector-bucket-split-decode
branch
2 times, most recently
from
August 26, 2026 09:19
c09d03c to
a3dee82
Compare
JunRuiLee
force-pushed
the
pkvector-bucket-split-decode
branch
from
August 26, 2026 09:53
a3dee82 to
0506b0d
Compare
A primary-key vector search has to be planned per bucket: deciding which ANN segments are current needs the bucket's complete active-file set, which an ordinary table-scan split does not carry. Java plans those splits and serializes them in a form a reader outside the JVM can consume; add that reader, so the search can then run natively. The message nests two things this crate could not read yet. The `DataSplit` is written inline and already had a decoder. Each payload is a `BinaryRow` of Java's `IndexFileMeta.SCHEMA`, which needed two array readers: `array<int>` uses 4-byte element slots rather than the 8-byte ones the existing `bigint` and string readers assume, and the deletion vector ranges are an array of rows, addressed the way variable-length fields are. Nested rows themselves reuse the existing pattern of reading a field's bytes and viewing them at the arity its schema fixes. Row ranges are checked against the data file they name, and a payload without global index metadata is rejected: the Java constructor enforces that, so it is not a state a real split reaches, but these bytes are untrusted. A malformed nested structure is reported as invalid data rather than as an internal fault, for the same reason. The checks are exact wherever the writer's layout is exact: an element body in an array of rows starts where the previous one ended and the array ends where its last element does, an `array<int>` is its word-padded fixed region and nothing more, and a negative row count on a data file is rejected rather than taken as a reason to skip the bound it sets. Element counts are bounded against the buffer before they are multiplied, so a forged one cannot wrap the offset arithmetic that follows. Pinned against two messages generated by the Java writer, one of them carrying the deletion vector ranges and the absent optionals the other cannot reach.
JunRuiLee
force-pushed
the
pkvector-bucket-split-decode
branch
from
August 26, 2026 10:55
0506b0d to
bdf0a92
Compare
JunRuiLee
marked this pull request as ready for review
August 26, 2026 10:57
Open
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
apache/paimon#9386 gave
BucketVectorSearchSplita byte form so a reader outside the JVM can read it. This is that reader: a Java planner enumerates bucket splits -- the ANN current-segment decision needs the bucket's complete active-file set, which an ordinary table scan split does not carry -- and dispatches each to a worker running this crate.Decode only. The writer that produced the fixtures has merged, as
088d4880ff. Part of #514.A payload is a
BinaryRowofIndexFileMeta.SCHEMA. TheDataSplitalready had a decoder, and nested rows reuse whatBinaryTableStatsdoes insideDataFileMeta, but two element layouts were missing:array<int>(_EXTRA_FIELD_IDS): 4-byte slots, so neither the existingbigint/string readers nor the bound check they share applies._DELETIONS_VECTORS_RANGES): elements are addressed like variable-length fields, so the reader returns element slices for the caller to decode at its own arity.VERSIONpins the nested layout too, sinceIndexFileMetaSerializercarries no version. TheDataSplitis the exception: it carries its own.Java re-checks little on the way back in, so this validates what a reader of untrusted bytes has to: row ranges against the data file they name, global index and source metadata present, counts and lengths bounded by what can follow, no trailing bytes, and a malformed nested structure reported as invalid data rather than as an internal fault. Where the writer's layout is exact the checks are exact rather than permissive, so nothing is accepted that the writer could not have produced.
Tests
Two fixtures dumped from the Java writer plus unit tests for the array readers. The first carries non-empty rows and stats, read back as values. The second covers deletion vector ranges (including a null cardinality), a second payload, and all-optionals-absent; without it both new readers would go untouched by real bytes. The rest is hostile input.
Fixtures are copies and the Java side keeps none of its own, so a change there has to be carried over by hand. Both payloads of the second fixture share one
_SOURCE_META, so the test cannot prove one payload's was not read off the other.Notes
BucketVectorPayloadrather thanspec::IndexFileMeta, which models the manifest form: no external path, row count narrowed toi32. Aligning that type with the schema looks worthwhile but touches every construction site.DataSplitdecoder behaviour left alone, since it predates this and every split path reaches it: absenttotalBucketsbecomes1,isStreaming = trueand snapshot-1are rejected, and only versions 8 and 9 are read where Java reads 1 through 9.deserialize_binary_array_strhas the aliasing gap the new array-of-rows reader rejects. Pre-existing, with other callers, so it is left for its own change.