Skip to content

feat(table): decode the BucketVectorSearchSplit byte form - #746

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JunRuiLee:pkvector-bucket-split-decode
Aug 27, 2026
Merged

feat(table): decode the BucketVectorSearchSplit byte form#746
JingsongLi merged 1 commit into
apache:mainfrom
JunRuiLee:pkvector-bucket-split-decode

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Purpose

apache/paimon#9386 gave BucketVectorSearchSplit a byte form so a reader outside the JVM can read it. This is that reader: a Java planner enumerates bucket splits -- the ANN current-segment decision needs the bucket's complete active-file set, which an ordinary table scan split does not carry -- and dispatches each to a worker running this crate.

Decode only. The writer that produced the fixtures has merged, as 088d4880ff. Part of #514.

i64 magic = "PKVSPLIT"; i32 version = 1
DataSplit                             // DataSplit#serialize
i32 payloadCount; IndexFileMeta*      // IndexFileMetaSerializer#serializeList
i32 rangeFileCount; rangeFile*
  rangeFile = utf fileName, i32 rangeCount, (i64 from, i64 to)*

A payload is a BinaryRow of IndexFileMeta.SCHEMA. The DataSplit already had a decoder, and nested rows reuse what BinaryTableStats does inside DataFileMeta, but two element layouts were missing:

  • array<int> (_EXTRA_FIELD_IDS): 4-byte slots, so neither the existing bigint/string readers nor the bound check they share applies.
  • array of rows (_DELETIONS_VECTORS_RANGES): elements are addressed like variable-length fields, so the reader returns element slices for the caller to decode at its own arity.

VERSION pins the nested layout too, since IndexFileMetaSerializer carries no version. The DataSplit is the exception: it carries its own.

Java re-checks little on the way back in, so this validates what a reader of untrusted bytes has to: row ranges against the data file they name, global index and source metadata present, counts and lengths bounded by what can follow, no trailing bytes, and a malformed nested structure reported as invalid data rather than as an internal fault. Where the writer's layout is exact the checks are exact rather than permissive, so nothing is accepted that the writer could not have produced.

Tests

Two fixtures dumped from the Java writer plus unit tests for the array readers. The first carries non-empty rows and stats, read back as values. The second covers deletion vector ranges (including a null cardinality), a second payload, and all-optionals-absent; without it both new readers would go untouched by real bytes. The rest is hostile input.

Fixtures are copies and the Java side keeps none of its own, so a change there has to be carried over by hand. Both payloads of the second fixture share one _SOURCE_META, so the test cannot prove one payload's was not read off the other.

Notes

  • BucketVectorPayload rather than spec::IndexFileMeta, which models the manifest form: no external path, row count narrowed to i32. Aligning that type with the schema looks worthwhile but touches every construction site.
  • Existing DataSplit decoder behaviour left alone, since it predates this and every split path reaches it: absent totalBuckets becomes 1, isStreaming = true and snapshot -1 are rejected, and only versions 8 and 9 are read where Java reads 1 through 9.
  • deserialize_binary_array_str has the aliasing gap the new array-of-rows reader rejects. Pre-existing, with other callers, so it is left for its own change.

@JunRuiLee
JunRuiLee marked this pull request as ready for review August 26, 2026 04:19
@JunRuiLee
JunRuiLee marked this pull request as draft August 26, 2026 04:20
@JunRuiLee
JunRuiLee force-pushed the pkvector-bucket-split-decode branch 2 times, most recently from c09d03c to a3dee82 Compare August 26, 2026 09:19
@JunRuiLee JunRuiLee changed the title feat(table): decode the BucketVectorSearchSplit byte protocol feat(table): decode the BucketVectorSearchSplit byte form Aug 26, 2026
@JunRuiLee
JunRuiLee force-pushed the pkvector-bucket-split-decode branch from a3dee82 to 0506b0d Compare August 26, 2026 09:53
A primary-key vector search has to be planned per bucket: deciding which
ANN segments are current needs the bucket's complete active-file set,
which an ordinary table-scan split does not carry. Java plans those
splits and serializes them in a form a reader outside the JVM can
consume; add that reader, so the search can then run natively.

The message nests two things this crate could not read yet. The
`DataSplit` is written inline and already had a decoder. Each payload is
a `BinaryRow` of Java's `IndexFileMeta.SCHEMA`, which needed two array
readers: `array<int>` uses 4-byte element slots rather than the 8-byte
ones the existing `bigint` and string readers assume, and the deletion
vector ranges are an array of rows, addressed the way variable-length
fields are. Nested rows themselves reuse the existing pattern of reading
a field's bytes and viewing them at the arity its schema fixes.

Row ranges are checked against the data file they name, and a payload
without global index metadata is rejected: the Java constructor enforces
that, so it is not a state a real split reaches, but these bytes are
untrusted. A malformed nested structure is reported as invalid data
rather than as an internal fault, for the same reason.

The checks are exact wherever the writer's layout is exact: an element
body in an array of rows starts where the previous one ended and the array
ends where its last element does, an `array<int>` is its word-padded fixed
region and nothing more, and a negative row count on a data file is
rejected rather than taken as a reason to skip the bound it sets. Element
counts are bounded against the buffer before they are multiplied, so a
forged one cannot wrap the offset arithmetic that follows.

Pinned against two messages generated by the Java writer, one of them
carrying the deletion vector ranges and the absent optionals the other
cannot reach.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 3dddf12 into apache:main Aug 27, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants