feat(table): plan a PK-vector search from a decoded bucket split - #757
Draft
JunRuiLee wants to merge 4 commits into
Draft
feat(table): plan a PK-vector search from a decoded bucket split#757JunRuiLee wants to merge 4 commits into
JunRuiLee wants to merge 4 commits into
Conversation
Index files were always read from `<table>/index/<file>`, ignoring both the
`_EXTERNAL_PATH` recorded in the index manifest and the
`index-file-in-data-file-dir` table option. A table that keeps index files
beside its bucket's data files fails to read them, e.g. a primary-key vector
search reports
failed to open ANN index file
'<table>/index/index-<uuid>-0' for range reads
while the file actually lives in the bucket directory.
Decode `_EXTERNAL_PATH` from the index manifest (Java `IndexFileMeta` SCHEMA
field 5) and add it to the write schema so a rewritten manifest keeps it — it is
currently dropped silently — add the `index-file-in-data-file-dir` option, and
resolve every index file through one place (`table/index_file_path.rs`) with two
modes:
* global, always `<table>/index`: the data-evolution global index, and vector
and full-text search over it;
* bucket-local, the data-file directory when the option is set: primary-key
vector ANN segments, primary-key full-text archives, deletion vectors, and
the dynamic-bucket hash index.
Each mode mirrors the factory Java uses for that consumer:
`DataEvolutionGlobalIndexScanner` resolves through `globalIndexFileFactory`,
while `IndexFileHandler` resolves hash, deletion-vector and primary-key vector
files through `pathFactories.get(partition, bucket)`, which selects
`IndexInDataFileDirPathFactory` when the option is set. An explicit external path
wins over both layouts, as in `toPath(IndexFileMeta)`.
For the two index kinds this crate writes itself, deletion vectors and the hash
index, reads and writes move together, so a file written here is found again:
* the data-evolution writer resolves an existing deletion vector through the
same path when merging, and writes a new one where the reader will look;
* the dynamic-bucket assigner resolves per partition and bucket for both
restore and commit. `BucketAssigner::prepare_commit_index` no longer takes an
index directory — three of its four implementations ignored it, and the
fourth now derives the layout itself;
* `TableCommit::abort` deletes a newly written index file where it was written,
mirroring Java `FileStoreCommitImpl.abort`, which deletes through
`indexFileFactory(partition, bucket)`. Deleting is best-effort, so the old
fixed path leaked the file silently instead of failing.
A bucket directory comes from the split that references the file when a split is
at hand, and otherwise from the partition and bucket being committed. Both go
through one `spec::bucket_path`, mirroring Java `FileStorePathFactory.bucketPath`:
the layout is only correct while every producer and consumer of a bucket
directory agrees byte for byte, and nothing else enforces that.
The option is immutable, as in Java, where it is annotated `@Immutable` and
`SchemaManager.checkAlterTableOption` rejects altering it: it selects the
directory index files are written to, so flipping it on a populated table would
hide every index file already written.
`$physical_files_size` now counts an `index-` prefixed file in a bucket directory
as an index file rather than dropping it, matching Java `FileType.classify`,
which maps any `index-*` basename to `BUCKET_INDEX` regardless of directory.
Classification follows the file's physical form, not the current option value, so
a file stays recognizable after the setting it was written under changes.
The BTree reader cache is keyed by the resolved path so two entries sharing a
file name cannot reuse each other's reader.
`data-file.path-directory` remains unsupported, as it is throughout this crate:
bucket paths are rooted directly at the table for data files as much as for
index files, so honoring it belongs with data-file path handling rather than
here.
`plan_and_search_pk_candidates_batch` resolved the query parameters, read the index manifest into a plan, and searched that plan in one body, so a caller that already holds a plan could not reuse the search path. Split it into three pieces with no behavior change: - `resolve_pk_vector_search_params` — the query-level parameters and the pre-filter guard: everything resolvable from the schema, the options and the queries alone, before planning. - `search_pk_raw_candidates_batch_with_plan` — search a supplied plan and return each query's raw indexed and exact candidate lists. Plan-dependent concurrency (segment count, batch-index parallelism, range-read bound) is derived from the plan actually being searched, so a narrowed plan can never be searched under limits computed for a wider one. - `search_pk_candidates_batch_with_plan` — the raw layer plus the optional exact rerank of the approximate candidates and the merge into one best-first list per query. `plan_and_search_pk_candidates_batch` keeps its signature and becomes a wrapper over the three. The empty-plan short circuit moves into the raw layer, still ahead of backend resolution, so a table with no searchable data does not error on an unrecognized index type.
A `BucketVectorSearchSplit` already carries everything a search needs for one bucket: the payload files, the rows each data file allows, and the snapshot the whole plan is pinned to. Planning could only be driven the other way round, by reading this table's index manifest, so a search could not be run over splits an engine planned elsewhere. `PkVectorScan::plan_for_bucket_vector_splits` builds a plan from such splits instead. The splits are authoritative -- no manifest is read -- and only the partition conjuncts of the scan's filter are re-applied, since a caller may narrow the query further than the planner that produced the splits. Payload resolution, bucket grouping, current-segment selection and exact-fallback eligibility all reuse the manifest route's `plan_from_inputs`, so both routes resolve index paths and pick segments the same way. Four inputs are rejected rather than planned around: - No splits at all, which pins no snapshot to report, and the plan's snapshot id has to stay authoritative even when nothing is searchable. - Splits pinning different snapshots, checked before partition pruning so an inconsistent input cannot hide behind an empty plan. - Two splits for one bucket, which would search its rows twice. Java emits one split per bucket, but independently decoded buffers cannot enforce that. - A nested data split carrying its own row ranges, which would be a second authority over which physical rows are readable, free to disagree with the per-file ranges the bucket form carries. Row ranges become a per-split allow-list of physical positions on the plan, and the search intersects it with the residual predicate's allow-list: both sides list what is permitted, so a position needs to survive both. The normalization is where the two formats disagree -- Java records ranges only for the files its own pre-filter narrowed and omits the rest, while the search kernel reads a missing entry as "no rows allowed" -- so an omitted file is turned into an explicit full-file range. An empty list stays empty and excludes its file. A payload's `deletion_vectors_ranges` is ignored on purpose. Java reserves that field for deletion-vector index files, builds vector payloads through the overload that leaves it null, and takes a read's deletion vectors from the bucket's data split, so a value there describes something the payload is not. Planning from the Java golden fixture is covered end to end: the external payload path wins over both directory layouts, the five-billion-byte size survives, and a six-row file listed as rows 0-1 and 4-5 plans to exactly those positions.
A producer that restricts the readable rows of only some data files leaves the rest unrestricted, and an adapter has to say so explicitly, as an allow-list covering the whole file. Building live row ids then walked that list one position at a time, costing an insert per row of the file, where the same statement made by omitting the residual entirely takes a single range insert. Recognize the whole-file shape and insert one range instead. A list whose length equals the file's row count and whose maximum is the last position can only be the full set, so the check also subsumes the per-position bound check it replaces. Deletion vectors still apply: the shortcut only replaces how positions enter the live set, not what happens to them afterwards.
JunRuiLee
force-pushed
the
feat/pk-vector-plan-from-bucket-split
branch
from
August 28, 2026 07:47
2ef30b3 to
aaac4b9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third step of #755: let a primary-key vector search run over bucket splits an engine planned elsewhere, instead of only over a plan read from this table's index manifest.
Stacked on #752, whose commit is included — the diff against
maincontains it until that merges, which is why this is a draft. This PR's own two commits are+1036/-13.refactor: separate PK-vector planning from searching(no behavior change)plan_and_search_pk_candidates_batchresolved parameters, read the manifest into a plan, and searched it in one body, so a caller already holding a plan could not reuse the search path. Split into parameter resolution, a raw layer that searches a supplied plan and returns each query's indexed/exact lists, and a merged layer adding the rerank. The old entry point keeps its signature and wraps the three. Plan-dependent concurrency is derived from the supplied plan, so a narrowed plan is never searched under limits computed for a wider one. The three moved blocks (97, 252, 28 lines) are byte-identical.feat: plan a PK-vector search from a decoded bucket splitPkVectorScan::plan_for_bucket_vector_splitstreats the splits as authoritative — their payloads, row ranges and pinned snapshot are used as given, no manifest is read — and re-applies only the partition conjuncts, since a caller may narrow further than the planner that produced them. Path resolution, bucket grouping, current-segment selection and exact-fallback eligibility reuse the manifest route'splan_from_inputs.Rejected rather than planned around: no splits (pins no snapshot, and the plan's snapshot id must stay authoritative when nothing is searchable); splits pinning different snapshots (checked before pruning, so an inconsistent input cannot hide behind an empty plan); two splits for one bucket; a nested data split carrying its own row ranges (a second authority over readable rows — Java builds it without them).
Row ranges become a per-split allow-list of physical positions, intersected with the residual predicate's allow-list. The two formats disagree on absence: Java omits files its pre-filter did not narrow, while the search kernel reads a missing entry as "no rows", so an omitted file becomes an explicit full-file range. An explicitly empty list still excludes its file.
A payload's
deletion_vectors_rangesis ignored: Java reserves that field for deletion-vector index files, builds vector payloads through the overload leaving it null, and takes deletion vectors from the data split.Tests
16 cases. Planning the Java golden fixture is covered end to end (external path wins over both layouts, the 5e9 size survives, rows 0-1 and 4-5 of a six-row file plan to exactly those positions), plus each rejection, unlisted-versus-empty ranges, foreign column/index type, mismatch outranking pruning, all-pruned keeping its snapshot, both layouts, and the intersection. The
Table-independent core is a free function so planning is testable without a table, as the manifest route already is.Notes
plan_for_bucket_vector_splitshas no in-tree caller yet — the C entry point is the next step. It carries#[allow(dead_code)]per this crate's convention and the tests drive the free function. Happy to fold the caller in here if you would rather not merge an uncalled entry point.row_countstays unused, here and on the manifest route: it is never carried intoBucketAnnSegment, so a zero-row payload is not short-circuited before its file is opened. Pre-existing and shared by both routes, so not changed for one of them.deserialize_binary_array_stris still unhardened (elements may overlap, sonof them can each clone one body,~len²/8). Reachable fromDataFileMetadecoding, hence from a nestedDataSplit. Nothing here feeds it untrusted input, but it wants the treatment feat(table): decode the BucketVectorSearchSplit byte form #746 gave the row-array variant before a C entry point accepts arbitrary split bytes.