fix(table): resolve index files by external path and bucket layout - #752
Open
JunRuiLee wants to merge 1 commit into
Open
fix(table): resolve index files by external path and bucket layout#752JunRuiLee wants to merge 1 commit into
JunRuiLee wants to merge 1 commit into
Conversation
JunRuiLee
force-pushed
the
feat/pk-scalar-pr1-index-paths
branch
3 times, most recently
from
August 26, 2026 14:05
8258587 to
a45c3e2
Compare
Open
2 tasks
JunRuiLee
force-pushed
the
feat/pk-scalar-pr1-index-paths
branch
from
August 27, 2026 08:06
a45c3e2 to
5290b66
Compare
JunRuiLee
marked this pull request as ready for review
August 27, 2026 08:07
Index files were always read from `<table>/index/<file>`, ignoring both the
`_EXTERNAL_PATH` recorded in the index manifest and the
`index-file-in-data-file-dir` table option. A table that keeps index files
beside its bucket's data files fails to read them, e.g. a primary-key vector
search reports
failed to open ANN index file
'<table>/index/index-<uuid>-0' for range reads
while the file actually lives in the bucket directory.
Decode `_EXTERNAL_PATH` from the index manifest (Java `IndexFileMeta` SCHEMA
field 5) and add it to the write schema so a rewritten manifest keeps it — it is
currently dropped silently — add the `index-file-in-data-file-dir` option, and
resolve every index file through one place (`table/index_file_path.rs`) with two
modes:
* global, always `<table>/index`: the data-evolution global index, and vector
and full-text search over it;
* bucket-local, the data-file directory when the option is set: primary-key
vector ANN segments, primary-key full-text archives, deletion vectors, and
the dynamic-bucket hash index.
Each mode mirrors the factory Java uses for that consumer:
`DataEvolutionGlobalIndexScanner` resolves through `globalIndexFileFactory`,
while `IndexFileHandler` resolves hash, deletion-vector and primary-key vector
files through `pathFactories.get(partition, bucket)`, which selects
`IndexInDataFileDirPathFactory` when the option is set. An explicit external path
wins over both layouts, as in `toPath(IndexFileMeta)`.
For the two index kinds this crate writes itself, deletion vectors and the hash
index, reads and writes move together, so a file written here is found again:
* the data-evolution writer resolves an existing deletion vector through the
same path when merging, and writes a new one where the reader will look;
* the dynamic-bucket assigner resolves per partition and bucket for both
restore and commit. `BucketAssigner::prepare_commit_index` no longer takes an
index directory — three of its four implementations ignored it, and the
fourth now derives the layout itself;
* `TableCommit::abort` deletes a newly written index file where it was written,
mirroring Java `FileStoreCommitImpl.abort`, which deletes through
`indexFileFactory(partition, bucket)`. Deleting is best-effort, so the old
fixed path leaked the file silently instead of failing.
A bucket directory comes from the split that references the file when a split is
at hand, and otherwise from the partition and bucket being committed. Both go
through one `spec::bucket_path`, mirroring Java `FileStorePathFactory.bucketPath`:
the layout is only correct while every producer and consumer of a bucket
directory agrees byte for byte, and nothing else enforces that.
The option is immutable, as in Java, where it is annotated `@Immutable` and
`SchemaManager.checkAlterTableOption` rejects altering it: it selects the
directory index files are written to, so flipping it on a populated table would
hide every index file already written.
`$physical_files_size` now counts an `index-` prefixed file in a bucket directory
as an index file rather than dropping it, matching Java `FileType.classify`,
which maps any `index-*` basename to `BUCKET_INDEX` regardless of directory.
Classification follows the file's physical form, not the current option value, so
a file stays recognizable after the setting it was written under changes.
The BTree reader cache is keyed by the resolved path so two entries sharing a
file name cannot reuse each other's reader.
`data-file.path-directory` remains unsupported, as it is throughout this crate:
bucket paths are rooted directly at the table for data files as much as for
index files, so honoring it belongs with data-file path handling rather than
here.
JunRuiLee
force-pushed
the
feat/pk-scalar-pr1-index-paths
branch
from
August 28, 2026 06:38
5290b66 to
af2167b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Index files are always read from
<table>/index/<file>, which ignores two things Java records andhonors:
_EXTERNAL_PATH—IndexFileMeta.SCHEMAfield 5. Not decoded here at all, so it is also droppedwhen this crate rewrites an index manifest.
index-file-in-data-file-dir— when set, Java'sindexFileFactory(partition, bucket)resolves anindex file against the bucket's data-file directory instead of the table
index/directory.Either one makes a table's index files unreadable. Observed as a primary-key vector search:
failed to open ANN index file '.../<table>/index/index-<uuid>-0' for range reads, while the file isin the bucket directory.
How
Decode
_EXTERNAL_PATHand add it to the write schema, add theindex-file-in-data-file-diroption,and route every index-file consumer through one resolver (
table/index_file_path.rs) with two modes,each mirroring the factory Java uses for that consumer:
<table>/indexglobalIndexFileFactory()pathFactories.get(partition, bucket)An explicit external path wins over both layouts, as in
toPath(IndexFileMeta).For the two index kinds this crate writes itself, reads and writes move together, so a file written
here is found again: the data-evolution writer resolves an existing deletion vector through the same
path when merging and writes a new one where the reader will look; the dynamic-bucket assigner resolves
per partition and bucket for both restore and commit (
prepare_commit_indexno longer takes an indexdirectory — three of its four implementations ignored it); and
TableCommit::abortdeletes where thefile was written, mirroring
FileStoreCommitImpl.abort. Deleting is best-effort, so the old fixed pathleaked silently — and
try_commitcallsabortitself on failure, as do the C and Python bindings.Bucket directories now come from one
spec::bucket_path(JavaFileStorePathFactory.bucketPath),replacing four copies of the same expression: the layout is only correct while every producer and
consumer agrees byte for byte, and nothing else enforces that.
Two smaller consequences of making the option live:
@Immutable, rejected bySchemaManager.checkAlterTableOption).It selects where index files are written, so flipping it on a populated table would hide every index
file the table has. It was inert here before, so altering it used to be a harmless no-op.
$physical_files_sizecounts anindex-prefixed file in a bucket directory as an index fileinstead of dropping it, matching Java
FileType.classify. Classification follows physical form, notthe current option value. Two tests that asserted the old counts are corrected.
The BTree reader cache is keyed by the resolved path — with the bare file name, two entries sharing a
name but resolving to different locations would reuse each other's reader.
Out of scope
data-file.path-directory: unsupported crate-wide (bucket paths are rooted at the table for datafiles as much as index files), so honoring it belongs with data-file path handling.
data files there either. Reading a Java-written external index file works.
higher-stakes direction (
LocalOrphanFilesCleanmaps candidate deletions byPath.getName()), andall names are UUID-based, so collisions are not reachable.
Behavior change
A table created with the option set but written only by older paimon-rust has its index files under
<table>/index, because this crate ignored the option — such a table is already unreadable by Java.After this PR Rust agrees with Java. No compatibility fallback on purpose: one would diverge from Java
and mask genuine missing-file errors.
Testing
New:
abortdeletes an index file in the bucket data-file directory, and one at an external path (bothverified to fail against the old path); ANN segment and full-text archive resolution with the option
set, through a real split whose bucket path is not derivable from the table root — the failure this PR
fixes, previously untested through a split;
directory()agrees withresolve()in every mode; decodetests for
_EXTERNAL_PATHpresent, present-but-null, and absent from the writer schema; the optioncannot be altered.
Gates:
cargo fmt --all -- --checkclean;clippy -D warnings --all-targetsclean forpaimon,paimon --features fulltext,paimon-datafusion,paimon-c(paimon-pythonneeds Python ≥ 3.10,unavailable locally);
cargo test -p paimon --lib2425 passed, 2500 with--features fulltext,--testsall pass.paimon-datafusion339 passed / 7 failed — all seven areTableNotExistfor the/tmp/paimon-warehousefixtures thatmake docker-upprovisions, unrelated to this change.