Repository navigation
Conversation
# Problem Closes lance-format#7227. The IoMetrics of the `take` operator are broken. The metrics such as `output_bytes=0.0 B`, `output_batches=0`, `bytes_read=0` are not visible. Example: `... Take: elapsed=350.781983256s, columns="_distance, _rowid, (vec)", metrics=[output_rows=10.00 K, elapsed_compute=302.52s, output_bytes=0.0 B, output_batches=0, batches_processed=1, bytes_read=0, iops=0, requests=0] ...` # Cause `TakeExec` was recording output row metrics inside `map_batch`, before the spawned take work had completed and before the final `RecordBatch` was emitted through DataFusion's metrics path. As a result, `output_batches`, `output_bytes`, and `ScanScheduler` I/O metrics stayed at zero even when take work read data successfully. # Solution This moves metric recording to the post-`try_buffered` result path, records the final `RecordBatch` with `BaselineMetrics::record_poll`, and records `ScanScheduler` I/O metrics after the actual take/read work completes. The stream also finalizes baseline metrics and records one final I/O snapshot when it finishes. Added a regression test covering output rows, output batches, output bytes, bytes read, IOPS, requests, and `batches_processed`.
## Bug Fix ### What is the bug? FTS phrase-query position cache hits clone `SharedPositionStream`. The stream already used shared `Bytes` for the compressed position payload, but `block_offsets` was stored as `Vec<u32>`, so each clone copied one offset per compressed position block. ### What issues or incorrect behavior does the bug cause? For large shared-position streams, repeated cache-hit clones can add avoidable O(N) CPU and memory traffic even when the position data itself is cached. ### How does this PR fix the problem? `SharedPositionStream` now stores `block_offsets` as `Arc<[u32]>`, keeping the existing `block_offsets() -> &[u32]` API while making clones share the offset allocation. A regression test verifies that cloned streams point at the same offsets allocation. ## Validation - `cargo test -p lance-index scalar::inverted::index::tests::test_shared_position_stream_clone_shares_block_offsets` - `cargo test -p lance-index shared_position` - `cargo test -p lance-index shared_stream` - `cargo test -p lance-index scalar::inverted::cache_codec::tests` - `cargo test -p lance-index` - `cargo fmt --all` - `cargo clippy --all --tests --benches -- -D warnings` - `git diff --check`
…e-format#7256) ## Summary The LSM scan planner (`LsmScanPlanner`) applied a query's `LIMIT`/`OFFSET` *above* the per-source union, so every flushed generation and the base table were fully scanned before the limit was imposed at the top. For a bounded read over a fresh tier with many flushed generations this materialized the entire tier on every query. This threads an `overfetch_factor` through `LsmScanner` → `LsmScanPlanner` and pushes a per-source fetch of `(offset + limit) * overfetch` into each `Base`/`Flushed` source scan. The active memtable is exempt (its within-source dedup needs the full match set). Block-listed sources over-fetch by `overfetch_factor` so cross-generation PK dedup still leaves ~`limit` live rows; a `LocalLimit` re-imposes the exact `skip`/`fetch` cap above the merge. Unbounded reads (no limit) are unchanged. ## Changes - `scanner/planner.rs`: `overfetch_factor` field + `with_overfetch_factor` on `LsmScanPlanner`; per-source limit pushdown in `plan_scan`; `fetch` parameter on `build_source_scan` (pushed into Base/Flushed scans, ActiveMemTable exempt). - `scanner/builder.rs`: `overfetch_factor` on `LsmScanner` + `with_overfetch_factor`, threaded into both the scan and FTS planners. ## Validation Validated end-to-end against a WAL benchmark on minikube with object storage behind a latency proxy: a bounded `offset+limit` read over an 18-generation fresh tier dropped from a full-tier scan (~360 generation scans) to a bounded per-source read. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This closes a v2.2 write-path hole where legacy blob metadata was rejected only on top-level fields. A nested field such as `summary_image_nested.image_bytes` could still carry `lance-encoding:blob` into a v2.2 dataset manifest. The write guard now scans the full Lance schema preorder and reports the offending field path. This keeps v2.2 dataset creation from silently preserving BlobV1 columns while leaving valid `lance.blob.v2` inputs untouched.
Blob v2 write support only handled top-level fields, so blob v2 columns nested under structs could reach the writer with the logical layout while the file footer expected descriptor fields. This PR makes blob v2 preprocessing and footer descriptor unloading walk nested structs, while preserving the existing per-field blob threshold handling. It also applies the same storage-version and external-reference checks to nested blob v2 fields that top-level blob v2 fields already use. Validated locally with the required Rust and Python checks plus targeted nested blob v2 read/write coverage.
## Summary Make distributed FTS finalization idempotent and object-store friendly. The old finalize flow was not retry-safe because it moved build-only `part_<old_id>_*` partition files before writing the final `metadata.lance`; if the process stopped in that window, a retry saw no final metadata but the original source files had already been deleted. The new flow treats `metadata.lance` as the commit marker and keeps all pre-commit partition sources under stable `staging/` paths. Finalization copies staged files to dense final names without deleting the staged sources or canonical final paths first, writes final metadata, then removes staging best-effort. Any leftover `staging/` objects are filtered from committed FTS `IndexMetadata.files`, so cleanup failures do not make transient files part of the committed index. ## Implementation - Write distributed build-only partition data and per-partition metadata under stable `staging/part_<old_id>_*` paths. - Keep the non-distributed builder path writing final `part_<id>_*` files and final `metadata.lance` directly, with no dense rewrite. - Add `IndexStore::copy_index_file_to(source, dest, dest_store)` for source-preserving copy-to-new-name; `LanceIndexStore` implements it with object-store copy and supports nested relative paths like `staging/...`. - Copy existing root partition files into staging for the public distributed-from-existing path, `from_existing_index(..., fragment_mask=Some(...))`, before advertising those partitions in staged metadata. - During `merge_index_files`, discover staged metadata, sort original partition IDs, map them densely to `0..N`, copy `staging/part_<old_id>_*` to final `part_<new_id>_*` without pre-deleting final paths, and then write final `metadata.lance`. - Delete staged partition data and staged per-partition metadata only after final metadata is written, and keep that cleanup best-effort. - Filter leftover `staging/` files from FTS segment `IndexMetadata.files` during segment commit. - On retry, existing final `metadata.lance` makes finalize a no-op; without it, staged sources remain authoritative and any partial final files are replaceable from staging. Added regressions for staged distributed writes, distributed-from-existing finalize, partial final-file retry without root `part_*` deletion, preserving staging when final metadata writing fails, and excluding stale staging files from committed metadata. Validated locally with Rust fmt/tests/clippy, segmented FTS integration tests, Python extension build, the targeted Python progress test, and `uv run make lint-rust`.
## Summary `CompactionOptions` did not expose `io_buffer_size`, even though the scanner used during compaction supports it. Compaction builds its scan reader in `prepare_reader`, which only forwarded `batch_size` to the scanner — the `io_buffer_size` knob was never set. This matters because a single batch larger than the I/O buffer size causes the scanner to **deadlock** (documented on `Scanner::io_buffer_size`), and since the backpressure warning was downgraded to `debug`, this deadlock is now silent at the default. Users had no way to raise the buffer to avoid it during compaction. Resolves lance-format#4946. ## Changes - Add `io_buffer_size: Option<u64>` to `CompactionOptions` (with `Default` = `None`). - Support the `lance.compaction.io_buffer_size` manifest config key in `apply_dataset_config`. - Plumb the value through `prepare_reader` → `scanner.io_buffer_size(...)`, mirroring the existing `batch_size` handling. - Update the Python binding to keep parameter names consistent across languages: - `parse_compaction_options` accepts `io_buffer_size` - `CompactionOptions` TypedDict documents the field ## Testing - Extended `test_from_dataset_config` to assert the `lance.compaction.io_buffer_size` key round-trips. - Added `test_compact_with_io_buffer_size` (parametrized over `Legacy`/`Stable` file versions) that runs `compact_files` with an explicit `io_buffer_size` and verifies the compaction succeeds and preserves all rows. - `cargo test -p lance --lib dataset::optimize::tests` → **76 passed, 0 failed**. - `cargo clippy -p lance --lib --tests` → clean. - `cargo fmt --all -- --check` → clean. ## Notes This is a non-breaking, additive change — the new field defaults to `None`, preserving existing behavior when unset.
…format#7243) ## What The per-partition top-k scan of multi-bit IVF_RQ search (`accumulate_raw_query_multi_bit_topk_with_scratch`, Normal/Accurate modes) walked all `n` rows with a scalar lower-bound computation — per row: 4 bounds-checked loads, ~5 FLOPs, two compares, iterator and `Option` plumbing — even though ~99.9% of rows are pruned once the heap is tight. On dbpedia-openai-1M (1536d, num_bits=5, nprobes=24, k=10) this loop profiled at ~10–13% of query self time. This PR vectorizes the classification: - **Dense path** (`accumulate_topk_with_scratch`, rows `0..n`): new `bq::prune` kernels evaluate the lower bound and both pruning compares for 16 rows per call, returning bit masks. Mask-zero groups (the common case) are skipped whole; surviving lanes run the existing scalar rerank with live values. The `row_id` mapping is now only invoked for rows that reach the scalar tail, and no scratch buffer is needed — everything stays in registers. - **Sparse path** (prefiltered `accumulate_filtered_topk_with_scratch`): unchanged scalar loop. Kernels follow the `ex_dot` dispatch pattern: `#[target_feature]` AVX-512 and AVX2 implementations behind a `LazyLock` runtime-dispatched fn pointer, with a portable 16-wide fallback that LLVM auto-vectorizes (NEON is baseline on aarch64). `accumulate_distances_into_heap` (Fast-mode bypass) is left as is: it has no factor arrays and would need a separate kernel shape, and Fast mode doesn't take the gated path. ## Correctness The dense path is bit-identical to the scalar implementation, not just statistically equivalent: - The kernels keep the scalar operation order (multiplies and adds, **no FMA**), so the lower bounds match `raw_query_lower_bound` bit for bit, and comparisons use ordered-quiet GE (`_CMP_GE_OQ`) matching scalar `>=` (a NaN lower bound is never pruned). - The heap threshold snapshot taken at each 16-row group start can be stale, but the threshold only ever tightens, so the masks can only over-select survivors — and survivors are re-checked per row against live values. - Heap contents, processing order, and the `LANCE_RQ_PRUNE_STATS` counters are unchanged. ## Tests - `vector::bq::prune`: every available kernel (portable, AVX2, AVX-512, dispatched) against a per-lane scalar reference on random inputs, exact `>=` boundary ties, and NaN/±inf semantics. - `test_raw_query_multi_bit_topk_dense_matches_sparse`: differential test of the dense path against the unchanged sparse scalar path with crafted factor columns controlling lower bounds and exact distances — n ∈ {1, 15, 16, 17, 100, 4109} × k ∈ {1, 10, n+7} × bounds, distance orderings descending (constant heap churn), ascending (mass pruning), random, duplicates, and exact ties, with a second pass on the shared heap (the carried tight-threshold regime). Asserts identical heap contents (row ids + distance bit patterns) and the k-smallest-distances reference. - `cargo test -p lance-index --lib vector::bq` and `cargo test -p lance ivf_rq` pass on aarch64 (portable kernel) and on x86_64 with AVX-512 (GCP c4-standard-16). ## Benchmark New `RQ heap topk` bench (binary FastScan + pruning scan + exact rerank; 4096 rows, k=10, DIM=1536, num_bits=5, error factors present so gating is enabled), GCP c4-standard-16 (AVX-512), pinned core: | mode | before | after | change | |---|---|---|---| | normal | 70.7 µs | 64.9 µs | **−9.4%** (p = 0.00) | | accurate | 93.4 µs | 87.9 µs | **−5.7%** (p = 0.00) | The binary FastScan portion of the bench is unchanged; the delta is the pruning scan itself, matching the ~5–9% end-to-end win predicted from the profile. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…nce-format#7139) ## What Extends the ngram index (today only `contains(col, 'substr')`) to also accelerate `regexp_like(col, pat)` / `regexp_match(col, pat)` and infix `LIKE` (`col LIKE '%foo%bar%'`), which until now fell through to a full table scan. Closes lance-format#7130. ## How Following Postgres `pg_trgm` and Russ Cox's [trigram-index approach](https://swtch.com/~rsc/regexp/regexp4.html), a pattern is compiled into a boolean trigram condition (`TrigramQuery`, an AND/OR tree) that is a *necessary* condition for any match. This maps onto the inverted index's set algebra: AND is posting-list intersection, OR is union. The index returns a candidate superset and the scan rechecks the true predicate, exactly as `contains` does. The derivation walks the `regex-syntax` HIR bottom-up, tracking per-node `(emptyable, exact, prefix, suffix)` sets and folding boundary trigrams across concatenation, with bounds that fold-then-discard so precision loss never drops a necessary trigram. ### Soundness The derived condition never requires a trigram a matching string could lack, so no real match is dropped. Requirements come from the index's own tokenizer (so sub-trigram runs contribute nothing); character classes and case-insensitive folds are treated as a single unknown character (the index's normalization disagrees with Unicode case folding - `(?i)c` also matches U+2102); and patterns with no derivable trigram (`a.b`, `.*`) are left to a full scan. ### Why infix LIKE A plain-literal `regexp_like(col, 'foo')` is rewritten to `col LIKE '%foo%'` by the optimizer before it reaches the index, so without infix-LIKE the most common "regex" query would not accelerate. The LIKE is translated to a loose regex for candidate generation only; the original LIKE stays as the recheck filter, so the candidate regex need only be a sound superset. ## Benchmark `cargo bench -p lance --bench regex_ngram`, 200k rows, before (main) vs after: | Query | Before | After | Change | |---|---|---|---| | `regexp_match(doc, 'zqxwvu.*needlexyz')` | 45.8 ms | 8.2 ms | -82% | | `regexp_match(doc, '(zqxwvu\|qwerasdf\|needlexyz)')` | 51.7 ms | 11.6 ms | -77% | | `regexp_match(doc, 'zqxwvu')` (rewritten to LIKE) | 36.6 ms | 8.0 ms | -78% | | `regexp_match(doc, 'a.b')` (non-accelerable) | 77.8 ms | 76.2 ms | within noise | ## Testing Unit tests for the regex-to-trigram derivation and the regex-flags folding; index-level search tests (AND across `.*`, alternation union, NULL exclusion, absent trigram); a multi-fragment end-to-end scan test asserting correct results and index engagement for `regexp_like`, `regexp_match`, and `LIKE`, plus a case-insensitive query that must fall back to a full recheck; and a regression test ensuring non-accelerable patterns still return all correct matches via full recheck. No binding changes are needed - Python/Java pass these filter strings through the existing scan API, so acceleration applies transparently. Co-authored-by: Vova Kolmakov <wombatukun@apache.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ance-format#7257) ## Summary The LSM FTS and vector search planners (`LsmFtsSearchPlanner`, `LsmVectorSearchPlanner`) built each source's plan in a sequential `for` loop and unioned the arms under a single `SortPreservingMergeExec`. The merge polls every union arm from one task, so per-arm CPU (posting/index decode, BM25 and distance scoring) serialized even though the underlying IO awaits interleave — wall time grew linearly with the flushed-generation count. This: - builds the per-source plans concurrently with `try_join_all` (FTS + vector), - runs the cross-source block-list PK hashing concurrently (`block_list.rs`), - wraps the union in a round-robin `RepartitionExec` via a new `spawn_union_arms` helper so each arm gets its own driver task. Rows stay disjoint across partitions, so the per-partition TopK + sort-preserving merge semantics are unchanged. ## Changes - `scanner/exec.rs`: `spawn_union_arms` helper (round-robin repartition over the union). - `scanner/fts_search.rs`, `scanner/vector_search.rs`: concurrent per-source plan builds + `spawn_union_arms` over the union. - `scanner/block_list.rs`: concurrent flushed-generation PK-hash loads. ## Validation Validated end-to-end against a WAL FTS benchmark on minikube with object storage behind a 10ms/GET latency proxy. Read latency over a fresh tier as a function of flushed-generation count, p50: | generations | before | after | |---|---|---| | 2 | 1,164ms | 565ms | | 5 | 1,983ms | 610ms | | 10 | 3,585ms | 660ms | | 18 | 6,071ms | 759ms | Per-generation slope dropped from ~290ms/gen to ~12ms/gen. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Closes lance-format#7209 - `python/src/dataset/optimize.rs`: `Compaction.commit` gains an optional `options` dict parameter (`#[pyo3(signature = (dataset, rewrites, options = None))]`), parsed through the same `parse_compaction_options` helper that `plan` and `execute` already use, seeded from the dataset's `manifest.config` exactly like `execute`. Absent or `None` preserves the old behavior. The TODO comment is removed. - `python/python/lance/lance/optimize.pyi`: stub updated to `commit(dataset, rewrites, options: Optional[CompactionOptions] = None)`. - `python/python/tests/test_optimize.py`: new parametrized test `test_defer_index_remap_via_commit_options`. With `Compaction.commit(dataset, rewrites, options={"defer_index_remap": True})` a `__lance_frag_reuse` system index appears in `describe_indices()`. Without options it does not (inline remap occurs). Mirrors the adjacent `test_defer_index_remap` setup.
…ance-format#7158) BREAKING CHANGE: Callers that previously depended on the silent index-drop behavior must now explicitly call `drop_index()` before `alter_columns` with a cast, then rebuild the index afterward. ## Summary Currently, calling `alter_columns` with a cast on a column that has an attached index **silently drops the index** without any warning or error. This has caused production incidents where vector search silently regressed to brute-force scan. This PR adds a **fail-fast check** before any cast is performed: if a target column has any index attached, the operation is rejected with a clear error message that names the affected column(s), the index(es), and the required remediation (`drop_index()` → cast → rebuild index). ## Changes - Added index-awareness check in `alter_columns` that scans cast target columns for attached indices - Returns a descriptive `Error::invalid_input` with column names, index names, and remediation steps when a conflict is detected - Updated existing tests that previously relied on silent index-drop behavior to explicitly `drop_index()` before casting - Added new test `test_alter_columns_cast_fails_with_attached_index` covering the IVF_PQ index path: verifies the error message, schema integrity, index preservation, and that the cast succeeds after a manual drop
…lance-format#7254) ## Context Linear: [GEN-476](https://linear.app/lancedb/issue/GEN-476) lance-namespace [0.8.6](https://github.com/lance-format/lance-namespace/releases) added an optional `source_task_size` field to `RefreshMaterializedViewRequest` (chunker materialized-view refresh work-item size, bounds per-actor memory). The reqwest client in this repo was pinned at `0.8.4`, so the field was dropped when `RestNamespace` deserialized the request — never reaching the server. ## Change Bumps `lance-namespace-reqwest-client` `0.8.4` → `0.8.6` (workspace `Cargo.toml` + `Cargo.lock`). Additive and **code-free**: the PyO3 binding (`python/src/namespace.rs`) `depythonize`s the typed `RefreshMaterializedViewRequest`, so the new field flows automatically once the crate carries it. ## Test plan - `cargo update -p lance-namespace-reqwest-client` → `v0.8.6` - `cargo check -p lance-namespace` → clean (builds against reqwest-client 0.8.6) - *(Local full build of the AWS-credential-vendor path hits a pre-existing `aws-smithy-types` E0119 under rustc 1.91, unrelated to this bump — CI's pinned toolchain builds it.)* ## Follow-up Publish a pylance beta from this so geneva can pin `lance-namespace>=0.8.6` + the new pylance and have `source_task_size` transit `db://` end-to-end. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Will Jones <willjones127@gmail.com>
…ce-format#7233) ## What Fixes a panic in `add_columns` when a fragment's deleted rows fill an entire read batch. Closes lance-format#7232. ## Root cause In `add_columns_from_stream`, the updater reads each fragment in physical batches and applies the deletion vector via `filter_record_batch`. When **every row in a physical batch is deleted**, the filter yields a **0-row batch**. The outer loop then sets `rows_remaining = 0`, the inner `while rows_remaining > 0` loop never runs, `batches` stays empty, and `concat_batches(&batches[0].schema(), ..)` panics with `index out of bounds: the len is 0 but the index is 0`. This was hit in production using a merge-columns workflow over a dataset whose fragment had a deletion file, where the deleted rows happened to align with a read-batch boundary. ## Fix When the updater yields a 0-row batch, feed an empty batch back to the updater (to keep it in sync) and continue: ```rust if rows_remaining == 0 { updater.update(RecordBatch::new_empty(stream.schema())).await?; continue; } ``` ## Test Adds `test_add_columns_with_fully_deleted_batch`: writes a single fragment with 105 rows, deletes the trailing 5 rows so that — read with `batch_size=50` — the last batch `[100..105)` is fully deleted, then verifies `add_columns` succeeds and the new column has the correct values. The test panics on the unpatched code and passes with the fix. Co-authored-by: 张杨 <zhangyang01@bilibili.com> Co-authored-by: Will Jones <willjones127@gmail.com>
The Rust core's MergeInsertBuilder supports source_dedupe_behavior (Fail/FirstSeen) to control how duplicate source rows that match the same target row are handled, but the Java binding never wired it. Java callers were stuck on the default (Fail) with no way to opt into FirstSeen. Add SourceDedupeBehavior enum + withSourceDedupeBehavior() builder to the Java MergeInsertParams, pass it through the JNI layer, and re-export SourceDedupeBehavior from lance::dataset so the binding can import it alongside the other merge types. Tests cover both enum values across the JNI boundary: FirstSeen keeps the first duplicate source row, Fail errors on duplicate source keys and leaves the dataset unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Recreating — fork main was stale at PR-creation time, inflating the commit list. |
Xuanwo
added a commit
that referenced
this pull request
Oct 28, 2026
…rmat#8878) ## What this changes A `linalg-asan` job in `nightly_run.yml` runs `cargo test -p lance-linalg --tests` under `-Zsanitizer=address`. The public entry points already validate the lengths a caller passes: `sum_4bit_dist_table` asserts `dists.len() >= n`, and `hamming_batch_u64` asserts the output length with a comment explaining why it is not a `debug_assert`. What nothing checks is the pointer arithmetic the kernels then do inside those bounds. `_mm256_storeu_si256(dists.as_mut_ptr().add(16), ...)` in the AVX2 dist-table kernels and `*results_ptr.add(1)` through `.add(3)` in `hamming_batch_avx2` are raw stores at offsets the kernel computes itself, so an off-by-one in a chunk count writes past the buffer while every entry assert still passes. It lands in `nightly_run.yml` rather than `rust.yml` because `-Zsanitizer` needs a nightly toolchain, and the only nightly this repo uses is in `daily-code-coverage.yml`, which is `schedule` plus `workflow_dispatch`. The pin is the same one that file uses, so there is one nightly date in the repo rather than two. Three details in the job are load-bearing and each is there for a reason a reader would otherwise have to rediscover. `RUSTUP_TOOLCHAIN` rather than `rustup default`. A rustup override file outranks the default toolchain, and `rust-toolchain.toml` pins a stable release, so `rustup default nightly-...` followed by `cargo test` runs the sanitizer flag into a stable compiler and stops at `error: the option 'Z' is only accepted on the nightly compiler`. `daily-code-coverage.yml` solves the same problem the other way, with `cargo +nightly-...`. `RUSTFLAGS` repeats the baseline from `.cargo/config.toml`. Setting the variable replaces that file's per-target `rustflags` wholesale rather than appending, so leaving it at just the sanitizer flag would instrument a build no release ships, and would compile out the arms gated on `target_feature = "avx2"`. One of those is where the batch call that writes into a heap buffer lives (`cosine_batch_avx512` into `vec![0.0; batch.len() / dimension]`), which is the case a sanitizer detects best. `--target x86_64-unknown-linux-gnu` keeps the sanitizer off build scripts and proc macros, which run on the host. There is no protoc step, unlike every other Rust job here. `lance-linalg`'s dependency closure contains no `prost-build` or `tonic-build`, dev-dependencies included, and the job runs green without it. ## Test plan On aarch64-apple-darwin, with one deliberate 2-byte store planted past the 32-slot output of `sum_dist_table_32bytes_batch_neon`, ASan reports `heap-buffer-overflow, WRITE of size 2` at `dist_table.rs` with the chain up through `sum_4bit_dist_table_uninit` to the test. The same binary without the sanitizer dies with a bare `SIGTRAP` and no location, so what this job adds on that mutation is the location rather than the detection. It has run for real on Linux, four times, on a fork branch with the `github.repository` guard dropped since that guard is what stops it running anywhere but here. Two clean runs: [33305364459](https://github.com/LuciferYang/lance/actions/runs/33305364459) and, after dropping the protoc step, [33312991796](https://github.com/LuciferYang/lance/actions/runs/33312991796). Both green, about a minute and a half to build and three to four minutes to run. Then two runs with a deliberate out-of-bounds store planted in a kernel, to check the claim that this job names a file and a line. That is where the `ASAN_SYMBOLIZER_PATH` step comes from. Without it the report was detected but useless: ``` WRITE of size 4 at 0x7cb5cf675100 thread T428 #0 0x5620f81ba20f (/home/runner/work/lance/lance/target/.../lance_linalg-...+0x...) #1 0x5620f81b1619 (/home/runner/work/lance/lance/target/.../lance_linalg-...+0x...) ``` ubuntu-24.04 ships `llvm-symbolizer` only under a versioned name, and ASan looks for the unversioned one on `PATH`. With the step, the same overflow reports: ``` WRITE of size 4 at 0x7ce193dcf280 thread T428 #0 ... sum_hacc_dist_table_32bytes_batch_avx2 rust/lance-linalg/src/simd/dist_table.rs:479:5 #1 ... sum_4bit_hacc_dist_table_avx2 rust/lance-linalg/src/simd/dist_table.rs:361:21 #2 ... sum_4bit_hacc_dist_table_uninit rust/lance-linalg/src/simd/dist_table.rs:244:13 #3 ... sum_4bit_hacc_dist_table rust/lance-linalg/src/simd/dist_table.rs:214:14 #4 ... tests::test_sum_4bit_hacc_dist_table_matches_u16_reference_multi_batch ``` So the failing test names itself in the stack, which is why the run does not need `--test-threads=1` to attribute a report. That exercise also turned up a coverage limit worth stating. I first planted the store in `sum_dist_table_32bytes_batch_avx2` and the job stayed green: on a runner with AVX-512BW, `sum_4bit_dist_table_uninit` selects the AVX-512 C kernel, so the Rust AVX2 one never executes. The hacc entry point has no AVX-512 kernel, which is why the planted store there did fire. So on an AVX-512 runner this job covers the hacc AVX2 kernel, the scalar paths and everything else in the crate, but not the AVX2 dist_table kernel that a smaller host would take. One gap remains: the C is not instrumented. `dist_table.c` compiles without `-fsanitize=address`, so the AVX-512 dist_table kernel is not covered. I tried `CFLAGS=-fsanitize=address` and it broke a host link in `lance-arrow`'s build script, so it is left out rather than half-applied. ## Scope note `--tests` selects the lib's unit tests and every integration target, so `tests/panic_location.rs` runs under the sanitizer too, and a target added later is included without editing this job. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Expose the Rust core's
source_dedupe_behavior(Fail/FirstSeen) on the JavaMergeInsertParams. TheMergeInsertBuilderin lance-core already supports it, but the Java binding never wired it — Java callers were stuck on the default (Fail) with no way to opt intoFirstSeen.Changes
rust/lance/src/dataset.rs— re-exportSourceDedupeBehaviorfromlance::datasetso the JNI crate can import it alongside the other merge types.MergeInsertParams.java— addSourceDedupeBehavior { Fail, FirstSeen }enum,withSourceDedupeBehavior()builder, getters, andtoString()entry. Defaults toFail, matching the Rust default.merge_insert.rs(JNI) —extract_source_dedupe_behavior()helper (mirrorsextract_when_matched), passed through to the builder.MergeInsertTest.java— cover both enum values across the JNI boundary:FirstSeenkeeps the first duplicate source row;Failerrors on duplicate source keys and leaves the dataset unchanged.Why
A merge source with duplicate join keys (e.g. a CDC stream with multiple updates per key) currently fails on the Java path with no recourse.
FirstSeenis the documented escape hatch in lance-core; this makes it reachable from Java.This is also a prerequisite for wiring native
MergeInsertParamsinto the lance-spark connector, where Spark'sMERGE INTOsemantics require a defined behavior for duplicate source keys.Test
MergeInsertTest: 11/11 pass (9 existing + 2 new)cargo fmt/cargo clippy(lance-jni) clean;cargo check -p lanceclean./mvnw spotless:checkclean🤖 Generated with Claude Code