Repository navigation
feat(index): share IVF partition scans across batch vector queries - #7640
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
📝 WalkthroughWalkthroughAdds indexed ANN batch vector search with shared IVF partition reads, per-query result grouping, cosine-safe normalization, correctness-gated scanner planning, fallback coverage, equivalence tests, and batch-versus-repeated-search benchmarks. ChangesIndexed ANN batch search
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Scanner
participant ANNIvfBatchExec
participant IVFIndex
Client->>Scanner: batch nearest query
Scanner->>ANNIvfBatchExec: plan indexed batch search
ANNIvfBatchExec->>IVFIndex: search multiple queries across partitions
IVFIndex-->>ANNIvfBatchExec: per-query top-k results
ANNIvfBatchExec-->>Scanner: grouped query_index results
Scanner-->>Client: batch result table
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
python/python/tests/test_vector_index.py (1)
240-241: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSeed the RNG so equivalence failures are reproducible.
This test asserts exact
idequality between the shared-scan batch path and repeated single-query search over an approximate IVF_PQ index. With unseedednp.random.randn, any tie-break or ordering discrepancy (see theHashMapscoring-order note inrust/lance/src/index/vector/ivf/v2.rs) surfaces as an intermittent failure that cannot be replayed.♻️ Suggested change
+ rng = np.random.default_rng(42) scales = np.linspace(0.1, 10.0, query_count).reshape(-1, 1) - queries = (np.random.randn(query_count, 128) * scales).astype(np.float32) + queries = (rng.standard_normal((query_count, 128)) * scales).astype(np.float32)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/python/tests/test_vector_index.py` around lines 240 - 241, Seed NumPy’s random number generator before the randomized query construction in the test containing the shared-scan and repeated single-query equivalence assertions. Use a fixed, explicit seed so failures involving exact id ordering are deterministic and reproducible, while leaving the existing query generation and assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance/src/dataset/scanner.rs`:
- Around line 4089-4096: Update the refine-factor gate in the batch
indexed-search path to reject any Some value, not only factors greater than one,
so refine(1) and refine(0) fall back to the per-query loop for reranking and
validation. Extend test_batch_knn_indexed_refine_falls_back to cover both refine
factors while preserving the existing refine(2) coverage.
- Around line 4137-4158: Update the batch fast-path gate in the surrounding
scanner method to fall back whenever
self.overlay_stale_vector_rows(index_segments)? is non-empty, before
constructing the batch executor. Preserve the existing batch behavior when no
stale overlay rows exist, and add coverage using an overlaid fragment that
verifies execution takes the per-query fallback path.
In `@rust/lance/src/index/vector/ivf/v2.rs`:
- Around line 1920-1938: The batch search flow around load_parallelism and the
try_collect::<Vec<_>>() currently retains every probed partition before scoring.
Replace this all-at-once collection with bounded-window processing, using
STREAMING_SEARCH_BATCH_SIZE or the existing bounded-channel pattern: load one
window, dispatch its scoring through spawn_cpu, release partition entries, then
continue until all assignments are processed while preserving result ordering
and error propagation.
- Around line 1902-1916: Update OrderedNode::cmp to use row_id as the final
comparison key after distance, ensuring equal-distance candidates are
deterministically selected and ordered. Preserve the existing heap behavior and
make the resulting global_heap_to_batch output match repeated single-query
ordering.
In `@rust/lance/src/io/exec/knn.rs`:
- Around line 2401-2411: Update the per-query batch handling around the loop
over per_query: first validate that the returned batch count matches the query
count, returning a diagnosable Internal error on mismatch before indexing
candidates. Then replace positional batch.column access with column_by_name
using the schema’s distance and row-id field names, preserving the existing
candidate extension behavior.
---
Nitpick comments:
In `@python/python/tests/test_vector_index.py`:
- Around line 240-241: Seed NumPy’s random number generator before the
randomized query construction in the test containing the shared-scan and
repeated single-query equivalence assertions. Use a fixed, explicit seed so
failures involving exact id ordering are deterministic and reproducible, while
leaving the existing query generation and assertions unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: c6efabac-70c1-4356-8665-8f6423a9f771
📒 Files selected for processing (6)
python/python/benchmarks/test_search.pypython/python/tests/test_vector_index.pyrust/lance-index/src/vector.rsrust/lance/src/dataset/scanner.rsrust/lance/src/index/vector/ivf/v2.rsrust/lance/src/io/exec/knn.rs
| // Invert the per-query partition lists so each distinct partition is | ||
| // loaded once and scored against every query that probes it. | ||
| let mut assignments: HashMap<u32, Vec<(usize, f32)>> = HashMap::new(); | ||
| for (query_index, (parts, dists)) in partitions_per_query | ||
| .iter() | ||
| .zip(q_c_dists_per_query.iter()) | ||
| .enumerate() | ||
| { | ||
| for (part_id, dist_q_c) in parts.values().iter().zip(dists.values().iter()) { | ||
| assignments | ||
| .entry(*part_id) | ||
| .or_default() | ||
| .push((query_index, *dist_q_c)); | ||
| } | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Does the global top-k heap impose a total order that includes the row id?
rg -nP -C6 'struct OrderedNode|impl .*Ord.* for OrderedNode' --type=rust
ast-grep run --pattern 'fn global_heap_to_batch($$$) { $$$ }' --lang rust rust/lance/src/index/vector/ivf/v2.rsRepository: lance-format/lance
Length of output: 156
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files 'rust/lance/src/index/vector/ivf/v2.rs' 'rust/lance/src/dataset/scanner.rs' | sed -n '1,20p'
echo "== outline around top of v2.rs =="
wc -l rust/lance/src/index/vector/ivf/v2.rs rust/lance/src/dataset/scanner.rs
ast-grep outline rust/lance/src/index/vector/ivf/v2.rs --length 200 | sed -n '1,200p'
echo "== search relevant terms =="
rg -n "assignments|partitions_per_query|global_heap_to_batch|TestVectorDataset|assert_batch_matches_single_queries|OrderedNode|OrderedCandidate|global_top|topk" rust/lance/src -S
echo "== lines 1860-1945 in ivf/v2.rs =="
sed -n '1860,1945p' rust/lance/src/index/vector/ivf/v2.rs
echo "== lines around scanner tests =="
rg -n "assert_batch_matches_single_queries|TestVectorDataset" rust/lance/src/dataset/scanner.rs -A 80 -B 20Repository: lance-format/lance
Length of output: 548
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== search relevant terms =="
rg -n "assignments|partitions_per_query|global_heap_to_batch|TestVectorDataset|assert_batch_matches_single_queries|OrderedNode|OrderedCandidate|global_top|topk" rust/lance/src -S || true
echo "== lines 1860-1945 in ivf/v2.rs =="
sed -n '1860,1945p' rust/lance/src/index/vector/ivf/v2.rs
echo "== lines around scanner tests =="
rg -n "assert_batch_matches_single_queries|TestVectorDataset" rust/lance/src/dataset/scanner.rs -A 100 -B 20 || true
echo "== import headers for v2.rs =="
sed -n '1,120p' rust/lance/src/index/vector/ivf/v2.rsRepository: lance-format/lance
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== ivf/v2.rs relevant scoring/heap/export lines =="
sed -n '800,935p' rust/lance/src/index/vector/ivf/v2.rs
sed -n '1945,2025p' rust/lance/src/index/vector/ivf/v2.rs
echo "== OrderedNode definition and Ord impl =="
rg -n -C 12 'struct OrderedNode|BinaryHeap<OrderedNode|Ord for Ordered|impl PartialOrd|impl PartialEq' rust/lance/src -S | sed -n '1,240p'
echo "== relevant tests around batch indexed equivalence =="
sed -n '6950,7825p' rust/lance/src/dataset/scanner.rsRepository: lance-format/lance
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== exact OrderedNode references in graph submodule =="
rg -n -C 20 'OrderedNode' rust/lance/src/index/vector/graph.rs rust/lance/src/index/vector -S | sed -n '1,260p'
echo "== ivf.rs OrderedNode/topk area =="
sed -n '3980,4095p' rust/lance/src/index/vector/ivf.rs
echo "== Behavioral probe: Rust BinaryHeap Ord ties (if cargo availability) =="
if cargo --version >/dev/null 2>&1 && rustc --version >/dev/null 2>&1; then
tmp="$(mktemp -d)"
cat > "$tmp/heap_order_probe.rs" <<'RS'
fn main() {
let mut heap = std::collections::BinaryHeap::new();
for row_id in 0u64..10 {
heap.push((0.0f32, row_id));
}
let out: Vec<_> = heap.into_iter().map(|(d, id)| format!("({}, {})", d, id)).collect();
for (i, item) in out.iter().enumerate() {
println!("heap_order_probe.{}. {}", i, item);
}
}
RS
cargo run --manifest-path "$tmp/HeapOrderProbeCargo.toml" 2>/dev/null || (
cat > "$tmp/Cargo.toml" <<'TOML'
[package]
name = "heap_order_probe"
version = "0.0.0"
edition = "2021"
TOML
mkdir -p "$tmp/src"
mv "$tmp/heap_order_probe.rs" "$tmp/src/main.rs"
cargo run --manifest-path "$tmp/Cargo.toml"
)
else
echo "cargo/rustc unavailable"
fiRepository: lance-format/lance
Length of output: 14303
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate graph.rs =="
git ls-files | rg '(^|/)graph\.rs$|lance_index/src|index/vector' | sed -n '1,120p'
echo "== graph.rs OrderedNode definitions/usages =="
while IFS= read -r f; do
rg -n -C 8 'struct OrderedNode|impl .*OrderedNode|derive\(.*Ord.*\)|impl PartialOrd|impl Ord|BinaryHeap<OrderedNode' "$f"
done < <(git ls-files | rg '(^|/)graph\.rs$')
echo "== ivf storage topk calls into graph =="
rg -n -C 12 'accumulate_topk_with_scratch|topk|OrderedNode' rust/lance/src/index/vector -S | sed -n '1,260p'Repository: lance-format/lance
Length of output: 1335
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== rust/lance-index/src/vector/graph.rs OrderedNode area =="
sed -n '1,180p' rust/lance-index/src/vector/graph.rs
echo "== binary heap tie ordering probe =="
python3 - <<'PY'
# Rust std::collections::BinaryHeap iterates from the maximum heap element;
# equal-order elements can emerge in different relative orders.
import random
heap = []
for row_id in range(9):
random.shuffle(heap)
heap.append((0.0, row_id))
heap.sort(reverse=True)
seen = set()
for _ in range(200):
q = heap.copy()
out = []
while q:
_, row_id = q.pop()
out.append(row_id)
seen.add(tuple(out))
print("seen orderings:", len(seen))
PYRepository: lance-format/lance
Length of output: 4451
Use a deterministic tie-break for batch top-k results.
OrderedNode only orders by distance, so equal-distance candidates are not deterministically selected or ordered before global_heap_to_batch emits them. With ties present, the shared batch path can produce the same top-k set in a different order than repeated single-query search, causing the indexed equivalence assertions to become flaky. Add the row id as the final comparison key in OrderedNode::cmp, or sort the heap output by (distance, row_id) before returning the batch.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance/src/index/vector/ivf/v2.rs` around lines 1902 - 1916, Update
OrderedNode::cmp to use row_id as the final comparison key after distance,
ensuring equal-distance candidates are deterministically selected and ordered.
Preserve the existing heap behavior and make the resulting global_heap_to_batch
output match repeated single-query ordering.
There was a problem hiding this comment.
Addressed the underlying nondeterminism, though not where the heap comment pointed. search_partitions_batch accumulated partitions in HashMap iteration order, so when rows tie at the k-th distance, which one survives the capped per-query heap varied run to run — fixed with loaded.sort_by_key(part_id) for a stable scan order (recall-neutral: any tied row is an equally valid k-th neighbor).
On emit order specifically: both global_heap_to_batch consumers already re-sort by (distance, row_id) — a total order — before truncating, so the final ordering was never user-visibly nondeterministic. I therefore did not add a sort inside global_heap_to_batch itself, since it would be redundant and would also tax the shared single-query path.
New test test_batch_knn_indexed_orders_ties_by_row_id pins the end-to-end guarantee: five neighbors tied at distance 0 come back in ascending-row-id order, and the batch result matches repeated single-query search (both stable/unstable row-id modes).
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== current batch partition ordering code =="
rg -n -C 10 'loaded\.sort_by_key|let loaded =|search_partitions_batch' \
rust/lance/src/index/vector/ivf/v2.rs
echo "== tie-order regression test =="
rg -n -C 12 'test_batch_knn_indexed_orders_ties_by_row_id' rust/lance --glob '*.rs'
echo "== current changed files relevant to the reported fix =="
git diff -- rust/lance/src/index/vector/ivf/v2.rs rust/lance/src/dataset/scanner.rsLength of output: 2686
The checkout currently does not show either reported addition. I’ll make one non-failing inspection to distinguish an out-of-date checkout from a missing change.
There was a problem hiding this comment.
The change is on the current PR head (135df234f): the tie-break sort is at rust/lance/src/index/vector/ivf/v2.rs:1946 (loaded.sort_by_key(|(part_id, _, _)| *part_id)), and the regression test is test_batch_knn_indexed_orders_ties_by_row_id in rust/lance/src/dataset/scanner.rs:7472. Looks like the analysis ran against an out-of-date checkout.
| // Load each distinct partition's storage exactly once. This shared I/O | ||
| // is the whole point of batch search versus repeated single queries. | ||
| let load_parallelism = get_num_compute_intensive_cpus().max(1); | ||
| let load_index = self.clone(); | ||
| let load_metrics = metrics.clone(); | ||
| let loaded = stream::iter(assignments) | ||
| .map(move |(part_id, probing_queries)| { | ||
| let index = load_index.clone(); | ||
| let metrics = load_metrics.clone(); | ||
| async move { | ||
| let part_entry = index | ||
| .load_partition(part_id as usize, true, metrics.as_ref()) | ||
| .await?; | ||
| Result::Ok((part_id as usize, part_entry, probing_queries)) | ||
| } | ||
| }) | ||
| .buffered(load_parallelism) | ||
| .try_collect::<Vec<_>>() | ||
| .await?; |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
All probed partitions are materialized before any scoring — unbounded memory for wide batches.
try_collect::<Vec<_>>() holds every distinct probed partition's storage alive at once. For a batch of q queries with nprobes probes each, that is up to min(q * nprobes, num_partitions) partitions resident, and the Arc clones also pin those entries in the index cache. The single-query path deliberately avoids this by streaming prepared partitions through a bounded channel (STREAMING_SEARCH_BATCH_SIZE, Line 1663) after #7642.
Consider processing the load stream in chunks (load a bounded window, dispatch scoring for it via spawn_cpu, drop the entries, repeat) so peak memory is bounded by the window rather than by the probe fan-out.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance/src/index/vector/ivf/v2.rs` around lines 1920 - 1938, The batch
search flow around load_parallelism and the try_collect::<Vec<_>>() currently
retains every probed partition before scoring. Replace this all-at-once
collection with bounded-window processing, using STREAMING_SEARCH_BATCH_SIZE or
the existing bounded-channel pattern: load one window, dispatch its scoring
through spawn_cpu, release partition entries, then continue until all
assignments are processed while preserving result ordering and error
propagation.
There was a problem hiding this comment.
Agreed this is real — materializing all probed partitions before scoring is unbounded for wide batches. It's a larger change to the accumulation model (streaming / bounded partition loading) than the other findings, so I'm tracking it separately rather than expanding this PR's scope.
There was a problem hiding this comment.
@sezruby, agreed. Deferring the bounded-window loading/scoring redesign is reasonable to keep this PR focused. The finding remains open for the follow-up work.
Would you like me to create a GitHub issue with the affected path, rationale, and acceptance criteria?
🐇📦
There was a problem hiding this comment.
Given this is being hooked into the Scanner already, I don't think we should defer this. As written, this would be a serious regression in memory use, right?
There was a problem hiding this comment.
Fixed in 8109f6f — not deferred.
search_partitions_batch now streams the loaded partitions through scoring in STREAMING_SEARCH_BATCH_SIZE chunks instead of try_collect-ing them all first. Each chunk is scored into the per-query top-k heaps in one spawn_cpu dispatch and then dropped, so resident partition storage is bounded to the load window plus one chunk rather than scaling with the batch width (previously up to min(query_count * nprobes, num_partitions) — potentially the whole index).
Determinism is preserved: the distinct partitions are sorted by part_id and buffered keeps that order, so the across-partition tie-break at the k-th distance is unchanged. The partition-loading awaits stay in the async loop — only pure-CPU scoring goes to spawn_cpu — per the pool-safety constraint in #7642.
While here I also fixed a related divergence: nprobes(0) (min == max == 0) passed the fixed-nprobes gate, but the node clamped it up to one partition while the single-query path probes nothing and returns empty. The scanner now falls back when minimum_nprobes == 0, and the .max(1) clamp is replaced with a debug_assert! documenting the invariant.
Tests: test_batch_knn_indexed_streams_multiple_chunks (a wide batch spanning multiple chunks, pinned to repeated single-query search) and test_batch_knn_indexed_zero_nprobes_falls_back.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rust/lance/src/io/exec/knn.rs (1)
2374-2382: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
.max(1)silently clampsnprobes(0), breaking the equivalence the batch path is gated on.
Scanner::nprobes(0)is not rejected (scanner.rsLine 1634 sets both bounds with no validation), sominimum_nprobes == maximum_nprobes == Some(0)passesbatch_index_search_supported. The single-query path then reachesIVFIndex::find_partitionswithmax_nprobes = 0and probes nothing (empty result), while this node clamps to 1 and returns a partition's worth of neighbors. That is exactly the batch-vs-repeated-single-query divergence the fixed-nprobes gate exists to rule out, and it silently adjusts a caller-supplied value.Either propagate
minimum_nprobesunchanged here, or make the scanner gate fall back whenminimum_nprobes == 0(arguablynprobes(0)should be rejected outright at the builder, likerefine(0)is).As per coding guidelines, "Validate inputs at API boundaries and reject invalid values with descriptive errors; never silently clamp or adjust them."
🐛 Proposed fix
- let nprobes = single_query.minimum_nprobes.max(1); - single_query.maximum_nprobes = Some(nprobes); + // The scanner gate guarantees `minimum_nprobes == maximum_nprobes`, so + // pass it through unchanged: clamping here would diverge from the + // single-query path this node must reproduce exactly. + single_query.maximum_nprobes = Some(single_query.minimum_nprobes);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/io/exec/knn.rs` around lines 2374 - 2382, Remove the .max(1) clamp when assigning nprobes in the batch search setup so single_query.minimum_nprobes, including zero, is propagated unchanged and remains equivalent to repeated single-query search. Keep the existing fixed-probe gating behavior intact.Source: Coding guidelines
🧹 Nitpick comments (1)
rust/lance/src/dataset/scanner.rs (1)
4063-4068: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDoc's "Requires all of" list is now incomplete — add the overlay condition.
The list reads as exhaustive but omits the overlay-staleness gate added at Lines 4124-4135, which is a correctness requirement on par with the others.
As per coding guidelines, "Ensure doc comments match actual semantics".
📝 Proposed doc update
/// Requires all of: /// - no refine step (the batch path does not yet rerank); /// - fixed nprobes (`minimum_nprobes == maximum_nprobes`) — see below; /// - every segment an IVF index with a flat-style sub-index (i.e. not HNSW); + /// - no overlay-stale indexed rows (the batch path neither blocks nor + /// re-scores them); /// - all target fragments indexed (or `fast_search`, which ignores /// unindexed fragments).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/scanner.rs` around lines 4063 - 4068, Update the “Requires all of” documentation for the batch path to include the overlay-staleness gate enforced near the scan eligibility logic, alongside the existing refine, nprobes, index, and fragment requirements. Describe the condition using the same semantics as the overlay check in the surrounding scanner implementation.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@rust/lance/src/io/exec/knn.rs`:
- Around line 2374-2382: Remove the .max(1) clamp when assigning nprobes in the
batch search setup so single_query.minimum_nprobes, including zero, is
propagated unchanged and remains equivalent to repeated single-query search.
Keep the existing fixed-probe gating behavior intact.
---
Nitpick comments:
In `@rust/lance/src/dataset/scanner.rs`:
- Around line 4063-4068: Update the “Requires all of” documentation for the
batch path to include the overlay-staleness gate enforced near the scan
eligibility logic, alongside the existing refine, nprobes, index, and fragment
requirements. Describe the condition using the same semantics as the overlay
check in the surrounding scanner implementation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 1bd1048c-b78e-4028-9866-9616c04698b9
📒 Files selected for processing (4)
rust/lance/src/dataset/scanner.rsrust/lance/src/dataset/tests/dataset_overlay_index_masking.rsrust/lance/src/index/vector/ivf/v2.rsrust/lance/src/io/exec/knn.rs
wjones127
left a comment
There was a problem hiding this comment.
This is a worthwhile optimization, but I can't accept it with the current memory use regression.
Is there any follow up design for how we can make this fully supported, including by refine factor and min/max nprobes?
| /// late-search expansion used by the single-query path is not applied, so recall | ||
| /// matches repeated single-query search when `minimum_nprobes == maximum_nprobes`. | ||
| #[derive(Debug)] | ||
| pub struct ANNIvfBatchExec { |
There was a problem hiding this comment.
nitpick(modularity): Could we give this execution node it's own module / file? This file is already very long.
| let mut query_index_builder = Int32Builder::new(); | ||
| let mut distance_builder = Float32Builder::new(); | ||
| let mut row_id_builder = UInt64Builder::new(); |
There was a problem hiding this comment.
nitpick(performance): please pre-size these array allocations.
| } | ||
| let properties = Arc::new(PlanProperties::new( | ||
| EquivalenceProperties::new(knn_empty_result_schema(true)), | ||
| Partitioning::RoundRobinBatch(1), |
There was a problem hiding this comment.
note: going forward I'd encourage us to support multiple partitions so we have CPU parallelism in DataFusion. This node could be split so that query vectors are round-robined across partitions. The IO to load partitions could be done from a background IO task that provides them to an async MPSC channel. That channel should be bounded by a MemoryConsumer, so it doesn't load too many at once. This can be an optional follow up.
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The shared-scan direction is worthwhile, and this revision fixes the earlier unbounded partition retention. Acceptance still requires preserving fragment-selection completeness and keeping centroid ranking off async executor threads. The dedicated batch node can remain, with eligibility computed from the selected segment coverage and its per-query ranking dispatched through the CPU runtime.
There was a problem hiding this comment.
Both earlier Gate findings are fixed: fragment coverage is now computed from the selected index segments, and batch centroid ranking runs on the dedicated CPU runtime. The native shared-scan path is the right mechanism and preserves the covered per-query and fallback semantics.
The remaining non-blocking risk is performance for low-overlap or very wide batches: batch CPU work is sequential within each dispatched job, while the benchmark does not compare against the former in-request per-query parallel plan. Treat the speedup as workload-dependent until low-overlap and query-count scaling are measured.
|
@wjones127 gentle ping on this one. The memory-regression concern is resolved — The modularity and pre-sizing nitpicks are still open — happy to fold both into this pass if you'd like them in before re-review, or as a quick follow-up. Whenever you have a chance to take another look, thanks! |
wjones127
left a comment
There was a problem hiding this comment.
Looks better. Thanks for addressing the memory use.
|
Looks like there's a simple rebase that needs to happen before this can be merged. |
Extend batch vector search (lance-format#6821) to the indexed/ANN path so a single multi-query request reads each IVF partition's storage once and scores every query that probes it, instead of re-running a full single-query plan per vector and unioning the results (which re-opens the index and rebuilds the prefilter for each query). - Add `VectorIndex::search_partitions_batch` + `supports_batch_partition_search` (defaulted so non-IVF indices stay explicitly unsupported). - Implement them for `IVFIndex` with a flat-style sub-index (IVF_FLAT/PQ/SQ/RQ): load each distinct partition once and accumulate one top-k heap per query, sharing the prefilter across the whole batch. - Add `ANNIvfBatchExec`, which ranks every query against the centroids, runs the shared-scan batch search, merges per-query top-k across deltas, and emits `query_index`-tagged results; route to it from `Scanner::batch_indexed_vector_search` when the gate below holds. - Normalize each query vector independently for cosine (`normalize_batch_query_for_index`): normalizing the concatenated batch key with one global norm would scale each vector by a batch-composition-dependent factor and break equivalence with single-query search. The shared-scan fast path is gated to cases that are provably equivalent to repeated single-query search: fixed nprobes (`minimum_nprobes == maximum_nprobes`), no refine step, an IVF flat-style index, and fully-indexed fragments. With adaptive nprobes the single-query path applies an `early_pruning` floor and late-search expansion that the batch path does not, so those queries fall back to the per-query loop, which stays exact. HNSW, refine, and mixed indexed/unindexed scans also fall back. Tests: plan shape; exact batch-vs-repeated-single equivalence (nprobes pinned); cosine regression; shared prefilter; multi-delta cross-delta merge; and fallbacks for refine and adaptive nprobes. Python parametrized over L2 + cosine; a batch-vs-repeated-single ANN benchmark. Closes lance-format#6822 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hunk `test_batch_knn_indexed_streams_multiple_chunks` relies on the 20-partition index probing more distinct partitions than one `STREAMING_SEARCH_BATCH_SIZE` chunk (16) holds. That premise was implicit: if the default chunk size were raised past the partition count, the test would silently collapse to a single chunk and stop covering the heap-threading seam it is named for, while still passing. Assert `num_partitions > chunk_size` so that regression fails loudly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rd partitions_searched Three follow-ups on the shared-scan batch IVF path: - Gate eligibility on whether the *selected* index_segments cover every requested fragment, not whether the whole logical index does. A subset selected via with_index_segments could otherwise let the batch node search only the selected segments and silently drop a fragment covered solely by an unselected segment. Extracted the coverage check into fragments_missing_from_index_segments, shared by the gate and knn_combined so eligibility and fallback stay in lockstep. - Run the per-query centroid ranking on the dedicated CPU runtime (find_partitions_batch_on_cpu) instead of the async worker: the ranking is pure CPU and batch width multiplies it, so a wide batch over a large centroid set could monopolize a Tokio worker. - Record partitions_searched on ANNIvfBatchExec. It built the metric but never incremented it, so EXPLAIN ANALYZE reported 0 for every batch query. Report the distinct partitions read -- the shared I/O this node exists to save -- mirroring the single-query ANNIvfSubIndexExec. Tests: partial-segment fallback, CPU-runtime ranking, and a partitions_searched=2 assertion (distinct union, not the per-query sum of 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
a1060cd to
8fc0e0c
Compare
|
Rebased onto One conflict, in
|
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The rebase note identifies the integration with the merged external-mask feature, but the shared indexed batch path does not preserve that contract: it remains eligible when a mask is present and drops the mask before building its prefilter. The earlier fragment-coverage and CPU-runtime fixes remain intact; acceptance now requires either carrying the external mask into the shared batch prefilter or falling back to the existing per-query path.
The prior low-overlap and very-wide-batch performance uncertainty remains non-blocking.
| // across the batch. HNSW, refine, and mixed indexed/unindexed scans fall | ||
| // back to the per-query loop below, which never regresses behavior. | ||
| if self | ||
| .batch_index_search_supported(index_name, index_segments, q) |
There was a problem hiding this comment.
This fast path can return rows excluded by with_row_addr_prefilter. batch_index_search_supported does not reject self.external_row_mask, and new_knn_batch_exec has no mask input; ANNIvfBatchExec then supplies None to the shared prefilter. The merged mask contract requires the mask to constrain ANN top-k. Either pass the mask into the batch prefilter or return false from this eligibility check when a mask exists, so the per-query path preserves it.
Reproducer run on this head
I added this temporary test beside test_batch_knn_indexed and ran:
cargo test -p lance --lib gate_repro_batch_knn_indexed_respects_external_mask -- --nocapture
#[tokio::test]
async fn gate_repro_batch_knn_indexed_respects_external_mask() {
let mut test_ds = TestVectorDataset::new(LanceFileVersion::Stable, true)
.await
.unwrap();
test_ds.make_vector_index().await.unwrap();
let dataset = &test_ds.dataset;
let (queries, _) = batch_knn_two_queries();
let mut scan = dataset.scan();
scan.nearest("vec", &queries, 2).unwrap();
scan.nprobes(2);
scan.with_row_addr_prefilter(RowAddrMask::allow_nothing());
scan.project(&["i"]).unwrap();
let plan = scan.explain_plan(false).await.unwrap();
assert!(plan.contains("ANNIvfBatch"), "unexpected plan:\n{plan}");
let batch = scan.try_into_batch().await.unwrap();
assert_eq!(batch.num_rows(), 0, "allow-nothing mask must exclude all rows");
}The plan used ANNIvfBatch; the assertion observed 4 rows instead of 0. The test itself finished in 0.36s.
There was a problem hiding this comment.
Fixed in 717b3e129: every batch query with an external row mask now falls back to the per-query ANN path, which forwards the mask into each prefilter. The focused regression passed and confirms the plan fallback and that every returned row stays within the allowlist.
… is set The shared-scan batch path builds one prefilter across the batch and does not carry a caller-supplied external row-address mask (`with_row_addr_prefilter`), whereas the per-query path threads it into each query via `with_external_mask`. After the merge with the external-mask feature, an otherwise batch-eligible query with a mask would have selected the batch node and silently dropped the mask, returning masked-out rows. Disqualify the batch path in `batch_index_search_supported` whenever an external mask is present so the query falls back to the per-query indexed loop, which honors the mask. Matches the existing fallback pattern (refine, adaptive/zero nprobes, HNSW, overlay, partial coverage). Adds `test_batch_knn_indexed_external_mask_falls_back`: the same query is batch-eligible without a mask, falls back to ANNSubIndex with one, and every returned row is in the allowlist.
|
Addressed the external-mask gap in 717b3e1. Took the fallback route: New test |
There was a problem hiding this comment.
The external-mask blocker is fixed in 717b3e129 following the author's update: masked batch searches now use the existing per-query ANN fallback, which preserves the caller's mask. The focused regression covers both plan selection and returned-row eligibility, and the earlier fragment-coverage and CPU-runtime fixes remain intact.
The remaining non-blocking risk is performance for low-overlap or very wide batches: batch CPU work is sequential within each dispatched job, while the benchmark does not compare against the former in-request per-query parallel plan. Treat the speedup as workload-dependent until low-overlap and query-count scaling are measured.
|
@wjones127 Rebased onto The rebase surfaced the merged external-mask feature (#7288): the shared batch path wasn't carrying a caller-supplied external row mask. Fixed in 717b3e1 — All checks pass except |
|
@wjones127 gentle ping on this one — it's approved and Every check is green except @BubbleCal looping you in since you've been active in the vector-index area and landing PRs lately — would either of you be able to kick off a re-run of the mac lane / land this when you get a chance? Thanks! |
#10443) ## Summary The shared-scan batch IVF node (`ANNIvfBatchExec`, #7640) required fixed probe bounds (`minimum_nprobes == maximum_nprobes`). Adaptive requests, including the default bounds and any explicit `search_effort`, fell back to one indexed search per query. This PR applies the single-query adaptive rules inside the batch node for IVF_FLAT, IVF_PQ, and IVF_SQ. Partition loads stay shared across queries. There are no changes to the index trait, the public API, or the index format. #10480, now merged, scores each loaded chunk's queries on separate CPU tasks. Before it, `search_partitions_batch` scored each chunk on a single CPU task, so routing adaptive bounds to the shared node showed up as a 3.3x slowdown on the existing `query_ann_batch` benchmark (768-d IVF_PQ, `k = 100`). The scanner changes are in `dataset/scanner/vector.rs`. `ANNIvfBatchExec` now lives in `io/exec/knn/batch.rs` and its adaptive tests in `dataset/tests/dataset_scanner_knn_adaptive.rs`, so no module grows past the size check. This follows up the min/max-nprobes question from the #7640 review. It builds on the segment-ownership fix in #9638. With #9618, namespace `multi_vector` requests that use adaptive bounds can now reach the shared path. ## Commits 1. `perf(vector): support adaptive nprobes in shared-scan batch IVF search` is the change itself. 2. `test(vector): move adaptive batch KNN tests into their own file` moves tests without changing them. 3. `fix(vector): align batch late search with single-query skip and stop rules` adds the selects-nothing skip, the budget-based ranking release, and the late-search helpers shared with the single-query path. 4. `refactor(vector): move ANNIvfBatchExec into its own module` moves code without changing it. Only imports and three intra-doc link paths change. 5. `Merge origin/main into perf/batch-ivf-adaptive-nprobes` merges `main` at `cd0a24899`; it had no conflicts. 6. `test(vector): cover hybrid scalar prefilter in adaptive batch KNN` adds the #10001 producer regression. ## Execution - Rank each query on the CPU runtime and apply the existing `AutoProbePolicy`, including `search_effort`. This is the same policy call the single-query path makes for each delta. - Search every query's initial budget in one shared call per delta. Finish the early phase for all deltas before deciding which queries need late search. - Keep only rankings that late search can still read. A query's ranking is released once it holds `k` candidates, holds every row the prefilter allows, or has no late range. Deltas with no remaining late work, including fixed, exhausted, and empty-ownership deltas, are released before the next delta is scanned. Retained deltas are checked again at the early-search barrier. - Run late search in rounds through the existing batch API, with one virtual query per (query, partition) pair. Each query consumes its partitions in rank order and stops where a sequential search would stop. Rows after that cut are discarded. - Admit whole queries per round, up to `max(1024, compute CPUs)` probes **across all queries and deltas**. The first query rotates between rounds so none starves. A query's deltas therefore advance together, and its rows do not depend on the rest of the batch. A query whose own waves exceed the cap is clipped and resumes at the first delta it could not fully admit. Admitted deltas run concurrently, and their results are consumed round-robin. - Score each loaded chunk on up to one CPU task per compute CPU, splitting its queries into contiguous ranges with at least 8 probes each when the chunk has enough. This applies to fixed and adaptive batches. - Apply segment ownership to early and late results and to the stopping budgets. Segments that own nothing skip late work, and so do segments whose own prefilter selects nothing. As in the single-query path (#10485), the latter are counted once in `segments_skipped`. The fewer-than-k shortcut returns only owned, unseen rows from eligible segments. - The single-query and batch late searches share the stop budget, the fewer-than-k shortcut condition, and the segment-ownership check (`LateSearchBudget`, `segment_owns_rows` in `knn.rs`), so the two paths cannot drift on when a query stops. - Keep the per-query fallback for explicit parallel adaptive probing (`query_parallelism > 1` or `-1`) and for adaptive IVF_RQ. Neither is equivalent to the shared sequential path today. Parallel late search can admit partitions past the sequential cut. Multi-bit IVF_RQ prunes with a probabilistic lower bound, so batch and single-query search can drop different rows (#10778). Fixed-probe IVF_RQ keeps its existing shared path and shows the same intermittent difference on `main`, independent of this PR. ## Memory - Per-query candidates stay in the bounded `TopHits` collector from #9688, so retained candidates are `Q * k`. `TopHits` gains an `iter()` so the fewer-than-k shortcut can skip rows it already holds. - For Float32 vectors, the temporary key copies in a late round hold at most `B * dim` floats, and a round returns at most `B * k` rows, where `B` is the round cap. With the default cap, a round of 1024-d queries copies at most 4 MiB of keys. - Adaptive rankings cover every partition. They are released per query once the query holds `k` candidates (or every row the prefilter allows) or has no late range, so an unfiltered batch does not keep `Q * num_partitions` ranking entries per delta after the delta's early scan. While a delta is being ranked, all of its queries' rankings are held at once. These bound the round scratch, not total execution memory. Ranked plans for genuine adaptive work, partition storage, and output have their own costs. Measured peak memory is under [Peak memory and latency](#peak-memory-and-latency). ## Behavioral scope For a single delta, shared adaptive FLAT/PQ/SQ follows the same sequential probe ranges and stopping rules as single-query search. Equal-distance ties can select different valid rows. With multiple deltas, results are consumed in a deterministic round-robin order rather than by completion timing, so identical row sets are not promised. The multi-delta tests check progression, uniqueness, counts, and recall. A query's rows do not depend on the other queries in the batch unless its own waves exceed the round cap. Batched exact refinement (#9353) covers fixed nprobes. Exact refinement with adaptive bounds keeps the per-query path, and `test_batch_knn_indexed_refine`'s `adaptive` case asserts that. **Metric change:** for adaptive batches that take the shared path, `partitions_searched` counts the shared scan's partition loads, as the fixed-probe batch path already does, instead of summing each query's searches. This includes late-search partitions loaded in a wave past a query's cut. Identical queries in one batch therefore count a partition once. The Python `test_vector_search_effort` batch assertion is updated to match. Adaptive IVF_RQ still reports the per-query sum. ## Tests - IVF_FLAT, IVF_PQ, IVF_SQ: batch results equal per-query results on single-delta fixtures. Cases cover unfiltered, selective, capped, oversized-maximum, fewer-than-k, and empty-filter queries. - `search_effort` at 0, 0.25, 0.75, and 1, with and without a filter: rows and distances match single-query search. Forcing the batch planner to ignore effort fails 18 of the 24 cases. - IVF_RQ: adaptive cases assert the per-query fallback and compare rows and distances with single-query search. Fixed-probe RQ still uses the shared node. - Distance ranges on IVF_FLAT and IVF_PQ, with and without a filter: rows and distances match single-query search. Narrow ranges leave queries short of `k` rows, so they exercise late search to exhaustion. - Cosine and dot: results equal single-query search, and recall is at least 0.5 against brute force, with and without filters. - Late search: deterministic tests cover row cuts, delta progression, ownership budgets, the fewer-than-k shortcut, and empty ownership. The cut test is arranged so that a row past the cut would displace a retained row from the bounded top-k if it were consumed. - Batch independence: a two-delta case where the round cap falls between a query's deltas returns the same rows as an uncapped query. Under per-(query, delta) admission, that query returned a farther row. - Round planning: cap and fairness tests cover partial waves, 1,025 queries, 129 deltas, and a cap of 1. A three-delta execution test records the actual batch calls. It asserts that each round admits only whole queries and never exceeds the cap. - Lifetime: weak-reference tests show that rankings, keys, and index handles for fixed and empty-ownership deltas are released before the next early delta, while state needed for adaptive late search stays alive. Rankings of queries that already hold `k` candidates are released, including when a later delta completes the query. So are rankings of queries that hold every row the prefilter allows. - Chunk scoring: range splitting is tested for empty, small, even, capped, skewed, and trailing-empty probe counts, and every query with probes belongs to exactly one range. All batch-vs-single equality tests run with the parallel scorer, including with `LANCE_CPU_THREADS=1`. - A delta whose own prefilter selects nothing (the filter keeps only rows of a newer delta): batch search reports the same `segments_skipped` as single-query search and returns the same rows. A unit test checks that such a delta does not join late search. Reverting the skip fails both. - `LateSearchBudget` and `segment_owns_rows`: unit tests cover allow lists below, at, and above `k`, empty masks, and masks that cannot count their rows (block lists, whole fragments). - `query_parallelism` 0 and 1 take the shared path, and their rows and distances match single-query search. Explicit parallel probing keeps the per-query path. - Existing HNSW, zero-probe, external-mask, overlay, and coverage fallbacks are unchanged. #9353's `test_batch_knn_indexed_refine` still requires exact refinement with adaptive bounds to take the per-query path. ## Validation The integrated head merges `main` at `cd0a24899` (#10753) into the branch in `2f63075ad`. Git merged cleanly, and the result keeps #10001's hybrid scalar prefilter producer, #10721's `lance_core::utils::tokio` spawn boundary, and #10727's extracted `partition_search` scoring; no old inline implementation was restored. - `cargo test --profile ci -p lance --lib dataset_scanner_knn_adaptive::`: 74 passed. - `cargo test --profile ci -p lance --lib dataset_scanner_knn::`: 152 passed. - `cargo test --profile ci -p lance --lib io::exec::knn::`: 1,219 passed, covering the batch, late-search, ownership, segment-contract, and budget units. - `cargo test --profile ci -p lance --lib dataset_scanner_segments::test_hybrid_prefilter_consumers`: 12 passed, so #10001's ANN, batch, and FTS consumers keep their union prefilter. - `cargo test --profile ci -p lance --lib io::exec::prefilter_union::` and `--test integration_tests prefilter_union`: 2 and 16 passed. - `cargo test --profile ci -p lance-index --lib -- minhash`: 63 passed. - `cargo clippy --profile ci --all --tests --benches -- -D warnings` and `cargo fmt --all -- --check`: clean. - Python `test_vector_index.py -k "search_effort or batch"` against a native build of the integrated head: 68 passed. - `uv run make lint` from `python/`: clean. The integration adds `test_batch_knn_adaptive_hybrid_scalar_prefilter`, which reaches the adaptive batch scan through #10001's `PrefilterUnionExec`. An exact B-tree predicate on `i` (`i >= 250 AND i < 1750`) covers both producer shapes: a scalar index over every fragment while the scan selects the middle two (`scalar_fragments=2, evaluated_fragments=2`), and a scalar index over two of four fragments (`scalar_fragments=2, evaluated_fragments=4`). Each case asserts that the union producer and `ANNIvfBatch` are planned and `ANNSubIndex` is not, that per-query rows and distances equal single-query search, that every returned row satisfies the predicate and stays inside the selected fragments, and a recall floor of 0.5 against the flat oracle. ### Author-reported validation before integration Recorded against `58e62bd54`, which includes #10695 (search-effort calibration by metric and k) and #10761 (segment contract checks), before this branch was merged with `main`: - `cargo test -p lance --lib`: 10,075 passed, 0 failed, 9 ignored. - Batch, adaptive, `search_effort`, range-splitting, and late-search budget tests with `LANCE_CPU_THREADS=1` and `LANCE_IVF_STREAMING_SEARCH_BATCH_SIZE=1`: 1,049 passed. - `cargo test -p lance-index --lib -- minhash`: 63 passed. - Python `test_vector_index.py -k "search_effort or batch"` on a fresh `make build`: 68 passed. - `cargo clippy --all --tests --benches -- -D warnings`, `cargo clippy -p lance --all-features --tests -- -D warnings`, `cargo fmt --all -- --check`, and `RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps`: clean. On aarch64 macOS, clippy also needs `-A unreachable_code` for the NEON byte-stream-split code already on `main`. - With the selects-nothing skip reverted, the end-to-end `segments_skipped` test and the late-search unit test fail. With the ranking release reverted to `k`, the budget retention test fails. - Earlier, on `0db573921` before the rebase, the full Python suite (`pytest python/tests -n auto`) passed 1,902 tests. ## Benchmark suite Everything in this section and the next two is author-reported: it was measured on the author's hardware against the revisions named below and was not re-run on the integrated head. `python/python/benchmarks/test_search.py` gains a `fixed` (`nprobes = 10`) and an `adaptive` (default bounds) case for the two `query_ann_batch` benchmarks. These are pylance release builds of this branch, measured before it was rebased onto `0db573921`, and of `main` at `65df9b7c4`, run on one Apple M5 Max. The dataset has 100k x 768-d vectors on local disk, IVF_PQ with 32 partitions, `k = 100`, and 32 queries. Values are pytest-benchmark medians in ms. | Benchmark | `main` | This PR | |---|---|---| | `test_batch_ann_search[clean-adaptive]` | 10.20 | 6.14 | | `test_batch_ann_search[clean-fixed]` | 12.68 | 4.53 | | `test_batch_ann_search[with_delete_files-adaptive]` | 11.10 | 7.00 | | `test_batch_ann_search[with_delete_files-fixed]` | 16.71 | 5.04 | | `test_batch_ann_search[with_new_rows-adaptive]` | 57.75 | 55.78 | | `test_batch_ann_search[with_new_rows-fixed]` | 53.80 | 50.15 | | `test_repeated_single_ann_search[clean-adaptive]` | 46.66 | 46.01 | | `test_repeated_single_ann_search[clean-fixed]` | 26.98 | 27.12 | `with_new_rows` has unindexed fragments, so it keeps the per-query path. The repeated single-query cases do not use the changed code and serve as a noise reference. Without #10480, `clean-adaptive` took 33.67 ms. ## Peak memory and latency This is a release-build measurement of this branch, taken before it was rebased onto `0db573921`, and of `main` at `65df9b7c4`. It is not checked in. The dataset is on local SSD: 500k x 16-d Float32 vectors, IVF_FLAT with 2,048 partitions, either as one index segment or as four (an initial index plus three appended deltas indexed with `optimize_indices`). Queries are uniform random vectors drawn independently of the data, with `k = 10` and the default adaptive bounds. A batch of 256 queries has low partition overlap; 8,192 queries puts the per-query rankings at about 134 MB per segment. Each configuration runs in a fresh process. Peak growth is the process high-water RSS (`getrusage` `ru_maxrss`) after the query minus its value just before. Values are the median of 3 runs on one Apple M5 Max. On `main`, an adaptive batch runs one indexed search per query inside the same plan. | Segments | Queries | Filter | Peak RSS growth (`main` → this PR) | Latency (`main` → this PR) | |---|---|---|---|---| | 1 | 256 | none | 133 MB → 34 MB | 56 ms → 56 ms | | 1 | 256 | `i % 97 = 0` | 507 MB → 36 MB | 128 ms → 58 ms | | 1 | 8,192 | none | 2,172 MB → 208 MB | 2,150 ms → 297 ms | | 1 | 8,192 | `i % 97 = 0` | 14,448 MB → 195 MB | 4,994 ms → 311 ms | | 4 | 256 | none | 316 MB → 45 MB | 162 ms → 143 ms | | 4 | 256 | `i % 97 = 0` | 677 MB → 46 MB | 236 ms → 149 ms | | 4 | 8,192 | none | 5,716 MB → 226 MB | 3,693 ms → 1,023 ms | | 4 | 8,192 | `i % 97 = 0` | 16,961 MB → 277 MB | 6,320 ms → 1,099 ms | Both builds return the same number of rows in every configuration. On `main`, the baseline already includes planning the per-query searches, so its growth is understated. Object-store latency is measured separately below. ## Object-store latency This measures release builds of `main` at `56963cb29` and of this PR at `58725f2fe`, through the Java binding on a Spark cluster in Azure: 8 executors with 4 cores each, data on Azure Data Lake Storage (ABFS). The dataset has 10M x 1024-d embeddings in 10 fragments, with one IVF_PQ segment: L2, 256 partitions, and 64 sub-vectors. Queries are dataset rows with 5% Gaussian noise, `k = 10`, and the Java `Query` sets no nprobes, so probing is adaptive from a minimum of 1 with no maximum. Each of 32 Spark tasks sends its share of the queries as one batch call (batch) or as one call per query (per-query). Values are the median wall time of 5 runs after a warmup, with the best run in parentheses, in ms. Each build ran in its own job on a shared cluster. | Filter | Queries | `main` batch | This PR batch | Per-query (`main` / this PR) | |---|---|---|---|---| | none | 1,000 | 1,204 (1,175) | 1,190 (1,172) | 1,360 / 1,309 | | none | 4,000 | 4,370 (4,153) | 3,835 (3,758) | 4,586 / 4,190 | | `rid % 20000 = 0` (500 rows) | 1,000 | 6,236 (6,091) | 1,294 (1,246) | — | | `rid % 20000 = 0` (500 rows) | 4,000 | 19,786 (19,510) | 3,469 (3,403) | — | - The per-query path does not change in this PR, so its column shows the difference between jobs: 4–9%. Without a filter, the shared path is within that range of `main`. - With the filter, about 2 matching rows fall in each partition, so queries need late search. The shared path is 4.8x faster at 1,000 queries and 5.7x faster at 4,000. Per-query search was not run with the filter. - Recall@10 against an exact ground truth (with the same filter) over 100 queries is the same for both builds: 0.726 without the filter and 0.553 with it. Without the filter, per-query and batch rows agree for every recall query in both builds. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io>
Which issue does this PR close?
Closes #6822.
Rationale for this change
Batch vector search (#6821, PR #6828) made indexed multi-query search work by looping the full single-query plan once per query vector — re-opening the index and rebuilding the prefilter for every query — and unioning the results. That throws away the main opportunity of a batch: query vectors mostly probe overlapping IVF partitions, so the same partition storage is read and decoded many times. This PR shares that index-level state across the batch.
What changes are included in this PR?
The indexed/ANN path now reads each probed IVF partition's storage once and scores every query that probes it, with the prefilter built once and shared.
VectorIndextrait (lance-index): defaultedsupports_batch_partition_search()+search_partitions_batch(...)(default returnsnot_supported). Additive — no breaking change.IVFIndex(ivf/v2.rs): batch search for flat-style sub-indices (IVF_FLAT/PQ/SQ/RQ). Invert per-query partition lists, load each distinct partition once, accumulate one top-k heap per query, reusingaccumulate_prepared_partition_search/global_heap_to_batch.ANNIvfBatchExec(io/exec/knn.rs): ranks each query against the centroids, runs the shared-scan batch search per delta, merges per-query top-k across deltas, emits{query_index, _distance, _rowid}. Prefilter wiring shared with the single-query node viabuild_dataset_prefilter.normalize_batch_query_for_index).Fallback matrix (no regression):
refine_factor/ IVF_HNSW_* / mixed indexed+unindexedDesign notes (anticipating review):
build_dataset_prefilter, the index's per-partition accumulate) and leaves the single-query nodes untouched.supports_batch_partition_search()? The gate is a planning-time decision and the single-query path likewise doesn't open the index there;derive_vector_index_typereads metadata with no I/O. The opened index re-checks the trait as a defensive invariant.minimum_nprobespartitions/query, but the single-query path is adaptive (early_pruningfloor + late-search expansion), so the two only match when nprobes is fixed. The fast path is gated tominimum_nprobes == maximum_nprobes; adaptive nprobes falls back to the per-query loop (verified: an unpinned batch diverged on every query before the gate, 0 divergence after). Open question: fixed-nprobes-first with batched early/late as a follow-up, or the full adaptive path in one PR?search_partitions) uses, widened to the batch's partition union; per-delta output is k-bounded, so cross-delta accumulation is O(deltas × k).Are these changes tested?
Yes.
cargo test -p lance --lib test_batch_knn— 15 tests: plan shape, exact batch-vs-repeated-single equivalence (nprobes pinned), cosine regression, shared prefilter, multi-delta cross-delta merge, and explicit fallbacks for refine, adaptive nprobes, and IVF_HNSW (acceptance criterion: "unsupported index types have explicit behavior and tests").cargo test -p lance --lib dataset::scanner::test::test_knn(29) — no single-query regression (exercises the sharedbuild_dataset_prefilter).cargo fmt --all&&cargo clippy -p lance -p lance-index --tests --benches -- -D warnings.pytest -k batch(L2 + cosine × three/single queries); index-type sweep (IVF_FLAT/PQ/SQ/HNSW_PQ/HNSW_SQ) matches repeated single-query;ruffclean;pyrightclean on changed lines.benchmarks/test_search.py): batch vs repeated-single ANN; local run (50k rows, dim 128, IVF_PQ 64 partitions, m=32, k=10, nprobes=10) → ~2.5× faster (ratio; absolute ms machine-dependent).Are there any user-facing changes?
No API changes. The batch-query surface (2-D / list query with a
query_indexoutput column) already shipped in #6828; this PR only changes how indexed batch queries execute, so Python and Java bindings are unaffected. It is a performance improvement for fixed-nprobes indexed batch search; results are identical to issuing the queries one at a time. No format or compatibility change.