Skip to content

feat(index): share IVF partition scans across batch vector queries - #7640

Merged
wjones127 merged 7 commits into
lance-format:mainfrom
sezruby:knn-batch-6822
Sep 12, 2026
Merged

wjones127 merged 7 commits into
lance-format:mainfrom
sezruby:knn-batch-6822

Conversation

@sezruby

@sezruby sezruby commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6822.

Rationale for this change

Batch vector search (#6821, PR #6828) made indexed multi-query search work by looping the full single-query plan once per query vector — re-opening the index and rebuilding the prefilter for every query — and unioning the results. That throws away the main opportunity of a batch: query vectors mostly probe overlapping IVF partitions, so the same partition storage is read and decoded many times. This PR shares that index-level state across the batch.

What changes are included in this PR?

The indexed/ANN path now reads each probed IVF partition's storage once and scores every query that probes it, with the prefilter built once and shared.

  • VectorIndex trait (lance-index): defaulted supports_batch_partition_search() + search_partitions_batch(...) (default returns not_supported). Additive — no breaking change.
  • IVFIndex (ivf/v2.rs): batch search for flat-style sub-indices (IVF_FLAT/PQ/SQ/RQ). Invert per-query partition lists, load each distinct partition once, accumulate one top-k heap per query, reusing accumulate_prepared_partition_search / global_heap_to_batch.
  • ANNIvfBatchExec (io/exec/knn.rs): ranks each query against the centroids, runs the shared-scan batch search per delta, merges per-query top-k across deltas, emits {query_index, _distance, _rowid}. Prefilter wiring shared with the single-query node via build_dataset_prefilter.
  • Each query vector is normalized independently for cosine (normalize_batch_query_for_index).

Fallback matrix (no regression):

Case Behavior
IVF_FLAT/PQ/SQ/RQ, fixed nprobes, fully indexed shared-scan fast path
adaptive nprobes / refine_factor / IVF_HNSW_* / mixed indexed+unindexed per-query indexed loop (exact)

Design notes (anticipating review):

  • Why a new exec node, not a mode on the existing ANN nodes? The two-node single-query pipeline streams one partition-list per delta through a per-query top-k; sharing the scan requires inverting queries onto partitions and keeping one heap per query in a single pass — a different dataflow. The new node reuses the underlying primitives (partition load, build_dataset_prefilter, the index's per-partition accumulate) and leaves the single-query nodes untouched.
  • Why gate on index-type metadata, not supports_batch_partition_search()? The gate is a planning-time decision and the single-query path likewise doesn't open the index there; derive_vector_index_type reads metadata with no I/O. The opened index re-checks the trait as a defensive invariant.
  • nprobes gate (correctness). The shared path searches exactly minimum_nprobes partitions/query, but the single-query path is adaptive (early_pruning floor + late-search expansion), so the two only match when nprobes is fixed. The fast path is gated to minimum_nprobes == maximum_nprobes; adaptive nprobes falls back to the per-query loop (verified: an unpinned batch diverged on every query before the gate, 0 divergence after). Open question: fixed-nprobes-first with batched early/late as a follow-up, or the full adaptive path in one PR?
  • Memory. Peak = the union of probed partitions held during scoring — the same buffering the existing single-query global-heap path (search_partitions) uses, widened to the batch's partition union; per-delta output is k-bounded, so cross-delta accumulation is O(deltas × k).

Are these changes tested?

Yes.

  • cargo test -p lance --lib test_batch_knn — 15 tests: plan shape, exact batch-vs-repeated-single equivalence (nprobes pinned), cosine regression, shared prefilter, multi-delta cross-delta merge, and explicit fallbacks for refine, adaptive nprobes, and IVF_HNSW (acceptance criterion: "unsupported index types have explicit behavior and tests").
  • cargo test -p lance --lib dataset::scanner::test::test_knn (29) — no single-query regression (exercises the shared build_dataset_prefilter).
  • cargo fmt --all && cargo clippy -p lance -p lance-index --tests --benches -- -D warnings.
  • Python: pytest -k batch (L2 + cosine × three/single queries); index-type sweep (IVF_FLAT/PQ/SQ/HNSW_PQ/HNSW_SQ) matches repeated single-query; ruff clean; pyright clean on changed lines.
  • Benchmark (benchmarks/test_search.py): batch vs repeated-single ANN; local run (50k rows, dim 128, IVF_PQ 64 partitions, m=32, k=10, nprobes=10) → ~2.5× faster (ratio; absolute ms machine-dependent).

Are there any user-facing changes?

No API changes. The batch-query surface (2-D / list query with a query_index output column) already shipped in #6828; this PR only changes how indexed batch queries execute, so Python and Java bindings are unaffected. It is a performance improvement for fixed-nprobes indexed batch search; results are identical to issuing the queries one at a time. No format or compatibility change.

@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer enhancement New feature or request labels Jul 6, 2026
@codecov

codecov Bot commented Jul 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@wjones127
wjones127 self-requested a review July 23, 2026 21:46
@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds indexed ANN batch vector search with shared IVF partition reads, per-query result grouping, cosine-safe normalization, correctness-gated scanner planning, fallback coverage, equivalence tests, and batch-versus-repeated-search benchmarks.

Changes

Indexed ANN batch search

Layer / File(s) Summary
Batch search contracts
rust/lance-index/src/vector.rs
Adds optional VectorIndex batch partition-search capability with an explicit unsupported default.
IVF partition execution
rust/lance/src/index/vector/ivf/v2.rs
Implements shared partition loading and independent per-query top-k accumulation for supported IVF sub-indexes.
Batch KNN execution
rust/lance/src/io/exec/knn.rs
Adds batch execution planning, cosine query normalization, shared prefilters, candidate aggregation, and grouped output assembly.
Scanner planning and validation
rust/lance/src/dataset/scanner.rs, rust/lance/src/dataset/tests/dataset_overlay_index_masking.rs, python/python/tests/test_vector_index.py
Gates the shared path on index and probe conditions, validates fallback behavior for unsupported cases and overlays, and compares indexed batch results with repeated single-query results.
ANN benchmarks
python/python/benchmarks/test_search.py
Adds benchmarks for one batched ANN request and repeated single-query ANN requests.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Scanner
  participant ANNIvfBatchExec
  participant IVFIndex
  Client->>Scanner: batch nearest query
  Scanner->>ANNIvfBatchExec: plan indexed batch search
  ANNIvfBatchExec->>IVFIndex: search multiple queries across partitions
  IVFIndex-->>ANNIvfBatchExec: per-query top-k results
  ANNIvfBatchExec-->>Scanner: grouped query_index results
  Scanner-->>Client: batch result table
Loading

Possibly related PRs

Suggested labels: performance

Suggested reviewers: wjones127, bubblecal, jackye1995

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: sharing IVF partition scans for batch vector queries.
Description check ✅ Passed The description matches the implemented batch ANN/indexed search sharing, fallbacks, tests, and benchmarks.
Linked Issues check ✅ Passed The changes satisfy #6822 by adding indexed batch search, shared scans, explicit fallbacks, equivalence tests, and benchmark coverage.
Out of Scope Changes check ✅ Passed The file changes are aligned with the PR goals and do not show unrelated or extra-scope additions.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (1)
python/python/tests/test_vector_index.py (1)

240-241: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Seed the RNG so equivalence failures are reproducible.

This test asserts exact id equality between the shared-scan batch path and repeated single-query search over an approximate IVF_PQ index. With unseeded np.random.randn, any tie-break or ordering discrepancy (see the HashMap scoring-order note in rust/lance/src/index/vector/ivf/v2.rs) surfaces as an intermittent failure that cannot be replayed.

♻️ Suggested change
+    rng = np.random.default_rng(42)
     scales = np.linspace(0.1, 10.0, query_count).reshape(-1, 1)
-    queries = (np.random.randn(query_count, 128) * scales).astype(np.float32)
+    queries = (rng.standard_normal((query_count, 128)) * scales).astype(np.float32)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/python/tests/test_vector_index.py` around lines 240 - 241, Seed
NumPy’s random number generator before the randomized query construction in the
test containing the shared-scan and repeated single-query equivalence
assertions. Use a fixed, explicit seed so failures involving exact id ordering
are deterministic and reproducible, while leaving the existing query generation
and assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance/src/dataset/scanner.rs`:
- Around line 4089-4096: Update the refine-factor gate in the batch
indexed-search path to reject any Some value, not only factors greater than one,
so refine(1) and refine(0) fall back to the per-query loop for reranking and
validation. Extend test_batch_knn_indexed_refine_falls_back to cover both refine
factors while preserving the existing refine(2) coverage.
- Around line 4137-4158: Update the batch fast-path gate in the surrounding
scanner method to fall back whenever
self.overlay_stale_vector_rows(index_segments)? is non-empty, before
constructing the batch executor. Preserve the existing batch behavior when no
stale overlay rows exist, and add coverage using an overlaid fragment that
verifies execution takes the per-query fallback path.

In `@rust/lance/src/index/vector/ivf/v2.rs`:
- Around line 1920-1938: The batch search flow around load_parallelism and the
try_collect::<Vec<_>>() currently retains every probed partition before scoring.
Replace this all-at-once collection with bounded-window processing, using
STREAMING_SEARCH_BATCH_SIZE or the existing bounded-channel pattern: load one
window, dispatch its scoring through spawn_cpu, release partition entries, then
continue until all assignments are processed while preserving result ordering
and error propagation.
- Around line 1902-1916: Update OrderedNode::cmp to use row_id as the final
comparison key after distance, ensuring equal-distance candidates are
deterministically selected and ordered. Preserve the existing heap behavior and
make the resulting global_heap_to_batch output match repeated single-query
ordering.

In `@rust/lance/src/io/exec/knn.rs`:
- Around line 2401-2411: Update the per-query batch handling around the loop
over per_query: first validate that the returned batch count matches the query
count, returning a diagnosable Internal error on mismatch before indexing
candidates. Then replace positional batch.column access with column_by_name
using the schema’s distance and row-id field names, preserving the existing
candidate extension behavior.

---

Nitpick comments:
In `@python/python/tests/test_vector_index.py`:
- Around line 240-241: Seed NumPy’s random number generator before the
randomized query construction in the test containing the shared-scan and
repeated single-query equivalence assertions. Use a fixed, explicit seed so
failures involving exact id ordering are deterministic and reproducible, while
leaving the existing query generation and assertions unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: c6efabac-70c1-4356-8665-8f6423a9f771

📥 Commits

Reviewing files that changed from the base of the PR and between 5deddec and e0f8167.

📒 Files selected for processing (6)
  • python/python/benchmarks/test_search.py
  • python/python/tests/test_vector_index.py
  • rust/lance-index/src/vector.rs
  • rust/lance/src/dataset/scanner.rs
  • rust/lance/src/index/vector/ivf/v2.rs
  • rust/lance/src/io/exec/knn.rs

Comment thread rust/lance/src/dataset/scanner.rs Resolved
Comment thread rust/lance/src/dataset/scanner.rs Resolved
Comment on lines +1902 to +1916
// Invert the per-query partition lists so each distinct partition is
// loaded once and scored against every query that probes it.
let mut assignments: HashMap<u32, Vec<(usize, f32)>> = HashMap::new();
for (query_index, (parts, dists)) in partitions_per_query
.iter()
.zip(q_c_dists_per_query.iter())
.enumerate()
{
for (part_id, dist_q_c) in parts.values().iter().zip(dists.values().iter()) {
assignments
.entry(*part_id)
.or_default()
.push((query_index, *dist_q_c));
}
}

@coderabbitai coderabbitai Bot Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Does the global top-k heap impose a total order that includes the row id?
rg -nP -C6 'struct OrderedNode|impl .*Ord.* for OrderedNode' --type=rust
ast-grep run --pattern 'fn global_heap_to_batch($$$) { $$$ }' --lang rust rust/lance/src/index/vector/ivf/v2.rs

Repository: lance-format/lance

Length of output: 156


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== files =="
git ls-files 'rust/lance/src/index/vector/ivf/v2.rs' 'rust/lance/src/dataset/scanner.rs' | sed -n '1,20p'

echo "== outline around top of v2.rs =="
wc -l rust/lance/src/index/vector/ivf/v2.rs rust/lance/src/dataset/scanner.rs
ast-grep outline rust/lance/src/index/vector/ivf/v2.rs --length 200 | sed -n '1,200p'

echo "== search relevant terms =="
rg -n "assignments|partitions_per_query|global_heap_to_batch|TestVectorDataset|assert_batch_matches_single_queries|OrderedNode|OrderedCandidate|global_top|topk" rust/lance/src -S

echo "== lines 1860-1945 in ivf/v2.rs =="
sed -n '1860,1945p' rust/lance/src/index/vector/ivf/v2.rs

echo "== lines around scanner tests =="
rg -n "assert_batch_matches_single_queries|TestVectorDataset" rust/lance/src/dataset/scanner.rs -A 80 -B 20

Repository: lance-format/lance

Length of output: 548


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== search relevant terms =="
rg -n "assignments|partitions_per_query|global_heap_to_batch|TestVectorDataset|assert_batch_matches_single_queries|OrderedNode|OrderedCandidate|global_top|topk" rust/lance/src -S || true

echo "== lines 1860-1945 in ivf/v2.rs =="
sed -n '1860,1945p' rust/lance/src/index/vector/ivf/v2.rs

echo "== lines around scanner tests =="
rg -n "assert_batch_matches_single_queries|TestVectorDataset" rust/lance/src/dataset/scanner.rs -A 100 -B 20 || true

echo "== import headers for v2.rs =="
sed -n '1,120p' rust/lance/src/index/vector/ivf/v2.rs

Repository: lance-format/lance

Length of output: 50375


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== ivf/v2.rs relevant scoring/heap/export lines =="
sed -n '800,935p' rust/lance/src/index/vector/ivf/v2.rs
sed -n '1945,2025p' rust/lance/src/index/vector/ivf/v2.rs

echo "== OrderedNode definition and Ord impl =="
rg -n -C 12 'struct OrderedNode|BinaryHeap<OrderedNode|Ord for Ordered|impl PartialOrd|impl PartialEq' rust/lance/src -S | sed -n '1,240p'

echo "== relevant tests around batch indexed equivalence =="
sed -n '6950,7825p' rust/lance/src/dataset/scanner.rs

Repository: lance-format/lance

Length of output: 50374


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== exact OrderedNode references in graph submodule =="
rg -n -C 20 'OrderedNode' rust/lance/src/index/vector/graph.rs rust/lance/src/index/vector -S | sed -n '1,260p'

echo "== ivf.rs OrderedNode/topk area =="
sed -n '3980,4095p' rust/lance/src/index/vector/ivf.rs

echo "== Behavioral probe: Rust BinaryHeap Ord ties (if cargo availability) =="
if cargo --version >/dev/null 2>&1 && rustc --version >/dev/null 2>&1; then
  tmp="$(mktemp -d)"
  cat > "$tmp/heap_order_probe.rs" <<'RS'
fn main() {
    let mut heap = std::collections::BinaryHeap::new();
    for row_id in 0u64..10 {
        heap.push((0.0f32, row_id));
    }
    let out: Vec<_> = heap.into_iter().map(|(d, id)| format!("({}, {})", d, id)).collect();
    for (i, item) in out.iter().enumerate() {
        println!("heap_order_probe.{}. {}", i, item);
    }
}
RS
  cargo run --manifest-path "$tmp/HeapOrderProbeCargo.toml" 2>/dev/null || (
    cat > "$tmp/Cargo.toml" <<'TOML'
[package]
name = "heap_order_probe"
version = "0.0.0"
edition = "2021"
TOML
    mkdir -p "$tmp/src"
    mv "$tmp/heap_order_probe.rs" "$tmp/src/main.rs"
    cargo run --manifest-path "$tmp/Cargo.toml"
  )
else
  echo "cargo/rustc unavailable"
fi

Repository: lance-format/lance

Length of output: 14303


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate graph.rs =="
git ls-files | rg '(^|/)graph\.rs$|lance_index/src|index/vector' | sed -n '1,120p'

echo "== graph.rs OrderedNode definitions/usages =="
while IFS= read -r f; do
  rg -n -C 8 'struct OrderedNode|impl .*OrderedNode|derive\(.*Ord.*\)|impl PartialOrd|impl Ord|BinaryHeap<OrderedNode' "$f"
done < <(git ls-files | rg '(^|/)graph\.rs$')

echo "== ivf storage topk calls into graph =="
rg -n -C 12 'accumulate_topk_with_scratch|topk|OrderedNode' rust/lance/src/index/vector -S | sed -n '1,260p'

Repository: lance-format/lance

Length of output: 1335


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== rust/lance-index/src/vector/graph.rs OrderedNode area =="
sed -n '1,180p' rust/lance-index/src/vector/graph.rs

echo "== binary heap tie ordering probe =="
python3 - <<'PY'
# Rust std::collections::BinaryHeap iterates from the maximum heap element;
# equal-order elements can emerge in different relative orders.
import random
heap = []
for row_id in range(9):
    random.shuffle(heap)
    heap.append((0.0, row_id))
    heap.sort(reverse=True)

seen = set()
for _ in range(200):
    q = heap.copy()
    out = []
    while q:
        _, row_id = q.pop()
        out.append(row_id)
    seen.add(tuple(out))
print("seen orderings:", len(seen))
PY

Repository: lance-format/lance

Length of output: 4451


Use a deterministic tie-break for batch top-k results.

OrderedNode only orders by distance, so equal-distance candidates are not deterministically selected or ordered before global_heap_to_batch emits them. With ties present, the shared batch path can produce the same top-k set in a different order than repeated single-query search, causing the indexed equivalence assertions to become flaky. Add the row id as the final comparison key in OrderedNode::cmp, or sort the heap output by (distance, row_id) before returning the batch.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/index/vector/ivf/v2.rs` around lines 1902 - 1916, Update
OrderedNode::cmp to use row_id as the final comparison key after distance,
ensuring equal-distance candidates are deterministically selected and ordered.
Preserve the existing heap behavior and make the resulting global_heap_to_batch
output match repeated single-query ordering.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed the underlying nondeterminism, though not where the heap comment pointed. search_partitions_batch accumulated partitions in HashMap iteration order, so when rows tie at the k-th distance, which one survives the capped per-query heap varied run to run — fixed with loaded.sort_by_key(part_id) for a stable scan order (recall-neutral: any tied row is an equally valid k-th neighbor).

On emit order specifically: both global_heap_to_batch consumers already re-sort by (distance, row_id) — a total order — before truncating, so the final ordering was never user-visibly nondeterministic. I therefore did not add a sort inside global_heap_to_batch itself, since it would be redundant and would also tax the shared single-query path.

New test test_batch_knn_indexed_orders_ties_by_row_id pins the end-to-end guarantee: five neighbors tied at distance 0 come back in ascending-row-id order, and the batch result matches repeated single-query search (both stable/unstable row-id modes).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== current batch partition ordering code =="
rg -n -C 10 'loaded\.sort_by_key|let loaded =|search_partitions_batch' \
  rust/lance/src/index/vector/ivf/v2.rs

echo "== tie-order regression test =="
rg -n -C 12 'test_batch_knn_indexed_orders_ties_by_row_id' rust/lance --glob '*.rs'

echo "== current changed files relevant to the reported fix =="
git diff -- rust/lance/src/index/vector/ivf/v2.rs rust/lance/src/dataset/scanner.rs

Length of output: 2686


The checkout currently does not show either reported addition. I’ll make one non-failing inspection to distinguish an out-of-date checkout from a missing change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is on the current PR head (135df234f): the tie-break sort is at rust/lance/src/index/vector/ivf/v2.rs:1946 (loaded.sort_by_key(|(part_id, _, _)| *part_id)), and the regression test is test_batch_knn_indexed_orders_ties_by_row_id in rust/lance/src/dataset/scanner.rs:7472. Looks like the analysis ran against an out-of-date checkout.

Comment thread rust/lance/src/index/vector/ivf/v2.rs Outdated
Comment on lines +1920 to +1938
// Load each distinct partition's storage exactly once. This shared I/O
// is the whole point of batch search versus repeated single queries.
let load_parallelism = get_num_compute_intensive_cpus().max(1);
let load_index = self.clone();
let load_metrics = metrics.clone();
let loaded = stream::iter(assignments)
.map(move |(part_id, probing_queries)| {
let index = load_index.clone();
let metrics = load_metrics.clone();
async move {
let part_entry = index
.load_partition(part_id as usize, true, metrics.as_ref())
.await?;
Result::Ok((part_id as usize, part_entry, probing_queries))
}
})
.buffered(load_parallelism)
.try_collect::<Vec<_>>()
.await?;

@coderabbitai coderabbitai Bot Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift

All probed partitions are materialized before any scoring — unbounded memory for wide batches.

try_collect::<Vec<_>>() holds every distinct probed partition's storage alive at once. For a batch of q queries with nprobes probes each, that is up to min(q * nprobes, num_partitions) partitions resident, and the Arc clones also pin those entries in the index cache. The single-query path deliberately avoids this by streaming prepared partitions through a bounded channel (STREAMING_SEARCH_BATCH_SIZE, Line 1663) after #7642.

Consider processing the load stream in chunks (load a bounded window, dispatch scoring for it via spawn_cpu, drop the entries, repeat) so peak memory is bounded by the window rather than by the probe fan-out.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/index/vector/ivf/v2.rs` around lines 1920 - 1938, The batch
search flow around load_parallelism and the try_collect::<Vec<_>>() currently
retains every probed partition before scoring. Replace this all-at-once
collection with bounded-window processing, using STREAMING_SEARCH_BATCH_SIZE or
the existing bounded-channel pattern: load one window, dispatch its scoring
through spawn_cpu, release partition entries, then continue until all
assignments are processed while preserving result ordering and error
propagation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed this is real — materializing all probed partitions before scoring is unbounded for wide batches. It's a larger change to the accumulation model (streaming / bounded partition loading) than the other findings, so I'm tracking it separately rather than expanding this PR's scope.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sezruby, agreed. Deferring the bounded-window loading/scoring redesign is reasonable to keep this PR focused. The finding remains open for the follow-up work.

Would you like me to create a GitHub issue with the affected path, rationale, and acceptance criteria?

🐇📦

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this is being hooked into the Scanner already, I don't think we should defer this. As written, this would be a serious regression in memory use, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8109f6f — not deferred.

search_partitions_batch now streams the loaded partitions through scoring in STREAMING_SEARCH_BATCH_SIZE chunks instead of try_collect-ing them all first. Each chunk is scored into the per-query top-k heaps in one spawn_cpu dispatch and then dropped, so resident partition storage is bounded to the load window plus one chunk rather than scaling with the batch width (previously up to min(query_count * nprobes, num_partitions) — potentially the whole index).

Determinism is preserved: the distinct partitions are sorted by part_id and buffered keeps that order, so the across-partition tie-break at the k-th distance is unchanged. The partition-loading awaits stay in the async loop — only pure-CPU scoring goes to spawn_cpu — per the pool-safety constraint in #7642.

While here I also fixed a related divergence: nprobes(0) (min == max == 0) passed the fixed-nprobes gate, but the node clamped it up to one partition while the single-query path probes nothing and returns empty. The scanner now falls back when minimum_nprobes == 0, and the .max(1) clamp is replaced with a debug_assert! documenting the invariant.

Tests: test_batch_knn_indexed_streams_multiple_chunks (a wide batch spanning multiple chunks, pinned to repeated single-query search) and test_batch_knn_indexed_zero_nprobes_falls_back.

Comment thread rust/lance/src/io/exec/knn.rs Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
rust/lance/src/io/exec/knn.rs (1)

2374-2382: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

.max(1) silently clamps nprobes(0), breaking the equivalence the batch path is gated on.

Scanner::nprobes(0) is not rejected (scanner.rs Line 1634 sets both bounds with no validation), so minimum_nprobes == maximum_nprobes == Some(0) passes batch_index_search_supported. The single-query path then reaches IVFIndex::find_partitions with max_nprobes = 0 and probes nothing (empty result), while this node clamps to 1 and returns a partition's worth of neighbors. That is exactly the batch-vs-repeated-single-query divergence the fixed-nprobes gate exists to rule out, and it silently adjusts a caller-supplied value.

Either propagate minimum_nprobes unchanged here, or make the scanner gate fall back when minimum_nprobes == 0 (arguably nprobes(0) should be rejected outright at the builder, like refine(0) is).

As per coding guidelines, "Validate inputs at API boundaries and reject invalid values with descriptive errors; never silently clamp or adjust them."

🐛 Proposed fix
-                    let nprobes = single_query.minimum_nprobes.max(1);
-                    single_query.maximum_nprobes = Some(nprobes);
+                    // The scanner gate guarantees `minimum_nprobes == maximum_nprobes`, so
+                    // pass it through unchanged: clamping here would diverge from the
+                    // single-query path this node must reproduce exactly.
+                    single_query.maximum_nprobes = Some(single_query.minimum_nprobes);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/io/exec/knn.rs` around lines 2374 - 2382, Remove the .max(1)
clamp when assigning nprobes in the batch search setup so
single_query.minimum_nprobes, including zero, is propagated unchanged and
remains equivalent to repeated single-query search. Keep the existing
fixed-probe gating behavior intact.

Source: Coding guidelines

🧹 Nitpick comments (1)
rust/lance/src/dataset/scanner.rs (1)

4063-4068: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Doc's "Requires all of" list is now incomplete — add the overlay condition.

The list reads as exhaustive but omits the overlay-staleness gate added at Lines 4124-4135, which is a correctness requirement on par with the others.

As per coding guidelines, "Ensure doc comments match actual semantics".

📝 Proposed doc update
     /// Requires all of:
     /// - no refine step (the batch path does not yet rerank);
     /// - fixed nprobes (`minimum_nprobes == maximum_nprobes`) — see below;
     /// - every segment an IVF index with a flat-style sub-index (i.e. not HNSW);
+    /// - no overlay-stale indexed rows (the batch path neither blocks nor
+    ///   re-scores them);
     /// - all target fragments indexed (or `fast_search`, which ignores
     ///   unindexed fragments).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/dataset/scanner.rs` around lines 4063 - 4068, Update the
“Requires all of” documentation for the batch path to include the
overlay-staleness gate enforced near the scan eligibility logic, alongside the
existing refine, nprobes, index, and fragment requirements. Describe the
condition using the same semantics as the overlay check in the surrounding
scanner implementation.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@rust/lance/src/io/exec/knn.rs`:
- Around line 2374-2382: Remove the .max(1) clamp when assigning nprobes in the
batch search setup so single_query.minimum_nprobes, including zero, is
propagated unchanged and remains equivalent to repeated single-query search.
Keep the existing fixed-probe gating behavior intact.

---

Nitpick comments:
In `@rust/lance/src/dataset/scanner.rs`:
- Around line 4063-4068: Update the “Requires all of” documentation for the
batch path to include the overlay-staleness gate enforced near the scan
eligibility logic, alongside the existing refine, nprobes, index, and fragment
requirements. Describe the condition using the same semantics as the overlay
check in the surrounding scanner implementation.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 1bd1048c-b78e-4028-9866-9616c04698b9

📥 Commits

Reviewing files that changed from the base of the PR and between e0f8167 and 135df23.

📒 Files selected for processing (4)
  • rust/lance/src/dataset/scanner.rs
  • rust/lance/src/dataset/tests/dataset_overlay_index_masking.rs
  • rust/lance/src/index/vector/ivf/v2.rs
  • rust/lance/src/io/exec/knn.rs

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a worthwhile optimization, but I can't accept it with the current memory use regression.

Is there any follow up design for how we can make this fully supported, including by refine factor and min/max nprobes?

/// late-search expansion used by the single-query path is not applied, so recall
/// matches repeated single-query search when `minimum_nprobes == maximum_nprobes`.
#[derive(Debug)]
pub struct ANNIvfBatchExec {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick(modularity): Could we give this execution node it's own module / file? This file is already very long.

Comment on lines +2444 to +2446
let mut query_index_builder = Int32Builder::new();
let mut distance_builder = Float32Builder::new();
let mut row_id_builder = UInt64Builder::new();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick(performance): please pre-size these array allocations.

}
let properties = Arc::new(PlanProperties::new(
EquivalenceProperties::new(knn_empty_result_schema(true)),
Partitioning::RoundRobinBatch(1),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: going forward I'd encourage us to support multiple partitions so we have CPU parallelism in DataFusion. This node could be split so that query vectors are round-robined across partitions. The IO to load partitions could be done from a background IO task that provides them to an async MPSC channel. That channel should be bounded by a MemoryConsumer, so it doesn't load too many at once. This can be an optional follow up.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Gate recommendation: request changes.

The shared-scan direction is worthwhile, and this revision fixes the earlier unbounded partition retention. Acceptance still requires preserving fragment-selection completeness and keeping centroid ranking off async executor threads. The dedicated batch node can remain, with eligibility computed from the selected segment coverage and its per-query ranking dispatched through the CPU runtime.

Comment thread rust/lance/src/dataset/scanner.rs Resolved
Comment thread rust/lance/src/io/exec/knn.rs Resolved
@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 18, 2026
@sezruby
sezruby requested a review from wjones127 August 20, 2026 16:59
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

Both earlier Gate findings are fixed: fragment coverage is now computed from the selected index segments, and batch centroid ranking runs on the dedicated CPU runtime. The native shared-scan path is the right mechanism and preserves the covered per-query and fallback semantics.

The remaining non-blocking risk is performance for low-overlap or very wide batches: batch CPU work is sequential within each dispatched job, while the benchmark does not compare against the former in-request per-query parallel plan. Treat the speedup as workload-dependent until low-overlap and query-count scaling are measured.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 21, 2026
@sezruby

sezruby commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

@wjones127 gentle ping on this one. The memory-regression concern is resolved — search_partitions_batch now streams probed partitions through scoring in bounded STREAMING_SEARCH_BATCH_SIZE chunks instead of materializing them all first (8109f6f), with loading overlapped against scoring in 8f494fa. The two follow-up findings from the automated pass (segment-coverage gate and centroid ranking on the CPU runtime) are fixed in 32120b8, and the branch is freshly merged with main.

The modularity and pre-sizing nitpicks are still open — happy to fold both into this pass if you'd like them in before re-review, or as a quick follow-up. Whenever you have a chance to take another look, thanks!

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 27, 2026

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks better. Thanks for addressing the memory use.

@wjones127

Copy link
Copy Markdown
Contributor

Looks like there's a simple rebase that needs to happen before this can be merged.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 2, 2026
Extend batch vector search (lance-format#6821) to the indexed/ANN path so a single
multi-query request reads each IVF partition's storage once and scores every
query that probes it, instead of re-running a full single-query plan per
vector and unioning the results (which re-opens the index and rebuilds the
prefilter for each query).

- Add `VectorIndex::search_partitions_batch` + `supports_batch_partition_search`
  (defaulted so non-IVF indices stay explicitly unsupported).
- Implement them for `IVFIndex` with a flat-style sub-index
  (IVF_FLAT/PQ/SQ/RQ): load each distinct partition once and accumulate one
  top-k heap per query, sharing the prefilter across the whole batch.
- Add `ANNIvfBatchExec`, which ranks every query against the centroids, runs
  the shared-scan batch search, merges per-query top-k across deltas, and emits
  `query_index`-tagged results; route to it from
  `Scanner::batch_indexed_vector_search` when the gate below holds.
- Normalize each query vector independently for cosine
  (`normalize_batch_query_for_index`): normalizing the concatenated batch key
  with one global norm would scale each vector by a batch-composition-dependent
  factor and break equivalence with single-query search.

The shared-scan fast path is gated to cases that are provably equivalent to
repeated single-query search: fixed nprobes (`minimum_nprobes ==
maximum_nprobes`), no refine step, an IVF flat-style index, and fully-indexed
fragments. With adaptive nprobes the single-query path applies an
`early_pruning` floor and late-search expansion that the batch path does not,
so those queries fall back to the per-query loop, which stays exact. HNSW,
refine, and mixed indexed/unindexed scans also fall back.

Tests: plan shape; exact batch-vs-repeated-single equivalence (nprobes pinned);
cosine regression; shared prefilter; multi-delta cross-delta merge; and
fallbacks for refine and adaptive nprobes. Python parametrized over L2 +
cosine; a batch-vs-repeated-single ANN benchmark.

Closes lance-format#6822

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sezruby and others added 2 commits September 2, 2026 17:47
…hunk

`test_batch_knn_indexed_streams_multiple_chunks` relies on the 20-partition
index probing more distinct partitions than one `STREAMING_SEARCH_BATCH_SIZE`
chunk (16) holds. That premise was implicit: if the default chunk size were
raised past the partition count, the test would silently collapse to a single
chunk and stop covering the heap-threading seam it is named for, while still
passing. Assert `num_partitions > chunk_size` so that regression fails loudly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rd partitions_searched

Three follow-ups on the shared-scan batch IVF path:

- Gate eligibility on whether the *selected* index_segments cover every
  requested fragment, not whether the whole logical index does. A subset
  selected via with_index_segments could otherwise let the batch node search
  only the selected segments and silently drop a fragment covered solely by an
  unselected segment. Extracted the coverage check into
  fragments_missing_from_index_segments, shared by the gate and knn_combined so
  eligibility and fallback stay in lockstep.

- Run the per-query centroid ranking on the dedicated CPU runtime
  (find_partitions_batch_on_cpu) instead of the async worker: the ranking is
  pure CPU and batch width multiplies it, so a wide batch over a large centroid
  set could monopolize a Tokio worker.

- Record partitions_searched on ANNIvfBatchExec. It built the metric but never
  incremented it, so EXPLAIN ANALYZE reported 0 for every batch query. Report
  the distinct partitions read -- the shared I/O this node exists to save --
  mirroring the single-query ANNIvfSubIndexExec.

Tests: partial-segment fallback, CPU-runtime ranking, and a
partitions_searched=2 assertion (distinct union, not the per-query sum of 4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sezruby

sezruby commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main and force-pushed — now linear, no merge commits.

One conflict, in knn.rs: main added external-mask (MaskAndLoader) handling to ANNIvfSubIndexExec's prefilter construction, which this PR had extracted into the shared build_dataset_prefilter helper. I folded the external-mask step into that helper (new external_mask param; the batch node passes None) so both call sites stay deduplicated and the behavior is unchanged.

cargo fmt --all, cargo clippy -p lance --lib --tests -- -D warnings, and the 20 batch-KNN tests are green.

@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 3, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Gate recommendation: request changes.

The rebase note identifies the integration with the merged external-mask feature, but the shared indexed batch path does not preserve that contract: it remains eligible when a mask is present and drops the mask before building its prefilter. The earlier fragment-coverage and CPU-runtime fixes remain intact; acceptance now requires either carrying the external mask into the shared batch prefilter or falling back to the existing per-query path.

The prior low-overlap and very-wide-batch performance uncertainty remains non-blocking.

// across the batch. HNSW, refine, and mixed indexed/unindexed scans fall
// back to the per-query loop below, which never regresses behavior.
if self
.batch_index_search_supported(index_name, index_segments, q)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fast path can return rows excluded by with_row_addr_prefilter. batch_index_search_supported does not reject self.external_row_mask, and new_knn_batch_exec has no mask input; ANNIvfBatchExec then supplies None to the shared prefilter. The merged mask contract requires the mask to constrain ANN top-k. Either pass the mask into the batch prefilter or return false from this eligibility check when a mask exists, so the per-query path preserves it.

Reproducer run on this head

I added this temporary test beside test_batch_knn_indexed and ran:

cargo test -p lance --lib gate_repro_batch_knn_indexed_respects_external_mask -- --nocapture
#[tokio::test]
async fn gate_repro_batch_knn_indexed_respects_external_mask() {
    let mut test_ds = TestVectorDataset::new(LanceFileVersion::Stable, true)
        .await
        .unwrap();
    test_ds.make_vector_index().await.unwrap();
    let dataset = &test_ds.dataset;
    let (queries, _) = batch_knn_two_queries();

    let mut scan = dataset.scan();
    scan.nearest("vec", &queries, 2).unwrap();
    scan.nprobes(2);
    scan.with_row_addr_prefilter(RowAddrMask::allow_nothing());
    scan.project(&["i"]).unwrap();

    let plan = scan.explain_plan(false).await.unwrap();
    assert!(plan.contains("ANNIvfBatch"), "unexpected plan:\n{plan}");
    let batch = scan.try_into_batch().await.unwrap();
    assert_eq!(batch.num_rows(), 0, "allow-nothing mask must exclude all rows");
}

The plan used ANNIvfBatch; the assertion observed 4 rows instead of 0. The test itself finished in 0.36s.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 717b3e129: every batch query with an external row mask now falls back to the per-query ANN path, which forwards the mask into each prefilter. The focused regression passed and confirms the plan fallback and that every returned row stays within the allowlist.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 3, 2026
… is set

The shared-scan batch path builds one prefilter across the batch and does
not carry a caller-supplied external row-address mask
(`with_row_addr_prefilter`), whereas the per-query path threads it into each
query via `with_external_mask`. After the merge with the external-mask
feature, an otherwise batch-eligible query with a mask would have selected
the batch node and silently dropped the mask, returning masked-out rows.

Disqualify the batch path in `batch_index_search_supported` whenever an
external mask is present so the query falls back to the per-query indexed
loop, which honors the mask. Matches the existing fallback pattern (refine,
adaptive/zero nprobes, HNSW, overlay, partial coverage).

Adds `test_batch_knn_indexed_external_mask_falls_back`: the same query is
batch-eligible without a mask, falls back to ANNSubIndex with one, and every
returned row is in the allowlist.
@sezruby

sezruby commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Addressed the external-mask gap in 717b3e1.

Took the fallback route: batch_index_search_supported now returns false when external_row_mask is set, so a masked query goes to the per-query indexed loop (which threads the mask via with_external_mask) instead of the shared batch node that doesn't carry it. This matches the path's existing fallbacks (refine, adaptive/zero nprobes, HNSW, overlay-stale, partial segment coverage) and never returns masked-out rows.

New test test_batch_knn_indexed_external_mask_falls_back: the same query uses the batch node without a mask, falls back to ANNSubIndex with one, and every returned row is in the allowlist. fmt/clippy -p lance --lib --tests -- -D warnings/all 21 batch-KNN tests green.

@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 3, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The external-mask blocker is fixed in 717b3e129 following the author's update: masked batch searches now use the existing per-query ANN fallback, which preserves the caller's mask. The focused regression covers both plan selection and returned-row eligibility, and the earlier fragment-coverage and CPU-runtime fixes remain intact.

The remaining non-blocking risk is performance for low-overlap or very wide batches: batch CPU work is sequential within each dispatched job, while the benchmark does not compare against the former in-request per-query parallel plan. Treat the speedup as workload-dependent until low-overlap and query-count scaling are measured.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 3, 2026
@sezruby

sezruby commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor Author

@wjones127 Rebased onto main (linear, no merge commits) and addressed the gatekeeper's post-rebase finding.

The rebase surfaced the merged external-mask feature (#7288): the shared batch path wasn't carrying a caller-supplied external row mask. Fixed in 717b3e1 — batch_index_search_supported now falls back to the per-query indexed loop when external_row_mask is set (matching the path's existing fallbacks: refine, adaptive/zero nprobes, HNSW, overlay-stale, partial coverage), so a masked query never routes to the batch node that would drop the mask. Added test_batch_knn_indexed_external_mask_falls_back. Gatekeeper is green again.

All checks pass except mac-build (stable), which failed in test_vector_search_respects_fragment_list — an index-build shuffle-offset validation in shuffler.rs (from #8894), unrelated to this PR. It passes on the other lanes and locally. I don't have re-run access on this repo, so it needs a maintainer re-run (or it'll re-run on the next push).

@sezruby

sezruby commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@wjones127 gentle ping on this one — it's approved and MERGEABLE, and the branch is current with main.

Every check is green except mac-build (stable), which fails only on test_vector_search_respects_fragment_list. That's the shuffler.rs shuffle-offset validation from #8894 (offset-derived count 196 for partition 0 does not match expected count 4), unrelated to this PR — it passes on all the other lanes and locally. So it just needs a re-run of that lane (I don't have re-run access here) or a merge.

@BubbleCal looping you in since you've been active in the vector-index area and landing PRs lately — would either of you be able to kick off a re-run of the mac lane / land this when you get a chance? Thanks!

@wjones127
wjones127 merged commit 577091e into lance-format:main Sep 12, 2026
37 of 39 checks passed
Xuanwo added a commit that referenced this pull request Nov 12, 2026
#10443)

## Summary

The shared-scan batch IVF node (`ANNIvfBatchExec`, #7640) required fixed
probe bounds (`minimum_nprobes == maximum_nprobes`). Adaptive requests,
including the default bounds and any explicit `search_effort`, fell back
to one indexed search per query. This PR applies the single-query
adaptive rules inside the batch node for IVF_FLAT, IVF_PQ, and IVF_SQ.
Partition loads stay shared across queries. There are no changes to the
index trait, the public API, or the index format.

#10480, now merged, scores each loaded chunk's queries on separate CPU
tasks. Before it, `search_partitions_batch` scored each chunk on a
single CPU task, so routing adaptive bounds to the shared node showed up
as a 3.3x slowdown on the existing `query_ann_batch` benchmark (768-d
IVF_PQ, `k = 100`). The scanner changes are in
`dataset/scanner/vector.rs`. `ANNIvfBatchExec` now lives in
`io/exec/knn/batch.rs` and its adaptive tests in
`dataset/tests/dataset_scanner_knn_adaptive.rs`, so no module grows past
the size check.

This follows up the min/max-nprobes question from the #7640 review. It
builds on the segment-ownership fix in #9638. With #9618, namespace
`multi_vector` requests that use adaptive bounds can now reach the
shared path.

## Commits

1. `perf(vector): support adaptive nprobes in shared-scan batch IVF
search` is the change itself.
2. `test(vector): move adaptive batch KNN tests into their own file`
moves tests without changing them.
3. `fix(vector): align batch late search with single-query skip and stop
rules` adds the selects-nothing skip, the budget-based ranking release,
and the late-search helpers shared with the single-query path.
4. `refactor(vector): move ANNIvfBatchExec into its own module` moves
code without changing it. Only imports and three intra-doc link paths
change.
5. `Merge origin/main into perf/batch-ivf-adaptive-nprobes` merges
`main` at `cd0a24899`; it had no conflicts.
6. `test(vector): cover hybrid scalar prefilter in adaptive batch KNN`
adds the #10001 producer regression.

## Execution

- Rank each query on the CPU runtime and apply the existing
`AutoProbePolicy`, including `search_effort`. This is the same policy
call the single-query path makes for each delta.
- Search every query's initial budget in one shared call per delta.
Finish the early phase for all deltas before deciding which queries need
late search.
- Keep only rankings that late search can still read. A query's ranking
is released once it holds `k` candidates, holds every row the prefilter
allows, or has no late range. Deltas with no remaining late work,
including fixed, exhausted, and empty-ownership deltas, are released
before the next delta is scanned. Retained deltas are checked again at
the early-search barrier.
- Run late search in rounds through the existing batch API, with one
virtual query per (query, partition) pair. Each query consumes its
partitions in rank order and stops where a sequential search would stop.
Rows after that cut are discarded.
- Admit whole queries per round, up to `max(1024, compute CPUs)` probes
**across all queries and deltas**. The first query rotates between
rounds so none starves. A query's deltas therefore advance together, and
its rows do not depend on the rest of the batch. A query whose own waves
exceed the cap is clipped and resumes at the first delta it could not
fully admit. Admitted deltas run concurrently, and their results are
consumed round-robin.
- Score each loaded chunk on up to one CPU task per compute CPU,
splitting its queries into contiguous ranges with at least 8 probes each
when the chunk has enough. This applies to fixed and adaptive batches.
- Apply segment ownership to early and late results and to the stopping
budgets. Segments that own nothing skip late work, and so do segments
whose own prefilter selects nothing. As in the single-query path
(#10485), the latter are counted once in `segments_skipped`. The
fewer-than-k shortcut returns only owned, unseen rows from eligible
segments.
- The single-query and batch late searches share the stop budget, the
fewer-than-k shortcut condition, and the segment-ownership check
(`LateSearchBudget`, `segment_owns_rows` in `knn.rs`), so the two paths
cannot drift on when a query stops.
- Keep the per-query fallback for explicit parallel adaptive probing
(`query_parallelism > 1` or `-1`) and for adaptive IVF_RQ. Neither is
equivalent to the shared sequential path today. Parallel late search can
admit partitions past the sequential cut. Multi-bit IVF_RQ prunes with a
probabilistic lower bound, so batch and single-query search can drop
different rows (#10778). Fixed-probe IVF_RQ keeps its existing shared
path and shows the same intermittent difference on `main`, independent
of this PR.

## Memory

- Per-query candidates stay in the bounded `TopHits` collector from
#9688, so retained candidates are `Q * k`. `TopHits` gains an `iter()`
so the fewer-than-k shortcut can skip rows it already holds.
- For Float32 vectors, the temporary key copies in a late round hold at
most `B * dim` floats, and a round returns at most `B * k` rows, where
`B` is the round cap. With the default cap, a round of 1024-d queries
copies at most 4 MiB of keys.
- Adaptive rankings cover every partition. They are released per query
once the query holds `k` candidates (or every row the prefilter allows)
or has no late range, so an unfiltered batch does not keep `Q *
num_partitions` ranking entries per delta after the delta's early scan.
While a delta is being ranked, all of its queries' rankings are held at
once.

These bound the round scratch, not total execution memory. Ranked plans
for genuine adaptive work, partition storage, and output have their own
costs. Measured peak memory is under [Peak memory and
latency](#peak-memory-and-latency).

## Behavioral scope

For a single delta, shared adaptive FLAT/PQ/SQ follows the same
sequential probe ranges and stopping rules as single-query search.
Equal-distance ties can select different valid rows. With multiple
deltas, results are consumed in a deterministic round-robin order rather
than by completion timing, so identical row sets are not promised. The
multi-delta tests check progression, uniqueness, counts, and recall. A
query's rows do not depend on the other queries in the batch unless its
own waves exceed the round cap.

Batched exact refinement (#9353) covers fixed nprobes. Exact refinement
with adaptive bounds keeps the per-query path, and
`test_batch_knn_indexed_refine`'s `adaptive` case asserts that.

**Metric change:** for adaptive batches that take the shared path,
`partitions_searched` counts the shared scan's partition loads, as the
fixed-probe batch path already does, instead of summing each query's
searches. This includes late-search partitions loaded in a wave past a
query's cut. Identical queries in one batch therefore count a partition
once. The Python `test_vector_search_effort` batch assertion is updated
to match. Adaptive IVF_RQ still reports the per-query sum.

## Tests

- IVF_FLAT, IVF_PQ, IVF_SQ: batch results equal per-query results on
single-delta fixtures. Cases cover unfiltered, selective, capped,
oversized-maximum, fewer-than-k, and empty-filter queries.
- `search_effort` at 0, 0.25, 0.75, and 1, with and without a filter:
rows and distances match single-query search. Forcing the batch planner
to ignore effort fails 18 of the 24 cases.
- IVF_RQ: adaptive cases assert the per-query fallback and compare rows
and distances with single-query search. Fixed-probe RQ still uses the
shared node.
- Distance ranges on IVF_FLAT and IVF_PQ, with and without a filter:
rows and distances match single-query search. Narrow ranges leave
queries short of `k` rows, so they exercise late search to exhaustion.
- Cosine and dot: results equal single-query search, and recall is at
least 0.5 against brute force, with and without filters.
- Late search: deterministic tests cover row cuts, delta progression,
ownership budgets, the fewer-than-k shortcut, and empty ownership. The
cut test is arranged so that a row past the cut would displace a
retained row from the bounded top-k if it were consumed.
- Batch independence: a two-delta case where the round cap falls between
a query's deltas returns the same rows as an uncapped query. Under
per-(query, delta) admission, that query returned a farther row.
- Round planning: cap and fairness tests cover partial waves, 1,025
queries, 129 deltas, and a cap of 1. A three-delta execution test
records the actual batch calls. It asserts that each round admits only
whole queries and never exceeds the cap.
- Lifetime: weak-reference tests show that rankings, keys, and index
handles for fixed and empty-ownership deltas are released before the
next early delta, while state needed for adaptive late search stays
alive. Rankings of queries that already hold `k` candidates are
released, including when a later delta completes the query. So are
rankings of queries that hold every row the prefilter allows.
- Chunk scoring: range splitting is tested for empty, small, even,
capped, skewed, and trailing-empty probe counts, and every query with
probes belongs to exactly one range. All batch-vs-single equality tests
run with the parallel scorer, including with `LANCE_CPU_THREADS=1`.
- A delta whose own prefilter selects nothing (the filter keeps only
rows of a newer delta): batch search reports the same `segments_skipped`
as single-query search and returns the same rows. A unit test checks
that such a delta does not join late search. Reverting the skip fails
both.
- `LateSearchBudget` and `segment_owns_rows`: unit tests cover allow
lists below, at, and above `k`, empty masks, and masks that cannot count
their rows (block lists, whole fragments).
- `query_parallelism` 0 and 1 take the shared path, and their rows and
distances match single-query search. Explicit parallel probing keeps the
per-query path.
- Existing HNSW, zero-probe, external-mask, overlay, and coverage
fallbacks are unchanged. #9353's `test_batch_knn_indexed_refine` still
requires exact refinement with adaptive bounds to take the per-query
path.

## Validation

The integrated head merges `main` at `cd0a24899` (#10753) into the
branch in `2f63075ad`. Git merged cleanly, and the result keeps #10001's
hybrid scalar prefilter producer, #10721's `lance_core::utils::tokio`
spawn boundary, and #10727's extracted `partition_search` scoring; no
old inline implementation was restored.

- `cargo test --profile ci -p lance --lib
dataset_scanner_knn_adaptive::`: 74 passed.
- `cargo test --profile ci -p lance --lib dataset_scanner_knn::`: 152
passed.
- `cargo test --profile ci -p lance --lib io::exec::knn::`: 1,219
passed, covering the batch, late-search, ownership, segment-contract,
and budget units.
- `cargo test --profile ci -p lance --lib
dataset_scanner_segments::test_hybrid_prefilter_consumers`: 12 passed,
so #10001's ANN, batch, and FTS consumers keep their union prefilter.
- `cargo test --profile ci -p lance --lib io::exec::prefilter_union::`
and `--test integration_tests prefilter_union`: 2 and 16 passed.
- `cargo test --profile ci -p lance-index --lib -- minhash`: 63 passed.
- `cargo clippy --profile ci --all --tests --benches -- -D warnings` and
`cargo fmt --all -- --check`: clean.
- Python `test_vector_index.py -k "search_effort or batch"` against a
native build of the integrated head: 68 passed.
- `uv run make lint` from `python/`: clean.

The integration adds `test_batch_knn_adaptive_hybrid_scalar_prefilter`,
which reaches the adaptive batch scan through #10001's
`PrefilterUnionExec`. An exact B-tree predicate on `i` (`i >= 250 AND i
< 1750`) covers both producer shapes: a scalar index over every fragment
while the scan selects the middle two (`scalar_fragments=2,
evaluated_fragments=2`), and a scalar index over two of four fragments
(`scalar_fragments=2, evaluated_fragments=4`). Each case asserts that
the union producer and `ANNIvfBatch` are planned and `ANNSubIndex` is
not, that per-query rows and distances equal single-query search, that
every returned row satisfies the predicate and stays inside the selected
fragments, and a recall floor of 0.5 against the flat oracle.

### Author-reported validation before integration

Recorded against `58e62bd54`, which includes #10695 (search-effort
calibration by metric and k) and #10761 (segment contract checks),
before this branch was merged with `main`:

- `cargo test -p lance --lib`: 10,075 passed, 0 failed, 9 ignored.
- Batch, adaptive, `search_effort`, range-splitting, and late-search
budget tests with `LANCE_CPU_THREADS=1` and
`LANCE_IVF_STREAMING_SEARCH_BATCH_SIZE=1`: 1,049 passed.
- `cargo test -p lance-index --lib -- minhash`: 63 passed.
- Python `test_vector_index.py -k "search_effort or batch"` on a fresh
`make build`: 68 passed.
- `cargo clippy --all --tests --benches -- -D warnings`, `cargo clippy
-p lance --all-features --tests -- -D warnings`, `cargo fmt --all --
--check`, and `RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps`:
clean. On aarch64 macOS, clippy also needs `-A unreachable_code` for the
NEON byte-stream-split code already on `main`.
- With the selects-nothing skip reverted, the end-to-end
`segments_skipped` test and the late-search unit test fail. With the
ranking release reverted to `k`, the budget retention test fails.
- Earlier, on `0db573921` before the rebase, the full Python suite
(`pytest python/tests -n auto`) passed 1,902 tests.

## Benchmark suite

Everything in this section and the next two is author-reported: it was
measured on the author's hardware against the revisions named below and
was not re-run on the integrated head.

`python/python/benchmarks/test_search.py` gains a `fixed` (`nprobes =
10`) and an `adaptive` (default bounds) case for the two
`query_ann_batch` benchmarks. These are pylance release builds of this
branch, measured before it was rebased onto `0db573921`, and of `main`
at `65df9b7c4`, run on one Apple M5 Max. The dataset has 100k x 768-d
vectors on local disk, IVF_PQ with 32 partitions, `k = 100`, and 32
queries. Values are pytest-benchmark medians in ms.

| Benchmark | `main` | This PR |
|---|---|---|
| `test_batch_ann_search[clean-adaptive]` | 10.20 | 6.14 |
| `test_batch_ann_search[clean-fixed]` | 12.68 | 4.53 |
| `test_batch_ann_search[with_delete_files-adaptive]` | 11.10 | 7.00 |
| `test_batch_ann_search[with_delete_files-fixed]` | 16.71 | 5.04 |
| `test_batch_ann_search[with_new_rows-adaptive]` | 57.75 | 55.78 |
| `test_batch_ann_search[with_new_rows-fixed]` | 53.80 | 50.15 |
| `test_repeated_single_ann_search[clean-adaptive]` | 46.66 | 46.01 |
| `test_repeated_single_ann_search[clean-fixed]` | 26.98 | 27.12 |

`with_new_rows` has unindexed fragments, so it keeps the per-query path.
The repeated single-query cases do not use the changed code and serve as
a noise reference. Without #10480, `clean-adaptive` took 33.67 ms.

## Peak memory and latency

This is a release-build measurement of this branch, taken before it was
rebased onto `0db573921`, and of `main` at `65df9b7c4`. It is not
checked in. The dataset is on local SSD: 500k x 16-d Float32 vectors,
IVF_FLAT with 2,048 partitions, either as one index segment or as four
(an initial index plus three appended deltas indexed with
`optimize_indices`). Queries are uniform random vectors drawn
independently of the data, with `k = 10` and the default adaptive
bounds. A batch of 256 queries has low partition overlap; 8,192 queries
puts the per-query rankings at about 134 MB per segment.

Each configuration runs in a fresh process. Peak growth is the process
high-water RSS (`getrusage` `ru_maxrss`) after the query minus its value
just before. Values are the median of 3 runs on one Apple M5 Max. On
`main`, an adaptive batch runs one indexed search per query inside the
same plan.

| Segments | Queries | Filter | Peak RSS growth (`main` → this PR) |
Latency (`main` → this PR) |
|---|---|---|---|---|
| 1 | 256 | none | 133 MB → 34 MB | 56 ms → 56 ms |
| 1 | 256 | `i % 97 = 0` | 507 MB → 36 MB | 128 ms → 58 ms |
| 1 | 8,192 | none | 2,172 MB → 208 MB | 2,150 ms → 297 ms |
| 1 | 8,192 | `i % 97 = 0` | 14,448 MB → 195 MB | 4,994 ms → 311 ms |
| 4 | 256 | none | 316 MB → 45 MB | 162 ms → 143 ms |
| 4 | 256 | `i % 97 = 0` | 677 MB → 46 MB | 236 ms → 149 ms |
| 4 | 8,192 | none | 5,716 MB → 226 MB | 3,693 ms → 1,023 ms |
| 4 | 8,192 | `i % 97 = 0` | 16,961 MB → 277 MB | 6,320 ms → 1,099 ms |

Both builds return the same number of rows in every configuration. On
`main`, the baseline already includes planning the per-query searches,
so its growth is understated. Object-store latency is measured
separately below.

## Object-store latency

This measures release builds of `main` at `56963cb29` and of this PR at
`58725f2fe`, through the Java binding on a Spark cluster in Azure: 8
executors with 4 cores each, data on Azure Data Lake Storage (ABFS). The
dataset has 10M x 1024-d embeddings in 10 fragments, with one IVF_PQ
segment: L2, 256 partitions, and 64 sub-vectors. Queries are dataset
rows with 5% Gaussian noise, `k = 10`, and the Java `Query` sets no
nprobes, so probing is adaptive from a minimum of 1 with no maximum.
Each of 32 Spark tasks sends its share of the queries as one batch call
(batch) or as one call per query (per-query). Values are the median wall
time of 5 runs after a warmup, with the best run in parentheses, in ms.
Each build ran in its own job on a shared cluster.

| Filter | Queries | `main` batch | This PR batch | Per-query (`main` /
this PR) |
|---|---|---|---|---|
| none | 1,000 | 1,204 (1,175) | 1,190 (1,172) | 1,360 / 1,309 |
| none | 4,000 | 4,370 (4,153) | 3,835 (3,758) | 4,586 / 4,190 |
| `rid % 20000 = 0` (500 rows) | 1,000 | 6,236 (6,091) | 1,294 (1,246) |
— |
| `rid % 20000 = 0` (500 rows) | 4,000 | 19,786 (19,510) | 3,469 (3,403)
| — |

- The per-query path does not change in this PR, so its column shows the
difference between jobs: 4–9%. Without a filter, the shared path is
within that range of `main`.
- With the filter, about 2 matching rows fall in each partition, so
queries need late search. The shared path is 4.8x faster at 1,000
queries and 5.7x faster at 4,000. Per-query search was not run with the
filter.
- Recall@10 against an exact ground truth (with the same filter) over
100 queries is the same for both builds: 0.726 without the filter and
0.553 with it. Without the filter, per-query and batch rows agree for
every recall query in both builds.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend batch vector queries to ANN and indexed search

2 participants