Skip to content

perf(vector): support adaptive nprobes in shared-scan batch IVF search - #9

Open
sezruby wants to merge 1 commit into
base/adaptive-nprobes-batchfrom
perf/adaptive-nprobes-batch
Open

sezruby wants to merge 1 commit into
base/adaptive-nprobes-batchfrom
perf/adaptive-nprobes-batch

Conversation

@sezruby

@sezruby sezruby commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Fork-only review PR. Stacked on #10 (fix/batch-ivf-segment-ownership), based on upstream main f4f7241. Not proposed upstream.

Summary

The shared-scan batch IVF executor previously required fixed probe bounds. Adaptive requests fell back to one indexed search per query. This PR supports adaptive minimum/maximum nprobes for IVF_FLAT, IVF_PQ, and IVF_SQ without changing the index trait, public API, or index format.

This follows up the min/max-nprobes design question in the review of lance-format#7640.

Namespace update: after lance-format#9184, legacy nprobes sets an exact probe count. Adaptive defaults and independent minimum/maximum bounds still benefit. Namespace batch routing is handled separately by lance-format#9618.

Ownership base: includes c9b459e from lance-format#9638. Batch preparation reuses prefilter_for_partition once per loaded partition, restoring NoFilter and bulk scoring for fully owned partitions while retaining ownership filtering before top-k. Both early and late searches inherit this safeguard.

Independent-review corrections: deltas without genuine late-search work now release their rankings and query-local state between early scans. This avoids retaining every delta's plans for fixed-probe batches. Adaptive RQ retains the existing per-query fallback: repeated validation reproduced intermittent RQ batch/scalar score and neighbor differences on the previous adaptive head as well as the working changes. This PR does not change the core RQ scorer or existing fixed-probe RQ eligibility.

Execution

  • Rank each query on the CPU runtime and apply the existing AutoProbePolicy.
  • Search every query's early partitions in one shared call per delta. Finish the early phase for all deltas before deciding which queries need late search.
  • Retain only deltas that can participate in late search. Fixed, exhausted, and empty-ownership delta state is released before processing the next delta.
  • Run late rounds through the existing batch API, with one virtual query per (query, partition) pair. Keep partition results separate and consume them in rank order, discarding results after the sequential per-query cut.
  • Admit at most B = max(1024, compute CPUs) probes per round across all queries and deltas. Clip the last wave to remaining capacity and rotate pairs to prevent starvation.
  • Run admitted deltas concurrently and consume their results round-robin rather than exhausting the first delta.
  • Apply segment ownership to early/late results and stopping budgets. Skip late work for segments owning nothing; the fewer-than-k shortcut returns only eligible segments' owned, unseen rows.
  • Preserve per-query fallback for explicit parallel adaptive probing (query_parallelism > 1 or -1) and adaptive RQ. Their existing semantics are not equivalent to the shared sequential path.

For Float32 vectors, temporary late-round key copies contain at most B * dim floats and the round returns at most B * k rows. This bounds round scratch, not total execution memory: genuine adaptive ranked plans, retained candidates, partition storage, and output have separate costs.

Regression coverage

  • IVF_FLAT, IVF_PQ, IVF_SQ: batch/per-query equality on single-delta fixtures with unfiltered, selective, capped, oversized-maximum, fewer-than-k, and empty-filter cases.
  • RQ: all adaptive matrix cases assert per-query fallback and compare rows/distances with single-query search. They also verify fixed-probe RQ still uses the shared node and preserves output counts.
  • Cosine/Dot: equality with single-query search and recall >= 0.5 against brute force, with and without filters.
  • Deterministic late tests verify row cuts, actual delta progression, ownership budgets, shortcuts, and empty ownership.
  • Planner cap/fairness cases cover partial waves, 1,025 queries, 129 deltas, and cap 1. A three-delta execution test also records actual batch calls and asserts their combined round admission equals, but never exceeds, the cap.
  • Weak-reference lifetime tests prove fixed/empty-ownership rankings, keys, and index handles are released before the next early delta; necessary adaptive state remains alive. The old unconditional retention fails these tests.
  • Inherited FLAT/PQ4/PQ8 regressions protect partition-level bulk scoring and filtering before top-k; in-place-update coverage includes physical and stable row IDs.
  • Existing refine, HNSW, zero-probe, external-mask, overlay, and coverage safeguards remain intact.

Validation

  • cargo test -p lance --lib: 4,562 passed, 0 failed, 3 ignored.
  • Batch tests with LANCE_CPU_THREADS=1 and LANCE_IVF_STREAMING_SEARCH_BATCH_SIZE=1: 231 passed.
  • Probe/lifetime module: 16 passed under both default and one-worker settings.
  • Previously intermittent RQ case: 20 consecutive one-worker repetitions pass with fallback; it reproduced on pre-fix head 514b6c9a6.
  • Workspace clippy (--all --tests --benches -- -D warnings), lance all-features clippy, and formatting are clean.
  • Rebuilt package artifacts when switching worktrees to avoid shared-target cache reuse.

Earlier release benchmark

Measured on earlier adaptive implementation 8b37d8aa5; not rerun on this head. Synthetic 100k x 64 vectors, IVF_FLAT with 256 partitions, k=10, minimum_nprobes=1.

Filter Queries Batch Per-query loop Speedup
none 16 1.4ms 4.9ms 3.5x
none 256 15.6ms 74.4ms 4.8x
i % 997 = 0 16 2.0ms 9.8ms 4.9x
i % 997 = 0 256 18.1ms 164.9ms 9.1x

Final-head latency/RSS benchmarks remain useful follow-up evidence; no new measured speedup or peak-memory claim is made here.

Behavioral scope

Single-delta sequential probe ranges and stopping rules match single-query search for shared adaptive FLAT/PQ/SQ. Equal-distance ties can select different valid rows. Multi-delta results use deterministic round-robin consumption rather than nondeterministic completion timing, so identical row sets are not promised; tests check progression, uniqueness, counts, and recall. Adaptive RQ continues through per-query execution until core scoring equivalence is established.

@sezruby
sezruby force-pushed the perf/adaptive-nprobes-batch branch from 4103ca2 to 8b37d8a Compare September 30, 2026 21:48
@sezruby
sezruby changed the base branch from base/adaptive-nprobes-batch to fix/batch-ivf-segment-ownership September 30, 2026 21:49
@sezruby
sezruby force-pushed the fix/batch-ivf-segment-ownership branch from 7274544 to 8dce5b7 Compare October 1, 2026 04:19
@sezruby
sezruby force-pushed the perf/adaptive-nprobes-batch branch from 8b37d8a to 6a7d27f Compare October 1, 2026 04:19
@sezruby
sezruby force-pushed the perf/adaptive-nprobes-batch branch from 6a7d27f to 514b6c9 Compare October 1, 2026 05:11
The shared-scan batch IVF node (`ANNIvfBatchExec`) required fixed probe
bounds; adaptive requests fell back to one indexed search per query. It now
applies the single-query adaptive rules per query while sharing partition
loads across the batch:

- rank each query and apply `AutoProbePolicy` (including `search_effort`)
  on the CPU runtime;
- search every query's initial budget in one shared call per delta;
- after all deltas finish, run late search in rounds with one virtual query
  per (query, partition), consumed in rank order and cut where a sequential
  search would stop;
- admit whole queries per round, up to max(1024, compute CPUs) probes across
  all queries and deltas, so a query's rows do not depend on the rest of
  the batch;
- release a query's ranking once it holds k candidates or has no late range;
- apply segment ownership to early and late results and stopping budgets.

Per-query candidates stay in the bounded `TopHits` heap. Explicit parallel
adaptive probing and adaptive IVF_RQ keep the per-query path. For batches
on the shared path, `partitions_searched` counts shared partition loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sezruby
sezruby changed the base branch from fix/batch-ivf-segment-ownership to base/adaptive-nprobes-batch November 2, 2026 04:37
@sezruby
sezruby force-pushed the perf/adaptive-nprobes-batch branch from a734e62 to 979ba78 Compare November 2, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant