Repository navigation
Conversation
sezruby
force-pushed
the
perf/adaptive-nprobes-batch
branch
from
September 30, 2026 21:48
4103ca2 to
8b37d8a
Compare
sezruby
changed the base branch from
base/adaptive-nprobes-batch
to
fix/batch-ivf-segment-ownership
September 30, 2026 21:49
sezruby
force-pushed
the
fix/batch-ivf-segment-ownership
branch
from
October 1, 2026 04:19
7274544 to
8dce5b7
Compare
sezruby
force-pushed
the
perf/adaptive-nprobes-batch
branch
from
October 1, 2026 04:19
8b37d8a to
6a7d27f
Compare
sezruby
force-pushed
the
perf/adaptive-nprobes-batch
branch
from
October 1, 2026 05:11
6a7d27f to
514b6c9
Compare
The shared-scan batch IVF node (`ANNIvfBatchExec`) required fixed probe bounds; adaptive requests fell back to one indexed search per query. It now applies the single-query adaptive rules per query while sharing partition loads across the batch: - rank each query and apply `AutoProbePolicy` (including `search_effort`) on the CPU runtime; - search every query's initial budget in one shared call per delta; - after all deltas finish, run late search in rounds with one virtual query per (query, partition), consumed in rank order and cut where a sequential search would stop; - admit whole queries per round, up to max(1024, compute CPUs) probes across all queries and deltas, so a query's rows do not depend on the rest of the batch; - release a query's ranking once it holds k candidates or has no late range; - apply segment ownership to early and late results and stopping budgets. Per-query candidates stay in the bounded `TopHits` heap. Explicit parallel adaptive probing and adaptive IVF_RQ keep the per-query path. For batches on the shared path, `partitions_searched` counts shared partition loads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sezruby
changed the base branch from
fix/batch-ivf-segment-ownership
to
base/adaptive-nprobes-batch
November 2, 2026 04:37
sezruby
force-pushed
the
perf/adaptive-nprobes-batch
branch
from
November 2, 2026 04:37
a734e62 to
979ba78
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The shared-scan batch IVF executor previously required fixed probe bounds. Adaptive requests fell back to one indexed search per query. This PR supports adaptive minimum/maximum nprobes for IVF_FLAT, IVF_PQ, and IVF_SQ without changing the index trait, public API, or index format.
This follows up the min/max-nprobes design question in the review of lance-format#7640.
Namespace update: after lance-format#9184, legacy
nprobessets an exact probe count. Adaptive defaults and independent minimum/maximum bounds still benefit. Namespace batch routing is handled separately by lance-format#9618.Ownership base: includes c9b459e from lance-format#9638. Batch preparation reuses
prefilter_for_partitiononce per loaded partition, restoringNoFilterand bulk scoring for fully owned partitions while retaining ownership filtering before top-k. Both early and late searches inherit this safeguard.Independent-review corrections: deltas without genuine late-search work now release their rankings and query-local state between early scans. This avoids retaining every delta's plans for fixed-probe batches. Adaptive RQ retains the existing per-query fallback: repeated validation reproduced intermittent RQ batch/scalar score and neighbor differences on the previous adaptive head as well as the working changes. This PR does not change the core RQ scorer or existing fixed-probe RQ eligibility.
Execution
AutoProbePolicy.B = max(1024, compute CPUs)probes per round across all queries and deltas. Clip the last wave to remaining capacity and rotate pairs to prevent starvation.query_parallelism > 1or-1) and adaptive RQ. Their existing semantics are not equivalent to the shared sequential path.For Float32 vectors, temporary late-round key copies contain at most
B * dimfloats and the round returns at mostB * krows. This bounds round scratch, not total execution memory: genuine adaptive ranked plans, retained candidates, partition storage, and output have separate costs.Regression coverage
Validation
cargo test -p lance --lib: 4,562 passed, 0 failed, 3 ignored.LANCE_CPU_THREADS=1andLANCE_IVF_STREAMING_SEARCH_BATCH_SIZE=1: 231 passed.514b6c9a6.--all --tests --benches -- -D warnings), lance all-features clippy, and formatting are clean.Earlier release benchmark
Measured on earlier adaptive implementation
8b37d8aa5; not rerun on this head. Synthetic 100k x 64 vectors, IVF_FLAT with 256 partitions, k=10, minimum_nprobes=1.i % 997 = 0i % 997 = 0Final-head latency/RSS benchmarks remain useful follow-up evidence; no new measured speedup or peak-memory claim is made here.
Behavioral scope
Single-delta sequential probe ranges and stopping rules match single-query search for shared adaptive FLAT/PQ/SQ. Equal-distance ties can select different valid rows. Multi-delta results use deterministic round-robin consumption rather than nondeterministic completion timing, so identical row sets are not promised; tests check progression, uniqueness, counts, and recall. Adaptive RQ continues through per-query execution until core scoring equivalence is established.