Repository navigation
test(index): cover Float16 streaming coreset for FLAT/PQ/SQ - #10554
Conversation
Without Float16 centroids from streaming coreset training, IVF_FLAT and
IVF_SQ fail partition assignment ("Centroids and vectors have
incompatible types") and IVF_PQ fails residual computation, the same as
IVF_RQ. Extend the float-type regression with those index types and
rename it, since it no longer covers RQ only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The added cases exercise the downstream Float16 paths left uncovered by #10350. All eight focused build-and-search cases passed locally on this revision; no material issues found.
Xuanwo
left a comment
There was a problem hiding this comment.
Reviewed the test-only extension on the integrated head with the latest main. The eight-case matrix keeps the existing RQ/dtype controls, forces the 257-partition streaming coreset path, and verifies the persisted Float16 centroid type, the ANN scan plan, finite top-k distances, and a flat-scan recall oracle for FLAT/PQ/SQ. No production-code or format change.
The focused eight-case matrix, fmt, workspace Clippy, the required hosted checks, and Gatekeeper pass on this revision.
Summary
Follow-up to #10350, the fix for #10349. It casts the centroids from streaming coreset training back to Float16, and its regression test covers IVF_RQ only. Without the cast, IVF_FLAT and IVF_SQ fail partition assignment ("Centroids and vectors have incompatible types") and IVF_PQ fails residual computation, so the same bug can come back for those index types without a failing test.
This extends
test_ivf_rq_build_and_search_float_typeswith Float16 streaming cases forflat,pqandsq, using the same fixture, full rerank and recall threshold. The test is renamed totest_ivf_build_and_search_float_typesbecause it no longer covers RQ only. Test change only, inrust/lance/src/index/vector/ivf.rs.I shared this on #10350 before it merged, as a cherry-pick or a follow-up.
Testing
Run on
mainat 46ab5a4 in arust:1.98.1-bookwormcontainer, with--profile ci --locked:cargo test -p lance --lib -- test_ivf_build_and_search_float_types: all 8 cases pass, in 20 consecutive runs.cargo test -p lance --lib -- index::vector::ivf::tests::: 143 passed, 0 failed.cargo fmt --all -- --checkandcargo clippy -p lance --tests -- -D warnings: clean. I ran clippy on thelancecrate only, the one this PR touches, not--all --benches.Written with help from Claude Code.