Skip to content

test(index): cover Float16 streaming coreset for FLAT/PQ/SQ - #10554

Merged
Xuanwo merged 2 commits into
lance-format:mainfrom
aimanmalib:test-f16-coreset-index-types
Nov 5, 2026
Merged

Xuanwo merged 2 commits into
lance-format:mainfrom
aimanmalib:test-f16-coreset-index-types

Conversation

@aimanmalib

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #10350, the fix for #10349. It casts the centroids from streaming coreset training back to Float16, and its regression test covers IVF_RQ only. Without the cast, IVF_FLAT and IVF_SQ fail partition assignment ("Centroids and vectors have incompatible types") and IVF_PQ fails residual computation, so the same bug can come back for those index types without a failing test.

This extends test_ivf_rq_build_and_search_float_types with Float16 streaming cases for flat, pq and sq, using the same fixture, full rerank and recall threshold. The test is renamed to test_ivf_build_and_search_float_types because it no longer covers RQ only. Test change only, in rust/lance/src/index/vector/ivf.rs.

I shared this on #10350 before it merged, as a cherry-pick or a follow-up.

Testing

Run on main at 46ab5a4 in a rust:1.98.1-bookworm container, with --profile ci --locked:

  • cargo test -p lance --lib -- test_ivf_build_and_search_float_types: all 8 cases pass, in 20 consecutive runs.
  • cargo test -p lance --lib -- index::vector::ivf::tests::: 143 passed, 0 failed.
  • cargo fmt --all -- --check and cargo clippy -p lance --tests -- -D warnings: clean. I ran clippy on the lance crate only, the one this PR touches, not --all --benches.
  • With the Float16 centroid cast from fix(index): preserve Float16 centroids after streaming training #10350 disabled, the six Float16 streaming cases fail, including the three new ones. The Float32 streaming and Float16 in-memory controls pass.

Written with help from Claude Code.

Without Float16 centroids from streaming coreset training, IVF_FLAT and
IVF_SQ fail partition assignment ("Centroids and vectors have
incompatible types") and IVF_PQ fails residual computation, the same as
IVF_RQ. Extend the float-type regression with those index types and
rename it, since it no longer covers RQ only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the chore label Nov 1, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 1, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 5, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The added cases exercise the downstream Float16 paths left uncovered by #10350. All eight focused build-and-search cases passed locally on this revision; no material issues found.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 5, 2026

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the test-only extension on the integrated head with the latest main. The eight-case matrix keeps the existing RQ/dtype controls, forces the 257-partition streaming coreset path, and verifies the persisted Float16 centroid type, the ANN scan plan, finite top-k distances, and a flat-scan recall oracle for FLAT/PQ/SQ. No production-code or format change.

The focused eight-case matrix, fmt, workspace Clippy, the required hosted checks, and Gatekeeper pass on this revision.

@Xuanwo
Xuanwo merged commit 250e5dc into lance-format:main Nov 5, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants