vector-search: support scalar pre-filter before Top-K

[shyjsarah]

Summary

Support scalar pre-filtering before Top-K for data-evolution/global-index vector search, including DataFusion literal and lateral vector_search queries.

The scalar predicate is evaluated against the same pinned snapshot as vector search, producing global row IDs that are localized per vector-index shard and passed to the vector backend as an allow-list. A normal Paimon read is used so scalar global indexes such as BTree can prune the filter read while residual evaluation, partial index coverage, and deletion visibility remain exact.

Changes

• Enable VectorSearchBuilder::with_filter and BatchVectorSearchBuilder::with_filter on the data-evolution path.
• Pin scalar filtering, deletion-vector handling, raw fallback, refine, and materialization to one resolved snapshot.
• Apply the same global row-ID allow-list to indexed search and unindexed raw fallback.
• Localize a shared bitmap once per vector-index shard, skip empty shards, and share bitmaps across batch queries without deep copies.
• Add a reusable prepared-filter API so lateral execution evaluates a target-side scalar predicate once per physical plan.
• Push fully translated DataFusion predicates into literal vector_search while retaining an Inexact residual correctness check.
• Push fully translated target-only conjuncts into lateral vector search while preserving left-side, cross-side, unsupported, and inexact residual predicates.
• Reject NaN predicates from vector pre-filter pushdown because Paimon and Arrow comparison semantics differ.
• Show pushed scalar filters in logical and physical EXPLAIN output.
• Document scalar pre-filter semantics and lateral-query behavior.

Testing

• cargo fmt --all --check
• git diff --check
• cargo test -p paimon --lib table::vector_search_builder::tests
• cargo test -p paimon-datafusion --lib filter_pushdown::tests::test_vector_prefilter_rejects_nan_literals
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vindex_build_then_vector_search_query -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture
• cargo clippy -p paimon --lib --tests -- -D warnings
• cargo clippy -p paimon-datafusion --lib --tests -- -D warnings

Covered cases include filter-before-Top-K ordering, empty matches, batch reuse, non-zero shard offsets, raw fallback for unindexed rows, time-travel snapshot pinning, literal SQL, lateral SQL, mixed/cross-side conjuncts, and NaN pushdown safety.

Notes

• No on-disk format change.
• No SQL function signature change.
• Existing VectorSearch::include_row_ids public API remains unchanged; this PR adds PreparedVectorSearchFilter and shared-filter builder support.
• This is scalar pre-filtering for vector search, not the existing vector/full-text route-fusion hybrid_search feature.

[jerry-024]

[MAJOR] The prepared scalar filter is cached for the lifetime of the physical execution plan, so re-executing the same plan after the target table advances keeps querying the snapshot selected by the first execution.

Scope the OnceCell to one query execution rather than storing it permanently on the reusable ExecutionPlan; share only an execution-scoped cache across partitions and batches.

[jerry-024]

[MAJOR] Detecting the shared pre-filter compares the entire RoaringTreemap once for every query in the batch, even though BatchVectorSearchBuilder installs the same Arc on every query.

Inspect shared_include_row_ids directly and use Arc::ptr_eq against the first query Arc; fall back to per-query localization if any query does not share that Arc.