[refactor](lance) improve lance catalog implementation in FE - #66581
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
I found one blocking compatibility issue with the
Consequently, a table that can be listed and described successfully by the currently supported REST fixture can be reported as non-existent (or fail with an exception, depending on the SDK error mapping) whenever a caller uses Please update the REST fixture and unit-test mock to implement the table-exists endpoint, and add coverage for:
If Doris intends to remain compatible with namespace implementations that support List/Describe but not TableExists, please also fall back to Also, the PR checklist currently says that this is a pure refactor with no behavior change and no tests needed. That is inaccurate: the previous implementation converted every |
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: N/A Related PR: #65730, #66581 Problem Summary: Ordinary Lance scans currently read **every row of a fragment** even when the query only needs the first N rows (e.g. `SELECT ... LIMIT 10`). Lance applies its own LIMIT *after* the scanner's filter, so the query LIMIT can be forwarded to each fragment scanner and let it stop early, cutting IO and decode cost. **How it is fixed** - `thrift`: add an optional `TLanceFileDesc.limit`. - `FE` (`LanceScanNode`): push the query limit into each fragment split via `canPushDownLimit()`, and surface `lanceLimit` in the explain output. - `BE` (`lance_reader`): forward it to the scanner through `lance_scanner_set_limit` for ordinary scans; vector search keeps its own `top_k` limit. **Correctness** The limit is pushed **only when all predicates are already pushed into Lance** (no residual Doris conjunct). Otherwise Doris still re-filters the returned rows, and truncating a fragment early could drop valid results. `OFFSET` needs no special handling: Nereids' `SplitLimit` rewrites `Limit(limit, offset)` into a global `Limit(limit, offset)` over a local `Limit(limit + offset, 0)`, and that local bound is what reaches the scan node. So `getLimit()` already includes the offset; each fragment fetches up to `limit + offset` rows and the upper global LIMIT still applies the offset and the final bound. Per-fragment truncation is therefore always safe. **Behavior change** Query results are unchanged. Only the number of rows scanned per fragment is reduced for LIMIT queries; the explain output shows an extra `lanceLimit=N` line when the limit is pushed. ### Release note Push down LIMIT into Lance fragment scanners to reduce the rows scanned for `LIMIT` / `LIMIT ... OFFSET` queries over Lance tables. ### Check List (For Author) - Test - [x] Unit Test (`LanceThriftContractTest` covers the limit round-trip and the no-limit case) - [ ] Manual test — `SELECT * FROM <lance_tbl> LIMIT 10` returns 10 rows and `EXPLAIN` shows `lanceLimit=10`; a query with a non-pushable predicate keeps the limit out of the scan - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
4 participants
What problem does this PR solve?
Problem Summary:
refactor some code including:
tableExistsAPI for table existence checks.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)