[opt](lance) push down LIMIT into Lance fragment scanners - #66608
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Ordinary Lance scans read every row of a fragment even when the query only needs the first N. Lance applies its LIMIT after the scanner's own filter, so we can forward the query LIMIT to each fragment scanner and let it stop early. To stay correct we only push the LIMIT when all predicates are already pushed into Lance (no residual Doris conjunct). Otherwise Doris still filters the returned rows and an early truncation could drop valid results. The upper LIMIT operator keeps enforcing the global bound across fragments, so per-fragment truncation is always safe. OFFSET needs no extra work: Nereids' SplitLimit rewrites Limit(limit, offset) into a global Limit(limit, offset) over a local Limit(limit + offset, 0), and that local bound is what reaches the scan node. So getLimit() already includes the offset and each fragment simply fetches up to limit + offset rows. - thrift: add optional TLanceFileDesc.limit - FE: set it in LanceScanNode.setScanParams via canPushDownLimit() (hasLimit() and no residual conjunct); also surface it in explain - BE: forward it via lance_scanner_set_limit for ordinary scans (vector search keeps its own top_k limit)
5ba2501 to
7fb0a61
Compare
Thanks for the reminder! I've just updated the PR description with the |
|
@hello-stephen Please take a look when you have time, thanks. |
|
设计和正确性判断总体是合理的，尤其是： 它没有真正覆盖核心逻辑。我会建议至少再补： @Jay-ju Hello, please add some more tests. |
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
3 participants
What problem does this PR solve?
Issue Number: N/A
Related PR: #65730, #66581
Problem Summary:
Ordinary Lance scans currently read every row of a fragment even when the
query only needs the first N rows (e.g.
SELECT ... LIMIT 10). Lance appliesits own LIMIT after the scanner's filter, so the query LIMIT can be forwarded
to each fragment scanner and let it stop early, cutting IO and decode cost.
How it is fixed
thrift: add an optionalTLanceFileDesc.limit.FE(LanceScanNode): push the query limit into each fragment split viacanPushDownLimit(), and surfacelanceLimitin the explain output.BE(lance_reader): forward it to the scanner throughlance_scanner_set_limitfor ordinary scans; vector search keeps its owntop_klimit.Correctness
The limit is pushed only when all predicates are already pushed into Lance
(no residual Doris conjunct). Otherwise Doris still re-filters the returned rows,
and truncating a fragment early could drop valid results.
OFFSETneeds no special handling: Nereids'SplitLimitrewritesLimit(limit, offset)into a globalLimit(limit, offset)over a localLimit(limit + offset, 0), and that local bound is what reaches the scan node.So
getLimit()already includes the offset; each fragment fetches up tolimit + offsetrows and the upper global LIMIT still applies the offset andthe final bound. Per-fragment truncation is therefore always safe.
Behavior change
Query results are unchanged. Only the number of rows scanned per fragment is
reduced for LIMIT queries; the explain output shows an extra
lanceLimit=Nline when the limit is pushed.
Release note
Push down LIMIT into Lance fragment scanners to reduce the rows scanned for
LIMIT/LIMIT ... OFFSETqueries over Lance tables.Check List (For Author)
LanceThriftContractTestcovers the limit round-trip and the no-limit case)SELECT * FROM <lance_tbl> LIMIT 10returns 10 rows andEXPLAINshowslanceLimit=10; a query with a non-pushable predicate keeps the limit out of the scan