[feature](lance) Add Rust-based Lance format reader for AI-native dat… by tomz-alt · Pull Request #62182 · apache/doris

tomz-alt · 2026-04-07T17:01:50Z

Summary

Add native Lance format support to Doris via Rust FFI integration, enabling SQL queries over AI-native Lance datasets from local disk and S3.

Lance is a columnar format designed for vector search, multimodal data (images, embeddings), and fast random access -- widely used in AI/ML pipelines.

Quick Examples

Read from S3:

SELECT * FROM s3(
    "uri" = "s3://bucket/embeddings.lance/data/fragment.lance",
    "format" = "lance",
    "s3.access_key" = "...",
    "s3.secret_key" = "...",
    "s3.region" = "us-east-1",
    "s3.endpoint" = "https://s3.us-east-1.amazonaws.com"
) ORDER BY id LIMIT 10;

Read from local disk (for testing):

-- Get backend_id from: SHOW BACKENDS;
SELECT * FROM local(
    "file_path" = "data/my_dataset.lance/data/fragment.lance",
    "backend_id" = "<backend_id from SHOW BACKENDS>",
    "format" = "lance"
) ORDER BY id LIMIT 10;

Aggregation across multi-fragment dataset:

SELECT count(*), min(id), max(id) FROM s3(
    "uri" = "s3://bucket/large.lance/data/fragment.lance",
    "format" = "lance",
    "s3.access_key" = "...", "s3.secret_key" = "...",
    "s3.region" = "us-east-1",
    "s3.endpoint" = "https://s3.us-east-1.amazonaws.com"
);

Architecture

Data exchange: Arrow C Data Interface (zero-copy between Rust and C++)
Async containment: block_on() with single-threaded tokio runtime (zero extra OS threads)
Build gating: BUILD_RUST_READERS=OFF by default, zero impact on existing builds

What Works (Verified on Live Cluster)

Feature	Status
SELECT * / column projection	Tested
WHERE filter / LIMIT / COUNT(*)	Tested
SUM() / AVG() aggregation	Tested
Multi-fragment datasets (3 fragments, 15 rows)	Tested
S3 access with AWS credentials	Tested
Schema inference (fetch_table_schema)	Tested
Time travel version (config wired)	Config ready
Vector ANN search / FTS / filter pushdown	Config ready

Known Limitations

TVF path only: No CREATE CATALOG support yet. Must use local() or s3() TVF
Directory-based format workaround: Lance datasets are directories. The TVF file_path must point to a single .lance data file inside the dataset; the reader auto-strips the path back to the dataset root and reads all fragments. If the TVF glob matches multiple .lance files (multi-fragment dataset), each scan range reopens the full dataset causing duplicate rows. Workaround: ensure the file_path glob matches exactly one data file per dataset
No Doris data cache integration: Lance reads bypass BlockFileCache. S3 reads are not cached on local SSD
No filter/vector pushdown from FE: The Rust config supports filter, vector_search, full_text_search but the FE planner does not populate them yet
BUILD_RUST_READERS=OFF default: Requires explicit opt-in and Rust toolchain
Binary size: Rust static lib ~430MB (.a), adds ~50-80MB to final doris_be after LTO

How to Build

# Rust tests only:
cd be/src/rust/doris-native && cargo test

# Full BE with Lance:
BUILD_RUST_READERS=ON ./build.sh --be

# Regression test:
./run-regression-test.sh --run -s test_lance_tvf

Changes

Thrift (2 files): FORMAT_LANCE = 19, TLanceFileDesc, enable_rust_lance_reader

FE (4 files): LanceFileFormatProperties, FileFormatProperties factory, FileFormatConstants, SessionVariable

BE - Rust (be/src/rust/doris-native/, 6 files): error.rs, lance_reader.rs (LanceReaderConfig with S3/version/vector/FTS support), ffi.rs (extern C functions), lib.rs

BE - C++ (5 files): lance_ffi.h, lance_rust_reader.h/cpp (GenericReader with Arrow import), file_scanner.cpp (FORMAT_LANCE dispatch), internal_service.cpp (fetch_table_schema)

Build (3 files): rust.cmake (Corrosion v0.5), CMakeLists.txt (BUILD_RUST_READERS option), format/CMakeLists.txt

Tests (4 files): 24 Rust tests, C++ GTest (7), C++ standalone (8), Groovy regression (9)

Future Work

Lance Catalog: CREATE CATALOG for dataset discovery (eliminates TVF path limitations and backend_id requirement)
Fragment-level scan ranges: FE lists fragments and creates one scan range per fragment with fragment ID, avoiding duplicate reads
FE filter/vector pushdown: Pass WHERE predicates and ANN queries to lance-rs scanner
Doris BlockFileCache integration: Route lance I/O through Doris cache for S3 data caching
Lance Session cache: Shared IndexCache for vector/FTS index reuse across queries
Time travel SQL: FOR VERSION AS OF N via LanceMvccSnapshot

Thearas · 2026-04-07T17:01:55Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-04-07T18:17:20Z

run buildall

hello-stephen · 2026-04-07T19:41:12Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

tomz-alt · 2026-04-07T21:00:26Z

we need to install cargo toolchain to make rust ffi compile

hello-stephen · 2026-04-07T22:19:06Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.98% (20105/37947)
Line Coverage	36.53% (188940/517204)
Region Coverage	32.78% (146566/447093)
Branch Coverage	33.93% (64209/189233)

hello-stephen · 2026-04-07T22:26:00Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.69% (27386/37162)
Line Coverage	57.31% (295496/515622)
Region Coverage	54.51% (245979/451226)
Branch Coverage	56.18% (106642/189815)

hello-stephen · 2026-04-08T08:15:42Z

run buildall

hello-stephen · 2026-04-08T09:16:57Z

TeamCity pipeline will use regression-test/pipeline/common/custom_env.sh when compiling, please set BUILD_RUST_READERS=on in it for test.

hello-stephen · 2026-04-08T10:26:03Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.00% (20112/37947)
Line Coverage	36.56% (189072/517225)
Region Coverage	32.81% (146716/447111)
Branch Coverage	33.94% (64233/189251)

hello-stephen · 2026-04-08T10:46:05Z

BE Regression && UT Coverage Report

Increment line coverage 89.87% (71/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.72% (27397/37162)
Line Coverage	57.36% (295782/515643)
Region Coverage	54.61% (246412/451244)
Branch Coverage	56.29% (106852/189833)

hello-stephen · 2026-04-12T14:32:45Z

run buildall

hello-stephen · 2026-04-12T16:02:58Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

morningman · 2026-04-13T17:39:47Z

run buildall

tomz-alt · 2026-04-13T18:58:56Z

run buildall

hello-stephen · 2026-04-13T19:35:27Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.17% (32304/50345)
Region Coverage	65.13% (16260/24967)
Branch Coverage	55.63% (8689/15620)

hello-stephen · 2026-04-13T20:23:06Z

FE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉
Increment coverage report
Complete coverage report

tomz-alt · 2026-04-20T20:51:25Z

run buildall

Upstream master refactored GenericReader virtual methods: get_next_block -> _do_get_next_block (protected hook) get_columns(name_to_type, missing_cols) -> _get_columns_impl(name_to_type) Update LanceRustReader's overrides to match the new signatures so CI (which builds our PR on top of current master) compiles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…o lance-support

morningman · 2026-04-20T22:48:10Z

run buildall

tomz-alt · 2026-04-21T06:52:43Z

run buildall

suxiaogang223 · 2026-04-21T10:42:05Z

run buildall

morningman · 2026-04-21T15:41:26Z

run buildall

tomz-alt · 2026-04-21T18:09:55Z

so many PR per day....

hello-stephen · 2026-04-21T19:31:20Z

FE Regression Coverage Report

Increment line coverage 90.91% (10/11) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-21T19:51:36Z

BE Regression && UT Coverage Report

Increment line coverage 0.68% (2/296) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.71% (26833/37418)
Line Coverage	54.95% (285305/519212)
Region Coverage	51.76% (235098/454223)
Branch Coverage	53.27% (101803/191106)

morningman · 2026-04-07T18:04:44Z

+            fuzzy = true,
+            description = {"使用 Rust Lance 读取器读取 Lance 格式数据",
+                    "Use Rust-based Lance reader for Lance format data"})
+    private boolean enableRustLanceReader = false;


Do we really need this variable?

github-actions · 2026-04-21T20:09:48Z

PR approved by at least one committer and no changes requested.

dismiss

github-actions · 2026-04-21T20:22:49Z

PR approved by anyone and no changes requested.

jja725 · 2026-05-06T23:53:54Z

Hi @BiteTheDDDDt @tomz-alt , just saw this great integration! We are working on https://github.com/lance-format/lance-c which is a standard c/c++ api for lance and want to see if we can unify our interface. We are running it with Velox and happy to talk more if you guys are interested

vkingnew · 2026-05-09T11:29:17Z

which version of apache doris is released this furture GA ?

apache#62182) Add native Lance format support to Doris via Rust FFI integration, enabling SQL queries over AI-native Lance datasets from local disk and S3. [Lance](https://lancedb.github.io/lance/) is a columnar format designed for vector search, multimodal data (images, embeddings), and fast random access -- widely used in AI/ML pipelines. ## Quick Examples **Read from S3:** ```sql SELECT * FROM s3( "uri" = "s3://bucket/embeddings.lance/data/fragment.lance", "format" = "lance", "s3.access_key" = "...", "s3.secret_key" = "...", "s3.region" = "us-east-1", "s3.endpoint" = "https://s3.us-east-1.amazonaws.com" ) ORDER BY id LIMIT 10; ``` **Read from local disk (for testing):** ```sql -- Get backend_id from: SHOW BACKENDS; SELECT * FROM local( "file_path" = "data/my_dataset.lance/data/fragment.lance", "backend_id" = "<backend_id from SHOW BACKENDS>", "format" = "lance" ) ORDER BY id LIMIT 10; ``` **Aggregation across multi-fragment dataset:** ```sql SELECT count(*), min(id), max(id) FROM s3( "uri" = "s3://bucket/large.lance/data/fragment.lance", "format" = "lance", "s3.access_key" = "...", "s3.secret_key" = "...", "s3.region" = "us-east-1", "s3.endpoint" = "https://s3.us-east-1.amazonaws.com" ); ``` ## Architecture - **Data exchange**: Arrow C Data Interface (zero-copy between Rust and C++) - **Async containment**: block_on() with single-threaded tokio runtime (zero extra OS threads) - **Build gating**: BUILD_RUST_READERS=OFF by default, zero impact on existing builds ## What Works (Verified on Live Cluster) | Feature | Status | |---------|--------| | SELECT * / column projection | Tested | | WHERE filter / LIMIT / COUNT(*) | Tested | | SUM() / AVG() aggregation | Tested | | Multi-fragment datasets (3 fragments, 15 rows) | Tested | | S3 access with AWS credentials | Tested | | Schema inference (fetch_table_schema) | Tested | | Time travel version (config wired) | Config ready | | Vector ANN search / FTS / filter pushdown | Config ready | ## Known Limitations - **TVF path only**: No CREATE CATALOG support yet. Must use local() or s3() TVF - **Directory-based format workaround**: Lance datasets are directories. The TVF file_path must point to a single `.lance` data file inside the dataset; the reader auto-strips the path back to the dataset root and reads all fragments. If the TVF glob matches multiple `.lance` files (multi-fragment dataset), each scan range reopens the full dataset causing **duplicate rows**. Workaround: ensure the file_path glob matches exactly one data file per dataset - **No Doris data cache integration**: Lance reads bypass BlockFileCache. S3 reads are not cached on local SSD - **No filter/vector pushdown from FE**: The Rust config supports filter, vector_search, full_text_search but the FE planner does not populate them yet - **BUILD_RUST_READERS=OFF default**: Requires explicit opt-in and Rust toolchain - **Binary size**: Rust static lib ~430MB (.a), adds ~50-80MB to final doris_be after LTO ## How to Build ```bash # Rust tests only: cd be/src/rust/doris-native && cargo test # Full BE with Lance: BUILD_RUST_READERS=ON ./build.sh --be # Regression test: ./run-regression-test.sh --run -s test_lance_tvf ``` ## Changes **Thrift** (2 files): FORMAT_LANCE = 19, TLanceFileDesc, enable_rust_lance_reader **FE** (4 files): LanceFileFormatProperties, FileFormatProperties factory, FileFormatConstants, SessionVariable **BE - Rust** (be/src/rust/doris-native/, 6 files): error.rs, lance_reader.rs (LanceReaderConfig with S3/version/vector/FTS support), ffi.rs (extern C functions), lib.rs **BE - C++** (5 files): lance_ffi.h, lance_rust_reader.h/cpp (GenericReader with Arrow import), file_scanner.cpp (FORMAT_LANCE dispatch), internal_service.cpp (fetch_table_schema) **Build** (3 files): rust.cmake (Corrosion v0.5), CMakeLists.txt (BUILD_RUST_READERS option), format/CMakeLists.txt **Tests** (4 files): 24 Rust tests, C++ GTest (7), C++ standalone (8), Groovy regression (9) ## Future Work - **Lance Catalog**: CREATE CATALOG for dataset discovery (eliminates TVF path limitations and backend_id requirement) - **Fragment-level scan ranges**: FE lists fragments and creates one scan range per fragment with fragment ID, avoiding duplicate reads - **FE filter/vector pushdown**: Pass WHERE predicates and ANN queries to lance-rs scanner - **Doris BlockFileCache integration**: Route lance I/O through Doris cache for S3 data caching - **Lance Session cache**: Shared IndexCache for vector/FTS index reuse across queries - **Time travel SQL**: FOR VERSION AS OF N via LanceMvccSnapshot --------- Co-authored-by: tom zhang <tom.zhang@velodb.io> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>

…tive data (#62182) (#65303) ### What problem does this PR solve? This reverts commit b613c37. #62182 will be later support lance catalog full. Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

tomz-alt requested review from BiteTheDDDDt and zclllyybb as code owners April 7, 2026 17:01

morningman self-assigned this Apr 7, 2026

tomz-alt force-pushed the lance-support branch 5 times, most recently from 9825d0e to 6f655a4 Compare April 7, 2026 18:15

tomz-alt force-pushed the lance-support branch from 6f655a4 to 2b57665 Compare April 7, 2026 22:45

tomz-alt force-pushed the lance-support branch 3 times, most recently from a02a47a to 9dcf75e Compare April 13, 2026 06:19

tomz-alt force-pushed the lance-support branch 2 times, most recently from d306c78 to f0bad39 Compare April 13, 2026 18:41

tom zhang and others added 3 commits April 20, 2026 15:42

Merge remote-tracking branch 'upstream/master' into lance-support

4025e89

Merge branch 'lance-support' of https://github.com/tomz-alt/doris int…

daf1e42

…o lance-support

morningman added the kind/need-document label Apr 21, 2026

Merge branch 'master' into lance-support

f12a8bb

fix conflict

5b01233

morningman approved these changes Apr 21, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2026

github-actions Bot added the reviewed label Apr 21, 2026

BiteTheDDDDt approved these changes Apr 22, 2026

View reviewed changes

BiteTheDDDDt merged commit b613c37 into apache:master Apr 22, 2026
30 of 33 checks passed

morningman added kind/need-document-merged and removed kind/need-document labels May 23, 2026

zhangstar333 mentioned this pull request Jul 7, 2026

Revert "[feature](lance) Add Rust-based Lance format reader for AI-native data (#62182) #65303

Merged

16 tasks

Jay-ju mentioned this pull request Aug 23, 2026

[feature](lance) push COUNT(*) down to Lance dataset metadata #66999

Open

Uh oh!

Conversation

tomz-alt commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Quick Examples

Architecture

What Works (Verified on Live Cluster)

Known Limitations

How to Build

Changes

Future Work

Uh oh!

Thearas commented Apr 7, 2026

Uh oh!

morningman commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

FE UT Coverage Report

Uh oh!

tomz-alt commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 7, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 8, 2026

Uh oh!

hello-stephen commented Apr 8, 2026

Uh oh!

hello-stephen commented Apr 8, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 8, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 12, 2026

Uh oh!

hello-stephen commented Apr 12, 2026

FE UT Coverage Report

Uh oh!

morningman commented Apr 13, 2026

Uh oh!

tomz-alt commented Apr 13, 2026

Uh oh!

hello-stephen commented Apr 13, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 13, 2026

FE UT Coverage Report

Uh oh!

tomz-alt commented Apr 20, 2026

Uh oh!

morningman commented Apr 20, 2026

Uh oh!

tomz-alt commented Apr 21, 2026

Uh oh!

suxiaogang223 commented Apr 21, 2026

Uh oh!

morningman commented Apr 21, 2026

Uh oh!

tomz-alt commented Apr 21, 2026

Uh oh!

hello-stephen commented Apr 21, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Apr 21, 2026

BE Regression && UT Coverage Report

Uh oh!

morningman Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 21, 2026

Uh oh!

github-actions Bot commented Apr 21, 2026

Uh oh!

Uh oh!

jja725 commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

tomz-alt commented Apr 7, 2026 •

edited

Loading

jja725 commented May 6, 2026 •

edited

Loading