[chore](thirdparty) Upgrade Apache Arrow to 24.0.0

[hubgeter]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Doris used Apache Arrow 17.0.0, which blocked consuming newer Arrow fixes and APIs. Upgrade the C++ third-party dependency to 24.0.0, align xsimd with Arrow 24, port the Paimon row-group, INT96, and Parquet LZO compatibility patches, add Arrow Compute to the static dependency graph, and migrate removed Arrow and Parquet APIs while preserving Decimal128 semantics.

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[hubgeter]

run buildall

[hubgeter]

/review

[github-actions]

Request changes: the Arrow 24 upgrade has two build-delivery blockers. The clean Linux third-party job succeeds, but both supported macOS third-party jobs fail at Arrow configuration, and an incremental Paimon source tree skips the new compatibility patch set.

Checkpoint conclusions:

• Goal and test sufficiency: not achieved on all supported paths; no upgrade-specific behavioral test was added, and the failed paths must be rerun after the fixes.
• Scope: all 13 authoritative changed paths and their Arrow/Paimon/build consumers were reviewed; the source changes are otherwise focused.
• Concurrency: no new shared-state or locking defect was introduced; Arrow's parallel column-reader ownership remains valid.
• Lifecycle and cleanup: runtime ownership/error cleanup is preserved, but extracted Paimon patch state is stale across an incremental upgrade (inline finding 2).
• Configuration and dynamic behavior: Arrow 24's CMake 3.25 floor conflicts with Doris's pinned 3.22.1 supported toolchains (inline finding 1).
• Compatibility and rolling upgrade: no persisted-data, journal, RPC, or storage-format rolling-upgrade change was found; source/API and prebuilt-library compatibility are blocked by the two findings.
• Parallel and equivalent paths: Linux fresh-source third-party CI passes; macOS and macOS-arm64 fail, macOS BE lacks the matching Compute archive, and an existing Paimon source directory follows the stale-marker path.
• Special conditions: malformed LZO, INT96 unit variants, projected/selected row groups, and decimal precision boundaries were traced. The empty-row-group truncation is real but behaviorally identical in the base Arrow 17 patch, so it is not attributed to this PR.
• Tests and results: review-only, so no local build/test was run. Live CI currently has Linux third-party passing; both macOS third-party jobs and macOS BE failing; Doris compile and BE UT pending.
• Observability: the build failures are explicit; no additional runtime observability regression was found.
• Persistence, transactions, data writes, and new variables: no new transaction/persistence or FE/BE variable path applies; the ported writer accounting and data semantics showed no PR-introduced defect.
• Performance: no substantiated CPU, memory, I/O, or link-time performance regression was found.
• Other issues: no additional nonduplicate issue remained after two convergence rounds.

No additional user-provided review focus was supplied; the whole PR was reviewed.

[github-actions]

[P1] Raise the supported CMake floor before selecting Arrow 24

Arrow 24's cpp/CMakeLists.txt now requires CMake 3.25, but both macOS third-party workflows explicitly install 3.22.1 (and the gcc10 compilation image pins the same version). The current macOS and macOS-arm64 jobs both fail at this configure step with CMake 3.25 or higher is required; CMAKE_POLICY_VERSION_MINIMUM cannot override cmake_minimum_required. Please update every supported Arrow build path to provide CMake >=3.25 and regenerate the matching prebuilt archives before adopting this version, otherwise the standard macOS/toolchain builds remain unusable.

[github-actions]

[P1] Invalidate the old Paimon patch marker

PAIMON_CPP_SOURCE is unchanged, so an incremental tree already patched on the base commit still has patched_mark. This guard then skips this whole new loop, including paimon-cpp-arrow-24-compatibility.patch, leaving the removed three-argument unique_ptr out-parameter GetRecordBatchReader calls and bare RETURN_NOT_OK uses in place when building against Arrow 24. Please invalidate/re-extract the source when the patch set changes (for example with a patch-set-specific marker or a bumped source identity), so the documented rerunnable download flow cannot silently retain incompatible sources.

[hubgeter]

run buildall

[hubgeter]

/review

[github-actions]

Request changes: the Arrow 24 upgrade still has one build-delivery blocker and one CI integration-coverage defect. The targeted macOS path cannot obtain xsimd 14, and the generic third-party trigger rebuilds only Arrow/Paimon.

Checkpoint conclusions:

• Goal and test sufficiency: not achieved on the targeted supported macOS path. The changed BE tests adapt Arrow APIs but do not exercise third-party source acquisition or generic third-party-to-BE integration.
• Scope: all 17 authoritative changed paths and their Arrow/Paimon/build consumers were reviewed. The upgrade is mostly focused, but the new macOS trigger is broader than its implementation.
• Concurrency: no runtime shared-state, locking, or static-initialization defect applies to these build and API changes.
• Lifecycle and cleanup: runtime reader/writer ownership and error propagation remain intact; workflow source/prebuilt artifact lifecycle is broken by inline finding 1, and dependency selection is stale in inline finding 2.
• Configuration: no dynamic Doris configuration item was added. Current-head CMake/toolchain selection is compatible with Arrow 24, and no separate configuration defect survived validation.
• Compatibility: the inspected Arrow 24 API migrations, INT96 property propagation, row-group behavior, and Doris external-Arrow Paimon path are consistent. No persisted-data, journal, RPC, or rolling-upgrade format defect was found.
• Parallel paths: Linux, Intel/arm64 macOS, fresh/incremental source, Doris external-Arrow Paimon, and final static-link paths were traced. The unsupported standalone Paimon bundled-Arrow branch was not treated as a Doris defect.
• Special conditions: the broad thirdparty/** condition does not match the hard-coded dependency set rebuilt (inline finding 2); no other incorrect conditional path survived validation.
• Test coverage: no upgrade-specific end-to-end test covers source-bundle closure or generic third-party-to-BE integration.
• Test results: review-only, so no local build/test was run. Live BE UT (macOS) currently fails exactly because 14.0.0.tar.gz is missing; style, license, secret, and dependency checks pass, while the full third-party and Doris compile checks were still pending at freeze time.
• Observability: the CI failure is explicit and diagnostic; no additional runtime logs or metrics are needed for this change.
• Persistence, transactions, and data writes: no FE journal, transaction, storage-version, or atomic data-write path is modified.
• FE/BE propagation: no new FE/BE variable or protocol field is introduced.
• Performance: no substantiated runtime CPU, memory, or I/O regression was found. The broad macOS trigger adds avoidable CI work while still providing stale integration coverage, as noted inline.
• Other issues: the remaining CMake-floor and incremental patch-marker concerns are already covered by existing P1 threads, so they were not duplicated. The apparent LZO include failure was disproved because thrift::thrift propagates the installed include root to Parquet's object target.

No additional user-provided review focus was supplied; the whole PR was reviewed.

[github-actions]

[P1] Fetch xsimd 14 before this targeted Arrow build

Passing only arrow paimon_cpp restricts download-thirdparty.sh to those two archives, but Arrow is configured with xsimd_SOURCE=BUNDLED and a local ${TP_SOURCE_DIR}/14.0.0.tar.gz URL. The shared source bundle does not contain that newly selected archive, so the live macOS job fails in xsimd_ep with File not found: .../thirdparty/src/14.0.0.tar.gz before Arrow is compiled. Please download Arrow's complete source-only dependency closure (at least xsimd) before this targeted build, or provide a source bundle guaranteed to match the checked-out vars.sh.

[github-actions]

[P2] Align this broad trigger with the dependency set actually rebuilt

thirdparty_changes matches every file under thirdparty/**, but this branch always rebuilds only Arrow and Paimon. For example, a protobuf version or patch change triggers this BE job while both Arrow and BE still consume protobuf from the shared prebuilt, so a green result would not validate the PR-selected dependency as the new comments imply. Please either narrow the filter to the Arrow/Paimon inputs this step covers or derive and rebuild the changed dependency set (including the required transitive closure).

[hubgeter]

run buildall

[hello-stephen]

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.05% (25558/43285)
Line Coverage	43.13% (256279/594178)
Region Coverage	38.82% (203053/523034)
Branch Coverage	40.16% (92577/230543)

[hello-stephen]

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.06% (25571/43296)
Line Coverage	43.14% (256363/594222)
Region Coverage	38.83% (203102/523058)
Branch Coverage	40.17% (92604/230557)

[hubgeter]

run buildall

[hubgeter]

/review

[github-actions]

Reviewed current head 1f017bb6c4b83d0b6ca16f25456ae1ef16b387b0. I found no new nonduplicate issue. The xsimd source-closure, Paimon marker, and broad-trigger defects are addressed on this head; the existing CMake-floor P1 still applies to docker/compilation/Dockerfile.gcc7, which remains on CMake 3.12.3 while Arrow 24 requires 3.25.

Checkpoint conclusions:

• Goal and test sufficiency: the Arrow 24/xsimd/Paimon migration is coherent across source acquisition, patching, compilation targets, runtime API migrations, and the changed semantic tests apart from the already-threaded gcc7 CMake floor.
• Scope: all 20 authoritative changed files and their relevant build/runtime consumers were reviewed; the changes remain focused on the dependency upgrade and its integration.
• Concurrency: no new shared-state, locking, static-initialization, or parallel-reader ownership defect was found.
• Lifecycle and cleanup: fresh and incremental Paimon patch states converge, source/prebuilt selection is consistent, and reader/writer ownership and error propagation remain intact.
• Configuration and dynamic behavior: current Linux, macOS, gcc10, and arm paths select CMake >=3.25 and the complete Arrow source-only dependency closure. The maintained gcc7 image remains below the floor under the existing P1; no Doris runtime configuration path is added.
• Compatibility and rolling upgrade: Decimal128 preserves Doris/Iceberg's supported precision, Flight/Paimon API migrations are complete, and no persisted-data, journal, RPC, or storage-format incompatibility was found.
• Parallel and equivalent paths: Linux, Intel/arm64 macOS, gcc10/arm/gcc7 images, fresh/incremental sources, Paimon external-Arrow targets, and final BE static linkage were traced. The gcc7 residue is covered by the existing broad CMake-floor thread rather than duplicated here.
• Special conditions: selected/final/empty row groups, compressed-byte thresholds, forced INT96, malformed/truncated LZO framing, decimal precision boundaries, and source-marker partial states were checked. The empty-row-group termination edge is inherited unchanged from the Arrow 17 patch.
• Tests and results: review-only, so no local build or test command was run. The changed tests exercise Decimal128 alignment, Flight ticket serialization, external-reader behavior, and Iceberg Parquet metrics; live third-party and BE jobs were still running at the review freeze.
• Observability: build/setup failures remain explicit in CI and no runtime logging or metrics regression was introduced.
• Persistence, transactions, data writes, and new variables: no FE journal, transaction protocol, storage-version, atomic-write, or FE/BE variable-propagation path is added; writer property and statistics semantics remain consistent.
• Performance: no substantiated CPU, memory, I/O, archive-link, or row-group-size regression was found.
• Other issues: the apparent LZO include failure was disproved because thrift::thrift propagates the installed include root to Parquet's object target. No additional nonduplicate issue survived main-agent validation; the outstanding gcc7 build path remains covered by the live CMake-floor P1.

No additional user-provided review focus was supplied; the whole PR was reviewed.

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.73% (31978/42228)
Line Coverage	60.39% (356476/590253)
Region Coverage	56.94% (298796/524780)
Branch Coverage	58.30% (134434/230588)

[hubgeter]

run buildall

[hubgeter]

/review

[hubgeter]

run buildall

[github-actions]

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30601197902

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[hubgeter]

/review

[github-actions]

Request changes: the current Arrow 24 head leaves the scheduled macOS BE path on an incompatible Arrow 17 prebuilt. The PR-triggered overlay is green, but the scheduled build skips that overlay while BE now imports the separate Arrow Compute archive unconditionally.

Checkpoint conclusions:

• Goal and test sufficiency: not achieved for the scheduled/default-branch path. The changed tests cover API migrations and Iceberg output, but no check validates the prebuilt's Arrow version or archive closure.
• Scope: all 22 authoritative changed files and their relevant Arrow/Paimon, BE-link, workflow, patch, and test consumers were reviewed; the changes remain focused on the dependency upgrade and integration.
• Concurrency: no new shared-state, locking, static-initialization, or parallel-reader ownership defect was found.
• Lifecycle and cleanup: runtime reader/writer ownership and error propagation remain intact. The prebuilt/source artifact lifecycle is incompatible on the skipped scheduled path (inline finding).
• Configuration and dynamic behavior: current-head Linux, macOS, gcc7, gcc10, and arm build definitions select CMake >=3.25; no Doris runtime configuration path is added. The scheduled workflow condition is the remaining configuration defect.
• Compatibility and rolling upgrade: the Arrow 24 API migrations, Decimal128 mapping, INT96 property propagation, LZO port, and Paimon link graph are consistent. No persisted-data, journal, RPC, or storage-format rolling-upgrade defect was found; the current macOS prebuilt is binary-incompatible with the new unconditional Compute import.
• Parallel and equivalent paths: clean Linux, macOS, and macOS-arm64 third-party builds and the PR-triggered macOS BE overlay pass. Scheduled and later BE-only runs skip that overlay and use the stale asset.
• Special conditions: scheduled/default-branch event behavior, clean/incremental sources, selected/final/empty row groups, malformed LZO framing, INT96 units, Decimal128 precision, Flight tickets, and source-marker states were traced. The real zero-row-group termination edge is inherited unchanged from the Arrow 17 patch and is not attributed to this PR.
• Tests and results: review-only, so no local build or test was run. Exact-head full third-party and targeted macOS BE checks pass. The TeamCity BE UT failure is in a test added on a newer integration revision and absent from the authoritative diff; external regressions were still pending at freeze time.
• Observability: build and patch failures are explicit, but this workflow has no preflight assertion for the prebuilt Arrow version/archive set; that omission is part of the inline finding.
• Persistence, transactions, data writes, and variables: no FE journal, transaction protocol, storage-version, atomic-write, or FE/BE variable-propagation path is introduced.
• Performance: no substantiated runtime CPU, memory, I/O, or link-time performance regression was found.
• Other issues: all other candidates were either duplicates of existing live threads or disproved as inherited base behavior.

No additional user-provided review focus was supplied; the whole PR was reviewed.

[github-actions]

[P1] Keep scheduled builds off the stale Arrow 17 prebuilt

Paths Filter is skipped for schedule, so arrow_paimon_changes is empty and this branch never runs on the scheduled job. The current automation prebuilt used above contains libarrow.a, libarrow_dataset.a, and libarrow_acero.a but no libarrow_compute.a (and its source bundle still contains Arrow 17); only this PR-triggered branch installs libarrow_compute.a. Once the unconditional Compute import in be/cmake/thirdparty.cmake lands, scheduled master builds—and later BE-only PRs that do not set this filter—will try to link against that stale archive and fail. Please either regenerate/publish the version-matched prebuilt before landing and validate its Arrow version, or rebuild the Arrow/Paimon closure on the scheduled path as well. This is separate from the existing xsimd thread because scheduled runs never reach the targeted rebuild at all.

[hello-stephen]

skip check_coverage