Skip to content

[refactor](thirdparty) Align Arrow upgrade with standard thirdparty workflow - #67432

Draft
hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:worktree/apache-master-20260902
Draft

[refactor](thirdparty) Align Arrow upgrade with standard thirdparty workflow#67432
hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:worktree/apache-master-20260902

Conversation

@hubgeter

@hubgeter hubgeter commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #66221

Problem Summary: The Arrow 24 upgrade introduced an Arrow/Paimon-specific semantic-fingerprint system, legacy marker allowlists, installed-artifact validators, focused recovery paths, and dedicated lifecycle CI. This diverged from the standard Doris thirdparty workflow and made dependency maintenance unnecessarily complex. Follow the existing model used by other dependency upgrades: keep source versions and checksums in vars.sh, use patch markers for source preparation, and rebuild the complete thirdparty installation when the platform completion sentinel or libarrow_compute.a is missing. Remove the special fingerprint and macOS recovery paths while preserving the Arrow 24/Paimon patches, bundled xsimd/Brotli source closure, and later ADBC/Paimon configuration tests.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
[refactor](thirdparty) Align Arrow upgrade with standard thirdparty w…
d5b7f4e 
…orkflow

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66221

Problem Summary: The Arrow 24 upgrade introduced an Arrow/Paimon-specific semantic-fingerprint system, legacy marker allowlists, installed-artifact validators, focused recovery paths, and dedicated lifecycle CI. This diverged from the standard Doris thirdparty workflow and made dependency maintenance unnecessarily complex. Follow the existing model used by other dependency upgrades: keep source versions and checksums in vars.sh, use patch markers for source preparation, rely on matching thirdparty prebuilts, and use the existing platform completion sentinel to trigger a full rebuild. Remove the special fingerprint and macOS recovery paths while preserving the Arrow 24/Compute/Paimon build changes, bundled xsimd/Brotli source closure, and later ADBC/Paimon configuration tests.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Bash syntax checks for all changed shell scripts
    - YAML parsing for the changed GitHub workflows
    - Thirdparty download, checksum, Azure retry, ADBC, and Paimon codec tests
    - Clean Arrow/Paimon/xsimd/Brotli download and patch application
    - Repeated download for idempotence and Paimon patch-marker migration
    - ShellCheck with error severity and git diff --check
    - Full thirdparty compilation was not run
- Behavior changed: Yes. Arrow/Paimon now follow the standard matching-prebuilt and full-rebuild thirdparty workflow instead of dedicated installed fingerprints and focused recovery.
- Does this need documentation: No
@hubgeter
hubgeter force-pushed the worktree/apache-master-20260902 branch from 5a43852 to d5b7f4e Compare September 2, 2026 06:52
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 2, 2026
@suxiaogang223
[chore](paimon) remove paimon-cpp dependency
7e3596d 
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67385, apache#67432

Problem Summary:

Remove the deprecated paimon-cpp source, build, reader, configuration, and test dependencies while retaining deprecated Thrift wire values for rolling-upgrade compatibility. Keep branch-4.1 aligned with master by using the standard thirdparty manifest and removing the temporary Arrow fingerprint, recovery, lifecycle, and Paimon-specific patch files. The shared thirdparty package continues to be produced from master.

### Release note

Remove the deprecated paimon-cpp reader and build dependency.

### Check List (For Author)

- Test: Unit Test and manual test
  - Paimon FE unit tests
  - Clean Arrow 24 patch application and idempotent download flow
  - Thirdparty script tests, shell syntax, workflow YAML, and diff checks
- Behavior changed: Yes. Paimon logical splits use JNI; deprecated wire values remain for compatibility.
- Does this need documentation: No
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 2, 2026
@suxiaogang223
[chore](paimon) Remove paimon-cpp dependency
5d08a80 
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67378, apache#67432

Problem Summary:

Remove the deprecated paimon-cpp package, reader implementation, build integration, tests, and Paimon-specific Arrow patch. Retain deprecated wire and configuration values for compatibility. Fold Arrow source variables back into the standard thirdparty manifest and remove the temporary fingerprint, recovery, and lifecycle scripts so the next master automation package is the final cleaned artifact.

### Release note

Remove the deprecated paimon-cpp reader and build dependency. Paimon reads continue through the JNI and native readers.

### Check List (For Author)

- Test: Unit Test and manual test
  - Paimon FE connector reactor: 553 tests passed, 1 live test skipped
  - Clean Arrow 24 patch application and Arrow package build
  - Thirdparty script tests, build hygiene, shell syntax, workflow YAML, and diff checks
- Behavior changed: Yes. The paimon-cpp reader and enable_paimon_cpp_reader session variable are removed; deprecated wire values remain for compatibility.
- Does this need documentation: No
This was referenced Sep 2, 2026
Merged
Merged
@hubgeter
hubgeter marked this pull request as draft September 2, 2026 08:12
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 2, 2026
@suxiaogang223
[chore](paimon) Remove paimon-cpp dependency
2a32705 
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67378, apache#67432

Problem Summary:

Remove the deprecated paimon-cpp package, reader implementation, build integration, tests, and Paimon-specific Arrow patch. Retain deprecated wire and configuration values for compatibility. Fold Arrow source variables back into the standard thirdparty manifest and remove the temporary fingerprint, recovery, and lifecycle scripts so the next master automation package is the final cleaned artifact.

### Release note

Remove the deprecated paimon-cpp reader and build dependency. Paimon reads continue through the JNI and native readers.

### Check List (For Author)

- Test: Unit Test and manual test
  - Paimon FE connector reactor: 553 tests passed, 1 live test skipped
  - Clean Arrow 24 patch application and Arrow package build
  - Thirdparty script tests, build hygiene, shell syntax, workflow YAML, and diff checks
- Behavior changed: Yes. The paimon-cpp reader and enable_paimon_cpp_reader session variable are removed; deprecated wire values remain for compatibility.
- Does this need documentation: No
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 2, 2026
@suxiaogang223
[chore](paimon) remove paimon-cpp dependency
77d29b7 
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67385, apache#67432

Problem Summary:

Remove the deprecated paimon-cpp source, build, reader, configuration, and test dependencies while retaining deprecated Thrift wire values for rolling-upgrade compatibility. Keep branch-4.1 aligned with master by using the standard thirdparty manifest and removing the temporary Arrow fingerprint, recovery, lifecycle, and Paimon-specific patch files. The shared thirdparty package continues to be produced from master.

### Release note

Remove the deprecated paimon-cpp reader and build dependency.

### Check List (For Author)

- Test: Unit Test and manual test
  - Paimon FE unit tests
  - Clean Arrow 24 patch application and idempotent download flow
  - Thirdparty script tests, shell syntax, workflow YAML, and diff checks
- Behavior changed: Yes. Paimon logical splits use JNI; deprecated wire values remain for compatibility.
- Does this need documentation: No
yiguolei pushed a commit that referenced this pull request Sep 2, 2026
@suxiaogang223
branch-4.1: [chore](paimon) remove paimon-cpp dependency (#67378)
57636ca 
### What problem does this PR solve?

Issue Number: N/A

Related PR: #67385, #67432

Remove paimon-cpp completely from branch-4.1 while retaining deprecated
Thrift wire values for rolling-upgrade compatibility. Keep its
thirdparty sources, Arrow patches, and standard scripts aligned with
master; branch-4.1 does not publish a separate thirdparty package and
consumes the package produced from master.

### Release note

Remove the deprecated paimon-cpp reader and build dependency.

### Check List (For Author)

- Test:
    - [x] Paimon FE unit tests
    - [x] Clean Arrow 24 patch application and idempotent download flow
- [x] Thirdparty scripts, shell syntax, workflow YAML, and diff checks
- Behavior changed:
- [x] Yes. Paimon logical splits use JNI; deprecated wire values remain
for compatibility.
- Does this need documentation:
    - [x] No
hello-stephen pushed a commit that referenced this pull request Sep 3, 2026
@suxiaogang223
[chore](paimon) remove paimon-cpp dependency (#67385)
7829cb7 
### What problem does this PR solve?

Issue Number: N/A

Related PR: #67378, #67432

Remove paimon-cpp completely from the thirdparty manifest, BE reader,
build integration, tests, and Arrow patch set. Also fold the Arrow
cleanup from #67432 into the standard thirdparty workflow, removing the
temporary fingerprint, recovery, and lifecycle scripts.

Deprecated Thrift enum/field values and `IGNORE_PAIMON_CPP` remain only
for compatibility. This master change is the source used to publish the
shared master/branch-4.1 thirdparty package.

### Release note

Remove the deprecated paimon-cpp reader and build dependency. Paimon
reads continue through JNI and native readers.

### Check List (For Author)

- Test:
    - [x] Paimon FE connector: 553 passed, 1 live test skipped
    - [x] Clean Arrow 24 patch application and package build
- [x] Thirdparty scripts, build hygiene, shell syntax, workflow YAML,
and diff checks
- Behavior changed:
- [x] Yes. paimon-cpp is removed; deprecated wire values remain for
compatibility.
- Does this need documentation:
    - [x] No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei will be requested when the pull request is marked ready for review yiguolei is a code owner
@morningman morningman Awaiting requested review from morningman morningman will be requested when the pull request is marked ready for review morningman is a code owner
@hello-stephen hello-stephen Awaiting requested review from hello-stephen hello-stephen will be requested when the pull request is marked ready for review hello-stephen is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hubgeter @hello-stephen