Skip to content

branch-4.1: [fix](paimon) Keep write cache coherent and extend coverage - #67023

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
suxiaogang223:codex/paimon-write-cases
Aug 24, 2026
Merged

branch-4.1: [fix](paimon) Keep write cache coherent and extend coverage#67023
yiguolei merged 2 commits into
apache:branch-4.1from
suxiaogang223:codex/paimon-write-cases

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Aug 21, 2026

Copy link
Copy Markdown
Member

What

  • move Paimon write suites and golden outputs under external_table_p0/paimon/write
  • extend write coverage for sequence groups, MERGE semantics, sequence/rowkind, deletion vectors, row tracking/data evolution, external paths, partition deletes, bucket modes, and JDBC catalog concurrency
  • keep the Doris Paimon metacache as the owner of table instances used by writes
  • invalidate the same table in Paimon's CachingCatalog before reloading it after a Doris cache miss
  • add FE unit coverage for cache invalidation and table reload behavior
  • validate stable Doris query results through checked-in .out files
  • keep unsupported behavior disabled with targeted TODO markers

Why

Doris currently has its own Paimon table metacache on top of Paimon's CachingCatalog. Invalidating only the Doris cache could still make the loader receive a stale Table from the underlying Paimon cache. As a result, writes after external metadata changes could continue using outdated bucket, path, or table options.

This change keeps the existing Doris cache boundary, documents why both cache layers exist, and makes a Doris cache reload invalidate the corresponding Paimon table entry first.

Testing

  • source ./custom_env.sh && ./build.sh --fe
  • source ./custom_env.sh && ./run-fe-ut.sh --run org.apache.doris.datasource.paimon.PaimonExternalMetaCacheTest,org.apache.doris.datasource.paimon.PaimonExternalTableTest
    • 31 tests, 0 failures, 0 errors
  • normal-mode regression execution for the added Paimon write suites and JDBC catalog concurrency coverage
  • focused .out verification after the cache fix:
    • test_paimon_write_bucket_modes
    • test_paimon_write_sequence_group
    • test_paimon_write_deletion_vector
    • test_paimon_write_external_paths
  • git diff --check

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@suxiaogang223
suxiaogang223 marked this pull request as ready for review August 21, 2026 08:12
@suxiaogang223 suxiaogang223 changed the title branch-4.1: [test](paimon) Add extended write regression coverage branch-4.1: [fix](paimon) Keep write cache coherent and extend coverage Aug 21, 2026
@suxiaogang223
suxiaogang223 force-pushed the codex/paimon-write-cases branch from 82841f2 to 845476a Compare August 21, 2026 11:24
@suxiaogang223
suxiaogang223 force-pushed the codex/paimon-write-cases branch from 845476a to 2268f86 Compare August 21, 2026 11:25
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@yiguolei
yiguolei merged commit a614930 into apache:branch-4.1 Aug 24, 2026
28 of 32 checks passed
@suxiaogang223
suxiaogang223 deleted the codex/paimon-write-cases branch August 24, 2026 01:59
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 3, 2026
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#67023

Problem Summary: Doris and Paimon maintain separate table caches. After a Doris metadata-cache miss, Paimon CachingCatalog could return the same stale table handle, leaving writes planned with outdated bucket, path, or table options. Invalidate the inner Paimon entry before every outer-cache reload and verify the invalidate-before-get ordering.

### Release note

Ensure refreshed Paimon writes use current table metadata.

### Check List (For Author)

- Test: Unit Test
    - Added cache reload ordering coverage; FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon table reloads no longer reuse stale inner-cache entries.
- Does this need documentation: No
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 3, 2026
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#65868, apache#66325, apache#67023, apache#67085

Problem Summary: Forward-port the final branch-4.1 Paimon write regression matrix to master, including append and primary-key modes, bucket routing, transactions, schema evolution, row-level DML, Variant V2, snapshot references, writer lifecycle, cache coherence, and stabilized external paths. Generated golden outputs are preserved byte-for-byte from branch-4.1.

### Release note

None

### Check List (For Author)

- Test: Regression test
    - Test cases and generated expected outputs were forward-ported; execution is deferred until the final integrated validation.
- Behavior changed: No
- Does this need documentation: No
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 3, 2026
### What problem does this PR solve?

Issue Number: close apache#65086

Related PR: apache#67023

Problem Summary: The forward-ported Paimon table cache coherence test uses List to record catalog calls but did not import java.util.List, causing FE test compilation to fail. Add the missing import.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Maven checkstyle validation for the Paimon connector
- Behavior changed: No
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants