[fix](table stream) remove base table cache & use dynamic schema - #67173
Merged
morrySnow merged 2 commits intoAug 28, 2026
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
|
/review |
morrySnow
previously approved these changes
Aug 26, 2026
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Requesting changes for two correctness issues.
Goal and scope
The production change is focused, and the added regression covers live-FE schema reflection across ADD/DROP. However, removing the serialized stream schema breaks mixed-version metadata replay, so the PR does not yet meet its compatibility goal safely.
Critical checkpoints
- Correctness and compatibility: a stream newly persisted by this code has no
fullSchemafield for an older FE reader, which can expose a zero-column stream during rolling upgrade or failover. This is blocking. - Persistence and lifecycle: create-table journal/image serialization and old-reader reconstruction were traced; no compatibility fallback or format gate is present.
- Concurrency: statement collection/binding, stream/base locks, DROP/rename, and allowed row-binlog schema-change paths were checked. No additional PR-introduced lock-order, race, or identity defect was substantiated.
- Parallel paths: local and Cloud stream construction and metadata access paths were checked; no distinct additional defect was found.
- Tests and expected results: the golden output is ordered and consistent with the intended live schema behavior, but the ALTER wait helper does not actually assert completion. Mixed-version journal/image coverage is also missing. I did not run builds or tests in this review-only environment.
- Configuration, FE/BE protocol variables, transactions, and data writes: this PR adds none and introduces no separate checkpoint issue.
- Performance and observability: dynamic schema allocation/catalog lookup was reviewed; planning snapshots the wrapper and no actionable hot-path or observability issue was established.
- User focus: no additional focus was provided; the whole PR was reviewed.
- CI snapshot: BE UT, Cloud UT, and coverage are successful; compile, FE UT, performance, and code-review checks are still pending at submission time.
| public BaseTableStream(long id, String streamName, List<Column> fullSchema, TableIf baseTable) { | ||
| super(id, streamName, TableType.STREAM, fullSchema); | ||
| public BaseTableStream(long id, String streamName, TableIf baseTable) { | ||
| super(id, streamName, TableType.STREAM, null); |
Contributor
There was a problem hiding this comment.
[P1] Preserve a serialized schema for older FE readers
Passing null here leaves fullSchema absent because Doris's Gson configuration does not serialize null fields. A stream created by this FE is therefore written without the only schema representation understood by the pre-change reader: when an older FE replays the CreateTableInfo journal entry or loads an image during rolling upgrade/failover, its no-arg construction leaves an empty schema and its inherited accessors expose zero columns. Please retain a serialized compatibility snapshot (new code can still ignore it at runtime), or explicitly version/gate this metadata format and cover the new-writer/old-reader path.
Contributor
There was a problem hiding this comment.
Confirmed this is still unresolved in the merged head 6268e7a5548. BaseTableStream still passes null as fullSchema, and the Doris Gson builder still omits null fields. A stream created by this version is therefore persisted without the schema snapshot required by pre-change FE readers; replaying the journal or loading the image on an older FE leaves an empty schema. Please track a follow-up that preserves a compatibility snapshot or explicitly gates the metadata format, with new-writer/old-reader coverage for both journal replay and image loading.
Contributor
TPC-H: Total hot run time: 16817 ms |
Contributor
TPC-DS: Total hot run time: 82299 ms |
Contributor
ClickBench: Total hot run time: 14.53 s |
Contributor
Author
|
run cloud_p0 |
2 similar comments
Contributor
Author
|
run cloud_p0 |
Contributor
Author
|
run cloud_p0 |
TsukiokaKogane
force-pushed
the
remove_stream_base_table_cache_and_refactor_dynamic_schema
branch
from
August 27, 2026 06:29
13bb414 to
fa9b9fd
Compare
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 16692 ms |
Contributor
TPC-DS: Total hot run time: 81403 ms |
Contributor
ClickBench: Total hot run time: 14.44 s |
fix format remove redundant check remove redundant check
TsukiokaKogane
force-pushed
the
remove_stream_base_table_cache_and_refactor_dynamic_schema
branch
from
August 27, 2026 10:05
86f6ea4 to
6268e7a
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 16964 ms |
Contributor
TPC-DS: Total hot run time: 81981 ms |
Contributor
ClickBench: Total hot run time: 14.45 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
morrySnow
approved these changes
Aug 28, 2026
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Aug 28, 2026
### What problem does this PR solve? Issue Number: close apache#67109 Related PR: apache#67173 Problem Summary: Sync the branch with the latest master and resolve the overlapping table stream dynamic-schema implementation. Keep the upstream base-table ID lookup and stream LSN support while retaining schema generation under the base-table read lock, explicit failure when the base table is unavailable, and an empty persisted stream schema. ### Release note None ### Check List (For Author) - Test: Unit Test - CreateTableStreamTest - ExplainTableStreamPlanTest - TableStreamBaseTableInfoTest - Behavior changed: No (merge conflict resolution preserves the intended PR behavior) - Does this need documentation: No
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
69 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #67095 #67109 #67101
Related PR: #65418
Problem Summary:
Table Streams currently retain both a cached
TableIfreference and a copied schema from the base table. These snapshots can become stale in two scenarios. First, if the original base table is dropped and another table is created with the same name, relation collection may lock the replacement table by name while stream binding continues to use the cached object of the dropped table, causing the planner lock target and scan target to refer to different table identities. Second, when the base table schema changes through supported ADD/DROP COLUMN operations, especially light schema changes, the stream continues exposing its creation-time schema through DESCRIBE, SHOW COLUMNS, information_schema, and query column resolution. The stale schema can also remain after an FE metadata reload.This change removes the cached base-table object and resolves internal base tables from the persisted catalog, database, and table IDs on each access. It also removes the copied stream schema and derives the current stream schema dynamically from the visible columns of the base table, followed by the hidden stream sequence and change-type columns. As a result, a dropped and recreated table with the same name is not treated as the original base table, while supported base-table schema changes are reflected by stream metadata and query planning. The added regression case covers schema visibility before and after ADD/DROP COLUMN and verifies stream query output after ADD COLUMN.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)