[feat](row binlog)：support cloud binlog and independent binlog tablet - #65810
Merged
Conversation
Userwhite
requested review from
924060929,
Gabriel39,
airborne12,
csun5285,
deardeng,
eldenmoon,
englefly,
gavinchou,
liaoxin01,
luwei16,
morningman,
morrySnow,
mymeiyi,
starocean999 and
yiguolei
as code owners
July 20, 2026 07:54
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Userwhite
changed the title
Userwhite
force-pushed
the
support_cloud_binlog
branch
from
July 20, 2026 11:52
185bbfe to
a42d600
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29573 ms |
Contributor
TPC-DS: Total hot run time: 176800 ms |
Contributor
ClickBench: Total hot run time: 25.03 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29328 ms |
Contributor
TPC-DS: Total hot run time: 178121 ms |
Contributor
ClickBench: Total hot run time: 25.01 s |
16 tasks
gavinchou
reviewed
Jul 22, 2026
gavinchou
reviewed
Jul 22, 2026
morningman
approved these changes
Aug 7, 2026
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Aug 7, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810, apache#65859 Problem Summary: Merge the Cloud Row Binlog data-plane implementation and reconcile commit TSO metadata with Cloud Table Stream offset updates. The integration initializes empty-partition TSO metadata, waits for lazy publish through the actual clone-chain owner, rereads a stable Stream snapshot after publish, and keeps target transaction commit metadata distinct from source Stream offsets. ### Release note Integrate Cloud Row Binlog data-plane metadata with the Cloud Table Stream control plane. ### Check List (For Author) - Test: - Unit Test: Cloud Table Stream 28/28, CloneChainReader 25/25, empty partition TSO initialization 1/1, FE Cloud Stream 26/26 - Behavior changed: Yes. Cloud Table Stream consumes stable Row Binlog commit TSO metadata and atomically advances source offsets with target transaction commit. - Does this need documentation: Yes. The local implementation design was updated.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Aug 7, 2026
### What problem does this PR solve? Issue Number: close apache#65418 Related PR: apache#65810 Problem Summary: Table Stream keeps per-partition consumption offsets in FE for storage-compute integrated deployments. In Cloud mode, target-table transactions are committed by MetaService, so FE-owned offsets cannot be advanced atomically with target data. This change adds the Cloud Table Stream control plane, stores Latest and Versioned offsets in MetaService, reuses the Index and recycler lifecycle, resolves statement-level read snapshots through clone chains, and commits expected-offset CAS updates with target data in the same immediate FDB transaction. It also integrates the Row Binlog commit TSO contract, initializes empty-partition TSO metadata, and waits for pending lazy publish using the actual clone-chain owner before rereading the complete snapshot. ### Release note Add the storage-compute separated Table Stream control plane. ### Check List (For Author) - Test: - Unit Test: Cloud Table Stream 28/28, CloneChainReader 25/25, empty partition TSO initialization 1/1, FE Cloud Stream 26/26 - Behavior changed: Yes. Cloud Table Stream uses MetaService-managed offsets and immediate atomic consumption commits while preserving non-Cloud behavior. - Does this need documentation: Yes. The implementation and test designs have been updated.
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Aug 7, 2026
### What problem does this PR solve? Issue Number: close apache#65951 Related PR: apache#65810 Problem Summary: Row Binlog AFTER value columns are nullable even when their source columns are NOT NULL. Complex source columns then provide no top-level null map, so nullable ARRAY, MAP, and STRUCT writers can leave the physical null child empty and abort during segment finalization. Wrap only Row Binlog normal non-key AFTER inputs before OlapBlockDataConvertor, including missing columns filled from historical rows, while preserving the original shared input and full block. Restore the generic ColumnWriter behavior so other write paths are unaffected. ### Release note Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter=SegmentFlusherTransformFormatTest.RowBinlogWritesNotNullComplexColumnsToNullableAfterColumns -j 16` - `./build.sh --be -j96` - `build-support/check-format.sh` - `build-support/run-clang-tidy.sh --base HEAD^ --build-dir be/ut_build_ASAN` - Behavior changed: Yes (limited to Row Binlog AFTER conversion) - Does this need documentation: No
gavinchou
pushed a commit
that referenced
this pull request
Aug 10, 2026
…sion (#66553) Issue Number: close #65951 Related PR: #65810 Problem Summary: Row Binlog makes normal non-key AFTER columns nullable even when their source columns are NOT NULL. `OlapBlockDataConvertor` derives the null map from the runtime source column, so a non-nullable complex source provides no top-level null map. Nullable ARRAY, MAP, and STRUCT writers can then leave their physical null child empty and abort during segment finalization. This PR keeps the fix local to Row Binlog: - Wrap normal non-key AFTER source columns as `ColumnNullable<original>` with an all-zero null map before conversion. - Keep key columns unchanged, preserve original source columns in `full_block`, and do not mutate the shared input Block. - For partial-update missing AFTER columns, wrap each column filled from the historical Rowset before conversion and append. - Restore the original generic `ColumnWriter::append()` behavior and remove the implicit null-map state, avoiding behavior changes in Scalar, VARIANT, and other common write paths. The test covers full updates, explicit partial updates, and partial updates whose NOT NULL MAP/STRUCT values are filled from a historical Rowset. It verifies logical readback and the physical top-level null-child row count. ### Release note Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns.
luwei16
pushed a commit
that referenced
this pull request
Aug 10, 2026
…5859) ### What problem does this PR solve? Issue Number: #65418 Related PR: #65810 Problem Summary: Table Stream already provides incremental consumption in storage-compute integrated deployments, where FE owns each partition offset. In Cloud mode, the target-table commit is finalized by MetaService, so keeping offsets in FE cannot atomically commit target data and advance Stream offsets. This PR adds the Cloud Table Stream control plane: - treats FE Catalog as the authority for Stream existence and binding, and stores only Latest and Versioned per-partition offsets in MetaService; - reuses the existing Index prepare, commit, Drop Index Log, Recycle Index, and InstanceRecycler lifecycle without creating Stream Mapping keys or physical tablets; - resolves one statement-level read snapshot, including clone-chain Offset inheritance; - performs expected-offset CAS and target-table commit in the same immediate FDB transaction, with no lazy-commit fallback; - keeps Create retries idempotent without overwriting offsets that consumption has already advanced; - carries affected Stream identities through DROP PARTITION so the partition recycler removes the corresponding Latest and Versioned offsets; - preserves the existing non-Cloud Table Stream path. This draft depends on #65810 for the Cloud Row Binlog data plane. Before Cloud E2E is enabled, the integration must confirm that every visible partition, including an empty partition and a newly added partition, has an authoritative `VersionPB.visible_tso` and empty-boundary TSO. The control plane intentionally fails when that metadata is missing instead of treating metadata loss as an empty partition. Rolling upgrade rule: upgrade all MetaService nodes before enabling the FE Cloud Table Stream path. Once enabled, MetaService rollback requires stopping Stream consumption first because an older MetaService does not understand Stream offset updates. ### Release note Add the storage-compute separated Table Stream control plane. End-to-end availability depends on the Cloud Row Binlog data plane in #65810. ### Check List (For Author) - Test - [x] Regression test - `table_stream_p0` and `row_binlog_p0`: 15 suites passed on the branch baseline. - [x] Unit Test - Cloud MetaService Create, Query, Update, key, clone-chain, recycler, checker, and operation-log tests passed. - FE Cloud catalog and Table Stream manager tests: 8 tests passed after the Offset-only metadata change. - The earlier full FE Table Stream, transaction, drop, planner, and MV rewrite baseline passed. - [x] Manual test - `./build.sh --cloud` passed. - `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe` passed with Checkstyle. - Cloud C++ formatting passed, and changed-line clang-tidy passed for all modified production C++ files. The existing Cloud HTTP test target still cannot be analyzed because its compile command exposes pre-existing private-API diagnostics. - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Cloud Table Stream uses MetaService-managed offsets, immediate atomic consumption commits, and FORCE-only drop semantics. Non-Cloud behavior is unchanged. - Does this need documentation? - [ ] No. - [x] Yes. The implementation and test designs have been reviewed; public user documentation should be published when #65810 completes the end-to-end feature.
AntiTopQuark
pushed a commit
to AntiTopQuark/doris
that referenced
this pull request
Aug 11, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Generic balance and storage-medium migration could move a row-binlog companion or its base tablet independently, while wrong-path repair deleted before rebuilding and could not make progress when the misplaced replica was the only latest copy. Redundant-pair deletion, invalid directional metadata, and colocate-group health also lacked complete safety boundaries. This change uses apache#65810 directional links as the only pair model, repairs wrong paths in place on the exact base path, blocks independent pair movement, validates pair metadata before destructive actions, and keeps companion repair state out of queryable colocate-group stability. ### Release note Keep row-binlog companion tablets on the same backend and storage path as their base tablets during FE repair and automatic migration. ### Check List (For Author) - Test: Not run per request; only static source, conflict-marker, legacy-symbol, and git diff checks were completed - Behavior changed: Yes. FE repair and automatic migration now preserve base/row-binlog tablet locality and reject unsafe storage relocation changes - Does this need documentation: No
AntiTopQuark
pushed a commit
to AntiTopQuark/doris
that referenced
this pull request
Aug 11, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: ModifyPartitionOpTest imported JUnit 4 APIs, which are rejected by the FE ImportControl rules. Use the JUnit 5 Assertions and Test APIs already used by neighboring Nereids command tests. ### Release note None ### Check List (For Author) - Test: Not run per request; static import and diff checks only - Behavior changed: No - Does this need documentation: No
AntiTopQuark
pushed a commit
to AntiTopQuark/doris
that referenced
this pull request
Aug 12, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: RowBinlogTabletSchedulerTest passed a cast null Tag through Deencapsulation. The reflection helper rejects untyped null arguments before invoking the scheduler, so the test observed IllegalArgumentException instead of the expected SchedException. Use the helper's Class token convention to resolve the Tag parameter while still passing null to the colocate scheduling path. ### Release note None ### Check List (For Author) - Test: Not run per request; only static source review and git diff checks were completed - Behavior changed: No. This only fixes a test reflection argument - Does this need documentation: No
Contributor
Author
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…apache#65810) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: This PR implements Row Binlog based on an independent row-binlog tablet model for both non-cloud mode and cloud mode. Before this change, Row Binlog data was tightly coupled with the base tablet write/compaction path, and cloud mode did not have a complete end- to-end implementation. This made it difficult to keep Row Binlog behavior consistent across storage-compute integrated and storage-compute separated deployments, especially for MOW tables, historical values, partial update/upsert publish conflicts, and compaction. The new implementation introduces row-binlog companion tablets and wires them through FE metadata, BE write/publish/compaction/query paths, and Cloud Meta Service. In non-cloud mode, Row Binlog write and compaction are refactored around the independent tablet model. In cloud mode, the same model is extended with cloud-specific rowset commit, publish, delete-bitmap, metadata, and compaction handling. Main changes: 1. Independent Row Binlog tablet model - Store Row Binlog data in companion row-binlog tablets instead of coupling it directly with base tablet rowsets. - Track the relationship between base tablets and row-binlog tablets through aligned tablet ids. - Represent Row Binlog indexes with dedicated materialized-index states. - Keep Row Binlog schema aligned with master: `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`. 2. Non-cloud write, publish, and compaction path - Support writing Row Binlog data through independent tablets in non- cloud mode. - Propagate Row Binlog tablet information through FE sink planning and BE load/write context. - Refactor Row Binlog compaction to reuse cumulative-compaction style logic. - Add `CumulativeCompactionBinlogPolicy` for non-cloud Row Binlog compaction. - Preserve MOW historical-value behavior, partial update/upsert semantics, and delete-bitmap correctness. 3. Cloud write, publish, and metadata path - Add cloud Row Binlog companion tablet creation and metadata flow. - Add paired base-rowset + row-binlog-rowset commit through Meta Service. - Support Cloud publish conflict handling for partial update and upsert. - Make Cloud delete-bitmap cache, rowset builder/writer, tablet manager, and transaction paths aware of Row Binlog rowsets. - Persist Row Binlog tablet statistics and compaction metadata in Cloud Meta Service. 4. Cloud Row Binlog compaction - Add `CloudBinlogCumulativeCompactionPolicy`. - Keep binlog-specific candidate selection separate from normal compaction paths. - Add `filter_new_visible_rowsets` to avoid compacting newly visible rowsets too early unless backlog requires it. - Fix cloud candidate selection so cloud rowsets are not incorrectly filtered as non-local rowsets. 5. Query, schema change, restore, and observability - Update Row Binlog scan/read paths to use TSO/LSN correctly. - Support Row Binlog behavior across schema change, restore, tablet report, SHOW/proc paths, and compaction actions. - Expose Row Binlog tablet/index information where needed for debugging and operation. 6. Tests - Add/update BE unit tests for non-cloud and cloud Row Binlog write, rowset builder/writer, transaction, and compaction policy behavior. - Add/update regression tests for Row Binlog basic behavior, compaction, schema change, multi-segment reads, Cloud compaction, and Cloud publish conflict. - Update regression ordering to use `__DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__` where result order depends on commit/visible TSO semantics.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…sion (apache#66553) Issue Number: close apache#65951 Related PR: apache#65810 Problem Summary: Row Binlog makes normal non-key AFTER columns nullable even when their source columns are NOT NULL. `OlapBlockDataConvertor` derives the null map from the runtime source column, so a non-nullable complex source provides no top-level null map. Nullable ARRAY, MAP, and STRUCT writers can then leave their physical null child empty and abort during segment finalization. This PR keeps the fix local to Row Binlog: - Wrap normal non-key AFTER source columns as `ColumnNullable<original>` with an all-zero null map before conversion. - Keep key columns unchanged, preserve original source columns in `full_block`, and do not mutate the shared input Block. - For partial-update missing AFTER columns, wrap each column filled from the historical Rowset before conversion and append. - Restore the original generic `ColumnWriter::append()` behavior and remove the implicit null-map state, avoiding behavior changes in Scalar, VARIANT, and other common write paths. The test covers full updates, explicit partial updates, and partial updates whose NOT NULL MAP/STRUCT values are filled from a historical Rowset. It verifies logical readback and the physical top-level null-child row count. ### Release note Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…ache#65859) Issue Number: apache#65418 Related PR: apache#65810 Problem Summary: Table Stream already provides incremental consumption in storage-compute integrated deployments, where FE owns each partition offset. In Cloud mode, the target-table commit is finalized by MetaService, so keeping offsets in FE cannot atomically commit target data and advance Stream offsets. This PR adds the Cloud Table Stream control plane: - treats FE Catalog as the authority for Stream existence and binding, and stores only Latest and Versioned per-partition offsets in MetaService; - reuses the existing Index prepare, commit, Drop Index Log, Recycle Index, and InstanceRecycler lifecycle without creating Stream Mapping keys or physical tablets; - resolves one statement-level read snapshot, including clone-chain Offset inheritance; - performs expected-offset CAS and target-table commit in the same immediate FDB transaction, with no lazy-commit fallback; - keeps Create retries idempotent without overwriting offsets that consumption has already advanced; - carries affected Stream identities through DROP PARTITION so the partition recycler removes the corresponding Latest and Versioned offsets; - preserves the existing non-Cloud Table Stream path. This draft depends on apache#65810 for the Cloud Row Binlog data plane. Before Cloud E2E is enabled, the integration must confirm that every visible partition, including an empty partition and a newly added partition, has an authoritative `VersionPB.visible_tso` and empty-boundary TSO. The control plane intentionally fails when that metadata is missing instead of treating metadata loss as an empty partition. Rolling upgrade rule: upgrade all MetaService nodes before enabling the FE Cloud Table Stream path. Once enabled, MetaService rollback requires stopping Stream consumption first because an older MetaService does not understand Stream offset updates. Add the storage-compute separated Table Stream control plane. End-to-end availability depends on the Cloud Row Binlog data plane in apache#65810. - Test - [x] Regression test - `table_stream_p0` and `row_binlog_p0`: 15 suites passed on the branch baseline. - [x] Unit Test - Cloud MetaService Create, Query, Update, key, clone-chain, recycler, checker, and operation-log tests passed. - FE Cloud catalog and Table Stream manager tests: 8 tests passed after the Offset-only metadata change. - The earlier full FE Table Stream, transaction, drop, planner, and MV rewrite baseline passed. - [x] Manual test - `./build.sh --cloud` passed. - `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe` passed with Checkstyle. - Cloud C++ formatting passed, and changed-line clang-tidy passed for all modified production C++ files. The existing Cloud HTTP test target still cannot be analyzed because its compile command exposes pre-existing private-API diagnostics. - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Cloud Table Stream uses MetaService-managed offsets, immediate atomic consumption commits, and FORCE-only drop semantics. Non-Cloud behavior is unchanged. - Does this need documentation? - [ ] No. - [x] Yes. The implementation and test designs have been reviewed; public user documentation should be published when apache#65810 completes the end-to-end feature.
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Aug 18, 2026
…RowBinlogSegmentWriter Last PR of the block-transform-chain stack. RowBinlogDeriveStage rebuilds the load block into a full-width block over the binlog schema -- key + AFTER values, optional __BEFORE__* values, and the TSO/LSN/op columns -- so the ordinary segment writers write it like any other block. build_transform_chain picks PlainRowBinlogDerive (no history probe) or MowRowBinlogDerive (partial update and/or BEFORE image) per flush, and keeps an empty chain for every non-direct write type, whose rows are already binlog shaped. RowBinlogSegmentWriter is deleted, and with it the dead VerticalRowBinlogRowsetWriter template, the retriever's clear() bridge and SegmentWriter::build_key_index's protected visibility. SegmentFlusher no longer special-cases binlog when it picks a writer. Three later fixes to the deleted writer are carried into the stage, since deleting the file would drop them silently: - apache#66046: the op revision from the old delete sign runs between the BEFORE fill and the op column fill, the order the writer used. - apache#65810: the cloud-mode rejection is gone, and the historical lookup uses source.base_tablet. The writer fell back to its own tablet when that was unset; the stage requires it instead, because on a binlog write the writer context's tablet is the binlog tablet, which holds no source rows. Every site that fills source.tablet_schema fills source.base_tablet too. - apache#66553: AFTER value columns are wrapped nullable before they land in the binlog block, which declares them nullable. Behavior change to ack: binlog segments are written by the ordinary segment writer, which is the vertical one under the default enable_vertical_segment_writer=true; the deleted writer was always horizontal. The derived rows are identical -- the plain, MoW and BEFORE binlog goldens recorded through the vertical writer are byte-identical to the baselines recorded from the legacy writer -- but the writer type and the physical page layout are visible to operational tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 tasks
gavinchou
pushed a commit
that referenced
this pull request
Aug 24, 2026
…66743) Related PR: #65810, #65859 Problem Summary: Cloud Row Binlog and Cloud Table Stream now have an integrated data and control plane, but the Cloud SQL path and several cross-module correctness boundaries did not have complete automated coverage. This PR adds layered coverage for the four Table Stream flows: - Create: batched Offset initialization, retries, partial-state cleanup, empty and newly added partitions. - Query: statement-level snapshots, initial/snapshot/reset reads, clone-chain inheritance, partial responses, and read-only behavior. - Update: Offset CAS, target-table atomicity, idempotent retry, multiple Streams, immediate commit, size limits, and source publish concurrency. - Delete: typed drop requests, partition identity propagation, operation-log recovery, clone-safe GC, paginated cleanup, and instance deletion. It also adds Cloud SQL suites for initial rows, repeated consumption, explicit partitions, multiple references, partition lifecycle, schema changes, metadata, and object lifecycle. A Docker specialty test verifies real FE Catalog and MetaChecker orphan-Offset detection. The expanded tests found and fix two operational validation issues: malformed versioned set_value input now returns INVALID_ARGUMENT instead of dereferencing a null protobuf, and check_instance now validates the typed RecycleIndex binding/state before suppressing a transient projection gap.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Aug 25, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Cloud Row Binlog currently does not refresh the implicit end TSO for direct table@incr scans. Keep the flexible partial-update Cloud coverage on base-table and raw Row Binlog results while skipping only the three direct @incr assertions until that generic FE issue is fixed. ### Release note None ### Check List (For Author) - Test: No local test run; this is a Cloud regression routing adjustment and buildall will validate it. - Behavior changed: No. Production code is unchanged. - Does this need documentation: No
hutiefang76
added a commit
to hutiefang76/doris
that referenced
this pull request
Aug 30, 2026
Issue Number: close apache#65383 Problem Summary: ALTER TABLE changes to binlog.format and binlog.need_historical_value must be routed through the binlog configuration path so immutable-setting validation reports the specific error. The production routing is now present on master via apache#65810, but that broader change did not add focused regression coverage for these properties. Add unit coverage for property classification and end-to-end ALTER TABLE error reporting for both immutable binlog settings. Test: JAVA_HOME=/Users/hutiefang/Library/Java/JavaVirtualMachines/ms-17.0.18/Contents/Home DORIS_THIRDPARTY=/Users/hutiefang/project/oss/doris/thirdparty bash run-fe-ut.sh --run org.apache.doris.alter.AlterOperationsTest,org.apache.doris.alter.SchemaChangeHandlerTest#testChangeBinlogFormatReportsBinlogError+testChangeBinlogNeedHistoricalValueReportsBinlogError Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
8 tasks
69 tasks
Yukang-Lian
added a commit
that referenced
this pull request
Sep 1, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: #65810 Problem Summary: Row Binlog supports fixed-column partial updates, but did not support flexible partial updates (`UPDATE_FLEXIBLE_COLUMNS`). Flexible updates carry a per-row skip bitmap and may merge rows before writing, so Row Binlog must reconstruct complete rows while keeping BEFORE/AFTER values, operations, row counts, and LSNs aligned. This PR: - Routes flexible partial updates through the existing `MowRowBinlogDeriveStage` transform chain. - Reuses the existing flexible-update aggregation, a read-only MOW key probe, and historical-row read plans to materialize complete AFTER rows and optional complete BEFORE rows. - Keeps per-row LSN and same-batch DELETE-to-INSERT state aligned through aggregation. For DELETE-to-INSERT, omitted AFTER cells use their default or null value, while a live historical row remains the UPDATE BEFORE image. - Preserves Base Rowset and Row Binlog `(segment_id, row_id)` alignment for sequence losers; the synchronized delete bitmap makes those rows logically invisible. - Uses a writer-private copy-on-write Block, so mutated columns detach without changing the Base Rowset writer's input. - Keeps the existing Row Binlog on-disk format, query protocol, and APPEND / UPDATE / DELETE semantics. The implementation is shared by local and Cloud storage. It does not change FE syntax, Table Stream metadata, Stream Offset handling, or protobuf definitions. Scope note: this PR does not lift the existing flexible partial-update multi-segment load restriction and does not add a zero-row Segment format.
This was referenced Sep 3, 2026
morningman
added a commit
to morningman/doris
that referenced
this pull request
Sep 3, 2026
### What problem does this PR solve? Issue Number: close apache#65388, related apache#65418 Related PR: apache#65810 Problem Summary: Review follow-up for this PR. Two problems were found in the first revision. 1. `visitAlterStreamComment` decoded the comment with `stripQuotes`, which only removes the outer delimiters. `ALTER STREAM s SET COMMENT 'a''b'` therefore stored `a''b` instead of `a'b`, and with the default sql mode `COMMENT '\n'` stored a backslash plus `n` instead of a newline. The value is exposed by `information_schema.table_streams.STREAM_COMMENT` and printed by `SHOW CREATE STREAM`, so the stored metadata was simply wrong. `visitCreateStream` had the other half of the problem: it decoded backslash escapes (`LogicalPlanBuilderAssistant.escapeBackSlash`) but did not collapse doubled quotes and ignored `NO_BACKSLASH_ESCAPES`, so the same literal produced different comments depending on whether it went through CREATE or ALTER. Both now use `SqlLiteralUtils.parseStringLiteral`, which collapses doubled quotes and honours the sql mode exactly like the lexer does (`NereidsParser` feeds `SqlModeHelper.hasNoBackSlashEscapes()` to the lexer). Not addressed here: `Env.addTableComment` wraps the comment in single quotes but only escapes double quotes (`SqlUtils.escapeQuota`), so a comment holding a single quote makes `SHOW CREATE` emit non-parsable DDL. That is pre-existing, shared by all 19 call sites of every table type, and is not stream specific. 2. The two new regression suites did not follow the repository test standards: names were indirected through `def`, deterministic results were asserted by hand instead of a generated `.out`, and both dropped their fixtures at the end. Fixed all three. While converting, the skip guard of `test_binlog_property_alter_exception` turned out to be dead: it compared the config name against `enable_feature_binlog`, but `SHOW FRONTEND CONFIG` reports the EXPERIMENTAL config as `experimental_enable_feature_binlog`, so the suite skipped itself unconditionally, in CI as well. It now uses the framework helper `getSyncer().checkEnableFeatureBinlog()`, and the suite actually runs. ### Release note None (the behaviour change of this PR is described in the first commit). ### Check List (For Author) - Test: - Unit Test: `AlterTableStreamCommentTest#testAlterStreamCommentStringLiteral` covers doubled single quotes, doubled double quotes, `\n`/`\t` and `NO_BACKSLASH_ESCAPES`. Both tests of the class pass locally (`Tests run: 2, Failures: 0, Errors: 0`). - Regression test: `test_table_stream_alter_comment` (the `.out` was generated with `-genOut` against a local cluster, then verified by a second run without it) and `test_binlog_property_alter_exception`, both pass locally and the six expected error messages were observed in the run log. - Behavior changed: Yes. A stream comment containing a doubled quote or a backslash escape is now stored decoded, and the sql mode is honoured. - Does this need documentation: No (the statement itself is documented by the first commit's doc PR). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xx7TgjXJCiChnzLYa6hgtL
morningman
added a commit
that referenced
this pull request
Sep 4, 2026
### What problem does this PR solve? Issue Number: close #65388, related #65418 Related PR: #65810 Problem Summary: **1. `ALTER STREAM ... SET COMMENT` was not supported (#65388)** A table stream can be created with a comment and the comment is fully wired up everywhere except for changing it: | step | before this PR | | --- | --- | | `CREATE STREAM s ON TABLE t COMMENT 'x'` | supported (`InternalCatalog#createTableStream`) | | persisted in the image | supported (`Table#comment`) | | `SHOW CREATE STREAM s` | prints the comment | | `information_schema.table_streams.STREAM_COMMENT` | exposes the comment | | changing the comment | **not possible** | There was no `ALTER STREAM` rule in `DorisParser.g4` at all — `STREAM` only appeared in `CREATE STREAM`, `DROP STREAM`, `SHOW STREAMS` and `SHOW CREATE STREAM` — so the statement failed at parser stage: ``` errCode = 2, detailMessage = no viable alternative at input 'ALTER STREAM'(line 1, pos 6) ``` `ALTER TABLE` is not an alternative either: `Alter#processAlterTable` rejects the `STREAM` table type with `Do not support alter STREAM table[...]`. This PR adds: ```sql ALTER STREAM <name> SET COMMENT 'new comment'; ALTER STREAM <name> MODIFY COMMENT 'new comment'; -- same thing ``` `MODIFY` is accepted alongside `SET` so the syntax stays consistent with `ALTER TABLE ... MODIFY COMMENT`, which is the existing Doris spelling for the same operation on a table. Implementation notes: - The comment of a stream lives in the `Table` metadata only, so `Alter#processAlterStreamComment` reuses `ModifyCommentOperationLog.forTable(...)` and the existing replay path `Alter#replayModifyComment`, which already resolves a generic `Table`. **No new edit log operation and no meta version bump.** - Cloud Meta Service only stores stream offsets and ids (`CloudInternalCatalog#beforeCreateTableStream` / `#afterCreateTableStream`), so no extra RPC is needed and the behaviour is the same in cloud mode. - `AlterStreamCommand` extends `AlterCommand`, which already provides `ForwardWithSync` and `StmtType.ALTER`. It carries an `AlterType` enum so that other `ALTER STREAM` clauses can be added later without reshaping the command. - Privilege required is `ALTER` on the stream, matching `ALTER TABLE`. Altering a non-stream table through `ALTER STREAM` reports `ERR_WRONG_OBJECT`, the same way `SHOW CREATE STREAM` does. - `Config.enable_table_stream` gates the operation, consistent with `CREATE STREAM` and `DROP STREAM`. - The comment literal is decoded with `SqlLiteralUtils.parseStringLiteral`, so a doubled quote collapses to one quote and backslash escapes follow the session sql mode, matching the lexer (`NereidsParser` drives the lexer with `SqlModeHelper.hasNoBackSlashEscapes()`). `CREATE STREAM ... COMMENT` was decoding the same literal differently -- it unescaped backslashes but never collapsed doubled quotes and ignored `NO_BACKSLASH_ESCAPES` -- so it was moved onto the same decoder, otherwise the comment stored by CREATE and by ALTER would differ for the same text. Not fixed here: `Env#addTableComment` quotes the value with single quotes while escaping only double quotes, so a comment holding a `'` makes `SHOW CREATE` emit non-parsable DDL. That is pre-existing, shared by all 19 call sites of every table type, and will be filed separately. **2. Regression coverage for immutable binlog properties (#65383)** `ALTER TABLE ... SET ("binlog.format" = ...)` on a ROW binlog table used to fail with a misleading light-schema-change error, because `AlterOperations#checkBinlogConfigChange` did not list `binlog.format` / `binlog.need_historical_value` and the statement was dispatched to the generic schema change path. That was fixed as a side effect of #65810 (`f745ddf9e22`), but no test locked the behaviour in. This PR adds `test_binlog_property_alter_exception.groovy` covering: | statement (on a `binlog.format = ROW` MOW table) | expected | | --- | --- | | `SET ("binlog.format" = "STATEMENT_AND_SNAPSHOT")` | `not support change binlog format from ROW to STATEMENT_AND_SNAPSHOT` | | `SET ("binlog.need_historical_value" = "false")` | `not support change binlog.need_historical_value from true to false` | | `SET ("binlog.enable" = "false")` | `can't disable binlog when format is [Row]` | | `SET ("binlog.format" = "ROW")` (same value) | accepted, no-op | | `SET ("binlog.ttl_seconds" = "7200")` | accepted | | `SET ("binlog.format" = "ROW")` on a table without binlog | `not support change binlog format from STATEMENT_AND_SNAPSHOT to ROW` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
This PR implements Row Binlog based on an independent row-binlog tablet
model for both non-cloud mode and cloud mode.
Before this change, Row Binlog data was tightly coupled with the base
tablet write/compaction path, and cloud mode did not have a complete end-
to-end implementation. This made it difficult to keep Row Binlog behavior
consistent across storage-compute integrated and storage-compute
separated deployments, especially for MOW tables, historical values,
partial update/upsert publish conflicts, and compaction.
The new implementation introduces row-binlog companion tablets and wires
them through FE metadata, BE write/publish/compaction/query paths, and
Cloud Meta Service. In non-cloud mode, Row Binlog write and compaction
are refactored around the independent tablet model. In cloud mode, the
same model is extended with cloud-specific rowset commit, publish,
delete-bitmap, metadata, and compaction handling.
Main changes:
coupling it directly with base tablet rowsets.
through aligned tablet ids.
__DORIS_BINLOG_TSO__,__DORIS_BINLOG_LSN__,__DORIS_BINLOG_OP__.cloud mode.
load/write context.
logic.
CumulativeCompactionBinlogPolicyfor non-cloud Row Binlogcompaction.
semantics, and delete-bitmap correctness.
and transaction paths aware of Row Binlog rowsets.
Meta Service.
CloudBinlogCumulativeCompactionPolicy.compaction paths.
filter_new_visible_rowsetsto avoid compacting newly visiblerowsets too early unless backlog requires it.
filtered as non-local rowsets.
report, SHOW/proc paths, and compaction actions.
and operation.
rowset builder/writer, transaction, and compaction policy behavior.
schema change, multi-segment reads, Cloud compaction, and Cloud publish
conflict.
__DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__where result order depends on commit/visible TSOsemantics.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)