[feature](coordinator) Add backend selection extension framework - #65173
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
10b4692 to
01868c3
Compare
|
run buildall |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated review summary:
I found three issues that need changes before this should land. The main problems are that two newly exposed FE configs are not wired into the runtime paths they describe, and the prepared point-query fast path does not preserve the new query affinity semantics.
Critical checkpoint conclusions:
- Goal/test proof: the no-op public framework is mostly scoped, but the extension/config contract is incomplete in query and repair paths. Unit tests cover basic no-op/session validation, but not the disabled config paths or prepared point-query reuse.
- Scope/focus: the change is generally focused on FE resource-group affinity plumbing; no existing inline threads or user focus items needed duplicate suppression.
- Concurrency/lifecycle: no new FE locking issue found in the reviewed heartbeat, ServiceLoader, and scheduler paths.
- Config/compatibility: optional thrift fields are compatible, but
resource_tag_location_checkandenable_repair_src_replica_local_affinitycurrently do not control behavior. - Parallel paths: query scan, prepared point query, stream/mysql/group-commit load, repair scheduling, frontends metadata, audit, and metrics paths were reviewed. The point-query path needs special handling because it reuses a cached scan node and shuffles candidate backends.
- Validation: static review plus
git diff --checkon the correct one-commit PR range. I did not run FE tests/build because this checkout is missingthirdparty/installed/bin/protoc, andfe/AGENTS.mdrequires stopping FE build setup in that case.
Subagent conclusions: optimizer-rewrite OR-1 and OR-2 were merged into M-1 and submitted as one inline comment. tests-session-config TS-1 became M-2 and TS-2 became M-3. No candidates were dismissed or duplicate-suppressed. Convergence round 2, after anchor repair, ended with both live subagents returning NO_NEW_VALUABLE_FINDINGS for this exact final comment set.
01868c3 to
92408bf
Compare
|
/review |
There was a problem hiding this comment.
Requesting changes based on four affinity-framework gaps that are still live in the current head.
Critical checkpoint conclusions:
- Goal/test: the PR adds the public Resource Group affinity extension surfaces, but the current code does not fully apply them across query/load scheduling paths. Unit tests cover no-op behavior, session validation, and two config gates, but not the affected cloud load and query scheduling cases below.
- Scope/focus: the framework is mostly localized, but it touches several parallel FE scheduling paths; the remaining issues are exactly in those parallel paths.
- Concurrency/lifecycle: no new FE lock or edit-log lifecycle issue found in this pass. Prepared short-circuit stale affinity was already covered by existing review context and is addressed in the current head.
- Configuration/session propagation: the previously raised config gates are now wired in the current head. Remaining load-affinity session behavior is not propagated to cloud non-group stream/MySQL load placement.
- Compatibility: new thrift fields are optional. I could not run generation/build validation here because
.worktree_initialized,thirdparty/installed, and executablethirdparty/installed/bin/protocare missing. - Parallel paths: issues remain between local vs cloud load paths, legacy vs Nereids scan assignment, and plan-time vs audit-time query affinity decisions.
- Tests/observability: missing tests for effective downstream policies on cloud load, ordinary OLAP scan placement, and statement-level query-affinity audit consistency. The new audit fields currently compute a fresh decision instead of reporting the statement decision.
- Transactions/persistence/data writes/performance: no storage, transaction, or persistence correctness issue found; the main performance concern is semantic scheduling behavior, not overhead.
User focus: no additional user-provided focus was supplied.
Subagent conclusions: tests-session-config proposed TSC-1, accepted as M-2. optimizer-rewrite proposed OR-1 and OR-2 during convergence, accepted as M-3 and M-4. M-1 was found by the main pass. Existing threads 3517152408, 3517152410, and 3517152404 were treated as already-known/stale context and not resubmitted. Convergence completed in round 3 with both optimizer-rewrite and tests-session-config returning NO_NEW_VALUABLE_FINDINGS for the same four-comment set.
TPC-H: Total hot run time: 29787 ms |
TPC-DS: Total hot run time: 173516 ms |
ClickBench: Total hot run time: 25.02 s |
92408bf to
a71cbb6
Compare
|
/review |
There was a problem hiding this comment.
Static review against head a71cbb6. I found three correctness issues in the new affinity handling: skip_missing_version replica ordering can be weakened, group-commit backend caching bypasses the effective load-affinity decision, and the default load helper can return an unavailable backend when cloud callers pass unfiltered candidates.
a71cbb6 to
0675b7e
Compare
|
/review |
There was a problem hiding this comment.
Static review found two issues that need changes before merge: forwarded SQL can resolve the requester local resource group from stale heartbeat cache, and the frontends() TVF regression expectations were not updated for the new column. I could not run FE/regression tests because thirdparty/installed/bin/protoc is missing in this checkout.
0675b7e to
c438e64
Compare
TPC-DS: Total hot run time: 158915 ms |
ClickBench: Total hot run time: 23.82 s |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Introduce a public backend selection extension framework for FE query, load, and replica-repair scheduling paths. Align the provider, manager, selection profile, forwarding, asynchronous load, planning, audit, and profile hooks with the downstream public interfaces so downstream implementations can be maintained without carrying kernel API differences. Apache Doris does not ship an affinity provider: the default provider remains strictly no-op and preserves existing candidate ordering and placement behavior.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-fe-ut.sh --run org.apache.doris.planner.OlapTableSinkBackendSelectionExplainTest`
- Behavior changed: No. The Apache Doris default provider remains no-op.
- Does this need documentation: No
3f30d48 to
ec82396
Compare
|
run buildall |
TPC-H: Total hot run time: 29370 ms |
TPC-DS: Total hot run time: 157975 ms |
ClickBench: Total hot run time: 23.77 s |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
|
失败case和pr无关 |
|
skip buildall |
github-actions
Bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
morrySnow
changed the title
morrySnow
left a comment
morrySnow
left a comment
There was a problem hiding this comment.
Review summary
Reviewed the backend selection extension framework. The overall structure is clean: SPI default methods preserve existing behavior, and BackendSelectionManager consistently validates provider contract violations (identity-preserving ordering/partitioning), which is the right fail-fast approach. Tests are extensive.
Key issues found (inline comments):
-
Coordinator observability for broker loads is misleading — the
coordinator_backendshown in SHOW LOAD/audit is captured inBrokerLoadJob.createTaskbefore the coordinator is selected, from the file-scan candidate list (FileLoadScanNoderecordsorderedBackends.get(0)without an availability check), and the first-writer-wins guard inBackendSelectionProfile.recordLoadCoordinatordrops the real coordinator record that happens later inexecuteOnce. -
REQUIRED-mode group commit failure is surfaced as a generic "No suitable backend" error instead of the required-selection error used everywhere else, and REQUIRE cache entries are written but never read.
-
A provider contract violation in repair-source ordering escalates to
UNRECOVERABLE, permanently abandoning tablet repairs. -
Two user-visible/behavioral changes on the default no-op path worth attention:
MysqlLoadManagernow usesselectBackendIdsByPolicy(policy, -1), which dedups to one backend per host (not the case with the previousnumber == 1); and the auditcompute_groupfield semantics change in non-cloud mode once a provider is installed. -
Design question: REQUIRED load selection is also applied to source-table replicas of INSERT-SELECT, so a load can fail when the source table's backends are outside the preferred partition.
morrySnow
left a comment
morrySnow
left a comment
There was a problem hiding this comment.
对于代码里面众多的策略选择if else和他们对应的成员变量，缺少comment解释使用场景和原因，代码很难看懂
| @@ -152,6 +166,10 @@ public DistributedPlanWorker randomAvailableWorker(long catalogId) { | |||
| try { | |||
| Reference<Long> selectedBackendId = new Reference<>(); | |||
| ImmutableMap<Long, Backend> backends = this.currentClusterBackends.get(catalogId); | |||
There was a problem hiding this comment.
不应该去修改这个类，而是创建一个他的子类，覆盖你修改的这两个函数，给load用
| ConnectContext context, boolean useLoadBackendSelection) { | ||
| this.workerManager = workerManager; | ||
| this.context = context; | ||
| this.useLoadBackendSelection = useLoadBackendSelection; |
There was a problem hiding this comment.
这个变量很奇怪，这个类本来就是给load用的，然后又加了一个useLoadBackendSelection 需要给这个变量加注释，解释为false的时候处理什么场景，为true的时候处理什么场景
| static boolean shouldShuffleCandidateBackends(OlapScanNode scanNode) { | ||
| return !scanNode.isScanBackendOrderBySelection(); | ||
| } |
| return execHostport; | ||
| } | ||
|
|
||
| private boolean isLoadSelectionCoordinator() { |
There was a problem hiding this comment.
为load单独写一个子类，或者将choose的逻辑，封装在不同selector里实现，避免if else，会更为健壮
| } | ||
| // In cloud mode, compute_group keeps its existing cloud compute group meaning. In integrated | ||
| // mode, resource groups provide compute affinity, so reuse compute_group for the preferred group. | ||
| String cluster = Config.isCloudMode() ? cloudCluster : selectionHint.getPreferredKey(); |
There was a problem hiding this comment.
这种最好能有接口来提供，现在cloud的 if else满天飞，后面没法维护
|
Thanks for the work on this framework. The SPI design with However, I have a few concerns that I think should be addressed before merging: 1. Cloud mode guard is inconsistent across overloads
Currently the call sites happen to guard correctly (e.g. Suggestion: Add 2. No exception isolation for provider implementations All provider method calls in Since this framework is explicitly designed for downstream extensions via Suggestion: Wrap all provider calls in 3. REQUIRE mode has no graceful degradation
The user setting REQUIRE (possibly via a global session variable) has no visibility into how partition pruning interacts with the selection constraint. The error message ("No candidate satisfies required backend selection key '...'") doesn't explain why no candidate was found. Suggestion: At minimum, enrich the error message with diagnostic context (how many candidates existed, what tags they had, why none matched). Consider also providing a session variable to allow REQUIRE to degrade to PREFER when no preferred candidates survive pruning, rather than failing hard. 4. Duplicate selection computation in the query path In the query path, selection is computed twice on essentially the same candidates:
Same hint, same candidates (different representations), two different SPI methods. This means the provider must implement consistent logic in both More importantly, the I understand this is partly due to the scan range protocol ( |
### What problem does this PR solve? Restore the FE unit and regression coverage from apache#65173 that was omitted from the branch-4.1 backport. The tests also exposed two master prerequisites missing on this branch: bounded group-commit selection caches and no journal wait for the update-load-data shortcut. ### Release note None ### Check List (For Author) - Test - [x] Unit Test - [ ] Regression test - Behavior changed: - [x] Yes. - [ ] No. - Does this need documentation? - [x] No.
6 participants
This PR introduces a public backend selection extension framework for query, load, and replica-repair scheduling paths.
The public interfaces are aligned with the downstream implementation so that downstream extensions can be maintained without carrying kernel API differences.
Apache Doris does not provide or register an affinity
BackendSelectionProvider. The default community provider is strictly no-op:This PR does not introduce any resource-group affinity implementation, enterprise provider, enterprise configuration, or other enterprise-only behavior. Actual affinity policies remain downstream extensions.
What problem does this PR solve?
Issue Number: None
Related PR: None
Check List (For Author)
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)