Repository navigation
Lxz/daos 19616 b28 drain - #19233
Closed
liuxuezhao wants to merge 7 commits into
Closed
Lxz/daos 19616 b28 drain#19233liuxuezhao wants to merge 7 commits into
liuxuezhao wants to merge 7 commits into
Conversation
1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. Before handling some modification on DTX leader, need to check whether the epoch for current modification is newer than pool's rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to the client that will retry the modification sometime later. 3. In theory, it is possible that when exclude some engine, related ULT maybe trying to abort a DTX because of some failure happened during local modification (but related modification has succeeded on remote targets). Related DTX_ABORT RPC has been sent out but not been received or processed by related remote targets. Under such case, the DTX leader is switched because of related engine is exlcuded. And then DTX resync is triggered by new DTX leader who may regard such DTX as committable since all alive targets claim 'PREPARED'. Then the new DTX leader will make different decision that conflict with old DTX leader did. That may cause related DTX to be aborted on some targets but committed on the others, as to data inconsistency among related redundancy group. The new logic forbids to abort DTX from old DTX leader to avoid such inconsistency. Signed-off-by: Fan Yong <fan.yong@hpe.com> Co-authored-by: Wang Shilong <shilong.wang@hpe.com>
Commit 4978f8d makes rebuild_container_scan_cb() wait until the container's global stable epoch (GSE) reaches rpt->rt_stable_epoch before scanning. A target in DRAIN status is still serving I/O, is counted in the PS leader's min stable epoch calculation and takes part in rebuild scanning (is_rebuild_scanning_tgt() includes DRAIN), but cont_refresh_track_eph_one() only updated the GSE for targets in UPIN status. The GSE on a draining target therefore stayed frozen at its pre-drain value, so every container on it hit the 600s timeout (-DER_TIMEDOUT) in rebuild_container_scan_cb() and drain failed. Allow the GSE refresh for UPIN | DRAIN targets. UP (reintegrating) targets are intentionally still excluded so their GSE remains frozen at the value they had when excluded, which incremental reintegration depends on. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
rebuild_container_scan_cb() waits until the container's global stable epoch (GSE) reaches rpt->rt_stable_epoch. But a container created with an old layout has no container extension (cd_ext == UMOFF_NULL) and a pool with pd_version < VOS_POOL_DF_2_8 never tracks the GSE at all; vos_cont_get_global_stable_epoch() always returns zero for them and vos_cont_set_global_stable_epoch() returns -DER_NOTSUPPORTED. Pool upgrade does not allocate the extension for existing containers. As a result, after upgrading from a release without the extension, every rebuild/reintegrate/drain/extend waited for the full timeout on each such container and then failed with -DER_TIMEDOUT, forever. Add vos_cont_global_stable_epoch_supported() and, when the container cannot track the GSE, skip the wait (falling back to the former behavior of scanning right after DTX resync) with an INFO message. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
The wait in rebuild_container_scan_cb() for the container global stable epoch (GSE) to exceed the rebuild epoch gave up after a hard coded 600 seconds and returned -DER_TIMEDOUT, which aborts the pool iteration and fails the whole rebuild; the leader then retries the rebuild and hits the same wait again. The GSE is a pool-wide minimum over all healthy targets, so it can be held back for a long time by a single slow target (local stable epoch is only re-evaluated once per VOS aggregation round), by any long lived prepared DTX, or by a reintegrating target whose reindexed old DTXs are not yet resolved by DTX resync. In such cases failing and retrying does not improve anything, it just adds more churn, while the periodic WARN already makes the stall visible. Keep waiting by default (the loop still honors rt_abort/rt_finishing) and only time out when an explicit limit is configured through the new DAOS_REBUILD_GSE_WAIT_TIMEOUT environment variable (seconds). Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
The DTX_CHECK / DTX_COLL_CHECK handlers (and the local pre-check in dtx_check()/dtx_coll_check()) now call vos_dtx_check() with DCI_RESYNC and pass the checker's pool map version via @pm_ver so that it can be recorded in dae_known_max_version. Formerly these callers passed a NULL @pm_ver. A non-NULL @pm_ver however also enables the tail check in vos_dtx_check(): if *pm_ver > vc_dtx_resync_ver the prepared DTX is reported as -DER_INPROGRESS. vc_dtx_resync_ver is only bumped at the very end of the local dtx_resync(). So when two targets are leaders for each other's DTXs under the same new map version, each one's resync asks the other and gets -DER_INPROGRESS -> DSHR_NEED_RETRY -> re-queued and retried without bound, both waiting for the other to finish first. This stalls DTX resync and therefore rebuild start. The resync checker is the one deciding the DTX fate, it never needs to wait for the checked target's own resync. For DCI_RESYNC record the version and return DTX_ST_PREPARED / DTX_ST_INITED directly, restoring the previous behavior for these callers. Also document that dae_known_max_version is DRAM-only. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
obj_handle_resend() now passes the request's pool map version to vos_dtx_abort() when aborting the old-epoch DTX of a resent RPC. vos_dtx_abort() refuses the abort with -DER_NO_PERM when a DTX check with a higher map version (the new leader's DTX resync) has already seen the DTX, or when the DTX turned out to be committed by race. That -DER_NO_PERM was returned unchanged to the client, which treats it as a permanent failure and does not retry, so an application I/O failed for what is only a transient leader hand-over. Map -DER_NO_PERM to -DER_INPROGRESS so the client retries (with a refreshed pool map) and the new leader decides the DTX fate. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
With the container global stable epoch (GSE) barrier in place, the rebuild scan/migration should never meet a prepared DTX whose epoch is not newer than the rebuild epoch, and the MIGRATION branch of vos_dtx_check_availability() reports such a hit with D_ERROR. It then still returns ALB_UNAVAILABLE, i.e. silently skips the entry. Skipping is the dangerous part: this is exactly the race that caused the DAOS-19616 data corruption. If the barrier is bypassed (e.g. for an old-layout container that cannot track the GSE) a prepared DTX on a non-leader replica may already be committable on the leader while the batched commit has not arrived; skipping it makes the migration fetch miss the record or return an older version, which is then written over the data already on the in-rebuilding target. Add a defense-in-depth fallback for that unexpected case, keeping the D_ERROR as the alarm: - in-flight DTX (not prepared / being prepared / still owned by the RPC handler): skip, as the DEFAULT intent does; - prepared DTX on a non-leader: return -DER_INPROGRESS through dtx_inprogress() so the fetch/enumeration handler refreshes with the DTX leader and retries, same as a regular non-leader read; - prepared DTX on the leader without owner (abort in progress or leader restart, to be resolved by DTX resync): skip. The rebuild object scan (dth_ignore_uncommitted) cannot refresh and keeps skipping. DTX newer than the rebuild epoch keep the fast path. The dsc client does not retry -DER_INPROGRESS for migration (obj_retriable_migrate()), so make mrone_obj_fetch_internal() retry it like it already does for -DER_TIMEDOUT / -DER_FETCH_AGAIN, instead of failing the whole rebuild. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
|
Errors are component not formatted correctly,Ticket number prefix incorrect,PR title is malformatted. See https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments,Unable to load ticket data |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19616_b28
branch
15 times, most recently
from
October 11, 2026 14:50
27b7deb to
40b2780
Compare
Collaborator
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19233/1/execution/node/1763/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19616_b28
branch
from
October 11, 2026 16:25
40b2780 to
9600127
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Steps for the author:
After all prior steps are complete: