Repository navigation
Conversation
|
Ticket title is 'FIO crc mismatch detected on 4x node IB config while running HSN cable pulls test' |
github-actions
Bot
added
the
priority
Collaborator
|
Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/1/execution/node/1693/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19616_1
branch
from
October 11, 2026 16:59
77c56a7 to
c6aab37
Compare
Collaborator
|
Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/2/execution/node/1699/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19616_1
branch
2 times, most recently
from
October 12, 2026 08:34
5de1b93 to
aa291fc
Compare
Collaborator
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/3/execution/node/1369/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19616_1
branch
3 times, most recently
from
October 12, 2026 13:38
775a238 to
69332d3
Compare
Collaborator
|
Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/6/execution/node/1683/log |
Collaborator
|
Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/6/execution/node/1698/log |
Collaborator
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/6/execution/node/1755/log |
Collaborator
|
Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/7/execution/node/1705/log |
1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. Before handling some modification on DTX leader, need to check whether the epoch for current modification is newer than pool's rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to the client that will retry the modification sometime later. 3. Fix some potential issues that may block container global stable epoch refresh: cont_iv_track_eph_retry() retried indefinitely on any retryable error. When a rank goes down, the CRT_IV_SYNC_EAGER update to that rank keeps returning -DER_HG until the pool map evicts it, so the loop spun forever while cont_agg_eph_sync() held cs_cont_ephs_mutex, blocking the GSE refresh for every container of the pool and making its existing network-error recovery path unreachable. Bound the retry to 30 seconds, return the last error so the caller resumes with the latest pool map on its next round, and log the retries. The local stable epoch is only sampled once per iteration of the VOS aggregation ULT, before a possibly very long (rate controlled) aggregation round. Sample it also from agg_rate_ctl(), at most once per second, so a long round cannot starve the reporting. Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT to control whether continue rebuild process or not if got timeout when wait for global stable epoch. It is false by default and rebuild will failout (and restart) when got timeout. If explicitly set it as true, then there is risk of data lost in subsequent rebuild process if race with inflight IO. It is a temporary solution and will be replaced in subsequent release. Signed-off-by: Fan Yong <fan.yong@hpe.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19616_1
branch
from
October 13, 2026 03:56
69332d3 to
feff308
Compare
A rank whose targets are PO_COMP_ST_NEW (an extend that has not finished) is not a member of the pool group yet, so it never receives the pool map through IV, its eph_report_ult() stays in "Wait for pool map" and it never reports any epoch. However cont_agg_eph_sync() on the PS leader built its per-rank table from all ranks in the pool map and only skipped DOWN/DOWNOUT ones, so such a rank contributed its initial 0 to the minimum. The leader then refreshed every container with stable epoch 0, which vos_cont_set_global_stable_epoch() ignores, and the persisted global stable epoch stayed frozen (two weeks in the reported case). Rebuild waits for the global stable epoch to pass the rebuild epoch, timed out after 600 seconds, failed out and was rescheduled every ten minutes for ever. Skip NEW ranks in cont_agg_eph_sync() and NEW targets in ds_cont_eph_report(), consistently with the collective masks already used when applying the refreshed epochs and with rebuild_leader_status_check(). Skip the refresh when no rank is left. Also add non-debug diagnostics for this case: the leader warns (rate limited) which rank has never reported a stable epoch, and eph_report_ult() warns every ten minutes while it is still waiting for the pool map. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
rebuild_container_scan_cb() waits for the container's global stable epoch (GSE) to pass the rebuild epoch before scanning it. The GSE is refreshed by the PS leader, and ds_cont_leader_update_track_eph() only tracks containers that exist in the container service RDB. An orphan container, which exists on this engine's VOS but not on the PS leader, is therefore never refreshed, its GSE stays at 0 and the scanner spins on it until the pool-wide 600 seconds wait budget is exhausted, then the rebuild fails out although the container should simply be skipped. The orphan check (ds_cont_fetch_snaps() returning -DER_CONT_NONEXIST or -DER_CONT_DESTROYING) already existed, but it ran after the GSE wait. Move the snapshot fetch in front of the wait so an orphan container is skipped right away, the same way the puller side does it in migrate_cont_iter_cb(). The fetch is done once per container, not on every retry of the wait loop. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
comp_may_change_layout() decides whether a component makes the PRE_REBUILD and POST_REBUILD layouts differ. Only the transitional states UP, DOWN and DRAIN do; NEW is skipped by the jump hash in both modes, so it must not be reported. Replace the negative check (!= UPIN && != DOWNOUT), which would wrongly flag NEW, with an explicit list of the transitional states. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
The PS leader tracks the stable epoch of a container per rank and takes the minimum as the global stable epoch (GSE). If some target of an UPIN rank has no shard of the container (in 2.8 shards are created lazily on open/migrate and only recovered on reintegration), that target contributes 0 to the per-rank minimum, so the rank never reports a stable epoch for the container, the leader keeps 0 for the rank and the GSE stays pinned forever. Rebuild then waits on the GSE until the 600 seconds timeout and fails. Self-heal the missing shard on the target side. The periodic track epoch refresh (IV, xstream 0 on every rank) already detects the case as -DER_CONT_NONEXIST from the per-target collective; when that happens verify the container still exists (IV props fetch, fails once the container is destroyed) and re-create the missing shard(s) with ds_cont_child_open_create(), the same way the container recovery does on reintegration. NEW/DOWN/DOWNOUT targets are skipped. On the leader, force a refresh of the container each time the "no stable epoch reported from rank" warning is issued (every 600 s while the GSE is pinned at 0), otherwise the unchanged epochs short- circuit the refresh and the heal would be attempted only once. Also initialise cte_stable_warn_ts at allocation to avoid a spurious warning right after a container starts being tracked. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Collaborator
|
Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19239/8/execution/node/1683/log |
For RB_OP_RECLAIM or RB_OP_FAIL_RECLAIM, it is unnecessary to wait stable epoch, instead, just notify DTX logic in VOS to ignore non-committed DTX. Increase CI test time for checksum + container to avoid fake timeout. Signed-off-by: Fan Yong <fan.yong@hpe.com>
Check buffer capacity against current write offset, not a potentially stale iov_len. Set iov_len to the bytes merged this round, rather than accumulating across rounds. Signed-off-by: Fan Yong <fan.yong@hpe.com>
Contributor
Author
|
Not need it any longer. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DAOS-19616 object: handle inflight IO during rebuild start
On each target, before scanning the container shard for rebuild,
the logic will wait its global stable epoch (per container based)
to exceed rebuild epoch. It will guarantee that all the inflight
IO result that are triggered before rebuild start is visible to
the subsequent rebuild logic whether on DTX leader or not.
Before handling some modification on DTX leader, need to check
whether the epoch for current modification is newer than pool's
rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to
the client that will retry the modification sometime later.
Fix some potential issues that may block container global stable
epoch refresh:
cont_iv_track_eph_retry() retried indefinitely on any retryable
error. When a rank goes down, the CRT_IV_SYNC_EAGER update to that
rank keeps returning -DER_HG until the pool map evicts it, so the
loop spun forever while cont_agg_eph_sync() held cs_cont_ephs_mutex,
blocking the GSE refresh for every container of the pool and making
its existing network-error recovery path unreachable. Bound the
retry to 30 seconds, return the last error so the caller resumes
with the latest pool map on its next round, and log the retries.
The local stable epoch is only sampled once per iteration of the VOS
aggregation ULT, before a possibly very long (rate controlled)
aggregation round. Sample it also from agg_rate_ctl(), at most once
per second, so a long round cannot starve the reporting.
Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT
to control whether continue rebuild process or not if got timeout when
wait for global stable epoch. It is false by default and rebuild will
failout (and restart) when got timeout. If explicitly set it as true,
then there is risk of data lost in subsequent rebuild process if race
with inflight IO. It is a temporary solution and will be replaced in
subsequent release.
Signed-off-by: Fan Yong fan.yong@hpe.com
Signed-off-by: Xuezhao Liu xuezhao.liu@hpe.com
Signed-off-by: Wang Shilong shilong.wang@hpe.com
Steps for the author:
After all prior steps are complete: