Repository navigation
DAOS-19632 object: combined patch for test - 2 - #19235
Draft
daltonbohning wants to merge 15 commits into
Draft
daltonbohning wants to merge 15 commits into
daltonbohning wants to merge 15 commits into
Conversation
Layout refresh during rebuild or reintegration can replace shard arrays between selection, target preparation, and dispatch. Protect each layout access and preserve request identities rather than relying only on a pre-dispatch version check. - Revalidate an opened shard against the current layout under cob_lock. Copy its endpoint, target ID, logical shard ID, EC index, and rebuild flags consistently instead of looking up the target in a newer layout. - Validate map versions and layout bounds during EC fetch and degraded shard selection. Pass a copied target ID to failure-list checks so they no longer dereference the layout without protection. - Make parity and enumeration selection version-aware, retain the group size used for selection, and propagate DER_STALE instead of treating a refresh as unavailable parity and falling back to data shards. - Reject stale target lists before fanout and explicitly complete the object task so completion callbacks can arrange map refresh and retry. - Skip success-output processing on failed attempts, preserving enumeration counts and anchors. Obtain key-to-anchor group boundaries from a version-checked snapshot and return stale errors rather than asserting or combining an old group index with a new group size. - Serialize leader-fetch timestamp updates with the layout write lock. - Resolve the CPD leader from classified request/shard entries, retaining the saved target and logical shard ID. Reject unclassified leaders before dispatch construction and protect forced-leader lookup against concurrent layout refresh. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
obj_ptr2shards() read cob_shards_nr, cob_grp_size and cob_grp_nr without cob_lock. A concurrent obj_layout_refresh() (e.g. right after reintegration) frees and rebuilds the layout under the write lock, so a stat/punch/sync could see cob_shards_nr == 0 and trip: Assertion '*grp_nr == obj->cob_grp_nr' failed: Unmatched grp nr: 0/16 Read the layout under cob_lock, check it matches the request map version, and return -DER_STALE instead of asserting, as obj_dkey2grpidx() does. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
The 1 MiB chunk-read cache was never invalidated, was used even with caching disabled, and could leave a FUSE request unanswered or touch freed inode state after replying. - Drop overlapping buckets on write, truncate and O_TRUNC; take alock for truncate by path. - Only use chunk reads on caching handles. - Reference count buckets so nothing touches the inode after a reply. - Fall back to a normal read if the slot already has a waiting request. - Add NLT tests for invalidation, concurrent stress and two mounts. Features: dfuse Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…anges dfuse aborted in obj_shard_list_prep() on D_ASSERT(idx != -1) while daos_array_stat() listed the dkeys of an EC_2P2 file during reintegration. A reintegration or extend adds extra shards to each group, so the in-memory group size (cob_grp_size) changes between enumeration calls when the layout is refreshed. Two things in the saved anchors depended on that size: - The dkey anchor stored grp_idx * cob_grp_size. When the group size dropped back from 6 to 4, the stored layout index (30) mapped to a different group (7 instead of 5). - EC sub-anchors were keyed by layout index (st_shard). The new request targets had no matching sub-anchor. shard_anchors_eof_check() only ignored untracked targets when the target count grew, so the shard reached obj_shard_list_prep() and the assert fired. If the indices happened to overlap, enumeration silently resumed on the wrong group with another group's anchors, which could return the wrong set of dkeys (e.g. a wrong array size). Make the anchors independent of layout extension: - Encode and decode the dkey anchor group with the object class group size (obj_get_replicas()), and use cob_grp_nr for the last-group check. key2anchor stores the logical shard id, and daos_obj_anchor_split()/daos_obj_anchor_set() use the class group size, matching the existing DIOF_TO_SPEC_GROUP decoding. - Key EC sub-anchors by logical shard id (st_shard_id) instead of layout index. - Always ignore forwarded shards that no sub-anchor tracks, not only when the shard count grew. Without a layout extension the class and layout group sizes are equal and logical ids equal layout indices, so behavior is unchanged. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Use saved EC target indices for recx conversion and failure tracking. Normalize enumeration/query-key shard positions within each group, and use a version-checked layout stride for query-key selection. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Document that intercepted I/O (libioil, libpil4dfs) bypasses DFuse and is not coherent with data-cached DFuse handles. Name the per-chunk slot count and log the busy-slot fallback. Add a best-effort test that a chunk read sees a write acknowledged early by the write-back cache, with the writer kept open. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…handle writes Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Protect written-range updates, truncate resets and synthetic-zero checks with the active-inode lock to prevent lost updates and inconsistent reads. Disable cached EOF after local mutations for the active-inode lifetime. Add regression tests for concurrent writes and cross-handle EOF invalidation. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…t caching Serve pre-read replies only until a write or truncate through any handle, the buffer may predate that change. Make the invalidation flag atomic so reads do not take the inode lock to check it, and only take the lock for the written-range and chunk-cache state when the container can cache data. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. When target receives rebuild epoch for the leader, it needs to update modification boundary as such rebuild epoch for all the containers in the pool being rebuilt. It will cause subsequent modification with old epoch to be refused and restarted. 3. In theory, it is possible that when exclude some engine, related ULT maybe trying to abort a DTX because of some failure happened during local modification (but related modification has succeeded on remote targets). Related DTX_ABORT RPC has been sent out but not been received or processed by related remote targets. Under such case, the DTX leader is switched because of related engine is exlcuded. And then DTX resync is triggered by new DTX leader who may regard such DTX as committable since all alive targets claim 'PREPARED'. Then the new DTX leader will make different decision that conflict with old DTX leader did. That may cause related DTX to be aborted on some targets but committed on the others, as to data inconsistency among related redundancy group. The new logic forbids to abort DTX from old DTX leader to avoid such inconsistency. 4. If the modification is in rebuilding/reintegrating, then related modification will use synchronous DTX that will help to refresh global stable epoch. 5. During DTX resync, the DTX CHECK RPC should not be sent to the target with UP status to avoid unexpected DTX status. Signed-off-by: Fan Yong <fan.yong@hpe.com> Co-authored-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Wang Shilong <shilong.wang@hpe.com>
…hning/daos-19632-combined-2
…cache-invalidate' into dbohning/daos-19632-combined-2
…into dbohning/daos-19632-combined-2
…ohning/daos-19632-combined-2
|
Ticket title is 'fio jobs were getting killed after second engine failure' |
Collaborator
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19235/1/execution/node/1467/log |
Collaborator
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19235/1/execution/node/1650/log |
Collaborator
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19235/1/execution/node/1807/log |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines:
Steps for the author:
After all prior steps are complete: