Repository navigation
DAOS-19632 object: combined patch for test - #19224
Draft
daltonbohning wants to merge 23 commits into
Draft
daltonbohning wants to merge 23 commits into
daltonbohning wants to merge 23 commits into
Conversation
Layout refresh during rebuild or reintegration can replace shard arrays between selection, target preparation, and dispatch. Protect each layout access and preserve request identities rather than relying only on a pre-dispatch version check. - Revalidate an opened shard against the current layout under cob_lock. Copy its endpoint, target ID, logical shard ID, EC index, and rebuild flags consistently instead of looking up the target in a newer layout. - Validate map versions and layout bounds during EC fetch and degraded shard selection. Pass a copied target ID to failure-list checks so they no longer dereference the layout without protection. - Make parity and enumeration selection version-aware, retain the group size used for selection, and propagate DER_STALE instead of treating a refresh as unavailable parity and falling back to data shards. - Reject stale target lists before fanout and explicitly complete the object task so completion callbacks can arrange map refresh and retry. - Skip success-output processing on failed attempts, preserving enumeration counts and anchors. Obtain key-to-anchor group boundaries from a version-checked snapshot and return stale errors rather than asserting or combining an old group index with a new group size. - Serialize leader-fetch timestamp updates with the layout write lock. - Resolve the CPD leader from classified request/shard entries, retaining the saved target and logical shard ID. Reject unclassified leaders before dispatch construction and protect forced-leader lookup against concurrent layout refresh. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Retry updates, punches, and CPD writes until their epoch is newer than the rebuild stable epoch. Features: rebuild Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
Features: rebuild Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
Keep the stable-epoch gate and add a persistent resync-start watermark. Drain old-map DTX handles with a shared five-minute wait limit. Use DTX_RESYNC_NOWAIT only for container-open resync. Warn and proceed on wait expiry without aborting unresolved DTXs. Warn and skip unresolved migration records instead of failing rebuild. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
obj_ptr2shards() read cob_shards_nr, cob_grp_size and cob_grp_nr without cob_lock. A concurrent obj_layout_refresh() (e.g. right after reintegration) frees and rebuilds the layout under the write lock, so a stat/punch/sync could see cob_shards_nr == 0 and trip: Assertion '*grp_nr == obj->cob_grp_nr' failed: Unmatched grp nr: 0/16 Read the layout under cob_lock, check it matches the request map version, and return -DER_STALE instead of asserting, as obj_dkey2grpidx() does. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
The 1 MiB chunk-read cache was never invalidated, was used even with caching disabled, and could leave a FUSE request unanswered or touch freed inode state after replying. - Drop overlapping buckets on write, truncate and O_TRUNC; take alock for truncate by path. - Only use chunk reads on caching handles. - Reference count buckets so nothing touches the inode after a reply. - Fall back to a normal read if the slot already has a waiting request. - Add NLT tests for invalidation, concurrent stress and two mounts. Features: dfuse Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
The server previously relied on ORF_REINTEGRATING_IO to reject updates while VOS discard was running. An update missing this client-provided flag could therefore race with discard. Fetch requests also bypassed the discard check entirely. Use the server-side discard state and local target status to protect only targets participating in reintegration. Reject fetches with DER_FETCH_AGAIN and reject all modification requests with DER_UPDATE_AGAIN while the target is being discarded. Log unexpected fetches and modification requests missing ORF_REINTEGRATING_IO to help identify placement or flag propagation issues. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
…anges dfuse aborted in obj_shard_list_prep() on D_ASSERT(idx != -1) while daos_array_stat() listed the dkeys of an EC_2P2 file during reintegration. A reintegration or extend adds extra shards to each group, so the in-memory group size (cob_grp_size) changes between enumeration calls when the layout is refreshed. Two things in the saved anchors depended on that size: - The dkey anchor stored grp_idx * cob_grp_size. When the group size dropped back from 6 to 4, the stored layout index (30) mapped to a different group (7 instead of 5). - EC sub-anchors were keyed by layout index (st_shard). The new request targets had no matching sub-anchor. shard_anchors_eof_check() only ignored untracked targets when the target count grew, so the shard reached obj_shard_list_prep() and the assert fired. If the indices happened to overlap, enumeration silently resumed on the wrong group with another group's anchors, which could return the wrong set of dkeys (e.g. a wrong array size). Make the anchors independent of layout extension: - Encode and decode the dkey anchor group with the object class group size (obj_get_replicas()), and use cob_grp_nr for the last-group check. key2anchor stores the logical shard id, and daos_obj_anchor_split()/daos_obj_anchor_set() use the class group size, matching the existing DIOF_TO_SPEC_GROUP decoding. - Key EC sub-anchors by logical shard id (st_shard_id) instead of layout index. - Always ignore forwarded shards that no sub-anchor tracks, not only when the shard count grew. Without a layout extension the class and layout group sizes are equal and logical ids equal layout indices, so behavior is unchanged. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Use saved EC target indices for recx conversion and failure tracking. Normalize enumeration/query-key shard positions within each group, and use a version-checked layout stride for query-key selection. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…ry (#19216) Restarting rebuild for every non-ready DTX below the stable epoch livelocks rebuild under continuous I/O: each restart samples a new stable epoch, and in-flight writes waiting for async commit always sit just below it. A DTX prepared with the rebuild (or newer) pool map version was sent with a layout that already includes the in-rebuilding target, so it is safe to keep it invisible to migration. Only restart for DTXs from an older map version, which DTX resync should have resolved. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
If pool map is changed during DTX process, then do not mark related DTX as committable, instead, return -DER_STALE to require retry the operation with new pool map. Signed-off-by: Fan Yong <fan.yong@hpe.com>
Document that intercepted I/O (libioil, libpil4dfs) bypasses DFuse and is not coherent with data-cached DFuse handles. Name the per-chunk slot count and log the busy-slot fallback. Add a best-effort test that a chunk read sees a write acknowledged early by the write-back cache, with the writer kept open. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…handle writes Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Protect written-range updates, truncate resets and synthetic-zero checks with the active-inode lock to prevent lost updates and inconsistent reads. Disable cached EOF after local mutations for the active-inode lifetime. Add regression tests for concurrent writes and cross-handle EOF invalidation. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…t caching Serve pre-read replies only until a write or truncate through any handle, the buffer may predate that change. Make the invalidation flag atomic so reads do not take the inode lock to check it, and only take the lock for the written-range and chunk-cache state when the container can cache data. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
|
Ticket title is 'fio jobs were getting killed after second engine failure' |
otherwise it can get inconsistent view of data for online rebuild Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…hning/daos-19632-combined
…cache-invalidate' into dbohning/daos-19632-combined
…ohning/daos-19632-combined
…into dbohning/daos-19632-combined
daltonbohning
force-pushed
the
dbohning/daos-19632-combined
branch
from
October 9, 2026 14:47
c10b8b0 to
eb1259b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines:
Steps for the author:
After all prior steps are complete: