DAOS-19636 object: fix EC selection and CPD layout refresh races by gnailzenh · Pull Request #19189 · daos-stack/daos

gnailzenh · 2026-10-02T18:31:01Z

Layout refresh during rebuild or reintegration can replace shard arrays between selection, target preparation, and dispatch. Protect each layout access and preserve request identities rather than relying only on a pre-dispatch version check.

Revalidate an opened shard against the current layout under cob_lock. Copy its endpoint, target ID, logical shard ID, EC index, and rebuild flags consistently instead of looking up the target in a newer layout.
Validate map versions and layout bounds during EC fetch and degraded shard selection. Pass a copied target ID to failure-list checks so they no longer dereference the layout without protection.
Make parity and enumeration selection version-aware, retain the group size used for selection, and propagate DER_STALE instead of treating a refresh as unavailable parity and falling back to data shards.
Reject stale target lists before fanout and explicitly complete the object task so completion callbacks can arrange map refresh and retry.
Skip success-output processing on failed attempts, preserving enumeration counts and anchors. Obtain key-to-anchor group boundaries from a version-checked snapshot and return stale errors rather than asserting or combining an old group index with a new group size.
Serialize leader-fetch timestamp updates with the layout write lock.
Resolve the CPD leader from classified request/shard entries, retaining the saved target and logical shard ID. Reject unclassified leaders before dispatch construction and protect forced-leader lookup against concurrent layout refresh.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Layout refresh during rebuild or reintegration can replace shard arrays between selection, target preparation, and dispatch. Protect each layout access and preserve request identities rather than relying only on a pre-dispatch version check. - Revalidate an opened shard against the current layout under cob_lock. Copy its endpoint, target ID, logical shard ID, EC index, and rebuild flags consistently instead of looking up the target in a newer layout. - Validate map versions and layout bounds during EC fetch and degraded shard selection. Pass a copied target ID to failure-list checks so they no longer dereference the layout without protection. - Make parity and enumeration selection version-aware, retain the group size used for selection, and propagate DER_STALE instead of treating a refresh as unavailable parity and falling back to data shards. - Reject stale target lists before fanout and explicitly complete the object task so completion callbacks can arrange map refresh and retry. - Skip success-output processing on failed attempts, preserving enumeration counts and anchors. Obtain key-to-anchor group boundaries from a version-checked snapshot and return stale errors rather than asserting or combining an old group index with a new group size. - Serialize leader-fetch timestamp updates with the layout write lock. - Resolve the CPD leader from classified request/shard entries, retaining the saved target and logical shard ID. Reject unclassified leaders before dispatch construction and protect forced-leader lookup against concurrent layout refresh. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

github-actions · 2026-10-02T18:31:19Z

Ticket title is 'Object client may use inconsistent layout data during replica retry'
Status is 'Reopened'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-19636

obj_ptr2shards() read cob_shards_nr, cob_grp_size and cob_grp_nr without cob_lock. A concurrent obj_layout_refresh() (e.g. right after reintegration) frees and rebuilds the layout under the write lock, so a stat/punch/sync could see cob_shards_nr == 0 and trip: Assertion '*grp_nr == obj->cob_grp_nr' failed: Unmatched grp nr: 0/16 Read the layout under cob_lock, check it matches the request map version, and return -DER_STALE instead of asserting, as obj_dkey2grpidx() does. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

Use saved EC target indices for recx conversion and failure tracking. Normalize enumeration/query-key shard positions within each group, and use a version-checked layout stride for query-key selection. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

daosbuild3 · 2026-10-07T04:58:22Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19189/3/execution/node/1682/log

Nasf-Fan · 2026-10-08T05:55:46Z

-bool
-obj_need_coll(struct dc_object *obj, uint32_t *start_shard, uint32_t *shard_nr,
+int
+obj_ptr2shards(struct dc_object *obj, uint32_t map_ver, uint32_t *start_shard, uint32_t *shard_nr,


Suggest to move it to cli_obj.c since it is not special for collective operation.

Nasf-Fan · 2026-10-08T05:55:59Z

 		if (i < 0)
 			D_GOTO(out, rc = i);

+		rc = dc_tx_leader_resolve(dtrgs, tgt_cnt, dc_tx_leftmost_req(tx, true) - start, i,


Why need this? Why not directly check pool map version?

…anges (#19207) dfuse aborted in obj_shard_list_prep() on D_ASSERT(idx != -1) while daos_array_stat() listed the dkeys of an EC_2P2 file during reintegration. A reintegration or extend adds extra shards to each group, so the in-memory group size (cob_grp_size) changes between enumeration calls when the layout is refreshed. Two things in the saved anchors depended on that size: - The dkey anchor stored grp_idx * cob_grp_size. When the group size dropped back from 6 to 4, the stored layout index (30) mapped to a different group (7 instead of 5). - EC sub-anchors were keyed by layout index (st_shard). The new request targets had no matching sub-anchor. shard_anchors_eof_check() only ignored untracked targets when the target count grew, so the shard reached obj_shard_list_prep() and the assert fired. If the indices happened to overlap, enumeration silently resumed on the wrong group with another group's anchors, which could return the wrong set of dkeys (e.g. a wrong array size). Make the anchors independent of layout extension: - Encode and decode the dkey anchor group with the object class group size (obj_get_replicas()), and use cob_grp_nr for the last-group check. key2anchor stores the logical shard id, and daos_obj_anchor_split()/daos_obj_anchor_set() use the class group size, matching the existing DIOF_TO_SPEC_GROUP decoding. - Key EC sub-anchors by logical shard id (st_shard_id) instead of layout index. - Always ignore forwarded shards that no sub-anchor tracks, not only when the shard count grew. Without a layout extension the class and layout group sizes are equal and logical ids equal layout indices, so behavior is unchanged. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

Encode key2anchor groups with the class width, replace CPD leader resolution with a locked version-checked snapshot, and move obj_ptr2shards to cli_obj.c. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

Snapshot the version-checked group size for allocation and iteration, and reject layout refresh before completing classification. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

Invalidate the initial shard marker when a replicated fetch retries with a different pool map, and reselect a replica in the new layout. Record the new cycle start to avoid looping on an old layout index. Preserve the original redundancy error when no replica remains. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

daosbuild3 · 2026-10-12T15:56:39Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19189/7/execution/node/1362/log

phender · 2026-10-13T04:39:47Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19189/7/execution/node/1362/log

The three functional test failures in the VM stage look to be related to https://daosio.atlassian.net/browse/DAOS-19766.

gnailzenh · 2026-10-13T04:53:30Z

https://daosio.atlassian.net/browse/DAOS-19766

gnailzenh · 2026-10-13T04:53:58Z

Sorry I clicked the wrong bottom...I wanted to add a comment but I closed the PR by mistake.

Nasf-Fan · 2026-10-14T08:05:45Z

 	cb_args.epoch = &args->ka_auxi.epoch;
 	cb_args.th = &obj_args->th;
 	cb_args.anchor = args->ka_anchor;
-	cb_args.shard = obj_shard->do_shard_idx;


Is it correct? during reintegration, shard index maybe different from shard ID. Which one is expected to be used?

do_shard_idx is the position in client layout array, which can be invalid after refresh, do_shard is the logic shard ID, which can identify the actual shard in the layout. I agree naming is confusing, we need a cleanup in the future.

What will happen if two shards with the identifiers (one old, the other is new) for the same object in the same redundancy group during reintegration?

Nasf-Fan

According to the discussion with Liang, he will resolve daos_anchor_t::shard related issue in the future.

github-actions Bot added the priority Ticket has high priority (automatically managed) label Oct 2, 2026

gnailzenh requested review from Nasf-Fan and wangshilong October 2, 2026 23:53

Michael-Hennecke added the release-2.8.1 Targeted for release 2.8.1 label Oct 5, 2026

Nasf-Fan reviewed Oct 8, 2026

View reviewed changes

This was referenced Oct 8, 2026

DAOS-19632 object: combined patch for test #19224

Draft

DAOS-19632 object: combined patch for test - 2 #19235

Draft

This was referenced Oct 11, 2026

DAOS-19632 object: combined patch for test - OBSOLETE #19241

Closed

DAOS-19632 object: combined patch for test - 3 #19242

Draft

gnailzenh added 4 commits October 12, 2026 16:02

DAOS-19636 object: simplify layout helpers and CPD leader lookup

f164d57

Encode key2anchor groups with the class width, replace CPD leader resolution with a locked version-checked snapshot, and move obj_ptr2shards to cli_obj.c. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

DAOS-19636 object: use a stable layout stride for CPD classification

933befd

Snapshot the version-checked group size for allocation and iteration, and reject layout refresh before completing classification. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

DAOS-19636 object: fixes for clang format

ece700d

Signed-off-by: Liang Zhen <gnailzenh@gmail.com>

gnailzenh requested a review from liuxuezhao October 12, 2026 12:40

daltonbohning mentioned this pull request Oct 12, 2026

DAOS-19632 object: combined patch for test - 4 #19250

Draft

6 tasks

gnailzenh closed this Oct 13, 2026

gnailzenh reopened this Oct 13, 2026

daltonbohning mentioned this pull request Oct 13, 2026

DAOS-19632 object: combined patch for test - 5 #19261

Draft

6 tasks

Nasf-Fan reviewed Oct 14, 2026

View reviewed changes

wangshilong approved these changes Oct 14, 2026

View reviewed changes

Nasf-Fan approved these changes Oct 14, 2026

View reviewed changes

This was referenced Oct 14, 2026

DAOS-19632 object: combined patch for test - 5 #19271

Closed

DAOS-19632 object: combined patch for test - 6 #19272

Draft

gnailzenh merged commit 98ec319 into release/2.8 Oct 14, 2026
59 of 60 checks passed

gnailzenh deleted the liang/b2_8_layout_race branch October 14, 2026 15:42

Navigation Menu

Repository navigation

DAOS-19636 object: fix EC selection and CPD layout refresh races - #19189

DAOS-19636 object: fix EC selection and CPD layout refresh races#19189
gnailzenh merged 8 commits into
release/2.8from
liang/b2_8_layout_race

gnailzenh commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

daosbuild3 commented Oct 7, 2026

Uh oh!

Nasf-Fan Oct 8, 2026

Uh oh!

Nasf-Fan Oct 8, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

phender commented Oct 13, 2026

Uh oh!

gnailzenh commented Oct 13, 2026

Uh oh!

gnailzenh commented Oct 13, 2026 •

edited

Loading

Uh oh!

Nasf-Fan Oct 14, 2026

Uh oh!

gnailzenh Oct 14, 2026

Uh oh!

Nasf-Fan Oct 14, 2026

Uh oh!

Nasf-Fan left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

Conversation

gnailzenh commented Oct 2, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

daosbuild3 commented Oct 7, 2026

Uh oh!

Nasf-Fan Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

phender commented Oct 13, 2026

Uh oh!

gnailzenh commented Oct 13, 2026

Uh oh!

gnailzenh commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nasf-Fan Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

gnailzenh Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

gnailzenh commented Oct 13, 2026 •

edited

Loading