Skip to content

DAOS-19636 object: fix client layout refresh races - #19248

Merged
gnailzenh merged 1 commit into
masterfrom
liang/b_layout_race
Oct 14, 2026
Merged

gnailzenh merged 1 commit into
masterfrom
liang/b_layout_race

Conversation

@gnailzenh

Copy link
Copy Markdown
Collaborator

Layout refresh during rebuild or reintegration can invalidate physical shard indices between selection, target preparation, and RPC dispatch. Protect layout accesses and retain consistent request shard identities.

  • Revalidate opened shards under cob_lock and snapshot endpoints, target IDs, logical shard IDs, EC indices, and rebuild flags consistently.
  • Make EC fetch, degraded reads, parity, and enumeration selection version-aware; propagate DER_STALE instead of using mismatched layouts.
  • Reject stale target lists before fanout via normal task completion.
  • Preserve saved EC indices for checksum and enumeration processing.
  • Identify sub-anchors by logical shard ID and use fixed class widths for group-anchor arithmetic across extended layout changes.
  • Skip success-output processing on failed attempts and validate key-to-anchor group boundaries against the requested map version.
  • Use a stable stride for CPD classification and resolve leaders from classified entries; protect forced-leader lookup and timestamp updates.
  • Reselect replicas after map changes, establish a new retry-cycle start, and preserve redundancy errors when no eligible replica remains.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@github-actions

Copy link
Copy Markdown

Ticket title is 'Object client may use inconsistent layout data during replica retry'
Status is 'Reopened'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-19636

@github-actions github-actions Bot added the priority Ticket has high priority (automatically managed) label Oct 12, 2026
Layout refresh during rebuild or reintegration can invalidate physical
shard indices between selection, target preparation, and RPC dispatch.
Protect layout accesses and retain consistent request shard identities.

- Revalidate opened shards under cob_lock and snapshot endpoints, target
  IDs, logical shard IDs, EC indices, and rebuild flags consistently.
- Make EC fetch, degraded reads, parity, and enumeration selection
  version-aware; propagate DER_STALE instead of using mismatched layouts.
- Reject stale target lists before fanout via normal task completion.
- Preserve saved EC indices for checksum and enumeration processing.
- Identify sub-anchors by logical shard ID and use fixed class widths
  for group-anchor arithmetic across extended layout changes.
- Skip success-output processing on failed attempts and validate
  key-to-anchor group boundaries against the requested map version.
- Use a stable stride for CPD classification and resolve leaders from
  classified entries; protect forced-leader lookup and timestamp updates.
- Reselect replicas after map changes, establish a new retry-cycle start,
  and preserve redundancy errors when no eligible replica remains.

Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19248/2/execution/node/1987/log

@gnailzenh
gnailzenh merged commit 8d81c0b into master Oct 14, 2026
37 of 39 checks passed
@gnailzenh
gnailzenh deleted the liang/b_layout_race branch October 14, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority Ticket has high priority (automatically managed)

Development

Successfully merging this pull request may close these issues.

4 participants