DAOS-18161 object: refine EC aggregation processing by liuxuezhao · Pull Request #17092 · daos-stack/daos

liuxuezhao · 2025-11-07T08:13:01Z

1. After restart take the sc_ec_agg_eph_boundary as ec aggregation's
   min epoch to avoid scan from epoch 0.
2. Consume more credits for layout calculate in EC agg.
3. Don't bump sc_ec_agg_eph after it reset during EC agg, to avoid data corruption.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-11-07T08:13:22Z

Ticket title is 'Aurora daos_user: After PM work daos_user cluster is very slow to repond'
Status is 'In Review'
Labels: 'scrubbed_2.6.5,scrubbed_2.8'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-18161

daosbuild3 · 2025-11-07T11:48:47Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17092/3/display/redirect

NiuYawei · 2025-11-10T09:59:50Z

-	agg_reset_entry(&agg_param->ap_agg_entry, entry, &oca);
+	agg_reset_entry(&agg_param->ap_agg_entry, entry, &oca, false /* fini */, &layout_create);
+	if (!layout_create)
+		return rc;


vos_iterate() ensures the iterator callback will never be called more than once, so the object layout won't be re-calculated on re-probe.

right, reverted, thanks.

liuxuezhao · 2025-11-13T08:44:04Z

we should before landing this. saw several engines asserting with:

#0 0x00007f686997cc6b in raise () from /lib64/libc.so.6 #1 0x00007f686997e305 in abort () from /lib64/libc.so.6 #2 0x00007f6869974c6a in __assert_fail_base () from /lib64/libc.so.6 #3 0x00007f6869974cf2 in __assert_fail () from /lib64/libc.so.6 #4 0x00007f6864704338 in cont_ec_aggregate_cb (cont=cont@entry=0x7e43723ff840, epr=epr@entry=0x7e437244c620, flags=0, agg_param=agg_param@entry=0x7e437244c720) at src/object/srv_ec_aggregate.c:2743 #5 0x00007f686477b7e9 in cont_child_aggregate (param=, agg_cb=0x7f6864703590 <cont_ec_aggregate_cb>, cont=0x7e43723ff840) at src/container/srv_target.c:432 #6 cont_aggregate_interval (cont=cont@entry=0x7e43723ff840, cb=cb@entry=0x7f6864703590 <cont_ec_aggregate_cb>, param=param@entry=0x7e437244c720) at src/container/srv_target.c:478 #7 0x00007f686470766e in ds_obj_ec_aggregate (arg=0x7e43723ff840) at src/object/srv_ec_aggregate.c:2882 #8 0x00007f6869c52f48 in ABTD_ythread_func_wrapper () from /scratchbox/daos/builds/20251111-162956_lxz_ec_agg_start_eph_26/install/bin/../prereq/release/argobots/lib/libabt.so.1 #9 0x00007f6869c428e9 in ABTD_ythread_context_func_wrapper () from /scratchbox/daos/builds/20251111-162956_lxz_ec_agg_start_eph_26/install/bin/../prereq/release/argobots/lib/libabt.so.1 #10 0x0000000000000000 in ?? ()

Thanks for testing. I checked some details and think that assertion cannot be satisfied for some special case, so I just removed it. @NiuYawei please re-check if the latest code is good for you, thanks.

NiuYawei · 2025-11-13T08:54:56Z

we should before landing this. saw several engines asserting with:
#0 0x00007f686997cc6b in raise () from /lib64/libc.so.6 #1 0x00007f686997e305 in abort () from /lib64/libc.so.6 #2 0x00007f6869974c6a in __assert_fail_base () from /lib64/libc.so.6 #3 0x00007f6869974cf2 in __assert_fail () from /lib64/libc.so.6 #4 0x00007f6864704338 in cont_ec_aggregate_cb (cont=cont@entry=0x7e43723ff840, epr=epr@entry=0x7e437244c620, flags=0, agg_param=agg_param@entry=0x7e437244c720) at src/object/srv_ec_aggregate.c:2743 #5 0x00007f686477b7e9 in cont_child_aggregate (param=, agg_cb=0x7f6864703590 <cont_ec_aggregate_cb>, cont=0x7e43723ff840) at src/container/srv_target.c:432 #6 cont_aggregate_interval (cont=cont@entry=0x7e43723ff840, cb=cb@entry=0x7f6864703590 <cont_ec_aggregate_cb>, param=param@entry=0x7e437244c720) at src/container/srv_target.c:478 #7 0x00007f686470766e in ds_obj_ec_aggregate (arg=0x7e43723ff840) at src/object/srv_ec_aggregate.c:2882 #8 0x00007f6869c52f48 in ABTD_ythread_func_wrapper () from /scratchbox/daos/builds/20251111-162956_lxz_ec_agg_start_eph_26/install/bin/../prereq/release/argobots/lib/libabt.so.1 #9 0x00007f6869c428e9 in ABTD_ythread_context_func_wrapper () from /scratchbox/daos/builds/20251111-162956_lxz_ec_agg_start_eph_26/install/bin/../prereq/release/argobots/lib/libabt.so.1 #10 0x0000000000000000 in ?? ()

Thanks for testing. I checked some details and think that assertion cannot be satisfied for some special case, so I just removed it. @NiuYawei please re-check if the latest code is good for you, thanks.

It might because the rank was excluded during the test? @mchaarawi is exclusion involved in the test?

mchaarawi · 2025-11-13T13:54:14Z

we should before landing this. saw several engines asserting with:
#0 0x00007f686997cc6b in raise () from /lib64/libc.so.6 #1 0x00007f686997e305 in abort () from /lib64/libc.so.6 #2 0x00007f6869974c6a in __assert_fail_base () from /lib64/libc.so.6 #3 0x00007f6869974cf2 in __assert_fail () from /lib64/libc.so.6 #4 0x00007f6864704338 in cont_ec_aggregate_cb (cont=cont@entry=0x7e43723ff840, epr=epr@entry=0x7e437244c620, flags=0, agg_param=agg_param@entry=0x7e437244c720) at src/object/srv_ec_aggregate.c:2743 #5 0x00007f686477b7e9 in cont_child_aggregate (param=, agg_cb=0x7f6864703590 <cont_ec_aggregate_cb>, cont=0x7e43723ff840) at src/container/srv_target.c:432 #6 cont_aggregate_interval (cont=cont@entry=0x7e43723ff840, cb=cb@entry=0x7f6864703590 <cont_ec_aggregate_cb>, param=param@entry=0x7e437244c720) at src/container/srv_target.c:478 #7 0x00007f686470766e in ds_obj_ec_aggregate (arg=0x7e43723ff840) at src/object/srv_ec_aggregate.c:2882 #8 0x00007f6869c52f48 in ABTD_ythread_func_wrapper () from /scratchbox/daos/builds/20251111-162956_lxz_ec_agg_start_eph_26/install/bin/../prereq/release/argobots/lib/libabt.so.1 #9 0x00007f6869c428e9 in ABTD_ythread_context_func_wrapper () from /scratchbox/daos/builds/20251111-162956_lxz_ec_agg_start_eph_26/install/bin/../prereq/release/argobots/lib/libabt.so.1 #10 0x0000000000000000 in ?? ()

Thanks for testing. I checked some details and think that assertion cannot be satisfied for some special case, so I just removed it. @NiuYawei please re-check if the latest code is good for you, thanks.

It might because the rank was excluded during the test? @mchaarawi is exclusion involved in the test?

no.. the cluster was idle with no IO running to it either

liuxuezhao · 2025-11-14T02:15:14Z

the assertion should due to ds_cont_child_reset_ec_agg_eph_all() followed by a sc_ec_agg_eph_boundary refresh. unrelated with this PR.

daosbuild3 · 2025-11-14T02:46:44Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17092/16/testReport/

liuxuezhao · 2025-11-14T06:50:51Z

the assertion should due to ds_cont_child_reset_ec_agg_eph_all() followed by a sc_ec_agg_eph_boundary refresh. unrelated with this PR.

yes, things is clear after some discussion with @NiuYawei, and I added the 3rd commit to fix a potential data corruption issue.

To avoid possible data corruption, dont bump sc_ec_agg_eph after it reset during EC agg. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

daosbuild3 · 2025-11-15T00:23:08Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17092/18/execution/node/1435/log

daosbuild3 · 2025-11-17T10:28:51Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17092/19/execution/node/1411/log

daosbuild3 · 2025-11-21T07:15:07Z

Test stage Build RPM on Leap 15.5 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17092/23/execution/node/359/log

daosbuild3 · 2025-11-24T09:49:05Z

Test stage Build RPM on Leap 15.5 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17092/24/execution/node/334/log

daosbuild3 · 2025-11-24T09:49:09Z

Test stage Build RPM on EL 8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17092/24/execution/node/407/log

gnailzenh

It's a clean cherry-pick and only has one irrelevant failure.

* DAOS-18161 object: refine EC aggregation processing 1. After restart take the sc_ec_agg_eph_boundary as ec aggregation's min epoch to avoid scan from epoch 0. 2. Consume more credits for layout calculate in EC agg. * DAOS-18161 object: remove an assertion D_ASSERTF(cont->sc_ec_agg_eph >= cont->sc_ec_agg_eph_boundary) possibly cannot be satisfied with considering ap_min_unagg_eph. So remove it. Refine Sluggish EC boundary bumping comparison. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> * DAOS-18161 object: don't bump sc_ec_agg_eph after it reset To avoid possible data corruption, dont bump sc_ec_agg_eph after it reset during EC agg. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

liuxuezhao requested review from a team as code owners November 7, 2025 08:13

liuxuezhao requested review from a team, NiuYawei, gnailzenh and wangshilong November 7, 2025 08:13

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch from a00301a to 0dda299 Compare November 7, 2025 08:28

liuxuezhao removed request for NiuYawei, gnailzenh and wangshilong November 7, 2025 10:32

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch from 0dda299 to 4dae11f Compare November 7, 2025 11:22

liuxuezhao removed request for a team November 7, 2025 11:23

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch from 4dae11f to 109d9ec Compare November 7, 2025 11:47

liuxuezhao requested a review from NiuYawei November 10, 2025 01:29

NiuYawei reviewed Nov 10, 2025

View reviewed changes

Comment thread src/container/srv_target.c Outdated

Comment thread src/object/srv_ec_aggregate.c Outdated

Comment thread src/container/srv_target.c

Comment thread src/object/srv_ec_aggregate.c

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch from 109d9ec to 259b3b2 Compare November 10, 2025 09:10

liuxuezhao changed the title ~~DAOS-18161 object: refine EC aggregation's epoch_min~~ DAOS-18161 object: refine EC aggregation processing Nov 10, 2025

liuxuezhao requested review from Nasf-Fan, NiuYawei, gnailzenh and wangshilong November 10, 2025 09:12

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch 2 times, most recently from d08f374 to f936c07 Compare November 10, 2025 09:24

NiuYawei reviewed Nov 10, 2025

View reviewed changes

liuxuezhao dismissed wangshilong’s stale review via 229d0a9 November 13, 2025 08:42

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch from 1023eb4 to 229d0a9 Compare November 13, 2025 08:42

liuxuezhao requested review from NiuYawei and wangshilong November 13, 2025 08:44

DAOS-18161 object: don't bump sc_ec_agg_eph after it reset

7dc2110

To avoid possible data corruption, dont bump sc_ec_agg_eph after it reset during EC agg. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

liuxuezhao force-pushed the lxz/ec_agg_start_eph_26 branch from a07d2a5 to 7dc2110 Compare November 14, 2025 08:38

NiuYawei approved these changes Nov 14, 2025

View reviewed changes

wangshilong approved these changes Nov 16, 2025

View reviewed changes

liuxuezhao added 3 commits November 18, 2025 01:42

Merge branch 'release/2.6' into lxz/ec_agg_start_eph_26

848c3dd

Merge branch 'release/2.6' into lxz/ec_agg_start_eph_26

a0459bf

Merge branch 'release/2.6' into lxz/ec_agg_start_eph_26

4d4f099

Merge branch 'release/2.6' into lxz/ec_agg_start_eph_26

38b97c7

github-actions Bot added the priority Ticket has high priority (automatically managed) label Nov 26, 2025

liuxuezhao added 2 commits November 27, 2025 07:16

Merge branch 'release/2.6' into lxz/ec_agg_start_eph_26

50a9f59

Merge branch 'release/2.6' into lxz/ec_agg_start_eph_26

f0eed98

gnailzenh approved these changes Nov 28, 2025

View reviewed changes

gnailzenh merged commit bc8d9a2 into release/2.6 Nov 28, 2025
40 of 41 checks passed

gnailzenh deleted the lxz/ec_agg_start_eph_26 branch November 28, 2025 06:19

Conversation

liuxuezhao commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Nov 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NiuYawei Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

liuxuezhao commented Nov 13, 2025

Uh oh!

NiuYawei commented Nov 13, 2025

Uh oh!

mchaarawi commented Nov 13, 2025

Uh oh!

liuxuezhao commented Nov 14, 2025

Uh oh!

daosbuild3 commented Nov 14, 2025

Uh oh!

liuxuezhao commented Nov 14, 2025

Uh oh!

daosbuild3 commented Nov 15, 2025

Uh oh!

daosbuild3 commented Nov 17, 2025

Uh oh!

daosbuild3 commented Nov 21, 2025

Uh oh!

daosbuild3 commented Nov 24, 2025

Uh oh!

daosbuild3 commented Nov 24, 2025

Uh oh!

gnailzenh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

liuxuezhao commented Nov 7, 2025 •

edited

Loading

github-actions Bot commented Nov 7, 2025 •

edited

Loading