DAOS-18597 ddb: enforce SPDK re-init rules by janekmi · Pull Request #17838 · daos-stack/daos

janekmi · 2026-03-30T17:18:21Z

SPDK, in all known applications—including daos_engine—is initialized only once during the lifetime of a process. DDB uses SPDK differently, allowing the user to initialize and re‑initialize SPDK multiple times within the same process.

However, SPDK does not fully support re‑initialization. At the moment, this issue appears only when the SPDK configuration uses the VMD subsystem. When a VMD‑enabled SPDK configuration is in use, two conditions must be respected:

The VMD‑enabled configuration must be used during the first SPDK initialization. The VMD subsystem and DPDK will not be initialized on subsequent SPDK re‑initializations.
After a VMD‑enabled configuration has been used, the user cannot re‑initialize SPDK—whether with VMD enabled or disabled. The internal state of the VMD subsystem and DPDK becomes unsafe to use, even if the next SPDK configuration does not include VMD.

These rules if not adhered to causes DDB to crash.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

SPDK, in all known applications—including daos_engine—is initialized only once during the lifetime of a process. DDB uses SPDK differently, allowing the user to initialize and re‑initialize SPDK multiple times within the same process. However, SPDK does not fully support re‑initialization. At the moment, this issue appears only when the SPDK configuration uses the VMD subsystem. When a VMD‑enabled SPDK configuration is in use, two conditions must be respected: - The VMD‑enabled configuration must be used during the first SPDK initialization. The VMD subsystem and DPDK will not be initialized on subsequent SPDK re‑initializations. - After a VMD‑enabled configuration has been used, the user cannot re‑initialize SPDK—whether with VMD enabled or disabled. The internal state of the VMD subsystem and DPDK becomes unsafe to use, even if the next SPDK configuration does not include VMD. These rules if not adhered to causes DDB to crash. Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

github-actions · 2026-03-30T17:18:57Z

Ticket title is 'ddb dev_list with --vos_path in interactive mode causes segmentation fault'
Status is 'In Progress'
Labels: 'VMD,ddb,scrubbed_2.8,test_2.8'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-18597

Nasf-Fan · 2026-03-31T06:14:48Z

+
+	if (opt->db_path != NULL && strnlen(opt->db_path, PATH_MAX) != 0) {
+		memset(path_parts.vf_db_path, 0, sizeof(path_parts.vf_db_path));
+		strncpy(path_parts.vf_db_path, opt->db_path, sizeof(path_parts.vf_db_path) - 1);


Can we move this section into above vos_path_parse(), then the other similar process in the patch can share that, and nobody will miss db_path.

It is already done here: #17340 by @knard38

I do not want to repeat the effort.

Nasf-Fan · 2026-03-31T06:15:36Z

+	int                 rc;
+
+	rc = json_object_object_get_ex(vmd_subsystem, KEY_SUBSYSTEM_CONFIG, &config);
+	D_ASSERTF(rc == 1, "VMD subsystem does not have a '%s' key.\n", KEY_SUBSYSTEM_CONFIG);


Do not assertion since the input source maybe out of control. Similarly for other places.

Done.

But I still left two cases where, in my opinion, we have all the reason to expect a given json_*() call to succeed. e.g.

methods_num = json_object_array_length(config); for (int i = 0; i < methods_num; i++) { struct json_object *method = json_object_array_get_idx(config, i); D_ASSERT(method != NULL);

The number of objects in the array is already established. If they are there it should be possible to get them one-by-one.

knard38

If I am correct, one call to vmd_wa_can_proceed() is missing ddb_main.c

Are you planning to add some unit tests or do not want to bother with that as it is a short term workaround ?

knard38 · 2026-03-31T13:20:23Z

+is_vmd_enabled(struct json_object *vmd_subsystem)
+{
+	struct json_object *config;
+	struct json_object *method;


NIT, the scope of the variable method can be reduced.

knard38 · 2026-03-31T13:22:10Z

+}
+
+static struct json_object *
+get_vmd_subsystem(struct json_object *subsystems)


NIT, the scope of the variable subsystem and rc can be reduced

knard38 · 2026-03-31T13:30:23Z

+			memset(path_parts.vf_db_path, 0, sizeof(path_parts.vf_db_path));
+			strncpy(path_parts.vf_db_path, pa.pa_db_path,
+				sizeof(path_parts.vf_db_path) - 1);
+		}


Should we also run vmd_wa_can_proceed() to update the status of the global variable.
If we are using a command file, several open and close could be done.

If we are using a command file, several open and close could be done.

If I am not mistaken this is already covered. As far as I can tell the call sequence will go something like: main() -> parseOpts() -> runFileCmds() -> runCmdStr() -> app.RunCommand() -> ...

So, all open and close commands from a command file should go through ddb_run_open() as normal.

Should we also run vmd_wa_can_proceed() to update the status of the global variable.

Done.

+ limit scope for some local variables Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

janekmi · 2026-03-31T15:12:13Z

@knard38 wrote:

Are you planning to add some unit tests or do not want to bother with that as it is a short term workaround ?

Lack of unit tests always come back to bite you and from experience short term workarounds are longer lived than expected. 😅

Let's see how the review and validation will go. I do not want to slow down the release or something and I have a few more things to take care of. If we end up still working on this review few days later it may give me time to write unit tests. If it makes sense.

daosbuild3 · 2026-03-31T17:38:38Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17838/3/testReport/

daosbuild3 · 2026-03-31T17:56:24Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17838/3/testReport/

Nasf-Fan · 2026-04-01T01:37:03Z

+	char               *nvme_conf;
+	int                 rc;
+
+	D_ASPRINTF(nvme_conf, "%s/%s", db_path, VOS_NVME_CONF);


Where will nvme_conf be released?

Great point. Done.

knard38

If I am correct there is some possible memory leak with the json object which needs to be fixed.
I also agree with @Nasf-Fan on missing memory free of the string nvme_conf.

knard38 · 2026-04-01T06:46:58Z

+
+	rc = json_object_object_get_ex(root, KEY_SUBSYSTEMS, &subsystems);
+	if (rc != JSON_TRUE) {
+		ddb_errorf(ctx, "File %s does not have '%s' key\n", nvme_conf, KEY_SUBSYSTEMS);


missing call of json_object_put(root) ?

You are right! Done. Thank you.

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

…DDB-limit-VMD-usage Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

daosbuild3 · 2026-04-01T11:29:29Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17838/4/testReport/

knard38

LGTM

daosbuild3 · 2026-04-01T11:47:16Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17838/4/testReport/

gnailzenh · 2026-04-01T12:45:38Z

it still has failed UT, please check

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

…DDB-limit-VMD-usage Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

knard38

LGTM

daltonbohning · 2026-04-02T14:23:37Z

FYI master is now 3.0 development so, if needed, this will need to be backported to release/2.8

SPDK, in all known applications—including daos_engine—is initialized only once during the lifetime of a process. DDB uses SPDK differently, allowing the user to initialize and re‑initialize SPDK multiple times within the same process. However, SPDK does not fully support re‑initialization. At the moment, this issue appears only when the SPDK configuration uses the VMD subsystem. When a VMD‑enabled SPDK configuration is in use, two conditions must be respected: - The VMD‑enabled configuration must be used during the first SPDK initialization. The VMD subsystem and DPDK will not be initialized on subsequent SPDK re‑initializations. - After a VMD‑enabled configuration has been used, the user cannot re‑initialize SPDK—whether with VMD enabled or disabled. The internal state of the VMD subsystem and DPDK becomes unsafe to use, even if the next SPDK configuration does not include VMD. These rules if not adhered to causes DDB to crash. Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

SPDK, in all known applications—including daos_engine—is initialized only once during the lifetime of a process. DDB uses SPDK differently, allowing the user to initialize and re‑initialize SPDK multiple times within the same process. However, SPDK does not fully support re‑initialization. At the moment, this issue appears only when the SPDK configuration uses the VMD subsystem. When a VMD‑enabled SPDK configuration is in use, two conditions must be respected: - The VMD‑enabled configuration must be used during the first SPDK initialization. The VMD subsystem and DPDK will not be initialized on subsequent SPDK re‑initializations. - After a VMD‑enabled configuration has been used, the user cannot re‑initialize SPDK—whether with VMD enabled or disabled. The internal state of the VMD subsystem and DPDK becomes unsafe to use, even if the next SPDK configuration does not include VMD. These rules if not adhered to causes DDB to crash. Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com> Co-authored-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

DAOS-18597 (PR #17838)'s guard against a crash-prone second in-process SPDK/VOS init only applied when VMD was involved, but the same SPDK re-init limitation affects any NVMe-backed pool. Generalize the guard (ddb_vmd_wa.c/vmd_wa_can_proceed() becomes ddb_spdk_reinit_wa.c/spdk_reinit_wa_can_proceed()) to cover all NVMe-backed pools, and move every call site into ddb_vos.c next to the vos_self_init() call it protects. This also fixes dv_pool_open() checking the guard against the wrong db_path when --db_path was omitted, the common case that reproduced the crash on real hardware. Add a diagnostic-only DAOS_DDB_ALLOW_SPDK_REINIT environment variable (non-release builds) to bypass the guard when validating a newer SPDK/DPDK version. Expand tests to cover the generalized guard and its new call sites, and document the limitation in ddb's README, --help, and man page. Features: recovery Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

DAOS-18597 (PR #17838)'s guard against a crash-prone second in-process SPDK/VOS init only applied when VMD was involved, but the same SPDK re-init limitation affects any NVMe-backed pool. Generalize the guard (ddb_vmd_wa.c/vmd_wa_can_proceed() -> ddb_spdk_reinit_wa.c/ dwa_can_proceed()) to cover all NVMe-backed pools by checking for <db_path>/daos_nvme.conf; pools with no NVMe config stay safe to reuse indefinitely. dwa_can_proceed(ctx, nvme_conf_dir, can_proceed) is now the single entry point for all 7 affected commands, called from ddb_vos.c next to each vos_self_init() it protects (fixing a bug where dv_pool_open() checked the wrong, unresolved db_path). NULL nvme_conf_dir means "always configured" (smd_sync's SPDK config is a separate nvme_conf argument, independent of db_path); the int return propagates a real internal check error separately from can_proceed's actual allow/refuse decision. Add a diagnostic-only DAOS_DDB_ALLOW_SPDK_REINIT environment variable (compiled out of release builds) to bypass the guard when validating a newer SPDK/DPDK version against real hardware. Document the limitation in ddb's README, --help, and man page. Expand ddb_ut coverage: a dedicated ddb_spdk_reinit_wa_ut.c suite (sequencing, the diagnostic override, NULL-argument asserts), and a gurt-style __wrap_dwa_can_proceed mock (ddb_mocks.c, shared by ddb_commands_ut.c) covering every call site's rc/can_proceed propagation. Features: recovery Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

DAOS-18597 (PR #17838)'s guard against a crash-prone second in-process SPDK/VOS init only applied when VMD was involved, but the same SPDK re-init limitation affects any NVMe-backed pool. Generalize the guard (ddb_vmd_wa.c/vmd_wa_can_proceed() -> ddb_spdk_reinit_wa.c/ dwa_can_proceed()) to cover all NVMe-backed pools by checking for <db_path>/daos_nvme.conf; pools with no NVMe config stay safe to reuse indefinitely. dwa_can_proceed(ctx, nvme_conf_dir, can_proceed) is now the single entry point for all 7 affected commands, called from ddb_vos.c next to each vos_self_init() it protects (fixing a bug where dv_pool_open() checked the wrong, unresolved db_path). NULL nvme_conf_dir means "always configured" (smd_sync's SPDK config is a separate nvme_conf argument, independent of db_path); the int return propagates a real internal check error separately from can_proceed's actual allow/refuse decision. Add a diagnostic-only DAOS_DDB_ALLOW_SPDK_REINIT environment variable (compiled out of release builds) to bypass the guard when validating a newer SPDK/DPDK version against real hardware. Document the limitation in ddb's README, --help, and man page. Expand ddb_ut coverage: a dedicated ddb_spdk_reinit_wa_ut.c suite (sequencing, the diagnostic override, NULL-argument asserts), and a gurt-style __wrap_dwa_can_proceed mock (ddb_mocks.c, shared by ddb_commands_ut.c) covering every call site's rc/can_proceed propagation, including feature_uses_derived_db_path_test, which proves ddb_run_feature() checks the guard against the resolved db_path the same way ddb_run_open() does. Features: recovery Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

DAOS-18597 (PR #17838) guarded against a crash-prone second in-process SPDK/VOS init, but only when VMD was involved. Generalize the guard to cover any NVMe-backed pool: ddb_vmd_wa.c/vmd_wa_can_proceed() becomes ddb_spdk_reinit_wa.c/dwa_can_proceed(), which checks for <db_path>/daos_nvme.conf; pools with no NVMe config stay safe to reuse indefinitely, and a NULL nvme_conf_dir always allows the operation (smd_sync's SPDK config comes from a separate nvme_conf argument, independent of db_path). dwa_can_proceed() is now called from the ddb_vos.c dv_*() VOS API functions right before each vos_self_init()/smd_init() call it protects, ahead of the real init in every case (fixing a bug where dv_pool_open() previously checked an unresolved db_path). The int return propagates internal check errors separately from the can_proceed allow/refuse decision. Add a diagnostic-only DAOS_DDB_ALLOW_SPDK_REINIT environment variable (compiled out of release builds) to bypass the guard when validating a newer SPDK/DPDK version against real hardware, and document the limitation in ddb's README, --help, and man page. Rename parse_vos_file_parts() to ddb_parse_vos_file_parts() for consistency with the rest of the ddb function names, and wrap it in unit tests. Add ddb_spdk_reinit_wa_ut.c and extend ddb_vos_ut.c to exercise the guard directly through the dv_*() VOS API for dv_pool_open/destroy, dev_list, dev_replace, prov_mem, and smd_sync, backed by new __wrap_dwa_can_proceed, __wrap_ddb_parse_vos_file_parts, and __wrap_d_asprintf2 mocks (ddb_mocks.c/.h, shared with ddb_commands_ut.c), plus a ddb_fake_print.c/.h test helper. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

janekmi requested review from Nasf-Fan, NiuYawei, gnailzenh and knard38 March 30, 2026 17:18

janekmi requested review from a team as code owners March 30, 2026 17:18

github-actions Bot added the priority Ticket has high priority (automatically managed) label Mar 30, 2026

Nasf-Fan reviewed Mar 31, 2026

View reviewed changes

Nasf-Fan self-requested a review March 31, 2026 06:17

knard38 requested changes Mar 31, 2026

View reviewed changes

janekmi added 2 commits March 31, 2026 14:50

DAOS-18597 ddb: handle gently malformed/unexpected daos_nvme.conf

630ef8f

+ limit scope for some local variables Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

DAOS-18597 ddb: add a missing vmd_wa_can_proceed() call

e3e18d5

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

janekmi requested a review from knard38 March 31, 2026 15:16

daltonbohning added the approved-to-merge PR has received release branch merge approval label Mar 31, 2026

Nasf-Fan reviewed Apr 1, 2026

View reviewed changes

Nasf-Fan self-requested a review April 1, 2026 01:40

knard38 requested changes Apr 1, 2026

View reviewed changes

janekmi added 2 commits April 1, 2026 10:37

DAOS-18597 ddb: add missing D_FREE and json_object_put()

a13ab1d

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

Merge remote-tracking branch 'origin/master' into janekmi/DAOS-18597-…

5e2cf74

…DDB-limit-VMD-usage Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

janekmi requested a review from knard38 April 1, 2026 10:44

Nasf-Fan previously approved these changes Apr 1, 2026

View reviewed changes

knard38 previously approved these changes Apr 1, 2026

View reviewed changes

DAOS-18597 ddb: handle the case where the SPDK conf file does not exist

dac3ba1

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

janekmi dismissed stale reviews from knard38 and Nasf-Fan via dac3ba1 April 1, 2026 13:42

janekmi requested review from Nasf-Fan and knard38 April 1, 2026 13:42

Nasf-Fan previously approved these changes Apr 1, 2026

View reviewed changes

knard38 previously approved these changes Apr 1, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into janekmi/DAOS-18597-…

f4e5a04

…DDB-limit-VMD-usage Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

janekmi dismissed stale reviews from knard38 and Nasf-Fan via f4e5a04 April 1, 2026 15:03

DAOS-18597 ddb: restore mistakenly removed error message

79e943e

Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>

janekmi requested review from Nasf-Fan and knard38 April 1, 2026 15:19

knard38 approved these changes Apr 1, 2026

View reviewed changes

Nasf-Fan approved these changes Apr 1, 2026

View reviewed changes

janekmi requested a review from a team April 2, 2026 11:23

daltonbohning removed the approved-to-merge PR has received release branch merge approval label Apr 2, 2026

daltonbohning merged commit ed2bc08 into master Apr 2, 2026
42 checks passed

daltonbohning deleted the janekmi/DAOS-18597-DDB-limit-VMD-usage branch April 2, 2026 14:22

janekmi restored the janekmi/DAOS-18597-DDB-limit-VMD-usage branch April 4, 2026 14:45

janekmi deleted the janekmi/DAOS-18597-DDB-limit-VMD-usage branch April 4, 2026 14:46

knard38 mentioned this pull request Aug 31, 2026

DAOS-19513 ddb: extend SPDK re-init guard to all NVMe devices #18962

Merged

2 of 6 tasks

Conversation

janekmi commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Mar 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knard38 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janekmi Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janekmi commented Mar 31, 2026

Uh oh!

daosbuild3 commented Mar 31, 2026

Uh oh!

daosbuild3 commented Mar 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Apr 1, 2026

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Apr 1, 2026

Uh oh!

gnailzenh commented Apr 1, 2026

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daltonbohning commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

janekmi commented Mar 30, 2026 •

edited

Loading

knard38 left a comment •

edited

Loading

janekmi Mar 31, 2026 •

edited

Loading