DAOS-19513 ddb: extend SPDK re-init guard to all NVMe devices

[knard38]

Extends DAOS-18597's guard (introduced in #17838) against ddb's SPDK re-initialization crash so that it covers every NVMe-backed pool, not just pools using the VMD subsystem.

Problem

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Why extend the guard rather than fix SPDK or add a CLI-level guard

A real fix (making SPDK's spdk_env_init() reinit path rebuild mem_map/vtophys instead of assuming they're still valid) would be a new SPDK capability that doesn't exist today — the pieces it needs are internal-only APIs, so it would have to be designed and accepted upstream with no committed timeline. A blanket CLI-level "one lifecycle command per process" counter was also considered, but it's less precise than a per-pool-init guard: for example feature can safely reuse an already-open pool without a new init, something a command counter can't distinguish from a genuine second init. Generalizing the existing per-process "used once" guard to all NVMe-backed pools is the minimal, low-risk option with no upstream dependency.

Bugs found and fixed while generalizing the guard

Moving the guard's call sites next to each vos_self_init() surfaced two pre-existing bugs in how the old, VMD-only guard was invoked. Both are fixed by this PR:

1. open (and feature) checked the wrong, unresolved db_path. Both commands take an optional -p/--db_path that, when omitted — the normal way to invoke either command — defaults to an empty string and is only derived from the pool path argument inside dv_pool_open()/parse_vos_file_parts(). The old guard instead ran in ddb_commands.c before that derivation, against the raw opt->db_path directly. With no --db_path given, that's NULL/empty, so vmd_subsystem_required()'s <dir>/daos_nvme.conf lookup built an unusable path (e.g. "(null)/daos_nvme.conf") that can never exist, and access() always failed — the guard silently assumed "no VMD/NVMe" regardless of the pool's actual, derived configuration. In other words, the guard was effectively bypassed whenever open/feature were used the normal way (no explicit --db_path override). Fixed by moving the check inside dv_pool_open(), after path resolution, so it always evaluates the real, resolved vf_db_path.

   Example: run open <pool-a>/vos-0 then, later in the same interactive/-f session, open <pool-b>/vos-0 — two different NVMe-backed pools, neither open call passing --db_path. Both calls hit the bogus, nonexistent <dir>/daos_nvme.conf path described above, so both are reported as "no VMD/NVMe" and allowed unconditionally. vos_self_init() therefore runs a second time, in the same process, for a second NVMe-backed pool — exactly the unsafe re-init this guard exists to prevent — and ddb crashes. This is hit through completely ordinary, default-argument use of open, not some obscure edge case.

   Introduced by DAOS-18304 ddb: Introduce DdbAPI interface and fix db_path propagation #18124.

   Test coverage: tests/ddb_commands_ut.c::open_uses_derived_db_path_test and the new feature_uses_derived_db_path_test lock this in for open/feature respectively — both leave opt.db_path NULL and assert, via the __wrap_dwa_can_proceed mock, that the guard is still called with the path derived from opt.path ("/mnt/daos2"), not the omitted raw value. rm_pool_uses_derived_db_path_test proves the same property for dv_pool_destroy()/rm_pool (not itself buggy pre-fix, but sharing the same derivation code). feature_uses_derived_db_path_test specifically targets the case where feature opens the pool itself (ctx.dc_poh not already valid, i.e. ddb_pool_is_open(ctx) is false); when a pool is already open in the session (e.g. after a preceding open), ddb_run_feature reuses it via a goto skip shortcut and never reaches dv_pool_open()/the guard at all, so that reuse path isn't (and can't usefully be) exercised by this kind of test.

2. smd_sync checked the wrong file entirely. The old guard also probed <db_path>/daos_nvme.conf for smd_sync, but smd_sync's SPDK/NVMe configuration always comes from its separate nvme_conf argument — it has no relationship to db_path (used only for the unrelated SMD/VOS system DB). So the old guard's VMD/NVMe determination for smd_sync was based on the wrong file and could come out wrong in either direction. Fixed by having smd_sync call dwa_can_proceed(ctx, NULL, ...), which unconditionally treats it as NVMe/SPDK-backed instead of probing db_path.

   Example: run smd_sync --nvme_conf /path/to/nvme.conf --db_path /mnt/daos0 where /mnt/daos0 has no daos_nvme.conf of its own (or one that doesn't match what /path/to/nvme.conf actually configures), then open <nvme-backed-pool>/vos-0 in the same session. The old guard checks /mnt/daos0/daos_nvme.conf for the smd_sync call, finds nothing relevant, and reports "no NVMe here" even though smd_sync really does initialize SPDK from /path/to/nvme.conf. The following open then sees an NVMe-backed pool for what it thinks is "the first time" and proceeds — a real second SPDK init in the same process that the guard should have refused.

   Introduced by DAOS-18597 ddb: enforce SPDK re-init rules #17838.

   Test coverage: tests/ddb_commands_ut.c::smd_sync_checks_guard_unconditionally_test deliberately sets mismatched opt.nvme_conf = "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/configs/engine.json" and opt.db_path = "/mnt/daos_without_a_daos_nvme_conf", then asserts via the mock (expect_value(dwa_can_proceed_mock, nvme_conf_dir, 0)) that the guard is still called with nvme_conf_dir == NULL — i.e. that it's checked unconditionally rather than against the (irrelevant) db_path. vos_self_init_ext()/smd_init() are mocked too, since the guard sits after they succeed.

Changes

ddb_spdk_reinit_wa.c / .h (replaces ddb_vmd_wa.c)

vmd_wa_can_proceed() is replaced by dwa_can_proceed(ctx, nvme_conf_dir, can_proceed), the single entry point for the guard:

• If nvme_conf_dir is non-NULL and <nvme_conf_dir>/daos_nvme.conf doesn't exist, the operation is PMEM-only and always allowed.
• Otherwise (including nvme_conf_dir == NULL, meaning "always treat as NVMe-backed") a process-lifetime nvme_used_once flag gates the decision: the first NVMe-backed operation is allowed, later ones are refused with an actionable "restart ddb" message reported via ctx.
• The function's int return is a real internal error (e.g. -DER_NOMEM from the daos_nvme.conf check), kept separate from can_proceed's allow/refuse decision so callers don't conflate the two.
• A diagnostic-only DAOS_DDB_ALLOW_SPDK_REINIT environment variable unconditionally bypasses the guard. It's compiled out of release builds (DAOS_BUILD_RELEASE) and is meant only for developers re-validating a newer SPDK/DPDK version against real hardware — not a supported workflow.

ddb_vos.c

The guard call moves next to each vos_self_init()/vos_self_init_ext() it protects, instead of living in the caller:

• dv_pool_open() (open, feature) and dv_pool_destroy() (rm_pool) check the resolved vf_db_path from create_vos_file_parts() — this is bug fix Adding a .travis.yml file to trigger CI builds. #1 above.
• dv_sync_smd() (smd_sync) calls the guard with nvme_conf_dir == NULL — this is bug fix Update README.md #2 above.
• dv_dev_list() (dev_list), dv_dev_replace() (dev_replace), and dv_run_prov_mem() (prov_mem) each gain a new struct ddb_ctx *ctx parameter and check db_path directly.

ddb_commands.c

Removes the old DDB_CAN_PROCEED/vmd_wa_can_proceed() forward-declaration and macro. Call sites now simply forward ctx into the dv_* VOS API instead of checking before calling it.

Documentation

Documents the limitation in three places: a new "Limitations" section in src/utils/ddb/README.md, a short paragraph in ddb's --help/interactive help text (main.go), and a new "SPDK RE-INITIALIZATION" section in the generated man page (manpage.go), including the special-cased smd_sync behavior.

Tests

• New tests/ddb_spdk_reinit_wa_ut.c suite covering the guard's state machine (sequencing of allow/refuse decisions), the diagnostic override, and NULL-argument asserts.
• New tests/ddb_mocks.c/.h with a gurt-style __wrap_dwa_can_proceed mock, shared by ddb_commands_ut.c, covering every call site's rc/can_proceed propagation, including the new feature_uses_derived_db_path_test (see bug Adding a .travis.yml file to trigger CI builds. #1 above).
• tests/ddb_vos_tests.c and tests/ddb_commands_tests.c updated for the new ctx parameter on the dv_* API.
• ddb_commands_test.go extends the Go man-page ordered-content assertion to include the new SPDK re-init section.

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'ddb: SPDK re-initialization fails for standard (non-VMD) NVMe devices within a single ddb process'
Status is 'In Review'
Labels: 'ddb'
https://daosio.atlassian.net/browse/DAOS-19513

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18962/2/execution/node/1781/log

[tanabarr]

LGTM but we probably want a review from someone else who works more regularly in ddb. Perhaps @janekmi ?

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18962/6/execution/node/1813/log

[knard38]

@janekmi , your concerns should have been fixed with the commit 1a396b4
Thanks for your in depth review allowing to noticeably improve my PR :-)

[janekmi]

Please note my reply in this thread as well:

• #18962 (comment)

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18962/7/execution/node/1770/log

[Nasf-Fan]

Why need to overwrite the old one before re-assign it as new fn?

[knard38]

if mock_fn is NULL, the default value should be equal to the real dwa_can_proceed() function.

[Nasf-Fan]

Why someone return -DER_SUCCESS, but some others return -DER_SUCCESS, such as L45? Can we unify them?

[knard38]

Oups error from my side

• Replace DER_SUCCESS with -DER_SUCCESS

[knard38]

Last concerns from @janekmi and @Nasf-Fan should have been fixed with commit 7825e14

[daosbuild3]

Test stage Test RPMs on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18962/9/execution/node/1251/log

[tanabarr]

This looks like a good improvement, maybe @NiuYawei could review the src/utils/ddb/ddb_spdk_reinit_wa.c and following changes as listed in the github UI review page to check that this seems sound as it's an important change in terms of data integrity.

[NiuYawei]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

[knard38]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

It depends of the commands.
As indicated in the README or the man page, ddb is offering two types of commands: Pool-content commands and pool-lifecycle commands.

• Pool-content commands which operates on the content of an already-open VOS pool and thus do not reintialize SPDK.
• Pool-lifecycle commands (open, close, feature, rm_pool, etc.) which operate on the pool itself and thus need to reinitialize SPDK.

This patch is thus dedicated to manage this second type of commands.

Doest it makes sense ?

[NiuYawei]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

It depends of the commands. As indicated in the README or the man page, ddb is offering two types of commands: Pool-content commands and pool-lifecycle commands.

* **Pool-content** commands which operates on the content of an already-open VOS pool and thus do not reintialize SPDK.

* **Pool-lifecycle** commands (open, close, feature, rm_pool, etc.) which operate on the pool itself and thus need to reinitialize SPDK.

This patch is thus dedicated to manage this second type of commands.

Doest it makes sense ?

Why the second type of commands need to reinitialize SPDK? DAOS engine also offers this type of function (open, close, pool destroy, etc.) without reinitializing SPDK, right?

[knard38]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

It depends of the commands. As indicated in the README or the man page, ddb is offering two types of commands: Pool-content commands and pool-lifecycle commands.

* **Pool-content** commands which operates on the content of an already-open VOS pool and thus do not reintialize SPDK.

* **Pool-lifecycle** commands (open, close, feature, rm_pool, etc.) which operate on the pool itself and thus need to reinitialize SPDK.

This patch is thus dedicated to manage this second type of commands.
Doest it makes sense ?

Why the second type of commands need to reinitialize SPDK? DAOS engine also offers this type of function (open, close, pool destroy, etc.) without reinitializing SPDK, right?

TBH, I'm not the original author of this part, but the reasoning seems to be architectural: unlike daos_engine, which is started once against one fixed set of NVMe/SCM targets for its whole lifetime, a single ddb process isn't tied to one storage config. In an interactive or -f session, pool-lifecycle commands can each target a different pool/db_path back-to-back (rm_pool, dev_list, dev_replace, prov_mem, and smd_sync all take their own path/db_path argument, independent of whatever open last opened). ddb uses VOS's "standalone" init (vos_self_init()/vos_self_fini()), a mode the engine never uses at all - the engine has its own xstream-based init tied to its one fixed config.

So each command claims the SPDK/NVMe environment its own target needs, then releases it, instead of assuming the whole session only ever touches the config initialized first. rm_pool is a clear example: it requires the pool to already be closed, so dv_pool_destroy() has no open handle to reuse and must init/fini VOS itself. dev_list, dev_replace, prov_mem, and smd_sync are the same. feature, on the other hand, skips re-init when a pool is already open and it's targeting that same pool - showing the real rule is "does this call need a possibly-different pool," not "this command class always re-inits."

[NiuYawei]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

It depends of the commands. As indicated in the README or the man page, ddb is offering two types of commands: Pool-content commands and pool-lifecycle commands.

* **Pool-content** commands which operates on the content of an already-open VOS pool and thus do not reintialize SPDK.

* **Pool-lifecycle** commands (open, close, feature, rm_pool, etc.) which operate on the pool itself and thus need to reinitialize SPDK.

This patch is thus dedicated to manage this second type of commands.
Doest it makes sense ?

Why the second type of commands need to reinitialize SPDK? DAOS engine also offers this type of function (open, close, pool destroy, etc.) without reinitializing SPDK, right?

TBH, I'm not the original author of this part, but the reasoning seems to be architectural: unlike daos_engine, which is started once against one fixed set of NVMe/SCM targets for its whole lifetime, a single ddb process isn't tied to one storage config. In an interactive or -f session, pool-lifecycle commands can each target a different pool/db_path back-to-back (rm_pool, dev_list, dev_replace, prov_mem, and smd_sync all take their own path/db_path argument, independent of whatever open last opened). ddb uses VOS's "standalone" init (vos_self_init()/vos_self_fini()), a mode the engine never uses at all - the engine has its own xstream-based init tied to its one fixed config.

So each command claims the SPDK/NVMe environment its own target needs, then releases it, instead of assuming the whole session only ever touches the config initialized first. rm_pool is a clear example: it requires the pool to already be closed, so dv_pool_destroy() has no open handle to reuse and must init/fini VOS itself. dev_list, dev_replace, prov_mem, and smd_sync are the same. feature, on the other hand, skips re-init when a pool is already open and it's targeting that same pool - showing the real rule is "does this call need a possibly-different pool," not "this command class always re-inits."

Thanks for clarifying. Implementing ddb to re-initialize SPDK per command appears to be a design oversight, unless we explicitly need to switch daos_nvme.conf configurations dynamically during a single ddb process run.

[knard38]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

It depends of the commands. As indicated in the README or the man page, ddb is offering two types of commands: Pool-content commands and pool-lifecycle commands.

* **Pool-content** commands which operates on the content of an already-open VOS pool and thus do not reintialize SPDK.

* **Pool-lifecycle** commands (open, close, feature, rm_pool, etc.) which operate on the pool itself and thus need to reinitialize SPDK.

This patch is thus dedicated to manage this second type of commands.
Doest it makes sense ?

Why the second type of commands need to reinitialize SPDK? DAOS engine also offers this type of function (open, close, pool destroy, etc.) without reinitializing SPDK, right?

TBH, I'm not the original author of this part, but the reasoning seems to be architectural: unlike daos_engine, which is started once against one fixed set of NVMe/SCM targets for its whole lifetime, a single ddb process isn't tied to one storage config. In an interactive or -f session, pool-lifecycle commands can each target a different pool/db_path back-to-back (rm_pool, dev_list, dev_replace, prov_mem, and smd_sync all take their own path/db_path argument, independent of whatever open last opened). ddb uses VOS's "standalone" init (vos_self_init()/vos_self_fini()), a mode the engine never uses at all - the engine has its own xstream-based init tied to its one fixed config.
So each command claims the SPDK/NVMe environment its own target needs, then releases it, instead of assuming the whole session only ever touches the config initialized first. rm_pool is a clear example: it requires the pool to already be closed, so dv_pool_destroy() has no open handle to reuse and must init/fini VOS itself. dev_list, dev_replace, prov_mem, and smd_sync are the same. feature, on the other hand, skips re-init when a pool is already open and it's targeting that same pool - showing the real rule is "does this call need a possibly-different pool," not "this command class always re-inits."

Thanks for clarifying. Implementing ddb to re-initialize SPDK per command appears to be a design oversight, unless we explicitly need to switch daos_nvme.conf configurations dynamically during a single ddb process run.

That's a fair point. Whether ddb genuinely needs to switch daos_nvme.conf configurations mid-session, or should instead avoid re-initializing SPDK at all when the config hasn't changed (closer to the engine's model), is a relevant architectural question worth investigating on its own. I looked at the code and even a plain close unconditionally tears SPDK down today, so nothing is currently reused even when the very next command targets the exact same db_path.

I'll open a new JIRA ticket to track that investigation, since it's out of scope for this PR.

Thanks for pushing as it could noticeably help improve the ddb user experience.

[knard38]

Unlike daos_engine, which initializes SPDK only once during the lifetime of the process, ddb allows several pool-lifecycle commands (open, rm_pool, feature, dev_list, dev_replace, prov_mem, smd_sync) to run within the same interactive or -f command-file session, each potentially performing its own vos_self_init()/SPDK initialization. SPDK does not fully support being re-initialized within the same process: its re-init path only rescans the PCI bus and never rebuilds the DPDK memory/address-translation tables torn down by the previous vos_self_fini(). This is unsafe for any pool backed by an NVMe device (i.e. whose db_path has a daos_nvme.conf), whether or not VMD is involved, but the existing guard only checked for VMD, so a second in-process init against a plain (non-VMD) NVMe device could still crash ddb.

Is there a specific reason we initialize SPDK per ddb command? Initializing it once on startup (like the DAOS engine) seems more efficient.

It depends of the commands. As indicated in the README or the man page, ddb is offering two types of commands: Pool-content commands and pool-lifecycle commands.

* **Pool-content** commands which operates on the content of an already-open VOS pool and thus do not reintialize SPDK.

* **Pool-lifecycle** commands (open, close, feature, rm_pool, etc.) which operate on the pool itself and thus need to reinitialize SPDK.

This patch is thus dedicated to manage this second type of commands.
Doest it makes sense ?

Why the second type of commands need to reinitialize SPDK? DAOS engine also offers this type of function (open, close, pool destroy, etc.) without reinitializing SPDK, right?

TBH, I'm not the original author of this part, but the reasoning seems to be architectural: unlike daos_engine, which is started once against one fixed set of NVMe/SCM targets for its whole lifetime, a single ddb process isn't tied to one storage config. In an interactive or -f session, pool-lifecycle commands can each target a different pool/db_path back-to-back (rm_pool, dev_list, dev_replace, prov_mem, and smd_sync all take their own path/db_path argument, independent of whatever open last opened). ddb uses VOS's "standalone" init (vos_self_init()/vos_self_fini()), a mode the engine never uses at all - the engine has its own xstream-based init tied to its one fixed config.
So each command claims the SPDK/NVMe environment its own target needs, then releases it, instead of assuming the whole session only ever touches the config initialized first. rm_pool is a clear example: it requires the pool to already be closed, so dv_pool_destroy() has no open handle to reuse and must init/fini VOS itself. dev_list, dev_replace, prov_mem, and smd_sync are the same. feature, on the other hand, skips re-init when a pool is already open and it's targeting that same pool - showing the real rule is "does this call need a possibly-different pool," not "this command class always re-inits."

Thanks for clarifying. Implementing ddb to re-initialize SPDK per command appears to be a design oversight, unless we explicitly need to switch daos_nvme.conf configurations dynamically during a single ddb process run.

That's a fair point. Whether ddb genuinely needs to switch daos_nvme.conf configurations mid-session, or should instead avoid re-initializing SPDK at all when the config hasn't changed (closer to the engine's model), is a relevant architectural question worth investigating on its own. I looked at the code and even a plain close unconditionally tears SPDK down today, so nothing is currently reused even when the very next command targets the exact same db_path.

I'll open a new JIRA ticket to track that investigation, since it's out of scope for this PR.

Thanks for pushing as it could noticeably help improve the ddb user experience.

Created followup ticket DAOS-19628

[knard38]

@daos-stack/daos-gatekeeper could you lend this PR with the following message:

• title: DAOS-19513 ddb: extend SPDK re-init guard to all NVMe devices
• body:

DAOS-18597 (PR #17838) guarded against a crash-prone second in-process
SPDK/VOS init, but only when VMD was involved. Generalize the guard to
cover any NVMe-backed pool: ddb_vmd_wa.c/vmd_wa_can_proceed() becomes
ddb_spdk_reinit_wa.c/dwa_can_proceed(), which checks for
<db_path>/daos_nvme.conf; pools with no NVMe config stay safe to reuse
indefinitely, and a NULL nvme_conf_dir always allows the operation
(smd_sync's SPDK config comes from a separate nvme_conf argument,
independent of db_path).

dwa_can_proceed() is now called from the ddb_vos.c dv_*() VOS API
functions right before each vos_self_init()/smd_init() call it
protects, ahead of the real init in every case (fixing a bug where
dv_pool_open() previously checked an unresolved db_path). The int
return propagates internal check errors separately from the
can_proceed allow/refuse decision.

Add a diagnostic-only DAOS_DDB_ALLOW_SPDK_REINIT environment variable
(compiled out of release builds) to bypass the guard when validating a
newer SPDK/DPDK version against real hardware, and document the
limitation in ddb's README, --help, and man page.

Rename parse_vos_file_parts() to ddb_parse_vos_file_parts() for
consistency with the rest of the ddb function names, and wrap it in
unit tests. Add ddb_spdk_reinit_wa_ut.c and extend ddb_vos_ut.c to
exercise the guard directly through the dv_*() VOS API
for dv_pool_open/destroy, dev_list, dev_replace, prov_mem, and
smd_sync, backed by new __wrap_dwa_can_proceed,
__wrap_ddb_parse_vos_file_parts, and __wrap_d_asprintf2 mocks
(ddb_mocks.c/.h, shared with ddb_commands_ut.c), plus a
ddb_fake_print.c/.h test helper.