build(cco): compile the SDMA path by default

[jhchouuu]

BUILD_CCO_SDMA was OFF unless BUILD_BENCHMARK=ON turned it on. What OFF produces is a build whose SDMA puts compile away: every symbol is still there, every kernel still launches, every put still returns, and no bytes move. The all-reduce then returns mostly the local slice.

GemmAllReduceOp.self_test exists because of exactly one such run. It measured faster than the real thing (-6.8% against -2.3%) precisely because it was not moving data, and the only signal that caught it was perplexity: 862511 against 3.26.

What it costs

Same tree, clean build each side:

	wall	libmori_cco.so
OFF	27s	1,701,544 B
ON	26s	1,707,632 B

+6,088 B (+0.36%), no measurable build time.

It adds no dependency. anvil.cpp is in MORI_APP_SOURCES unconditionally and find_package(hsakmt REQUIRED) is too, so the host transport and its link deps were already there either way. The flag reaches only cco's own layer: the queue setup in cco_init.cpp, the cco_sdma_* wrapper symbols (58 lines of 316), the device API in cco.hpp, and one extra JIT bitcode variant that is cached per config.

Why it is safe

It creates nothing on its own. ccoSdmaSetupCommQueues returns before any allocation unless MORI_ENABLE_SDMA is set, and that runtime gate is independent and still defaults off.

Verified on the new default build, two ranks:

	MORI_ENABLE_SDMA	sdmaQueueCount	result
A	unset	0 (default)	comm builds, no queues, nothing logged
B	unset	8	comm builds, no queues, diagnosed
C	1	8	comm builds, queues created

So for anyone not opting in this changes nothing; for anyone who does, it is the difference between working and silently inert.

CI already built with BUILD_CCO_SDMA=ON in both ci_cco.yml and ci.yml, so ON was already the configuration under test and OFF was the less-covered one.

The diagnostic

Case B above used to be silent too. When a caller explicitly asks for queues (sdmaQueueCount > 0; 0 means "whatever the env wants") and none will be created, cco_init.cpp now says which of the two reasons applies:

[shmem] [error] sdmaQueueCount=8 requested but no SDMA queues will be created
(MORI_ENABLE_SDMA is not set): every ccoSdma put on this comm will move no bytes

At ERROR level rather than WARN because the default global level is ERROR — a warning here would be invisible to exactly the person it is for. Not fatal: a comm without SDMA is legal and every other path still works. Silent for the default sdmaQueueCount=0, so a comm that never wanted SDMA logs nothing.

Verification

On MI355X, from this branch:

• plain pip install . bakes BUILD_CCO_SDMA=True; BUILD_CCO_SDMA=OFF still builds and bakes False — the escape hatch works
• tests/python/cco/test_sdma_api.py — 12 passed
• test_rocm_bootstrap.py + test_rocm_import_order.py with MORI_ENABLE_SDMA unset — 9 passed
• the three runtime-gate cases above

One note on the box used: an unrelated container was holding SDMA engines part of the time, and sdma_queue_count=8 failed with ERROR code: 6 at anvil.cpp:237 while 1 and 2 succeeded. That is the contention kernels_sdma.py already documents ("56 queues to use 7"), and 6 is HSAKMT_STATUS_NO_MEMORY — a process-memory condition, not queue exhaustion (see #685). It cleared on its own and case C passes above; this change does not touch queue creation.

Docs

Updated the two places that called OFF "the default" and the build instructions that told users to pass the flag by hand. The BUILD_CCO_SDMA=ON requirement is unchanged everywhere — only the need to type it.

🤖 Generated with Claude Code