fix: correct MiniMax H3 reference audio encoding

[jk212h20]

Summary

Fix MiniMax-H3 stereo reference-audio encoding so the C++ audio VAE matches the official PyTorch implementation.

The current encoder reshapes stereo into a batched conv1d input. On the tested GGML Metal backend, convolution output storage interleaves the stream dimension with feature channels, while later layers read each stream's feature channels as contiguous. This corrupts the reference latent before Ref2VA conditioning.

This change:

• keeps the prepared stereo waveform planar;
• runs the mono audio encoder independently for each stereo stream;
• concatenates the completed normalized stream latents;
• registers and applies the checkpoint's zero_k_bias in causal attention.

Fixes #1882.

Verification

Tested on Apple M4 Max with the Metal backend and minimax_h3_audio_vae_fp32.safetensors.

Cross-decoding isolated the failure to C++ encoding:

• PyTorch encode -> C++ decode: intelligible speech
• original C++ encode -> PyTorch decode: buzzing
• original C++ encode -> C++ decode: buzzing
• fixed C++ encode -> C++ decode: intelligible speech

For the same 3.04-second stereo WAV, fixed C++ versus PyTorch normalized latents:

• shape: 122 x 2 x 32 x 1
• correlation: 0.9999837
• mean absolute error: 0.0004343
• C++ standard deviation: 0.4534302
• PyTorch standard deviation: 0.4534051

The original latent correlation was approximately -0.066.

A clean checkout at de298c2 builds successfully with:

cmake -B build -DSD_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 8 --target sd-cli

A clean-checkout 4-step Ref2VA smoke test also completed successfully, although its generated output was music and is not used as speech-quality evidence. The latent parity and cross-decoding results above came from an instrumented build of the same encoder implementation. Separate longer repaired-input tests produced intelligible, input-dependent generated speech. H3 still generates a new target soundtrack; this patch does not claim waveform copying or exact transcript preservation.

Scope

This is focused on MiniMax-H3 reference-audio encoding. T2AV primarily exercises the audio decoder and therefore does not expose this encoder defect.

AI assistance

AI tools assisted with investigation and patch preparation. I reviewed the changed lines and validated the implementation against the official PyTorch encoder, cross-decoding, a clean build, and end-to-end runs.

[stduhpf]

Same settings and assets as in #1882 repro:

output.mp4

Voice matches the reference now at least so it's much better than before, but the speech is still incoherent, which I believe should not happen if the model is asked to reuse the audio exactly...

[jk212h20]

Thanks for testing this. I downloaded the result and compared it with the original reference from #1882.

Whisper Large-v3 transcribes the reference as:

This is a simple sentence to test lip sync with audio reference mode.

It transcribes the new generated result approximately as:

and do the use and rescission to the volume of your tigris generation.

So I agree that the generated transcript is still incoherent. The fact that the voice now matches the reference is useful confirmation that the repaired reference latent is reaching Ref2VA; the remaining transcript behavior is separate from the stereo encoder corruption fixed here.

At present, full_copy / fully_copy is prompt text rather than an implemented media-copy operation in stable-diffusion.cpp. The backend encodes the reference WAV as conditioning, jointly generates a new video/audio latent, and decodes a newly generated soundtrack. It does not copy the source waveform into the output container. Therefore exact audio reuse currently requires muxing the original WAV after generation.

For generated speech, including the literal sentence in the text prompt should improve semantic adherence, for example:

The character says clearly, “This is a simple sentence to test lip sync with audio reference mode.”

Using the prompt guide's fully_copy spelling and complete reference-prompt structure may also help, but neither spelling invokes literal waveform copying in the backend.

I think H3's transcript adherence can still be investigated separately. This PR is intentionally narrower: it restores the C++ audio encoder to near-numerical parity with the official PyTorch encoder and fixes the corrupted reference conditioning. It does not implement deterministic soundtrack copying or guarantee exact generated speech.

[stduhpf]

Even if I edit the prompt file to replace my typo "<Audio 1>: full_copy" to the exact example syntax from the official prompting guide: "<Audio 1>: fully_copy - <Audio 1> is reused 1:1 as the target video's complete final audio track." , the output is still the same kind of gibberish, with the correct voice timbre.

If I add <d>[english]This is a simple sentence to test lip sync with audio reference mode.</d>, then the result is extremely close to the original audio, down to the the pacing and intonation, but this should not be necessary in fully_copy mode.

I'm thinking maybe there is an issue either with the way the reference audio latents are ordered or with their positionnal encodings, but i cannot find what exactly.

[stduhpf]

i found something:

diff --git a/src/model/diffusion/minimax_h3.hpp b/src/model/diffusion/minimax_h3.hpp
index d0683166..c9175769 100644
--- a/src/model/diffusion/minimax_h3.hpp
+++ b/src/model/diffusion/minimax_h3.hpp
@@ -835,6 +835,18 @@ namespace MiniMaxH3 {
             }
         };
 
+        auto append_condition_audio_positions = [&](int64_t length,
+                                                    float cursor,
+                                                    float w_low,
+                                                    float w_high) {
+            for (int64_t t = 0; t < length; ++t) {
+                for (int channel = 0; channel < 2; ++channel) {
+                    float w = channel == 0 ? w_low : w_high;
+                    append_position(cursor + static_cast<float>(t) * 2.0f, 0.f, w);
+                }
+            }
+        };
+
         float cursor = static_cast<float>(text_len);
         if (reference_blocks.empty()) {
             float video_duration = 0.f;
@@ -888,7 +900,7 @@ namespace MiniMaxH3 {
                         w_high    = axes.second.back();
                     }
                     int64_t count = ref_audio->shape()[0] * 2;
-                    append_audio_positions(ref_audio->shape()[0], cursor, w_low, w_high);
+                    append_condition_audio_positions(ref_audio->shape()[0], cursor, w_low, w_high);
                     layout.sequence_segments.push_back({row,
                                                         row + count,
                                                         SequenceKind::CONDITION_AUDIO,

output-dirtyfix.mp4

Of course this is most likely not the correct way to fix this issue, but that's a clue to whatever is going on. Maybe w_high is not correct for audio?

[jk212h20]

@stduhpf Thanks again for investigating this. I found a separate generated-audio issue that appears to explain the remaining malformed speech, especially with H3 Turbo at 4 steps: #1908.

H3 uses different shifted flow schedules for video (12) and audio (3). The current C++ path converts audio velocity with the local schedule derivative, but then Euler applies that value over a finite video-sigma interval. At low step counts this substantially over-advances the audio trajectory. #1908 uses the exact audio/video sigma interval ratio for each Euler step.

I tested the fix with H3 Ref2VA + Turbo, Euler/simple, 4 steps, and the same kind of previously malformed generated audio became clear and correct.

I also checked the proposed reference-audio position change against the Hugging Face Diffusers and ComfyUI H3 layouts. Both use channel-major audio packing with the same time position for the two channels (cursor + t), rather than advancing time by t * 2, so I did not include that positional change.

This still does not turn fully_copy into literal waveform muxing—the model generates a new soundtrack—but it fixes a concrete scheduler error affecting that generated soundtrack. PR #1886 remains the reference-encoder fix; #1908 addresses the subsequent generated-audio sampling problem.

[leejet]

Thank you for your contribution.