[Bug] DFlash 2 speculative decoding crashes and fails in Python due to missing ctx_other and non-autoregressive graph execution

[ilperev]

Prerequisites

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

When loading the target model utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF (Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf) together with the DFlash 2 draft model incoai/Qwen3.8-27B-DFlash2-GGUF (Qwen3.8-27B-DFlash2-Q4_K_M.gguf), llama-cpp-python should link the draft context to the target context via cparams.ctx_other = target_context and execute the native C++ block-diffusion speculative decoding pipeline, producing speculative speedups.

Current Behavior

1. Calling draft_llm = Llama(model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf") fails in GGML with:
   dflash requires ctx_other to be set -> ValueError: Failed to create llama_context
   because DFlash 2 sidecar GGUF files do not contain their own lm_head / output.weight tensors and require ctx_other to be set during context initialization.
2. Setting draft_llm.model = draft_model_ptr fails with:
   AttributeError: property 'model' of 'Llama' object has no setter.
3. Calling LlamaDraftModel(draft_llm, num_pred_tokens=5) fails with:
   TypeError: LlamaDraftModel() takes no arguments because LlamaDraftModel is an abstract base class.
4. When subclassing LlamaDraftModel and returning a Python list, it crashes inside llama.py with:
   AttributeError: 'list' object has no attribute 'astype'.
5. When returning a numpy.ndarray with dtype=np.intc, the Python loop calls draft_llm.sample(). This bypasses the C++ DFlash 2 pipeline (target hidden layer extraction llama_get_embeddings_layer_inp, encoder pass llama_encode, and candidate lattice selection build_post_sampling), resulting in a 0% draft acceptance rate and dropping generation speed to baseline autoregressive throughput (~34.27 tok/s).

Environment and Context

• Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition (sm_120), 24+ GB VRAM
• Environment: Hugging Face ZeroGPU Space
• Operating System: Debian GNU/Linux 13 (trixie) / Linux 6.12.94-123.192.amzn2023.x86_64
• glibc Version: Debian GLIBC 2.41-12
• Python Version: 3.12.12
• NVIDIA Driver: 580.159.03 (CUDA 13.0)
• llama-cpp-python: v0.3.35 compiled against submodule vendor/llama.cpp using:
  • Repository: https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4
  • Branch: dflash2
  • Commit Hash: 5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4
  • Upstream Pull Request: https://github.com/ggml-org/llama.cpp/pull/27342/changes

Steps to Reproduce

1. Build llama-cpp-python with vendor/llama.cpp checked out at z-lab/llama.cpp-fork commit 5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4.
2. Download target model utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF and draft model incoai/Qwen3.8-27B-DFlash2-GGUF.
3. Attempt to initialize the draft model in Python:

import llama_cpp
from llama_cpp import Llama

# 1. Target model initialization succeeds:
target_llm = Llama(
    model_path="Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
)

# 2. Draft model initialization fails here:
draft_llm = Llama(
    model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
)

Failure Logs

Context creation failure:

  File "app.py", line 58, in get_or_load_model
    draft_llm = Llama(model_path=DRAFT_MODEL_PATH, ...)
  File ".../llama_cpp/llama.py", line 415, in __init__
    internals.LlamaContext(
  File ".../llama_cpp/_internals.py", line 266, in __init__
    raise ValueError("Failed to create llama_context")
ValueError: Failed to create llama_context

Attribute setter failure:

  File "app.py", line 94, in get_or_load_dflash2_model
    draft_llm.model = draft_model_ptr
AttributeError: property 'model' of 'Llama' object has no setter

LlamaDraftModel constructor failure:

  File "app.py", line 108, in get_or_load_dflash2_model
    target_llm.draft_model = LlamaDraftModel(draft_model=draft_llm, num_pred_tokens=5)
TypeError: LlamaDraftModel() takes no arguments

List vs NumPy array failure:

  File ".../llama_cpp/llama.py", line 1022, in generate
    draft_tokens.astype(int)[
AttributeError: 'list' object has no attribute 'astype'

---

Additional Notes & Disclaimers

• Goal: I am trying to run DFlash 2 (Qwen3.8-27B-DFlash2-Q4_K_M.gguf) on a Hugging Face ZeroGPU Space with llama-cpp-python.
• Specific Fork: The C++ code is from https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (PR https://github.com/ggml-org/llama.cpp/pull/27342/changes).
• Disclaimer on other speculative methods: The mention of EAGLE was suggested by an AI assistant during our discussion; I cannot personally confirm EAGLE's implementation details. I also do not know with certainty about built-in MTP, DFlash v1, or DSpark in llama-cpp-python, though DFlash v1 and DSpark may already be present in upstream llama.cpp.
• AI Disclosure: This bug report was formatted with the help of an AI assistant at my request during a live debugging session. All steps, code snippets, stack traces, and reproduction logs come directly from my testing on Hugging Face ZeroGPU.

[ilperev]

Following up with a quick finding regarding the build system:

In llama-cpp-python's root CMakeLists.txt, llama_cpp_python_install_target is currently called for llama, ggml, ggml-cuda, etc., but not for llama-common.

Because llama.cpp's native speculative decoding implementation (common/speculative.cpp) lives in the llama-common target, adding:

llama_cpp_python_install_target(llama-common)

is needed so that libllama-common.so is included alongside libllama.so in the wheel distribution.

Exposing bindings to common_speculative from libllama-common.so would allow llama-cpp-python to delegate the entire speculative drafting and context linking (ctx_other) pipeline directly to llama.cpp's native C++ engine.