Skip to content

feat: add SenseNova U1.5 support - #1935

Merged
leejet merged 11 commits into
leejet:masterfrom
Maphist0:dev-u1
Sep 11, 2026
Merged

leejet merged 11 commits into
leejet:masterfrom
Maphist0:dev-u1

Conversation

@Maphist0

@Maphist0 Maphist0 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Add SenseNova U1.5 8B MoT support for non-thinking text-to-image generation.

SenseNova U1.5 performs flow matching directly in RGB pixel space, so it does not require a separate text encoder or VAE. This PR adds:

  • loading for the model's root-level sharded Safetensors checkpoint;
  • a dedicated SenseNova U1.5 tokenizer and prompt template;
  • model-local weight-based configuration detection;
  • vision patch/downsample embeddings and progressive PixelShuffle RGB decoding;
  • separate understanding and generation MoT branches with grouped-query attention and three-axis RoPE;
  • prefix K/V caching across denoising steps;
  • shifted-flow Euler sampling and the model-specific unconditional prompt path; and
  • user documentation with a reproducible CUDA invocation.

The implementation supports non-thinking text-to-image generation only. Thinking-mode prompt expansion, image editing, visual understanding, and interleaved generation are outside this PR.

Related Issue / Discussion

Related #1841 #1894

Additional Information

cmake -S . -B build \
  -DSD_BUILD_SHARED_LIBS=ON \
  -DGGML_NATIVE=OFF \
  -DSD_BUILD_SHARED_GGML_LIB=ON \
  -DGGML_BACKEND_DL=ON \
  -DGGML_CPU_ALL_VARIANTS=ON \
  -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
  -DCMAKE_INSTALL_RPATH='$ORIGIN'
cmake --build build --config Release
ctest --test-dir build --output-on-failure
./build/bin/sd-cli --help

Native generation was compared with the pinned official implementation using the same checkpoint, prompt, empty negative prompt, seed, dimensions, Euler sampler, CFG, flow shift, and BF16 precision.

./build/bin/sd-cli \
  --model /path/to/SenseNova-U1.5-8B-MoT \
  --prompt "a red cube on a white background" \
  --width 2048 \
  --height 2048 \
  --steps 50 \
  --cfg-scale 4 \
  --flow-shift 3 \
  --seed 42 \
  --sampling-method euler \
  --rng cuda \
  --fa \
  --output output.png

The one-step output is nearly pixel-identical. Differences accumulate over the full 50-step run because the official pipeline uses SDPA while the sdcpp invocation uses the ggml CUDA flash-attention path.

Sample output:

The (long) prompt that generates the following image
Create a polished English-language infographic titled SenseNova-U1.5 in a modern minimalist technology-matrix style, closely following the visual structure of the official SenseNova-U infographic. Use a wide 16:9 horizontal three-column grid on premium matte-white paper with a very pale silver-gray dot pattern. Use strict alignment, generous whitespace, thin charcoal borders, crisp bold sans-serif headings, and clear modern body type. Make it colorful yet professional: sapphire blue for the left column, violet and indigo for the center, emerald green for the right column, with soft pastel gradient panels and richly colored line icons. Keep every visible word in English only.

At the top center, place the exact large title SenseNova-U1.5. Directly below it, place the exact subtitle NEXT-GENERATION END-TO-END UNIFIED MULTIMODAL MODEL.

Divide the body into three equal vertical sections separated by ample whitespace.

The left section is titled OVERVIEW with a blue magnifying-glass-and-gear icon. Arrange three vertical points. First: a colorful document-and-photo icon, the heading UNIFIED MULTIMODAL FAMILY, and the caption Text & Image Understanding and Generation. Second: an interlocking-ring architecture icon, the heading NEO-UNIFY ARCHITECTURE, and the caption End-to-End Unified Understanding and Generation. Third: crossed-out eye and funnel icons, the heading NO VE · NO VAE, and the caption No Separate Visual or Variational Encoder.

The center section is titled MODEL FAMILY with a violet branching-network icon. Place two large stacked rounded cards with subtle blue-to-violet gradients. The first card contains a dense geometric cube icon, the exact model name SenseNova-U1.5-8B-MoT, and the caption 8B MoT Post-Trained Model. The second card contains a glowing neural-brain icon with a lightning bolt, the exact model name SenseNova-U1.5-8B-MoT-SFT, and the caption 8B MoT Supervised Fine-Tuned Model. Below the cards place two small badges: OPEN WEIGHTS with a smiling face icon, and TECHNICAL REPORT with a document icon.

The right section is titled HIGHLIGHTS with an emerald rising-chart icon. Arrange four separate rounded rectangular cards with pale mint, cyan, lavender, and warm coral backgrounds. Card one has a Möbius-loop icon, heading NATIVE UNIFIED ARCHITECTURE, caption No VE · No VAE. Card two has a trophy icon, heading UNDERSTAND & GENERATE, caption One Unified Model. Card three has interleaved text-and-image icons, heading INTERLEAVED REASONING, caption Native Image Generation During Reasoning. Card four has a detailed pie-chart icon, heading INFOGRAPHIC SYNTHESIS, caption Text-Rich Diagrams and Complex Charts.
sensenova-u1 5-official-style-english-color-2720x1536-v1

Checklist

Maphist0 and others added 11 commits September 3, 2026 23:40
Support root-level sharded Safetensors directories while preserving Diffusers directory precedence.

Detect SenseNova U1.5 using independent tensor markers for its MoT generation branch, vision input path, and pixel-flow output head. Register the architecture as a DiT model and ignore checkpoint tensors that are unused by the text-to-image runtime.
Add a dedicated SenseNova tokenizer built on the Qwen2 BPE
implementation.
Add model-local configuration detection, checkpoint-layout
convolutions, timestep and vision embeddings, and the progressive
PixelShuffle RGB decoder.
Add separate understanding and generation transformer branches with
three-axis RoPE and grouped-query attention.
Register the U1 conditioner, MoT runner, and full-resolution RGB path.

Add shifted-flow Euler sampling, pass prompt token IDs to the diffusion
runner, and route CFG negatives through the U1-specific unconditional
prompt builder.
Copy per-layer prefix K/V tensors into dedicated graph outputs so ggml
keeps their buffers alive until the runner migrates them into persistent
cache storage.
@leejet
leejet merged commit 7f986a9 into leejet:master Sep 11, 2026
7 of 9 checks passed
@leejet

leejet commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Thank you for your contribution.

@Maphist0
Maphist0 deleted the dev-u1 branch September 12, 2026 14:06
danielhanchen added a commit to unslothai/stable-diffusion.cpp that referenced this pull request Sep 21, 2026
* fix: preserve "token_refiner" token for MiniMax H3 LoRAs (leejet#1864)

* fix: fail with a message when MiniMax-H3 is run in img_gen mode (leejet#1863)

* feat: support INT8 ConvRot safetensors (leejet#1857)

* fix: replace free_compute_buffer with runner_done in vae (leejet#1872)

* sync: update ggml (leejet#1873)

* fix(ci): trigger builds for ggml updates

* feat: add taeh3 support (leejet#1874)

* fix: prevent gallocr hash overflow in tiny graph-cut segments (leejet#1880)

* fix: re-clamp streaming VRAM budget to currently free memory (leejet#1878)

* fix: mark graph cuts with both a prefix and a suffix (leejet#1883)

* fix: make max_order of lms sampler configurable (leejet#1885)

* fix: guard against missing sampler/scheduler names (leejet#1887)

* chore: format code

* fix: use sd_get_preview_interval() (leejet#1907)

* feat: configurable image / video compression (leejet#1909)

* feat: support standard Qwen3-VL weights for MiniMax-H3 (leejet#1910)

* feat: load scaled FP8 weights without upfront conversion (leejet#1913)

* fix: match exact weights in LLM config detection (leejet#1923)

* feat: add LTX-2.5 support (leejet#1893)

Co-authored-by: leejet <leejet714@gmail.com>

* fix: correct MiniMax H3 reference audio encoding (leejet#1886)

* fix: correct MiniMax H3 audio Euler steps (leejet#1908)

* feat: use backend-native FP8 matmul when supported (leejet#1916)

* sync: update ggml

* feat: additional `--preview-interval` values (leejet#1915)

Co-authored-by: leejet <leejet714@gmail.com>

* feat: support numbering for preview images (leejet#1895)

* fix: use carrier sampling for MiniMax H3 audio (leejet#1924)

* feat: generalize temporal tiling across video VAEs (leejet#1926)

* feat: prefetch streamed layers during compute (leejet#1905)

Co-authored-by: leejet <leejet714@gmail.com>

* refactor: unify runner lifecycles and weight residency (leejet#1940)

* feat: add verbose logging and log-level selection (leejet#1941)

* feat: enable single-GPU auto-fit with tiered parameter placement (leejet#1942)

* fix: reuse graph cut plans across CFG passes (leejet#1943)

* refactor: split ggml extensions and move implementations to cpp files (leejet#1945)

* fix: preserve K-quantized embedding weights (leejet#1936)

* fix: correct SDXL embeddings loading (leejet#1939)

* refactor: unify model source and weight lifecycle management (leejet#1956)

* docs: reflect GGML_MAX_NAME value change in rpc docs (and in ggml_extend assert) (leejet#1950)

* refactor: split generation pipeline out of stable-diffusion.cpp (leejet#1957)

* fix: enable VAE decode tiling fallback without auto-fit (leejet#1932)

* fix: preserve BF16 embedding weights for get_rows (leejet#1959)

* fix: handle invalid option numbers (leejet#1961)

* feat: expose the loaded model version name through the public API (leejet#1962)

* feat: add SenseNova U1.5 support (leejet#1935)

* fix: reuse graph plans when scale parameters change (leejet#1963)

* feat: add linear and attention scale overrides (leejet#1964)

* feat: preserve explicit backend assignments during auto-fit (leejet#1967)

* fix: guard GPU memory capacity and propagate encoding failures (leejet#1958)

* fix: bound plain-text runs in parse_prompt_attention regex (leejet#1919)

* feat: add Wan2.2 S2V (audio+img-to-video) support (leejet#1925)

Co-authored-by: leejet <leejet714@gmail.com>

* fix: validate vision projector output dim against LLM hidden size (leejet#1918)

* fix: resolve MSVC narrowing conversion warnings (leejet#1969)

* feat: Add generation parameters into video metadata (leejet#1901)

Co-authored-by: leejet <leejet714@gmail.com>

* feat: support external Hugging Face tokenizer JSON files (leejet#1973)

* refactor: require external Gemma 2 and GPT-OSS tokenizers (leejet#1974)

* feat: support Brownian tree noise in all noise injection samplers (leejet#1899)

* fix: use tokenizer-specific pre-tokenization rules (leejet#1975)

* fix: remove vision_model. from ununsed tensors (leejet#1983)

* perf: eliminate temporary allocations in Philox rounds (leejet#1982)

* fix: honor flash attention flag in LLM text encoder attention (leejet#1987)

* refactor: remove obsolete unused tensor filtering (leejet#1984)

* perf: pad small attention heads to 64 for MMA Flash Attention (leejet#1992)

* perf: update ggml for faster direct convolutions (leejet#1993)

* perf: accelerate VAE direct 3D convolutions (leejet#1996)

* fix: propagate CUDA driver dependency to shared library consumers

* fix: prevent clip_preprocess center crop from exceeding the resized image (leejet#1995)

* perf: reduce CPU overhead in graph execution and sampling (leejet#1997)

* perf: parallelize host tensor elementwise and broadcast ops (leejet#1998)

* feat: support building with upstream ggml (leejet#1999)

* feat: add Qwen Image 2.1 support (leejet#1994)

* feat: restore legacy fp8 handling when building with upstream ggml (leejet#2001)

* fix: avoid passing ggml logs as format strings (leejet#2002)

* feat: add LLaDA-Image support (leejet#1968)

Co-authored-by: leejet <leejet714@gmail.com>

* feat: add native CUDA SageAttention support (leejet#2005)

* fix: avoid narrowing conversion in SigVQ patch embedding and format code

* docs: update CONTRIBUTING.md

---------

Co-authored-by: stduhpf <stephduh@live.fr>
Co-authored-by: leejet <leejet714@gmail.com>
Co-authored-by: LostRuins Concedo <39025047+LostRuins@users.noreply.github.com>
Co-authored-by: fszontagh <51741446+fszontagh@users.noreply.github.com>
Co-authored-by: Wagner Bruna <wbruna@users.noreply.github.com>
Co-authored-by: vmobilis <75476228+vmobilis@users.noreply.github.com>
Co-authored-by: Piotr Wilkin (ilintar) <ilintar@gmail.com>
Co-authored-by: jk212h20 <101200018+jk212h20@users.noreply.github.com>
Co-authored-by: assouan <750048+assouan@users.noreply.github.com>
Co-authored-by: nan <zjn32202153@gmail.com>
Co-authored-by: Hmission <62598659+Hmission@users.noreply.github.com>
Co-authored-by: LED-M <105789115+xledx@users.noreply.github.com>
Co-authored-by: Maphist0 <28743569+Maphist0@users.noreply.github.com>
Co-authored-by: George <35490284+noctrex@users.noreply.github.com>
Co-authored-by: Санька Четвёртый <CAHbKA-IV@mail.ru>
Co-authored-by: Lin Xuhao <linxuhao84@gmail.com>
Co-authored-by: Fabrice Aneche <akhenakh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants