Repository navigation
Description
Our ExecuTorch fork (software-mansion-labs/executorch) carries Gemma 4 Vulkan
support that upstream does not have. It should go upstream so we stop carrying
it, but it is a re-port rather than a patch, so it needs planning.
Context
Commit 24f0f69 "Add Gemma4 Vulkan support (Android GPU)" is 45 files,
+4247/-1188. It was already a port ("Ports Mateusz Kopciński's Gemma4 Vulkan
work onto 1.3"), and the upstream files it rewrites have moved again since
1.3.1. Upstream now has its own RMSNorm.cpp and IndexSelect.cpp, and its
SDPA.cpp has evolved independently, so our version collides rather than
applies.
Everything else SWM carries has now been checked against upstream and is either
already there or has been sent:
- MLX getitem boundary fix (
1f9e5a4) — already upstream, verbatim. - MLX iOS support (
d454314) — upstream built its own (iOS 17.0+,
ios.toolchain.cmake, metallib viaMLX_METAL_JIT). Superseded. - Vulkan SymInt as a plain
IntEValue — extracted from the Gemma 4 commit and
sent as Vulkan: accept a SymInt delegate arg as a plain Int EValue pytorch/executorch#21848. - Android per-backend shared libraries — Android: allow building Vulkan/XNNPACK backends as separate shared libraries pytorch/executorch#21849.
- Vulkan
index.Tensorsupport check + delegate arg-count guard —
Vulkan: stop delegating nodes past their own support check pytorch/executorch#21847 (found while enabling LFM2.5-VL, see LFM2.5-VL-1.6B: Vulkan export for Android GPU? #1360).
So Gemma 4 Vulkan is the only substantial piece left on the fork.
Suggested split
Four independent pieces, easiest first:
- Shader precision/perf helpers —
linear_q4gsw/linear_dq8ca_q4gsw
changes plussdpa_fp32_acc.glslhandlinear_fp32_scale_dq_acc.glslh.
Additive, likely stand alone. - The
aten.rms_normlowering (replacing the legacyet_vk.rms_norm).
Upstream already has anRMSNorm.cpp; this needs a diff against theirs first
to establish whether ours is an improvement, a duplicate, or a regression. GemmaSDPA.cpp+ the gemma SDPA shaders — the actual feature, ~738 lines
plus ~10 shader files. New files, so it can land as an addition, but it
depends on the reworked SDPA tile-load helpers, which is exactly where it
collides with upstream's evolvedSDPA.cpp. This is the bulk of the work.- Do not upstream — the
gen_vulkan_spv.pybackward-compat shim (lets
ported shaders calldefine_required_extensionswith one arg; a porting
artifact) and theDispatchNode.cpptracer, see below.
Separate cleanup, worth doing regardless
The same commit added an env-gated dispatch tracer to
backends/vulkan/runtime/graph/ops/DispatchNode.cpp — ETVK_DISPATCH_TRACE,
with <cstdio>/fopen/setvbuf, commented "Iter 21 instrumentation". It is
still in the fork HEAD, so it ships in every runtime we build, including the one
RNE distributes. Inert unless the env var is set, but it is leftover debugging
in a shipped runtime and should probably come out of the fork.
Activity
msluszniak commentedon Oct 20, 2026
Upstream PRs: pytorch/executorch#23536 (merged), pytorch/executorch#23539, pytorch/executorch#24023.
Exporter changes to export Gemma 4 with upstream ET:
et_vk::gemma_sdpa_with_kv_cachewithllama.sdpa_with_kv_cache(scale=1.0, sliding mask squeezed to[S, max_context_len])._edge_adaptersdtype promotion and_to_dim_order_copyrewrite; they work around upstream_to_copyandembedding_q4gswfp16 bugs.The fp32 scale and accumulator shaders are not needed: no measurable accuracy gain on S26.
msluszniak commentedon Oct 20, 2026
Since all needed changes are now as upstream PRs and only changes are in exports I'm closing this issue as comleted.