Skip to content

refactor: define VAE tile dimensions in image pixels #2059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leejet merged 1 commit into from
Sep 25, 2026
Merged

refactor: define VAE tile dimensions in image pixels #2059

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/performance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,54 @@ CPU fallback. It excludes weights and cache buffers. Within a runner lifecycle,
the summary is printed only on the first graph or when backend capacities or the
segment count change.

## Use VAE tiling to reduce encode and decode memory usage.

`--vae-tiling` enables spatial tiling for both VAE encoding and decoding. The
default tile size is 256x256 **image pixels**, independent of the VAE scale factor:

```shell
--vae-tiling --vae-tile-size 256x256 --vae-tile-overlap 0.5
```

`--vae-tile-size` accepts one size or `WIDTHxHEIGHT`. A zero dimension uses the
256-pixel default. Sizes are rounded down to a multiple of the VAE scale factor
and capped at the current input dimensions. Explicit sizes below four latent
pixels per axis (or the full axis when it is smaller) are rejected. Encoding and
decoding use the same spatial sizes, without an additional encoding multiplier.
Inputs that fit within a tile are processed as one tile.

For a 512x512 image with the default 50% overlap, both encoding and decoding use
3x3 tiles. A 256-pixel tile corresponds to 32 latent pixels for an 8x VAE, 16 for
a 16x VAE, and 8 for a 32x VAE. Smaller tiles reduce each graph's memory demand,
but overlapping work can increase processing time and tiling can affect image
quality, especially during encoding. Use larger tiles when more context is needed.

`--vae-relative-tile-size` overrides the absolute size on each axis with a positive
value. Values up to and including 1 specify a fraction of the current input size;
values greater than 1 specify a target number of tiles per axis, accounting for
overlap. For example, `0.5x0.5` uses half the width and height in both encode and
decode. The target overlap is clamped to 0 through 0.5 and the actual overlap is
adjusted to fit the image. Size and overlap options require `--vae-tiling`.

**Migration:** `--vae-tile-size` and the C/JSON fields `tile_size_w` and
`tile_size_h` now use image pixels instead of latent units. The C/JSON fields
`tile_size_x/y` have been renamed to `tile_size_w/h`, and `rel_size_x/y` to
`rel_size_w/h`. The command-line option names are unchanged. For example, an old
decode tile size of 32 corresponds to 256 pixels for an 8x VAE or 512 pixels for a
16x VAE. Encoding no longer enlarges explicit or relative tile sizes.

The main VAE decode path retries allocation failures with smaller tiles, even
without `--vae-tiling`. Supported video VAEs first try temporal tiling; spatial
retries use at most 256-pixel tiles initially and then halve the effective tile
dimensions down to the minimum size. Each spatial retry must reduce the effective
tile size. These runtime adjustments do not change the caller's parameters.
Execution failures are not retried, and encoding has no automatic OOM retry.

`--temporal-tiling` remains independent of spatial tiling. MiniMax H3 always uses
spatial tiling (256x256 pixels and 25% overlap by default) and its own temporal
windows. With `--vae-tiling`, its overlap follows `--vae-tile-overlap`; explicit
spatial sizes are honored.

## Offload weights to the CPU to save VRAM without reducing generation speed.

Using `--offload-to-cpu` allows you to offload weights to the CPU, saving VRAM without reducing generation speed.
Expand Down
58 changes: 29 additions & 29 deletions examples/common/common.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1341,7 +1341,7 @@ ArgOptions SDGenerationParams::get_options() {
&embed_image_metadata},
{"",
"--vae-tiling",
"process vae in tiles to reduce memory usage",
"process vae encode and decode in spatial tiles to reduce memory usage (default: 256x256 image pixels)",
true,
&vae_tiling_params.enabled},
{"",
Expand Down Expand Up @@ -1605,12 +1605,12 @@ ArgOptions SDGenerationParams::get_options() {
size_t x_pos = tile_size_str.find('x');
try {
if (x_pos != std::string::npos) {
std::string tile_x_str = tile_size_str.substr(0, x_pos);
std::string tile_y_str = tile_size_str.substr(x_pos + 1);
vae_tiling_params.tile_size_x = std::stoi(tile_x_str);
vae_tiling_params.tile_size_y = std::stoi(tile_y_str);
std::string tile_w_str = tile_size_str.substr(0, x_pos);
std::string tile_h_str = tile_size_str.substr(x_pos + 1);
vae_tiling_params.tile_size_w = std::stoi(tile_w_str);
vae_tiling_params.tile_size_h = std::stoi(tile_h_str);
} else {
vae_tiling_params.tile_size_x = vae_tiling_params.tile_size_y = std::stoi(tile_size_str);
vae_tiling_params.tile_size_w = vae_tiling_params.tile_size_h = std::stoi(tile_size_str);
}
} catch (const std::invalid_argument&) {
return -1;
Expand All @@ -1628,12 +1628,12 @@ ArgOptions SDGenerationParams::get_options() {
size_t x_pos = rel_size_str.find('x');
try {
if (x_pos != std::string::npos) {
std::string rel_x_str = rel_size_str.substr(0, x_pos);
std::string rel_y_str = rel_size_str.substr(x_pos + 1);
vae_tiling_params.rel_size_x = std::stof(rel_x_str);
vae_tiling_params.rel_size_y = std::stof(rel_y_str);
std::string rel_w_str = rel_size_str.substr(0, x_pos);
std::string rel_h_str = rel_size_str.substr(x_pos + 1);
vae_tiling_params.rel_size_w = std::stof(rel_w_str);
vae_tiling_params.rel_size_h = std::stof(rel_h_str);
} else {
vae_tiling_params.rel_size_x = vae_tiling_params.rel_size_y = std::stof(rel_size_str);
vae_tiling_params.rel_size_w = vae_tiling_params.rel_size_h = std::stof(rel_size_str);
}
} catch (const std::invalid_argument&) {
return -1;
Expand Down Expand Up @@ -1763,11 +1763,11 @@ ArgOptions SDGenerationParams::get_options() {
on_scm_policy_arg},
{"",
"--vae-tile-size",
"tile size for vae tiling in latent units, not image pixels, format [X]x[Y] (default: 32x32)",
"tile size for vae encode and decode in image pixels, format [W]x[H] or [S] (default: 256x256; requires --vae-tiling)",
on_tile_size_arg},
{"",
"--vae-relative-tile-size",
"relative tile size for vae tiling, format [X]x[Y], in fraction of image size if < 1, in number of tiles per dim if >=1 (overrides --vae-tile-size)",
"relative tile size for vae encode and decode, format [W]x[H] or [S]: <=1 is a dimension fraction, >1 a target tile count (overrides --vae-tile-size; requires --vae-tiling)",
on_relative_tile_size_arg},
{"",
"--prompt-file",
Expand Down Expand Up @@ -2224,20 +2224,20 @@ bool SDGenerationParams::from_json_str(
if (tiling_json.contains("temporal_tiling") && tiling_json["temporal_tiling"].is_boolean()) {
vae_tiling_params.temporal_tiling = tiling_json["temporal_tiling"];
}
if (tiling_json.contains("tile_size_x") && tiling_json["tile_size_x"].is_number_integer()) {
vae_tiling_params.tile_size_x = tiling_json["tile_size_x"];
if (tiling_json.contains("tile_size_w") && tiling_json["tile_size_w"].is_number_integer()) {
vae_tiling_params.tile_size_w = tiling_json["tile_size_w"];
}
if (tiling_json.contains("tile_size_y") && tiling_json["tile_size_y"].is_number_integer()) {
vae_tiling_params.tile_size_y = tiling_json["tile_size_y"];
if (tiling_json.contains("tile_size_h") && tiling_json["tile_size_h"].is_number_integer()) {
vae_tiling_params.tile_size_h = tiling_json["tile_size_h"];
}
if (tiling_json.contains("target_overlap") && tiling_json["target_overlap"].is_number()) {
vae_tiling_params.target_overlap = tiling_json["target_overlap"];
}
if (tiling_json.contains("rel_size_x") && tiling_json["rel_size_x"].is_number()) {
vae_tiling_params.rel_size_x = tiling_json["rel_size_x"];
if (tiling_json.contains("rel_size_w") && tiling_json["rel_size_w"].is_number()) {
vae_tiling_params.rel_size_w = tiling_json["rel_size_w"];
}
if (tiling_json.contains("rel_size_y") && tiling_json["rel_size_y"].is_number()) {
vae_tiling_params.rel_size_y = tiling_json["rel_size_y"];
if (tiling_json.contains("rel_size_h") && tiling_json["rel_size_h"].is_number()) {
vae_tiling_params.rel_size_h = tiling_json["rel_size_h"];
}
if (tiling_json.contains("extra_tiling_args") && tiling_json["extra_tiling_args"].is_string()) {
extra_tiling_args = tiling_json["extra_tiling_args"].get<std::string>();
Expand Down Expand Up @@ -2934,11 +2934,11 @@ std::string SDGenerationParams::to_string() const {
<< " vae_tiling_params: { "
<< vae_tiling_params.enabled << ", "
<< vae_tiling_params.temporal_tiling << ", "
<< vae_tiling_params.tile_size_x << ", "
<< vae_tiling_params.tile_size_y << ", "
<< vae_tiling_params.tile_size_w << ", "
<< vae_tiling_params.tile_size_h << ", "
<< vae_tiling_params.target_overlap << ", "
<< vae_tiling_params.rel_size_x << ", "
<< vae_tiling_params.rel_size_y << ", "
<< vae_tiling_params.rel_size_w << ", "
<< vae_tiling_params.rel_size_h << ", "
<< "\"" << extra_tiling_args << "\" },\n"
<< "}";
return oss.str();
Expand Down Expand Up @@ -3140,11 +3140,11 @@ std::string build_sdcpp_image_metadata_json(const SDContextParams& ctx_params,
root["vae_tiling"] = {
{"enabled", gen_params.vae_tiling_params.enabled},
{"temporal_tiling", gen_params.vae_tiling_params.temporal_tiling},
{"tile_size_x", gen_params.vae_tiling_params.tile_size_x},
{"tile_size_y", gen_params.vae_tiling_params.tile_size_y},
{"tile_size_w", gen_params.vae_tiling_params.tile_size_w},
{"tile_size_h", gen_params.vae_tiling_params.tile_size_h},
{"target_overlap", gen_params.vae_tiling_params.target_overlap},
{"rel_size_x", gen_params.vae_tiling_params.rel_size_x},
{"rel_size_y", gen_params.vae_tiling_params.rel_size_y},
{"rel_size_w", gen_params.vae_tiling_params.rel_size_w},
{"rel_size_h", gen_params.vae_tiling_params.rel_size_h},
{"extra_tiling_args", gen_params.extra_tiling_args},
};
}
Expand Down
42 changes: 22 additions & 20 deletions examples/server/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -524,11 +524,11 @@ Shared default fields used by both `img_gen` and `vid_gen`:
| `vae_tiling_params` | `object` |
| `vae_tiling_params.enabled` | `boolean` |
| `vae_tiling_params.temporal_tiling` | `boolean` |
| `vae_tiling_params.tile_size_x` | `integer` |
| `vae_tiling_params.tile_size_y` | `integer` |
| `vae_tiling_params.tile_size_w` | `integer` |
| `vae_tiling_params.tile_size_h` | `integer` |
| `vae_tiling_params.target_overlap` | `number` |
| `vae_tiling_params.rel_size_x` | `number` |
| `vae_tiling_params.rel_size_y` | `number` |
| `vae_tiling_params.rel_size_w` | `number` |
| `vae_tiling_params.rel_size_h` | `number` |
| `vae_tiling_params.extra_tiling_args` | `string` |
| `cache_mode` | `string` |
| `cache_option` | `string` |
Expand All @@ -537,6 +537,8 @@ Shared default fields used by both `img_gen` and `vid_gen`:
| `output_format` | `string` |
| `output_compression` | `integer` |

`vae_tiling_params.tile_size_w` and `tile_size_h` are in **image pixels**, with `0` selecting the 256-pixel default. Both encode and decode use these sizes without an encoding multiplier. Positive `rel_size_w`/`rel_size_h` values override the corresponding absolute size: values up to 1 are dimension fractions, and values greater than 1 are target tile counts. Set `enabled` to use spatial tiling. Sizes are aligned down to the VAE scale factor and capped at the input dimensions; explicit sizes below the minimum supported tile size are rejected. These fields previously used latent units; see [VAE tiling](../../docs/performance.md#use-vae-tiling-to-reduce-encode-and-decode-memory-usage) for migration and OOM retry behavior.

`vae_tiling_params.extra_tiling_args` accepts a key=value list. Supported video VAEs accept `temporal_tile_frames` (alias `temporal_tile_size`, default `4`) and `temporal_tile_overlap` (default `1`).
LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEHV use overlap blending. MiniMax H3 keeps its model-specific fixed temporal windows because its latent-to-frame mapping is non-linear.

Expand Down Expand Up @@ -767,11 +769,11 @@ Example:
"vae_tiling_params": {
"enabled": false,
"temporal_tiling": false,
"tile_size_x": 0,
"tile_size_y": 0,
"tile_size_w": 0,
"tile_size_h": 0,
"target_overlap": 0.5,
"rel_size_x": 0.0,
"rel_size_y": 0.0,
"rel_size_w": 0.0,
"rel_size_h": 0.0,
"extra_tiling_args": ""
},

Expand Down Expand Up @@ -900,11 +902,11 @@ Other native fields:
| `vae_tiling_params` | `object` |
| `vae_tiling_params.enabled` | `boolean` |
| `vae_tiling_params.temporal_tiling` | `boolean` |
| `vae_tiling_params.tile_size_x` | `integer` |
| `vae_tiling_params.tile_size_y` | `integer` |
| `vae_tiling_params.tile_size_w` | `integer` |
| `vae_tiling_params.tile_size_h` | `integer` |
| `vae_tiling_params.target_overlap` | `number` |
| `vae_tiling_params.rel_size_x` | `number` |
| `vae_tiling_params.rel_size_y` | `number` |
| `vae_tiling_params.rel_size_w` | `number` |
| `vae_tiling_params.rel_size_h` | `number` |
| `vae_tiling_params.extra_tiling_args` | `string` |
| `cache_mode` | `string` |
| `cache_option` | `string` |
Expand Down Expand Up @@ -1115,11 +1117,11 @@ Example:
"vae_tiling_params": {
"enabled": false,
"temporal_tiling": false,
"tile_size_x": 0,
"tile_size_y": 0,
"tile_size_w": 0,
"tile_size_h": 0,
"target_overlap": 0.5,
"rel_size_x": 0.0,
"rel_size_y": 0.0,
"rel_size_w": 0.0,
"rel_size_h": 0.0,
"extra_tiling_args": ""
},

Expand Down Expand Up @@ -1240,11 +1242,11 @@ Other native fields:
| `vae_tiling_params` | `object` |
| `vae_tiling_params.enabled` | `boolean` |
| `vae_tiling_params.temporal_tiling` | `boolean` |
| `vae_tiling_params.tile_size_x` | `integer` |
| `vae_tiling_params.tile_size_y` | `integer` |
| `vae_tiling_params.tile_size_w` | `integer` |
| `vae_tiling_params.tile_size_h` | `integer` |
| `vae_tiling_params.target_overlap` | `number` |
| `vae_tiling_params.rel_size_x` | `number` |
| `vae_tiling_params.rel_size_y` | `number` |
| `vae_tiling_params.rel_size_w` | `number` |
| `vae_tiling_params.rel_size_h` | `number` |
| `vae_tiling_params.extra_tiling_args` | `string` |
| `cache_mode` | `string` |
| `cache_option` | `string` |
Expand Down
2 changes: 1 addition & 1 deletion examples/server/frontend
Open in desktop
Submodule frontend updated 5 files
+5 −5 src/App.vue
+4 −4 src/lib/form-capabilities.ts
+4 −4 src/lib/form-defaults.ts
+4 −4 src/lib/form-request.ts
+4 −4 src/lib/types.ts
8 changes: 4 additions & 4 deletions examples/server/routes_sdcpp.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,11 @@ static json make_vae_tiling_json(const sd_tiling_params_t& params) {
return {
{"enabled", params.enabled},
{"temporal_tiling", params.temporal_tiling},
{"tile_size_x", params.tile_size_x},
{"tile_size_y", params.tile_size_y},
{"tile_size_w", params.tile_size_w},
{"tile_size_h", params.tile_size_h},
{"target_overlap", params.target_overlap},
{"rel_size_x", params.rel_size_x},
{"rel_size_y", params.rel_size_y},
{"rel_size_w", params.rel_size_w},
{"rel_size_h", params.rel_size_h},
{"extra_tiling_args", params.extra_tiling_args ? params.extra_tiling_args : ""},
};
}
Expand Down
10 changes: 6 additions & 4 deletions include/stable-diffusion.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,11 +173,13 @@ enum lora_apply_mode_t {
typedef struct {
bool enabled;
bool temporal_tiling;
int tile_size_x;
int tile_size_y;
// Spatial tile dimensions in image pixels for both encode and decode; 0 uses 256.
int tile_size_w;
int tile_size_h;
float target_overlap;
float rel_size_x;
float rel_size_y;
// Positive values override tile_size: <= 1 is a dimension fraction, > 1 a target tile count.
float rel_size_w;
float rel_size_h;
const char* extra_tiling_args;
} sd_tiling_params_t;

Expand Down
41 changes: 29 additions & 12 deletions src/core/backend_fit.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,12 @@ namespace sd::backend_fit {
return true;
}

bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params, bool prefer_temporal_tiling, ggml_status status) {
bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params,
bool prefer_temporal_tiling,
ggml_status status,
int latent_tile_size_w,
int latent_tile_size_h,
int scale_factor) {
// Execution failures can leave the device unusable; tiling only helps with allocation failures.
if (status != GGML_STATUS_ALLOC_FAILED) {
return false;
Expand All @@ -487,19 +492,31 @@ namespace sd::backend_fit {
if (prefer_temporal_tiling && !tiling_params.temporal_tiling) {
tiling_params.temporal_tiling = true;
retry_mode = tiling_params.enabled ? "spatial+temporal" : "temporal";
} else if (!tiling_params.enabled) {
tiling_params.enabled = true;
tiling_params.rel_size_x = 0.5f;
tiling_params.rel_size_y = 0.5f;
if (tiling_params.tile_size_x <= 0) {
tiling_params.tile_size_x = 256;
} else {
if (latent_tile_size_w <= 0 || latent_tile_size_h <= 0 || scale_factor <= 0) {
return false;
}
if (tiling_params.tile_size_y <= 0) {
tiling_params.tile_size_y = 256;
auto smaller_tile = [&](int size) {
int next_size = size / 2;
if (!tiling_params.enabled) {
next_size = std::min(next_size, 256 / scale_factor);
}
return std::min(size, std::max(4, next_size));
};
const int tile_size_w = smaller_tile(latent_tile_size_w);
const int tile_size_h = smaller_tile(latent_tile_size_h);
if (tile_size_w == latent_tile_size_w && tile_size_h == latent_tile_size_h) {
return false;
}
retry_mode = tiling_params.temporal_tiling ? "spatial+temporal" : "spatial";
} else {
return false;
tiling_params.enabled = true;
tiling_params.rel_size_w = 0.0f;
tiling_params.rel_size_h = 0.0f;
tiling_params.tile_size_w = tile_size_w * scale_factor;
tiling_params.tile_size_h = tile_size_h * scale_factor;
retry_mode = tiling_params.temporal_tiling ? "spatial+temporal" : "spatial";
LOG_WARN("Reducing VAE decode tiles from %dx%d to %dx%d image pixels",
latent_tile_size_w * scale_factor, latent_tile_size_h * scale_factor,
tiling_params.tile_size_w, tiling_params.tile_size_h);
}

LOG_WARN("VAE decode ran out of memory; retrying with %s tiling",
Expand Down
5 changes: 4 additions & 1 deletion src/core/backend_fit.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ namespace sd::backend_fit {

bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params,
bool prefer_temporal_tiling,
ggml_status status);
ggml_status status,
int latent_tile_size_w,
int latent_tile_size_h,
int scale_factor);

} // namespace sd::backend_fit

Expand Down
Loading
Loading