Skip to content

marigold model/pipeline review #13628

Open
Open
marigold model/pipeline review#13628

Description

@hlky
hlky
opened on Apr 29, 2026

marigold model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Issue 1: NumPy HWC images validate returned latents against the wrong shape

Affected code:

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py

Lines 273 to 312 in 0f1abc4

if isinstance(img, np.ndarray) or torch.is_tensor(img):
if img.ndim not in (2, 3, 4):
raise ValueError(f"`image[{i}]` has unsupported dimensions or shape: {img.shape}.")
H_i, W_i = img.shape[-2:]
N_i = 1
if img.ndim == 4:
N_i = img.shape[0]
elif isinstance(img, Image.Image):
W_i, H_i = img.size
N_i = 1
else:
raise ValueError(f"Unsupported `image[{i}]` type: {type(img)}.")
if W is None:
W, H = W_i, H_i
elif (W, H) != (W_i, H_i):
raise ValueError(
f"Input `image[{i}]` has incompatible dimensions {(W_i, H_i)} with the previous images {(W, H)}"
)
num_images += N_i
# latents checks
if latents is not None:
if not torch.is_tensor(latents):
raise ValueError("`latents` must be a torch.Tensor.")
if latents.dim() != 4:
raise ValueError(f"`latents` has unsupported dimensions or shape: {latents.shape}.")
if processing_resolution > 0:
max_orig = max(H, W)
new_H = H * processing_resolution // max_orig
new_W = W * processing_resolution // max_orig
if new_H == 0 or new_W == 0:
raise ValueError(f"Extreme aspect ratio of the input image: [{W} x {H}]")
W, H = new_W, new_H
w = (W + self.vae_scale_factor - 1) // self.vae_scale_factor
h = (H + self.vae_scale_factor - 1) // self.vae_scale_factor
shape_expected = (num_images * ensemble_size, self.vae.config.latent_channels, h, w)
if latents.shape != shape_expected:
raise ValueError(f"`latents` has unexpected shape={latents.shape} expected={shape_expected}.")

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py

Lines 285 to 324 in 0f1abc4

if isinstance(img, np.ndarray) or torch.is_tensor(img):
if img.ndim not in (2, 3, 4):
raise ValueError(f"`image[{i}]` has unsupported dimensions or shape: {img.shape}.")
H_i, W_i = img.shape[-2:]
N_i = 1
if img.ndim == 4:
N_i = img.shape[0]
elif isinstance(img, Image.Image):
W_i, H_i = img.size
N_i = 1
else:
raise ValueError(f"Unsupported `image[{i}]` type: {type(img)}.")
if W is None:
W, H = W_i, H_i
elif (W, H) != (W_i, H_i):
raise ValueError(
f"Input `image[{i}]` has incompatible dimensions {(W_i, H_i)} with the previous images {(W, H)}"
)
num_images += N_i
# latents checks
if latents is not None:
if not torch.is_tensor(latents):
raise ValueError("`latents` must be a torch.Tensor.")
if latents.dim() != 4:
raise ValueError(f"`latents` has unsupported dimensions or shape: {latents.shape}.")
if processing_resolution > 0:
max_orig = max(H, W)
new_H = H * processing_resolution // max_orig
new_W = W * processing_resolution // max_orig
if new_H == 0 or new_W == 0:
raise ValueError(f"Extreme aspect ratio of the input image: [{W} x {H}]")
W, H = new_W, new_H
w = (W + self.vae_scale_factor - 1) // self.vae_scale_factor
h = (H + self.vae_scale_factor - 1) // self.vae_scale_factor
shape_expected = (num_images * ensemble_size, self.unet.config.out_channels, h, w)
if latents.shape != shape_expected:
raise ValueError(f"`latents` has unexpected shape={latents.shape} expected={shape_expected}.")

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py

Lines 258 to 297 in 0f1abc4

if isinstance(img, np.ndarray) or torch.is_tensor(img):
if img.ndim not in (2, 3, 4):
raise ValueError(f"`image[{i}]` has unsupported dimensions or shape: {img.shape}.")
H_i, W_i = img.shape[-2:]
N_i = 1
if img.ndim == 4:
N_i = img.shape[0]
elif isinstance(img, Image.Image):
W_i, H_i = img.size
N_i = 1
else:
raise ValueError(f"Unsupported `image[{i}]` type: {type(img)}.")
if W is None:
W, H = W_i, H_i
elif (W, H) != (W_i, H_i):
raise ValueError(
f"Input `image[{i}]` has incompatible dimensions {(W_i, H_i)} with the previous images {(W, H)}"
)
num_images += N_i
# latents checks
if latents is not None:
if not torch.is_tensor(latents):
raise ValueError("`latents` must be a torch.Tensor.")
if latents.dim() != 4:
raise ValueError(f"`latents` has unsupported dimensions or shape: {latents.shape}.")
if processing_resolution > 0:
max_orig = max(H, W)
new_H = H * processing_resolution // max_orig
new_W = W * processing_resolution // max_orig
if new_H == 0 or new_W == 0:
raise ValueError(f"Extreme aspect ratio of the input image: [{W} x {H}]")
W, H = new_W, new_H
w = (W + self.vae_scale_factor - 1) // self.vae_scale_factor
h = (H + self.vae_scale_factor - 1) // self.vae_scale_factor
shape_expected = (num_images * ensemble_size, self.vae.config.latent_channels, h, w)
if latents.shape != shape_expected:
raise ValueError(f"`latents` has unexpected shape={latents.shape} expected={shape_expected}.")

Problem:
check_inputs() reads every tensor/array size with img.shape[-2:]. That is correct for torch CHW/NCHW, but wrong for NumPy HWC/NHWC. A valid latent for a (32, 64, 3) NumPy image should be (1, 4, 4, 8), but validation expects (1, 4, 8, 1).

Impact:
Users following the documented output_latent=True reuse path can round-trip PIL/torch inputs, but NumPy image inputs reject their own valid latents or allow invalid latents that fail later in denoising.

Reproduction:

import numpy as np
import torch
from types import SimpleNamespace
from diffusers import MarigoldDepthPipeline

pipe = object.__new__(MarigoldDepthPipeline)
pipe.vae_scale_factor = 8
pipe.vae = SimpleNamespace(config=SimpleNamespace(block_out_channels=[1, 1, 1, 1], latent_channels=4))
pipe.scale_invariant = False
pipe.shift_invariant = False

pipe.check_inputs(
    image=np.zeros((32, 64, 3), dtype=np.float32),
    num_inference_steps=1,
    ensemble_size=1,
    processing_resolution=0,
    resample_method_input="bilinear",
    resample_method_output="bilinear",
    batch_size=1,
    ensembling_kwargs=None,
    latents=torch.zeros(1, 4, 4, 8),
    generator=None,
    output_type="pt",
    output_uncertainty=False,
)

Relevant precedent:
MarigoldImageProcessor.load_image_canonical() already treats NumPy as HWC/NHWC before converting to NCHW.

diffusers/src/diffusers/pipelines/marigold/marigold_image_processing.py

Lines 175 to 185 in 0f1abc4

if isinstance(image, np.ndarray):
if np.issubdtype(image.dtype, np.integer) and not np.issubdtype(image.dtype, np.unsignedinteger):
raise ValueError(f"Input image dtype={image.dtype} cannot be a signed integer.")
if np.issubdtype(image.dtype, np.complexfloating):
raise ValueError(f"Input image dtype={image.dtype} cannot be complex.")
if np.issubdtype(image.dtype, bool):
raise ValueError(f"Input image dtype={image.dtype} cannot be boolean.")
if np.issubdtype(image.dtype, np.unsignedinteger):
image_dtype_max = np.iinfo(image.dtype).max
image = image.astype(np.float32) # because torch does not have unsigned dtypes beyond torch.uint8
image = MarigoldImageProcessor.numpy_to_pt(image)

Suggested fix:

if isinstance(img, np.ndarray):
    if img.ndim == 2:
        H_i, W_i = img.shape
        N_i = 1
    elif img.ndim == 3:
        H_i, W_i = img.shape[:2]
        N_i = 1
    else:
        N_i, H_i, W_i = img.shape[:3]
else:
    H_i, W_i = img.shape[-2:]
    N_i = img.shape[0] if img.ndim == 4 else 1

Issue 2: Generator lists are reused unsliced across Marigold manual batches

Affected code:

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py

Lines 514 to 528 in 0f1abc4

for i in self.progress_bar(
range(0, num_images * ensemble_size, batch_size), leave=True, desc="Marigold predictions..."
):
batch_image_latent = image_latent[i : i + batch_size] # [B,4,h,w]
batch_pred_latent = pred_latent[i : i + batch_size] # [B,4,h,w]
effective_batch_size = batch_image_latent.shape[0]
text = batch_empty_text_embedding[:effective_batch_size] # [B,2,1024]
self.scheduler.set_timesteps(num_inference_steps, device=device)
for t in self.progress_bar(self.scheduler.timesteps, leave=False, desc="Diffusion steps..."):
batch_latent = torch.cat([batch_image_latent, batch_pred_latent], dim=1) # [B,8,h,w]
noise = self.unet(batch_latent, t, encoder_hidden_states=text, return_dict=False)[0] # [B,4,h,w]
batch_pred_latent = self.scheduler.step(
noise, t, batch_pred_latent, generator=generator
).prev_sample # [B,4,h,w]

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py

Lines 520 to 534 in 0f1abc4

for i in self.progress_bar(
range(0, num_images * ensemble_size, batch_size), leave=True, desc="Marigold predictions..."
):
batch_image_latent = image_latent[i : i + batch_size] # [B,4,h,w]
batch_pred_latent = pred_latent[i : i + batch_size] # [B,T*4,h,w]
effective_batch_size = batch_image_latent.shape[0]
text = batch_empty_text_embedding[:effective_batch_size] # [B,2,1024]
self.scheduler.set_timesteps(num_inference_steps, device=device)
for t in self.progress_bar(self.scheduler.timesteps, leave=False, desc="Diffusion steps..."):
batch_latent = torch.cat([batch_image_latent, batch_pred_latent], dim=1) # [B,(1+T)*4,h,w]
noise = self.unet(batch_latent, t, encoder_hidden_states=text, return_dict=False)[0] # [B,T*4,h,w]
batch_pred_latent = self.scheduler.step(
noise, t, batch_pred_latent, generator=generator
).prev_sample # [B,T*4,h,w]

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py

Lines 491 to 505 in 0f1abc4

for i in self.progress_bar(
range(0, num_images * ensemble_size, batch_size), leave=True, desc="Marigold predictions..."
):
batch_image_latent = image_latent[i : i + batch_size] # [B,4,h,w]
batch_pred_latent = pred_latent[i : i + batch_size] # [B,4,h,w]
effective_batch_size = batch_image_latent.shape[0]
text = batch_empty_text_embedding[:effective_batch_size] # [B,2,1024]
self.scheduler.set_timesteps(num_inference_steps, device=device)
for t in self.progress_bar(self.scheduler.timesteps, leave=False, desc="Diffusion steps..."):
batch_latent = torch.cat([batch_image_latent, batch_pred_latent], dim=1) # [B,8,h,w]
noise = self.unet(batch_latent, t, encoder_hidden_states=text, return_dict=False)[0] # [B,4,h,w]
batch_pred_latent = self.scheduler.step(
noise, t, batch_pred_latent, generator=generator
).prev_sample # [B,4,h,w]

Problem:
The pipelines validate a generator list of length num_images * ensemble_size, then process predictions in smaller manual batches. Each scheduler step receives the full list instead of the current slice. LCMScheduler.step() samples per-step noise, and randn_tensor() uses only the first shape[0] generators, so later batches reuse the wrong generators.

Impact:
Batched Marigold LCM inference is not equivalent to separate seeded calls when batch_size < num_images * ensemble_size and num_inference_steps > 1.

Reproduction:

import torch
from transformers import CLIPTextConfig, CLIPTextModel, CLIPTokenizer
from diffusers import AutoencoderTiny, LCMScheduler, MarigoldDepthPipeline, UNet2DConditionModel

def make_pipe():
    torch.manual_seed(0)
    unet = UNet2DConditionModel(
        block_out_channels=(32, 64), layers_per_block=1, sample_size=4,
        in_channels=8, out_channels=4,
        down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
        up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
        cross_attention_dim=32,
    )
    vae = AutoencoderTiny(in_channels=3, out_channels=3, latent_channels=4)
    scheduler = LCMScheduler(prediction_type="v_prediction", beta_schedule="scaled_linear")
    text_encoder = CLIPTextModel(CLIPTextConfig(
        bos_token_id=0, eos_token_id=2, hidden_size=32, intermediate_size=37,
        num_attention_heads=4, num_hidden_layers=1, pad_token_id=1, vocab_size=1000,
    ))
    tokenizer = CLIPTokenizer.from_pretrained("hf-internal-testing/tiny-random-clip")
    pipe = MarigoldDepthPipeline(unet, vae, scheduler, text_encoder, tokenizer, "depth", True, True).to("cpu")
    pipe.set_progress_bar_config(disable=True)
    return pipe

def gen(seed):
    return torch.Generator(device="cpu").manual_seed(seed)

image = torch.full((1, 3, 32, 32), 0.5)
batched = make_pipe()(image=[image[0], image[0]], num_inference_steps=2, processing_resolution=0,
                      batch_size=1, generator=[gen(0), gen(1)], output_type="pt").prediction
single_seed_1 = make_pipe()(image=image, num_inference_steps=2, processing_resolution=0,
                            generator=gen(1), output_type="pt").prediction

print((batched[1] - single_seed_1[0]).abs().max().item())  # non-zero

Relevant precedent:
randn_tensor() consumes generator lists by batch position, so callers must pass a list matching the current batch.

diffusers/src/diffusers/utils/torch_utils.py

Lines 167 to 195 in 0f1abc4

batch_size = shape[0]
layout = layout or torch.strided
device = device or torch.device("cpu")
if generator is not None:
gen_device_type = generator.device.type if not isinstance(generator, list) else generator[0].device.type
if gen_device_type != device.type and gen_device_type == "cpu":
rand_device = "cpu"
if device != "mps":
logger.info(
f"The passed generator was created on 'cpu' even though a tensor on {device} was expected."
f" Tensors will be created on 'cpu' and then moved to {device}. Note that one can probably"
f" slightly speed up this function by passing a generator that was created on the {device} device."
)
elif gen_device_type != device.type and gen_device_type == "cuda":
raise ValueError(f"Cannot generate a {device} tensor from a generator of type {gen_device_type}.")
# make sure generator list of length 1 is treated like a non-list
if isinstance(generator, list) and len(generator) == 1:
generator = generator[0]
if isinstance(generator, list):
shape = (1,) + shape[1:]
latents = [
torch.randn(shape, generator=generator[i], device=rand_device, dtype=dtype, layout=layout)
for i in range(batch_size)
]
latents = torch.cat(latents, dim=0).to(device)

Suggested fix:

batch_generator = generator
if isinstance(generator, list):
    batch_generator = generator[i : i + effective_batch_size]

batch_pred_latent = self.scheduler.step(
    noise, t, batch_pred_latent, generator=batch_generator
).prev_sample

Issue 3: Absolute depth ensembling is documented but always raises

Affected code:

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py

Lines 806 to 827 in 0f1abc4

requires_aligning = scale_invariant or shift_invariant
ensemble_size = depth.shape[0]
if requires_aligning:
param = compute_param(depth)
depth = align(depth, param)
depth, uncertainty = ensemble(depth, return_uncertainty=output_uncertainty)
depth_max = depth.max()
if scale_invariant and shift_invariant:
depth_min = depth.min()
elif scale_invariant:
depth_min = 0
else:
raise ValueError("Unrecognized alignment.")
depth_range = (depth_max - depth_min).clamp(min=1e-6)
depth = (depth - depth_min) / depth_range
if output_uncertainty:
uncertainty /= depth_range
return depth, uncertainty # [1,1,H,W], [1,1,H,W]

Problem:
The ensemble_depth() docstring says absolute predictions (scale_invariant=False, shift_invariant=False) skip alignment and only ensemble, but the post-ensemble normalization branch raises ValueError("Unrecognized alignment.") whenever scale_invariant is false.

Impact:
Any absolute-depth Marigold checkpoint config can run single predictions, but ensemble_size > 1 crashes.

Reproduction:

import torch
from diffusers import MarigoldDepthPipeline

depth = torch.rand(3, 1, 8, 8)
MarigoldDepthPipeline.ensemble_depth(
    depth,
    scale_invariant=False,
    shift_invariant=False,
    output_uncertainty=True,
    reduction="mean",
)

Relevant precedent:
The method’s own docstring describes absolute-prediction ensembling as supported.

diffusers/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py

Lines 685 to 691 in 0f1abc4

Ensembles the depth maps represented by the `depth` tensor with expected shape `(B, 1, H, W)`, where B is the
number of ensemble members for a given prediction of size `(H x W)`. Even though the function is designed for
depth maps, it can also be used with disparity maps as long as the input tensor values are non-negative. The
alignment happens when the predictions have one or more degrees of freedom, that is when they are either
affine-invariant (`scale_invariant=True` and `shift_invariant=True`), or just scale-invariant (only
`scale_invariant=True`). For absolute predictions (`scale_invariant=False` and `shift_invariant=False`)
alignment is skipped and only ensembling is performed.

Suggested fix:

if scale_invariant:
    depth_max = depth.max()
    depth_min = depth.min() if shift_invariant else 0
    depth_range = (depth_max - depth_min).clamp(min=1e-6)
    depth = (depth - depth_min) / depth_range
    if output_uncertainty:
        uncertainty /= depth_range

Issue 4: Visualization helpers advertise list[np.ndarray] but list paths assume torch tensors

Affected code:

diffusers/src/diffusers/pipelines/marigold/marigold_image_processing.py

Lines 487 to 538 in 0f1abc4

def visualize_normals(
normals: np.ndarray | torch.Tensor | list[np.ndarray] | list[torch.Tensor],
flip_x: bool = False,
flip_y: bool = False,
flip_z: bool = False,
) -> list[PIL.Image.Image]:
"""
Visualizes surface normals, such as predictions of the `MarigoldNormalsPipeline`.
Args:
normals (`np.ndarray | torch.Tensor | list[np.ndarray, list[torch.Tensor]]`):
Surface normals.
flip_x (`bool`, *optional*, defaults to `False`): Flips the X axis of the normals frame of reference.
Default direction is right.
flip_y (`bool`, *optional*, defaults to `False`): Flips the Y axis of the normals frame of reference.
Default direction is top.
flip_z (`bool`, *optional*, defaults to `False`): Flips the Z axis of the normals frame of reference.
Default direction is facing the observer.
Returns: `list[PIL.Image.Image]` with surface normals visualization.
"""
flip_vec = None
if any((flip_x, flip_y, flip_z)):
flip_vec = torch.tensor(
[
(-1) ** flip_x,
(-1) ** flip_y,
(-1) ** flip_z,
],
dtype=torch.float32,
)
def visualize_normals_one(img, idx=None):
img = img.permute(1, 2, 0)
if flip_vec is not None:
img *= flip_vec.to(img.device)
img = (img + 1.0) * 0.5
img = (img * 255).to(dtype=torch.uint8, device="cpu").numpy()
img = PIL.Image.fromarray(img)
return img
if normals is None or isinstance(normals, list) and any(o is None for o in normals):
raise ValueError("Input normals is `None`")
if isinstance(normals, (np.ndarray, torch.Tensor)):
normals = MarigoldImageProcessor.expand_tensor_or_array(normals)
if isinstance(normals, np.ndarray):
normals = MarigoldImageProcessor.numpy_to_pt(normals) # [N,3,H,W]
if not (normals.ndim == 4 and normals.shape[1] == 3):
raise ValueError(f"Unexpected input shape={normals.shape}, expecting [N,3,H,W].")
return [visualize_normals_one(img, idx) for idx, img in enumerate(normals)]
elif isinstance(normals, list):
return [visualize_normals_one(img, idx) for idx, img in enumerate(normals)]

diffusers/src/diffusers/pipelines/marigold/marigold_image_processing.py

Lines 543 to 626 in 0f1abc4

def visualize_intrinsics(
prediction: np.ndarray | torch.Tensor | list[np.ndarray] | list[torch.Tensor],
target_properties: dict[str, Any],
color_map: str | dict[str, str] = "binary",
) -> list[dict[str, PIL.Image.Image]]:
"""
Visualizes intrinsic image decomposition, such as predictions of the `MarigoldIntrinsicsPipeline`.
Args:
prediction (`np.ndarray | torch.Tensor | list[np.ndarray, list[torch.Tensor]]`):
Intrinsic image decomposition.
target_properties (`dict[str, Any]`):
Decomposition properties. Expected entries: `target_names: list[str]` and a dictionary with keys
`prediction_space: str`, `sub_target_names: list[str | Null]` (must have 3 entries, null for missing
modalities), `up_to_scale: bool`, one for each target and sub-target.
color_map (`str | dict[str, str]`, *optional*, defaults to `"Spectral"`):
Color map used to convert a single-channel predictions into colored representations. When a dictionary
is passed, each modality can be colored with its own color map.
Returns: `list[dict[str, PIL.Image.Image]]` with intrinsic image decomposition visualization.
"""
if "target_names" not in target_properties:
raise ValueError("Missing `target_names` in target_properties")
if not isinstance(color_map, str) and not (
isinstance(color_map, dict)
and all(isinstance(k, str) and isinstance(v, str) for k, v in color_map.items())
):
raise ValueError("`color_map` must be a string or a dictionary of strings")
n_targets = len(target_properties["target_names"])
def visualize_targets_one(images, idx=None):
# img: [T, 3, H, W]
out = {}
for target_name, img in zip(target_properties["target_names"], images):
img = img.permute(1, 2, 0) # [H, W, 3]
prediction_space = target_properties[target_name].get("prediction_space", "srgb")
if prediction_space == "stack":
sub_target_names = target_properties[target_name]["sub_target_names"]
if len(sub_target_names) != 3 or any(
not (isinstance(s, str) or s is None) for s in sub_target_names
):
raise ValueError(f"Unexpected target sub-names {sub_target_names} in {target_name}")
for i, sub_target_name in enumerate(sub_target_names):
if sub_target_name is None:
continue
sub_img = img[:, :, i]
sub_prediction_space = target_properties[sub_target_name].get("prediction_space", "srgb")
if sub_prediction_space == "linear":
sub_up_to_scale = target_properties[sub_target_name].get("up_to_scale", False)
if sub_up_to_scale:
sub_img = sub_img / max(sub_img.max().item(), 1e-6)
sub_img = sub_img ** (1 / 2.2)
cmap_name = (
color_map if isinstance(color_map, str) else color_map.get(sub_target_name, "binary")
)
sub_img = MarigoldImageProcessor.colormap(sub_img, cmap=cmap_name, bytes=True)
sub_img = PIL.Image.fromarray(sub_img.cpu().numpy())
out[sub_target_name] = sub_img
elif prediction_space == "linear":
up_to_scale = target_properties[target_name].get("up_to_scale", False)
if up_to_scale:
img = img / max(img.max().item(), 1e-6)
img = img ** (1 / 2.2)
elif prediction_space == "srgb":
pass
img = (img * 255).to(dtype=torch.uint8, device="cpu").numpy()
img = PIL.Image.fromarray(img)
out[target_name] = img
return out
if prediction is None or isinstance(prediction, list) and any(o is None for o in prediction):
raise ValueError("Input prediction is `None`")
if isinstance(prediction, (np.ndarray, torch.Tensor)):
prediction = MarigoldImageProcessor.expand_tensor_or_array(prediction)
if isinstance(prediction, np.ndarray):
prediction = MarigoldImageProcessor.numpy_to_pt(prediction) # [N*T,3,H,W]
if not (prediction.ndim == 4 and prediction.shape[1] == 3 and prediction.shape[0] % n_targets == 0):
raise ValueError(f"Unexpected input shape={prediction.shape}, expecting [N*T,3,H,W].")
N_T, _, H, W = prediction.shape
N = N_T // n_targets
prediction = prediction.reshape(N, n_targets, 3, H, W)
return [visualize_targets_one(img, idx) for idx, img in enumerate(prediction)]
elif isinstance(prediction, list):
return [visualize_targets_one(img, idx) for idx, img in enumerate(prediction)]

diffusers/src/diffusers/pipelines/marigold/marigold_image_processing.py

Lines 631 to 670 in 0f1abc4

def visualize_uncertainty(
uncertainty: np.ndarray | torch.Tensor | list[np.ndarray] | list[torch.Tensor],
saturation_percentile=95,
) -> list[PIL.Image.Image]:
"""
Visualizes dense uncertainties, such as produced by `MarigoldDepthPipeline`, `MarigoldNormalsPipeline`, or
`MarigoldIntrinsicsPipeline`.
Args:
uncertainty (`np.ndarray | torch.Tensor | list[np.ndarray, list[torch.Tensor]]`):
Uncertainty maps.
saturation_percentile (`int`, *optional*, defaults to `95`):
Specifies the percentile uncertainty value visualized with maximum intensity.
Returns: `list[PIL.Image.Image]` with uncertainty visualization.
"""
def visualize_uncertainty_one(img, idx=None):
prefix = "Uncertainty" + (f"[{idx}]" if idx else "")
if img.min() < 0:
raise ValueError(f"{prefix}: unexpected data range, min={img.min()}.")
img = img.permute(1, 2, 0) # [H,W,C]
img = img.squeeze(2).cpu().numpy() # [H,W] or [H,W,3]
saturation_value = np.percentile(img, saturation_percentile)
img = np.clip(img * 255 / saturation_value, 0, 255)
img = img.astype(np.uint8)
img = PIL.Image.fromarray(img)
return img
if uncertainty is None or isinstance(uncertainty, list) and any(o is None for o in uncertainty):
raise ValueError("Input uncertainty is `None`")
if isinstance(uncertainty, (np.ndarray, torch.Tensor)):
uncertainty = MarigoldImageProcessor.expand_tensor_or_array(uncertainty)
if isinstance(uncertainty, np.ndarray):
uncertainty = MarigoldImageProcessor.numpy_to_pt(uncertainty) # [N,C,H,W]
if not (uncertainty.ndim == 4 and uncertainty.shape[1] in (1, 3)):
raise ValueError(f"Unexpected input shape={uncertainty.shape}, expecting [N,C,H,W] with C in (1,3).")
return [visualize_uncertainty_one(img, idx) for idx, img in enumerate(uncertainty)]
elif isinstance(uncertainty, list):
return [visualize_uncertainty_one(img, idx) for idx, img in enumerate(uncertainty)]

Problem:
visualize_normals(), visualize_intrinsics(), and visualize_uncertainty() accept list[np.ndarray] in their annotations/docstrings, but their list branches call helpers that immediately use tensor-only methods like .permute().

Impact:
Batch arrays work, but equivalent lists of arrays fail with AttributeError, which is a public API mismatch for post-processing utilities.

Reproduction:

import numpy as np
from diffusers.pipelines.marigold import MarigoldImageProcessor

MarigoldImageProcessor.visualize_normals([np.zeros((4, 4, 3), dtype=np.float32)])

Relevant precedent:
visualize_depth() handles list elements individually, and the non-list branches of these helpers already know how to convert NumPy arrays.

diffusers/src/diffusers/pipelines/marigold/marigold_image_processing.py

Lines 415 to 446 in 0f1abc4

def visualize_depth_one(img, idx=None):
prefix = "Depth" + (f"[{idx}]" if idx else "")
if isinstance(img, PIL.Image.Image):
if img.mode != "I;16":
raise ValueError(f"{prefix}: invalid PIL mode={img.mode}.")
img = np.array(img).astype(np.float32) / (2**16 - 1)
if isinstance(img, np.ndarray) or torch.is_tensor(img):
if img.ndim != 2:
raise ValueError(f"{prefix}: unexpected shape={img.shape}.")
if isinstance(img, np.ndarray):
img = torch.from_numpy(img)
if not torch.is_floating_point(img):
raise ValueError(f"{prefix}: unexpected dtype={img.dtype}.")
else:
raise ValueError(f"{prefix}: unexpected type={type(img)}.")
if val_min != 0.0 or val_max != 1.0:
img = (img - val_min) / (val_max - val_min)
img = MarigoldImageProcessor.colormap(img, cmap=color_map, bytes=True) # [H,W,3]
img = PIL.Image.fromarray(img.cpu().numpy())
return img
if depth is None or isinstance(depth, list) and any(o is None for o in depth):
raise ValueError("Input depth is `None`")
if isinstance(depth, (np.ndarray, torch.Tensor)):
depth = MarigoldImageProcessor.expand_tensor_or_array(depth)
if isinstance(depth, np.ndarray):
depth = MarigoldImageProcessor.numpy_to_pt(depth) # [N,H,W,1] -> [N,1,H,W]
if not (depth.ndim == 4 and depth.shape[1] == 1): # [N,1,H,W]
raise ValueError(f"Unexpected input shape={depth.shape}, expecting [N,1,H,W].")
return [visualize_depth_one(img[0], idx) for idx, img in enumerate(depth)]
elif isinstance(depth, list):
return [visualize_depth_one(img, idx) for idx, img in enumerate(depth)]

Suggested fix:

elif isinstance(normals, list):
    return [
        out
        for item in normals
        for out in MarigoldImageProcessor.visualize_normals(item, flip_x=flip_x, flip_y=flip_y, flip_z=flip_z)
    ]

Apply the same recursive list handling to visualize_intrinsics() and visualize_uncertainty().

Duplicate-search status

Searched GitHub issues and PRs for marigold, the affected class/function/file names, and the specific failure modes above. I found broad Marigold integration/docs items, but no duplicate issues or PRs for these four findings.

Test coverage status

Fast and slow tests exist for depth, normals, and intrinsics under tests/pipelines/marigold/. The gaps are the cases above: NumPy HWC latent reuse, generator-list batching with multi-step LCM, absolute-depth ensembling, and list-of-NumPy visualization inputs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions