Skip to content

consisid model/pipeline review #13586

Open
Open
consisid model/pipeline review#13586

Description

@hlky
hlky
opened on Apr 29, 2026

consisid model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Duplicate search performed with gh search issues/prs for ConsisID, ConsisIDPipeline, ConsisIDTransformer3DModel, num_videos_per_prompt, id_cond, latents dtype, _no_split_modules, attention backend, and prepare_face_models CUDAExecutionProvider. I found no direct duplicate for the items below. Closed issue #10659 contains a related ONNX provider warning in logs, but it does not track the provider/device bug directly.

Issue 1: num_videos_per_prompt is silently ignored

Affected code:

diffusers/src/diffusers/pipelines/consisid/pipeline_consisid.py

Line 674 in 0f1abc4

num_videos_per_prompt: int = 1,

diffusers/src/diffusers/pipelines/consisid/pipeline_consisid.py

Line 793 in 0f1abc4

num_videos_per_prompt = 1

Problem:
__call__ exposes num_videos_per_prompt, but resets it to 1 before prompt encoding and latent preparation. Users requesting multiple videos per prompt get one output without an error.

Impact:
The public API lies about batch semantics and downstream tests do not catch it.

Reproduction:

import torch
from PIL import Image
from transformers import AutoConfig, AutoTokenizer, T5EncoderModel
from diffusers import AutoencoderKLCogVideoX, ConsisIDPipeline, ConsisIDTransformer3DModel, DDIMScheduler

transformer = ConsisIDTransformer3DModel(
    num_attention_heads=2, attention_head_dim=16, in_channels=8, out_channels=4,
    time_embed_dim=2, text_embed_dim=32, num_layers=1, sample_width=2,
    sample_height=2, sample_frames=9, patch_size=2, temporal_compression_ratio=4,
    max_text_seq_length=16, use_rotary_positional_embeddings=True,
    use_learned_positional_embeddings=True, is_train_face=False,
)
vae = AutoencoderKLCogVideoX(
    in_channels=3, out_channels=3, down_block_types=("CogVideoXDownBlock3D",) * 4,
    up_block_types=("CogVideoXUpBlock3D",) * 4, block_out_channels=(8, 8, 8, 8),
    latent_channels=4, layers_per_block=1, norm_num_groups=2, temporal_compression_ratio=4,
)
config = AutoConfig.from_pretrained("hf-internal-testing/tiny-random-t5")
pipe = ConsisIDPipeline(
    AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-t5"),
    T5EncoderModel(config), vae, transformer, DDIMScheduler(),
)
pipe.set_progress_bar_config(disable=True)
frames = pipe(
    image=Image.new("RGB", (16, 16)), prompt="dance monkey", negative_prompt="",
    generator=torch.Generator(device="cpu").manual_seed(0), num_inference_steps=1,
    guidance_scale=1.0, height=16, width=16, num_frames=8, max_sequence_length=16,
    num_videos_per_prompt=2, output_type="pt",
).frames
print(frames.shape)  # torch.Size([1, 8, 3, 16, 16]), expected batch 2

Relevant precedent:
WanPipeline keeps and uses num_videos_per_prompt through prompt expansion and latent batch sizing:

diffusers/src/diffusers/pipelines/wan/pipeline_wan.py

Lines 537 to 560 in 0f1abc4

num_videos_per_prompt=num_videos_per_prompt,
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_prompt_embeds,
max_sequence_length=max_sequence_length,
device=device,
)
transformer_dtype = self.transformer.dtype if self.transformer is not None else self.transformer_2.dtype
prompt_embeds = prompt_embeds.to(transformer_dtype)
if negative_prompt_embeds is not None:
negative_prompt_embeds = negative_prompt_embeds.to(transformer_dtype)
# 4. Prepare timesteps
self.scheduler.set_timesteps(num_inference_steps, device=device)
timesteps = self.scheduler.timesteps
# 5. Prepare latent variables
num_channels_latents = (
self.transformer.config.in_channels
if self.transformer is not None
else self.transformer_2.config.in_channels
)
latents = self.prepare_latents(
batch_size * num_videos_per_prompt,

Suggested fix:
Support the argument instead of resetting it. Repeat image latents and identity tensors per prompt, or reject unsupported values explicitly:

if num_videos_per_prompt != 1:
    raise ValueError("`num_videos_per_prompt > 1` is not currently supported by ConsisIDPipeline.")

Issue 2: Identity tensors are not CFG/batch expanded

Affected code:

diffusers/src/diffusers/pipelines/consisid/pipeline_consisid.py

Lines 893 to 912 in 0f1abc4

latent_model_input = torch.cat([latents] * 2) if do_classifier_free_guidance else latents
latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)
latent_image_input = torch.cat([image_latents] * 2) if do_classifier_free_guidance else image_latents
latent_model_input = torch.cat([latent_model_input, latent_image_input], dim=2)
# broadcast to batch dimension in a way that's compatible with ONNX/Core ML
timestep = t.expand(latent_model_input.shape[0])
# predict noise model_output
noise_pred = self.transformer(
hidden_states=latent_model_input,
encoder_hidden_states=prompt_embeds,
timestep=timestep,
image_rotary_emb=image_rotary_emb,
attention_kwargs=attention_kwargs,
return_dict=False,
id_vit_hidden=id_vit_hidden,
id_cond=id_cond,
)[0]

diffusers/src/diffusers/models/transformers/consisid_transformer_3d.py

Lines 637 to 645 in 0f1abc4

valid_face_emb = None
if self.is_train_face:
id_cond = id_cond.to(device=hidden_states.device, dtype=hidden_states.dtype)
id_vit_hidden = [
tensor.to(device=hidden_states.device, dtype=hidden_states.dtype) for tensor in id_vit_hidden
]
valid_face_emb = self.local_facial_extractor(
id_cond, id_vit_hidden
) # torch.Size([1, 1280]), list[5](torch.Size([1, 577, 1024])) -> torch.Size([1, 32, 2048])

diffusers/src/diffusers/models/transformers/consisid_transformer_3d.py

Lines 687 to 691 in 0f1abc4

if self.is_train_face:
if i % self.cross_attn_interval == 0 and valid_face_emb is not None:
hidden_states = hidden_states + self.local_face_scale * self.perceiver_cross_attention[ca_idx](
valid_face_emb, hidden_states
) # torch.Size([2, 32, 2048]) torch.Size([2, 17550, 3072])

Problem:
CFG doubles latents, image_latents, and prompt_embeds, but forwards id_cond and id_vit_hidden unchanged. Batched per-sample identity embeddings then fail in the facial cross-attention path.

Impact:
Batched ConsisID generation with per-image identities crashes under normal CFG settings.

Reproduction:

import torch
from diffusers import ConsisIDTransformer3DModel

model = ConsisIDTransformer3DModel(
    num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
    time_embed_dim=2, text_embed_dim=8, num_layers=1, sample_width=8, sample_height=8,
    sample_frames=8, patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
    cross_attn_interval=1, is_train_face=True, cross_attn_dim_head=1, cross_attn_num_heads=1,
    LFE_id_dim=2, LFE_vit_dim=2, LFE_depth=5, LFE_dim_head=8, LFE_num_heads=2,
    LFE_num_id_token=1, LFE_num_querie=1, LFE_output_dim=10, LFE_ff_mult=1, LFE_num_scale=1,
)
model(
    hidden_states=torch.randn(4, 1, 4, 8, 8),      # CFG-expanded batch
    encoder_hidden_states=torch.randn(4, 8, 8),
    timestep=torch.arange(4),
    id_cond=torch.ones(2, 2),                     # original prompt batch
    id_vit_hidden=[torch.ones(2, 2, 2)],
)

Relevant precedent:
Pipelines that duplicate conditional inputs for CFG keep all denoiser inputs aligned before calling the transformer.

Suggested fix:
Validate and expand identity inputs in the pipeline before the denoising loop:

def expand_face_tensor(tensor):
    if tensor.shape[0] == 1 and batch_size > 1:
        tensor = tensor.expand(batch_size, *tensor.shape[1:])
    tensor = tensor.repeat_interleave(num_videos_per_prompt, dim=0)
    if do_classifier_free_guidance:
        tensor = torch.cat([tensor, tensor], dim=0)
    return tensor.to(device=device, dtype=prompt_embeds.dtype)

Issue 3: Provided latents are not cast to the pipeline dtype

Affected code:

diffusers/src/diffusers/pipelines/consisid/pipeline_consisid.py

Lines 512 to 518 in 0f1abc4

if latents is None:
latents = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
else:
latents = latents.to(device)
# scale the initial noise by the standard deviation required by the scheduler
latents = latents * self.scheduler.init_noise_sigma

Problem:
When users pass latents, prepare_latents only moves them to device, not dtype. In a bf16 pipeline, fp32 latents promote the concatenated transformer input to fp32 and then hit bf16 weights.

Impact:
Common reproducibility workflows using pre-generated fp32 latents fail in mixed/bfloat16 inference.

Reproduction:

# Same tiny pipe setup as Issue 1, then:
pipe = pipe.to(dtype=torch.bfloat16)
pipe(
    image=Image.new("RGB", (16, 16)), prompt="dance monkey", negative_prompt="",
    generator=torch.Generator(device="cpu").manual_seed(0),
    latents=torch.randn(1, 2, 4, 2, 2),  # fp32
    num_inference_steps=1, guidance_scale=1.0, height=16, width=16,
    num_frames=8, max_sequence_length=16, output_type="pt",
)
# RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16

Relevant precedent:
Most pipeline latent preparation paths cast supplied latents to the requested execution dtype.

Suggested fix:

latents = latents.to(device=device, dtype=dtype)

Issue 4: device_map="auto" is unsupported because _no_split_modules is missing

Affected code:

diffusers/src/diffusers/models/transformers/consisid_transformer_3d.py

Lines 351 to 460 in 0f1abc4

class ConsisIDTransformer3DModel(ModelMixin, AttentionMixin, ConfigMixin, PeftAdapterMixin):
"""
A Transformer model for video-like data in [ConsisID](https://github.com/PKU-YuanGroup/ConsisID).
Parameters:
num_attention_heads (`int`, defaults to `30`):
The number of heads to use for multi-head attention.
attention_head_dim (`int`, defaults to `64`):
The number of channels in each head.
in_channels (`int`, defaults to `16`):
The number of channels in the input.
out_channels (`int`, *optional*, defaults to `16`):
The number of channels in the output.
flip_sin_to_cos (`bool`, defaults to `True`):
Whether to flip the sin to cos in the time embedding.
time_embed_dim (`int`, defaults to `512`):
Output dimension of timestep embeddings.
text_embed_dim (`int`, defaults to `4096`):
Input dimension of text embeddings from the text encoder.
num_layers (`int`, defaults to `30`):
The number of layers of Transformer blocks to use.
dropout (`float`, defaults to `0.0`):
The dropout probability to use.
attention_bias (`bool`, defaults to `True`):
Whether to use bias in the attention projection layers.
sample_width (`int`, defaults to `90`):
The width of the input latents.
sample_height (`int`, defaults to `60`):
The height of the input latents.
sample_frames (`int`, defaults to `49`):
The number of frames in the input latents. Note that this parameter was incorrectly initialized to 49
instead of 13 because ConsisID processed 13 latent frames at once in its default and recommended settings,
but cannot be changed to the correct value to ensure backwards compatibility. To create a transformer with
K latent frames, the correct value to pass here would be: ((K - 1) * temporal_compression_ratio + 1).
patch_size (`int`, defaults to `2`):
The size of the patches to use in the patch embedding layer.
temporal_compression_ratio (`int`, defaults to `4`):
The compression ratio across the temporal dimension. See documentation for `sample_frames`.
max_text_seq_length (`int`, defaults to `226`):
The maximum sequence length of the input text embeddings.
activation_fn (`str`, defaults to `"gelu-approximate"`):
Activation function to use in feed-forward.
timestep_activation_fn (`str`, defaults to `"silu"`):
Activation function to use when generating the timestep embeddings.
norm_elementwise_affine (`bool`, defaults to `True`):
Whether to use elementwise affine in normalization layers.
norm_eps (`float`, defaults to `1e-5`):
The epsilon value to use in normalization layers.
spatial_interpolation_scale (`float`, defaults to `1.875`):
Scaling factor to apply in 3D positional embeddings across spatial dimensions.
temporal_interpolation_scale (`float`, defaults to `1.0`):
Scaling factor to apply in 3D positional embeddings across temporal dimensions.
is_train_face (`bool`, defaults to `False`):
Whether to use enable the identity-preserving module during the training process. When set to `True`, the
model will focus on identity-preserving tasks.
is_kps (`bool`, defaults to `False`):
Whether to enable keypoint for global facial extractor. If `True`, keypoints will be in the model.
cross_attn_interval (`int`, defaults to `2`):
The interval between cross-attention layers in the Transformer architecture. A larger value may reduce the
frequency of cross-attention computations, which can help reduce computational overhead.
cross_attn_dim_head (`int`, optional, defaults to `128`):
The dimensionality of each attention head in the cross-attention layers of the Transformer architecture. A
larger value increases the capacity to attend to more complex patterns, but also increases memory and
computation costs.
cross_attn_num_heads (`int`, optional, defaults to `16`):
The number of attention heads in the cross-attention layers. More heads allow for more parallel attention
mechanisms, capturing diverse relationships between different components of the input, but can also
increase computational requirements.
LFE_id_dim (`int`, optional, defaults to `1280`):
The dimensionality of the identity vector used in the Local Facial Extractor (LFE). This vector represents
the identity features of a face, which are important for tasks like face recognition and identity
preservation across different frames.
LFE_vit_dim (`int`, optional, defaults to `1024`):
The dimension of the vision transformer (ViT) output used in the Local Facial Extractor (LFE). This value
dictates the size of the transformer-generated feature vectors that will be processed for facial feature
extraction.
LFE_depth (`int`, optional, defaults to `10`):
The number of layers in the Local Facial Extractor (LFE). Increasing the depth allows the model to capture
more complex representations of facial features, but also increases the computational load.
LFE_dim_head (`int`, optional, defaults to `64`):
The dimensionality of each attention head in the Local Facial Extractor (LFE). This parameter affects how
finely the model can process and focus on different parts of the facial features during the extraction
process.
LFE_num_heads (`int`, optional, defaults to `16`):
The number of attention heads in the Local Facial Extractor (LFE). More heads can improve the model's
ability to capture diverse facial features, but at the cost of increased computational complexity.
LFE_num_id_token (`int`, optional, defaults to `5`):
The number of identity tokens used in the Local Facial Extractor (LFE). This defines how many
identity-related tokens the model will process to ensure face identity preservation during feature
extraction.
LFE_num_querie (`int`, optional, defaults to `32`):
The number of query tokens used in the Local Facial Extractor (LFE). These tokens are used to capture
high-frequency face-related information that aids in accurate facial feature extraction.
LFE_output_dim (`int`, optional, defaults to `2048`):
The output dimension of the Local Facial Extractor (LFE). This dimension determines the size of the feature
vectors produced by the LFE module, which will be used for subsequent tasks such as face recognition or
tracking.
LFE_ff_mult (`int`, optional, defaults to `4`):
The multiplication factor applied to the feed-forward network's hidden layer size in the Local Facial
Extractor (LFE). A higher value increases the model's capacity to learn more complex facial feature
transformations, but also increases the computation and memory requirements.
LFE_num_scale (`int`, optional, defaults to `5`):
The number of different scales visual feature. A higher value increases the model's capacity to learn more
complex facial feature transformations, but also increases the computation and memory requirements.
local_face_scale (`float`, defaults to `1.0`):
A scaling factor used to adjust the importance of local facial features in the model. This can influence
how strongly the model focuses on high frequency face-related content.
"""
_supports_gradient_checkpointing = True

Problem:
ConsisIDTransformer3DModel inherits ModelMixin, but does not define _no_split_modules. The shared loader rejects automatic device maps for such models.

Impact:
Users cannot use device_map="auto"/balanced placement for a large video transformer that needs memory-aware loading.

Reproduction:

from diffusers import ConsisIDTransformer3DModel

model = ConsisIDTransformer3DModel(
    num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
    time_embed_dim=2, text_embed_dim=8, num_layers=1, sample_width=8, sample_height=8,
    sample_frames=8, patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
    cross_attn_interval=1, is_train_face=False,
)
print(model._get_no_split_modules("auto"))
# ValueError: ConsisIDTransformer3DModel does not support `device_map='auto'`.

Relevant precedent:

diffusers/src/diffusers/models/transformers/cogvideox_transformer_3d.py

Lines 217 to 218 in 0f1abc4

_supports_gradient_checkpointing = True
_no_split_modules = ["CogVideoXBlock", "CogVideoXPatchEmbed"]

Suggested fix:

_no_split_modules = ["ConsisIDBlock", "CogVideoXPatchEmbed", "LocalFacialExtractor", "PerceiverCrossAttention"]

Issue 5: Attention backend selection cannot affect ConsisID attention

Affected code:

diffusers/src/diffusers/models/transformers/consisid_transformer_3d.py

Lines 25 to 28 in 0f1abc4

from ..attention import Attention, AttentionMixin, FeedForward
from ..attention_processor import CogVideoXAttnProcessor2_0
from ..embeddings import CogVideoXPatchEmbed, TimestepEmbedding, Timesteps
from ..modeling_outputs import Transformer2DModelOutput

diffusers/src/diffusers/models/transformers/consisid_transformer_3d.py

Lines 289 to 298 in 0f1abc4

self.attn1 = Attention(
query_dim=dim,
dim_head=attention_head_dim,
heads=num_attention_heads,
qk_norm="layer_norm" if qk_norm else None,
eps=1e-6,
bias=attention_bias,
out_bias=attention_out_bias,
processor=CogVideoXAttnProcessor2_0(),
)

diffusers/src/diffusers/models/attention_processor.py

Lines 2277 to 2330 in 0f1abc4

class CogVideoXAttnProcessor2_0:
r"""
Processor for implementing scaled dot-product attention for the CogVideoX model. It applies a rotary embedding on
query and key vectors, but does not include spatial normalization.
"""
def __init__(self):
if not hasattr(F, "scaled_dot_product_attention"):
raise ImportError("CogVideoXAttnProcessor requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.")
def __call__(
self,
attn: Attention,
hidden_states: torch.Tensor,
encoder_hidden_states: torch.Tensor,
attention_mask: torch.Tensor | None = None,
image_rotary_emb: torch.Tensor | None = None,
) -> torch.Tensor:
text_seq_length = encoder_hidden_states.size(1)
hidden_states = torch.cat([encoder_hidden_states, hidden_states], dim=1)
batch_size, sequence_length, _ = hidden_states.shape
if attention_mask is not None:
attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)
attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1])
query = attn.to_q(hidden_states)
key = attn.to_k(hidden_states)
value = attn.to_v(hidden_states)
inner_dim = key.shape[-1]
head_dim = inner_dim // attn.heads
query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
if attn.norm_q is not None:
query = attn.norm_q(query)
if attn.norm_k is not None:
key = attn.norm_k(key)
# Apply RoPE if needed
if image_rotary_emb is not None:
from .embeddings import apply_rotary_emb
query[:, :, text_seq_length:] = apply_rotary_emb(query[:, :, text_seq_length:], image_rotary_emb)
if not attn.is_cross_attention:
key[:, :, text_seq_length:] = apply_rotary_emb(key[:, :, text_seq_length:], image_rotary_emb)
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False

Problem:
ConsisID uses CogVideoXAttnProcessor2_0, which calls F.scaled_dot_product_attention directly and has no _attention_backend field. The review rules require model-local attention processors using dispatch_attention_fn.

Impact:
model.set_attention_backend(...) cannot route ConsisID attention through configured backends or context-parallel-aware dispatch.

Reproduction:

from diffusers import ConsisIDTransformer3DModel

model = ConsisIDTransformer3DModel(
    num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
    time_embed_dim=2, text_embed_dim=8, num_layers=1, sample_width=8, sample_height=8,
    sample_frames=8, patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
    cross_attn_interval=1, is_train_face=False,
)
processor = next(iter(model.attn_processors.values()))
print(processor.__class__.__name__, hasattr(processor, "_attention_backend"))
# CogVideoXAttnProcessor2_0 False

Relevant precedent:
transformer_wan.py and transformer_flux.py define model-local processors that call dispatch_attention_fn.

Suggested fix:
Port the ConsisID/CogVideoX joint attention processor into consisid_transformer_3d.py and call dispatch_attention_fn(..., backend=self._attention_backend, parallel_config=self._parallel_config).

Issue 6: prepare_face_models hard-codes CUDA ONNX providers

Affected code:

diffusers/src/diffusers/pipelines/consisid/consisid_utils.py

Lines 294 to 350 in 0f1abc4

def prepare_face_models(model_path, device, dtype):
"""
Prepare all face models for the facial recognition task.
Parameters:
- model_path: Path to the directory containing model files.
- device: The device (e.g., 'cuda', 'xpu', 'cpu') where models will be loaded.
- dtype: Data type (e.g., torch.float32) for model inference.
Returns:
- face_helper_1: First face restoration helper.
- face_helper_2: Second face restoration helper.
- face_clip_model: CLIP model for face extraction.
- eva_transform_mean: Mean value for image normalization.
- eva_transform_std: Standard deviation value for image normalization.
- face_main_model: Main face analysis model.
"""
# get helper model
face_helper_1 = FaceRestoreHelper(
upscale_factor=1,
face_size=512,
crop_ratio=(1, 1),
det_model="retinaface_resnet50",
save_ext="png",
device=device,
model_rootpath=os.path.join(model_path, "face_encoder"),
)
face_helper_1.face_parse = None
face_helper_1.face_parse = init_parsing_model(
model_name="bisenet", device=device, model_rootpath=os.path.join(model_path, "face_encoder")
)
face_helper_2 = insightface.model_zoo.get_model(
f"{model_path}/face_encoder/models/antelopev2/glintr100.onnx", providers=["CUDAExecutionProvider"]
)
face_helper_2.prepare(ctx_id=0)
# get local facial extractor part 1
model, _, _ = create_model_and_transforms(
"EVA02-CLIP-L-14-336",
os.path.join(model_path, "face_encoder", "EVA02_CLIP_L_336_psz14_s6B.pt"),
force_custom_clip=True,
)
face_clip_model = model.visual
eva_transform_mean = getattr(face_clip_model, "image_mean", OPENAI_DATASET_MEAN)
eva_transform_std = getattr(face_clip_model, "image_std", OPENAI_DATASET_STD)
if not isinstance(eva_transform_mean, (list, tuple)):
eva_transform_mean = (eva_transform_mean,) * 3
if not isinstance(eva_transform_std, (list, tuple)):
eva_transform_std = (eva_transform_std,) * 3
eva_transform_mean = eva_transform_mean
eva_transform_std = eva_transform_std
# get local facial extractor part 2
face_main_model = FaceAnalysis(
name="antelopev2", root=os.path.join(model_path, "face_encoder"), providers=["CUDAExecutionProvider"]
)
face_main_model.prepare(ctx_id=0, det_size=(640, 640))

Problem:
The helper accepts device, documents CPU/XPU-style values, but always creates InsightFace ONNX models with providers=["CUDAExecutionProvider"] and ctx_id=0.

Impact:
CPU-only or non-CUDA users get provider warnings/fallbacks or incorrect device setup from a public helper that appears device-aware.

Reproduction:

from pathlib import Path

source = Path("src/diffusers/pipelines/consisid/consisid_utils.py").read_text()
for needle in ['providers=["CUDAExecutionProvider"]', "prepare(ctx_id=0)"]:
    print(needle, needle in source)

Relevant precedent:
Diffusers device helpers usually derive provider/device behavior from the requested execution device instead of hard-coding CUDA.

Suggested fix:

device_type = torch.device(device).type
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"] if device_type == "cuda" else ["CPUExecutionProvider"]
ctx_id = 0 if device_type == "cuda" else -1

Issue 7: Tests exist but key assertions are placeholders

Affected code:

diffusers/tests/pipelines/consisid/test_consisid.py

Lines 179 to 181 in 0f1abc4

expected_video = torch.randn(8, 3, 16, 16)
max_diff = np.abs(generated_video - expected_video).max()
self.assertLessEqual(max_diff, 1e10)

diffusers/tests/pipelines/consisid/test_consisid.py

Lines 363 to 366 in 0f1abc4

expected_video = torch.randn(1, 16, 480, 720, 3).numpy()
max_diff = numpy_cosine_similarity_distance(video.cpu(), expected_video)
assert max_diff < 1e-3, f"Max diff is too high. got {video}"

Problem:
The fast test compares output to random noise with a 1e10 threshold. The slow test also compares against freshly generated random noise instead of a fixed expected slice. Slow tests are present, but they do not validate model/pipeline correctness.

Impact:
The runtime bugs above are not caught, and slow coverage can pass or fail for reasons unrelated to ConsisID behavior.

Reproduction:

from pathlib import Path

text = Path("tests/pipelines/consisid/test_consisid.py").read_text()
print("1e10" in text)
print("expected_video = torch.randn" in text)

Relevant precedent:
Other pipeline slow tests pin deterministic expected output slices or cosine distances against fixed fixtures.

Suggested fix:
Replace placeholder assertions with stable expected slices, and add fast tests for:

# num_videos_per_prompt=2 returns batch 2 or raises a clear ValueError
# batched id_cond/id_vit_hidden with guidance_scale > 1
# fp32 user latents in a bf16 pipeline
# ConsisIDTransformer3DModel._get_no_split_modules("auto")

Test execution note: I attempted ./.venv/Scripts/python.exe -m pytest tests/models/transformers/test_models_transformer_consisid.py tests/pipelines/consisid/test_consisid.py -q -m "not slow", but collection failed in this local .venv because the installed Windows PyTorch build lacks torch._C._distributed_c10d, imported through shared test mixins.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions