Repository navigation
Conversation
With `output_type="latent"` the SDXL img2img pipeline sets `image = latents` but still called `watermark.apply_watermark()` and `image_processor.postprocess()` on it, so raw latents were treated as decoded RGB. The text-to-image pipeline already guards both behind `output_type != "latent"`. Apply the same guard in the img2img pipeline, the PAG img2img variant that shares the pattern, and the modular SDXL decoder step. Add a regression test that installs a watermarker which fails if called and asserts the latent output keeps its latent channel count. Ref huggingface#13610 (Issue 4) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBYeq5vB4DNZDEaqUsroKR
|
Hi @Devam0311, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Once the PR links an issue (or gets the |
adi-IL
left a comment
There was a problem hiding this comment.
Tested locally on Linux x86_64.
Before this change, running output_type='latent' on StableDiffusionXLImg2ImgPipeline, PAG img2img, and the modular SDXL decoder step passed raw 4-channel latents through RGB watermarking and image_processor.postprocess().
Guarding both behind output_type != 'latent' matches the behavior in the base text-to-image SDXL pipeline and preserves the latent dimensions. The new test asserting watermark bypass passes.
|
This PR has been automatically closed because it does not link an issue and the reminder above was not addressed within 10 days. If this PR is still relevant, please link the issue it fixes (e.g. We are experimenting with this process to keep the review queue manageable, and it will sometimes get it wrong. If you think this PR should stay open, please just say so here and we will reopen it — no need to justify it at length. Thanks again for contributing, and sorry for the noise if we closed this by mistake! |
What does this PR do?
Fixes Issue 4 of #13610 (
stable_diffusion_xlmodel/pipeline review).With
output_type="latent"the SDXL img2img pipeline setsimage = latentsand then runs the watermarker andimage_processor.postprocess()over it, so raw latents are handled as if they were decoded RGB.The text-to-image pipeline already guards both:
With a real watermarker this can corrupt or fail on larger latent tensors; with any custom watermarker it is simply called with the wrong data.
Solution
Applies the text-to-image guard to the sites that lack it.
The issue names two. I swept the codebase for the same unguarded shape and found one more —
pipeline_pag_sd_xl_img2img.py— so all three are fixed together:pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.pymodular_pipelines/stable_diffusion_xl/decoders.pypipelines/pag/pipeline_pag_sd_xl_img2img.py(found by sweep, not listed in the issue)The SDXL inpaint pipeline was checked and already guards correctly.
Testing
test_latent_output_skips_watermark_and_postprocessinstalls a watermarker that raises if called, requestsoutput_type="latent", and asserts the result is still a latent tensor — checking the channel count, sincepostprocesswould have collapsed it to 3 channels.Verified it catches the bug: with the fix reverted it fails.
The
TestStableDiffusionXL*PipelineIPAdapterclasses are deselected above: they fail on a clean checkout ofmainin my environment with aModuleNotFoundErrorfrom an optional dependency I do not have installed, unrelated to this change.Before submitting
self-reviewskill on the diff?Self-review notes
Reviewed against
.ai/references/review-rules.mdandpipelines.md. No blocking issues.For the reviewer:
output_type="latent"returns in these pipelines — previously postprocessed, now raw latents. That is the documented and intended behaviour and matches text-to-image, but it is a behaviour change for anyone who was relying on the postprocessed result.pipeline_pag_sd_xl_img2img.py), found by sweeping for the identical unguarded pattern. Happy to drop it if you'd rather keep this to the two files named.hasattr(components, "watermark"); I left that guard as-is and only wrapped it in theoutput_typecheck.Who can review?
@yiyixuxu @asomoza @hlky