Repository navigation
Support multiple control nets in the StableDiffusionControlNetXSPipeline/StableDiffusionXLControlNetXSPipeline #8434
Description
Is your feature request related to a problem? Please describe.
The StableDiffusionControlNetPipeline/StableDiffusionXLControlNetPipeline allow you to specify a list of control nets (i.e. depth and canny)
However, the new ControlNetXS pipelines only allow you provide a single control net
Describe the solution you'd like.
I want to be able to do:
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
canny_controlnet = ControlNetXSAdapter.from_pretrained(
"UmerHA/Testing-ConrolNetXS-SDXL-canny", torch_dtype=torch.float16
)
depth_controlnet = ControlNetXSAdapter.from_pretrained(
"UmerHA/Testing-ConrolNetXS-SDXL-depth", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetXSPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", controlnet=[canny_controlnet, depth_controlnet], torch_dtype=torch.float16
)
Activity
sayakpaul commentedon Jun 8, 2024
Thanks for your interest. Would you want to open a PR to add this support? We'd be more than happy to guide you. Cc: @UmerHA as well.
github-actions commentedon Sep 14, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
KarthikSundar2002 commentedon Nov 29, 2024
I would like to work in this issue, and this is my first time contributing to Diffusers, so would really like some guidelines on where to start
sayakpaul commentedon Nov 29, 2024
Thanks for that. Feel free to start working on it and ask questions along the way.
KarthikSundar2002 commentedon Dec 1, 2024
I looked into the codebase for a while and also read the corresponding ControlNetXS Paper. I have observed the following
Possible Solutions
I personally feel the first way is good and solves the issue. What are your thoughts? Any Suggestions? Is this the right direction?
sayakpaul commentedon Dec 1, 2024
UmerHA commentedon Dec 1, 2024
@KarthikSundar2002 @sayakpaul
I'm not sure this is possible in principle.
For controlnet (not xs), the controlnet part processes information independently of the main model, and its information is injected into the main model at a single point only. Thinks of it like computing an embedding and then adding it to a representation. So if we have 2 controlnets, they can process information independently of each other (ie compute their embeddings respectively), and then inject their info into the main model (ie add both embeddings to the representation).
For controlnet-xs the information flows to and from the main model many times. So if we have 2 controlnet-xs's, their information becomes muddled together. Ie the embeddings can't be computed correctly in the first place.
Happy to be convinced otherwise by empirical data!
KarthikSundar2002 commentedon Dec 2, 2024
Oh that makes sense, like the base model's feedback signal will have information corresponding to both the controlnet's and we won't be able to separate it into it's constituents. I thought of adding an encoding (something similar to that of positional encoding) to bypass this issue, but then we have to modify the base model's architecture, thus neglecting the premise of controlnets
naomili0924 commentedon Aug 8, 2025
#12100
I created a pull request to address this feature request and verify it with the code provided in description. The injection methods follows the same mechanism from single controlnet.
Due to this is my first contribution pull request, feel free to give me any feedback, thx.
LeonxLJX commentedon Sep 1, 2026
I'd like to take this one (
Support multiple control nets in theStableDiffusionControl`). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)LeonxLJX commentedon Sep 1, 2026
I'd like to take this one (
Support multiple control nets in theStableDiffusionControl`). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)