Ever wished you could generate Stable Diffusion XL images with transparent backgrounds? Well, your wish has been answered by the smart people behind the Transparent Image Layer Diffusion using Latent Transparency paper. They have made their code and models available, and what do you know, Chenlei Hu has ported it to ComfyUI!

Installation

I don’t think huchenlei/ComfyUI-layerdiffusion is available in the ComfyUI-Manager yet, so I installed it manually. Assuming you are in the folder where ComfyUI is installed:

cd ComfyUI\custom_nodes
git clone https://github.com/huchenlei/ComfyUI-layerdiffusion

When the custom nodes are used for the first time, the Layer Diffusion models will be downloaded from Hugging Face and saved to to \ComfyUI\models\layer_model\:

  • layer_xl_transparent_attn.safetensors - 709 MB
  • layer_xl_transparent_conv.safetensors - 3.37 GB
  • vae_transparent_decoder.safetensors -199 MB

I only use the Attention Injection model, since the authors of sd-forge-layerdiffusion say:

This layer_xl_transparent_conv.safetensors is still included for some special use cases that needs special prompt understanding.
However, in practice, I find the layer_xl_transparent_attn.safetensors will lead to better results.

(order of sentences swapped to make more sense in this context)

Workflow

ComfyUI Layer Diffusion workflow

This workflow saves the generated image and mask as two separate files- the mask must be applied to the image in order to get the transparent background. There may be a node to do this, but for simplicity, the workflow just saves them separately. It is a very standard SDXL workflow, except:

  • The SDXL model, in this case Dreamshaper XL Turbo, is passed to LayerDiffusionApply node with the Attention Injection method selected.
  • The output image from the VAE is passed to LayerDiffusionDecode node, along with the latents, and...
  • The mask output converted to an image with a MaskToImage node and saved.

If you use Paint.net, just open the mask and copy it to clipboard. Then open the generated image, and run the Alpha Mask Import Plugin (2.0) with Paste from Clipboard checked, and this will make the background transparent. You can then copy the resulting image and paste it onto a background image of your choosing.

Paint.net Alpha Mask Plugin

Update 8 Mar 2024: Here is a workflow that merges the generated image and mask (as an alpha channel) and saves the image as a transparent PNG - notice the background color of the final image is the color of the SaveImage node. The trick is:

  • first to InvertMask,
  • then to JoinImageWithAlpha where alpha is the inverted mask.

ComfyUI Layer Diffusion workflow merged as a transparent PNG

❮ Older
Newer ❯