在 Diffusers 中使用 Kandinsky 系列管线文生图、图生图、Inpainting、插值与 ControlNet 实战指南【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusersKandinsky 2.1 是 Diffusers 官方支持的多模态扩散模型它继承自 DALL·E 2 与 Latent Diffusion 的最佳实践并通过 CLIP 文本/图像编码器配合扩散图像先验prior在 CLIP 模态的隐空间之间建立映射从而显著提升视觉质量并支持图像混合与文本引导的图像编辑。本文基于 Kandinsky API 文档完整讲解 Kandinsky 2.1/2.2/3 系列管线在文本生成图像、图生图、Inpainting、隐空间插值以及 Kandinsky 2.2 ControlNet 深度条件生成中的端到端用法并结合 src/diffusers/pipelines/kandinsky 下的源码剖析 prior 管线与解码管线的工作机制最后给出推理优化建议。读完本文你将能够独立搭建 Kandinsky 生成流程并针对不同任务灵活切换管线。前置知识Kandinsky 的先验 解码双管线架构Kandinsky 2.1/2.2 与常见扩散模型的最大区别在于必须先用 prior 管线编码提示词、生成图像嵌入image embeddings再用第二个管线把隐变量解码成图像。从源码结构看这一设计对应仓库中的两类文件prior 管线pipeline_kandinsky_prior.py 中的KandinskyPriorPipeline核心组件为PriorTransformerunCLIP 先验、CLIP 文本编码器CLIPTextModelWithProjection、CLIP 图像编码器CLIPVisionModelWithProjection与UnCLIPScheduler解码管线pipeline_kandinsky.py 中的KandinskyPipeline核心组件为多语言 CLIP 文本编码器MultilingualCLIP、XLMRobertaTokenizer、条件 U-NetUNet2DConditionModel与 MoVQ 解码器VQModel默认搭配DDIMScheduler。在 pipeline_kandinsky.py 中可以看到解码管线的 CPU offload 顺序定义为text_encoder-unet-movq其 MoVQ 缩放因子由2 ** (len(self.movq.config.block_out_channels) - 1)计算得到并在get_new_h_w中把用户指定的高宽向上对齐到缩放因子的整数倍保证送入 U-Net 的隐变量尺寸合法。版本差异提醒Kandinsky 2.1 与 2.2 的用法非常相似唯一区别是Kandinsky 2.2 解码时不接收prompt只接收image_embeds而Kandinsky 3 架构更精简不需要 prior 模型其用法与 Stable Diffusion XL 等其他扩散模型完全一致。安装依赖Colab 中取消注释执行#!pip install -q diffusers transformers accelerate文本生成图像Text-to-image无论执行哪种任务第一步都是搭建 prior 管线将提示词编码并生成图像嵌入。prior 管线同时还会生成与负提示词对应的negative_image_embeds。如果想获得更好效果可以给 prior 管线传入真实的negative_prompt但这会使 prior 管线的有效 batch size 翻倍。Kandinsky 2.1两阶段调用from diffusers import KandinskyPriorPipeline, KandinskyPipeline import torch prior_pipeline KandinskyPriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-1-prior, dtypetorch.float16).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyPipeline.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16).to(cuda) prompt A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting negative_prompt low quality, bad quality # 负提示词可选但通常能改善结果 image_embeds, negative_image_embeds prior_pipeline(prompt, negative_prompt, guidance_scale1.0).to_tuple()接着把提示词与两组嵌入一并传给KandinskyPipeline生成图像image pipeline( prompt, image_embedsimage_embeds, negative_promptnegative_prompt, negative_image_embedsnegative_image_embeds, height768, width768, ).images[0] image从 pipeline_kandinsky.py 的__call__实现看KandinskyPipeline会把image_embeds与negative_image_embeds拼接后与文本嵌入一起作为 U-Net 的added_cond_kwargs条件输入并执行无分类器引导CFG当guidance_scale 1.0时噪声预测与方差预测会先按通道拆分、再按 CFG 公式noise_pred_uncond guidance_scale * (noise_pred_text - noise_pred_uncond)合并最后经 MoVQ 解码并归一化到 01 输出 PIL 图像。Kandinsky 2.2解码阶段不接收 promptfrom diffusers import KandinskyV22PriorPipeline, KandinskyV22Pipeline import torch prior_pipeline KandinskyV22PriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-2-prior, dtypetorch.float16).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyV22Pipeline.from_pretrained(kandinsky-community/kandinsky-2-2-decoder, dtypetorch.float16).to(cuda) prompt A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting negative_prompt low quality, bad quality image_embeds, negative_image_embeds prior_pipeline(prompt, guidance_scale1.0).to_tuple() image pipeline(image_embedsimage_embeds, negative_image_embedsnegative_image_embeds, height768, width768).images[0] imageKandinsky 3无需 prior直接生成from diffusers import Kandinsky3Pipeline import torch pipeline Kandinsky3Pipeline.from_pretrained(kandinsky-community/kandinsky-3, variantfp16, dtypetorch.float16) pipeline.enable_model_cpu_offload() prompt A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting image pipeline(prompt).images[0] image端到端合并管线与 AutoPipeline Diffusers 还提供端到端 APIKandinskyCombinedPipeline与KandinskyV22CombinedPipeline二者会自动加载 prior 模型与解码器无需分别加载。若想单独调整 prior 阶段的采样仍可通过prior_guidance_scale与prior_num_inference_steps参数控制——在 pipeline_kandinsky_combined.py 的实现中合并管线正是把这两个参数透传给内部 prior 管线的__call__分别映射为guidance_scale与num_inference_steps。使用AutoPipelineForText2Image即可在底层自动调用合并管线Kandinsky 2.1from diffusers import AutoPipelineForText2Image import torch pipeline AutoPipelineForText2Image.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16) pipeline.enable_model_cpu_offload() prompt A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting negative_prompt low quality, bad quality image pipeline(promptprompt, negative_promptnegative_prompt, prior_guidance_scale1.0, guidance_scale4.0, height768, width768).images[0] imageKandinsky 2.2from diffusers import AutoPipelineForText2Image import torch pipeline AutoPipelineForText2Image.from_pretrained(kandinsky-community/kandinsky-2-2-decoder, dtypetorch.float16) pipeline.enable_model_cpu_offload() prompt A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting negative_prompt low quality, bad quality image pipeline(promptprompt, negative_promptnegative_prompt, prior_guidance_scale1.0, guidance_scale4.0, height768, width768).images[0] image图像到图像Image-to-image图生图需要把初始图像与文本提示词同时传给管线。先加载 prior 管线Kandinsky 2.1import torch from diffusers import KandinskyImg2ImgPipeline, KandinskyPriorPipeline prior_pipeline KandinskyPriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-1-prior, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyImg2ImgPipeline.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16, use_safetensorsTrue).to(cuda)Kandinsky 2.2import torch from diffusers import KandinskyV22Img2ImgPipeline, KandinskyPriorPipeline prior_pipeline KandinskyPriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-2-prior, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyV22Img2ImgPipeline.from_pretrained(kandinsky-community/kandinsky-2-2-decoder, dtypetorch.float16, use_safetensorsTrue).to(cuda)Kandinsky 3不需要 prior直接加载图生图管线from diffusers import Kandinsky3Img2ImgPipeline from diffusers.utils import load_image import torch pipeline Kandinsky3Img2ImgPipeline.from_pretrained(kandinsky-community/kandinsky-3, variantfp16, dtypetorch.float16) pipeline.enable_model_cpu_offload()下载一张用于条件约束的图像load_image来自diffusers.utils返回 PIL 图像并自动转为 RGBfrom diffusers.utils import load_image # 下载示例图像 url https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg original_image load_image(url) original_image original_image.resize((768, 512))用 prior 管线生成image_embeds与negative_image_embedsprompt A fantasy landscape, Cinematic lighting negative_prompt low quality, bad quality image_embeds, negative_image_embeds prior_pipeline(prompt, negative_prompt).to_tuple()把原始图像、提示词与嵌入全部传给管线生成新图像Kandinsky 2.1strength0.3表示保留较多原始图像结构数值越大去噪越彻底、与输入图差异越大from diffusers.utils import make_image_grid image pipeline(prompt, negative_promptnegative_prompt, imageoriginal_image, image_embedsimage_embeds, negative_image_embedsnegative_image_embeds, height768, width768, strength0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows1, cols2)Kandinsky 2.2同样不接收promptfrom diffusers.utils import make_image_grid image pipeline(imageoriginal_image, image_embedsimage_embeds, negative_image_embedsnegative_image_embeds, height768, width768, strength0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows1, cols2)Kandinsky 3image pipeline(prompt, negative_promptnegative_prompt, imageimage, strength0.75, num_inference_steps25).images[0] image同样存在端到端合并管线KandinskyImg2ImgCombinedPipeline与KandinskyV22Img2ImgCombinedPipeline并可用AutoPipelineForImage2Image一键调用Kandinsky 2.1from diffusers import AutoPipelineForImage2Image from diffusers.utils import make_image_grid, load_image import torch pipeline AutoPipelineForImage2Image.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16, use_safetensorsTrue) pipeline.enable_model_cpu_offload() prompt A fantasy landscape, Cinematic lighting negative_prompt low quality, bad quality url https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg original_image load_image(url) original_image.thumbnail((768, 768)) image pipeline(promptprompt, negative_promptnegative_prompt, imageoriginal_image, strength0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows1, cols2)Kandinsky 2.2from diffusers import AutoPipelineForImage2Image from diffusers.utils import make_image_grid, load_image import torch pipeline AutoPipelineForImage2Image.from_pretrained(kandinsky-community/kandinsky-2-2-decoder, dtypetorch.float16) pipeline.enable_model_cpu_offload() prompt A fantasy landscape, Cinematic lighting negative_prompt low quality, bad quality url https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg original_image load_image(url) original_image.thumbnail((768, 768)) image pipeline(promptprompt, negative_promptnegative_prompt, imageoriginal_image, strength0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows1, cols2)图像修补Inpainting⚠️ 重要变更Kandinsky 模型使用白色像素而非黑色像素表示遮罩区域。若在KandinskyInpaintPipeline的生产环境中使用旧掩码需要将其反转为白色# PIL 输入 import PIL.ImageOps mask PIL.ImageOps.invert(mask) # PyTorch 与 NumPy 输入 mask 1 - maskInpainting 需要三样东西原始图像、标记待替换区域的掩码、以及描述修补内容的文本提示词。先加载 prior 管线Kandinsky 2.1from diffusers import KandinskyInpaintPipeline, KandinskyPriorPipeline from diffusers.utils import load_image, make_image_grid import torch import numpy as np from PIL import Image prior_pipeline KandinskyPriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-1-prior, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyInpaintPipeline.from_pretrained(kandinsky-community/kandinsky-2-1-inpaint, dtypetorch.float16, use_safetensorsTrue).to(cuda)Kandinsky 2.2from diffusers import KandinskyV22InpaintPipeline, KandinskyV22PriorPipeline from diffusers.utils import load_image, make_image_grid import torch import numpy as np from PIL import Image prior_pipeline KandinskyV22PriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-2-prior, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyV22InpaintPipeline.from_pretrained(kandinsky-community/kandinsky-2-2-decoder-inpaint, dtypetorch.float16, use_safetensorsTrue).to(cuda)加载初始图像并构造掩码下面示例把猫头上方区域置 1表示待修补位置init_image load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png) mask np.zeros((768, 768), dtypenp.float32) # 遮住猫头上方的区域 mask[:250, 250:-250] 1用 prior 管线生成嵌入prior_output是KandinskyPriorPipelineOutput可直接用**解包prompt a hat prior_output prior_pipeline(prompt)把初始图像、掩码、提示词与嵌入传给 Inpaint 管线Kandinsky 2.1num_inference_steps150提供更充分的去噪迭代output_image pipeline(prompt, imageinit_image, mask_imagemask, **prior_output, height768, width768, num_inference_steps150).images[0] mask Image.fromarray((mask*255).astype(uint8), L) make_image_grid([init_image, mask, output_image], rows1, cols3)Kandinsky 2.2output_image pipeline(imageinit_image, mask_imagemask, **prior_output, height768, width768, num_inference_steps150).images[0] mask Image.fromarray((mask*255).astype(uint8), L) make_image_grid([init_image, mask, output_image], rows1, cols3)同样可以使用端到端的KandinskyInpaintCombinedPipeline/KandinskyV22InpaintCombinedPipeline通过AutoPipelineForInpainting在底层串联 prior 与解码器Kandinsky 2.1import torch import numpy as np from PIL import Image from diffusers import AutoPipelineForInpainting from diffusers.utils import load_image, make_image_grid pipe AutoPipelineForInpainting.from_pretrained(kandinsky-community/kandinsky-2-1-inpaint, dtypetorch.float16) pipe.enable_model_cpu_offload() init_image load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png) mask np.zeros((768, 768), dtypenp.float32) # 遮住猫头上方的区域 mask[:250, 250:-250] 1 prompt a hat output_image pipe(promptprompt, imageinit_image, mask_imagemask).images[0] mask Image.fromarray((mask*255).astype(uint8), L) make_image_grid([init_image, mask, output_image], rows1, cols3)Kandinsky 2.2import torch import numpy as np from PIL import Image from diffusers import AutoPipelineForInpainting from diffusers.utils import load_image, make_image_grid pipe AutoPipelineForInpainting.from_pretrained(kandinsky-community/kandinsky-2-2-decoder-inpaint, dtypetorch.float16) pipe.enable_model_cpu_offload() init_image load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png) mask np.zeros((768, 768), dtypenp.float32) # 遮住猫头上方的区域 mask[:250, 250:-250] 1 prompt a hat output_image pipe(promptprompt, imageinit_image, mask_imagemask).images[0] mask Image.fromarray((mask*255).astype(uint8), L) make_image_grid([init_image, mask, output_image], rows1, cols3)隐空间插值Interpolation插值允许你在图像嵌入与文本嵌入之间的隐空间中进行探索能够直观看到 prior 模型的中间产物。加载 prior 管线与两张想插值的图像Kandinsky 2.1from diffusers import KandinskyPriorPipeline, KandinskyPipeline from diffusers.utils import load_image, make_image_grid import torch prior_pipeline KandinskyPriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-1-prior, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu img_1 load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png) img_2 load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/starry_night.jpeg) make_image_grid([img_1.resize((512, 512)), img_2.resize((512, 512))], rows1, cols2)Kandinsky 2.2from diffusers import KandinskyV22PriorPipeline, KandinskyV22Pipeline from diffusers.utils import load_image, make_image_grid import torch prior_pipeline KandinskyV22PriorPipeline.from_pretrained(kandinsky-community/kandinsky-2-2-prior, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu img_1 load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png) img_2 load_image(https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/starry_night.jpeg) make_image_grid([img_1.resize((512, 512)), img_2.resize((512, 512))], rows1, cols2)指定要插值的文本或图像并为每一项设置权重权重之和不必为 1但会影响融合比例建议多尝试不同组合images_texts [a cat, img_1, img_2] weights [0.3, 0.3, 0.4]调用interpolate方法生成嵌入再传给解码管线生成图像Kandinsky 2.1# prompt 可以留空 prompt prior_out prior_pipeline.interpolate(images_texts, weights) pipeline KandinskyPipeline.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu image pipeline(prompt, **prior_out, height768, width768).images[0] imageKandinsky 2.2# prompt 可以留空 prompt prior_out prior_pipeline.interpolate(images_texts, weights) pipeline KandinskyV22Pipeline.from_pretrained(kandinsky-community/kandinsky-2-2-decoder, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu image pipeline(prompt, **prior_out, height768, width768).images[0] image从 pipeline_kandinsky_prior.py 的实现可以看到interpolate的底层逻辑它遍历images_and_prompts列表字符串经 prior 扩散生成image_embedsPIL 图像或张量则直接经image_processor预处理后由CLIPVisionModelWithProjection编码最后把所有嵌入按权重加权求和image_emb * weight后sum得到融合嵌入同时以空字符串为条件生成负嵌入zero_image_emb返回KandinskyPriorPipelineOutput。这也解释了为何插值完成后解码阶段可以把prompt留空。Kandinsky 2.2 ControlNet深度图条件生成⚠️ 注意ControlNet 仅支持 Kandinsky 2.2ControlNet 允许用深度图、边缘检测等额外输入来约束大型预训练扩散模型。例如给 Kandinsky 2.2 提供深度图模型就能理解并保留深度图像的结构信息。首先加载一张图像并提取其深度图from diffusers.utils import load_image img load_image( https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinskyv22/cat.png ).resize((768, 768)) img用 Transformers 的depth-estimationpipeline 处理图像、提取深度图make_hint把深度图复制成三通道并归一化到 01再转为 CHW 格式import torch import numpy as np from transformers import pipeline def make_hint(image, depth_estimator): image depth_estimator(image)[depth] image np.array(image) image image[:, :, None] image np.concatenate([image, image, image], axis2) detected_map torch.from_numpy(image).float() / 255.0 hint detected_map.permute(2, 0, 1) return hint depth_estimator pipeline(depth-estimation) hint make_hint(img, depth_estimator).unsqueeze(0).half().to(cuda) # 或 mps、xpu、cpu深度条件文生图加载 prior 管线与KandinskyV22ControlnetPipelinefrom diffusers import KandinskyV22PriorPipeline, KandinskyV22ControlnetPipeline prior_pipeline KandinskyV22PriorPipeline.from_pretrained( kandinsky-community/kandinsky-2-2-prior, dtypetorch.float16, use_safetensorsTrue ).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyV22ControlnetPipeline.from_pretrained( kandinsky-community/kandinsky-2-2-controlnet-depth, dtypetorch.float16 ).to(cuda)从提示词与负提示词生成图像嵌入这里使用固定随机种子保证结果可复现prompt A robot, 4k photo negative_prior_prompt lowres, text, error, cropped, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, out of frame, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, username, watermark, signature generator torch.Generator(devicecuda).manual_seed(43) # 或 mps、xpu、cpu image_emb, zero_image_emb prior_pipeline( promptprompt, negative_promptnegative_prior_prompt, generatorgenerator ).to_tuple()最后把图像嵌入与深度图一起传给KandinskyV22ControlnetPipeline生成图像image pipeline(image_embedsimage_emb, negative_image_embedszero_image_emb, hinthint, num_inference_steps50, generatorgenerator, height768, width768).images[0] image深度条件图生图图生图 ControlNet 需要两个管线KandinskyV22PriorEmb2EmbPipeline根据文本提示词与初始图像生成图像嵌入KandinskyV22ControlnetImg2ImgPipeline根据初始图像与图像嵌入生成新图像。先用depth-estimation处理初始猫图得到深度图import torch import numpy as np from diffusers import KandinskyV22PriorEmb2EmbPipeline, KandinskyV22ControlnetImg2ImgPipeline from diffusers.utils import load_image from transformers import pipeline img load_image( https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinskyv22/cat.png ).resize((768, 768)) def make_hint(image, depth_estimator): image depth_estimator(image)[depth] image np.array(image) image image[:, :, None] image np.concatenate([image, image, image], axis2) detected_map torch.from_numpy(image).float() / 255.0 hint detected_map.permute(2, 0, 1) return hint depth_estimator pipeline(depth-estimation) hint make_hint(img, depth_estimator).unsqueeze(0).half().to(cuda) # 或 mps、xpu、cpu加载两个管线prior_pipeline KandinskyV22PriorEmb2EmbPipeline.from_pretrained( kandinsky-community/kandinsky-2-2-prior, dtypetorch.float16, use_safetensorsTrue ).to(cuda) # 或 mps、xpu、cpu pipeline KandinskyV22ControlnetImg2ImgPipeline.from_pretrained( kandinsky-community/kandinsky-2-2-controlnet-depth, dtypetorch.float16 ).to(cuda)把文本提示词与初始图像传给 prior 管线生成嵌入strength控制原始图像被扰动/覆盖的程度正嵌入与负嵌入可分别设置prompt A robot, 4k photo negative_prior_prompt lowres, text, error, cropped, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, out of frame, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, username, watermark, signature generator torch.Generator(devicecuda).manual_seed(43) # 或 mps、xpu、cpu img_emb prior_pipeline(promptprompt, imageimg, strength0.85, generatorgenerator) negative_emb prior_pipeline(promptnegative_prior_prompt, imageimg, strength1, generatorgenerator)运行KandinskyV22ControlnetImg2ImgPipeline从初始图像与嵌入生成新图像image pipeline(imageimg, strength0.5, image_embedsimg_emb.image_embeds, negative_image_embedsnegative_emb.image_embeds, hinthint, num_inference_steps50, generatorgenerator, height768, width768).images[0] make_image_grid([img.resize((512, 512)), image.resize((512, 512))], rows1, cols2)推理优化技巧Kandinsky 的独特之处在于需要 prior 管线生成映射、再由第二个管线把隐变量解码为图像。优化应聚焦在第二个管线因为大部分计算量都集中在这里。以下是提升推理性能的几条建议启用 xFormersPyTorch 2.0 时from diffusers import DiffusionPipeline import torch pipe DiffusionPipeline.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16) pipe.enable_xformers_memory_efficient_attention()使用torch.compilePyTorch 2.0 时自动利用缩放点积注意力SDPApipe.unet.to(memory_formattorch.channels_last) pipe.unet torch.compile(pipe.unet, modereduce-overhead, fullgraphTrue)这等价于显式把注意力处理器设置为AttnAddedKVProcessor2_0from diffusers.models.attention_processor import AttnAddedKVProcessor2_0 pipe.unet.set_attn_processor(AttnAddedKVProcessor2_0())CPU offload避免显存不足OOMfrom diffusers import DiffusionPipeline import torch pipe DiffusionPipeline.from_pretrained(kandinsky-community/kandinsky-2-1, dtypetorch.float16) pipe.enable_model_cpu_offload()更换调度器默认文生图管线使用DDIMScheduler可以替换为DDPMScheduler等观察推理速度与图像质量之间的权衡from diffusers import DDPMScheduler from diffusers import DiffusionPipeline scheduler DDPMScheduler.from_pretrained(kandinsky-community/kandinsky-2-1, subfolderddpm_scheduler) pipe DiffusionPipeline.from_pretrained(kandinsky-community/kandinsky-2-1, schedulerscheduler, dtypetorch.float16, use_safetensorsTrue).to(cuda) # 或 mps、xpu、cpu需要说明的是在 pipeline_kandinsky_prior.py 中prior 管线将prior排除在 CPU offload 之外_exclude_from_cpu_offload [prior]且其 offload 顺序为text_encoder-prior而解码管线 pipeline_kandinsky.py 的 offload 顺序为text_encoder-unet-movq说明两个阶段的显存压力点不同offload 策略也应区别对待。相关管线速查表任务Kandinsky 2.1Kandinsky 2.2Kandinsky 3文本生成图像KandinskyPriorPipelineKandinskyPipelineKandinskyV22PriorPipelineKandinskyV22PipelineKandinsky3Pipeline无需 prior图像到图像KandinskyPriorPipelineKandinskyImg2ImgPipelineKandinskyV22PriorPipelineKandinskyV22Img2ImgPipelineKandinsky3Img2ImgPipeline图像修补KandinskyPriorPipelineKandinskyInpaintPipelineKandinskyV22PriorPipelineKandinskyV22InpaintPipeline—隐空间插值KandinskyPriorPipeline.interpolateKandinskyV22PriorPipeline.interpolate—深度条件生成—KandinskyV22PriorPipelineKandinskyV22ControlnetPipeline—端到端合并KandinskyCombinedPipeline/KandinskyImg2ImgCombinedPipeline/KandinskyInpaintCombinedPipelineKandinskyV22CombinedPipeline/KandinskyV22Img2ImgCombinedPipeline/KandinskyV22InpaintCombinedPipeline—Kandinsky 系列在仓库中的实现位于 src/diffusers/pipelines/kandinsky2.1、src/diffusers/pipelines/kandinsky2_22.2与 src/diffusers/pipelines/kandinsky33对应测试覆盖在 tests/pipelines/kandinsky含test_kandinsky.py、test_kandinsky_prior.py、test_kandinsky_combined.py、test_kandinsky_img2img.py、test_kandinsky_inpaint.py社区脚本还包括 examples/community/unclip_image_interpolation.py 与 examples/community/unclip_text_interpolation.py 等可参考用例。关于调度器速度与质量的权衡可进一步阅读 Schedulers 指南关于如何在多条管线间复用组件可参考 复用跨管线组件 一节。【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考