AnimateDiff 视频生成实战指南从文本/视频到动画的完整 Diffusers 管线解析【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers导读AnimateDiff 是 Diffusers 生态中一套面向文生视频 / 视频生视频的官方管线家族。其核心思想是在冻结的 Stable DiffusionSD 1.4/1.5乃至 SDXLUNet 中插入一组额外的运动模块Motion Modules用视频片段蒸馏出通用运动先验从而把任意已有个性化文生图模型一键动画化——既不需要针对每个模型重新微调也能保留原有模型生成的风格与多样性。本指南将围绕 docs/source/en/api/pipelines/animatediff.md 展开逐一讲解 AnimateDiff 家族 6 条管线的配置与用法并结合 src/diffusers/pipelines/animatediff 源码与 tests/pipelines/animatediff 测试深入 MotionAdapter、UNetMotionModel、FreeInit、FreeNoise、Motion LoRA 等底层实现帮助读者从会跑示例进阶到理解原理、按需调参。核心原理冻结的文生图模型 插入式运动模块AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning 提出的框架要点是冻结基础文生图模型Stable Diffusion 的 UNet、VAE、文本编码器全部保持不动保证已有个性化能力DreamBooth、LoRA 微调等不丢失插入新初始化的运动建模模块在 UNet 的 ResNet 与 Attention 块之后插入运动模块并用视频片段训练使其蒸馏出通用的运动先验即插即用训练完成后只需把运动模块注入到任何基于同一基础模型SD 1.4/1.5派生出的个性化版本中即可输出多样、个性化的动画片段。从当前仓库源码看这一机制被实现为三个关键类src/diffusers/models/unets/unet_motion_model.pyMotionModulesL1019每个 UNet 块内的一组AnimateDiffTransformer3D模块列表采用正弦位置编码、gelu 激活、可配置注意力头数与最大序列长度MotionAdapterL1060一个容器按block_out_channels(320, 640, 1280, 1280)为 UNet 的 down/up 块和 mid 块分别装配MotionModules并通过FromOriginalModelMixin支持从 AnimateDiff 原始权重格式加载UNetMotionModelL1201在UNet2DConditionModel基础上扩展出的运动版 UNet其默认down_block_types为CrossAttnDownBlockMotion×3 DownBlockMotionup_block_types对应为UpBlockMotionCrossAttnUpBlockMotion×3。在 pipeline_animatediff.py 的构造函数中如果传入的unet是普通UNet2DConditionModel管线会调用UNetMotionModel.from_unet2d(unet, motion_adapter)自动完成冻结 2D UNet 注入运动模块的转换——这正是无需针对性微调即可动画化在代码层面的落点。可用管线一览本仓库 src/diffusers/pipelines/animatediff 目录下提供了 6 条官方管线另有社区实现的 pipeline_animatediff_ipex.py 等位于 examples/community管线任务源码文件AnimateDiffPipeline文生视频pipeline_animatediff.pyAnimateDiffControlNetPipeline使用 ControlNet 的受控视频生成pipeline_animatediff_controlnet.pyAnimateDiffSparseControlNetPipeline使用 SparseCtrl 的受控视频生成pipeline_animatediff_sparsectrl.pyAnimateDiffSDXLPipeline基于 SDXL 的视频生成pipeline_animatediff_sdxl.pyAnimateDiffVideoToVideoPipeline视频生视频pipeline_animatediff_video2video.pyAnimateDiffVideoToVideoControlNetPipeline使用 ControlNet 的视频生视频pipeline_animatediff_video2video_controlnet.pyMotion Adapter 检查点官方适配器权重由guoyww提供如guoyww/animatediff-motion-adapter-v1-5-2、animatediff-motion-adapter-v1-5-3、SDXL 测试版animatediff-motion-adapter-sdxl-beta等它们与任何基于 Stable Diffusion 1.4/1.5 的模型兼容。测试目录 tests/pipelines/animatediff 中 6 个测试文件与 6 条官方管线一一对应可作为各管线输入参数与输出格式的权威参考。AnimateDiffPipeline文生视频基础用法AnimateDiffPipeline由MotionAdapter 检查点 Stable Diffusion 模型检查点组成适用于 SD 1.4/1.5 及其微调版本import torch from diffusers import AnimateDiffPipeline, DDIMScheduler, MotionAdapter from diffusers.utils import export_to_gif # 加载运动适配器 adapter MotionAdapter.from_pretrained(guoyww/animatediff-motion-adapter-v1-5-2, dtypetorch.float16) # 加载基于 SD 1.5 的微调模型 model_id SG161222/Realistic_Vision_V5.1_noVAE pipe AnimateDiffPipeline.from_pretrained(model_id, motion_adapteradapter, dtypetorch.float16) # 关键为 AnimateDiff 定制 DDIM 调度器 scheduler DDIMScheduler.from_pretrained( model_id, subfolderscheduler, clip_sampleFalse, timestep_spacinglinspace, beta_schedulelinear, steps_offset1, ) pipe.scheduler scheduler # 内存优化VAE 切片 模型 CPU offload pipe.vae.enable_slicing() pipe.enable_model_cpu_offload() output pipe( prompt( masterpiece, bestquality, highlydetailed, ultradetailed, sunset, orange sky, warm lighting, fishing boats, ocean waves seagulls, rippling water, wharf, silhouette, serene atmosphere, dusk, evening glow, golden hour, coastal landscape, seaside scenery ), negative_promptbad quality, worse quality, num_frames16, guidance_scale7.5, num_inference_steps25, generatortorch.Generator(cpu).manual_seed(42), ) frames output.frames[0] export_to_gif(frames, animation.gif)关键参数与调度器调优从 pipeline_animatediff.py 的__call__签名可确认如下核心参数prompt/negative_prompt支持字符串或字符串列表guidance_scale 1时启用无分类器引导默认 7.5值越高越贴合提示词但可能降低质量num_frames默认 16生成帧数16 帧在 8fps 下约 2 秒视频num_inference_steps默认 50去噪步数越多质量越高、速度越慢latents可传入预生成的噪声 latent形状(batch, channel, num_frames, height, width)用于多提示词复用同一噪声decode_chunk_size默认 16解码 latent 时每次处理的帧数用于控制解码显存峰值output_type可选pil默认/torch.Tensor/np.arrayreturn_dictFalse时返回元组。[!TIP] 调度器三个关键设置AnimateDiff 与微调后的 SD 模型配合效果更好若调度器支持 clip 采样务必设clip_sampleFalse否则对生成样本有负面影响AnimateDiff 检查点对调度器的 beta 调度敏感官方建议beta_schedulelinear同时配合timestep_spacinglinspace与steps_offset1。AnimateDiffControlNetPipeline用 ControlNet 做受控视频生成ControlNet 通过额外的控制图像如深度图条件化 Stable Diffusion 的生成从而保留深度图的空间信息地生成视频是更灵活、更精确的视频控制手段论文 Adding Conditional Control to Text-to-Image Diffusion Models。完整示例见 docs/source/en/api/pipelines/animatediff.md要点如下import torch from diffusers import AnimateDiffControlNetPipeline, AutoencoderKL, ControlNetModel, MotionAdapter, LCMScheduler from diffusers.utils import export_to_gif, load_video from controlnet_aux.processor import ZoeDetector # pip install controlnet_aux用于预处理视频帧 # 单文件格式 ControlNetcontrol_v11f1p_sd15_depth.pth用 .from_single_file # 或 Diffusers 格式lllyasviel/sd-controlnet-depth用 .from_pretrained controlnet ControlNetModel.from_single_file(control_v11f1p_sd15_depth.pth, dtypetorch.float16) # AnimateLCM 运动模块也可用 guoyww/animatediff-motion-adapter-v1-5-3 motion_adapter MotionAdapter.from_pretrained(wangfuyun/AnimateLCM) vae AutoencoderKL.from_pretrained(stabilityai/sd-vae-ft-mse, dtypetorch.float16) pipe: AnimateDiffControlNetPipeline AnimateDiffControlNetPipeline.from_pretrained( SG161222/Realistic_Vision_V5.1_noVAE, motion_adaptermotion_adapter, controlnetcontrolnet, vaevae, ).to(devicecuda, dtypetorch.float16) # 或 mps / xpu / cpu # LCM 加速推理 pipe.scheduler LCMScheduler.from_config(pipe.scheduler.config, beta_schedulelinear) pipe.load_lora_weights(wangfuyun/AnimateLCM, weight_nameAnimateLCM_sd15_t2v_lora.safetensors, adapter_namelcm-lora) pipe.set_adapters([lcm-lora], [0.8]) # 逐帧提取深度图作为条件 depth_detector ZoeDetector.from_pretrained(lllyasviel/Annotators).to(cuda) video load_video(https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-vid2vid-input-1.gif) conditioning_frames [] with pipe.progress_bar(totallen(video)) as progress_bar: for frame in video: conditioning_frames.append(depth_detector(frame)) progress_bar.update() video pipe( prompta panda, playing a guitar, sitting in a pink boat, in the ocean, mountains in background, realistic, high quality, negative_promptbad quality, worst quality, num_frameslen(video), num_inference_steps10, guidance_scale2.0, conditioning_framesconditioning_frames, # 控制条件帧序列 generatortorch.Generator().manual_seed(42), ).frames[0] export_to_gif(video, animatediff_controlnet.gif, fps8)controlnet_conditioning_scale的取值规则源码 pipeline_animatediff_controlnet.py 对缩放系数做了严格校验单个 ControlNetcontrolnet_conditioning_scale必须是float默认 1.0ControlNet 输出按该系数缩放后叠加进 UNet 主干多个 ControlNetMultiControlNetModel若传float会自动广播为与 ControlNet 数量等长的列表L896-L897若传list其长度必须与 ControlNet 数量一致否则抛错调低该系数可减弱控制条件的影响力用于在忠实于控制图与跟随文本提示词之间平衡。AnimateDiffSparseControlNetPipelineSparseCtrl 稀疏控制SparseCtrl论文 SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models解决的是稀疏时序信号控制问题只需 1 张或少数几张条件帧涂鸦、深度图或 RGB 图即可为视频生成提供结构控制无需逐帧标注密集条件。它额外引入一个条件编码器处理稀疏信号而预训练 T2V 模型保持不动兼容素描、深度图、RGB 图像等多种模态适合分镜绘制、深度渲染、关键帧动画与插帧等应用。官方提供两个检查点guoyww/animatediff-sparsectrl-scribbleSparseCtrl Scribbleguoyww/animatediff-sparsectrl-rgbSparseCtrl RGBSparseCtrl Scribble用涂鸦关键帧控制视频import torch from diffusers import AnimateDiffSparseControlNetPipeline from diffusers.models import AutoencoderKL, MotionAdapter, SparseControlNetModel from diffusers.schedulers import DPMSolverMultistepScheduler from diffusers.utils import export_to_gif, load_image model_id SG161222/Realistic_Vision_V5.1_noVAE motion_adapter_id guoyww/animatediff-motion-adapter-v1-5-3 controlnet_id guoyww/animatediff-sparsectrl-scribble lora_adapter_id guoyww/animatediff-motion-lora-v1-5-3 vae_id stabilityai/sd-vae-ft-mse device cuda # 或 mps / xpu / cpu motion_adapter MotionAdapter.from_pretrained(motion_adapter_id, dtypetorch.float16).to(device) controlnet SparseControlNetModel.from_pretrained(controlnet_id, dtypetorch.float16).to(device) vae AutoencoderKL.from_pretrained(vae_id, dtypetorch.float16).to(device) scheduler DPMSolverMultistepScheduler.from_pretrained( model_id, subfolderscheduler, beta_schedulelinear, algorithm_typedpmsolver, use_karras_sigmasTrue, ) pipe AnimateDiffSparseControlNetPipeline.from_pretrained( model_id, motion_adaptermotion_adapter, controlnetcontrolnet, vaevae, schedulerscheduler, dtypetorch.float16, ).to(device) pipe.load_lora_weights(lora_adapter_id, adapter_namemotion_lora) pipe.fuse_lora(lora_scale1.0) prompt an aerial view of a cyberpunk city, night time, neon lights, masterpiece, high quality negative_prompt low quality, worst quality, letterboxed # 只有 3 张涂鸦关键帧分别指定其生效的帧索引 image_files [ https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-scribble-1.png, https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-scribble-2.png, https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-scribble-3.png, ] condition_frame_indices [0, 8, 15] conditioning_frames [load_image(img_file) for img_file in image_files] video pipe( promptprompt, negative_promptnegative_prompt, num_inference_steps25, conditioning_framesconditioning_frames, controlnet_conditioning_scale1.0, controlnet_frame_indicescondition_frame_indices, # 关键帧与帧索引一一对应 generatortorch.Generator().manual_seed(1337), ).frames[0] export_to_gif(video, output.gif)SparseCtrl RGB单帧图片即可驱动视频RGB 变体甚至只需要1 帧作为起点例如一张烟花人像其余帧由运动模块与文本引导生成image load_image(https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-firework.png) video pipe( promptcloseup face photo of man in black clothes, night city street, bokeh, fireworks in background, negative_promptlow quality, worst quality, num_inference_steps25, conditioning_framesimage, controlnet_frame_indices[0], # 只在第 0 帧施加控制 controlnet_conditioning_scale1.0, generatortorch.Generator().manual_seed(42), ).frames[0] export_to_gif(video, output.gif)AnimateDiffSDXLPipeline基于 SDXL 的视频生成实验性AnimateDiff 同样可以用于 SDXL 模型。需要说明的是目前仅有 beta 版运动适配器检查点guoyww/animatediff-motion-adapter-sdxl-beta属于实验性功能。示例完整代码见 docs/source/en/api/pipelines/animatediff.mdimport torch from diffusers.models import MotionAdapter from diffusers import AnimateDiffSDXLPipeline, DDIMScheduler from diffusers.utils import export_to_gif adapter MotionAdapter.from_pretrained(guoyww/animatediff-motion-adapter-sdxl-beta, dtypetorch.float16) model_id stabilityai/stable-diffusion-xl-base-1.0 scheduler DDIMScheduler.from_pretrained( model_id, subfolderscheduler, clip_sampleFalse, timestep_spacinglinspace, beta_schedulelinear, steps_offset1, ) pipe AnimateDiffSDXLPipeline.from_pretrained( model_id, motion_adapteradapter, schedulerscheduler, dtypetorch.float16, variantfp16, ).to(cuda) # 或 mps / xpu / cpu # SDXL 分辨率更高VAE 显存压力更大切片 tiling 双管齐下 pipe.vae.enable_slicing() pipe.vae.enable_tiling() output pipe( prompta panda surfing in the ocean, realistic, high quality, negative_promptlow quality, worst quality, num_inference_steps20, guidance_scale8, width1024, height1024, num_frames16, ) frames output.frames[0] export_to_gif(frames, animation.gif)注意这里显式指定了width1024, height1024以匹配 SDXL 的原生分辨率并用variantfp16加载半精度权重。测试文件 test_animatediff_sdxl.py 可用于核对 SDXL 管线的最小可用配置。AnimateDiffVideoToVideoPipeline视频生视频与风格迁移该管线接收一段初始视频生成视觉上相似的新视频可用于风格 / 角色 / 背景等编辑。核心在于strength参数控制保留原视频程度。完整示例在 docs/source/en/api/pipelines/animatediff.md关键调用output pipe( videovideo, # 输入视频帧列表PIL.Image promptpanda playing a guitar, on a boat, in the ocean, high quality, negative_promptbad quality, worse quality, guidance_scale7.5, num_inference_steps25, strength0.5, # 越低越贴近原视频 generatortorch.Generator(cpu).manual_seed(42), ) frames output.frames[0] export_to_gif(frames, animation.gif)strength语义与 img2img 一致数值越小对输入视频的改动越小去噪从更靠近原视频的噪声水平开始数值越大文本提示词主导程度越高。文档中给出的示例效果浣熊弹吉他 → 熊猫弹吉他、玛格特·罗比近景 → 托尼·斯塔克近景说明该管线在保持原视频构图、光影的前提下替换主体/风格的能力。AnimateDiffVideoToVideoControlNetPipeline视频 ControlNet 双重条件此管线同时以原视频和一组控制图像作为条件既能继承原始视频的光影与构图又能通过 ControlNet如 OpenPose 姿态图精确约束动作from diffusers import AnimateDiffVideoToVideoControlNetPipeline from diffusers import AutoencoderKL, ControlNetModel, MotionAdapter, LCMScheduler from controlnet_aux.processor import OpenposeDetector from diffusers.utils import export_to_gif, load_video controlnet ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-openpose, dtypetorch.float16) motion_adapter MotionAdapter.from_pretrained(wangfuyun/AnimateLCM) vae AutoencoderKL.from_pretrained(stabilityai/sd-vae-ft-mse, dtypetorch.float16) pipe AnimateDiffVideoToVideoControlNetPipeline.from_pretrained( SG161222/Realistic_Vision_V5.1_noVAE, motion_adaptermotion_adapter, controlnetcontrolnet, vaevae, ).to(devicecuda, dtypetorch.float16) pipe.scheduler LCMScheduler.from_config(pipe.scheduler.config, beta_schedulelinear) pipe.load_lora_weights(wangfuyun/AnimateLCM, weight_nameAnimateLCM_sd15_t2v_lora.safetensors, adapter_namelcm-lora) pipe.set_adapters([lcm-lora], [0.8]) video load_video(https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/dance.gif) video [frame.convert(RGB) for frame in video] open_pose OpenposeDetector.from_pretrained(lllyasviel/Annotators).to(cuda) conditioning_frames [open_pose(frame) for frame in video] # 逐帧提取姿态 strength 0.8 with torch.inference_mode(): video pipe( videovideo, promptastronaut in space, dancing, negative_promptbad quality, worst quality, jpeg artifacts, ugly, num_inference_steps10, guidance_scale2.0, controlnet_conditioning_scale0.75, # 控制强度略低于 1.0兼顾文本引导 conditioning_framesconditioning_frames, strengthstrength, generatortorch.Generator().manual_seed(42), ).frames[0] export_to_gif(video, animatediff_vid2vid_controlnet.gif, fps8)示例效果中宇航员在太空跳舞的光线与构图均由源视频迁移而来同时姿态由 OpenPose 控制帧锁定——这是视频到视频 结构控制组合的典型应用如换装、动作迁移。运动 LoRA为动画添加特定运动模式Motion LoRA是配合guoyww/animatediff-motion-adapter-v1-5-2使用的一组 LoRA负责为动画注入特定运动类型如animatediff-motion-lora-zoom-out拉远、animatediff-motion-lora-pan-left左摇。加载方式与普通 LoRA 完全一致pipe AnimateDiffPipeline.from_pretrained(model_id, motion_adapteradapter, dtypetorch.float16) pipe.load_lora_weights(guoyww/animatediff-motion-lora-zoom-out, adapter_namezoom-out)使用 PEFT 组合多个 Motion LoRA安装pip install peft后可以叠加多个 Motion LoRA 组合出更复杂的运动如同时拉远 左摇docs/source/en/api/pipelines/animatediff.md 给出了完整代码核心三行pipe.load_lora_weights(diffusers/animatediff-motion-lora-zoom-out, adapter_namezoom-out) pipe.load_lora_weights(diffusers/animatediff-motion-lora-pan-left, adapter_namepan-left) pipe.set_adapters([zoom-out, pan-left], adapter_weights[1.0, 1.0])set_adapters可以分别指定每个 LoRA 的权重用于调节各运动模式的混合强度。FreeInit免训练提升时序一致性FreeInit: Bridging Initialization Gap in Video Diffusion Models 是一种无需额外训练的推理期方法通过迭代精化 latent 初始噪声提升视频扩散模型的时序一致性与整体质量可无缝用于 AnimateDiff、ModelScope、VideoCrafter 等模型。使用方式pipe.enable_free_init(methodbutterworth, use_fast_samplingTrue) output pipe(...) # 正常推理 pipe.disable_free_init()从 src/diffusers/pipelines/free_init_utils.py 的enable_free_init签名可确认完整可调参数参数默认值说明num_iters3FreeInit 噪声重新初始化迭代次数use_fast_samplingFalse是否启用论文中的 Coarse-to-Fine Sampling 加速采样牺牲部分质量换取速度methodbutterworth低通滤波方法可选butterworth/ideal/gaussianorder4butterworth滤波器的阶数越大越接近ideal越小越接近gaussianspatial_stop_frequency0.25空间维度归一化截止频率论文记作d_s范围 0~1temporal_stop_frequency0.25时间维度归一化截止频率论文记作d_t范围 0~1[!WARNING] FreeInit 并非免费 质量提升以额外计算为代价取决于num_iters需要额外采样若干轮。use_fast_samplingTrue可显著提升整体性能代价是质量略低于False但仍优于 vanilla 生成。源码中_get_free_init_freq_filterL73-L115用 FFT 构造三维低通掩码_apply_freq_filterL117 起对噪声做频域滤波实现噪声重初始化disable_free_init只需将_free_init_num_iters置为None。AnimateLCM一致性蒸馏的少步数快速生成AnimateLCM 使用一致性学习策略、解耦蒸馏图像生成先验与运动生成先验得到运动模块检查点wangfuyun/AnimateLCM与配套 LCM LoRA可在极少的步数如 6 步内生成高质量动画adapter MotionAdapter.from_pretrained(wangfuyun/AnimateLCM) pipe AnimateDiffPipeline.from_pretrained(emilianJR/epiCRealism, motion_adapteradapter) pipe.scheduler LCMScheduler.from_config(pipe.scheduler.config, beta_schedulelinear) pipe.load_lora_weights(wangfuyun/AnimateLCM, weight_namesd15_lora_beta.safetensors, adapter_namelcm-lora) pipe.vae.enable_slicing() pipe.enable_model_cpu_offload() output pipe( promptA space rocket with trails of smoke behind it launching into space from the desert, 4k, high resolution, negative_promptbad quality, worse quality, low resolution, num_frames16, guidance_scale1.5, # LCM 通常需要更低的引导系数 num_inference_steps6, # 仅需 6 步 generatortorch.Generator(cpu).manual_seed(0), ) frames output.frames[0] export_to_gif(frames, animatelcm.gif)AnimateLCM 也与既有 Motion LoRA 兼容可同时加载 LCM LoRA 与运动 LoRA 并混合pipe.load_lora_weights(wangfuyun/AnimateLCM, weight_namesd15_lora_beta.safetensors, adapter_namelcm-lora) pipe.load_lora_weights(guoyww/animatediff-motion-lora-tilt-up, adapter_nametilt-up) pipe.set_adapters([lcm-lora, tilt-up], [1.0, 0.8])FreeNoise用噪声重调度生成超长视频FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling 通过噪声重调度 滑动窗口时序注意力 latent 帧加权平均让短视频模型也能生成长视频且支持多提示词插值实现剧情演进。当前支持 FreeNoise 的 AnimateDiff 管线为AnimateDiffPipeline、AnimateDiffControlNetPipeline、AnimateDiffVideoToVideoPipeline、AnimateDiffVideoToVideoControlNetPipeline。启用只需在加载管线后加一行 pipe.enable_free_noise()多提示词插值生成prompt可传帧索引→提示词的字典每个整数键表示该提示词影响力最大的帧位置未出现在字典中的中间帧提示词通过插值生成默认线性插值可通过prompt_interpolation_callback自定义。完整示例见 docs/source/en/api/pipelines/animatediff.md可生成毛毛虫→蝶蛹→蝴蝶→赛博蝴蝶的 256 帧长视频pipe.enable_free_noise(context_length16, context_stride4) pipe.to(cuda) # 或 mps / xpu / cpu prompt { 0: A caterpillar on a leaf, high quality, photorealistic, 40: A caterpillar transforming into a cocoon, on a leaf, near flowers, photorealistic, 80: A cocoon on a leaf, flowers in the background, photorealistic, 120: A cocoon maturing and a butterfly being born, flowers and leaves visible in the background, photorealistic, 160: A beautiful butterfly, vibrant colors, sitting on a leaf, flowers in the background, photorealistic, 200: A beautiful butterfly, flying away in a forest, photorealistic, 240: A cyberpunk butterfly, neon lights, glowing, } negative_prompt bad quality, worst quality, jpeg artifacts output pipe( promptprompt, negative_promptnegative_prompt, num_frames256, guidance_scale2.5, num_inference_steps10, generatortorch.Generator(cpu).manual_seed(0), ) frames output.frames[0] export_to_video(frames, output.mp4, fps16) # 长视频建议导出 mp4从 src/diffusers/pipelines/free_noise_utils.py 可确认enable_free_noise的完整参数context_length默认 16一次处理的视频帧数建议设为运动适配器训练时的最大帧数通常 16/24/32默认取motion_adapter.config.motion_max_seq_length超出会收到警告context_stride默认 4滑动窗口步长。例如context_length16, context_stride4时24 帧会以窗口[0,15], [4,19], [8,23]0 起索引处理weighting_schemeflat/pyramid默认/delayed_reverse_sawtooth控制滑动窗口重叠区域 latent 的加权平均模式noise_typeshuffle_context默认通常效果最佳但可能出现可见运动重复/repeat_context/random。FreeNoise 内存优化SplitInferenceModule由于 FreeNoise 需要同时处理多帧空间/时间注意力、上下采样、ResNet、前馈层等会成为显存瓶颈。由于这些块主要在通道/嵌入维度上运算可以在批次维度上做分块推理AnimateDiff 的批次维度在空间块中是[B×F, H×W, C]跨帧在时间块中是[B×H×W, F, C]跨空间因此分别做空间分块与时间分块是可行的。SplitInferenceModulefree_noise_utils.py正是为此设计的通用包装器按指定维度把输入切成多个 chunk逐个前向后再拼接还原。启用方式# 加载管线与适配器 # ... pipe.enable_free_noise_split_inference() pipe.unet.enable_forward_chunking(16)enable_free_noise_split_inference(spatial_split_size256, temporal_split_size16)L566-L593会把运动模块的投影层与 transformer 块按spatial_split_size沿空间批次维拆分把注意力、ResNet、上下采样块按temporal_split_size沿时间批次维拆分。分块越小显存占用越低但推理越慢反之更快但更耗显存可按自身 VRAM 调整。用from_single_file加载原始格式 MotionAdapterdiffusers0.30.0起支持通过MotionAdapter.from_single_file直接加载 AnimateDiff 原始格式检查点无需预先转换例如加载 LongAnimateDiff 的 32 帧长视频运动模块import torch from diffusers import MotionAdapter ckpt_path https://huggingface.co/Lightricks/LongAnimateDiff/blob/main/lt_long_mm_32_frames.ckpt adapter MotionAdapter.from_single_file(ckpt_path, dtypetorch.float16) pipe AnimateDiffPipeline.from_pretrained(emilianJR/epiCRealism, motion_adapteradapter)该能力由MotionAdapter继承的FromOriginalModelMixin提供见 src/diffusers/models/unets/unet_motion_model.py。API 参考与测试佐证6 条管线均继承自DiffusionPipeline支持通用的模型下载/保存/设备迁移能力并在相应 mixin 中提供了TextualInversionLoaderMixin.load_textual_inversion加载文本反转嵌入StableDiffusionLoraLoaderMixin.load_lora_weights/save_lora_weights加载/保存 LoRAIPAdapterMixin.load_ip_adapter加载 IP-Adapterpipeline_animatediff.py 的类定义可见全部 mixin 继承FreeInitMixin与AnimateDiffFreeNoiseMixin上文所述的 FreeInit 与 FreeNoise 能力FromSingleFileMixinfrom_single_file加载。__call__的返回类型为AnimateDiffPipelineOutputpipeline_output.py包含frames字段return_dictFalse时返回普通元组。每条管线的调用参数、调度器组合与边界条件均可对照 tests/pipelines/animatediff 下的测试文件如 test_animatediff.py、test_animatediff_controlnet.py进行验证与二次开发。进一步阅读调度器速度与质量权衡见 docs/source/en/using-diffusers/schedulers.md跨管线复用组件同一 VAE/文本编码器加载到多条管线见 docs/source/en/using-diffusers/loading.mdLCM LoRA 通用用法见 docs/source/en/using-diffusers/inference_with_lcm_lora.md。【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考