AI 绘画进阶：Stable Diffusion 原理与实战教程

前言

Stable Diffusion 是当前最流行的开源文本生成图像模型之一，由 Stability AI 联合多个研究机构于2022年发布。与闭源的 DALL-E 和 Midjourney 不同，Stable Diffusion 完全开源，允许开发者在本地硬件上运行和二次开发。本教程将从架构原理出发，深入讲解采样器、提示词工程、ControlNet、LoRA训练等核心技术，最终带你构建一个品牌视觉AI生成系统。

---

第一章：Stable Diffusion 架构深度解析

1.1 整体架构概览

Stable Diffusion 属于潜在扩散模型（Latent Diffusion Model, LDM），其核心思想是在低维潜在空间中进行扩散过程，而非直接在像素空间操作。这大大降低了计算成本，使得在消费级GPU上生成高质量图像成为可能。

整个模型由三个核心组件构成：

1. CLIP Text Encoder：将文本提示词编码为语义向量
2. UNet（含 Cross-Attention）：在潜在空间中执行去噪过程
3. VAE（Variational Autoencoder）：负责潜在空间与像素空间的转换

文本提示词 → [CLIP Text Encoder] → 文本嵌入向量
                                          ↓
随机噪声 → [UNet 去噪网络] ← 文本条件注入（Cross-Attention）
              ↓
         潜在表示 → [VAE Decoder] → 最终图像

1.2 VAE：变分自编码器

VAE 的作用是在像素空间和潜在空间之间进行转换：

• Encoder：将 512×512×3 的图像压缩为 64×64×4 的潜在表示（压缩比约 8 倍）
• Decoder：将 64×64×4 的潜在表示还原为 512×512×3 的图像

import torch
from diffusers import AutoencoderKL

# 加载预训练VAE
vae = AutoencoderKL.from_pretrained(
    "stabilityai/sd-vae-ft-mse",
    torch_dtype=torch.float16
).to("cuda")

# 编码：图像 → 潜在空间
def encode_image(image_tensor):
    with torch.no_grad():
        latent = vae.encode(image_tensor).latent_dist.sample()
        latent = latent * 0.18215  # 缩放因子
    return latent

# 解码：潜在空间 → 图像
def decode_latent(latent):
    with torch.no_grad():
        latent = latent / 0.18215
        image = vae.decode(latent).sample
    return image

VAE 的常见问题与优化：

• 颜色偏移：某些 VAE 在深色/浅色区域会出现颜色偏差，可切换为 sd-vae-ft-ema 解决
• 细节丢失：8倍压缩不可避免地丢失高频细节，SDXL 使用了改进的 VAE 架构
• NaN 问题：半精度推理时可能出现 NaN，建议使用 vae.to(torch.float32) 进行解码

1.3 CLIP Text Encoder：文本编码器

SD 1.x 使用 CLIP ViT-L/14 的文本编码器，SD 2.x 使用 OpenCLIP ViT-H/14，SDXL 则同时使用 CLIP ViT-L 和 OpenCLIP ViT-bigG 两个编码器。

from transformers import CLIPTokenizer, CLIPTextModel

# 加载分词器和编码器
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
text_encoder = CLIPTextModel.from_pretrained(
    "openai/clip-vit-large-patch14",
    torch_dtype=torch.float16
).to("cuda")

def encode_prompt(prompt, negative_prompt="", max_length=77):
    # 编码正面提示词
    text_input = tokenizer(
        prompt,
        padding="max_length",
        max_length=max_length,
        truncation=True,
        return_tensors="pt"
    )
    with torch.no_grad():
        text_embeddings = text_encoder(text_input.input_ids.to("cuda"))[0]

    # 编码负面提示词
    uncond_input = tokenizer(
        negative_prompt,
        padding="max_length",
        max_length=max_length,
        truncation=True,
        return_tensors="pt"
    )
    with torch.no_grad():
        uncond_embeddings = text_encoder(uncond_input.input_ids.to("cuda"))[0]

    # 拼接：用于 Classifier-Free Guidance
    text_embeddings = torch.cat([uncond_embeddings, text_embeddings])
    return text_embeddings

关于 77 个 token 的限制：

CLIP 的分词器最大长度为 77 个 token（含开始和结束标记，实际可用 75 个）。超过的部分会被截断。这也是为什么提示词写作中建议将最重要的描述放在前面。

1.4 UNet：去噪网络

UNet 是整个扩散模型的核心，负责在每一步去噪过程中预测噪声残差。SD 的 UNet 包含：

• 下采样块（DownBlock）：逐步降低空间分辨率，提取多尺度特征
• 中间块（MidBlock）：在最低分辨率上进行特征处理
• 上采样块（UpBlock）：逐步恢复空间分辨率
• Cross-Attention 层：将文本条件注入到去噪过程中

from diffusers import UNet2DConditionModel

unet = UNet2DConditionModel.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    subfolder="unet",
    torch_dtype=torch.float16
).to("cuda")

# UNet 的输入输出
# 输入: noisy_latent (B, 4, 64, 64), timestep (B,), encoder_hidden_states (B, 77, 768)
# 输出: noise_pred (B, 4, 64, 64)

Cross-Attention 机制详解：

Cross-Attention 是文本条件注入的关键。在每个 Attention 层中：

• Query（Q）来自图像特征
• Key（K）和 Value（V）来自文本嵌入
• 注意力权重决定了图像每个区域应该关注提示词的哪个部分

# Cross-Attention 的简化实现
import torch.nn as nn

class CrossAttention(nn.Module):
    def __init__(self, dim, context_dim=768):
        super().__init__()
        self.to_q = nn.Linear(dim, dim)
        self.to_k = nn.Linear(context_dim, dim)
        self.to_v = nn.Linear(context_dim, dim)
        self.scale = dim ** -0.5

    def forward(self, x, context):
        q = self.to_q(x)      # 图像特征 → Query
        k = self.to_k(context) # 文本嵌入 → Key
        v = self.to_v(context) # 文本嵌入 → Value

        attn = torch.softmax(q @ k.transpose(-2, -1) * self.scale, dim=-1)
        out = attn @ v
        return out

---

第二章：采样器详解与选择策略

2.1 扩散过程与采样器原理

扩散模型的推理过程是从纯噪声逐步去噪生成图像。采样器（Sampler）决定了这个去噪过程的具体算法。不同的采样器在速度、质量和风格上各有特点。

2.2 常用采样器对比

采样器	类型	步数推荐	特点
Euler	一阶	20-30	简单快速，风格偏柔和
Euler a	一阶随机	20-30	更多随机性，适合创意探索
DPM++ 2M	多步	20-30	高效，细节丰富
DPM++ 2M Karras	多步	15-25	当前最推荐，质量速度平衡
DPM++ SDE	随机	20-30	纹理细腻，适合写实
UniPC	统一	15-20	最新算法，少步数高质量
DDIM	确定性	50-100	经典算法，速度较慢

2.3 采样器代码实现

from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# 切换采样器为 DPM++ 2M Karras
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config,
    use_karras_sigmas=True,
    algorithm_type="dpmsolver++"
)

# 生成图像
image = pipe(
    prompt="a beautiful sunset over mountains, 8k, detailed",
    negative_prompt="blurry, low quality",
    num_inference_steps=25,
    guidance_scale=7.5,
    generator=torch.Generator("cuda").manual_seed(42)
).images[0]

image.save("sunset.png")

2.4 采样器选择建议

• 日常使用：DPM++ 2M Karras（20步），质量速度最佳平衡
• 快速预览：UniPC（10-15步），快速查看构图效果
• 写实人像：DPM++ SDE Karras（25步），皮肤纹理细腻
• 动漫风格：Euler a（25步），线条流畅自然
• 精确复现：DDIM（50步），确定性采样，相同种子相同结果

---

第三章：提示词工程进阶

3.1 提示词权重语法

Stable Diffusion WebUI（AUTOMATIC1111）支持多种权重语法来精细控制提示词的影响：

# 基本权重语法
(word:1.5)    # 将 "word" 的权重提升到 1.5 倍
(word:0.7)    # 将 "word" 的权重降低到 0.7 倍
(word)        # 等价于 (word:1.1)
((word))      # 等价于 (word:1.21)
(((word)))    # 等价于 (word:1.331)

# 注意：权重值建议在 0.5-1.5 之间，过高会导致图像失真

3.2 提示词分段与 BREAK

当提示词超过 75 个 token 时，需要使用 BREAK 关键字进行分段：

a beautiful woman with long hair, wearing red dress, standing in garden,
flowers, sunlight, photorealistic, 8k, detailed face BREAK
detailed eyes, soft skin, natural makeup, professional photography,
bokeh background, cinematic lighting BREAK
masterpiece, best quality, ultra detailed, sharp focus

每段会分别编码为 75 个 token 的向量，然后拼接后送入 UNet。

3.3 负面提示词模板

负面提示词用于排除不想要的元素，以下是常用模板：

# 通用负面提示词
(worst quality:1.4), (low quality:1.4), (normal quality:1.4),
lowres, blurry, text, watermark, signature, username,
bad anatomy, bad hands, extra fingers, fewer fingers,
missing fingers, extra limbs, missing limbs,
mutated hands, fused fingers, too many fingers,
long neck, deformed, disfigured, ugly

# 写实人像专用
3d, cartoon, anime, sketch, drawing, painting,
plastic skin, overexposed, underexposed,
face paint, face tattoo

# 动漫风格专用
photorealistic, 3d render, photograph,
realistic proportions, realistic body

3.4 提示词写作实战技巧

描述层次结构：

[主体] + [外观细节] + [场景/背景] + [风格/媒介] + [质量修饰词]

# 示例
a young woman (主体)
with flowing blonde hair, blue eyes, wearing white linen shirt (外观细节)
in a sunlit meadow with wildflowers (场景)
oil painting style, impressionist (风格)
masterpiece, best quality, highly detailed (质量)

风格关键词速查：

# 写实风格
photorealistic, photo, 8k uhd, dslr, film grain, Fujifilm XT3

# 艺术风格
oil painting, watercolor, acrylic, charcoal, pencil sketch

# 数字艺术
digital art, concept art, artstation, cgsociety, unreal engine

# 动漫风格
anime style, manga, cel shading, Studio Ghibli, Makoto Shinkai

---

第四章：ControlNet 全系列

4.1 ControlNet 简介

ControlNet 是一种为扩散模型添加空间条件控制的方法。它通过在 UNet 上添加旁路网络，允许用户使用边缘图、深度图、姿势图等作为额外控制信号，精确控制生成图像的构图和姿态。

4.2 Canny 边缘检测

Canny ControlNet 使用 Canny 边缘检测算法提取输入图像的边缘信息，生成的图像会严格遵循这些边缘轮廓。

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image
import cv2
import numpy as np

# 加载 ControlNet 模型
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# 边缘检测预处理
def canny_preprocess(image_path, low_threshold=100, high_threshold=200):
    image = cv2.imread(image_path)
    image = cv2.resize(image, (512, 512))
    canny = cv2.Canny(image, low_threshold, high_threshold)
    canny = canny[:, :, None]
    canny = np.concatenate([canny, canny, canny], axis=2)
    return Image.fromarray(canny)

canny_image = canny_preprocess("input.jpg")

# 使用 ControlNet 生成
image = pipe(
    prompt="a futuristic city, cyberpunk, neon lights, 8k",
    image=canny_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    controlnet_conditioning_scale=0.8
).images[0]

4.3 Depth 深度估计

Depth ControlNet 使用深度图来控制图像的空间结构，适合保持场景的前后景关系。

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from controlnet_aux import MidasDetector

# 加载深度估计器
depth_estimator = MidasDetector.from_pretrained(
    "lllyasviel/ControlNet"
)

# 加载 ControlNet-Depth 模型
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-depth",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# 深度图预处理
input_image = load_image("room.jpg").resize((512, 512))
depth_map = depth_estimator(input_image)

# 生成
image = pipe(
    prompt="a luxury modern living room, interior design, 8k render",
    image=depth_map,
    controlnet_conditioning_scale=0.7
).images[0]

4.4 OpenPose 人体姿势

OpenPose ControlNet 可以提取人体骨骼关键点，控制生成人物的姿势和动作。

from controlnet_aux import OpenposeDetector

# 加载 OpenPose 检测器
openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-openpose",
    torch_dtype=torch.float16
)

# 提取姿势
input_image = load_image("dancer.jpg").resize((512, 512))
pose_image = openpose(input_image)

# 使用姿势控制生成
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

image = pipe(
    prompt="a ballet dancer in a beautiful dress, studio lighting, professional photo",
    image=pose_image,
    controlnet_conditioning_scale=1.0
).images[0]

4.5 Lineart 线稿控制

Lineart ControlNet 适合将草稿或线稿转换为精细的彩色图像，是插画师的利器。

from controlnet_aux import LineartDetector

lineart = LineartDetector.from_pretrained("lllyasviel/ControlNet")

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-lineart",
    torch_dtype=torch.float16
)

# 提取线稿
sketch = load_image("sketch.png").resize((512, 512))
lineart_image = lineart(sketch)

image = pipe(
    prompt="a beautiful anime girl, detailed, masterpiece",
    image=lineart_image,
    controlnet_conditioning_scale=0.8
).images[0]

4.6 IP-Adapter 图像提示适配器

IP-Adapter 是一种新颖的条件控制方式，允许将参考图像的风格和内容"适配"到生成过程中，无需重新训练模型。

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# 加载 IP-Adapter
pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="models",
    weight_name="ip-adapter_sd15.bin"
)

# 使用参考图像
ref_image = load_image("style_ref.jpg")

image = pipe(
    prompt="a landscape painting",
    ip_adapter_image=ref_image,
    ip_adapter_scale=0.7,  # 控制参考图像的影响程度
    num_inference_steps=30
).images[0]

4.7 多 ControlNet 联合使用

实际项目中，往往需要同时使用多个 ControlNet 来实现精确控制：

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel

# 加载多个 ControlNet
controlnet_canny = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
)
controlnet_openpose = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=[controlnet_canny, controlnet_openpose],  # 列表形式
    torch_dtype=torch.float16
).to("cuda")

# 同时使用边缘图和姿势图
image = pipe(
    prompt="a fashion model, studio lighting",
    image=[canny_image, pose_image],
    controlnet_conditioning_scale=[0.6, 0.8]  # 分别控制权重
).images[0]

---

第五章：LoRA 训练实战

5.1 LoRA 原理

LoRA（Low-Rank Adaptation）是一种参数高效微调方法。它在 UNet 的 Cross-Attention 层旁边添加低秩矩阵，只训练这些新增参数（通常只有几十 MB），而保持原始模型参数不变。

原始权重 W (768×768) + ΔW = W + A×B
其中 A: (768×r), B: (r×768), r << 768（通常 r=4~128）

5.2 训练数据准备

import os
from PIL import Image
import json

# 目录结构
# dataset/
#   ├── 10_mycharacter/    # 数字前缀决定训练优先级
#   │   ├── img001.png
#   │   ├── img001.txt     # 对应的标签文件
#   │   ├── img002.png
#   │   └── img002.txt
#   └── metadata.json

def prepare_dataset(image_dir, output_dir, trigger_word="mychar"):
    """准备训练数据集"""
    os.makedirs(output_dir, exist_ok=True)

    # 使用 BLIP 自动生成标签
    from transformers import BlipProcessor, BlipForConditionalGeneration
    processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
    model = BlipForConditionalGeneration.from_pretrained(
        "Salesforce/blip-image-captioning-large"
    ).to("cuda")

    for img_file in os.listdir(image_dir):
        if not img_file.endswith(('.png', '.jpg', '.jpeg')):
            continue

        # 处理图像
        img_path = os.path.join(image_dir, img_file)
        image = Image.open(img_path).convert("RGB")
        image = image.resize((512, 512))

        # 自动生成描述
        inputs = processor(image, return_tensors="pt").to("cuda")
        caption = model.generate(**inputs, max_length=75)
        caption = processor.decode(caption[0], skip_special_tokens=True)

        # 添加触发词
        caption = f"{trigger_word}, {caption}"

        # 保存
        out_img = os.path.join(output_dir, img_file)
        out_txt = os.path.join(output_dir, img_file.rsplit('.', 1)[0] + '.txt')
        image.save(out_img)
        with open(out_txt, 'w') as f:
            f.write(caption)

        print(f"Processed: {img_file} -> {caption}")

5.3 训练配置

# kohya_ss 训练配置文件 (toml格式)
[model]
pretrained_model_name_or_path = "runwayml/stable-diffusion-v1-5"
output_name = "my_character_lora"
output_dir = "./output"

[training]
train_batch_size = 1
gradient_accumulation_steps = 1
learning_rate = 1e-4
unet_lr = 1e-4
text_encoder_lr = 5e-5
max_train_epochs = 10
network_dim = 32          # LoRA 秩，越大越精细但文件越大
network_alpha = 16        # 通常设为 dim 的一半
resolution = 512
mixed_precision = "fp16"

[scheduler]
lr_scheduler = "cosine_with_restarts"
lr_scheduler_num_cycles = 3

[optimizer]
optimizer_type = "AdamW8bit"

[dataset]
batch_size = 1
cache_latents = true
flip_aug = true
color_aug = false

5.4 使用 kohya_ss 训练

# 安装 kohya_ss
git clone https://github.com/kohya-ss/sd-scripts.git
cd sd-scripts
pip install -r requirements.txt

# 准备数据
python finetune/tag_images_by_wd14_tagger.py \
    --batch_size=8 \
    --repo_id=SmilingWolf/wd-v1-4-vit-tagger-v2 \
    --thresh=0.35 \
    --caption_extension=".txt" \
    ./dataset/10_mycharacter

# 开始训练 LoRA
accelerate launch --num_cpu_threads_per_process=1 train_network.py \
    --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
    --train_data_dir="./dataset" \
    --output_dir="./output" \
    --output_name="my_character_lora" \
    --network_module=networks.lora \
    --network_dim=32 \
    --network_alpha=16 \
    --resolution=512 \
    --train_batch_size=1 \
    --max_train_epochs=10 \
    --learning_rate=1e-4 \
    --unet_lr=1e-4 \
    --text_encoder_lr=5e-5 \
    --lr_scheduler="cosine_with_restarts" \
    --lr_scheduler_num_cycles=3 \
    --mixed_precision="fp16" \
    --cache_latents \
    --optimizer_type="AdamW8bit" \
    --flip_aug \
    --save_every_n_epochs=2

5.5 LoRA 效果调优

# 使用训练好的 LoRA
from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# 加载 LoRA
pipe.load_lora_weights("./output", weight_name="my_character_lora.safetensors")

# 使用触发词生成
image = pipe(
    prompt="mychar, a portrait, professional photography, 8k",
    negative_prompt="low quality, blurry",
    num_inference_steps=30,
    guidance_scale=7.5,
    cross_attention_kwargs={"scale": 0.8}  # 控制 LoRA 强度
).images[0]

调优建议：

• LoRA 文件大小：10-200MB，过小可能效果不明显，过大可能过拟合
• 训练轮数：通常 5-15 轮，观察 loss 曲线下降趋于平稳即可停止
• 学习率：过高导致过拟合（图像失真），过低导致欠拟合（效果不明显）
• 数据量：最少 15-20 张高质量图片，理想 50-100 张

---

第六章：图生图与 Inpainting 高级技巧

6.1 图生图（Image-to-Image）

图生图允许以一张现有图像为基础，通过文本提示词引导风格转换或内容修改。

from diffusers import StableDiffusionImg2ImgPipeline

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

init_image = load_image("photo.jpg").resize((512, 512))

# strength 参数控制修改程度：0.0=完全保留, 1.0=完全重绘
image = pipe(
    prompt="a beautiful oil painting of a landscape, impressionist style",
    image=init_image,
    strength=0.75,        # 修改强度
    guidance_scale=7.5,
    num_inference_steps=30
).images[0]

strength 参数选择指南：

• 0.3-0.4：轻微风格化，保留原图结构和细节
• 0.5-0.6：中等修改，保留大致构图，改变风格
• 0.7-0.8：大幅重绘，只保留颜色和轮廓暗示
• 0.9-1.0：几乎完全重绘，原图仅作为颜色参考

6.2 Inpainting 局部重绘

Inpainting 可以只修改图像的特定区域，保持其余部分不变。

from diffusers import StableDiffusionInpaintPipeline

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16
).to("cuda")

# 加载原图和遮罩
image = load_image("photo.png").resize((512, 512))
mask = load_image("mask.png").resize((512, 512))  # 白色区域为需要修改的部分

image = pipe(
    prompt="a beautiful garden with flowers",
    image=image,
    mask_image=mask,
    num_inference_steps=30,
    guidance_scale=7.5
).images[0]

6.3 Inpainting 进阶技巧

# 1. 使用 Outpainting 扩展图像
from diffusers import StableDiffusionInpaintPipeline
import numpy as np

def outpaint(pipe, original_image, direction="right", pixels=256):
    """向指定方向扩展图像"""
    w, h = original_image.size
    new_w = w + pixels if direction in ["left", "right"] else w
    new_h = h + pixels if direction in ["up", "down"] else h

    new_image = Image.new("RGB", (new_w, new_h), (0, 0, 0))
    mask = Image.new("L", (new_w, new_h), 255)

    # 粘贴原图并设置遮罩
    if direction == "right":
        new_image.paste(original_image, (0, 0))
        mask.paste(Image.new("L", (w, h), 0), (0, 0))

    return pipe(
        prompt="continue the scene naturally",
        image=new_image.resize((512, 512)),
        mask_image=mask.resize((512, 512)),
    ).images[0]

# 2. 逐区域精修工作流
def iterative_inpaint(pipe, image, regions, prompts):
    """对图像的不同区域分别进行精修"""
    for region, prompt in zip(regions, prompts):
        mask = create_mask_from_region(image.size, region)
        image = pipe(
            prompt=prompt,
            image=image,
            mask_image=mask,
            strength=0.6
        ).images[0]
    return image

---

第七章：模型融合与权重合并

7.1 Checkpoint Merger 原理

模型融合是将两个或多个模型的权重按比例混合，产生具有两者特征的新模型。这是一种无需训练即可获得新风格的方法。

# 使用 diffusers 进行模型合并
from diffusers import StableDiffusionPipeline
import torch

def merge_models(model_a_path, model_b_path, alpha=0.5, output_path="merged"):
    """
    合并两个模型
    alpha: 模型A的权重比例，0.0=完全使用B，1.0=完全使用A
    """
    pipe_a = StableDiffusionPipeline.from_pretrained(model_a_path, torch_dtype=torch.float16)
    pipe_b = StableDiffusionPipeline.from_pretrained(model_b_path, torch_dtype=torch.float16)

    # 合并 UNet 权重
    state_dict_a = pipe_a.unet.state_dict()
    state_dict_b = pipe_b.unet.state_dict()

    merged_state_dict = {}
    for key in state_dict_a:
        if key in state_dict_b:
            merged_state_dict[key] = (
                alpha * state_dict_a[key] + (1 - alpha) * state_dict_b[key]
            )
        else:
            merged_state_dict[key] = state_dict_a[key]

    pipe_a.unet.load_state_dict(merged_state_dict)
    pipe_a.save_pretrained(output_path)
    print(f"模型已保存至 {output_path}")

# 使用示例
merge_models(
    "runwayml/stable-diffusion-v1-5",    # 写实模型
    "Linaqruf/anything-v3.0",            # 动漫模型
    alpha=0.6,                            # 60%写实 + 40%动漫
    output_path="./merged_model"
)

7.2 常见融合策略

融合策略	Alpha 范围	效果
写实+动漫	0.5-0.7	半写实风格，适合二次元写实化
风景+人像	0.4-0.6	风景中的人物更协调
色彩+线稿	0.3-0.5	丰富色彩但保持线条感
三模型融合	分段 alpha	需要多次两两融合

7.3 Add Difference 方法

def add_difference(model_a, model_b, model_c, alpha=0.5, output_path="merged"):
    """
    Add Difference: A + alpha * (B - C)
    常见用法：A=基础模型, B=目标风格模型, C=基础模型的另一个版本
    """
    pipe_a = StableDiffusionPipeline.from_pretrained(model_a, torch_dtype=torch.float16)
    pipe_b = StableDiffusionPipeline.from_pretrained(model_b, torch_dtype=torch.float16)
    pipe_c = StableDiffusionPipeline.from_pretrained(model_c, torch_dtype=torch.float16)

    state_a = pipe_a.unet.state_dict()
    state_b = pipe_b.unet.state_dict()
    state_c = pipe_c.unet.state_dict()

    merged = {}
    for key in state_a:
        if key in state_b and key in state_c:
            merged[key] = state_a[key] + alpha * (state_b[key] - state_c[key])
        else:
            merged[key] = state_a[key]

    pipe_a.unet.load_state_dict(merged)
    pipe_a.save_pretrained(output_path)

---

第八章：SDXL 与 SD3 新特性对比

8.1 SDXL 架构改进

SDXL 相比 SD 1.5 有显著的架构升级：

• 双文本编码器：CLIP ViT-L (768维) + OpenCLIP ViT-bigG (1280维)
• 更大的 UNet：3.5B 参数（SD 1.5 约 860M）
• 改进的 VAE：更好的高频细节保留
• Refiner 模型：专门用于高分辨率精修
• 原生 1024×1024：不再局限于 512×512

from diffusers import DiffusionPipeline
import torch

# SDXL 基础模型
pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
).to("cuda")

# SDXL 生成（原生 1024x1024）
image = pipe(
    prompt="a majestic mountain landscape, golden hour, 8k uhd",
    negative_prompt="low quality, blurry",
    width=1024,
    height=1024,
    num_inference_steps=30,
    guidance_scale=7.5
).images[0]

# 使用 Refiner 进行精修
from diffusers import StableDiffusionXLImg2ImgPipeline

refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
).to("cuda")

# 两阶段生成
image = refiner(
    prompt="a majestic mountain landscape, golden hour, 8k uhd",
    image=image,
    strength=0.3,
    num_inference_steps=30
).images[0]

8.2 SD3 新特性

SD3（Stable Diffusion 3）引入了多项革命性改进：

• MMDiT 架构：多模态 Diffusion Transformer，替代传统 UNet
• 三重文本编码器：CLIP ViT-L + OpenCLIP ViT-bigG + T5-XXL
• Flow Matching：替代传统 DDPM 扩散过程
• Rectified Flow：更直的采样路径，更少的步数

from diffusers import StableDiffusion3Pipeline
import torch

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium",
    torch_dtype=torch.float16
).to("cuda")

image = pipe(
    prompt="a photorealistic cat wearing a top hat, studio lighting, 8k",
    negative_prompt="low quality, distorted",
    num_inference_steps=28,
    guidance_scale=7.0
).images[0]

8.3 版本选择建议

特性	SD 1.5	SDXL	SD3
推理显存	4-6 GB	8-12 GB	12-16 GB
原生分辨率	512×512	1024×1024	1024×1024
生态成熟度	★★★★★	★★★★	★★★
文字生成	差	一般	良好
速度	最快	中等	较慢
推荐场景	快速原型/低配	通用生产	高质量需求

---

第九章：批量生成与自动化工作流

9.1 Python 批量生成脚本

import os
import json
import torch
from diffusers import StableDiffusionPipeline
from datetime import datetime

class BatchGenerator:
    def __init__(self, model_path, device="cuda"):
        self.pipe = StableDiffusionPipeline.from_pretrained(
            model_path,
            torch_dtype=torch.float16
        ).to(device)
        self.pipe.enable_attention_slicing()  # 节省显存

    def generate_batch(self, prompts, output_dir, batch_config=None):
        """批量生成图像"""
        os.makedirs(output_dir, exist_ok=True)
        config = batch_config or {
            "num_inference_steps": 30,
            "guidance_scale": 7.5,
            "width": 512,
            "height": 512,
            "num_images_per_prompt": 1
        }

        results = []
        for i, prompt_data in enumerate(prompts):
            prompt = prompt_data.get("prompt", "")
            negative = prompt_data.get("negative_prompt", "")
            seed = prompt_data.get("seed", 42)

            generator = torch.Generator("cuda").manual_seed(seed)

            image = self.pipe(
                prompt=prompt,
                negative_prompt=negative,
                generator=generator,
                **config
            ).images[0]

            filename = f"{i:04d}_{datetime.now().strftime('%H%M%S')}.png"
            filepath = os.path.join(output_dir, filename)
            image.save(filepath)

            results.append({
                "index": i,
                "prompt": prompt,
                "seed": seed,
                "file": filepath
            })
            print(f"[{i+1}/{len(prompts)}] 生成完成: {filename}")

        # 保存生成日志
        with open(os.path.join(output_dir, "generation_log.json"), "w") as f:
            json.dump(results, f, indent=2, ensure_ascii=False)

        return results

# 使用示例
generator = BatchGenerator("runwayml/stable-diffusion-v1-5")

prompts = [
    {"prompt": "a red sports car on highway, cinematic", "seed": 100},
    {"prompt": "a cozy coffee shop interior, warm lighting", "seed": 200},
    {"prompt": "a futuristic robot, detailed, 8k", "seed": 300},
]

generator.generate_batch(prompts, "./batch_output")

9.2 ComfyUI 自动化工作流

import json
import requests

def run_comfyui_workflow(workflow_json, prompt_text, output_path):
    """通过 API 调用 ComfyUI 工作流"""
    # 加载工作流模板
    with open(workflow_json, 'r') as f:
        workflow = json.load(f)

    # 修改提示词
    workflow["6"]["inputs"]["text"] = prompt_text

    # 提交到 ComfyUI API
    response = requests.post(
        "http://127.0.0.1:8188/prompt",
        json={"prompt": workflow}
    )

    prompt_id = response.json()["prompt_id"]

    # 轮询等待完成
    while True:
        status = requests.get(f"http://127.0.0.1:8188/history/{prompt_id}")
        if prompt_id in status.json():
            break
        time.sleep(1)

    # 下载结果
    result = status.json()[prompt_id]
    for node_id, node_output in result["outputs"].items():
        if "images" in node_output:
            for img_info in node_output["images"]:
                img_url = f"http://127.0.0.1:8188/view?filename={img_info['filename']}"
                img_data = requests.get(img_url).content
                with open(output_path, 'wb') as f:
                    f.write(img_data)

9.3 定时任务与队列系统

from celery import Celery
import torch

# Celery 配置
app = Celery('sd_tasks', broker='redis://localhost:6379/0')

# 全局加载模型（避免重复加载）
pipe = None

def get_pipe():
    global pipe
    if pipe is None:
        pipe = StableDiffusionPipeline.from_pretrained(
            "runwayml/stable-diffusion-v1-5",
            torch_dtype=torch.float16
        ).to("cuda")
        pipe.enable_attention_slicing()
    return pipe

@app.task(bind=True, max_retries=3)
def generate_image_task(self, prompt, negative_prompt="", seed=42, **kwargs):
    """异步图像生成任务"""
    try:
        p = get_pipe()
        generator = torch.Generator("cuda").manual_seed(seed)

        image = p(
            prompt=prompt,
            negative_prompt=negative_prompt,
            generator=generator,
            num_inference_steps=kwargs.get("steps", 30),
            guidance_scale=kwargs.get("cfg", 7.5)
        ).images[0]

        output_path = f"/outputs/{self.request.id}.png"
        image.save(output_path)
        return {"status": "success", "path": output_path}

    except Exception as exc:
        self.retry(exc=exc, countdown=60)

---

第十章：AI 绘画商业化应用

10.1 电商产品图生成

class EcommerceImageGenerator:
    """电商产品图AI生成系统"""

    def __init__(self, model_path):
        self.pipe = StableDiffusionPipeline.from_pretrained(
            model_path, torch_dtype=torch.float16
        ).to("cuda")

    def generate_product_shot(self, product_desc, style="studio"):
        """生成产品展示图"""
        style_prompts = {
            "studio": "professional product photography, white background, studio lighting, commercial quality, 8k",
            "lifestyle": "lifestyle photography, natural lighting, interior setting, warm tones",
            "outdoor": "outdoor product photography, nature background, golden hour lighting",
        }

        prompt = f"{product_desc}, {style_prompts.get(style, style_prompts['studio'])}"
        negative = "low quality, blurry, distorted, watermark, text"

        return self.pipe(
            prompt=prompt,
            negative_prompt=negative,
            width=1024, height=1024,
            num_inference_steps=35,
            guidance_scale=7.5
        ).images[0]

    def generate_batch_products(self, products, style="studio"):
        """批量生成产品图"""
        results = {}
        for product in products:
            image = self.generate_product_shot(product, style)
            results[product] = image
        return results

10.2 游戏美术资产生成

class GameAssetGenerator:
    """游戏美术资产AI生成"""

    def generate_character_concept(self, description, art_style="fantasy"):
        """生成角色概念图"""
        style_map = {
            "fantasy": "fantasy character concept art, detailed armor, magical elements, artstation",
            "sci-fi": "sci-fi character design, futuristic suit, cyberpunk, concept art",
            "pixel": "pixel art character, 16-bit style, retro game, sprite sheet",
        }

        prompt = f"{description}, {style_map[art_style]}, character design sheet, multiple angles"
        return self.pipe(prompt=prompt, width=1024, height=768).images[0]

    def generate_environment(self, description, mood="neutral"):
        """生成游戏场景"""
        mood_map = {
            "dark": "dark atmospheric, moody lighting, ominous",
            "bright": "bright colorful, cheerful, vibrant",
            "neutral": "balanced lighting, detailed environment",
        }

        prompt = f"{description}, {mood_map[mood]}, game environment concept art, 4k"
        return self.pipe(prompt=prompt, width=1024, height=576).images[0]

---

第十一章：实战项目——构建品牌视觉 AI 生成系统

11.1 项目概述

我们将构建一个完整的品牌视觉AI生成系统，能够：

1. 根据品牌调性自动生成视觉素材
2. 保持品牌一致性（颜色、风格、构图）
3. 批量生成不同场景的品牌素材
4. 支持多种输出格式和尺寸

11.2 系统架构

import os
import json
import yaml
from dataclasses import dataclass
from typing import List, Dict, Optional
from PIL import Image
import torch
from diffusers import (
    StableDiffusionPipeline,
    StableDiffusionControlNetPipeline,
    ControlNetModel,
    DPMSolverMultistepScheduler
)

@dataclass
class BrandConfig:
    """品牌配置"""
    name: str
    trigger_word: str  # LoRA 触发词
    color_palette: List[str]  # 品牌主色
    style_keywords: List[str]  # 风格关键词
    negative_prompts: List[str]  # 品牌禁止元素
    lora_path: Optional[str] = None  # 品牌 LoRA 路径
    lora_weight: float = 0.8

class BrandVisualSystem:
    """品牌视觉AI生成系统"""

    def __init__(self, config_path: str):
        with open(config_path, 'r', encoding='utf-8') as f:
            config = yaml.safe_load(f)

        self.brand = BrandConfig(**config['brand'])
        self.model_path = config.get('model_path', 'runwayml/stable-diffusion-v1-5')
        self.output_dir = config.get('output_dir', './brand_output')

        self._load_models()

    def _load_models(self):
        """加载模型和LoRA"""
        print(f"正在加载模型: {self.model_path}")
        self.pipe = StableDiffusionPipeline.from_pretrained(
            self.model_path,
            torch_dtype=torch.float16
        ).to("cuda")

        # 设置采样器
        self.pipe.scheduler = DPMSolverMultistepScheduler.from_config(
            self.pipe.scheduler.config,
            use_karras_sigmas=True
        )

        # 加载品牌 LoRA
        if self.brand.lora_path and os.path.exists(self.brand.lora_path):
            self.pipe.load_lora_weights(self.brand.lora_path)
            print(f"品牌 LoRA 已加载: {self.brand.lora_path}")

        self.pipe.enable_attention_slicing()

    def _build_prompt(self, scene: str, extras: List[str] = None) -> str:
        """构建品牌化提示词"""
        parts = [
            self.brand.trigger_word,
            scene,
            ", ".join(self.brand.style_keywords),
            f"color palette: {', '.join(self.brand.color_palette)}",
        ]
        if extras:
            parts.extend(extras)
        parts.append("masterpiece, best quality, highly detailed, 8k")

        return ", ".join(parts)

    def _build_negative(self) -> str:
        """构建品牌化负面提示词"""
        base_negative = [
            "worst quality", "low quality", "blurry", "text",
            "watermark", "signature", "deformed"
        ]
        return ", ".join(base_negative + self.brand.negative_prompts)

    def generate(self, scene: str, size: tuple = (1024, 1024),
                 seed: int = None, extras: List[str] = None) -> Image.Image:
        """生成品牌视觉素材"""
        prompt = self._build_prompt(scene, extras)
        negative = self._build_negative()

        generator = None
        if seed is not None:
            generator = torch.Generator("cuda").manual_seed(seed)

        print(f"生成中: {scene}")
        print(f"提示词: {prompt[:100]}...")

        image = self.pipe(
            prompt=prompt,
            negative_prompt=negative,
            width=size[0],
            height=size[1],
            num_inference_steps=30,
            guidance_scale=7.5,
            generator=generator,
            cross_attention_kwargs={"scale": self.brand.lora_weight}
        ).images[0]

        return image

    def generate_marketing_set(self, base_scene: str, variants: List[str],
                                seed_base: int = 42) -> Dict[str, Image.Image]:
        """生成营销素材套装"""
        results = {}
        for i, variant in enumerate(variants):
            scene = f"{base_scene}, {variant}"
            image = self.generate(scene, seed=seed_base + i)
            key = variant.replace(" ", "_")[:30]
            results[key] = image

            # 保存
            path = os.path.join(self.output_dir, f"{self.brand.name}_{key}.png")
            os.makedirs(self.output_dir, exist_ok=True)
            image.save(path)
            print(f"已保存: {path}")

        return results

    def generate_social_media_set(self, post_text: str, platforms: List[str] = None):
        """生成社交媒体素材套装"""
        platforms = platforms or ["instagram", "twitter", "wechat"]

        size_map = {
            "instagram": (1080, 1080),
            "twitter": (1200, 675),
            "wechat": (900, 500),
            "xiaohongshu": (1080, 1440),
            "douyin": (1080, 1920),
        }

        results = {}
        for platform in platforms:
            size = size_map.get(platform, (1080, 1080))
            scene = f"social media post, {post_text}"
            image = self.generate(scene, size=size)
            results[platform] = image

            path = os.path.join(self.output_dir, f"social_{platform}.png")
            image.save(path)

        return results


# 配置文件示例
config_yaml = """
model_path: "runwayml/stable-diffusion-v1-5"
output_dir: "./brand_output"

brand:
  name: "TechBrand"
  trigger_word: "techbrand_style"
  color_palette:
    - "#0066CC"
    - "#FF6B00"
    - "#FFFFFF"
    - "#1A1A2E"
  style_keywords:
    - "modern minimalist"
    - "professional"
    - "tech aesthetic"
    - "clean design"
  negative_prompts:
    - "cartoon"
    - "anime"
    - "hand drawn"
    - "sketch"
  lora_path: "./lora/techbrand_v1.safetensors"
  lora_weight: 0.75
"""

# 使用示例
if __name__ == "__main__":
    # 保存配置
    with open("brand_config.yaml", "w") as f:
        f.write(config_yaml)

    # 初始化系统
    system = BrandVisualSystem("brand_config.yaml")

    # 生成产品宣传图
    product_image = system.generate(
        "a sleek smartphone floating in space, dramatic lighting, product showcase",
        size=(1024, 1024),
        seed=42
    )

    # 生成营销套装
    marketing_images = system.generate_marketing_set(
        base_scene="tech company promotional banner",
        variants=[
            "with team collaboration theme",
            "with innovation and future theme",
            "with customer success theme",
        ]
    )

    # 生成社交媒体素材
    social_images = system.generate_social_media_set(
        post_text="launching new AI product, exciting announcement",
        platforms=["instagram", "twitter", "wechat", "xiaohongshu"]
    )

    print("所有素材生成完成！")

11.3 运行效果与优化

# 安装依赖
pip install diffusers transformers accelerate pyyaml pillow

# 运行系统
python brand_visual_system.py

# 输出目录结构
# brand_output/
#   ├── TechBrand_with_team_collaboration_theme.png
#   ├── TechBrand_with_innovation_and_future_theme.png
#   ├── TechBrand_with_customer_success_theme.png
#   ├── social_instagram.png
#   ├── social_twitter.png
#   ├── social_wechat.png
#   └── social_xiaohongshu.png

---

总结与展望

本教程从 Stable Diffusion 的底层架构出发，系统讲解了：

1. 架构原理：理解 VAE、UNet、CLIP 三大组件的工作机制
2. 采样器选择：根据场景选择最合适的采样算法
3. 提示词工程：掌握权重语法、分段技巧、负面提示词模板
4. ControlNet：利用边缘、深度、姿势等条件精确控制生成
5. LoRA 训练：从数据准备到训练调优的完整流程
6. 模型融合：无需训练即可创造新风格的方法
7. SDXL/SD3：新一代模型的架构改进与选择策略
8. 自动化工作流：批量生成、API调用、队列系统
9. 商业化应用：电商、游戏、品牌视觉的实际落地方案

AI 绘画正在从"玩具"变为"工具"，从"灵感来源"变为"生产力"。掌握这些技术，你就能在创作和商业领域中充分释放 AI 绘画的潜力。

推荐学习资源

• Hugging Face Diffusers 文档：https://huggingface.co/docs/diffusers
• AUTOMATIC1111 WebUI：https://github.com/AUTOMATIC1111/stable-diffusion-webui
• ComfyUI：https://github.com/comfyanonymous/ComfyUI
• kohya_ss 训练工具：https://github.com/kohya-ss/sd-scripts
• CivitAI 模型社区：https://civitai.com

---

本教程内容为原创撰写，基于作者对 Stable Diffusion 生态的实际使用和项目经验整理而成。如有技术细节更新，请参考官方文档获取最新信息。