Docker + Kubernetes AI应用部署完全教程
前言
随着人工智能技术的飞速发展，越来越多的AI模型从实验室走向生产环境。然而，将一个在Jupyter Notebook中运行良好的AI模型部署到生产环境，面临着环境依赖复杂、GPU资源管理困难、服务弹性伸缩需求等诸多挑战。Docker和Kubernetes作为云原生技术的核心，为AI应用的部署提供了标准化、可扩展的解决方案。
本教程将从零基础出发，系统讲解如何使用Docker和Kubernetes部署AI应用，涵盖从基础概念到高级实战的完整知识体系，帮助AI工程师和DevOps工程师掌握AI应用容器化部署的核心技能。
第一章：为什么AI应用需要容器化部署
1.1 AI应用部署的传统痛点
在容器化技术普及之前，AI应用的部署通常面临以下问题：
环境依赖地狱：AI应用依赖大量的Python库、CUDA驱动、cuDNN等组件，不同模型可能需要不同版本的TensorFlow、PyTorch，甚至不同版本的CUDA。在一个服务器上同时部署多个AI服务时，版本冲突几乎不可避免。
典型的AI应用依赖链：
Python 3.9 → PyTorch 2.0 → CUDA 11.8 → cuDNN 8.6 → NVIDIA Driver 520.x
Python 3.10 → TensorFlow 2.12 → CUDA 12.0 → cuDNN 8.8 → NVIDIA Driver 525.x
GPU资源浪费：传统的物理服务器部署方式，GPU资源往往无法充分利用。一台配备8张A100的服务器，可能只运行了一个占用2张卡的推理服务，剩余6张卡处于闲置状态。
弹性伸缩困难：AI服务的流量往往是波动的。白天用户活跃时需要大量GPU算力，深夜则几乎空闲。传统的静态部署无法应对这种波动，要么过度配置浪费资源，要么配置不足导致服务降级。
部署一致性问题："在我机器上能跑"是AI工程师最常说的话。开发环境、测试环境、生产环境的差异，经常导致模型在不同环境中表现不一致。
1.2 容器化如何解决这些问题
容器化技术通过以下方式解决上述痛点：
环境隔离：每个容器拥有独立的文件系统、网络和进程空间，不同服务的依赖互不干扰。你可以在同一台服务器上运行需要CUDA 11.8的PyTorch服务和需要CUDA 12.0的TensorFlow服务。
标准化交付：Docker镜像将应用代码、依赖库、运行时环境打包成一个不可变的交付物，确保从开发到生产的环境一致性。
资源编排：Kubernetes提供了强大的资源调度能力，可以精确分配GPU、CPU、内存等资源，并根据负载自动伸缩。
快速扩缩容：通过Kubernetes的HPA（Horizontal Pod Autoscaler）和KEDA等组件，可以在几分钟内完成服务的扩容或缩容。
1.3 云原生AI部署架构概览
一个完整的云原生AI部署架构通常包含以下层次：
┌─────────────────────────────────────────────────────────┐
│ 用户请求层 │
│ Ingress / API Gateway │
├─────────────────────────────────────────────────────────┤
│ 服务编排层 │
│ Kubernetes (Deployment, Service, HPA) │
├─────────────────────────────────────────────────────────┤
│ 模型服务层 │
│ KServe / Triton Inference Server / vLLM │
├─────────────────────────────────────────────────────────┤
│ 容器运行时 │
│ Docker / containerd │
├─────────────────────────────────────────────────────────┤
│ 资源层 │
│ GPU (NVIDIA A100/H100) + CPU + 内存 │
├─────────────────────────────────────────────────────────┤
│ 监控层 │
│ Prometheus + Grafana + ELK Stack │
└─────────────────────────────────────────────────────────┘
第二章：Docker基础回顾
2.1 Docker核心概念
Docker是一个开源的容器化平台，它的核心概念包括：
镜像（Image）：镜像是一个只读的模板，包含了运行应用所需的一切——代码、运行时、库、环境变量和配置文件。你可以把镜像想象成一个"类"，而容器就是它的"实例"。
容器（Container）：容器是镜像的运行实例。它是一个轻量级、独立、可执行的软件包。容器之间相互隔离，共享宿主机的操作系统内核。
仓库（Registry）：Docker镜像的存储中心，最常用的是Docker Hub，企业通常会搭建私有仓库如Harbor。
2.2 Dockerfile编写最佳实践
Dockerfile是构建Docker镜像的"配方"。对于AI应用，以下是一个优化的Dockerfile示例：
# 使用NVIDIA CUDA基础镜像
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
# 设置非交互模式，避免安装过程中的交互提示
ENV DEBIAN_FRONTEND=noninteractive
# 安装系统依赖
RUN apt-get update && apt-get install -y \
python3.10 \
python3-pip \
git \
curl \
&& rm -rf /var/lib/apt/lists/*
# 设置工作目录
WORKDIR /app
# 先复制依赖文件，利用Docker缓存层
COPY requirements.txt .
# 安装Python依赖
RUN pip3 install --no-cache-dir -r requirements.txt
# 复制应用代码
COPY . .
# 暴露端口
EXPOSE 8000
# 设置启动命令
CMD ["python3", "serve.py"]
关键优化点：
- 基础镜像选择：NVIDIA提供了多种CUDA镜像变体。
runtime版本只包含运行时库，体积较小；devel版本包含编译工具链，适合需要编译CUDA代码的场景。 - 层缓存优化：将
requirements.txt的复制和安装放在代码复制之前，这样当只有代码变化时，依赖安装层可以利用缓存。 - 清理不必要文件：使用
rm -rf /var/lib/apt/lists/*清理apt缓存，减小镜像体积。
2.3 Docker常用命令速查
# 构建镜像
docker build -t my-ai-app:v1.0 .
# 运行容器（带GPU支持）
docker run --gpus all -p 8000:8000 my-ai-app:v1.0
# 查看运行中的容器
docker ps
# 查看容器日志
docker logs -f <container_id>
# 进入容器调试
docker exec -it <container_id> /bin/bash
# 查看镜像层历史
docker history my-ai-app:v1.0
# 导出/导入镜像
docker save my-ai-app:v1.0 > my-ai-app.tar
docker load < my-ai-app.tar
2.4 多阶段构建
对于需要编译的AI应用（如自定义CUDA算子），多阶段构建可以显著减小最终镜像体积：
# 第一阶段：编译阶段
FROM nvidia/cuda:11.8.0-devel-ubuntu22.04 AS builder
RUN apt-get update && apt-get install -y python3.10 python3-pip
WORKDIR /build
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt
# 编译自定义CUDA算子
COPY custom_ops/ ./custom_ops/
RUN cd custom_ops && python3 setup.py build_ext --inplace
# 第二阶段：运行阶段
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3.10 python3-pip \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# 从编译阶段复制编译好的依赖和算子
COPY --from=builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
COPY --from=builder /build/custom_ops ./custom_ops/
COPY . .
EXPOSE 8000
CMD ["python3", "serve.py"]
第三章：AI应用Docker化
3.1 GPU支持配置
要在Docker中使用GPU，需要在宿主机上安装NVIDIA驱动和NVIDIA Container Toolkit：
# 检查NVIDIA驱动是否正常
nvidia-smi
# 安装NVIDIA Container Toolkit（以Ubuntu为例）
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
验证GPU支持：
# 运行一个测试容器
docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
# 指定使用特定GPU
docker run --rm --gpus '"device=0,1"' nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
# 限制GPU内存
docker run --rm --gpus all \
-e NVIDIA_VISIBLE_DEVICES=0 \
-e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
3.2 模型文件挂载策略
AI模型文件通常体积巨大（几GB到几百GB），直接打包进镜像会导致镜像臃肿。常见的模型挂载策略有：
策略一：Volume挂载
# 将宿主机的模型目录挂载到容器
docker run --gpus all \
-v /data/models:/models \
-v /data/configs:/configs \
my-ai-app:v1.0
策略二：使用Docker Volume
# 创建命名卷
docker volume create model-storage
# 使用命名卷
docker run --gpus all \
-v model-storage:/models \
my-ai-app:v1.0
策略三：环境变量指定远程模型路径
# 在Dockerfile中设置默认模型路径
ENV MODEL_PATH=/models/llm
ENV MODEL_NAME=chatglm-6b
# 启动时通过环境变量覆盖
docker run --gpus all \
-e MODEL_PATH=/models/custom-llm \
-e MODEL_NAME=my-finetuned-model \
-v /data/models:/models \
my-ai-app:v1.0
策略四：运行时下载模型
# serve.py - 启动时自动下载模型
import os
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_NAME = os.getenv("MODEL_NAME", "THUDM/chatglm-6b")
MODEL_PATH = os.getenv("MODEL_PATH", f"/models/{MODEL_NAME}")
def load_model():
if os.path.exists(MODEL_PATH):
print(f"Loading model from local path: {MODEL_PATH}")
model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
else:
print(f"Downloading model: {MODEL_NAME}")
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
# 保存到本地以便下次使用
model.save_pretrained(MODEL_PATH)
tokenizer.save_pretrained(MODEL_PATH)
return model, tokenizer
3.3 镜像体积优化
AI应用的Docker镜像往往很大，以下是几种优化方法：
选择合适的基础镜像：
镜像类型 大小 适用场景
nvidia/cuda:11.8.0-base ~200MB 只需要CUDA运行时
nvidia/cuda:11.8.0-runtime ~500MB 需要cuDNN
nvidia/cuda:11.8.0-devel ~3GB 需要编译CUDA代码
pytorch/pytorch:2.0-cuda ~5GB PyTorch完整环境
使用.dockerignore：
# .dockerignore
.git
__pycache__
*.pyc
*.pyo
.pytest_cache
.mypy_cache
*.egg-info
dist
build
.env
*.md
tests/
docs/
清理pip缓存：
RUN pip install --no-cache-dir -r requirements.txt
第四章：Docker Compose编排多服务AI应用
4.1 Docker Compose简介
一个完整的AI应用通常不仅仅是模型推理服务，还可能包含API网关、数据库、消息队列、前端界面等多个组件。Docker Compose可以方便地定义和管理这些多容器应用。
4.2 AI应用Compose配置示例
以下是一个典型的AI应用Docker Compose配置：
version: '3.8'
services:
# 模型推理服务
model-server:
build:
context: ./model-server
dockerfile: Dockerfile
image: my-ai-app/model-server:v1.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
- model-data:/models
- ./configs:/app/configs
environment:
- MODEL_PATH=/models/chatglm-6b
- MAX_BATCH_SIZE=16
- DEVICE=cuda:0
ports:
- "8001:8001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8001/health"]
interval: 30s
timeout: 10s
retries: 3
restart: unless-stopped
# API网关
api-gateway:
build:
context: ./api-gateway
dockerfile: Dockerfile
ports:
- "8000:8000"
environment:
- MODEL_SERVER_URL=http://model-server:8001
- REDIS_URL=redis://redis:6379
- DATABASE_URL=postgresql://postgres:password@postgres:5432/aidb
depends_on:
model-server:
condition: service_healthy
redis:
condition: service_started
postgres:
condition: service_started
restart: unless-stopped
# Redis缓存
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redis-data:/data
command: redis-server --appendonly yes --maxmemory 1gb --maxmemory-policy allkeys-lru
# PostgreSQL数据库
postgres:
image: postgres:15-alpine
environment:
- POSTGRES_DB=aidb
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=password
volumes:
- postgres-data:/var/lib/postgresql/data
ports:
- "5432:5432"
# Nginx反向代理
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/ssl:/etc/nginx/ssl:ro
depends_on:
- api-gateway
volumes:
model-data:
redis-data:
postgres-data:
4.3 Compose常用操作
# 启动所有服务
docker compose up -d
# 查看服务状态
docker compose ps
# 查看日志
docker compose logs -f model-server
# 扩展服务实例数（非GPU服务）
docker compose up -d --scale api-gateway=3
# 重建并启动
docker compose up -d --build
# 停止并清理
docker compose down
# 停止并清理数据卷
docker compose down -v
4.4 环境配置管理
使用.env文件管理不同环境的配置：
# .env.production
MODEL_NAME=chatglm-6b
MODEL_PATH=/data/models/chatglm-6b
MAX_BATCH_SIZE=32
REDIS_MAXMEMORY=4gb
POSTGRES_PASSWORD=secure_password_here
# .env.development
MODEL_NAME=chatglm-6b-int4
MODEL_PATH=/data/models/chatglm-6b-int4
MAX_BATCH_SIZE=4
REDIS_MAXMEMORY=512mb
POSTGRES_PASSWORD=dev_password
# 使用指定环境文件启动
docker compose --env-file .env.production up -d
第五章：Kubernetes基础
5.1 Kubernetes核心概念
Kubernetes（K8s）是一个开源的容器编排平台，用于自动化容器化应用的部署、扩展和管理。
Pod：Kubernetes中最小的部署单元，包含一个或多个容器。Pod中的容器共享网络和存储。
apiVersion: v1
kind: Pod
metadata:
name: ai-model-pod
labels:
app: ai-model
spec:
containers:
- name: model-server
image: my-ai-app/model-server:v1.0
ports:
- containerPort: 8001
resources:
requests:
cpu: "2"
memory: "8Gi"
nvidia.com/gpu: "1"
limits:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
Deployment：管理Pod的控制器，确保指定数量的Pod副本始终运行。
apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-model-deployment
spec:
replicas: 3
selector:
matchLabels:
app: ai-model
template:
metadata:
labels:
app: ai-model
spec:
containers:
- name: model-server
image: my-ai-app/model-server:v1.0
ports:
- containerPort: 8001
resources:
requests:
cpu: "2"
memory: "8Gi"
nvidia.com/gpu: "1"
limits:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
env:
- name: MODEL_PATH
value: "/models/chatglm-6b"
volumeMounts:
- name: model-volume
mountPath: /models
volumes:
- name: model-volume
persistentVolumeClaim:
claimName: model-pvc
Service：为Pod提供稳定的网络访问入口。
apiVersion: v1
kind: Service
metadata:
name: ai-model-service
spec:
selector:
app: ai-model
ports:
- port: 8001
targetPort: 8001
type: ClusterIP # 集群内部访问
Ingress：管理外部访问集群服务的规则，通常用于HTTP/HTTPS路由。
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ai-model-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
spec:
rules:
- host: ai-model.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ai-model-service
port:
number: 8001
tls:
- hosts:
- ai-model.example.com
secretName: tls-secret
5.2 Kubernetes集群搭建
使用kubeadm搭建生产集群：
# 在master节点上初始化
sudo kubeadm init \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-advertise-address=<master-ip>
# 配置kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 安装网络插件（Calico）
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
# 在worker节点上加入集群
kubeadm join <master-ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
使用kind快速搭建开发集群：
# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30080
hostPort: 30080
- role: worker
- role: worker
kind create cluster --config kind-config.yaml --name ai-cluster
5.3 kubectl常用命令
# 查看集群信息
kubectl cluster-info
kubectl get nodes
# 部署应用
kubectl apply -f deployment.yaml
# 查看资源
kubectl get pods -o wide
kubectl get services
kubectl get deployments
# 查看详细信息
kubectl describe pod <pod-name>
kubectl logs <pod-name> -f
kubectl exec -it <pod-name> -- /bin/bash
# 扩缩容
kubectl scale deployment ai-model-deployment --replicas=5
# 滚动更新
kubectl set image deployment/ai-model-deployment model-server=my-ai-app/model-server:v2.0
# 回滚
kubectl rollout undo deployment/ai-model-deployment
第六章：GPU集群调度
6.1 NVIDIA Device Plugin
NVIDIA Device Plugin是Kubernetes中管理GPU资源的核心组件。它负责发现节点上的GPU设备，并将其暴露为可调度的资源。
# 安装NVIDIA Device Plugin
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.1/nvidia-device-plugin.yml
安装完成后，Kubernetes节点会显示可用的GPU数量：
kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'
6.2 GPU资源请求
在Pod中请求GPU资源：
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
containers:
- name: ai-container
image: my-ai-app:v1.0
resources:
limits:
nvidia.com/gpu: 1 # 请求1个GPU
多GPU请求：
resources:
limits:
nvidia.com/gpu: 4 # 请求4个GPU用于模型并行
6.3 节点亲和性与污点容忍
节点亲和性：将Pod调度到具有特定标签的节点。
apiVersion: v1
kind: Pod
metadata:
name: gpu-intensive-pod
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: gpu-type
operator: In
values:
- a100
- h100
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: ai-container
image: my-ai-app:v1.0
resources:
limits:
nvidia.com/gpu: 1
GPU节点污点设置：
# 给GPU节点添加污点，防止非GPU工作负载调度到GPU节点
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
# 给节点打标签
kubectl label nodes gpu-node-1 gpu-type=a100
kubectl label nodes gpu-node-2 gpu-type=h100
6.4 GPU拓扑感知调度
对于多GPU训练任务，GPU之间的通信拓扑（NVLink、PCIe）会影响性能。NVIDIA Topology Manager可以帮助调度器感知GPU拓扑：
apiVersion: v1
kind: Pod
metadata:
name: multi-gpu-training
spec:
containers:
- name: training-container
image: my-training-app:v1.0
resources:
limits:
nvidia.com/gpu: 8
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "all"
- name: NCCL_TOPO_FILE
value: "/etc/nccl/topo.xml"
第七章：模型服务部署
7.1 KServe部署
KServe（原KFServing）是Kubernetes上标准化的模型推理平台，支持多种推理引擎。
安装KServe：
# 安装Istio（KServe的依赖）
kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.10.0/net-istio.yaml
# 安装KServe
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.11.0/kserve.yaml
使用KServe部署模型：
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: chatglm-service
spec:
predictor:
model:
modelFormat:
name: pytorch
storageUri: "pvc://model-pvc/chatglm-6b"
resources:
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
runtime: "kserve-pytorch"
自定义推理服务：
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: custom-llm-service
spec:
predictor:
containers:
- name: kserve-container
image: my-registry/custom-llm-server:v1.0
ports:
- containerPort: 8080
protocol: TCP
env:
- name: MODEL_NAME
value: "chatglm-6b"
- name: MODEL_PATH
value: "/models/chatglm-6b"
resources:
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
volumeMounts:
- name: model-storage
mountPath: /models
volumes:
- name: model-storage
persistentVolumeClaim:
claimName: model-pvc
7.2 Triton Inference Server
Triton Inference Server是NVIDIA开源的高性能推理服务器，支持多种框架（TensorFlow、PyTorch、ONNX等），并提供动态批处理、模型并发执行等高级功能。
Triton模型仓库结构：
model-repository/
├── chatglm/
│ ├── config.pbtxt
│ └── 1/
│ └── model.py
├── bert-ner/
│ ├── config.pbtxt
│ └── 1/
│ └── model.onnx
└── stable-diffusion/
├── config.pbtxt
└── 1/
└── model.plan
config.pbtxt示例：
name: "chatglm"
platform: "pytorch_libtorch"
max_batch_size: 8
input [
{
name: "input_ids"
data_type: TYPE_INT64
dims: [ -1 ]
},
{
name: "attention_mask"
data_type: TYPE_INT64
dims: [ -1 ]
}
]
output [
{
name: "output"
data_type: TYPE_FP32
dims: [ -1, 130528 ]
}
]
instance_group [
{
count: 1
kind: KIND_GPU
gpus: [ 0 ]
}
]
dynamic_batching {
preferred_batch_size: [ 2, 4, 8 ]
max_queue_delay_microseconds: 100000
}
部署Triton到Kubernetes：
apiVersion: apps/v1
kind: Deployment
metadata:
name: triton-server
spec:
replicas: 2
selector:
matchLabels:
app: triton
template:
metadata:
labels:
app: triton
spec:
containers:
- name: triton
image: nvcr.io/nvidia/tritonserver:23.10-py3
command: ["tritonserver"]
args:
- "--model-repository=/models"
- "--strict-model-config=false"
- "--log-verbose=1"
ports:
- containerPort: 8000 # HTTP
name: http
- containerPort: 8001 # gRPC
name: grpc
- containerPort: 8002 # Metrics
name: metrics
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: model-repo
mountPath: /models
readinessProbe:
httpGet:
path: /v2/health/ready
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
livenessProbe:
httpGet:
path: /v2/health/live
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
volumes:
- name: model-repo
persistentVolumeClaim:
claimName: triton-model-pvc
第八章：自动伸缩
8.1 HPA（Horizontal Pod Autoscaler）
HPA根据CPU、内存或自定义指标自动调整Pod副本数。
基于GPU显存使用率的HPA：
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ai-model-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ai-model-deployment
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: gpu_memory_used_percent
target:
type: AverageValue
averageValue: "80"
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 2
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 25
periodSeconds: 120
8.2 VPA（Vertical Pod Autoscaler）
VPA自动调整Pod的资源请求和限制。
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: ai-model-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: ai-model-deployment
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: model-server
minAllowed:
cpu: "1"
memory: "4Gi"
maxAllowed:
cpu: "16"
memory: "64Gi"
controlledResources: ["cpu", "memory"]
8.3 KEDA事件驱动伸缩
KEDA（Kubernetes Event-driven Autoscaling）支持基于事件源的伸缩，如消息队列长度、HTTP请求速率等。
基于请求队列长度的伸缩：
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: ai-model-scaledobject
spec:
scaleTargetRef:
name: ai-model-deployment
minReplicaCount: 1
maxReplicaCount: 20
cooldownPeriod: 300
triggers:
- type: redis
metadata:
address: redis-service:6379
listName: request-queue
listLength: "10" # 每10个待处理请求扩容1个Pod
authenticationRef:
name: redis-auth
基于Prometheus指标的伸缩：
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: ai-model-scaledobject
spec:
scaleTargetRef:
name: ai-model-deployment
minReplicaCount: 1
maxReplicaCount: 20
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: inference_queue_length
query: sum(inference_queue_length{service="ai-model"})
threshold: "5"
第九章：配置管理
9.1 ConfigMap
ConfigMap用于存储非敏感的配置数据。
apiVersion: v1
kind: ConfigMap
metadata:
name: ai-model-config
data:
model-config.yaml: |
model:
name: "chatglm-6b"
path: "/models/chatglm-6b"
max_length: 2048
temperature: 0.7
top_p: 0.9
server:
host: "0.0.0.0"
port: 8001
workers: 4
batch_size: 16
logging:
level: "INFO"
format: "json"
在Deployment中使用ConfigMap：
spec:
containers:
- name: model-server
env:
- name: MODEL_NAME
valueFrom:
configMapKeyRef:
name: ai-model-config
key: model.name
volumeMounts:
- name: config-volume
mountPath: /app/configs
volumes:
- name: config-volume
configMap:
name: ai-model-config
9.2 Secret
Secret用于存储敏感信息，如API密钥、数据库密码等。
apiVersion: v1
kind: Secret
metadata:
name: ai-model-secrets
type: Opaque
stringData:
API_KEY: "sk-xxxxxxxxxxxxxxxxxxxx"
DATABASE_PASSWORD: "secure_password"
HUGGINGFACE_TOKEN: "hf_xxxxxxxxxxxxxxxx"
使用Secret：
spec:
containers:
- name: model-server
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: ai-model-secrets
key: API_KEY
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: ai-model-secrets
key: HUGGINGFACE_TOKEN
9.3 环境变量管理最佳实践
# 使用ConfigMap和Secret的组合
env:
# 来自ConfigMap的配置
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: app-config
key: log_level
# 来自Secret的敏感信息
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secrets
key: password
# 直接定义的环境变量
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
# 引用其他环境变量
- name: MODEL_CACHE_DIR
value: "/cache/$(MODEL_NAME)"
第十章：监控与日志
10.1 Prometheus + Grafana监控
安装Prometheus Operator：
# 使用Helm安装kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
自定义AI服务监控指标：
# metrics.py - 在AI服务中暴露Prometheus指标
from prometheus_client import Counter, Histogram, Gauge, generate_latest
import time
# 定义指标
REQUEST_COUNT = Counter(
'inference_requests_total',
'Total inference requests',
['model', 'status', 'method']
)
REQUEST_LATENCY = Histogram(
'inference_request_duration_seconds',
'Inference request latency',
['model'],
buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
)
GPU_MEMORY_USED = Gauge(
'gpu_memory_used_bytes',
'GPU memory used in bytes',
['gpu_id']
)
BATCH_SIZE = Histogram(
'inference_batch_size',
'Batch size of inference requests',
['model'],
buckets=[1, 2, 4, 8, 16, 32, 64]
)
QUEUE_LENGTH = Gauge(
'inference_queue_length',
'Number of requests waiting in queue'
)
# 使用示例
def inference_handler(request):
start_time = time.time()
try:
result = model.predict(request)
REQUEST_COUNT.labels(model='chatglm', status='success', method='predict').inc()
return result
except Exception as e:
REQUEST_COUNT.labels(model='chatglm', status='error', method='predict').inc()
raise
finally:
REQUEST_LATENCY.labels(model='chatglm').observe(time.time() - start_time)
ServiceMonitor配置：
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ai-model-monitor
labels:
release: prometheus
spec:
selector:
matchLabels:
app: ai-model
endpoints:
- port: metrics
interval: 15s
path: /metrics
Grafana Dashboard配置：
{
"dashboard": {
"title": "AI Model Monitoring",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(inference_requests_total[5m])",
"legendFormat": "{{model}} - {{status}}"
}
]
},
{
"title": "P95 Latency",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(inference_request_duration_seconds_bucket[5m]))",
"legendFormat": "{{model}}"
}
]
},
{
"title": "GPU Memory Usage",
"type": "gauge",
"targets": [
{
"expr": "gpu_memory_used_bytes / gpu_memory_total_bytes * 100",
"legendFormat": "GPU {{gpu_id}}"
}
]
}
]
}
}
10.2 ELK日志系统
使用EFK（Elasticsearch + Fluentd + Kibana）收集日志：
# Fluentd DaemonSet配置
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: logging
spec:
selector:
matchLabels:
name: fluentd
template:
metadata:
labels:
name: fluentd
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-1
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch.logging.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
volumeMounts:
- name: varlog
mountPath: /var/log
- name: containers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
- name: containers
hostPath:
path: /var/lib/docker/containers
结构化日志最佳实践：
import json
import logging
from datetime import datetime
class StructuredLogger:
def __init__(self, name):
self.logger = logging.getLogger(name)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(message)s'))
self.logger.addHandler(handler)
self.logger.setLevel(logging.INFO)
def info(self, message, **kwargs):
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"level": "INFO",
"message": message,
**kwargs
}
self.logger.info(json.dumps(log_entry))
def error(self, message, **kwargs):
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"level": "ERROR",
"message": message,
**kwargs
}
self.logger.error(json.dumps(log_entry))
# 使用示例
logger = StructuredLogger("ai-model-server")
logger.info("Inference completed",
model="chatglm-6b",
latency_ms=150,
input_tokens=128,
output_tokens=256,
request_id="req-12345"
)
第十一章：实战项目——部署可弹性伸缩的LLM推理服务
11.1 项目架构
我们将构建一个完整的LLM推理服务，包含以下组件：
┌──────────────┐
│ Ingress │
│ (Nginx) │
└──────┬───────┘
│
┌──────▼───────┐
│ API Gateway │
│ (FastAPI) │
└──────┬───────┘
│
┌────────────┼────────────┐
│ │ │
┌──────▼──────┐ ┌──▼──────────┐ ┌▼───────────┐
│ LLM Worker │ │ LLM Worker │ │ LLM Worker │
│ (vLLM) │ │ (vLLM) │ │ (vLLM) │
└──────┬──────┘ └──┬──────────┘ └┬───────────┘
│ │ │
└────────────┼────────────┘
│
┌──────▼───────┐
│ Redis │
│ (Queue) │
└──────────────┘
11.2 vLLM推理服务Dockerfile
FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
python3.11 \
python3-pip \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# 安装vLLM和依赖
RUN pip install --no-cache-dir \
vllm==0.2.7 \
fastapi==0.104.1 \
uvicorn==0.24.0 \
redis==5.0.1 \
prometheus-client==0.19.0
COPY server.py .
COPY metrics.py .
EXPOSE 8000
CMD ["python3", "-m", "uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]
11.3 推理服务代码
# server.py
import os
import json
import time
import asyncio
from typing import Optional, List
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
import redis.asyncio as redis
from metrics import REQUEST_COUNT, REQUEST_LATENCY, GPU_MEMORY_USED, QUEUE_LENGTH
# 配置
MODEL_NAME = os.getenv("MODEL_NAME", "Qwen/Qwen-7B-Chat")
TENSOR_PARALLEL_SIZE = int(os.getenv("TENSOR_PARALLEL_SIZE", "1"))
MAX_MODEL_LEN = int(os.getenv("MAX_MODEL_LEN", "4096"))
GPU_MEMORY_UTILIZATION = float(os.getenv("GPU_MEMORY_UTILIZATION", "0.9"))
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
# 全局变量
engine = None
redis_client = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global engine, redis_client
# 初始化vLLM引擎
engine_args = AsyncEngineArgs(
model=MODEL_NAME,
tensor_parallel_size=TENSOR_PARALLEL_SIZE,
max_model_len=MAX_MODEL_LEN,
gpu_memory_utilization=GPU_MEMORY_UTILIZATION,
trust_remote_code=True,
)
engine = AsyncLLMEngine.from_engine_args(engine_args)
# 初始化Redis
redis_client = redis.from_url(REDIS_URL, decode_responses=True)
yield
# 清理
if redis_client:
await redis_client.close()
app = FastAPI(title="LLM Inference Service", lifespan=lifespan)
class ChatRequest(BaseModel):
messages: List[dict]
model: str = MODEL_NAME
temperature: float = 0.7
top_p: float = 0.9
max_tokens: int = 2048
stream: bool = False
class ChatResponse(BaseModel):
id: str
model: str
choices: List[dict]
usage: dict
@app.post("/v1/chat/completions", response_model=ChatResponse)
async def chat_completions(request: ChatRequest):
start_time = time.time()
request_id = f"req-{int(time.time() * 1000)}"
try:
# 构建prompt
prompt = ""
for msg in request.messages:
role = msg.get("role", "user")
content = msg.get("content", "")
if role == "system":
prompt += f"System: {content}\n"
elif role == "user":
prompt += f"User: {content}\n"
elif role == "assistant":
prompt += f"Assistant: {content}\n"
prompt += "Assistant: "
# 设置采样参数
sampling_params = SamplingParams(
temperature=request.temperature,
top_p=request.top_p,
max_tokens=request.max_tokens,
)
# 执行推理
QUEUE_LENGTH.inc()
results = []
async for output in engine.generate(prompt, sampling_params, request_id):
results.append(output)
QUEUE_LENGTH.dec()
# 获取最终结果
final_output = results[-1]
generated_text = final_output.outputs[0].text
# 计算token数
prompt_tokens = len(final_output.prompt_token_ids)
completion_tokens = len(final_output.outputs[0].token_ids)
# 更新指标
latency = time.time() - start_time
REQUEST_COUNT.labels(model=MODEL_NAME, status='success').inc()
REQUEST_LATENCY.labels(model=MODEL_NAME).observe(latency)
return ChatResponse(
id=request_id,
model=MODEL_NAME,
choices=[{
"index": 0,
"message": {
"role": "assistant",
"content": generated_text
},
"finish_reason": "stop"
}],
usage={
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": prompt_tokens + completion_tokens
}
)
except Exception as e:
REQUEST_COUNT.labels(model=MODEL_NAME, status='error').inc()
raise HTTPException(status_code=500, detail=str(e))
@app.get("/health")
async def health_check():
return {"status": "healthy", "model": MODEL_NAME}
@app.get("/metrics")
async def metrics():
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST
from fastapi.responses import Response
return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)
11.4 Kubernetes部署清单
# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ai-inference
---
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: llm-config
namespace: ai-inference
data:
MODEL_NAME: "Qwen/Qwen-7B-Chat"
TENSOR_PARALLEL_SIZE: "1"
MAX_MODEL_LEN: "4096"
GPU_MEMORY_UTILIZATION: "0.9"
---
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: llm-secrets
namespace: ai-inference
type: Opaque
stringData:
HF_TOKEN: "hf_xxxxxxxxxxxxxxxx"
---
# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-cache-pvc
namespace: ai-inference
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-csi
resources:
requests:
storage: 100Gi
---
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-server
namespace: ai-inference
labels:
app: llm-server
spec:
replicas: 2
selector:
matchLabels:
app: llm-server
template:
metadata:
labels:
app: llm-server
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: gpu-type
operator: In
values:
- a100
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: llm-server
image: my-registry/llm-server:v1.0
ports:
- containerPort: 8000
name: http
envFrom:
- configMapRef:
name: llm-config
- secretRef:
name: llm-secrets
resources:
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
volumeMounts:
- name: model-cache
mountPath: /root/.cache/huggingface
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 30
timeoutSeconds: 5
volumes:
- name: model-cache
persistentVolumeClaim:
claimName: model-cache-pvc
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
name: llm-service
namespace: ai-inference
labels:
app: llm-server
spec:
selector:
app: llm-server
ports:
- port: 8000
targetPort: 8000
name: http
type: ClusterIP
---
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-hpa
namespace: ai-inference
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-server
minReplicas: 1
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: inference_queue_length
target:
type: AverageValue
averageValue: "5"
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 2
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 25
periodSeconds: 120
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-ingress
namespace: ai-inference
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
rules:
- host: llm-api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: llm-service
port:
number: 8000
tls:
- hosts:
- llm-api.example.com
secretName: llm-tls-secret
11.5 部署与验证
# 创建命名空间
kubectl apply -f namespace.yaml
# 部署配置
kubectl apply -f configmap.yaml
kubectl apply -f secret.yaml
kubectl apply -f pvc.yaml
# 部署服务
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f hpa.yaml
kubectl apply -f ingress.yaml
# 检查部署状态
kubectl get pods -n ai-inference
kubectl get svc -n ai-inference
kubectl get hpa -n ai-inference
# 查看日志
kubectl logs -f deployment/llm-server -n ai-inference
# 测试API
curl -X POST http://llm-api.example.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "你好，请介绍一下自己"}
],
"temperature": 0.7,
"max_tokens": 512
}'
11.6 性能调优建议
1. 模型优化：
- 使用量化（GPTQ、AWQ）减少显存占用
- 启用PagedAttention优化长序列推理
- 使用连续批处理（Continuous Batching）提高吞吐
2. 系统优化：
- 调整
GPU_MEMORY_UTILIZATION参数，平衡显存使用和性能 - 根据实际负载调整HPA的触发阈值
- 使用NVLink优化多GPU通信
3. 成本优化：
- 使用Spot/Preemptible实例降低GPU成本
- 实施请求队列，在低峰期合并请求
- 使用模型缓存避免重复加载
第十二章：总结与展望
12.1 关键要点回顾
本教程涵盖了AI应用容器化部署的完整知识体系：
- Docker基础：镜像构建、GPU支持、多阶段构建、体积优化
- Docker Compose：多服务编排、环境管理、依赖控制
- Kubernetes核心：Pod、Service、Deployment、Ingress
- GPU调度：Device Plugin、节点亲和性、拓扑感知
- 模型服务：KServe、Triton Inference Server
- 自动伸缩：HPA、VPA、KEDA
- 配置管理：ConfigMap、Secret、环境变量
- 监控日志：Prometheus、Grafana、ELK
- 实战部署：完整的LLM推理服务
12.2 生产环境检查清单
在将AI服务部署到生产环境前，请确认以下事项：
- 镜像安全扫描通过
- GPU资源请求和限制已正确配置
- 健康检查（Readiness/Liveness Probe）已配置
- HPA自动伸缩策略已测试
- 监控告警已配置
- 日志收集已启用
- TLS证书已配置
- 备份策略已制定
- 灾难恢复方案已准备
- 性能基准测试已完成
12.3 未来趋势
- Serverless AI：如Knative、AWS Lambda等，实现真正的按需计费
- 多模型服务网格：Istio/Linkerd在AI服务中的应用
- 边缘AI部署：K3s、KubeEdge等轻量级Kubernetes在边缘场景的应用
- AI编排平台：如Ray、Kubeflow等，简化AI工作流管理
- GPU虚拟化：MIG（Multi-Instance GPU）、vGPU等技术提高GPU利用率
参考资源
本教程持续更新，欢迎反馈和建议。
