Skip to content

Litmus 详解 (CNCF Chaos Engineering)

Litmus 是 CNCF Incubating 项目,提供丰富的故障实验库和强大的 Web UI 平台。

一、Litmus 概述

1.1 项目简介

项目:    LitmusChaos
官网:    https://litmuschaos.io/
GitHub:  https://github.com/litmuschaos/litmus
维护:    MayaData
License: Apache 2.0
CNCF:    Incubating 项目
当前版本: 3.x

1.2 核心特性

1. 丰富的实验库 (ChaosHub)
   - 50+ 预定义实验
   - 可共享 (类似镜像仓库)
   - 持续社区贡献

2. Web UI 平台 (Litmus Portal)
   - 实验编排
   - 实时监控
   - 报告生成
   - GitOps 集成

3. K8s 原生
   - CRD 形式
   - 多租户
   - 命名空间隔离

4. RBAC
   - 细粒度权限
   - 团队协作
   - 审批流程

5. ChaosCenter
   - 实验管理
   - 调度执行
   - 报告分析

二、核心概念

2.1 核心 CRD 资源

1. ChaosExperiment
   故障实验定义 (YAML)
   描述: 什么故障, 怎么注入

2. ChaosEngine
   实验运行实例
   绑定到目标 (app info)
   调度方式 (now / schedule)

3. ChaosResult
   实验结果
   verdict: pass / fail

4. ChaosHub
   实验共享仓库 (Git Repo)
   团队共享实验模板

5. ChaosSchedule
   定时调度实验

6. ChaosScenario
   多实验编排
   串行 / 并行

2.2 实验生命周期

1. 选择实验 (从 ChaosHub 或自定义)
2. 创建 ChaosEngine (绑定目标)
3. 实验调度
4. 故障注入
5. 监控与采集
6. 结果判定 (verdict)
7. 实验归档
8. 报告生成

三、安装部署

3.1 系统要求

- K8s 1.17+
- Helm 3.0+
- 至少 3 个节点 (生产)
- 存储 (etcd 后端)
- RBAC 启用

3.2 Helm 安装

# 1. 添加仓库
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update

# 2. 创建 namespace
kubectl create namespace litmus

# 3. 安装
helm install litmus litmuschaos/litmus \
  --namespace litmus \
  --create-namespace \
  --set portal.frontend.service.type=NodePort \
  --set portal.frontend.service.nodePort=30090

# 4. 验证
kubectl get pods -n litmus
# chaos-litmus-portal-frontend
# chaos-litmus-portal-backend
# chaos-litmus-auth-server
# chaos-litmus-event-tracker
# chaos-litmus-workflow-controller
# chaos-runner

3.3 访问 UI

# 获取端口
kubectl get svc chaos-litmus-portal-frontend -n litmus
# 默认 NodePort 30091

# 访问
http://<node-ip>:30091
# 默认账号: admin / litmus

四、ChaosHub (实验库)

4.1 官方实验库

GitHub: https://github.com/litmuschaos/chaos-charts

实验分类:

Pod:
  - pod-delete
  - pod-cpu-hog
  - pod-memory-hog
  - pod-network-latency
  - pod-network-loss
  - pod-disk-fill
  - pod-cpu-burn
  - container-kill

Network:
  - network-loss
  - network-latency
  - network-duplication
  - network-corruption
  - dns-error

Stress:
  - cpu-stress
  - memory-stress
  - io-stress
  - disk-stress

State:
  - pod-restart
  - container-restart
  - process-kill

Time:
  - time-chaos

4.2 添加 ChaosHub

# 添加官方 hub
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-charts/master/charts/generic/experiments.yaml -n litmus

# 添加自定义 hub
cat <<EOF | kubectl apply -f -
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosHub
metadata:
  name: my-hub
  namespace: litmus
spec:
  scope: Cluster
  charts:
  - name: my-custom-experiment
    version: 1.0.0
    repositoryURL: https://github.com/myorg/chaos-charts
EOF

五、ChaosExperiment (实验定义)

5.1 完整示例

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-delete
  namespace: litmus
spec:
  description:
    Kill the pod
  definition:
    scope: Namespaced
  method:
    image: "litmuschaos/go-runner:latest"
    commands:
    - /bin/bash
    - -c
    - |
      kubectl delete pod -n ${{ invocation.experimentObject.metadata.namespace }}
        -l ${{ invocation.experimentObject.spec.labelSelectors }}
        --force=true
      sleep 10
  imagePullPolicy: Always

5.2 带参数实验

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-cpu-hog
  namespace: litmus
spec:
  description:
    Consume CPU of the target pods
  definition:
    scope: Namespaced
  method:
    image: "litmuschaos/go-runner:latest"
    commands:
    - /bin/bash
    - -c
    - |
      stress-ng --cpu ${{ invocation.experimentObject.parameters.CPU_COUNT }} \
                 --timeout ${{ invocation.experimentObject.parameters.DURATION }}
  imagePullPolicy: Always
  definition:
    scope: Cluster
  kind: Experiment

5.3 完整实验文件结构

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: experiment-name
  namespace: litmus
spec:
  description: "实验描述"
  definition:
    scope: Cluster | Namespaced
  auth:
    # IAM 角色
  rank: 1  # 调度顺序
  version: 1.0.0
  method:
    image: "image:tag"
    commands:
    - 命令列表
    - 可用变量:
      - invocation.experimentObject.metadata.namespace
      - invocation.experimentObject.spec.labelSelectors
      - invocation.experimentObject.parameters.PARAM_NAME
    imagePullPolicy: IfNotPresent
    entrypoint:  # 可选
    args:
    - 可选参数
  resources:
    requests:
      cpu: 100m
      memory: 100Mi
    limits:
      cpu: 500m
      memory: 500Mi

六、ChaosEngine (运行引擎)

6.1 基础引擎

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-engine
  namespace: production
spec:
  appinfo:
    appkind: deployment
    applabel: app=nginx
    appns: production
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete

6.2 完整引擎

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: my-chaos
  namespace: production
spec:
  # 应用信息 (实验目标)
  appinfo:
    appkind: deployment          # deployment / statefulset / daemonset
    applabel: app=nginx,env=prod
    appns: production

  # 混沌服务账号
  chaosServiceAccount: chaos-admin

  # 监控服务账号
  jobsCleanUpPolicy: retain   # retain / delete

  # 实验列表
  experiments:
  - name: pod-delete
  - name: pod-cpu-hog
  - name: pod-network-latency

6.3 调度策略

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: scheduled-chaos
spec:
  appinfo:
    appkind: deployment
    applabel: app=nginx
  chaosServiceAccount: chaos-admin
  experiments:
  - name: pod-delete
  schedule:
    type: now                 # now (立即) / in (延迟) / cron
    when: ""                  # in: "5m" / cron: "0 2 * * *"
    historyLimit: 3

七、ChaosResult (实验结果)

7.1 查看结果

# 列出所有实验结果
kubectl get chaosresults -A

# 查看特定结果详情
kubectl describe chaosresult <name> -n production

7.2 结果字段

status:
  experimentstatus:
    phase: Completed        # Running / Completed / Failed
    verdict: Pass            # Pass / Fail
  probesstatus:
  - name: check-pod-status
    status: Pass
  - name: check-app-health
    status: Fail              # 实验失败,系统暴露脆弱点
  experimentpod: chaos-pod-xxx
  verfication:                # 验证结果
    result:
      verdict: Pass
      description: "Application remained healthy"

八、ChaosSchedule (定时调度)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosSchedule
metadata:
  name: weekly-pod-kill
  namespace: production
spec:
  schedule:
    type: cron
    cron:
      schedule: "0 2 * * 0"     # 每周日 2 点
      timezone: "Asia/Shanghai"
      startingDeadlineSeconds: 60
  chaosEngine: my-chaos-engine
  historyLimit: 5
  concurrencyPolicy: Forbid  # Forbid / Allow

九、ChaosScenario (场景编排)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosScenario
metadata:
  name: resilience-test
  namespace: litmus
spec:
  description: "Application resilience scenario"
  tasks:
  - name: pod-kill
    taskType: chaosengine
    engine: pod-kill-engine
  - name: network-delay
    taskType: chaosengine
    engine: network-delay-engine
    dependsOn: [pod-kill]
  - name: final-check
    taskType: probe
    dependsOn: [network-delay]

十、ChaosProbe (健康探测)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosProbe
metadata:
  name: app-health-check
  namespace: production
spec:
  type: httpProbe
  url: http://my-app/health
  insecureSkipVerify: true
  interval: 5
  timeout: 3
  retry: 3
  probeTimeout: 30
# Kubernetes 探针
spec:
  type: k8sProbe
  verb: get
  resource: pods
  labelSelectors: app=nginx
  fieldSelector: status.phase=Running
  timeout: 5
  retry: 3

十一、Litmus Portal (Web UI)

11.1 功能

1. 实验探索
   - 浏览 ChaosHub
   - 搜索实验
   - 预定义模板

2. 实验编排
   - 选择实验
   - 配置参数
   - 设置调度
   - 配置 Probe

3. 实验监控
   - 实时状态
   - 故障事件
   - 系统响应
   - Probe 结果

4. 报告生成
   - 实验详情
   - 时间线
   - 失败原因
   - 改进建议

5. RBAC
   - 用户管理
   - 团队协作
   - 审批流程

6. 集成
   - GitOps (ArgoCD)
   - 监控 (Prometheus)
   - 通知 (Slack, Teams)

11.2 RBAC 权限

# Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: chaos-experimenter
rules:
- apiGroups: ["litmuschaos.io"]
  resources: ["chaosengines", "chaosexperiments"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list"]

十二、实战案例

12.1 Pod 杀演练

# 1. ChaosHub
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosHub
metadata:
  name: litmus
  namespace: litmus
spec:
  scope: Cluster
  charts:
  - name: pod-delete
    version: 1.0.0
    repositoryURL: https://github.com/litmuschaos/chaos-charts
# 2. ChaosEngine
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-test
  namespace: production
spec:
  appinfo:
    appkind: deployment
    applabel: app=nginx
  chaosServiceAccount: chaos-admin
  experiments:
  - name: pod-delete
# 3. 应用
kubectl apply -f chaosengine.yaml

# 4. 监控
watch kubectl get chaosresult -n production

# 5. 完成后查看结果
kubectl get chaosresult pod-delete-test -n production -o yaml

12.2 网络延迟演练

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: network-delay-test
  namespace: production
spec:
  appinfo:
    appkind: deployment
    applabel: app=api
  chaosServiceAccount: chaos-admin
  experiments:
  - name: pod-network-latency
    spec:
      components:
        env:
        - name: NETWORK_LATENCY
          value: "2000"      # 2 秒
        - name: CONTAINER_RUNTIME
          value: containerd
        - name: SOCKET_PATH
          value: /run/containerd/containerd.sock
        - name: NETWORK_INTERFACE
          value: eth0
        - name: TARGET_CONTAINER
          value: api

12.3 数据库慢响应演练

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: mysql-latency
  namespace: production
spec:
  appinfo:
    appkind: statefulset
    applabel: app=mysql
  chaosServiceAccount: chaos-admin
  experiments:
  - name: pod-network-latency
    spec:
      components:
        env:
        - name: NETWORK_LATENCY
          value: "500"
        - name: TARGET_CONTAINER
          value: mysql

十三、集成与扩展

13.1 GitOps 集成

# ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: litmus
spec:
  source:
    repoURL: https://github.com/myorg/chaos-experiments
    path: experiments/
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

13.2 Prometheus 集成

# chaos-exporter 暴露实验指标
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/monitoring/chaos-exporter.yaml
# 实验执行状态
chaos_engine_status
chaos_experiment_status
chaos_result_verdict

13.3 通知集成

# Slack 通知 (Litmus Slack integration)
apiVersion: v1
kind: ConfigMap
metadata:
  name: slack-config
  namespace: litmus
data:
  SLACK_BOT_TOKEN: "xoxb-xxx"
  SLACK_CHANNEL: "#chaos-alerts"
  CHAOS_UI_URL: "https://litmus.example.com"

十四、核心要点速记

Litmus 核心 CRD

ChaosExperiment:  故障实验定义
ChaosEngine:      实验运行实例
ChaosResult:      实验结果
ChaosHub:         实验仓库
ChaosSchedule:    定时任务
ChaosScenario:    场景编排
ChaosProbe:       健康探测

实验库 (ChaosHub)

Pod:       pod-delete, pod-cpu-hog, pod-memory-hog
Network:   network-loss, network-latency, dns-error
Stress:    cpu-stress, memory-stress
State:     container-kill, process-kill
Time:      time-chaos
Disk:      disk-fill

核心流程

1. 创建 ChaosHub (从 Git)
2. 选择实验
3. 创建 ChaosEngine (绑定目标)
4. 调度执行
5. 监控 ChaosProbe
6. 查看 ChaosResult
7. 总结改进

快速上手

# 安装
helm install litmus litmuschaos/litmus -n litmus

# 访问 UI
http://<node-ip>:30091
# admin / litmus

# 准备实验
# 添加 ChaosHub
# 选实验
# 配 ChaosEngine
# 跑实验

与 Chaos Mesh 对比

Chaos Mesh: K8s 原生, CRD 形式
  - 内置 12+ 种故障类型
  - 完全 K8s 风格
  - CNCF Sandbox

Litmus: 实验库 + Web UI
  - 50+ 预定义实验
  - 强大 UI 平台
  - 实验共享 (ChaosHub)
  - CNCF Incubating

选择建议

- 想要丰富实验库 → Litmus
- 想要完整 UI 平台 → Litmus
- 想要纯 K8s 集成 → Chaos Mesh
- 中文支持 → ChaosBlade

参考

  • Litmus 官方: https://litmuschaos.io/
  • Litmus 文档: https://docs.litmuschaos.io/
  • GitHub: https://github.com/litmuschaos/litmus
  • ChaosHub 示例: https://github.com/litmuschaos/chaos-charts
  • CNCF 项目: https://www.cncf.io/projects/litmuschaos/
  • Slack: https://kubernetes.slack.com/archives/C0196Q5NZQM
  • Litmus 教程: https://docs.litmuschaos.io/docs/getting-started/