Kubernetes,  Operation

基于 k3s + gitea + argocd + tekton 部署 GitOps

2022-06-29 /

基于 k3s + gitea + argocd + tekton 部署 GitOps

  • 部署方案拓扑
①developers
  ↓
  ↓ (git push)
  ↓
②Gitea (/java-repo/springboot.git)
  ↓
  ↓ (webhook)
  ↓
③Tekton (EventListener/TriggerBinding/TriggerTemplate/PipelineRun) →  ⑤Gitea (/helm-repo/springboot.git)
  ↓                                                                     ↓
  ↓ (docker push)                                                       ↓ (webhook)
  ↓                                                                     ↓
④hub.docker (harbor/nexus3)  →  →   →  → docker pull  →  →  →  →  →  ⑥ArgoCD
                                                                        ↓
                                                                        ↓ (kubectl apply)
                                                                        ↓
                                                                      kube-apiserver / k3s server
                                                                        ↓
                                                                        ↓ (run deployement/rs/pod/svc/...)
                                                                       ...

  • 提示：在实际企业级环境中，webhook 点 ② 和 ⑤ 一般是独立仓库，其中 ② 一般会对应 Git Flow 中多种分支，如：features 分支开发完提交自动触发 tekton-cli (编译/静态扫描/单元测试)合并到 master，然后管理员将 master 合并到 release-1.0，再触发 tekton-cli 发布到 preparate 环境。

  • Git Flow 管理模型分析：https://plump-possum-683.notion.site/Git-a332efd1f5074521a527a78dd810a7d0

1. 部署 k3s

1.1 一键在线部署

curl -sfL https://get.k3s.io | sh -

1.2 二进制离线部署

curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true sh -
  • 安装完后
ls -al /usr/local/bin/ | grep k3s
k3s
k3s-killall.sh
k3s-uninstall.sh
  • 快捷命令
echo "alias k3='sudo k3s kubectl'" >> /etc/bash.bashrc

2. 部署 argocd

  • 2.1 安装 argocd 组件
# 必须是干净的 namespace，否则 argocd 可能会读取到其他资源导致权限检查失败.
k3 create namespace argocd
k3 apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 或使用代理
k3 apply -n argocd -f https://ghproxy.com/https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
k3 -n argocd edit service/argocd-server
  • 查看初始密码 (用户名: admin)
k3 get secret -n argocd argocd-initial-admin-secret -ojson | jq -r '.data.password' | base64 -d; echo

3. 配置 gitea webhook to ArgoCD

4. 部署 tekton

export baseUri='https://gitee.com/wl4g/blogs/raw/master/docs/articles/kubernetes/deploy-k3s-gitea-argocd-tekton-gitops/resources'
k3 apply -f ${baseUri}/tekton-dashboard-release.yaml
k3 apply -f ${baseUri}/tekton-release.yaml
k3 apply -f ${baseUri}/trigger-release.yaml
k3 apply -f ${baseUri}/trigger-interceptors.yaml
  • 安装 tekton 组件 (最新版, 官方默认模版可能需修改配置)
k3 apply -f https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashboard-release.yaml
k3 apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
k3 apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
k3 apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
  • 安装 tekton CLI (tkn)
curl -OL https://github.com/tektoncd/cli/releases/download/v0.24.0/tektoncd-cli-0.24.0_Linux-64bit.deb
sudo dpkg -i ./tektoncd-cli-*.deb
which tkn

# 或
curl -OL https://github.com/tektoncd/cli/releases/download/v0.24.0/tkn_0.24.0_Linux_x86_64.tar.gz
tar -xf tkn*
cp tkn* /bin/tkn
  • 查看 tekton 组件
k3 -n tekton-pipelines get all

5. 配置 gitea webhook to Tekton

  • 配置 tekton 接收 webhook 的 secret

TODO

  • gitee webhook 配置同理，参见 #3

6. 配置 tekton (Trigger/PipelineRun/Task/...)

TODO

7. FAQ

7.1 部署完 k3s 后 k3 get node 节点显示 NotReady

  • 日志现象
# 查看 k3s server 日志
sudo journalctl -afu k3s

desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory

# 查看 k3s 的 containerd 日志
sudo tail -f /var/lib/rancher/k3s/agent/containerd/containerd.log

7.2 无法删除状态为 Terminating 的 pod

# 强制删除
k3 -n argocd delete all --all --force --grace-period=0

若强制删除还不行，可尝试设置 finalizers 为空
如果一个容器已经在运行，这时需要对一些容器属性进行修改，又不想删除容器，或不方便通过replace的方式进行更新。kubernetes还提供了一种在容器运行时，直接对容器进行修改的方式，就是patch命令：

k3 -n xxx patch pod xxx -n xxx -p '{"metadata":{"finalizers":null}}'

7.3 pod/argocd-server 无法调度错误

  • 错误现象
k3 -n argocd describe po argocd-application-controller-0

0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.
  • 查看节点污点
k3 -n argocd get nodes -o json | jq '.items[].spec'
{
  "podCIDR": "10.42.0.0/24",
  "podCIDRs": [
    "10.42.0.0/24"
  ],
  "providerID": "k3s://wanglsir-pro",
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "node.kubernetes.io/disk-pressure",
      "timeAdded": "2021-12-17T10:54:31Z"
    }
  ]
}

# 查看节点信息
k3 describe no $(hostname)
  • 解决污点
k3 taint node $(hostname) node.kubernetes.io/disk-pressure:NoSchedule-
k3 taint node $(hostname) node.kubernetes.io/unreachable-

7.4 pod/argocd-server 显示 Evicted 被驱逐错误

k3 -n argocd describe po argocd-server-5674bcbc44-jbr9n

The node was low on resource: ephemeral-storage. Container argocd-server was using 44Ki, which exceeds its request of 0.
  • 检查 k3s 节点资源是否有问题？重启 k3s、重试部署?
  • 查看镜像: k3s ctr i ls，清理 argocd 所有资源: k3s kubectl -n argocd delete all --all

7.5 如何解决 ImagePullBackOff 错误

7.5.1 配置 mirror 在线加速（推荐, 工业级方案）

  • 首先使用 skopeo / image-sync / image-transfer 等工具将 gcr.io 镜像自动同步到私有仓库(nexus3 / harbor 等)，具体参见：skopeo - image mirror 加速方案

  • 再将 k3s 仓库 mirror 指向私有代理仓库

sudo cat <<-EOF>/etc/rancher/k3s/registries.yaml
# see:https://rancher.com/docs/k3s/latest/en/installation/private-registry/#rewrites
mirrors:
  quay.io:
    endpoint:
      - "https://mirror.registry.your-privaterepo./o"
    #rewrite:
    #  "^argoproj/(.*)": "public/argoproj/$1"
  k8s.gcr.io:
    endpoint:
      - "https://mirror.registry.your-privaterepo.io"
  gcr.io:
    endpoint:
      - "https://mirror.registry.your-privaterepo.io"
configs:
  "mirror.registry.your-privaterepo.io":
    auth:
      username: '<your username>'
      password: '<your password>'
EOF

sudo systemctl restart k3s

# 测试从 k8s.gcr.io 拉镜像，拉取成功后可以在 nexus/harbor 控制台看到缓存的这个镜像
sudo k3s crictl pull k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0

本质上是 k3s 启动时检查是否有 /etc/rancher/k3s/registries.yaml，若存在则底层会将配置内容同步到内置的containerd 配置文件：/var/lib/rancher/k3s/agent/etc/containerd/config.toml

7.5.2 手动离线导入导出（适合环境恶劣、初学个人场景）

sudo k3s ctr -n k8s.io i import --base-name foo/bar foobar.tar

留言

您的电子邮箱地址不会被公开。