Kubernetes

容器化部署之 image mirror 加速方案

2022-02-07 /

容器化部署之 image mirror 加速方案

1. 常用公共镜像仓库

2. 墙内镜像加速

tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://hjbu3ivg.mirror.aliyuncs.com"]
}
EOF
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
  • 重启 Docker 服务：
systemctl daemon-reload && sudo systemctl restart docker
  • 检查配置生效
docker info | grep Mirrors -A1
Registry Mirrors:
 https://hjbu3ivg.mirror.aliyuncs.com/
  • 验证 pull 速度
time docker pull centos

3. 公私有镜像仓库同步

以下工具都不依赖docker环境，同步镜像时不落本地盘：

可结合公网免费CICD工具githubAction、TravisCI、CircleCI、gitlabCI等，将gcr.io镜像定时同步到个人dockerhub账号下或国内阿里云镜像仓库中。

国内镜像源替换示例，针对部分国外镜像仓库无法访问问题，国内阿里云或dockerhub可能有转存镜像可用，拉取后需自行修改tag：

docker pull k8s.gcr.io/pause:3.2
# 替换为
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2

4. 使用 skopeo 同步（推荐）

为啥推荐 skopeo，一方面因为属于 Open container 社区的，star 比如上腾讯阿里的也很多，二是上面2个工具亲测发现同步到 nexus docker hosted 仓库后发现没有目录结构（因为ContentType不是application/vnd.docker.distribution.manifest.v2+json，暂未未找到设置参数），只有 skopeo copy 或 sync 能指定格式为 docker ，同步后的都有目录结构（与 docker push 的一致）

4.1 使用官方原始包

# Fedora:
sudo dnf install gpgme-devel libassuan-devel btrfs-progs-devel device-mapper-devel

# RHEL/CentOS ≤ 7.x
sudo yum -y install skopeo

# Ubuntu (`libbtrfs-dev` requires Ubuntu 18.10 and above):
sudo apt install libgpgme-dev libassuan-dev libbtrfs-dev libdevmapper-dev pkg-config

# macOS:
brew install gpgme

# Alpine
sudo apk add skopeo

4.2 方案1：使用 skopeo-docker 增强镜像安装 (推荐) 

docker run --rm docker.io/wl4g/skopeo:1.8.0 skopeo --help
# 或
docker run --rm registry.cn-shenzhen.aliyuncs.com/wl4g-k8s/skopeo:1.8.0 skopeo --help
  • 4.2.1 简单使用
# 查看指定仓库镜像的元数据
skopeo inspect docker://registry.fedoraproject.org/fedora:latest
skopeo inspect docker://registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.2

# copy指定仓库镜像到远程私有仓库
skopeo copy docker://registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.2 docker://mirror.registry.privaterepo.com/k8s/kube-scheduler
  • 4.2.2 将 aliyuncs 仓库中 kube-scheduer 镜像所有tags 全量同步到私有仓库
skopeo sync --src docker --dest docker registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler mirror.registry.privaterepo.com/k8s/kube-scheduler
  • 4.2.3 按需批量同步
# 指定需同步的 tags 规则
cat <<-'EOF' > sync.yml
registry.cn-hangzhou.aliyuncs.com:
    images-by-tag-regex:
        google_containers/kube-scheduler: ^v1\.1[89]\.[1234]$|^v1\.2[123456789]\.[1234]$
EOF

# 执行
skopeo sync --src yaml --dest docker sync.yml mirror.registry.privaterepo.com/public

如上将会从 aliyuncs 公共仓库 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler 同步 v1.18.1, v1.18.2 ... v1.21.1...v1.22.1,v1.22.2... 到私有仓库 mirror.registry.privaterepo.com/public/kube-scheduler

通常 skopeo 进程可能就在 privaterepo 私服上运行，可将 mirror.registry.privaterepo.com 解析到 127.0.0.1 以加速写入。

同步到私有 nexus3 docker hosted 截图：

5. 方案2：使用 image-syncer 同步

image-syncer 是阿里云开源的一个docker镜像同步工具，使用golang开发，可用来进行多对多镜像仓库同步，支持绝大多数主流docker镜像仓库服务。下面演示基本使用方法。

5.1 二进制安装

wget https://github.com/AliyunContainerService/image-syncer/releases/download/v1.3.0/image-syncer-v1.3.0-linux-amd64.tar.gz
tar -zxvf image-syncer-*.tar.gz -C /usr/local/bin

5.2 创建同步配置

如下示例为仓库级同步，更多同步参见官方文档：https://github.com/AliyunContainerService/image-syncer

cat > config.yaml <<EOF
auth:
  #registry.hub.docker.com:
  #  username: xxx
  #  password: xxx
  #k8s.gcr.io:
  #  username: xxx
  #  password: xxx
  #registry.cn-shenzhen.aliyuncs.com:
  #  username: xxx
  #  password: xxx
  mirror.registry.privaterepo.com:
    username: xxx
    password: xxx
images:
  k8s.gcr.io/metrics-server/metrics-server: mirror.registry.privaterepo.com/k8s/metrics-server
  registry.cn-hangzhou.aliyuncs.com/google_containers/pause: mirror.registry.privaterepo.com/k8s/pause
  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver: mirror.registry.privaterepo.com/k8s/kube-apiserver
  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager: mirror.registry.privaterepo.com/k8s/kube-controller-manager
  registry.cn-hangzhou.aliyuncs.com/google_containers/kubelet: mirror.registry.privaterepo.com/k8s/kubelet
  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler: mirror.registry.privaterepo.com/k8s/kube-scheduler
EOF
  • 执行同步，将从 gcr.io 同步至 dockerhub 及阿里云镜像仓库
image-syncer --proc=6 --config=config.yaml --retries=3

6. 方案3：使用 image-transfer 同步

wget https://github.com/tkestack/image-transfer/releases/download/v1.2.3/image-transfer-linux-amd64.tar.gz
tar -zxvf image-transfer-*.tar.gz -C /usr/local/bin
cat <<-'EOF' > registry-secret.yaml
#registry.hub.docker.com:
#  username: xxx
#  password: xxx
#k8s.gcr.io:
#  username: xxx
#  password: xxx
#registry.cn-hangzhou.aliyuncs.com:
#  username: xxx
#  password: xxx
mirror.registry.privaterepo.com:
  username: xxx
  password: xxx
EOF

cat <<-'EOF' > transfer-rule.yaml
k8s.gcr.io/metrics-server/metrics-server: mirror.registry.privaterepo.com/k8s/metrics-server
registry.cn-hangzhou.aliyuncs.com/google_containers/pause: mirror.registry.privaterepo.com/k8s/pause
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver: mirror.registry.privaterepo.com/k8s/kube-apiserver
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager: mirror.registry.privaterepo.com/k8s/kube-controller-manager
registry.cn-hangzhou.aliyuncs.com/google_containers/kubelet: mirror.registry.privaterepo.com/k8s/kubelet
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler: mirror.registry.privaterepo.com/k8s/kube-scheduler
EOF
  • 执行同步，将从 gcr.io 同步至 dockerhub 及阿里云镜像仓库
image-transfer --securityFile=./registry-secret.yaml --ruleFile=./transfer-rule.yaml \
--ns=default --registry=ccr.ccs.tencentyun.com --routines=5 --retry=3

7. 配置容器引擎

7.1 docker

# 向 deamon.json 中 registry-mirrors 字段追加新 mirror 地址
sudo jq --argjson new_mirror '["https://mirror.registry.privaterepo.com"]' '."registry-mirrors" += $new_mirror' /etc/docker/daemon.json

# 重启
sudo systemctl restart docker

7.2 containerd

sudo mkdir -p /etc/containerd/
sudo containerd config default > /etc/containerd/config.toml # 追加修改以下内容

...
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
          endpoint = ["https://mirror.registry.privaterepo.com"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."mirror.registry.privaterepo.com".auth]
          auth = "d2w0ZzoxMjM0NTY="
          ## Priority use username-password.
          ## see:https://blogs.wl4g.com/archives/66#4.2
          ## see:https://github.com/containerd/cri/blob/master/docs/registry.md#configure-registry-endpoint
          username = "wl4g"
          password = "123456"
          #identitytoken = ""
...

7.3 k3s

sudo cat <<-EOF>/etc/rancher/k3s/registries.yaml
# see:https://rancher.com/docs/k3s/latest/en/installation/private-registry/#rewrites
mirrors:
  quay.io:
    endpoint:
      - "https://mirror.registry.your-privaterepo./o"
    #rewrite:
    #  "^argoproj/(.*)": "public/argoproj/$1"
  k8s.gcr.io:
    endpoint:
      - "https://mirror.registry.your-privaterepo.io"
  gcr.io:
    endpoint:
      - "https://mirror.registry.your-privaterepo.io"
configs:
  "mirror.registry.your-privaterepo.io":
    auth:
      username: '<your username>'
      password: '<your password>'
EOF

# 重启
sudo systemctl restart k3s

# 测试从 k8s.gcr.io 拉镜像，拉取成功后可以在 nexus/harbor 控制台看到缓存的这个镜像
sudo k3s crictl pull k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0

8. 参考资料

留言

您的电子邮箱地址不会被公开。