minikube 搭建 k8s 应用本地开发调试环境

• KIND 搭建 K8S 应用本地开发调试环境

• 官方文档: minikube.sigs.k8s.io/docs/start/

1. Deploy local k8s cluster via minikube.

1.1 Installation k8s cluster.

• Installation docker

sudo apt install -y docker.io

• Installation minikube

# Way1:
curl -OL https://github.com/kubernetes/minikube/releases/download/v1.25.1/minikube-linux-amd64
# or
curl -OL https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo mv minikube-linux-amd64 /bin/minikube

# Way2:
git clone https://github.com/kubernetes/minikube
cd minkube && make linux # Build for linux

• Create local k8s cluster

  • Notes: Please make sure the host (or virtual machine) configuration is higher than minikube minimum requirements: 2C/2G

  minikube start \
  --driver=docker \
  --apiserver-ips=127.0.0.1,10.0.0.130 \
  --force

  • or

  minikube start \
  --driver=docker \
  --apiserver-ips=127.0.0.1,10.0.0.130 \
  --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
  --registry-mirror=https://hjbu3ivg.mirror.aliyuncs.com \
  --base-image=kicbase/stable \
  --force

1.2 Configuration on remote management

Allow to manage minikube k8s cluster with remote.

• Set alias commands

# Example ubuntu20.04
echo "alias k='minikube kubectl -- '" > /etc/bash.bashrc && . /etc/bash.bashrc

• Create apiserver pod forward tunnel. (on minikube k8s cluster node)

k port-forward pods/kube-apiserver-minikube --address=0.0.0.0 8443:8443 -n kube-system

• Copy minikube kubeconfig to remote manage PC node.

mkdir -p $HOME/.kube/minikube/ && cd $HOME/.kube/
scp root@10.0.0.130:/root/.kube/config .
cd $HOME/.kube/minikube/
scp root@10.0.0.130:/root/.minikube/ca.crt .
scp root@10.0.0.130:/root/.minikube/profiles/minikube/client.* .

• Modify kubeconfig.

cd $HOME/.kube/
sed -ri "s#certificate-authority:(.+)#certificate-authority: $HOME/.kube/minikube/ca.crt#g" config
sed -ri "s#server:(.+)#server: https://10.0.0.130:8443#g" config
sed -ri "s#client-certificate:(.+)#client-certificate: $HOME/.kube/minikube/client.crt#g" config
sed -ri "s#client-key:(.+)#client-key: $HOME/.kube/minikube/client.key#g" config

• Testing remote manage for k8s cluster.

k get po -A

NAMESPACE     NAME                               READY   STATUS    RESTARTS      AGE
kube-system   coredns-78fcd69978-hqtc9  1/1   Running   1 (39m ago)   77m
kube-system   etcd-minikube                      1/1   Running   1 (39m ago)   77m
kube-system   kube-apiserver-minikube     1/1   Running   1 (39m ago)   77m
kube-system   kube-controller-manager-minikube  1/1   Running   1 (39m ago)  77m
kube-system   kube-proxy-wcn62              1/1    Running   1 (39m ago)   77m
kube-system   kube-scheduler-minikube   1/1   Running   1 (39m ago)   77m
kube-system   storage-provisioner            1/1    Running   2 (38m ago)   77m

1.3 Debugging for minikube etcd

# enter minikube etcd pod.
minikube kubectl -- exec -it -n kube-system etcd-minikube -- sh

# example get etcd data for /.
export ETCDCTL_API=3
etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/minikube/certs/etcd/ca.crt \
--cert=/var/lib/minikube/certs/etcd/server.crt \
--key=/var/lib/minikube/certs/etcd/server.key \
get / --prefix --keys-only

# example get pods.
etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/minikube/certs/etcd/ca.crt \
--cert=/var/lib/minikube/certs/etcd/server.crt \
--key=/var/lib/minikube/certs/etcd/server.key \
get /registry/pods --prefix

2. FAQ

2.1 Exiting due to DRV_AS_ROOT: The “docker“ driver should not be used with root privileges.

[root@kubeadm-1 ~]# minikube start --driver=docker
😄  minikube v1.23.3 on Centos 7.8.2003 (amd64)
✨  Using the docker driver based on user configuration
🛑  The "docker" driver should not be used with root privileges.
💡  If you are running minikube within a VM, consider using --driver=none:
📘    https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❌  Exiting due to DRV_AS_ROOT: The "docker" driver should not be used with root privileges.

• 解决方案: 增加 --force 选项

2.2 Can't find the containers of the pods started by minikube use docker ps ?

• 当minikube start 时--driver=docker时（默认自适应），会在伪节点容器启动内置 dockerd 进程，可使用ps -ef | grep dockerd来验证，会输出2个不同的进程，其中进程号等于systemctl status docker则就是您在宿主机上安装的，另外一个就是 minikube 内置启动的 dockerd.

• 解决方案: 执行 minikube docker-env ，即将宿主机上 docker 命令的 socket 指向 minikube 的 dockerd，此时在宿主机上执行 docker ps 就能看到 minikube 运行的容器列表（注：但之前在宿主机上运行的容器会看不见，其原理就是如:export DOCKER_HOST="tcp://192.168.49.2:2376" 指向 minikube 内置启动的 dockerd，或重新打开 shell terminal 即可恢复）。

2.3 How to configure command line short aliases?

• curl -L -o /etc/profile.d/profile-aliases-k8s.sh 'https://gitee.com/wl4g/xcloud-blogs/raw/master/articles/kubernetes/k8s-app-local-devel-by-kind/script/profile-aliases-k8s.sh' && . /etc/profile