arthas 之 attach kubernetes Pod 中 JVM 失败解决

• Origin refer to: github.com/wl4g-k8s/docker-toolbox

• Origin refer to: gitee.com/wl4g-k8s/docker-toolbox

1. Quick start

1.1 将临时工具容器 Attach 到目标 Pod

• Scenario: James now finds that the coredns pod network is not connected,
  he needs to use network tools to troubleshooting, he can do that.

• Note: This attach ephemeral container can share the pid/utc/network/ipc namespace with the
  target pod, but not the mnt namespace, because the kubernetes design does not allow sharing of rootfs,
  because the authors are concerned that rootfs file writes will pollute each other.

# export imageUrl="docker.io/wl4g/toolbox-base"
export imageUrl="registry.cn-shenzhen.aliyuncs.com/wl4g-k8s/toolbox-arthas"

alias k="kubectl" # alias k="sudo k3s kubectl"

export podName=$(k -n kube-system get pods | grep coredns | awk -F ' ' '{print $1}')
k -n kube-system debug -it ${podName} --image=${imageUrl} --target=coredns

1.2 以 pod 副本运行 debug toolbox 容器

• The advantage of this method is that it is safer than directly attaching temporary containers, because direct attaching
  may cause pod resources to exceed the limit and be evicted, but the disadvantage is also obvious, because some failures
  may need to preserve the running memory state, the way to create pod replicas will lose this states information.

k -n kube-system debug -it ${podName} --image=${imageUrl} --copy-to debug-pod --share-processes

1.3 arthas attach 失败排查

• 1.3.1 前置条件: If it is an environment less than kubernetes 1.23, the target Pod must run with root privileges, such as: spec.containers[].securityContext.runAsNonRoot=false,runAsUser=0. because pods follow immutability, i.e. they cannot be modified once created (but openshift supports kubectl debug --as-root, see to: #3.6)

• 1.3.2 Attach 临时容器并进入终端

# export imageUrl="docker.io/wl4g/toolbox-arthas"
export imageUrl="registry.cn-shenzhen.aliyuncs.com/wl4g-k8s/toolbox-arthas"

alias k="kubectl" # alias k="sudo k3s kubectl"

export podName=$(k -n biz-app get pods | grep myapp | awk -F ' ' '{print $1}')
k -n biz-app debug -it ${podName} --image=${imageUrl} --target=myapp

• 1.3.3 手动复制 arthas jars 到目标 pod 的 rootfs

# Get myapp JVM pid
export jvmPid=$(ps -ef | grep java | grep -v grep | cut -c 9-16 | sed 's/ //g')

cp -r /tmp/.arthas/ /proc/${jvmPid}/root/root/

• 1.3.4 手动复制 jps 命令到目标 pod 的 rootfs

# 如下是镜像 openjdk:8u212-jre-alpine3.9 中jre的实际路径.
# 注：请根据实际路径修改目标rootfs中的jre路径。
cp $(which jps) /proc/${jvmPid}/root/usr/lib/jvm/java-1.8-openjdk/jre/../lib/

• 1.3.5 Run arthas.

# 虽然在调试临时容器的终端启动arthas通常更优雅，但为了attach成功率最好直接进入目标pod启动arthas。
k -n biz-app exec -it pods/${podName} -- /bin/sh

cd /root/.arthas/lib/*/arthas/
java -jar arthas-boot.jar

• 1.3.6 Attach 失败故障排除

  • 下面的错误可能是应用 istio proxy 或者应用 jvm 无法加载到 arthas jars，sock 没有监听成功，导致 attach 失败。 更多内容：#3.7

[INFO] arthas-boot version: 3.6.2
[INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
* [1]: 7 com.xxx.MyApp

[INFO] arthas home: /root/.arthas/lib/3.6.2/arthas
[INFO] Try to attach process 7
[ERROR] Start arthas failed, exception stack trace: 
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
        at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
        at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
        at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:102)
        at com.taobao.arthas.core.Arthas.<init>(Arthas.java:27)
        at com.taobao.arthas.core.Arthas.main(Arthas.java:151)
[ERROR] attach fail, targetPid: 7

• 新开一个目标 pod 的终端并 watch arthas-boot.jar 执行时的进程命令（不管 arthas 是在临时容器终端启动还是在目标容器终端启动）

while true; do ps -ef | grep arthas; sleep 0.2; done

/usr/lib/jvm/java-1.8-openjdk/jre/../bin/java -Xbootclasspath/a:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/tools.jar -jar /root/.arthas/lib/3.6.2/arthas/arthas-core.jar -pid 7 -core /root/.arthas/lib/3.6.2/arthas/arthas-core.jar -agent /root/.arthas/lib/3.6.2/arthas/arthas-agent.jar

2. Development Guide

./build.sh build
./build.sh push

3. References

• 3.1 kubernetes.io security context docs

• 3.2 kubernetes.io ephemeral-volumes docs

• 3.3 kubectl debug 之最佳实践. #411724

• 3.4 kubectl debug 之无法访问 non-root 启动的容器. #105846

• 3.5 kubectl debug 临时容器一旦附加无法删除， 且多次 attach 会导致临时容器过多，当超过 1.5M 时 etcd 会报错

• 3.6 openshift 支持启动具有 root 权限的 debug pod

• 3.7 arthas 在 ephemeral container attach 到目标 pod 失败. #1874

• 3.8 开发一个 arthas native agent 和远程管理平台，统一实时管理 JVM 集群. #2163