debugging Chainguard distroless containers

Chainguard images are lightweight, distroless images with minimum footprint, reduced attack surface and much less known vulnerabilities; so it serves as good starting point as a base image. Further, they provide both the dev and prod version of the most commonly used tech stack like java, python, php, etc. so i started to explore how to tap into it for my daily work.

I quickly realized that working with distroless images has its own pain and lacking of shell access is a top concern, and your usual tactics like ‘docker exec’ wouldn’t work by default, so I am exploring any possible workaround and I find the following master piece at https://iximiuz.com/en/posts/docker-debug-slim-containers/

my image dockerfile

FROM cgr.dev/chainguard/jre:latest-dev AS build
LABEL NAME = "WebGoat: built based on latest ChainGuard JRE image"

USER root

WORKDIR /tmp
RUN apk update && apk add curl
ENV URL=https://xxx.com
RUN curl -k -o agent.zip "${URL}/binaries/JAVA"
RUN mkdir -p /tmp/agent && unzip -d /tmp/agent agent.zip

WORKDIR /home/webgoat
RUN wget https://github.com/WebGoat/WebGoat/releases/download/v2023.4/webgoat-2023.4.jar -O /home/webgoat/webgoat.jar

FROM cgr.dev/chainguard/jre:latest
USER root
WORKDIR /home/webgoat
COPY --from=build /home/webgoat/webgoat.jar /home/webgoat/webgoat.jar
COPY --from=build /tmp/agent/agent.jar /tmp/agent/agent.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/tmp/agent/agent.jar"

ENTRYPOINT [ "java", \
   "-Duser.home=/home/webgoat", \
   "-Dfile.encoding=UTF-8", \
   "--add-opens", "java.base/java.lang=ALL-UNNAMED", \
   "--add-opens", "java.base/java.util=ALL-UNNAMED", \
   "--add-opens", "java.base/java.lang.reflect=ALL-UNNAMED", \
   "--add-opens", "java.base/java.text=ALL-UNNAMED", \
   "--add-opens", "java.desktop/java.beans=ALL-UNNAMED", \
   "--add-opens", "java.desktop/java.awt.font=ALL-UNNAMED", \
   "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED", \
   "--add-opens", "java.base/java.io=ALL-UNNAMED", \
   "--add-opens", "java.base/java.util=ALL-UNNAMED", \
   "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED", \
   "--add-opens", "java.base/java.io=ALL-UNNAMED", \
   "-Drunning.in.docker=true", \
   "-Dwebgoat.host=0.0.0.0", \
   "-Dwebwolf.host=0.0.0.0", \
   "-Dwebgoat.port=8080", \
   "-Dwebwolf.port=9090", \
   "-jar", "webgoat.jar" ]

so here we can easily leverage the dev image to install some utilities using apt and download some utilities and then copy to the prod images and ship out, and then we can launch the container with

docker run -it -p 8080:8080 --name target wg-undistro

so how do we that the process is actually running and listening on correct network ports?

our usual docker exec tactics woudn’t work

# docker exec aef751eec466 sh
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

we could acheive this by using the approach mentioned above

# docker ps
CONTAINER ID   IMAGE         COMMAND                  CREATED         STATUS         PORTS                                       NAMES
aef751eec466   wg-undistro   "java -Duser.home=/h…"   4 seconds ago   Up 4 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   target


# docker run --rm -it --name debugger --pid container:target --network container:target busybox sh
/ # hostname
aef751eec466

/ # netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN      1/java
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      1/java

/ # ps aux
PID   USER     TIME  COMMAND
    1 root      2:10 java -Duser.home=/home/webgoat -Dfile.encoding=UTF-8 --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --
  123 root      0:00 sh
  131 root      0:00 ps aux


/ # ls /proc/1/root/home
java     webgoat
/ # ls /proc/1/root/home/webgoat/
webgoat.jar