Tools

Tools: Complete Guide to Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

2026-04-16 0 views admin
Tools: Complete Guide to Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Why Distroless Breaks the Normal Debugging Workflow

Option 1: kubectl debug with Ephemeral Containers

Capabilities and Limitations

Accessing the Application Filesystem

RBAC Requirements

Option 2: kubectl debug --copy-to (Pod Copy Strategy)

Limitations

Option 3: Debug Image Variants

Security Considerations

Option 4: cdebug

Option 5: Node-Level Debugging

RBAC and Security

Choosing the Right Approach: Access Profile Matrix

Developer — Local or Development Cluster

Developer — Staging Cluster

Platform Engineer / SRE — Production

Platform Engineer — Production, Node-Level Issue

Common Errors and Solutions

"ephemeral containers are disabled for this cluster"

"cannot update ephemeralcontainers" (RBAC)

"container not found" with --target

Can see processes but cannot read /proc/1/root

tcpdump shows no traffic

Production RBAC Design

Summary Traditional container debugging assumes shell access with standard tools like ps, netstat, and curl. Distroless images intentionally exclude these utilities to reduce attack surface and CVEs. This creates an operational challenge: "when something goes wrong, you cannot use the tools that the process itself is not allowed to run." Kubernetes addresses this through ephemeral containers, stabilized in version 1.25, which enable temporary debug containers to be injected into running pods. The canonical solution uses ephemeral containers to inject a debug container sharing the target pod's network and process namespaces without modifying the original container or restarting the pod. The --target flag shares the process namespace of the specified container, enabling inspection via ps aux and /proc/ access. For network diagnostics, use a richer image: Ephemeral containers provide: Ephemeral containers do not provide: The workaround for filesystem access uses the /proc filesystem: The /proc//root symlink provides read access to the container's filesystem. Ephemeral containers require the pods/ephemeralcontainers subresource permission, separate from pods/exec: In production, scope this tightly with time-limited bindings and approval workflows. The --copy-to flag creates a full pod copy with modifications: This creates a new pod with the container image replaced. Add a debug container alongside the original: The copy strategy does not debug the original pod because: For crash debugging, combine with a modified entrypoint: Maintain a debug variant of your application image including shell tooling. Google distroless images provide :debug tags with BusyBox: Chainguard images follow a similar pattern with :latest-dev variants that include apk and shell: For custom images, use multi-stage builds: Build both targets and push my-app:${VERSION} (production) and my-app:${VERSION}-debug (debug) to your registry. Debug image variants undermine distroless security benefits if deployed to production. Track usage carefully, require explicit approval, and ensure removal after debugging. cdebug is an open-source CLI tool that simplifies ephemeral container debugging: The tradeoff is that it requires third-party tooling installation. For issues that ephemeral containers cannot address—pod crashing too fast, kernel-level problems, or tools requiring elevated privileges—node-level debugging provides direct container access from the host node: From the privileged pod, use nsenter to enter container namespaces: This approach enables running strace and other kernel-level tools: Node-level debugging requires nodes/proxy and ability to create privileged pods. The debug pod runs with hostPID: true and hostNetwork: true, providing visibility into all node processes. Treat this as a break-glass procedure with dual approval, complete audit logging, and immediate cleanup. Goal: Reproduce bugs, inspect configuration, verify service connectivity.

Approach: Debug image variants or cdebug. Speed and iteration take priority. Build the debug variant and deploy it directly, or use cdebug exec for automatic filesystem root access. Goal: Debug integration issues and environment-specific behavior.Approach: kubectl debug with ephemeral containers (--target), scoped to own namespace. Grant developers pods/ephemeralcontainers in their team's namespaces for self-service debugging without ops involvement. Goal: Diagnose live production incidents while minimizing risk.Approach: kubectl debug with ephemeral containers. Ephemeral containers satisfy production requirements: Avoid --copy-to in production incidents because it creates a pod that may not exhibit the issue and adds load during an incident. Goal: Diagnose kernel-level issues, container runtime problems, or multi-pod networking issues.

Approach: Node-level debug pod with nsenter. Treat as break-glass. Create a dedicated RBAC role that grants nodes/proxy access only on-demand with separate authentication and time-limited bindings. Log all access. Ephemeral containers require Kubernetes 1.16+ with the feature gate enabled. They are stable and always-on from Kubernetes 1.25. You have pods/exec but lack pods/ephemeralcontainers. These are separate subresources. The container name in --target must match exactly. Verify with: The ephemeral container may lack CAP_SYS_PTRACE capability. Use the Baseline PodSecurityStandards (PSS) profile for debug namespaces or explicitly add the capability: Use tcpdump -i any to capture on all interfaces including loopback, where inter-container traffic travels. Separate three privilege tiers: Tier 1: Developer self-service (team namespaces) Tier 2: SRE production incident access (all namespaces) Tier 3: Break-glass node access (time-limited binding recommended) Bind Tier 1 permanently to developers. Bind Tier 2 permanently to SREs with audit alerts on use. Bind Tier 3 only on-demand via a Kubernetes operator creating time-limited RoleBindings—never as a permanent ClusterRoleBinding. Distroless containers reduce attack surface and CVEs, forcing a clean separation between application and tooling. Kubernetes provides ephemeral containers and kubectl debug as the clean answer: inject a debug container with necessary tools into the running pod, sharing its network and process namespaces, without restarting or modifying the application. For scenarios ephemeral containers cannot address—filesystem access, crash debugging, kernel-level investigation—the copy strategy and node-level debug fill remaining gaps. The key to scaling this approach is the access model: developers get self-service ephemeral container access in their namespaces, SREs get cluster-wide ephemeral container access for production incidents, and node-level access is a break-glass procedure with audit trail and time limits. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Command

Copy

$ -weight: 500;">kubectl debug -it my-pod \ --image=busybox:latest \ --target=my-container -weight: 500;">kubectl debug -it my-pod \ --image=busybox:latest \ --target=my-container -weight: 500;">kubectl debug -it my-pod \ --image=busybox:latest \ --target=my-container -weight: 500;">kubectl debug -it my-pod \ --image=nicolaka/netshoot \ --target=my-container -weight: 500;">kubectl debug -it my-pod \ --image=nicolaka/netshoot \ --target=my-container -weight: 500;">kubectl debug -it my-pod \ --image=nicolaka/netshoot \ --target=my-container # Browse via /proc ls /proc/1/root/app/ cat /proc/1/root/etc/config.yaml # Or chroot into the application's filesystem chroot /proc/1/root /bin/sh # Browse via /proc ls /proc/1/root/app/ cat /proc/1/root/etc/config.yaml # Or chroot into the application's filesystem chroot /proc/1/root /bin/sh # Browse via /proc ls /proc/1/root/app/ cat /proc/1/root/etc/config.yaml # Or chroot into the application's filesystem chroot /proc/1/root /bin/sh apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ephemeral-debugger rules: - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ephemeral-debugger rules: - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ephemeral-debugger rules: - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] -weight: 500;">kubectl debug my-pod \ -it \ --copy-to=my-pod-debug \ --image=my-app:debug \ --share-processes -weight: 500;">kubectl debug my-pod \ -it \ --copy-to=my-pod-debug \ --image=my-app:debug \ --share-processes -weight: 500;">kubectl debug my-pod \ -it \ --copy-to=my-pod-debug \ --image=my-app:debug \ --share-processes -weight: 500;">kubectl debug my-pod \ -it \ --copy-to=my-pod-debug \ --image=busybox \ --share-processes \ --container=debugger -weight: 500;">kubectl debug my-pod \ -it \ --copy-to=my-pod-debug \ --image=busybox \ --share-processes \ --container=debugger -weight: 500;">kubectl debug my-pod \ -it \ --copy-to=my-pod-debug \ --image=busybox \ --share-processes \ --container=debugger -weight: 500;">kubectl debug my-crashing-pod \ -it \ --copy-to=my-pod-debug \ --image=busybox \ --share-processes \ -- sleep 3600 -weight: 500;">kubectl debug my-crashing-pod \ -it \ --copy-to=my-pod-debug \ --image=busybox \ --share-processes \ -- sleep 3600 -weight: 500;">kubectl debug my-crashing-pod \ -it \ --copy-to=my-pod-debug \ --image=busybox \ --share-processes \ -- sleep 3600 # Production image FROM gcr.io/distroless/java17-debian12 # Debug variant FROM gcr.io/distroless/java17-debian12:debug # Production image FROM gcr.io/distroless/java17-debian12 # Debug variant FROM gcr.io/distroless/java17-debian12:debug # Production image FROM gcr.io/distroless/java17-debian12 # Debug variant FROM gcr.io/distroless/java17-debian12:debug # Production FROM cgr.dev/chainguard/go:latest # Development/debug FROM cgr.dev/chainguard/go:latest-dev # Production FROM cgr.dev/chainguard/go:latest # Development/debug FROM cgr.dev/chainguard/go:latest-dev # Production FROM cgr.dev/chainguard/go:latest # Development/debug FROM cgr.dev/chainguard/go:latest-dev FROM golang:1.22 AS builder WORKDIR /app COPY . . RUN go build -o myapp . FROM gcr.io/distroless/static-debian12 AS production COPY --from=builder /app/myapp /myapp ENTRYPOINT ["/myapp"] FROM gcr.io/distroless/static-debian12:debug AS debug COPY --from=builder /app/myapp /myapp ENTRYPOINT ["/myapp"] FROM golang:1.22 AS builder WORKDIR /app COPY . . RUN go build -o myapp . FROM gcr.io/distroless/static-debian12 AS production COPY --from=builder /app/myapp /myapp ENTRYPOINT ["/myapp"] FROM gcr.io/distroless/static-debian12:debug AS debug COPY --from=builder /app/myapp /myapp ENTRYPOINT ["/myapp"] FROM golang:1.22 AS builder WORKDIR /app COPY . . RUN go build -o myapp . FROM gcr.io/distroless/static-debian12 AS production COPY --from=builder /app/myapp /myapp ENTRYPOINT ["/myapp"] FROM gcr.io/distroless/static-debian12:debug AS debug COPY --from=builder /app/myapp /myapp ENTRYPOINT ["/myapp"] # Install -weight: 500;">brew -weight: 500;">install cdebug # Debug a running pod cdebug exec -it my-pod # Specify namespace and container cdebug exec -it -n production my-pod -c my-container # Use specific debug image cdebug exec -it my-pod --image=nicolaka/netshoot # Install -weight: 500;">brew -weight: 500;">install cdebug # Debug a running pod cdebug exec -it my-pod # Specify namespace and container cdebug exec -it -n production my-pod -c my-container # Use specific debug image cdebug exec -it my-pod --image=nicolaka/netshoot # Install -weight: 500;">brew -weight: 500;">install cdebug # Debug a running pod cdebug exec -it my-pod # Specify namespace and container cdebug exec -it -n production my-pod -c my-container # Use specific debug image cdebug exec -it my-pod --image=nicolaka/netshoot -weight: 500;">kubectl debug node/my-node-name \ -it \ --image=nicolaka/netshoot -weight: 500;">kubectl debug node/my-node-name \ -it \ --image=nicolaka/netshoot -weight: 500;">kubectl debug node/my-node-name \ -it \ --image=nicolaka/netshoot # Find the container's PID crictl ps | grep my-container crictl inspect | grep pid # Enter the container's namespaces nsenter -t -m -u -i -n -p -- /bin/sh # Enter only network namespace nsenter -t -n -- ip a # Find the container's PID crictl ps | grep my-container crictl inspect | grep pid # Enter the container's namespaces nsenter -t -m -u -i -n -p -- /bin/sh # Enter only network namespace nsenter -t -n -- ip a # Find the container's PID crictl ps | grep my-container crictl inspect | grep pid # Enter the container's namespaces nsenter -t -m -u -i -n -p -- /bin/sh # Enter only network namespace nsenter -t -n -- ip a # Trace all syscalls from the application process nsenter -t -- strace -p -f -e trace=network # Trace all syscalls from the application process nsenter -t -- strace -p -f -e trace=network # Trace all syscalls from the application process nsenter -t -- strace -p -f -e trace=network -weight: 500;">kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' -weight: 500;">kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' -weight: 500;">kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' securityContext: capabilities: add: - SYS_PTRACE securityContext: capabilities: add: - SYS_PTRACE securityContext: capabilities: add: - SYS_PTRACE apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: distroless-debugger namespace: team-namespace rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: distroless-debugger namespace: team-namespace rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: distroless-debugger namespace: team-namespace rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: sre-distroless-debugger rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: sre-distroless-debugger rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: sre-distroless-debugger rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["-weight: 500;">update", "patch"] - apiGroups: [""] resources: ["pods/attach"] verbs: ["create", "get"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-debugger rules: - apiGroups: [""] resources: ["nodes/proxy"] verbs: ["get"] - apiGroups: [""] resources: ["pods"] verbs: ["create", "get", "list", "delete"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-debugger rules: - apiGroups: [""] resources: ["nodes/proxy"] verbs: ["get"] - apiGroups: [""] resources: ["pods"] verbs: ["create", "get", "list", "delete"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-debugger rules: - apiGroups: [""] resources: ["nodes/proxy"] verbs: ["get"] - apiGroups: [""] resources: ["pods"] verbs: ["create", "get", "list", "delete"] - Full network namespace visibility - Process inspection via /proc/ (open files, environment variables, memory maps) - Pod-level DNS resolution access - Outbound network calls from the pod's network context - Direct application container filesystem access - Container removal after creation - Volume mount modifications via CLI - Resource limits support in the -weight: 500;">kubectl debug CLI - It lacks the original pod's in-memory state - It creates a new Pod UID, potentially triggering different admission policies - For crashing pods, the copy will also crash unless the entrypoint is modified - Automatic filesystem chroot to the target container's filesystem - Docker container integration (cdebug exec) - No RBAC complications for Docker-based local development - They are recorded in API audit logs (who, when, which pod) - They do not modify the running application container - They are limited to the pod's network and process namespaces

🏷️ Tags

toolsutilitiessecurity toolscompleteguidedebuggingdistrolesscontainerskubectldebug

More from Tools

Tools: How to Set Up Let's Encrypt SSL on Any VPS

Tools: How to Set Up Let's Encrypt SSL on Any VPS

2026-04-16 0
Tools: I Built a Job Platform With Laravel 13, Livewire 4, and GPT-5 — Here's What I Learned

Tools: I Built a Job Platform With Laravel 13, Livewire 4, and GPT-5 — Here's What I Learned

2026-04-16 0
Tools: to Debug Microservices with Distributed Tracing How

Tools: to Debug Microservices with Distributed Tracing How

2026-04-16 0
Tools: Linux-Systemd-Nspawn: Leichtgewichtige Container (2026)

Tools: Linux-Systemd-Nspawn: Leichtgewichtige Container (2026)

2026-04-16 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views