Tools: Report: Kaniko Is Dead. Here's How I Build Tenant Images in Kubernetes Now.

Tools: Report: Kaniko Is Dead. Here's How I Build Tenant Images in Kubernetes Now.

Why Buildah

The Architecture

The Build Job

The Build Script

Image Tagging Strategy

GitHub Token Resolution

Registry Credentials

Polling, Failure Handling, and Cleanup

The Builder Interface Pattern

What I'd Do Differently

The Numbers

Try It Google archived Kaniko in June 2025. Docker-in-Docker requires privileged containers. I needed a third option for my project so I built a Buildah-based image pipeline that runs as Kubernetes Jobs, clones source, auto-generates Dockerfiles, and pushes to a private registry. Here's exactly how it works. If you're building container images inside a Kubernetes cluster in 2026, your options have narrowed faster than you might think. Docker-in-Docker requires mounting the Docker socket or running privileged containers, effectively giving the build container root access to the host. Every security guide tells you not to do this. OWASP's Docker Security Cheat Sheet says it plainly says it plainly: "Do not run containers with the --privileged flag." Kaniko was the answer for years, Google's purpose-built tool for building images inside unprivileged containers. No Docker daemon needed. Then in June 2025, Google archived the repository. It's now read-only. GitLab has already removed their Kaniko documentation and recommends Buildah or Podman instead. I was building Staxa, a multi-tenant PaaS that provisions isolated customer environments via API and I needed an image build pipeline that would run reliably inside k3s on a single ARM64 server. I started with Kaniko, saw the writing on the wall, and switched to Buildah. This post is a walkthrough of how that pipeline actually works. When a Staxa user deploys an app, here's what happens at the build stage: After Kaniko's archival, three alternatives emerged as the main contenders: BuildKit: Docker's modern build backend. Fast, parallel DAG-based execution and great caching. But running BuildKit as a standalone service inside Kubernetes adds another long-lived component to manage. For a platform that values minimal moving parts, that's overhead I didn't want. Buildah: Red Hat's OCI image builder, now part of the Podman Container Tools suite accepted into the CNCF Sandbox in January 2025. Buildah builds standard Dockerfiles, pushes to any registry, and runs as a one-shot process, no daemon required. It fits naturally into the Kubernetes Job model: spin up a pod, build, push, exit. werf / Kimia: Higher-level tools that wrap Buildah or BuildKit. More features, more opinions, more dependencies. Not what I wanted for a platform where every external dependency is a liability. Buildah won because it maps cleanly onto a single pattern: one Kubernetes Job per build, no persistent infrastructure. staxa-system namespace The Go API orchestrates everything through client-go. No shell scripts wrapping kubectl. No Helm charts. The builder is a Go struct that implements a Builder interface: This interface is the seam that let me swap Kaniko for Buildah without touching the rest of the pipeline. The deploy pipeline calls builder.Build() and gets back an image tag. It doesn't know or care which tool actually built the image. Here's what the Buildah Kubernetes Job looks like when the Go API creates it: So I made a few deliberate choices here: BackoffLimit: 0: If a build fails, it fails. No automatic retries that could mask a real error. The user sees the failure immediately via SSE events and gets build logs. RestartPolicy: Never: Same philosophy. The Job runs once. If something goes wrong, the user redeploys with a fix rather than the platform silently retrying with the same broken code. Minimal capabilities: Instead of running the build container with full privileges, only seven specific Linux capabilities are granted: CHOWN, DAC_OVERRIDE, and FOWNER for file ownership during COPY --chown; SETUID and SETGID for USER directives; SETFCAP for file capabilities in base images; and SYS_CHROOT for Buildah's chroot isolation mode. This is the minimum set that lets Buildah build arbitrary Dockerfiles. Labels: Every build Job is labeled with the deployment ID. This is how we find the associated pods later when we need to grab logs from a failed build. The actual build runs as a shell script inside the Buildah container: --storage-driver vfs: This is the key to running Buildah without overlayfs kernel support. VFS copies each layer fully rather than using filesystem overlays. It's slower than overlayfs, but it works everywhere including inside a container on k3s on an ARM64 Hetzner box. No special kernel modules required. --isolation chroot: Instead of spinning up a full OCI runtime (runc) for each build step, Buildah uses a simple chroot. This keeps the capability requirements minimal, chroot isolation only needs SYS_CHROOT rather than the broad set of privileges a full OCI runtime would demand. --layers: Enables layer caching. Subsequent builds reuse unchanged layers, which significantly speeds up redeploys where only application code changed. --depth 1: Shallow clone. We don't need git history, so we save time and bandwidth by only pulling the latest commit on the target branch. $BUILD_ARGS: Tenant environment variables are injected as --build-arg flags by the Go API. This is critical for frameworks like Next.js that inline NEXT_PUBLIC_* variables at build time they need to be available during the image build, not just at runtime. Every image is tagged with a deterministic, versioned path: For example: registry.staxa-system:5000/tenants/ten_abc/app:3 The Go code that generates this: Private repos need authentication. Staxa handles this with a two-tier fallback: If a GitHub App is configured, each deployment gets a short-lived installation access token (Tier 1). If not, it falls back to a cluster-level secret. If neither exists, the build proceeds without auth which works fine for public repos, fails fast for private ones with a clear error. Similarly, pushing to registries that require auth uses a Docker-compatible credentials mount: Buildah reads Docker-compatible auth from /root/.docker/config.json, so the same credentials format works for both tools which is another reason the Kaniko → Buildah migration was straightforward. The Go API doesn't use Kubernetes watches for build status. It polls: Why polling over watches? The build is a synchronous step in an async pipeline. The worker goroutine is already dedicated to this deployment. Polling every 5 seconds for up to 10 minutes is simple, predictable, and doesn't require setting up watch infrastructure. The Kubernetes API handles it fine. When a build fails, the builder grabs the last 50 lines of logs from the pod: These logs get surfaced to the user through the SSE event stream. When a build fails, they see exactly why, not a generic "build failed" message. After every build (success or failure), the Job is deleted: No TTL controllers, no cron cleanup. The build created the Job, the build cleans it up. Simple. The reason swapping from Kaniko to Buildah was a one-file change: the Builder interface. Both KanikoBuilder and BuildahBuilder implement this interface. The deploy pipeline doesn't know which one is active. Switching was a matter of changing which struct gets instantiated at startup no changes for the pipeline, the worker, or the API handlers. There's also a NoopBuilder for testing that returns a deterministic image tag without building anything. The entire deployment pipeline can be tested end-to-end without ever running a real build. VFS performance. The vfs storage driver works everywhere but copies every layer fully. For a platform optimized for initial deploy speed ("60 seconds to a running tenant"), build time is the largest chunk. I'm exploring whether fuse-overlayfs could work in this environment to speed up builds without requiring privileged overlayfs access. Build caching across tenants. Currently each build starts fresh. Tenants deploying the same framework (say, Next.js 15) all download and install the same base dependencies independently. A shared cache layer could cut repeat build times significantly. BuildKit as an alternative. BuildKit's parallel DAG execution would be faster for multi-stage Dockerfiles. If build performance becomes a bottleneck at scale, running a BuildKit daemon as a persistent service in the cluster rather than per-build Jobs would be worth the operational tradeoff. Staxa handles the entire build pipeline, framework detection, Dockerfile generation, image building, and deployment all behind a single API call or dashboard wizard. 👉 Join the waitlist at staxa.dev This is the second post in the Building Staxa series. The first post covered the multi-tenancy problem and why I built a platform that automates full tenant isolation. Next up: how the framework detection system auto-detects 19 frameworks without any AI using deterministic file-marker matching and dependency-file inspection. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Code Block

Copy

type Builder interface { Build(ctx context.Context, job BuildJob) (imageTag string, err error) } type Builder interface { Build(ctx context.Context, job BuildJob) (imageTag string, err error) } type Builder interface { Build(ctx context.Context, job BuildJob) (imageTag string, err error) } bJob := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ Name: jobName, // "build-dep_abc123" Namespace: "staxa-system", Labels: map[string]string{ "staxa.dev/deployment-id": job.DeploymentID, }, }, Spec: batchv1.JobSpec{ BackoffLimit: &zero, // No retries — fail fast Template: corev1.PodTemplateSpec{ Spec: corev1.PodSpec{ RestartPolicy: corev1.RestartPolicyNever, Containers: []corev1.Container{{ Name: "buildah", Image: cfg.BuilderImage, Command: []string{"sh", "-c", buildScript()}, Env: env, SecurityContext: &corev1.SecurityContext{ RunAsUser: &rootUser, Capabilities: &corev1.Capabilities{ Add: []corev1.Capability{ "CHOWN", "DAC_OVERRIDE", "FOWNER", "SETUID", "SETGID", "SETFCAP", "SYS_CHROOT", }, }, }, }}, }, }, }, } bJob := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ Name: jobName, // "build-dep_abc123" Namespace: "staxa-system", Labels: map[string]string{ "staxa.dev/deployment-id": job.DeploymentID, }, }, Spec: batchv1.JobSpec{ BackoffLimit: &zero, // No retries — fail fast Template: corev1.PodTemplateSpec{ Spec: corev1.PodSpec{ RestartPolicy: corev1.RestartPolicyNever, Containers: []corev1.Container{{ Name: "buildah", Image: cfg.BuilderImage, Command: []string{"sh", "-c", buildScript()}, Env: env, SecurityContext: &corev1.SecurityContext{ RunAsUser: &rootUser, Capabilities: &corev1.Capabilities{ Add: []corev1.Capability{ "CHOWN", "DAC_OVERRIDE", "FOWNER", "SETUID", "SETGID", "SETFCAP", "SYS_CHROOT", }, }, }, }}, }, }, }, } bJob := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ Name: jobName, // "build-dep_abc123" Namespace: "staxa-system", Labels: map[string]string{ "staxa.dev/deployment-id": job.DeploymentID, }, }, Spec: batchv1.JobSpec{ BackoffLimit: &zero, // No retries — fail fast Template: corev1.PodTemplateSpec{ Spec: corev1.PodSpec{ RestartPolicy: corev1.RestartPolicyNever, Containers: []corev1.Container{{ Name: "buildah", Image: cfg.BuilderImage, Command: []string{"sh", "-c", buildScript()}, Env: env, SecurityContext: &corev1.SecurityContext{ RunAsUser: &rootUser, Capabilities: &corev1.Capabilities{ Add: []corev1.Capability{ "CHOWN", "DAC_OVERRIDE", "FOWNER", "SETUID", "SETGID", "SETFCAP", "SYS_CHROOT", }, }, }, }}, }, }, }, } set -e # Inject GitHub token for private repos if available CLONE_URL="$REPO_URL" if [ -n "${GITHUB_TOKEN:-}" ]; then CLONE_URL="$(echo "$REPO_URL" | \ sed "s|https://|https://x-access-token:${GITHUB_TOKEN}@|")" fi # Shallow clone — only the branch we need, one commit deep git clone --depth 1 --branch "$BRANCH" "$CLONE_URL" /workspace cd /workspace # Build with VFS storage driver + chroot isolation (no privileged mode needed) buildah --storage-driver vfs build \ --isolation chroot \ --layers \ $BUILD_ARGS \ --tag "$IMAGE_TAG" \ --file "$DOCKERFILE_PATH" \ "${BUILD_CONTEXT:-.}" # Push to internal registry (TLS disabled for in-cluster registry) buildah --storage-driver vfs push \ --tls-verify=false \ "$IMAGE_TAG" \ "docker://$IMAGE_TAG" set -e # Inject GitHub token for private repos if available CLONE_URL="$REPO_URL" if [ -n "${GITHUB_TOKEN:-}" ]; then CLONE_URL="$(echo "$REPO_URL" | \ sed "s|https://|https://x-access-token:${GITHUB_TOKEN}@|")" fi # Shallow clone — only the branch we need, one commit deep git clone --depth 1 --branch "$BRANCH" "$CLONE_URL" /workspace cd /workspace # Build with VFS storage driver + chroot isolation (no privileged mode needed) buildah --storage-driver vfs build \ --isolation chroot \ --layers \ $BUILD_ARGS \ --tag "$IMAGE_TAG" \ --file "$DOCKERFILE_PATH" \ "${BUILD_CONTEXT:-.}" # Push to internal registry (TLS disabled for in-cluster registry) buildah --storage-driver vfs push \ --tls-verify=false \ "$IMAGE_TAG" \ "docker://$IMAGE_TAG" set -e # Inject GitHub token for private repos if available CLONE_URL="$REPO_URL" if [ -n "${GITHUB_TOKEN:-}" ]; then CLONE_URL="$(echo "$REPO_URL" | \ sed "s|https://|https://x-access-token:${GITHUB_TOKEN}@|")" fi # Shallow clone — only the branch we need, one commit deep git clone --depth 1 --branch "$BRANCH" "$CLONE_URL" /workspace cd /workspace # Build with VFS storage driver + chroot isolation (no privileged mode needed) buildah --storage-driver vfs build \ --isolation chroot \ --layers \ $BUILD_ARGS \ --tag "$IMAGE_TAG" \ --file "$DOCKERFILE_PATH" \ "${BUILD_CONTEXT:-.}" # Push to internal registry (TLS disabled for in-cluster registry) buildah --storage-driver vfs push \ --tls-verify=false \ "$IMAGE_TAG" \ "docker://$IMAGE_TAG" {registry}/tenants/{tenant_id}/{service_name}:{version} {registry}/tenants/{tenant_id}/{service_name}:{version} {registry}/tenants/{tenant_id}/{service_name}:{version} imageTag := fmt.Sprintf("%s/tenants/%s/%s:%d", job.RegistryURL, job.TenantID, name, job.Version) imageTag := fmt.Sprintf("%s/tenants/%s/%s:%d", job.RegistryURL, job.TenantID, name, job.Version) imageTag := fmt.Sprintf("%s/tenants/%s/%s:%d", job.RegistryURL, job.TenantID, name, job.Version) func (b *BuildahBuilder) resolveGitHubToken(ctx context.Context, token string) []corev1.EnvVar { // Tier 1: Per-deployment token (from GitHub App installation) if token != "" { return []corev1.EnvVar{{Name: "GITHUB_TOKEN", Value: token}} } // Tier 2: Cluster-level secret (for environments without GitHub App) _, err := b.client.CoreV1().Secrets(buildahNamespace). Get(ctx, "github-credentials", metav1.GetOptions{}) if err != nil { return nil // No token available — public repo only } return []corev1.EnvVar{{ Name: "GITHUB_TOKEN", ValueFrom: &corev1.EnvVarSource{ SecretKeyRef: &corev1.SecretKeySelector{ LocalObjectReference: corev1.LocalObjectReference{ Name: "github-credentials", }, Key: "token", }, }, }} } func (b *BuildahBuilder) resolveGitHubToken(ctx context.Context, token string) []corev1.EnvVar { // Tier 1: Per-deployment token (from GitHub App installation) if token != "" { return []corev1.EnvVar{{Name: "GITHUB_TOKEN", Value: token}} } // Tier 2: Cluster-level secret (for environments without GitHub App) _, err := b.client.CoreV1().Secrets(buildahNamespace). Get(ctx, "github-credentials", metav1.GetOptions{}) if err != nil { return nil // No token available — public repo only } return []corev1.EnvVar{{ Name: "GITHUB_TOKEN", ValueFrom: &corev1.EnvVarSource{ SecretKeyRef: &corev1.SecretKeySelector{ LocalObjectReference: corev1.LocalObjectReference{ Name: "github-credentials", }, Key: "token", }, }, }} } func (b *BuildahBuilder) resolveGitHubToken(ctx context.Context, token string) []corev1.EnvVar { // Tier 1: Per-deployment token (from GitHub App installation) if token != "" { return []corev1.EnvVar{{Name: "GITHUB_TOKEN", Value: token}} } // Tier 2: Cluster-level secret (for environments without GitHub App) _, err := b.client.CoreV1().Secrets(buildahNamespace). Get(ctx, "github-credentials", metav1.GetOptions{}) if err != nil { return nil // No token available — public repo only } return []corev1.EnvVar{{ Name: "GITHUB_TOKEN", ValueFrom: &corev1.EnvVarSource{ SecretKeyRef: &corev1.SecretKeySelector{ LocalObjectReference: corev1.LocalObjectReference{ Name: "github-credentials", }, Key: "token", }, }, }} } func (b *BuildahBuilder) registryCredentials(ctx context.Context) ([]corev1.Volume, []corev1.VolumeMount) { _, err := b.client.CoreV1().Secrets(buildahNamespace). Get(ctx, "registry-credentials", metav1.GetOptions{}) if err != nil { return nil, nil // No credentials, internal registry, no auth needed } volumes := []corev1.Volume{{ Name: "registry-credentials", VolumeSource: corev1.VolumeSource{ Secret: &corev1.SecretVolumeSource{ SecretName: "registry-credentials", }, }, }} mounts := []corev1.VolumeMount{{ Name: "registry-credentials", MountPath: "/root/.docker/config.json", SubPath: "config.json", ReadOnly: true, }} return volumes, mounts } func (b *BuildahBuilder) registryCredentials(ctx context.Context) ([]corev1.Volume, []corev1.VolumeMount) { _, err := b.client.CoreV1().Secrets(buildahNamespace). Get(ctx, "registry-credentials", metav1.GetOptions{}) if err != nil { return nil, nil // No credentials, internal registry, no auth needed } volumes := []corev1.Volume{{ Name: "registry-credentials", VolumeSource: corev1.VolumeSource{ Secret: &corev1.SecretVolumeSource{ SecretName: "registry-credentials", }, }, }} mounts := []corev1.VolumeMount{{ Name: "registry-credentials", MountPath: "/root/.docker/config.json", SubPath: "config.json", ReadOnly: true, }} return volumes, mounts } func (b *BuildahBuilder) registryCredentials(ctx context.Context) ([]corev1.Volume, []corev1.VolumeMount) { _, err := b.client.CoreV1().Secrets(buildahNamespace). Get(ctx, "registry-credentials", metav1.GetOptions{}) if err != nil { return nil, nil // No credentials, internal registry, no auth needed } volumes := []corev1.Volume{{ Name: "registry-credentials", VolumeSource: corev1.VolumeSource{ Secret: &corev1.SecretVolumeSource{ SecretName: "registry-credentials", }, }, }} mounts := []corev1.VolumeMount{{ Name: "registry-credentials", MountPath: "/root/.docker/config.json", SubPath: "config.json", ReadOnly: true, }} return volumes, mounts } err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 10*time.Minute, false, func(ctx context.Context) (bool, error) { j, err := b.client.BatchV1().Jobs(buildahNamespace). Get(ctx, jobName, metav1.GetOptions{}) if err != nil { return false, err } if j.Status.Succeeded >= 1 { return true, nil } if j.Status.Failed >= 1 { buildErr = fmt.Errorf("buildah: build failed for %s", imageTag) if logs := b.tailBuildLogs(ctx, jobName); logs != "" { buildErr = fmt.Errorf("%w\nbuild logs:\n%s", buildErr, logs) } return false, buildErr } return false, nil }, ) err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 10*time.Minute, false, func(ctx context.Context) (bool, error) { j, err := b.client.BatchV1().Jobs(buildahNamespace). Get(ctx, jobName, metav1.GetOptions{}) if err != nil { return false, err } if j.Status.Succeeded >= 1 { return true, nil } if j.Status.Failed >= 1 { buildErr = fmt.Errorf("buildah: build failed for %s", imageTag) if logs := b.tailBuildLogs(ctx, jobName); logs != "" { buildErr = fmt.Errorf("%w\nbuild logs:\n%s", buildErr, logs) } return false, buildErr } return false, nil }, ) err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 10*time.Minute, false, func(ctx context.Context) (bool, error) { j, err := b.client.BatchV1().Jobs(buildahNamespace). Get(ctx, jobName, metav1.GetOptions{}) if err != nil { return false, err } if j.Status.Succeeded >= 1 { return true, nil } if j.Status.Failed >= 1 { buildErr = fmt.Errorf("buildah: build failed for %s", imageTag) if logs := b.tailBuildLogs(ctx, jobName); logs != "" { buildErr = fmt.Errorf("%w\nbuild logs:\n%s", buildErr, logs) } return false, buildErr } return false, nil }, ) func (b *BuildahBuilder) tailBuildLogs(ctx context.Context, jobName string) string { pods, err := b.client.CoreV1().Pods(buildahNamespace).List(ctx, metav1.ListOptions{LabelSelector: "job-name=" + jobName}) if err != nil || len(pods.Items) == 0 { return "" } tailLines := int64(50) req := b.client.CoreV1().Pods(buildahNamespace). GetLogs(pods.Items[0].Name, &corev1.PodLogOptions{TailLines: &tailLines}) rc, err := req.Stream(ctx) if err != nil { return "" } defer rc.Close() var buf bytes.Buffer io.Copy(&buf, rc) return buf.String() } func (b *BuildahBuilder) tailBuildLogs(ctx context.Context, jobName string) string { pods, err := b.client.CoreV1().Pods(buildahNamespace).List(ctx, metav1.ListOptions{LabelSelector: "job-name=" + jobName}) if err != nil || len(pods.Items) == 0 { return "" } tailLines := int64(50) req := b.client.CoreV1().Pods(buildahNamespace). GetLogs(pods.Items[0].Name, &corev1.PodLogOptions{TailLines: &tailLines}) rc, err := req.Stream(ctx) if err != nil { return "" } defer rc.Close() var buf bytes.Buffer io.Copy(&buf, rc) return buf.String() } func (b *BuildahBuilder) tailBuildLogs(ctx context.Context, jobName string) string { pods, err := b.client.CoreV1().Pods(buildahNamespace).List(ctx, metav1.ListOptions{LabelSelector: "job-name=" + jobName}) if err != nil || len(pods.Items) == 0 { return "" } tailLines := int64(50) req := b.client.CoreV1().Pods(buildahNamespace). GetLogs(pods.Items[0].Name, &corev1.PodLogOptions{TailLines: &tailLines}) rc, err := req.Stream(ctx) if err != nil { return "" } defer rc.Close() var buf bytes.Buffer io.Copy(&buf, rc) return buf.String() } func (b *BuildahBuilder) cleanupBuildJob(ctx context.Context, jobName string) { prop := metav1.DeletePropagationBackground b.client.BatchV1().Jobs(buildahNamespace).Delete(ctx, jobName, metav1.DeleteOptions{PropagationPolicy: &prop}) } func (b *BuildahBuilder) cleanupBuildJob(ctx context.Context, jobName string) { prop := metav1.DeletePropagationBackground b.client.BatchV1().Jobs(buildahNamespace).Delete(ctx, jobName, metav1.DeleteOptions{PropagationPolicy: &prop}) } func (b *BuildahBuilder) cleanupBuildJob(ctx context.Context, jobName string) { prop := metav1.DeletePropagationBackground b.client.BatchV1().Jobs(buildahNamespace).Delete(ctx, jobName, metav1.DeleteOptions{PropagationPolicy: &prop}) } type Builder interface { Build(ctx context.Context, job BuildJob) (imageTag string, err error) } type Builder interface { Build(ctx context.Context, job BuildJob) (imageTag string, err error) } type Builder interface { Build(ctx context.Context, job BuildJob) (imageTag string, err error) } - User pushes source code (GitHub repo) or provides a Dockerfile - The platform needs to turn that source into a container image - The image gets pushed to an internal registry - The orchestrator deploys it into the tenant's isolated namespace All of this happens inside a Kubernetes cluster. There's no external CI service, no GitHub Actions, no cloud build service. The build runs as a Kubernetes Job in the staxa-system namespace. - Must work on a single-node k3s cluster (ARM64) - Must handle private GitHub repos (token injection) - Must support auto-generated Dockerfiles (framework detection feeds into this) - Must clean up after itself (no build job accumulation) - Must surface build logs back to the user on failure - staxad (Go API) receives deploy request - Creates a Kubernetes Job running Buildah - Job clones repo → builds image → pushes to registry - Go API polls Job status until success or failure - On failure: captures last 50 lines of build logs - On completion: cleans up the Job - Tenant isolation in the registry: each tenant's images live under their own path - Multi-service support: a tenant with a frontend and backend gets separate image paths - Version history: rollback is just redeploying a previous version number - No tag collisions: version is an incrementing integer from the database