Dockerfile Internals and the Image Build Pipeline

From Dockerfile to Build Graph ## Practical Impact: The .dockerignore Advantage ## Layer Creation Is Content, Not Commands ## Cache Key Composition ## Why BuildKit Changed Everything ## BuildKit vs Classic: A Performance Comparison ## Advanced BuildKit Features ## Multi-Stage Builds as a Security Boundary ## Security Impact Analysis ## Debugging Builds Means Debugging Inputs ## Diagnostic Toolkit ## Production Patterns ## 1. Deterministic Builds ## 2. Build-Time Optimization ## 3. Size Optimization ## The OCI Artifact: What Actually Gets Built ## Summary When engineers say "Docker builds an image," they usually mean a single command. In reality, docker build triggers a deterministic pipeline that transforms a text file into an OCI-compliant artifact, composed of immutable, content-addressed layers. Understanding this pipeline explains why cache behaves the way it does, why instruction order matters, and why small Dockerfile changes can dramatically impact build time and image size. The build process starts long before any filesystem changes occur. Docker first parses the Dockerfile into an internal instruction graph. This phase validates syntax, resolves build stages, and prepares the build context after applying .dockerignore. No layers are created here. The output is a dependency-aware plan for how the image could be built. Only after this plan is constructed does execution begin. Key files to exclude: Each filesystem-changing instruction such as RUN, COPY, or ADD produces a new layer. These layers are immutable and identified by a cryptographic hash derived from their content and their parent layer. This is why Docker caching is reliable. If the inputs are identical, the resulting layer hash is identical. The build system does not care why a command ran, only what it produced. Example Cache Behavior: This design is what allows Docker to reuse layers across images, hosts, and even registries. The classic Docker builder executed instructions sequentially, treating each step as an isolated operation. BuildKit replaces this with a graph-based execution model. With BuildKit, independent steps can execute in parallel, cache keys are more precise, and sensitive data such as credentials can be mounted at build time without ever becoming part of an image layer. 1. Build Secrets (Never in Image Layers) 2. Cache Mounts (Persistent Between Builds) This is not an optimization. It is a fundamental shift in how image builds are modeled. Multi-stage builds are often described as a size optimization. More importantly, they create a clean separation between build-time and runtime concerns. Compilers, package managers, and secrets exist only in intermediate stages. The final image contains exactly what is required to run the application, and nothing else. This reduces attack surface, simplifies vulnerability scanning, and makes image provenance easier to reason about. Most Docker build issues are not runtime problems. They are cache invalidation problems. Unexpected rebuilds almost always trace back to: 3. Context Troubleshooting Tools like docker build --progress=plain, docker history, and layer inspection utilities expose these relationships directly, turning "Docker magic" back into observable behavior. At the end of the pipeline, Docker produces: The Docker build pipeline transforms human-readable instructions into a secure, efficient, distributable artifact through: Understanding these internals moves teams from "Docker builds" to "engineered artifact pipelines." Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse COMMAND_BLOCK: # Without .dockerignore: Sending build context to Docker daemon 1.2GB # Slow transfer # With proper .dockerignore: Sending build context to Docker daemon 12.3kB # Fast transfer Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Without .dockerignore: Sending build context to Docker daemon 1.2GB # Slow transfer # With proper .dockerignore: Sending build context to Docker daemon 12.3kB # Fast transfer COMMAND_BLOCK: # Without .dockerignore: Sending build context to Docker daemon 1.2GB # Slow transfer # With proper .dockerignore: Sending build context to Docker daemon 12.3kB # Fast transfer COMMAND_BLOCK: node_modules/ .git/ *.log .env dist/ # For multi-stage builds Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: node_modules/ .git/ *.log .env dist/ # For multi-stage builds COMMAND_BLOCK: node_modules/ .git/ *.log .env dist/ # For multi-stage builds CODE_BLOCK: Layer Hash = SHA256( Parent Layer Hash + Instruction Content + File Content (for COPY/ADD) + Build Arguments at this point ) Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Layer Hash = SHA256( Parent Layer Hash + Instruction Content + File Content (for COPY/ADD) + Build Arguments at this point ) CODE_BLOCK: Layer Hash = SHA256( Parent Layer Hash + Instruction Content + File Content (for COPY/ADD) + Build Arguments at this point ) COMMAND_BLOCK: # Layer 1: Always cached (base image) FROM node:18-alpine # Layer 2: Cached unless WORKDIR changes WORKDIR /app # Layer 3: Cache breaks if package.json changes COPY package*.json ./ # Layer 4: Cache breaks if Layer 3 changes RUN npm ci # Layer 5: Cache breaks if ANY file changes COPY . . # Layer 6: Always cached (metadata) CMD ["npm", "start"] Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Layer 1: Always cached (base image) FROM node:18-alpine # Layer 2: Cached unless WORKDIR changes WORKDIR /app # Layer 3: Cache breaks if package.json changes COPY package*.json ./ # Layer 4: Cache breaks if Layer 3 changes RUN npm ci # Layer 5: Cache breaks if ANY file changes COPY . . # Layer 6: Always cached (metadata) CMD ["npm", "start"] COMMAND_BLOCK: # Layer 1: Always cached (base image) FROM node:18-alpine # Layer 2: Cached unless WORKDIR changes WORKDIR /app # Layer 3: Cache breaks if package.json changes COPY package*.json ./ # Layer 4: Cache breaks if Layer 3 changes RUN npm ci # Layer 5: Cache breaks if ANY file changes COPY . . # Layer 6: Always cached (metadata) CMD ["npm", "start"] COMMAND_BLOCK: # Classic Builder (sequential) Step 1/8 : FROM alpine:latest Step 2/8 : RUN apk add --no-cache python3 Step 3/8 : RUN pip install pandas ... # Each step waits for previous # BuildKit (concurrent possible) [+] Building 8.2s (15/15) FINISHED => CACHED [stage-1 2/6] ... => CACHED [stage-1 3/6] ... # Parallel execution => CACHED [stage-1 4/6] ... Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Classic Builder (sequential) Step 1/8 : FROM alpine:latest Step 2/8 : RUN apk add --no-cache python3 Step 3/8 : RUN pip install pandas ... # Each step waits for previous # BuildKit (concurrent possible) [+] Building 8.2s (15/15) FINISHED => CACHED [stage-1 2/6] ... => CACHED [stage-1 3/6] ... # Parallel execution => CACHED [stage-1 4/6] ... COMMAND_BLOCK: # Classic Builder (sequential) Step 1/8 : FROM alpine:latest Step 2/8 : RUN apk add --no-cache python3 Step 3/8 : RUN pip install pandas ... # Each step waits for previous # BuildKit (concurrent possible) [+] Building 8.2s (15/15) FINISHED => CACHED [stage-1 2/6] ... => CACHED [stage-1 3/6] ... # Parallel execution => CACHED [stage-1 4/6] ... COMMAND_BLOCK: RUN --mount=type=secret,id=npm_token \ echo "//registry.npmjs.org/:_authToken=$(cat /run/secrets/npm_token)" > .npmrc && \ npm ci Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: RUN --mount=type=secret,id=npm_token \ echo "//registry.npmjs.org/:_authToken=$(cat /run/secrets/npm_token)" > .npmrc && \ npm ci COMMAND_BLOCK: RUN --mount=type=secret,id=npm_token \ echo "//registry.npmjs.org/:_authToken=$(cat /run/secrets/npm_token)" > .npmrc && \ npm ci CODE_BLOCK: RUN --mount=type=cache,target=/var/cache/apt \ apt-get update && apt-get install -y packages Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: RUN --mount=type=cache,target=/var/cache/apt \ apt-get update && apt-get install -y packages CODE_BLOCK: RUN --mount=type=cache,target=/var/cache/apt \ apt-get update && apt-get install -y packages COMMAND_BLOCK: # Single-Stage (Vulnerable) FROM node:18 COPY . . RUN npm ci # 600+ dev dependencies RUN npm run build CMD ["node", "dist/app.js"] # Result: 1.2GB image with dev tools, compilers, secrets # Multi-Stage (Secure) FROM node:18 AS builder COPY . . RUN npm ci && npm run build # Dev dependencies here FROM node:18-alpine COPY --from=builder /app/dist ./dist COPY --from=builder /app/package*.json ./ RUN npm ci --only=production # Only 40 prod dependencies # Result: 180MB image, no dev tools, no build secrets Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Single-Stage (Vulnerable) FROM node:18 COPY . . RUN npm ci # 600+ dev dependencies RUN npm run build CMD ["node", "dist/app.js"] # Result: 1.2GB image with dev tools, compilers, secrets # Multi-Stage (Secure) FROM node:18 AS builder COPY . . RUN npm ci && npm run build # Dev dependencies here FROM node:18-alpine COPY --from=builder /app/dist ./dist COPY --from=builder /app/package*.json ./ RUN npm ci --only=production # Only 40 prod dependencies # Result: 180MB image, no dev tools, no build secrets COMMAND_BLOCK: # Single-Stage (Vulnerable) FROM node:18 COPY . . RUN npm ci # 600+ dev dependencies RUN npm run build CMD ["node", "dist/app.js"] # Result: 1.2GB image with dev tools, compilers, secrets # Multi-Stage (Secure) FROM node:18 AS builder COPY . . RUN npm ci && npm run build # Dev dependencies here FROM node:18-alpine COPY --from=builder /app/dist ./dist COPY --from=builder /app/package*.json ./ RUN npm ci --only=production # Only 40 prod dependencies # Result: 180MB image, no dev tools, no build secrets COMMAND_BLOCK: docker history myimage --no-trunc --format "{{.CreatedBy}}" dive myimage # Interactive layer explorer Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: docker history myimage --no-trunc --format "{{.CreatedBy}}" dive myimage # Interactive layer explorer COMMAND_BLOCK: docker history myimage --no-trunc --format "{{.CreatedBy}}" dive myimage # Interactive layer explorer COMMAND_BLOCK: # See why cache invalidated docker build --progress=plain . # Check specific layer docker inspect myimage --format='{{.RootFS.Layers}}' Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # See why cache invalidated docker build --progress=plain . # Check specific layer docker inspect myimage --format='{{.RootFS.Layers}}' COMMAND_BLOCK: # See why cache invalidated docker build --progress=plain . # Check specific layer docker inspect myimage --format='{{.RootFS.Layers}}' COMMAND_BLOCK: # See what's being sent to daemon docker build --no-cache . 2>&1 | grep "sending build context" Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # See what's being sent to daemon docker build --no-cache . 2>&1 | grep "sending build context" COMMAND_BLOCK: # See what's being sent to daemon docker build --no-cache . 2>&1 | grep "sending build context" COMMAND_BLOCK: # Pin everything FROM node:18.20.1-alpine3.19 # Not :latest RUN npm ci --frozen-lockfile # Not npm install Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Pin everything FROM node:18.20.1-alpine3.19 # Not :latest RUN npm ci --frozen-lockfile # Not npm install COMMAND_BLOCK: # Pin everything FROM node:18.20.1-alpine3.19 # Not :latest RUN npm ci --frozen-lockfile # Not npm install COMMAND_BLOCK: # Order matters: Stable → Changing COPY package*.json ./ # Infrequent changes RUN npm ci # Expensive operation COPY . . # Frequent changes Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Order matters: Stable → Changing COPY package*.json ./ # Infrequent changes RUN npm ci # Expensive operation COPY . . # Frequent changes COMMAND_BLOCK: # Order matters: Stable → Changing COPY package*.json ./ # Infrequent changes RUN npm ci # Expensive operation COPY . . # Frequent changes COMMAND_BLOCK: # Clean as you go RUN apt-get update && \ apt-get install -y build-essential && \ # Build something && \ apt-get remove -y build-essential && \ apt-get autoremove -y && \ rm -rf /var/lib/apt/lists/* Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Clean as you go RUN apt-get update && \ apt-get install -y build-essential && \ # Build something && \ apt-get remove -y build-essential && \ apt-get autoremove -y && \ rm -rf /var/lib/apt/lists/* COMMAND_BLOCK: # Clean as you go RUN apt-get update && \ apt-get install -y build-essential && \ # Build something && \ apt-get remove -y build-essential && \ apt-get autoremove -y && \ rm -rf /var/lib/apt/lists/* CODE_BLOCK: { "schemaVersion": 2, "layers": [ { "digest": "sha256:abc123...", // Content hash "size": 1234567 } ], "config": { "digest": "sha256:def456...", "Cmd": ["npm", "start"] } } Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: { "schemaVersion": 2, "layers": [ { "digest": "sha256:abc123...", // Content hash "size": 1234567 } ], "config": { "digest": "sha256:def456...", "Cmd": ["npm", "start"] } } CODE_BLOCK: { "schemaVersion": 2, "layers": [ { "digest": "sha256:abc123...", // Content hash "size": 1234567 } ], "config": { "digest": "sha256:def456...", "Cmd": ["npm", "start"] } } - Changing inputs in early layers - Overly broad COPY instructions - Uncontrolled build arguments - Image Manifest - Metadata and layer references - Image Config - Environment, entrypoint, working directory - Layer Tarballs - Compressed filesystem diffs - Index (multi-arch) - Platform-specific manifests - Graph-based planning - Not linear execution - Content-addressable storage - Deterministic layer creation - Stage isolation - Build/runtime separation - Observable behavior - Every layer is inspectable