Tools

Tools: Latest: The Six Things That Broke During My kubeadm Setup on Hetzner — and How I Fixed Them

2026-05-30 0 views admin
Tools: Latest: The Six Things That Broke During My kubeadm Setup on Hetzner — and How I Fixed Them

The Setup

Break 1 — The Node Types I Wanted Didn't Exist

What happened

The fix

Break 2 — conntrack Was Missing on Both Nodes

What happened

The fix

Break 3 — The Private Network Interface Wasn't eth1

What happened

The fix

Break 4 — The fluent-bit Helm Chart Was Deprecated

What happened

The fix

Break 5 — Falcosidekick Went Into CrashLoopBackOff

What happened

The fix

Break 6 — Loki Refused to Start on a Two-Node Cluster

What happened

The fix

Full Working Install Order

Summary

What's Next I set up a kubeadm cluster on Hetzner Cloud last week. It broke in 6 different ways before it worked. Here's every error, every fix, and the exact commands that solved each one. TL;DR: conntrack not installed, private NIC named enp7s0 not eth1, Falcosidekick nil pointer crash on missing secret, fluent-bit chart deprecated (use Promtail), Loki distributed defaults breaking on a two-node cluster (use SingleBinary + emptyDir), cpx21/cx32 unavailable in nbg1 (used cpx32/cpx22). All fixed. Commands below. Two-node kubeadm cluster on Hetzner Cloud (nbg1 region): I planned around cpx21 (control plane) and cx32 (worker). When I went to create them in nbg1: Not deprecated. Not removed. Just not available in that datacentre at that moment. Went one tier up: cpx32 and cpx22. Slightly more expensive but available immediately. 💡 Lesson: Hetzner inventory varies by location and changes without notice. Always run hcloud server-type list filtered by your target region before committing to a server type in your Terraform or scripts. First kubeadm init attempt on the control plane: conntrack handles network connection tracking and is required for kube-proxy. Not installed by default on Hetzner's Ubuntu 22.04 images. Not mentioned clearly in the official kubeadm docs. Add this to your node provisioning script before you ever run kubeadm: 💡 Lesson: conntrack is missing from Hetzner's Ubuntu default image and the kubeadm docs don't mention it clearly. Add it to every node bootstrap script before running anything else. Every tutorial, Stack Overflow answer, and blog post assumes Hetzner's private NIC is named eth1. On these nodes it wasn't: The private NIC was enp7s0. This caused two downstream problems: Find your actual interface name first: For kubeadm init, explicitly set the advertise address and node IP: For Flannel, patch the manifest to specify the interface: For the worker node, set the node IP before joining: 💡 Lesson: Never assume eth1. Run ip addr show on your Hetzner nodes before planning your networking. The private NIC name depends on the server type and can change. My original logging plan used fluent-bit. I added the Helm repo and ran the install: The stable/fluent-bit chart was deprecated and the ecosystem had moved to Promtail as the standard Loki log collector. Switch to Promtail — purpose-built for Loki with better Kubernetes metadata enrichment: Promtail runs as a DaemonSet, picks up pod logs automatically via the Kubernetes API, and enriches every line with namespace, pod name, container name, and node name. 💡 Lesson: Use Promtail over fluent-bit for Loki pipelines. Tighter integration, actively maintained, and Kubernetes metadata enrichment works out of the box with zero configuration. Falco installed cleanly. Falcosidekick — the component that routes Falco alerts to Slack — did not: The webhook URL wasn't being passed through correctly from Helm values. A nil pointer in config loading caused a crash rather than a clean validation error. Create the Slack webhook URL as a Kubernetes secret: Reference it in Helm values using existingSecret: Pod came up clean. First Slack alert arrived within 30 seconds. 💡 Lesson: Falcosidekick config errors crash rather than validate gracefully. Always put webhook URLs in a Kubernetes secret and reference existingSecret in Helm values — cleaner and avoids the nil pointer crash entirely. This was the most time-consuming of the six. Loki's default Helm chart assumes a distributed deployment with multiple replicas, persistent volumes, a gateway component, and a caching layer: The pending pods were waiting for PVCs that couldn't bind — no storage class configured. The gateway crashed because the backend wasn't ready. Classic dependency deadlock. SingleBinary deployment mode — Loki as a single process, no distributed components, no PVC required: 💡 Lesson: For Loki on small clusters (under 5 nodes, no storage class), deploymentMode: SingleBinary with emptyDir persistence is the correct starting point. The distributed defaults are built for production scale — not a two-node homelab cluster. With the stack running I moved straight into attack simulation. The next post in this series covers Attack 1 — deploying a cryptominer into the cluster and watching Falco catch it in 47 seconds with three correlated alerts, full Loki log correlation, and MITRE ATT&CK evidence. Full config files, patched Flannel manifest, and Helm values are in the repo: github.com/chrisazzo/k8s-soc-foundation Building a DevSecOps portfolio targeting AI Security Architect work in London and Zurich. Follow the series for the full attack simulation, hardening, and CKS build logs. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Code Block

Copy

Error: server type cpx21 is not available in location nbg1 Error: server type cpx21 is not available in location nbg1 Error: server type cpx21 is not available in location nbg1 # Check availability before planning hcloud server-type list | grep cpx # Check availability before planning hcloud server-type list | grep cpx # Check availability before planning hcloud server-type list | grep cpx [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileNotFound]: /usr/sbin/conntrack not found [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileNotFound]: /usr/sbin/conntrack not found [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileNotFound]: /usr/sbin/conntrack not found apt-get install -y conntrack apt-get install -y conntrack apt-get install -y conntrack apt-get update apt-get install -y \ apt-transport-https \ ca-certificates \ curl \ conntrack \ socat \ ipset apt-get update apt-get install -y \ apt-transport-https \ ca-certificates \ curl \ conntrack \ socat \ ipset apt-get update apt-get install -y \ apt-transport-https \ ca-certificates \ curl \ conntrack \ socat \ ipset ip addr show ip addr show ip addr show 1: lo: <LOOPBACK> 2: eth0: <BROADCAST> inet 5.x.x.x/32 ← public interface 3: enp7s0: <BROADCAST> inet 10.0.0.2/24 ← private interface 1: lo: <LOOPBACK> 2: eth0: <BROADCAST> inet 5.x.x.x/32 ← public interface 3: enp7s0: <BROADCAST> inet 10.0.0.2/24 ← private interface 1: lo: <LOOPBACK> 2: eth0: <BROADCAST> inet 5.x.x.x/32 ← public interface 3: enp7s0: <BROADCAST> inet 10.0.0.2/24 ← private interface ip route | grep "10.0.0" | awk '{print $3}' ip route | grep "10.0.0" | awk '{print $3}' ip route | grep "10.0.0" | awk '{print $3}' kubeadm init \ --apiserver-advertise-address=10.0.0.2 \ --pod-network-cidr=10.244.0.0/16 \ --node-ip=10.0.0.2 kubeadm init \ --apiserver-advertise-address=10.0.0.2 \ --pod-network-cidr=10.244.0.0/16 \ --node-ip=10.0.0.2 kubeadm init \ --apiserver-advertise-address=10.0.0.2 \ --pod-network-cidr=10.244.0.0/16 \ --node-ip=10.0.0.2 # In kube-flannel.yml, under kube-flannel container args: args: - --ip-masq - --kube-subnet-mgr - --iface=enp7s0 # Add this line # In kube-flannel.yml, under kube-flannel container args: args: - --ip-masq - --kube-subnet-mgr - --iface=enp7s0 # Add this line # In kube-flannel.yml, under kube-flannel container args: args: - --ip-masq - --kube-subnet-mgr - --iface=enp7s0 # Add this line echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \ >> /etc/default/kubelet systemctl daemon-reload systemctl restart kubelet echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \ >> /etc/default/kubelet systemctl daemon-reload systemctl restart kubelet echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \ >> /etc/default/kubelet systemctl daemon-reload systemctl restart kubelet Error: chart "fluent-bit" not found in stable repository WARNING: This chart is deprecated Error: chart "fluent-bit" not found in stable repository WARNING: This chart is deprecated Error: chart "fluent-bit" not found in stable repository WARNING: This chart is deprecated helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install promtail grafana/promtail \ --namespace monitoring \ --set config.clients[0].url=http://loki:3100/loki/api/v1/push helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install promtail grafana/promtail \ --namespace monitoring \ --set config.clients[0].url=http://loki:3100/loki/api/v1/push helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install promtail grafana/promtail \ --namespace monitoring \ --set config.clients[0].url=http://loki:3100/loki/api/v1/push kubectl get pods -n falco kubectl get pods -n falco kubectl get pods -n falco NAME READY STATUS RESTARTS AGE falco-abcd1 1/1 Running 0 4m falcosidekick-xyz99 0/1 CrashLoopBackOff 6 4m NAME READY STATUS RESTARTS AGE falco-abcd1 1/1 Running 0 4m falcosidekick-xyz99 0/1 CrashLoopBackOff 6 4m NAME READY STATUS RESTARTS AGE falco-abcd1 1/1 Running 0 4m falcosidekick-xyz99 0/1 CrashLoopBackOff 6 4m kubectl logs falcosidekick-xyz99 -n falco kubectl logs falcosidekick-xyz99 -n falco kubectl logs falcosidekick-xyz99 -n falco panic: runtime error: invalid memory address or nil pointer dereference error: failed to load configuration: SLACK_WEBHOOKURL is required when Slack output is enabled panic: runtime error: invalid memory address or nil pointer dereference error: failed to load configuration: SLACK_WEBHOOKURL is required when Slack output is enabled panic: runtime error: invalid memory address or nil pointer dereference error: failed to load configuration: SLACK_WEBHOOKURL is required when Slack output is enabled kubectl create secret generic falcosidekick-secrets \ --from-literal=slackWebhookUrl="https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \ -n falco kubectl create secret generic falcosidekick-secrets \ --from-literal=slackWebhookUrl="https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \ -n falco kubectl create secret generic falcosidekick-secrets \ --from-literal=slackWebhookUrl="https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \ -n falco # falcosidekick-values.yaml config: slack: webhookurl: "" minimumpriority: "notice" existingSecret: "falcosidekick-secrets" # falcosidekick-values.yaml config: slack: webhookurl: "" minimumpriority: "notice" existingSecret: "falcosidekick-secrets" # falcosidekick-values.yaml config: slack: webhookurl: "" minimumpriority: "notice" existingSecret: "falcosidekick-secrets" helm upgrade --install falcosidekick falcosecurity/falcosidekick \ --namespace falco \ -f falcosidekick-values.yaml helm upgrade --install falcosidekick falcosecurity/falcosidekick \ --namespace falco \ -f falcosidekick-values.yaml helm upgrade --install falcosidekick falcosecurity/falcosidekick \ --namespace falco \ -f falcosidekick-values.yaml kubectl get pods -n monitoring kubectl get pods -n monitoring kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE loki-backend-0 0/1 Pending 0 8m loki-read-0 0/1 Pending 0 8m loki-write-0 0/1 Pending 0 8m loki-gateway-xyz 0/1 CrashLoopBackOff 4 8m NAME READY STATUS RESTARTS AGE loki-backend-0 0/1 Pending 0 8m loki-read-0 0/1 Pending 0 8m loki-write-0 0/1 Pending 0 8m loki-gateway-xyz 0/1 CrashLoopBackOff 4 8m NAME READY STATUS RESTARTS AGE loki-backend-0 0/1 Pending 0 8m loki-read-0 0/1 Pending 0 8m loki-write-0 0/1 Pending 0 8m loki-gateway-xyz 0/1 CrashLoopBackOff 4 8m # loki-values.yaml loki: commonConfig: replication_factor: 1 storage: type: filesystem schemaConfig: configs: - from: "2024-01-01" store: tsdb object_store: filesystem schema: v13 index: prefix: loki_index_ period: 24h deploymentMode: SingleBinary singleBinary: replicas: 1 persistence: enabled: false extraVolumes: - name: loki-data emptyDir: {} extraVolumeMounts: - name: loki-data mountPath: /var/loki read: replicas: 0 write: replicas: 0 backend: replicas: 0 gateway: enabled: false chunksCache: enabled: false resultsCache: enabled: false lokiCanary: enabled: false test: enabled: false # loki-values.yaml loki: commonConfig: replication_factor: 1 storage: type: filesystem schemaConfig: configs: - from: "2024-01-01" store: tsdb object_store: filesystem schema: v13 index: prefix: loki_index_ period: 24h deploymentMode: SingleBinary singleBinary: replicas: 1 persistence: enabled: false extraVolumes: - name: loki-data emptyDir: {} extraVolumeMounts: - name: loki-data mountPath: /var/loki read: replicas: 0 write: replicas: 0 backend: replicas: 0 gateway: enabled: false chunksCache: enabled: false resultsCache: enabled: false lokiCanary: enabled: false test: enabled: false # loki-values.yaml loki: commonConfig: replication_factor: 1 storage: type: filesystem schemaConfig: configs: - from: "2024-01-01" store: tsdb object_store: filesystem schema: v13 index: prefix: loki_index_ period: 24h deploymentMode: SingleBinary singleBinary: replicas: 1 persistence: enabled: false extraVolumes: - name: loki-data emptyDir: {} extraVolumeMounts: - name: loki-data mountPath: /var/loki read: replicas: 0 write: replicas: 0 backend: replicas: 0 gateway: enabled: false chunksCache: enabled: false resultsCache: enabled: false lokiCanary: enabled: false test: enabled: false helm upgrade --install loki grafana/loki \ --namespace monitoring \ -f loki-values.yaml helm upgrade --install loki grafana/loki \ --namespace monitoring \ -f loki-values.yaml helm upgrade --install loki grafana/loki \ --namespace monitoring \ -f loki-values.yaml kubectl get pods -n monitoring # NAME READY STATUS RESTARTS AGE # loki-0 1/1 Running 0 45s kubectl get pods -n monitoring # NAME READY STATUS RESTARTS AGE # loki-0 1/1 Running 0 45s kubectl get pods -n monitoring # NAME READY STATUS RESTARTS AGE # loki-0 1/1 Running 0 45s # Step 1 — Node prerequisites (run on BOTH nodes) apt-get update && apt-get install -y \ conntrack socat ipset curl # Step 2 — Container runtime + kubeadm/kubelet/kubectl # (standard Ubuntu kubeadm installation docs) # Step 3 — Find your private NIC name ip addr show # Note the interface name next to your 10.x.x.x address # Step 4 — kubeadm init (control plane only) kubeadm init \ --apiserver-advertise-address=10.0.0.2 \ --pod-network-cidr=10.244.0.0/16 # Step 5 — Flannel with explicit interface # Download manifest, add --iface=enp7s0 to container args, apply kubectl apply -f kube-flannel-enp7s0.yml # Step 6 — Worker node prep (run on worker) echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \ >> /etc/default/kubelet systemctl daemon-reload && systemctl restart kubelet # Step 7 — Worker join (run on worker) kubeadm join 10.0.0.2:6443 \ --token <token> \ --discovery-token-ca-cert-hash sha256:<hash> # Step 8 — Namespaces kubectl create namespace monitoring kubectl create namespace falco # Step 9 — Falco secret first kubectl create secret generic falcosidekick-secrets \ --from-literal=slackWebhookUrl="YOUR_WEBHOOK_URL" \ -n falco # Step 10 — Falco + Falcosidekick helm repo add falcosecurity \ https://falcosecurity.github.io/charts helm install falco falcosecurity/falco \ --namespace falco -f falco-values.yaml helm install falcosidekick falcosecurity/falcosidekick \ --namespace falco -f falcosidekick-values.yaml # Step 11 — Loki SingleBinary helm repo add grafana https://grafana.github.io/helm-charts helm install loki grafana/loki \ --namespace monitoring -f loki-values.yaml # Step 12 — Promtail helm install promtail grafana/promtail \ --namespace monitoring \ --set config.clients[0].url=http://loki:3100/loki/api/v1/push # Step 13 — Grafana helm install grafana grafana/grafana \ --namespace monitoring \ --set adminPassword=changeme # Step 14 — Trivy Operator helm install trivy-operator \ aquasecurity/trivy-operator \ --namespace monitoring \ --set trivy.ignoreUnfixed=true # Step 1 — Node prerequisites (run on BOTH nodes) apt-get update && apt-get install -y \ conntrack socat ipset curl # Step 2 — Container runtime + kubeadm/kubelet/kubectl # (standard Ubuntu kubeadm installation docs) # Step 3 — Find your private NIC name ip addr show # Note the interface name next to your 10.x.x.x address # Step 4 — kubeadm init (control plane only) kubeadm init \ --apiserver-advertise-address=10.0.0.2 \ --pod-network-cidr=10.244.0.0/16 # Step 5 — Flannel with explicit interface # Download manifest, add --iface=enp7s0 to container args, apply kubectl apply -f kube-flannel-enp7s0.yml # Step 6 — Worker node prep (run on worker) echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \ >> /etc/default/kubelet systemctl daemon-reload && systemctl restart kubelet # Step 7 — Worker join (run on worker) kubeadm join 10.0.0.2:6443 \ --token <token> \ --discovery-token-ca-cert-hash sha256:<hash> # Step 8 — Namespaces kubectl create namespace monitoring kubectl create namespace falco # Step 9 — Falco secret first kubectl create secret generic falcosidekick-secrets \ --from-literal=slackWebhookUrl="YOUR_WEBHOOK_URL" \ -n falco # Step 10 — Falco + Falcosidekick helm repo add falcosecurity \ https://falcosecurity.github.io/charts helm install falco falcosecurity/falco \ --namespace falco -f falco-values.yaml helm install falcosidekick falcosecurity/falcosidekick \ --namespace falco -f falcosidekick-values.yaml # Step 11 — Loki SingleBinary helm repo add grafana https://grafana.github.io/helm-charts helm install loki grafana/loki \ --namespace monitoring -f loki-values.yaml # Step 12 — Promtail helm install promtail grafana/promtail \ --namespace monitoring \ --set config.clients[0].url=http://loki:3100/loki/api/v1/push # Step 13 — Grafana helm install grafana grafana/grafana \ --namespace monitoring \ --set adminPassword=changeme # Step 14 — Trivy Operator helm install trivy-operator \ aquasecurity/trivy-operator \ --namespace monitoring \ --set trivy.ignoreUnfixed=true # Step 1 — Node prerequisites (run on BOTH nodes) apt-get update && apt-get install -y \ conntrack socat ipset curl # Step 2 — Container runtime + kubeadm/kubelet/kubectl # (standard Ubuntu kubeadm installation docs) # Step 3 — Find your private NIC name ip addr show # Note the interface name next to your 10.x.x.x address # Step 4 — kubeadm init (control plane only) kubeadm init \ --apiserver-advertise-address=10.0.0.2 \ --pod-network-cidr=10.244.0.0/16 # Step 5 — Flannel with explicit interface # Download manifest, add --iface=enp7s0 to container args, apply kubectl apply -f kube-flannel-enp7s0.yml # Step 6 — Worker node prep (run on worker) echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \ >> /etc/default/kubelet systemctl daemon-reload && systemctl restart kubelet # Step 7 — Worker join (run on worker) kubeadm join 10.0.0.2:6443 \ --token <token> \ --discovery-token-ca-cert-hash sha256:<hash> # Step 8 — Namespaces kubectl create namespace monitoring kubectl create namespace falco # Step 9 — Falco secret first kubectl create secret generic falcosidekick-secrets \ --from-literal=slackWebhookUrl="YOUR_WEBHOOK_URL" \ -n falco # Step 10 — Falco + Falcosidekick helm repo add falcosecurity \ https://falcosecurity.github.io/charts helm install falco falcosecurity/falco \ --namespace falco -f falco-values.yaml helm install falcosidekick falcosecurity/falcosidekick \ --namespace falco -f falcosidekick-values.yaml # Step 11 — Loki SingleBinary helm repo add grafana https://grafana.github.io/helm-charts helm install loki grafana/loki \ --namespace monitoring -f loki-values.yaml # Step 12 — Promtail helm install promtail grafana/promtail \ --namespace monitoring \ --set config.clients[0].url=http://loki:3100/loki/api/v1/push # Step 13 — Grafana helm install grafana grafana/grafana \ --namespace monitoring \ --set adminPassword=changeme # Step 14 — Trivy Operator helm install trivy-operator \ aquasecurity/trivy-operator \ --namespace monitoring \ --set trivy.ignoreUnfixed=true - Control plane: cpx32 — 4 vCPU, 8GB RAM, Ubuntu 22.04 - Worker node: cpx22 — 3 vCPU, 4GB RAM, Ubuntu 22.04 - Private network enabled (Hetzner Cloud Networks) - CNI: Flannel - Goal: foundation for a Kubernetes security detection stack — Falco, Loki, Grafana, Trivy Operator, kube-bench - kubeadm advertised the public IP for the API server — worker joins routed over the public internet - Flannel defaulted to the public interface for pod-to-pod traffic

🏷️ Tags

toolsutilitiessecurity toolslatestthingsbrokeduringkubeadmsetuphetzner

More from Tools

Tools: Deploying Backend on Render? Read This First - Complete Guide

Tools: Deploying Backend on Render? Read This First - Complete Guide

2026-05-30 0
Tools: Breaking: Your CI Pipeline Is Lying to You — 5 Signals You're Ignoring (And How to Fix Them)

Tools: Breaking: Your CI Pipeline Is Lying to You — 5 Signals You're Ignoring (And How to Fix Them)

2026-05-30 0
Tools: I Tested Every Web Scraping Tool Against Lazada — Here's What Actually Works (May 2026) - Complete Guide

Tools: I Tested Every Web Scraping Tool Against Lazada — Here's What Actually Works (May 2026) - Complete Guide

2026-05-30 0
Tools: Complete Guide to My test suite was green. My software was lying to me.

Tools: Complete Guide to My test suite was green. My software was lying to me.

2026-05-30 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views