MINI PROJECT: Headless Service with StatefulSet (MySQL-style behavior)

🎯 Goal ## 🧠 Mental model (keep this in mind) ## 🧩 Project structure ## 1️⃣ Headless Service (NO Cluster IP) ## 2️⃣ StatefulSet (stable pod identity) ## 3️⃣ DNS test pod (to observe behavior) ## 4️⃣ Apply everything (ORDER MATTERS) ## 5️⃣ 🔥 THE MOST IMPORTANT PART — DNS BEHAVIOR ### Enter the test pod ### 🔍 DNS lookup of the HEADLESS service ### 🔥 DNS lookup of INDIVIDUAL PODS (THIS IS THE KEY) ## 6️⃣ Why databases NEED this ## 7️⃣ Visual intuition (what’s happening) ## 8️⃣ One-line interview answer (remember this) ## 🔥 SAME APP, TWO SERVICES ## 🟦 CASE 1 — ClusterIP Service (DEFAULT BEHAVIOR) ## YAML (normal service) ### What Kubernetes does ### DNS behavior ## 🔁 What happens to traffic ## ❌ Why this breaks databases ## 🟨 CASE 2 — Headless Service (NO CLUSTER IP) ## YAML (headless service) ## 🔍 DNS behavior (this is the key difference) ## 🧠 Still not enough alone (important) ## 🟩 ENTER STATEFULSET (THIS IS THE MISSING PIECE) ## 🔥 THIS IS THE MOMENT IT CLICKS ### Write traffic ### Read traffic ## 📊 SIDE-BY-SIDE SUMMARY (MEMORIZE THIS) ## 🧪 Why we used dns-test pod ## 🎯 ONE-LINE INTERVIEW ANSWER (IMPORTANT) By the end, you will clearly see: 📄 mysql-headless.yaml 📄 mysql-statefulset.yaml 📌 What StatefulSet guarantees: ✅ You will see MULTIPLE IPs (one per pod): 👉 This is DNS round-robin. ✅ Each one resolves to a specific pod IP. This is what ClusterIP can NEVER do. 💡 This is impossible with Deployment + ClusterIP. “A headless service removes the virtual IP and exposes pod identities via DNS. Combined with StatefulSets, it enables stable per-pod DNS, which is required for databases and leader-follower architectures.” That IP is NOT a pod.

It’s a virtual service IP. Kubernetes decides where each request goes. ClusterIP is stateless-friendly, stateful-hostile. Now we change only one line. ⚠️ No virtual IP
⚠️ DNS returns pod IPs directly This is DNS round-robin, not kube-proxy load balancing. If you stop here and use: Your app may still hit different pods, because DNS answers rotate. So headless alone is not the full solution. StatefulSet guarantees stable pod names: And Kubernetes automatically creates DNS records like: No guessing. No randomness. You cannot “see DNS” from your laptop. So we create a pod just to ask: That’s how SREs debug real prod issues. “ClusterIP services hide pod identity and load balance traffic, which is unsuitable for databases. Headless services expose pod IPs via DNS, and when combined with StatefulSets, provide stable per-pod DNS required for stateful workloads.” Templates let you quickly answer FAQs or store snippets for re-use. as well , this person and/or CODE_BLOCK:
headless-demo/
├── mysql-headless.yaml
├── mysql-statefulset.yaml
└── dns-test.yaml CODE_BLOCK:
headless-demo/
├── mysql-headless.yaml
├── mysql-statefulset.yaml
└── dns-test.yaml CODE_BLOCK:
headless-demo/
├── mysql-headless.yaml
├── mysql-statefulset.yaml
└── dns-test.yaml COMMAND_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-headless
spec: clusterIP: None # 👈 THIS MAKES IT HEADLESS selector: app: mysql ports: - port: 3306 COMMAND_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-headless
spec: clusterIP: None # 👈 THIS MAKES IT HEADLESS selector: app: mysql ports: - port: 3306 COMMAND_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-headless
spec: clusterIP: None # 👈 THIS MAKES IT HEADLESS selector: app: mysql ports: - port: 3306 COMMAND_BLOCK:
apiVersion: apps/v1
kind: StatefulSet
metadata: name: mysql
spec: serviceName: mysql-headless # 👈 REQUIRED replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: root ports: - containerPort: 3306 COMMAND_BLOCK:
apiVersion: apps/v1
kind: StatefulSet
metadata: name: mysql
spec: serviceName: mysql-headless # 👈 REQUIRED replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: root ports: - containerPort: 3306 COMMAND_BLOCK:
apiVersion: apps/v1
kind: StatefulSet
metadata: name: mysql
spec: serviceName: mysql-headless # 👈 REQUIRED replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: root ports: - containerPort: 3306 CODE_BLOCK:
mysql-0 mysql-1 mysql-2 CODE_BLOCK:
mysql-0 mysql-1 mysql-2 CODE_BLOCK:
mysql-0 mysql-1 mysql-2 CODE_BLOCK:
apiVersion: v1
kind: Pod
metadata: name: dns-test
spec: containers: - name: dns image: busybox:1.28 command: ["sleep", "3600"] CODE_BLOCK:
apiVersion: v1
kind: Pod
metadata: name: dns-test
spec: containers: - name: dns image: busybox:1.28 command: ["sleep", "3600"] CODE_BLOCK:
apiVersion: v1
kind: Pod
metadata: name: dns-test
spec: containers: - name: dns image: busybox:1.28 command: ["sleep", "3600"] COMMAND_BLOCK:
kubectl apply -f mysql-headless.yaml
kubectl apply -f mysql-statefulset.yaml
kubectl apply -f dns-test.yaml COMMAND_BLOCK:
kubectl apply -f mysql-headless.yaml
kubectl apply -f mysql-statefulset.yaml
kubectl apply -f dns-test.yaml COMMAND_BLOCK:
kubectl apply -f mysql-headless.yaml
kubectl apply -f mysql-statefulset.yaml
kubectl apply -f dns-test.yaml COMMAND_BLOCK:
kubectl get pods COMMAND_BLOCK:
kubectl get pods COMMAND_BLOCK:
kubectl get pods CODE_BLOCK:
mysql-0 Running
mysql-1 Running
mysql-2 Running
dns-test Running CODE_BLOCK:
mysql-0 Running
mysql-1 Running
mysql-2 Running
dns-test Running CODE_BLOCK:
mysql-0 Running
mysql-1 Running
mysql-2 Running
dns-test Running COMMAND_BLOCK:
kubectl exec -it dns-test -- sh COMMAND_BLOCK:
kubectl exec -it dns-test -- sh COMMAND_BLOCK:
kubectl exec -it dns-test -- sh CODE_BLOCK:
nslookup mysql-headless CODE_BLOCK:
nslookup mysql-headless CODE_BLOCK:
nslookup mysql-headless CODE_BLOCK:
Address: 10.244.0.12
Address: 10.244.0.13
Address: 10.244.0.14 CODE_BLOCK:
Address: 10.244.0.12
Address: 10.244.0.13
Address: 10.244.0.14 CODE_BLOCK:
Address: 10.244.0.12
Address: 10.244.0.13
Address: 10.244.0.14 CODE_BLOCK:
nslookup mysql-0.mysql-headless
nslookup mysql-1.mysql-headless
nslookup mysql-2.mysql-headless CODE_BLOCK:
nslookup mysql-0.mysql-headless
nslookup mysql-1.mysql-headless
nslookup mysql-2.mysql-headless CODE_BLOCK:
nslookup mysql-0.mysql-headless
nslookup mysql-1.mysql-headless
nslookup mysql-2.mysql-headless CODE_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-clusterip
spec: selector: app: mysql ports: - port: 3306 CODE_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-clusterip
spec: selector: app: mysql ports: - port: 3306 CODE_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-clusterip
spec: selector: app: mysql ports: - port: 3306 CODE_BLOCK:
nslookup mysql-clusterip CODE_BLOCK:
nslookup mysql-clusterip CODE_BLOCK:
nslookup mysql-clusterip CODE_BLOCK:
Name: mysql-clusterip
Address: 10.96.120.15 <-- ONE IP CODE_BLOCK:
Name: mysql-clusterip
Address: 10.96.120.15 <-- ONE IP CODE_BLOCK:
Name: mysql-clusterip
Address: 10.96.120.15 <-- ONE IP COMMAND_BLOCK:
todo-app | v
mysql-clusterip (10.96.120.15) | +--> mysql-0 +--> mysql-1 +--> mysql-2 COMMAND_BLOCK:
todo-app | v
mysql-clusterip (10.96.120.15) | +--> mysql-0 +--> mysql-1 +--> mysql-2 COMMAND_BLOCK:
todo-app | v
mysql-clusterip (10.96.120.15) | +--> mysql-0 +--> mysql-1 +--> mysql-2 COMMAND_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-headless
spec: clusterIP: None # 👈 THIS IS EVERYTHING selector: app: mysql ports: - port: 3306 COMMAND_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-headless
spec: clusterIP: None # 👈 THIS IS EVERYTHING selector: app: mysql ports: - port: 3306 COMMAND_BLOCK:
apiVersion: v1
kind: Service
metadata: name: mysql-headless
spec: clusterIP: None # 👈 THIS IS EVERYTHING selector: app: mysql ports: - port: 3306 CODE_BLOCK:
nslookup mysql-headless CODE_BLOCK:
nslookup mysql-headless CODE_BLOCK:
nslookup mysql-headless CODE_BLOCK:
Address: 10.244.0.10 (mysql-0)
Address: 10.244.0.11 (mysql-1)
Address: 10.244.0.12 (mysql-2) CODE_BLOCK:
Address: 10.244.0.10 (mysql-0)
Address: 10.244.0.11 (mysql-1)
Address: 10.244.0.12 (mysql-2) CODE_BLOCK:
Address: 10.244.0.10 (mysql-0)
Address: 10.244.0.11 (mysql-1)
Address: 10.244.0.12 (mysql-2) CODE_BLOCK:
mysql-headless CODE_BLOCK:
mysql-headless CODE_BLOCK:
mysql-headless CODE_BLOCK:
mysql-0
mysql-1
mysql-2 CODE_BLOCK:
mysql-0
mysql-1
mysql-2 CODE_BLOCK:
mysql-0
mysql-1
mysql-2 CODE_BLOCK:
mysql-0.mysql-headless
mysql-1.mysql-headless
mysql-2.mysql-headless CODE_BLOCK:
mysql-0.mysql-headless
mysql-1.mysql-headless
mysql-2.mysql-headless CODE_BLOCK:
mysql-0.mysql-headless
mysql-1.mysql-headless
mysql-2.mysql-headless CODE_BLOCK:
mysql-0.mysql-headless CODE_BLOCK:
mysql-0.mysql-headless CODE_BLOCK:
mysql-0.mysql-headless CODE_BLOCK:
mysql-1.mysql-headless
mysql-2.mysql-headless CODE_BLOCK:
mysql-1.mysql-headless
mysql-2.mysql-headless CODE_BLOCK:
mysql-1.mysql-headless
mysql-2.mysql-headless CODE_BLOCK:
nslookup <service-name> CODE_BLOCK:
nslookup <service-name> CODE_BLOCK:
nslookup <service-name> - Why ClusterIP hides pod identity
- Why Headless Service exposes pod identity
- How StatefulSet + Headless Service gives stable DNS per pod
- Why databases need this - No virtual IP
- DNS will return pod IPs - Pods named: - Names NEVER change
- Identity is stable - Writes → mysql-0.mysql-headless
- Reads → replicas
- Replication → stable target - Creates ONE virtual IP
- Hides all pod IPs
- kube-proxy load balances traffic - Signup request → mysql-2 (write happens there)
- Login request → mysql-0 (no data)
- ❌ login fails - write and read hit different pods
- pod identity is hidden
- you cannot say “always go to mysql-0”