How we solved cache invalidation in Kubernetes with a headless service

⬅️ Background ## 🛑 The Problem: Fast Caching & Scalable Invalidation ## 🛠️ The Solution: The K8s Headless Service ## Step 1: Headless Services for Pod Discovery ## Step 2 & 3: Broadcast Implementation ## 📈 Production Results and Takeaways At my previous organization, we were building a health platform, and one of the most critical, high-traffic components was our Lab test booking service. One of its core functions is to serve package details like inclusions, prices, and provider information for thousands of user requests daily. The initial architecture was simple: every single request meant a new query to a database. As traffic scaled, our database struggled. Latency spiked, and CPU usage became a constant concern. With mostly static data that rarely changes, the clear solution was to cache it. But the execution is where we got creative. This is the story of how we implemented a zero-cost, in-memory cache in our Node.js microservices and used a Kubernetes feature, “The Headless Service” to handle distributed cache invalidation flawlessly. We decided on in-memory caching using a library like @nestjs/cache-manager to slash DB load and boost response times. However, in a Kubernetes cluster, where our service runs across multiple, dynamic pods, this creates a major headache: Cache Inconsistency. If an admin updates a package price in the MySQL database, only the single pod that handled the write sees the change immediately. The other pods are serving users stale data - consistency, gone*.* Furthermore, Kubernetes pod IPs are temporary. They change during restarts, deployments, and scaling events. We couldn't rely on hardcoded lists. Our invalidation system needed a reliable way to discover every running pod at the exact moment of a data update. We needed a broadcasting mechanism. Our strategy was a hybrid approach: local caching for performance, and a targeted internal HTTP broadcast for invalidation. This is the key to the whole solution. A standard Kubernetes service acts as a load balancer with a single Virtual IP. But a Headless Service is different. By setting clusterIP: None in the Service manifest (.spec.clusterIP), Kubernetes skips the load balancer and, crucially, returns a list of individual DNS A records for every active pod’s IP address. Here is the simple(yet powerful) Headless Service manifest we deployed: Now, when we resolve booking-service-headless.default.svc.cluster.local, we get an array of all current, healthy pod IPs. For visualization, here's a diagram of Headless Services in action: In the next step, we added kubectl into our Docker image for API queries. This gives the pod the power to query the Kubernetes API directly. To implement this safely, we followed the Principle of Least Privilege. We created a dedicated ServiceAccount for our booking service and bound it to a specific Role using RBAC (Role-Based Access Control). This Role was strictly limited: it only allowed the list and get operations on the endpoints resource within its own namespace. This ensures that even if a pod is compromised, an attacker cannot delete resources or view sensitive secrets. Second, we created a simple, internal /invalidate-cache API endpoint on our booking service. When we hit this endpoint, it clears a specific cache key (passed in the request body) using the in-memory cache’s del() method. When an admin update happens, and the data is written to MySQL, the pod that handles the write executes a bash script using kubectl and curl: It uses kubectl to get the list of IPs from the Headless Service endpoints, then sends a direct HTTP POST request to the internal /invalidate-cache endpoint on each pod. This ensures the system is: Dynamic: The IP list is fresh at invalidation time, handling upscaling, downscaling and restarts seamlessly. Targeted: We clear only the stale package key, maximizing cache retention. Cost-Effective: Zero new service bills. Reliability: Add curl retries and logging for failed broadcasts. For those looking to keep their Docker images clean, you can avoid installing kubectl and curl entirely by using the official @kubernetes/client-node library. By using the library, your Node.js process can query the Kubernetes API directly from within the code. This makes the logic more testable, allows for better error handling, and removes the overhead of spawning shell processes. It turns your infrastructure logic into standard application code. Post-rollout, our MySQL database load has consistently remained low, and our cache hit rate has remained above 95%+. We have had zero reported incidents of users seeing stale package data. The Headless Service trick proved robust, even during high-traffic deployments. This solution proves that sometimes, the most elegant and cheapest solution isn't an expensive managed service, but a creative application of your existing tools*.* It’s about being smart with your infrastructure, not just throwing money at the problem. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse COMMAND_BLOCK: apiVersion: v1 kind: Service metadata: name: booking-service-headless spec: clusterIP: None # THE TRICK: Makes it Headless selector: app: booking-service ports: - protocol: TCP port: 80 targetPort: 3000 Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: apiVersion: v1 kind: Service metadata: name: booking-service-headless spec: clusterIP: None # THE TRICK: Makes it Headless selector: app: booking-service ports: - protocol: TCP port: 80 targetPort: 3000 COMMAND_BLOCK: apiVersion: v1 kind: Service metadata: name: booking-service-headless spec: clusterIP: None # THE TRICK: Makes it Headless selector: app: booking-service ports: - protocol: TCP port: 80 targetPort: 3000 COMMAND_BLOCK: # Executed by the Node.js process on update pod_ips=$(kubectl get endpoints listing-headless-service -o jsonpath='{.subsets[0].addresses[*].ip}') for ip in $pod_ips; do curl -X POST http://$ip:3000/v2/invalidate\ -H "Content-Type: application/json" \ -d '{"key": "package-123"}' # Clears the specific key done Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Executed by the Node.js process on update pod_ips=$(kubectl get endpoints listing-headless-service -o jsonpath='{.subsets[0].addresses[*].ip}') for ip in $pod_ips; do curl -X POST http://$ip:3000/v2/invalidate\ -H "Content-Type: application/json" \ -d '{"key": "package-123"}' # Clears the specific key done COMMAND_BLOCK: # Executed by the Node.js process on update pod_ips=$(kubectl get endpoints listing-headless-service -o jsonpath='{.subsets[0].addresses[*].ip}') for ip in $pod_ips; do curl -X POST http://$ip:3000/v2/invalidate\ -H "Content-Type: application/json" \ -d '{"key": "package-123"}' # Clears the specific key done - Dynamic: The IP list is fresh at invalidation time, handling upscaling, downscaling and restarts seamlessly. - Targeted: We clear only the stale package key, maximizing cache retention. - Cost-Effective: Zero new service bills. - Reliability: Add curl retries and logging for failed broadcasts.