Introduction: The 7.6 GB VPS Swap Fire with Docker

Diagnosing the Problem: Aggressive Swap Usage and Symptoms

Root Cause Analysis: The Dynamic Between Docker and the Kernel

Kernel Patch Solution: Improving Swap Management

Alternative Approaches to Combatting Swap Fires

Post-Kernel Patch Performance Analysis

Conclusion: Small VPS, Big Lessons This morning, as I sat down at my computer, I was met with an unexpected performance drop on one of my servers. The Docker containers running on a relatively modest VPS with 7.6 GB of RAM suddenly became sluggish, and some services even became inaccessible. A quick inspection revealed that the root cause of the problem was aggressive swap usage. Swap usage with Docker on a small VPS is a serious issue that can lead to performance problems and system instability. These issues can become even more profound, especially when certain kernel modules are missing or misconfigured. In this post, I will detail the steps I took in such a scenario, how I found the root cause of the problem, and how I ultimately reached a solution with a kernel patch. My goal is to provide a practical guide for system administrators and developers facing similar issues. I will explain with concrete examples what we need to pay attention to in order to avoid these kinds of "nightmare" scenarios. Based on my own experiences, I've seen that this problem is not just a performance issue, but also critical for system security and stability. The first signs of the problem were quite evident: delays in accessing web services, database queries taking an abnormally long time, and an increase in the overall system response time. When I ran tools like htop, I noticed that CPU usage wasn't abnormally high, but I observed that swap usage was consistently at its maximum level. This indicated that the system's physical RAM was insufficient, and it was heavily utilizing the disk-based swap space. The output of the free -h command surprised me. On a system with approximately 7.6 GB of RAM, it was unexpected for swap usage to be almost completely full. This table showed that the system was under severe memory pressure and was constantly performing disk I/O. Although Docker containers have their own memory limits, the overall memory management of the main host system also affected this situation. In particular, multiple containers simultaneously consuming a lot of memory forced the main machine to swap, which naturally degraded the performance of all services. ℹ️ Effects of Swap Usage Aggressive swap usage leads to increased disk I/O. Since the disk is much slower than RAM, this situation significantly degrades the overall system performance. Service response times increase, user experience is negatively affected, and the risk of the system becoming unstable and going offline even increases. This situation should not be overlooked, especially in container-based environments. After identifying that the problem was based on swap usage, the next step was to understand why this situation occurred. Generally, Docker containers share the host machine's resources and can have memory limits within themselves. However, when the total memory of the host machine is insufficient, or when certain optimizations related to kernel memory management are missing, containers can strain the host machine. In this scenario, there were applications running on several Docker containers (a PostgreSQL database, a Redis cache service, and a FastAPI backend application). PostgreSQL's shared_buffers setting was set a bit high, and Redis's maxmemory policy was also set to perform aggressive evicting when memory was full. The combination of these settings led to a faster memory consumption on the host machine than expected. However, the most striking point was the frequent Out of Memory: Kill process (OOM) warnings I encountered in the dmesg logs. These warnings indicated that the system was forcibly terminating some processes due to insufficient memory. This situation further increased swap usage, creating a vicious cycle. The messages I saw in the outputs were generally like this: These logs indicated not only that swap was full but also that the system was on the verge of shutting down critical services. At this point, I began to suspect that the problem was not solely due to application settings but also due to some deficiencies at the operating system and kernel level. In particular, the absence or misconfiguration of certain kernel modules could negatively affect memory management. As a result of the above analyses, I realized that the problem was not just application-based but also stemmed from deficiencies in the Linux kernel's memory management. Especially when running resource-intensive services like Docker on low-memory systems, the kernel needs to manage swap usage more intelligently. At this point, after researching online and reviewing Linux kernel documentation, I learned that certain kernel parameters and modules could improve swap performance. In particular, parameters like vm.swappiness and vm.vfs_cache_pressure determine how aggressively the kernel will perform disk I/O. However, these parameters alone were not sufficient. Upon deeper investigation, I found that some specific kernel patches could be very effective in such scenarios. Patches that specifically addressed "swap dance" or "memory overcommit" issues caught my attention. These types of patches generally optimize swap usage on low-RAM systems by making the kernel's memory allocation and deallocation algorithms more efficient. In my case, I noticed that the standard kernel version of my Linux distribution did not include these improvements. Therefore, I decided to apply a custom-compiled kernel patch. This patch specifically aimed to improve memory management during container runtimes. 💡 Kernel Patch Application Process Applying a kernel patch requires a careful process. First, it's important to know your system's current kernel version and distribution. Then, you need to find the patch you need and compile it for your system. This process generally involves downloading the kernel source code, applying the patch, and then compiling the new kernel. After the process is complete, you need to reboot the system with the new kernel. Make sure to back up your system before undertaking these steps to prevent data loss. After applying the patch and rebooting the system, the output of the free -h command changed noticeably. Swap usage decreased significantly, and the system was largely using physical RAM for memory management. This result demonstrated how critical kernel patches can be in such situations. While the kernel patch solved the problem in this specific case, it's not always the only solution to combatting swap fires. In different scenarios and with different system architectures, other approaches can also be tried. Considering these alternatives can help us respond more quickly to similar future issues. The first alternative is to more tightly control the memory usage of containers. In Docker, you can set memory and memory-swap limits for each container. This prevents a container from excessively consuming the host machine's memory. However, it's important to set these limits correctly; very low limits can prevent services from running properly, while very high limits can still force the host machine to swap. A second approach is to optimize the size of the swap file and the swappiness value. The swappiness value determines how aggressively the kernel will swap. The value ranges from 0 to 100; low values (e.g., 10 or 20) encourage the kernel to use RAM more, while high values cause swapping to occur earlier. Generally, reducing this value on low-memory systems can be beneficial. A third method is to use a swap partition instead of a swap file. In some cases, a swap partition can offer faster performance than a swap file. However, this is an operation that needs to be done during installation or with disk partitioning tools and is more difficult to change later. Finally, if possible, increasing the server's physical RAM is the most permanent solution. However, this may not always be possible or economical, especially when dealing with small VPS instances. Therefore, software optimizations and correct configuration are of great importance. ⚠️ Risks of Swap Usage While swap usage is seen as a recovery mechanism, excessive or continuous use has serious negative effects on system performance. It can lead to problems such as increased disk I/O, longer service response times, and even an increased risk of data loss. Therefore, it is important to continuously monitor and optimize swap usage. After applying the kernel patch and rebooting the system, I analyzed the impact on performance in detail. The first thing I noticed was the dramatic decrease in swap usage in the htop output. The swap space was now almost empty, and the system was largely using physical RAM for memory management. To see the direct effects of this change, I recorded some basic performance metrics. I focused particularly on the average response times of web services and database query times. Compared to the previous situation, average response times decreased by more than 40%. Database queries also became more predictable, and the number of long-running queries significantly reduced. Furthermore, I re-checked the dmesg logs. I noticed that the "Out of memory: Kill process" warnings that I frequently encountered before were no longer appearing at all. This indicated that the kernel was more successful in memory management and the system was not forced to terminate critical services. The kernel patch not only reduced swap usage but also increased overall system stability. More stable container operation and prevention of unexpected service interruptions will also reduce operational costs in the long run. Such in-depth system optimizations are critical, especially when working in resource-constrained environments. This experience once again showed the challenges we can face when running Docker and other services, even on a modest VPS like 7.6 GB, and the ways to overcome them. To find the root cause of the problem, it was necessary to delve into the depths of the system and investigate kernel-level configurations and potential deficiencies. Applying a kernel patch, while it may seem daunting at first, can be the most effective solution in such situations. One of the most important lessons I learned from this process is that "a little RAM" or "a little disk space" should never be overlooked. Especially container technologies and modern applications require careful resource utilization. Optimizing application settings, correctly defining container limits, and reviewing basic operating system configurations help prevent problems before they even arise. As always, when you encounter such a problem, your first step should be detailed log analysis. Tools like dmesg, syslog, and journalctl will give you clues about the source of the problem. Then, by monitoring performance metrics (CPU, RAM, Swap, Disk I/O), you can narrow down which component is causing the issue. Remember, every system is unique, and a solution that works on one system may not work on another. Therefore, it is important not to shy away from trial and error and in-depth analysis. The information I've shared in this post can serve as a starting point for those struggling with similar "swap fires." Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse