Introduction: A VPS Experience in the Shadow of OOM Killer

What is OOM Killer and Why Does It Occur?

A Concrete Case: Delayed Shipment Reports in Production ERP and Data Loss After OOM

Analyzing Memory Usage: top, htop, and free

Strategies to Prevent OOM Killer

1. Detecting and Fixing Memory Leaks

2. Setting Service-Based Memory Limits

3. Optimizing or Increasing Swap Space

4. Adding More RAM

Configuring OOM Killer: oom_score_adj and oom_score

Conclusion: System Balance and Continuous Monitoring For some time, I've been having some interesting and at times frustrating experiences with memory management for applications running on my own Virtual Private Server (VPS). Especially during peak periods or unexpected traffic surges, when the system's memory (RAM) consumption reached critical levels, I encountered the OOM Killer (Out-of-Memory Killer). This is a situation we could call the "silent death" of the system, because your applications can suddenly shut down, data loss can occur, or the entire system can become unresponsive. In this post, I will delve into what OOM Killer is, why it occurs, the specific cases I faced on my VPS, and the strategies I implemented to deal with this problem. The purpose of this article is not just to explain how a problem was solved, but also to delve into the depths of operating system memory management mechanisms and offer a perspective rooted in direct field experience, far from a "corporate consultant" tone. This adventure I had with my own VPS taught me invaluable lessons about the complex inner workings of Linux and showed me how delicate the balance of system reliability is. Especially in the first half of 2026, increasing workloads and more complex service deployments have made memory management issues more visible. The Linux kernel is designed to efficiently manage system resources. Memory (RAM) is one of the most critical of these resources. All processes running on a system occupy space in memory. If the total memory demand on the system exceeds the available physical RAM and swap space, the kernel faces an "out of memory" situation. At this point, a mechanism kicks in to prevent the system from completely crashing: the OOM Killer. The OOM Killer attempts to free up memory by selecting and terminating one or more processes currently running on the system. ℹ️ How OOM Killer Works When deciding which process to terminate, OOM Killer typically uses an oom_score. This score is calculated based on various factors such as the amount of memory the process occupies, its runtime, and the time spent in kernel mode. Processes with a higher oom_score are considered more likely targets by the OOM Killer. The ability to adjust this score is one of the methods used to prevent certain critical processes from being terminated. So, how does this "out of memory" situation arise? There can be multiple reasons: In my case, several scenarios converged. One of the backend services of a production ERP system was consuming much more memory than expected, especially when certain reporting queries were run. In addition, a Time Series Database (TSDB) and several small helper services running on the same VPS also created a continuous memory load. This combination began to push the system to its limits. A few months ago, while working on a large manufacturing company's ERP system, we noticed that shipment reports were consistently delayed. Reports typically arrived with a 2-hour delay, which made it difficult to make operational decisions. Initially, we thought the problem was with the reporting query itself. We added indexes to optimize the query, examined the query plan, but didn't get the expected performance. It took a full 3 days to find the root cause of the problem. The gist of the incident was this: when a shipment report for a specific date range was run, the backend service was pulling an enormous amount of data and processing it in memory. The PostgreSQL database also used intensive memory for this query. After about 1.5 hours, both the application service and the database began to push their memory limits. The total memory of the VPS was around 32GB, and memory usage exceeded 95% while this query was running. At this critical moment, OOM Killer intervened. However, OOM Killer first terminated the database process, and then the main process of the backend service. The result: The reporting process was interrupted, the database shut down unexpectedly, and the backend service crashed. Although the system tried to recover a few minutes later, the reporting process remained "incomplete" and data loss occurred. This situation not only caused reporting issues but also operational disruptions and potential financial losses. One of the most important lessons I learned from this case was that OOM Killer is not just a "bad thing," but also a recovery mechanism that prevents the system from completely crashing; however, the cost can be high. ⚠️ Post-OOM Killer Situation Assessment When OOM Killer terminates a process, it usually leaves a relevant entry in log files such as /var/log/syslog or /var/log/messages. These logs contain clues about which process was terminated, how much memory it used, and why OOM Killer made that decision. For example, you might see a log line like this: Regularly checking these logs is critical for understanding the source of OOM issues. To understand and prevent the excessive memory usage that triggers OOM Killer, it's essential to regularly monitor memory usage on the system. There are several basic tools we can use for this: top Command: Shows real-time CPU and memory usage of running processes. The %MEM column indicates the percentage of total memory a process occupies. You can press Shift + M to sort processes by memory usage. htop Command: A more user-friendly and interactive version of the top command. It's more useful with features like colored output, process trees, and mouse support. Memory usage, CPU usage, and other information are also presented graphically. free Command: Summarizes the system's overall memory usage. It shows information such as used memory, free memory, buffers, and cache. The free -h command displays the output in a human-readable format (like MB, GB). These tools became my closest companions during my VPS journey. The first thing I did every morning was to check the system's overall status with htop. Especially if memory usage was consistently in the 80-90% range, it was an alarm signal. The free -h command clearly showed the current total memory status. For example, once, the free -h output was like this: This output shows that 29GB of 31GB memory is used, only 1.2GB is free, and half of the swap space is also full. This situation was a clear indication that OOM Killer could intervene shortly. The low available memory amount is particularly critical. This value indicates how "comfortable" the system is to start new processes or meet the memory demands of existing ones. 💡 Swap Space and Performance Swap space is backup memory on disk used when physical RAM is full. Swap usage usually indicates a performance drop because disk access is much slower than RAM access. Using swap to avoid OOM Killer is a solution, but it reduces the overall speed of the system. The ideal is to keep swap usage to a minimum and provide sufficient RAM. Several different strategies can be followed to prevent OOM Killer from being triggered. These may vary depending on the source of the problem and the system's architecture: This is the most ideal and permanent solution. You can use various tools to detect memory leaks in your applications: In my ERP example, I analyzed query-based memory usage to identify the problem. PostgreSQL's own pg_stat_activity view and query logs provided clues about which queries consumed how much memory. Afterward, I realized that a function processing data on the Vue.js frontend was fetching too much data and holding it in memory. I optimized this function, rewriting it to fetch only necessary data and occupy less memory. With this change, the memory consumption of the reporting query decreased by 60%. Limiting how much memory each service can use prevents one service from overwhelming others. Container orchestration tools like Docker and Kubernetes are very effective in this regard. However, it's also possible to set limits via systemd service units in bare-metal or simple VPS setups. For example, you can set memory limits by adding these lines to a systemd unit file for a service: This configuration helps the system become more cautious when the relevant service starts using more than 1.5GB of memory. If it exceeds 2GB, the OOM Killer is more likely to target this service. This was a method I used to prevent critical services, especially those I developed myself and had direct control over, from harming the production ERP or database. For example, I set a limit like MemoryMax=512M for the service running the backend of my own financial calculator application, so my own application wouldn't affect the overall system. If there are no memory leaks and your existing applications' memory needs are known, but sometimes exceed this need, increasing swap space can be a temporary solution. However, this is not a long-term solution as it causes performance degradation. It's also possible to optimize swap by adjusting the swappiness parameter. The swappiness value ranges from 0 to 100. Lower values (e.g., 10) make the kernel avoid moving memory to swap space as much as possible. Higher values (e.g., 60, the default) encourage the kernel to aggressively move memory to swap space. To change the swappiness value: Lowering the swappiness value to 10 on my VPS allowed the system to retain physical memory longer and reduced swap usage. This decreased the frequency of OOM Killer interventions but did not eliminate them entirely. The simplest and most effective solution, if possible, is to add more RAM to your VPS. However, this increases costs and may not always be feasible. If cost and infrastructure allow, this is often the best and cleanest solution. In my situation, I already had the highest RAM capacity offered by my VPS provider, so this option was not applicable to me. You can adjust the oom_score_adj value to control more precisely which processes OOM Killer terminates. This value is found in the /proc/<PID>/oom_score_adj file for each process. The value can range from -1000 to 1000. For example, for a critical process like the ERP backend service, you can lower this value: This command applies to the currently running process. To make it permanent, you need to configure it in the systemd service unit or startup scripts. Similarly, I increased the oom_score_adj value for one of my helper services, ensuring that if a memory problem occurred, this service would be the first to shut down. oom_score, on the other hand, shows the current oom_score of the process. By reading this, you can see which processes are more likely to be targeted: These tools are invaluable for understanding and intervening in OOM Killer's "decision-making" mechanism. My OOM Killer adventures on my VPS once again showed me how complex and delicately balanced systems are. OOM Killer is not a "debugging" tool, but a "recovery" mechanism, and its activation usually signals the existence of a deeper problem. The most important lessons I learned during this process were: In the world of technology, "perfect" and "error-free" systems are rare. What matters is understanding how these systems work, being prepared for potential issues, and solving problems with a pragmatic approach. OOM Killer is a silent but powerful reminder from systems: use resources carefully and always have a Plan B. Such experiences have been a continuous driving force for me to become a better system architect and operator. As I mentioned in my previous post [related: my VPS migration experience], there is always something new to learn. This OOM adventure also gave me a deep understanding of memory management. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse