rsyslog[8421]: cannot create '/var/log/syslog.1': No space left on device
systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
rsyslog[8421]: cannot create '/var/log/syslog.1': No space left on device
systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
rsyslog[8421]: cannot create '/var/log/syslog.1': No space left on device
systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 100G 45G 50G 48% /
/dev/sda2 20G 8.0G 11G 42% /var/log
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 100G 45G 50G 48% /
/dev/sda2 20G 8.0G 11G 42% /var/log
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 100G 45G 50G 48% /
/dev/sda2 20G 8.0G 11G 42% /var/log
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 6553598 2 100% /
/dev/sda2 1310720 1310718 2 100% /var/log
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 6553598 2 100% /
/dev/sda2 1310720 1310718 2 100% /var/log
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 6553598 2 100% /
/dev/sda2 1310720 1310718 2 100% /var/log
$ find /var/log -type f | wc -l
1310715
$ find /var/log -type f | wc -l
1310715
$ find /var/log -type f | wc -l
1310715
$ ls /var/log/ | head
session_000001.log
session_000002.log
session_000003.log
session_000004.log
session_000005.log
...
$ ls /var/log/ | head
session_000001.log
session_000002.log
session_000003.log
session_000004.log
session_000005.log
...
$ ls /var/log/ | head
session_000001.log
session_000002.log
session_000003.log
session_000004.log
session_000005.log
...
$ find /var/log -name 'session_*' -printf '%s\n' | sort -u
0
$ find /var/log -name 'session_*' -printf '%s\n' | sort -u
0
$ find /var/log -name 'session_*' -printf '%s\n' | sort -u
0
$ find /var/log -type f -name 'session_*' -delete
$ find /var/log -type f -name 'session_*' -delete
$ find /var/log -type f -name 'session_*' -delete
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 2883 6550717 1% /
/dev/sda2 1310720 1003 1309717 1% /var/log
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 2883 6550717 1% /
/dev/sda2 1310720 1003 1309717 1% /var/log
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 2883 6550717 1% /
/dev/sda2 1310720 1003 1309717 1% /var/log
$ systemctl restart rsyslog
$ systemctl status rsyslog Active: active (running)
$ systemctl restart rsyslog
$ systemctl status rsyslog Active: active (running)
$ systemctl restart rsyslog
$ systemctl status rsyslog Active: active (running) - Disk space (what df -h shows) - how many bytes are used
- Inodes - how many files can exist - Monitor inode usage, not just disk space. Most monitoring setups check df -h but forget df -i. Add an alert at 85% inode usage.
- Set up logrotate for any directory that accumulates log files. The default logrotate config handles most system logs but custom paths need their own config.
- Code review any script that creates files in production. The script that caused this was "just a debug helper" that was never removed.
- Use find ... -delete for cleanup, not rm with glob patterns. Glob expansion will hit the ARG_MAX limit with millions of files. - Do you check the actual error message carefully? ("No space left" has two possible causes)
- Do you form a hypothesis before running commands? (Running df -h, df -i, find, each answering a specific question)
- Can you explain the underlying concept? (inodes as a separate resource)
- Do you think about prevention, not just the immediate fix?
