Tools: Linux File System Hunting: What Your OS Is Hiding in Plain Sight (2026)

Tools: Linux File System Hunting: What Your OS Is Hiding in Plain Sight (2026)

1. /etc/resolv.conf — The DNS Brain of Your Machine

2. /proc/net/route — Your Routing Table, Unfiltered

3. /etc/passwd and /etc/shadow — The Auth Split That Saved Linux Security

4. /proc/PID/fd/ — Every File Your Process Has Open

5. /etc/hosts and /etc/nsswitch.conf — DNS That Predates DNS

6. /etc/systemd/system/ — Where Services Are Born

7. /var/log/auth.log — Your Machine's Security Diary

8. /dev/null, /dev/zero, /dev/urandom — The Kernel's Built-In Utilities

9. /proc/meminfo — Why "Free Memory" Is a Lie

10. /boot/grub/grub.cfg — The First Thing That Actually Runs

The Pattern Behind All of This

Going Deeper Stop running commands. Start reading the OS like a story. Most engineers learn Linux by memorizing commands. But the real insight comes when you stop asking "what command does X?" and start asking "why does this file exist at all?" This post is a field report from a deep dive into the Linux file system — not a command tutorial, but an investigation. Here's what I found, why it matters, and what it reveals about how Linux actually thinks. When your system needs to resolve google.com, it doesn't magically know where to look. It opens /etc/resolv.conf. What's actually happening here? On modern Ubuntu/Debian systems, the nameserver points to 127.0.0.53 — a loopback address handled by systemd-resolved, not a real external DNS server. This is a stub resolver. Your queries hit systemd-resolved first, which checks its cache, applies policies, and only then forwards upstream. Why this matters for engineers: If you're debugging DNS failures inside a container or a VPN, this file is the first place to look — but remember, editing it directly on systemd-resolved systems often gets overwritten. The real config lives in /etc/systemd/resolved.conf. Insight: Linux separates "who handles DNS" from "where DNS queries go". That layering enables caching, split-horizon DNS, and link-specific resolvers. Many DNS bugs in containerized apps trace back to not understanding this indirection. Everyone uses ip route to see routes. But that command is just reading from a file: The values are in little-endian hexadecimal. Decoding 0101A8C0: reverse the byte pairs → C0.A8.01.01 → 192.168.1.1. That's your default gateway. Why this exists: The kernel exposes routing decisions through /proc as virtual files — no database, no daemon needed. Tools like ip, netstat, and route are essentially pretty-printers for this raw data. Insight: The kernel routing table isn't stored anywhere on disk — it's computed in memory and exposed live through /proc. You're always reading the present state of the kernel's mind. Notice the x in the password field? That x is a pointer — it means "go look in /etc/shadow". The hash prefix $6$ means SHA-512. The rounds= value controls iteration count — the computational cost of cracking. Why the split exists: Originally, /etc/passwd held password hashes directly — but it had to be world-readable so programs could look up usernames. This meant anyone on the system could grab hashes and crack them offline. The fix: move hashes to /etc/shadow, readable only by root. Insight: This is a textbook least-privilege design. Readable metadata in one file, sensitive secrets in another, permissions separating them. Vault, AWS SSM, and Kubernetes Secrets all follow the exact same principle. Pick any running process and inspect it live: File descriptors 0, 1, 2 are stdin, stdout, and stderr — symlinks to the terminal device. Now try it on a running service: You'll see open sockets, config files, log files, and PID files — everything that process is currently touching. Why this matters: This is exactly how lsof and strace work under the hood. In incident response, checking /proc/PID/fd can reveal if a process is holding onto a deleted file (keeping disk space locked) or has unexpected outbound connections open. This file is a relic from the ARPANET era — before DNS existed, machines shared a central HOSTS.TXT file. Today it's still consulted before DNS, governed by /etc/nsswitch.conf: files comes first — meaning /etc/hosts wins over any DNS server. Why engineers should care: Kubernetes, Docker, and many CI environments inject hostnames via this file. Understanding the lookup order explains why a DNS change doesn't take effect but a hosts file entry does — and why service resolution differs inside and outside a container. After= defines boot ordering. Restart=on-failure means systemd auto-restarts on crash. WantedBy=multi-user.target determines which boot stage activates the service. The key insight: Everything in systemd is a declarative dependency graph — not a sequential boot script. The OS calculates start order at runtime from After= and Wants= relationships across all unit files. This is fundamentally different from old SysV init where order was hardcoded line by line. If your machine is internet-facing, you almost certainly have hundreds of these every day — automated credential stuffing from bots scanning the entire IPv4 space. What else lives here: Sudo usage, su attempts, PAM authentication events, and SSH key fingerprints from successful logins. Cross-referencing with /var/log/syslog gives you a full timeline of system behavior around any suspicious event. Insight: Logs are only useful if you know what normal looks like. On a healthy server, SSH access comes from a handful of known IPs. Once you understand the baseline, anomalies are immediately obvious. These aren't real files. They're character devices — kernel interfaces that behave like files. Why /dev/urandom matters more than you think: When your Node.js app calls crypto.randomBytes() or Python calls secrets.token_hex(), they ultimately read from /dev/urandom. It's the root of all cryptographic randomness on Linux — every TLS handshake, every UUID, every session token traces back here. MemFree is nearly always the wrong number to watch. The kernel holds Buffers + Cached memory for disk I/O caching — it shows as "used" but is immediately reclaimable. MemAvailable is the real number. This is why a freshly booted server shows "high memory usage" — the kernel is aggressively using idle RAM for caching, which is the correct behavior. On cloud VMs, CPU flags like vmx (Intel VT-x) or svm (AMD-V) reveal whether the machine supports hardware virtualization — useful for detecting nested virtualization environments. Before the kernel, before init, before systemd — GRUB runs. Its config: The linux line passes parameters directly to the kernel. quiet splash suppresses boot messages. ro mounts root read-only initially (remounted read-write after filesystem checks). root=UUID= tells the kernel which block device is the root filesystem. Why this matters: Kernel parameters like init=/bin/bash (drop to shell), nomodeset (fix GPU issues), or systemd.unit=rescue.target (recovery mode) are configured here. Understanding GRUB is what separates engineers who can recover a broken system from those who reinstall it. After this investigation, one theme is unmistakable: Linux exposes almost everything as a file. This isn't just a design quirk — it's a philosophy. If it's a file, you can read it, watch it with inotify, pipe it through grep, version-control it with git, and audit it with standard Unix tools. No proprietary APIs, no special SDKs, no vendor lock-in. The Linux file system isn't just where your files live. It's the kernel's window into itself. If this sparked your curiosity, explore further: The more you read the filesystem, the more the OS reveals itself — not as a black box, but as a transparent, well-documented machine that's been explaining itself all along. You just have to know where to look. Part of the Web Dev Cohort 2026 — Linux File System Hunting. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Command

Copy

$ cat /etc/resolv.conf cat /etc/resolv.conf cat /etc/resolv.conf nameserver 127.0.0.53 options edns0 trust-ad search home nameserver 127.0.0.53 options edns0 trust-ad search home nameserver 127.0.0.53 options edns0 trust-ad search home cat /proc/net/route cat /proc/net/route cat /proc/net/route Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT eth0 00000000 0101A8C0 0003 0 0 100 00000000 0 0 0 eth0 0001A8C0 00000000 0001 0 0 100 00FFFFFF 0 0 0 Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT eth0 00000000 0101A8C0 0003 0 0 100 00000000 0 0 0 eth0 0001A8C0 00000000 0001 0 0 100 00FFFFFF 0 0 0 Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT eth0 00000000 0101A8C0 0003 0 0 100 00000000 0 0 0 eth0 0001A8C0 00000000 0001 0 0 100 00FFFFFF 0 0 0 cat /etc/passwd | head -3 cat /etc/passwd | head -3 cat /etc/passwd | head -3 root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin -weight: 600;">sudo cat /etc/shadow | head -1 -weight: 600;">sudo cat /etc/shadow | head -1 -weight: 600;">sudo cat /etc/shadow | head -1 root:$6$rounds=5000$xyz...:19200:0:99999:7::: root:$6$rounds=5000$xyz...:19200:0:99999:7::: root:$6$rounds=5000$xyz...:19200:0:99999:7::: ls -la /proc/$$/fd ls -la /proc/$$/fd ls -la /proc/$$/fd lrwxrwxrwx 1 user user 64 Apr 20 10:01 0 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 1 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 2 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 0 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 1 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 2 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 0 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 1 -> /dev/pts/0 lrwxrwxrwx 1 user user 64 Apr 20 10:01 2 -> /dev/pts/0 ls -la /proc/$(pgrep nginx | head -1)/fd ls -la /proc/$(pgrep nginx | head -1)/fd ls -la /proc/$(pgrep nginx | head -1)/fd cat /etc/hosts cat /etc/hosts cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 myhostname ::1 localhost ip6-localhost ip6-loopback 127.0.0.1 localhost 127.0.1.1 myhostname ::1 localhost ip6-localhost ip6-loopback 127.0.0.1 localhost 127.0.1.1 myhostname ::1 localhost ip6-localhost ip6-loopback grep hosts /etc/nsswitch.conf # hosts: files mdns4_minimal [NOTFOUND=return] dns grep hosts /etc/nsswitch.conf # hosts: files mdns4_minimal [NOTFOUND=return] dns grep hosts /etc/nsswitch.conf # hosts: files mdns4_minimal [NOTFOUND=return] dns cat /lib/systemd/system/ssh.-weight: 500;">service cat /lib/systemd/system/ssh.-weight: 500;">service cat /lib/systemd/system/ssh.-weight: 500;">service [Unit] Description=OpenBSD Secure Shell server After=network.target auditd.-weight: 500;">service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure [Install] WantedBy=multi-user.target [Unit] Description=OpenBSD Secure Shell server After=network.target auditd.-weight: 500;">service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure [Install] WantedBy=multi-user.target [Unit] Description=OpenBSD Secure Shell server After=network.target auditd.-weight: 500;">service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure [Install] WantedBy=multi-user.target -weight: 600;">sudo grep "Failed password" /var/log/auth.log | tail -5 -weight: 600;">sudo grep "Failed password" /var/log/auth.log | tail -5 -weight: 600;">sudo grep "Failed password" /var/log/auth.log | tail -5 Apr 20 03:14:22 host sshd[4821]: Failed password for root from 185.220.101.x port 52344 ssh2 Apr 20 03:14:25 host sshd[4821]: Failed password for root from 185.220.101.x port 52391 ssh2 Apr 20 03:14:22 host sshd[4821]: Failed password for root from 185.220.101.x port 52344 ssh2 Apr 20 03:14:25 host sshd[4821]: Failed password for root from 185.220.101.x port 52391 ssh2 Apr 20 03:14:22 host sshd[4821]: Failed password for root from 185.220.101.x port 52344 ssh2 Apr 20 03:14:25 host sshd[4821]: Failed password for root from 185.220.101.x port 52391 ssh2 # Silence all output command 2>/dev/null # Generate a 1MB file of zeros dd if=/dev/zero of=zeros.bin bs=1M count=1 # Generate cryptographically secure random bytes head -c 32 /dev/urandom | base64 # Silence all output command 2>/dev/null # Generate a 1MB file of zeros dd if=/dev/zero of=zeros.bin bs=1M count=1 # Generate cryptographically secure random bytes head -c 32 /dev/urandom | base64 # Silence all output command 2>/dev/null # Generate a 1MB file of zeros dd if=/dev/zero of=zeros.bin bs=1M count=1 # Generate cryptographically secure random bytes head -c 32 /dev/urandom | base64 cat /proc/meminfo | grep -E "MemTotal|MemFree|Buffers|Cached|MemAvailable" cat /proc/meminfo | grep -E "MemTotal|MemFree|Buffers|Cached|MemAvailable" cat /proc/meminfo | grep -E "MemTotal|MemFree|Buffers|Cached|MemAvailable" MemTotal: 16237568 kB MemFree: 423108 kB Buffers: 312440 kB Cached: 5621700 kB MemAvailable: 7840200 kB MemTotal: 16237568 kB MemFree: 423108 kB Buffers: 312440 kB Cached: 5621700 kB MemAvailable: 7840200 kB MemTotal: 16237568 kB MemFree: 423108 kB Buffers: 312440 kB Cached: 5621700 kB MemAvailable: 7840200 kB cat /proc/cpuinfo | grep "model name" | uniq cat /proc/cpuinfo | grep "model name" | uniq cat /proc/cpuinfo | grep "model name" | uniq -weight: 600;">sudo grep -A5 "menuentry" /boot/grub/grub.cfg | head -10 -weight: 600;">sudo grep -A5 "menuentry" /boot/grub/grub.cfg | head -10 -weight: 600;">sudo grep -A5 "menuentry" /boot/grub/grub.cfg | head -10 menuentry 'Ubuntu' { linux /boot/vmlinuz-6.5.0-generic root=UUID=xxxx ro quiet splash initrd /boot/initrd.img-6.5.0-generic } menuentry 'Ubuntu' { linux /boot/vmlinuz-6.5.0-generic root=UUID=xxxx ro quiet splash initrd /boot/initrd.img-6.5.0-generic } menuentry 'Ubuntu' { linux /boot/vmlinuz-6.5.0-generic root=UUID=xxxx ro quiet splash initrd /boot/initrd.img-6.5.0-generic } - /dev/null — a black hole. Writes disappear. Reads return EOF immediately. - /dev/zero — infinite stream of zero bytes. Used for zeroing disks and memory. - /dev/urandom — reads entropy from the kernel's CSPRNG, seeded by hardware interrupts, disk I/O timing, and network events. - /sys/class/net/ — live network interface stats including dropped packets per interface - /etc/cron.d/ — scheduled tasks; a favourite persistence location for attackers - /proc/net/tcp — all active TCP connections in raw hex format - /etc/ld.so.conf — controls which directories the dynamic linker searches for shared libraries - /proc/sys/kernel/ — live-tunable kernel parameters (also writable via sysctl)