Tools: Node Exporter Deep Dive: Connecting the Linux Kernel and Prometheus

Introduction ## 1. Architecture Overview ## The Flow of Metrics ## What is the Prometheus Format? ## 2. Deep Dive: Collectors ## Inside Major Collectors ## CPU Collector (cpu) ## Memory Collector (meminfo) ## Filesystem Collector (filesystem) ## 3. The Power of "Textfile Collector" ## Use Cases ## Architecture of Textfile Collector ## 1. Differences in Role ## 2. Architectural Differences ## 3. The Only Overlap: Textfile Collector ## Advanced Usage: Turning Application Metrics into Metrics ## Pattern 1: Textfile Collector ## Pattern 2: JSON Exporter ## 4. Hands-on: Testing on Mac (via Multipass) ## Step 1: Prepare the Linux Environment ## Step 2: Run Node Exporter ## Step 3: Verify Metrics ## Summary Many engineers think, "I want to monitor my system, so I'll just install Node Exporter." But have you ever stopped to consider what this binary is actually doing? Node Exporter doesn't use magic. It is simply a "translator" that takes information exposed by the Linux kernel and converts it into a format that Prometheus can understand. In this article, I will explain the internal architecture of Node Exporter, specifically the Collector mechanism and its interaction with the kernel, using diagrams (Mermaid). Node Exporter's role is very simple. It receives an HTTP GET request (/metrics) from the Prometheus server and returns the system's health status at that exact moment in text format. The key point is that Node Exporter itself is stateless. It doesn't remember "the CPU usage from one minute ago." It simply reads the current values from /proc/stat the moment Prometheus comes to fetch them. What Node Exporter ultimately returns is not binary data, but human-readable text data. This is called the "Prometheus Exposition Format." The structure is very simple: a list of "metric_name{labels} value". Before: Linux Kernel (/proc/loadavg) (By itself, it's hard to tell what these values represent.) After: Node Exporter (/metrics) (Meaningful names and type information are added, making it interpretable by Prometheus.) The internals of Node Exporter consist of a collection of modules called Collectors. When you run node_exporter --help, you'll see a massive list of flags; these are used to enable or disable specific Collectors. Most Collectors are built simply to parse the Linux virtual filesystem (/proc, /sys). If the standard Collectors aren't enough, do you need to write Go code and rebuild? No, that's where the Textfile Collector comes in. This is a simple yet powerful feature that "takes the contents of .prom files located in a specific directory and returns them directly to Prometheus." Important: Writing to the .prom file must be done atomically. If Node Exporter reads a file while it is being written, it might send corrupted data to Prometheus. The short answer is: "They don't conflict (they coexist)." Since their roles are distinct, it is common to run both on the same server. There is no technical collision as they use different default ports. In your Prometheus configuration (prometheus.yml), you register them as separate targets: A choice (and potential overlap) only arises when deciding how to pass JSON data to Prometheus: via a script that writes to a file for Node Exporter, or via JSON Exporter. When you want to monitor application-specific metrics (e.g., {"active_users": 150}), there are two main approaches. For simple numeric values, it's easy to convert them to Prometheus format using a shell script and let Node Exporter read them. If the JSON structure is complex or you want to monitor API endpoints directly, I recommend using json_exporter. Since its port does not conflict with Node Exporter, you can use them together for advanced application monitoring. For Mac users who think, "I get it, but I don't have a Linux machine handy," let's use Multipass to see how /proc values actually turn into metrics. While you can run Node Exporter directly on Mac, the macOS (Darwin) kernel doesn't have /proc, making it unsuitable for exploring the Linux-specific behaviors (CPU, meminfo, filesystem collectors) discussed here. Spin up a lightweight Ubuntu VM instantly. Install and start Node Exporter and Json Exporter inside the VM. Open another terminal and hit the metrics endpoint from outside the VM (from your Mac). Now you should see raw CPU counters pulled straight from a real Linux kernel (/proc/stat). Don't forget to run multipass stop node-exp-test when you're finished. Node Exporter is a reliable interpreter that translates the Linux kernel's /proc and /sys into the language of Prometheus (Metrics format). The next time a "High CPU Usage" alert sounds, being able to visualize how Node Exporter pulled those numbers from /proc/stat will raise the resolution of your troubleshooting by a whole level. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse CODE_BLOCK: 0.02 0.04 0.00 1/157 28941 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: 0.02 0.04 0.00 1/157 28941 CODE_BLOCK: 0.02 0.04 0.00 1/157 28941 COMMAND_BLOCK: # HELP node_load1 1m load average. # TYPE node_load1 gauge node_load1 0.02 Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # HELP node_load1 1m load average. # TYPE node_load1 gauge node_load1 0.02 COMMAND_BLOCK: # HELP node_load1 1m load average. # TYPE node_load1 gauge node_load1 0.02 COMMAND_BLOCK: # ❌️ Bad Practice echo "my_metric 1" > /var/lib/node_exporter/my.prom # ✅️ Good Practice (Atomic Rename) echo "my_metric 1" > /var/lib/node_exporter/my.prom.tmp mv /var/lib/node_exporter/my.prom.tmp /var/lib/node_exporter/my.prom Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # ❌️ Bad Practice echo "my_metric 1" > /var/lib/node_exporter/my.prom # ✅️ Good Practice (Atomic Rename) echo "my_metric 1" > /var/lib/node_exporter/my.prom.tmp mv /var/lib/node_exporter/my.prom.tmp /var/lib/node_exporter/my.prom COMMAND_BLOCK: # ❌️ Bad Practice echo "my_metric 1" > /var/lib/node_exporter/my.prom # ✅️ Good Practice (Atomic Rename) echo "my_metric 1" > /var/lib/node_exporter/my.prom.tmp mv /var/lib/node_exporter/my.prom.tmp /var/lib/node_exporter/my.prom COMMAND_BLOCK: scrape_configs: - job_name: 'node' static_configs: - targets: ['localhost:9100'] # Node Exporter (OS Monitoring) - job_name: 'json' static_configs: - targets: ['localhost:7979'] # JSON Exporter (App Data Monitoring) Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: scrape_configs: - job_name: 'node' static_configs: - targets: ['localhost:9100'] # Node Exporter (OS Monitoring) - job_name: 'json' static_configs: - targets: ['localhost:7979'] # JSON Exporter (App Data Monitoring) COMMAND_BLOCK: scrape_configs: - job_name: 'node' static_configs: - targets: ['localhost:9100'] # Node Exporter (OS Monitoring) - job_name: 'json' static_configs: - targets: ['localhost:7979'] # JSON Exporter (App Data Monitoring) COMMAND_BLOCK: # Extract value from JSON and output to .prom file (read by Node Exporter) echo "app_active_users $(cat app_status.json | jq '.active_users')" > /var/lib/node_exporter/metrics.prom Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Extract value from JSON and output to .prom file (read by Node Exporter) echo "app_active_users $(cat app_status.json | jq '.active_users')" > /var/lib/node_exporter/metrics.prom COMMAND_BLOCK: # Extract value from JSON and output to .prom file (read by Node Exporter) echo "app_active_users $(cat app_status.json | jq '.active_users')" > /var/lib/node_exporter/metrics.prom CODE_BLOCK: metrics: - name: app_active_users path: "{ .active_users }" help: "Active users count from JSON source" type: gauge Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: metrics: - name: app_active_users path: "{ .active_users }" help: "Active users count from JSON source" type: gauge CODE_BLOCK: metrics: - name: app_active_users path: "{ .active_users }" help: "Active users count from JSON source" type: gauge COMMAND_BLOCK: # Install Multipass (if not already installed) brew install --cask multipass # Launch Ubuntu instance multipass launch --name node-exp-test multipass shell node-exp-test Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Install Multipass (if not already installed) brew install --cask multipass # Launch Ubuntu instance multipass launch --name node-exp-test multipass shell node-exp-test COMMAND_BLOCK: # Install Multipass (if not already installed) brew install --cask multipass # Launch Ubuntu instance multipass launch --name node-exp-test multipass shell node-exp-test COMMAND_BLOCK: # Node Exporter sudo apt update sudo apt install -y prometheus-node-exporter # Json Exporter wget https://github.com/prometheus-community/json_exporter/releases/download/v0.7.0/json_exporter-0.7.0.linux-arm64.tar.gz tar xvfz json_exporter-0.7.0.linux-arm64.tar.gz # Setup config cat <<EOF > json_exporter_config.yml metrics: - name: app_active_users path: "{ .active_users }" help: "Dummy Application Active Users" type: gauge EOF ./json_exporter-0.7.0.linux-arm64/json_exporter --config.file json_exporter_config.yml & Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Node Exporter sudo apt update sudo apt install -y prometheus-node-exporter # Json Exporter wget https://github.com/prometheus-community/json_exporter/releases/download/v0.7.0/json_exporter-0.7.0.linux-arm64.tar.gz tar xvfz json_exporter-0.7.0.linux-arm64.tar.gz # Setup config cat <<EOF > json_exporter_config.yml metrics: - name: app_active_users path: "{ .active_users }" help: "Dummy Application Active Users" type: gauge EOF ./json_exporter-0.7.0.linux-arm64/json_exporter --config.file json_exporter_config.yml & COMMAND_BLOCK: # Node Exporter sudo apt update sudo apt install -y prometheus-node-exporter # Json Exporter wget https://github.com/prometheus-community/json_exporter/releases/download/v0.7.0/json_exporter-0.7.0.linux-arm64.tar.gz tar xvfz json_exporter-0.7.0.linux-arm64.tar.gz # Setup config cat <<EOF > json_exporter_config.yml metrics: - name: app_active_users path: "{ .active_users }" help: "Dummy Application Active Users" type: gauge EOF ./json_exporter-0.7.0.linux-arm64/json_exporter --config.file json_exporter_config.yml & COMMAND_BLOCK: VM_IP=$(multipass info node-exp-test | grep IPv4 | awk '{print $2}') # Fetch metrics ## Node Exporter curl http://$VM_IP:9100/metrics ## Json Exporter curl http://$VM_IP:7979/metrics Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: VM_IP=$(multipass info node-exp-test | grep IPv4 | awk '{print $2}') # Fetch metrics ## Node Exporter curl http://$VM_IP:9100/metrics ## Json Exporter curl http://$VM_IP:7979/metrics COMMAND_BLOCK: VM_IP=$(multipass info node-exp-test | grep IPv4 | awk '{print $2}') # Fetch metrics ## Node Exporter curl http://$VM_IP:9100/metrics ## Json Exporter curl http://$VM_IP:7979/metrics - Source: /proc/stat - Metric: node_cpu_seconds_total - Mechanism: The kernel maintains CPU usage time per CPU, categorized by modes like "USER," "NICE," "SYSTEM," and "IDLE." Node Exporter outputs these as counters. The actual calculation (e.g., using rate()) is the job of Prometheus. - Source: /proc/meminfo - Metric: node_memory_MemTotal_bytes, node_memory_MemAvailable_bytes, etc. - Mechanism: It parses the contents of /proc/meminfo and converts them into bytes. - Source: statfs system call - Metric: node_filesystem_size_bytes, node_filesystem_free_bytes - Mechanism: It issues system calls for each mounted filesystem to retrieve capacity and Inode information. Note that if an NFS mount hangs, this Collector might also hang and cause a timeout. - Cron Job Results: Success/failure of backup scripts (my_backup_last_run_timestamp). - Package Versions: Number of pending updates for apt or yum. - Hardware-Specific Info: RAID controller status or other metrics not supported by default. - Node Exporter: A tool to monitor "Infrastructure (OS)." CPU, Memory, Disk usage, Network traffic, etc. Visualizes information held by the Linux kernel. - CPU, Memory, Disk usage, Network traffic, etc. - Visualizes information held by the Linux kernel. - JSON Exporter: A tool to monitor "Application Internals (Data)." Business metrics returned by APIs (user count, sales, inventory, etc.). Parses application-specific JSON data that Node Exporter cannot reach. - Business metrics returned by APIs (user count, sales, inventory, etc.). - Parses application-specific JSON data that Node Exporter cannot reach. - CPU, Memory, Disk usage, Network traffic, etc. - Visualizes information held by the Linux kernel. - Business metrics returned by APIs (user count, sales, inventory, etc.). - Parses application-specific JSON data that Node Exporter cannot reach. - Node Exporter: Port 9100 (Default) - JSON Exporter: Port 7979 (Default) - Node Exporter: For OS metrics or simple custom metrics. - JSON Exporter: For parsing structured JSON data or monitoring external APIs. - Simple & Stateless: It holds no state and fetches fresh data from the kernel for every request. - Format Translation: It converts cryptic kernel data into simple text format (Key Value). - Extensible: Using the Textfile Collector, you can easily turn JSON logs or batch process results into metrics.