The Stack

How Detection Works

Layer 1: Sliding Windows (60 seconds)

Layer 2: Rolling Baseline (30 minutes)

Layer 3: Anomaly Evaluation

What Happens When an Anomaly Fires

Global Anomaly → Slack Only

Per-IP Anomaly → iptables DROP + Slack + Audit

Tiered Auto-Unban

The Audit Trail

The Dashboard

Baseline Over Time

Lessons Learned

Running It Yourself

What I'd Improve Every VPS running a public web app gets hit with traffic it didn't ask for, from scrapers, brute-force login attempts, or just someone's misconfigured bot hammering the same endpoint every second. Most tutorials say "install Fail2Ban and move on." But what if you want to understand the traffic before you block it? What if you need thresholds that adapt to your actual load instead of a hardcoded "5 failures in 10 minutes"? That's what I built for the HNG DevOps track: a Python daemon that tails Nginx access logs, compares live request rates to a rolling 30-minute baseline, and reacts — Slack alerts for global spikes, iptables DROP for abusive individual IPs, with tiered auto-unban so a single bad minute doesn't permanently lock someone out. Repository: github.com/Trojanhorse7/hng-anomaly-detector The whole system runs on a single Linux VPS with Docker Compose: The detector container runs with network_mode: host and cap_add: NET_ADMIN so its iptables calls affect the actual host firewall — not an isolated container network. The detection pipeline has three layers: sliding windows, rolling baseline, and anomaly evaluation. Every parsed log line feeds into collections.deque structures — one global deque for all requests, and one per source IP. Timestamps older than 60 seconds are continuously evicted from the left side. At any moment, RPS = count / 60. There's no "bucket per minute" approximation. Every request is tracked individually and aged out precisely. Parallel deques track 4xx/5xx errors separately for the error-surge path (more on that below). A background thread recomputes the baseline every 60 seconds. It builds a dense vector of per-second request counts over the last 1,800 seconds (30 minutes) and calculates: There's an important twist: if enough samples exist in the current UTC hour, the baseline uses only that hour's data instead of the full 30-minute window. This matters because traffic patterns shift — 2 AM is different from 2 PM, and the baseline should reflect current conditions, not a blend of quiet and busy periods. Floor values prevent divide-by-zero edge cases in z-score calculations. Every recompute is audited to a structured log file with the timestamp, source (hourly vs full window), and the computed mean/std. For each incoming request, the detector compares current RPS to the baseline. An anomaly fires if either condition is true: Error surge tightening: if an IP's error RPS (4xx/5xx responses) exceeds 3× the baseline error mean, thresholds tighten automatically — z-score drops to 2.0 and the rate multiplier drops to 3×. This means an IP generating lots of failed requests gets scrutinized more aggressively, which is exactly what you want for brute-force login attempts. The system distinguishes between global and per-IP anomalies, and they trigger different responses: If the aggregate RPS across all IPs spikes above the baseline, the detector sends a Slack notification. It does not apply iptables rules — blocking all traffic would take the service down. Global alerts are informational: "your server is seeing unusual load right now." A cooldown (default 120 seconds) prevents Slack spam if the global anomaly persists for minutes. If a single IP is responsible for anomalous traffic, the detector: Permanently banning IPs from a single spike is too aggressive. The system uses escalating timeouts: A background thread checks every 3 seconds for IPs whose ban has expired, removes the iptables rule, and sends an unban Slack notification. The strike counter persists across container restarts via a JSON file (ban_state.json). This means a legitimate user who triggered a false positive gets unblocked in 10 minutes. A repeat offender escalates through the tiers. By the 4th strike, they're gone for good. Every significant event is appended to a structured log file at data/audit.log: This file is the source of truth for debugging, compliance, and the baseline graph (more below). A FastAPI server on port 8080 serves a single-page dashboard with live metrics via WebSocket push (every 2.5 seconds). If WebSocket fails (e.g., behind a proxy without Upgrade support), the page falls back to HTTP polling automatically. The /api/state JSON endpoint returns: One of the requirements was demonstrating that the baseline actually adapts. By parsing BASELINE_RECALC lines from the audit log and plotting effective_mean over time, you can see the baseline shift as traffic patterns change between UTC hours. During a busy period, effective_mean climbs. When traffic drops, it falls. The hourly-slice preference means the baseline reacts to the current hour's pattern rather than being dragged by stale data from 25 minutes ago. 1. JSON logs are non-negotiable. Parsing regex against Nginx's default combined log format is fragile. One unusual user-agent string with spaces and quotes breaks your parser. JSON logs with escape=json in the Nginx config give you reliable field extraction every time. 2. Host networking in Docker is powerful but surprising. network_mode: host means the container shares the host's network stack — iptables rules apply to the actual server, not a virtual bridge. This is exactly what you want for blocking IPs, but it also means port conflicts are your problem. 3. Hardcoded thresholds are the enemy. "Block after 100 requests per minute" sounds reasonable until your app legitimately serves 200 req/s during peak hours. A rolling baseline that adapts to actual traffic means your thresholds stay meaningful whether you're serving 2 req/s at 3 AM or 50 req/s at noon. 4. Tiered responses prevent self-inflicted outages. The first time I tested with aggressive thresholds, my own monitoring IP got permanently banned. Escalating tiers (10m → 30m → 2h → permanent) give false positives a way to recover while still catching persistent abuse. 5. Audit everything. When something goes wrong — a legitimate user gets blocked, or an attack slips through — the audit log tells you exactly what the baseline was, what the detector saw, and why it made the decision it did. Without that, you're guessing. Nextcloud at http://<VPS_IP>/, dashboard at http://<VPS_IP>:8080/. Thresholds, window sizes, and ban durations are all in detector/config.yaml — no code changes needed to tune the system. Built for the HNG DevOps track. The full source is at github.com/Trojanhorse7/hng-anomaly-detector. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse