Tools: SSH died. Spent 3 hours fixing the wrong thing. (2026)

Tools: SSH died. Spent 3 hours fixing the wrong thing. (2026)

nmap first

Where it actually breaks

The fix

Round two

The actual reason

What to do from day one Three or four days ago I deployed a Gitea runner, tested it, closed the terminal and moved on. Today, for some reason I don't even remember, I tried to SSH into the server. Nothing. Opened the browser to check if the server was alive — course project running, Gitea up, Nextcloud up, everything in the k3s cluster fine. Went into VMmanager, rebooted. Still couldn't get in. Closed The Boys and went to fix it through VNC. filtered — not closed, not refused. Packets reaching the server and getting silently dropped. Firewall somewhere. Checked the INPUT chain: ACCEPT rule for port 22 at position 2. Looked fine. Zero packets on the accept rule. Connections weren't reaching it at all. The server has both iptables-nft (kube-router, UFW) and native nftables running at the same time. They fight. kube-router constantly reconciles and overwrites stuff. The output was already warning about it: The Gitea runner deploy from a few days ago triggered a reconciliation that messed up the rule ordering. I just hadn't tried SSH since then. Native nftables table at priority -150. Runs before any filter chains (priority 0), before kube-router, before everything. kube-router only manages the iptables-nft layer — doesn't touch native nftables tables. SSH came back. Wrote to my friend: "fixed". He said "eee". Made it a systemd service. Two minutes later wrote to him: "SSH died again". Back in VNC. Table still there, rules still there, zero packets on the accept rule again. systemctl start nftables had loaded the saved /etc/nftables.conf which had kube-router's warning comments in it and corrupted the nftables state. Re-added the table manually. SSHed from the server to itself — worked. sshd on 0.0.0.0:22 — fine. Still couldn't connect from outside. Then checked fail2ban, which had installed itself as a dependency somewhere during all this. It had banned two IPs — bots hammering port 22 the whole time. My repeated failed attempts got caught in the same drop. Ran fail2ban-client unban --all, connected immediately. Then SSH died again. Disabled fail2ban completely — still nothing. I was in the middle of debugging nftables chains when I remembered — my VPN is self-hosted on a server in Finland. The day before I'd been complaining to my friend that the VPN was slow. The provider had sent an email: Due to an increase in brute-force complaints via SSH (port 22), outgoing traffic on port 22 has been blocked for VPS servers located in Finland. I had read it. I knew about it. And I spent three hours in VNC chasing nftables, fail2ban and kube-router while the answer was sitting in my inbox the whole time. Turned off the VPN. SSH connected immediately. Add the ssh_rescue table before you need it: Make it a service so it survives reboots and kube-router reconciliation: After any deploy that touches networking — run nmap -p 22 <ip> immediately. And read your damn emails. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to ? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Code Block

Copy

nmap -p 22 2.27.42.100 nmap -p 22 2.27.42.100 nmap -p 22 2.27.42.100 PORT STATE SERVICE 22/tcp filtered ssh PORT STATE SERVICE 22/tcp filtered ssh PORT STATE SERVICE 22/tcp filtered ssh Chain INPUT (policy DROP) 1 KUBE-ROUTER-INPUT 2 ACCEPT tcp dpt:22 3 KUBE-PROXY-FIREWALL ... 8 ufw-before-input Chain INPUT (policy DROP) 1 KUBE-ROUTER-INPUT 2 ACCEPT tcp dpt:22 3 KUBE-PROXY-FIREWALL ... 8 ufw-before-input Chain INPUT (policy DROP) 1 KUBE-ROUTER-INPUT 2 ACCEPT tcp dpt:22 3 KUBE-PROXY-FIREWALL ... 8 ufw-before-input nft list ruleset | grep -E "22|drop|DROP" nft list ruleset | grep -E "22|drop|DROP" nft list ruleset | grep -E "22|drop|DROP" type filter hook input priority filter; policy drop; ... tcp dport 22 counter packets 0 bytes 0 accept type filter hook input priority filter; policy drop; ... tcp dport 22 counter packets 0 bytes 0 accept type filter hook input priority filter; policy drop; ... tcp dport 22 counter packets 0 bytes 0 accept # Warning: table ip filter is managed by iptables-nft, do not touch! # Warning: table ip filter is managed by iptables-nft, do not touch! # Warning: table ip filter is managed by iptables-nft, do not touch! nft add table inet ssh_rescue nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }' nft add rule inet ssh_rescue input tcp dport 22 accept nft add table inet ssh_rescue nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }' nft add rule inet ssh_rescue input tcp dport 22 accept nft add table inet ssh_rescue nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }' nft add rule inet ssh_rescue input tcp dport 22 accept systemctl start nftables /etc/nftables.conf fail2ban-client unban --all nft add table inet ssh_rescue nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }' nft add rule inet ssh_rescue input tcp dport 22 accept nft add table inet ssh_rescue nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }' nft add rule inet ssh_rescue input tcp dport 22 accept nft add table inet ssh_rescue nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }' nft add rule inet ssh_rescue input tcp dport 22 accept cat > /usr/local/bin/ssh-rescue.sh << 'EOF2' #!/bin/bash nft list table inet ssh_rescue 2>/dev/null || { nft add table inet ssh_rescue nft add chain inet ssh_rescue input { type filter hook input priority -150; } nft add rule inet ssh_rescue input tcp dport 22 accept } EOF2 chmod +x /usr/local/bin/ssh-rescue.sh cat > /usr/local/bin/ssh-rescue.sh << 'EOF2' #!/bin/bash nft list table inet ssh_rescue 2>/dev/null || { nft add table inet ssh_rescue nft add chain inet ssh_rescue input { type filter hook input priority -150; } nft add rule inet ssh_rescue input tcp dport 22 accept } EOF2 chmod +x /usr/local/bin/ssh-rescue.sh cat > /usr/local/bin/ssh-rescue.sh << 'EOF2' #!/bin/bash nft list table inet ssh_rescue 2>/dev/null || { nft add table inet ssh_rescue nft add chain inet ssh_rescue input { type filter hook input priority -150; } nft add rule inet ssh_rescue input tcp dport 22 accept } EOF2 chmod +x /usr/local/bin/ssh-rescue.sh nmap -p 22 <ip>