What I Was Trying to Do

What Happened — The Crash Loop

Understanding the Problem — Why It Was So Hard to Escape

The Solution — Masking the VM Start Service at Boot

Step 1: Stop the autoboot timer at GRUB

Step 2: Edit the boot command

What systemd.mask=pve-guests.service actually does

Step 3: Fix the VM config via web UI

Plot Twist — The GPU Renamed Itself

Root Cause Analysis

Lessons Learned

The Final State

Quick Reference — Commands Used How a Kubernetes worker VM with a passed-through AMD GPU sent my entire home lab into an infinite crash loop — and the GRUB-level trick that saved it. My home lab runs on a Mini PC with Proxmox as the hypervisor. The setup hosts a mix of LXC containers and KVM virtual machines — Jellyfin for media, a Pi-hole, a Kubernetes cluster (k3s), Cloudflare tunnels, and a bunch of other self-hosted services. I was upgrading the hardware configuration of VM 104 (k3s-vm-worker) — one of the worker nodes in my k3s Kubernetes cluster. The goal was straightforward: pass through the host's AMD GPU directly into the VM so the Kubernetes worker could handle GPU-accelerated workloads. In Proxmox, GPU passthrough (PCIe passthrough) requires a few things: I configured all of this. The hardware tab showed the PCI device (0000:e6:00, highlighted in orange — a warning sign I probably should have paid more attention to). I saved the config, felt good about it, and rebooted. The Mini PC came back up, Proxmox started loading, and then — nothing. The host became completely unresponsive. No web UI. No SSH. Nothing. I power-cycled it. Same thing. It would power on, start booting, and crash before I could even get to the Proxmox web interface. Here's what was actually happening, which I only understood after I could finally get in and check the logs: VM 104 had "Start at boot" set to Yes. The moment Proxmox finished loading, it triggered a "Bulk Start VMs" task — its automatic process for starting all VMs flagged to auto-start. VM 104 was in that queue. The second that task reached VM 104, Proxmox tried to initialise the PCIe passthrough, the VM grabbed the GPU from the host, and the entire system crashed — hard. I panicked and quickly got back into the web UI during a narrow window and changed "Start at boot" to No. Thought that would fix it. Proxmox had already queued the bulk start task the moment it turned on. Changing the setting mid-flight did nothing. The next boot, same crash. The boot after that, same crash. I was locked in an auto-start crash loop with no way out through the web UI. This is the part that makes this specific failure mode so nasty: The window between "Proxmox is up enough to serve the web UI" and "VM 104 has already been queued and grabbed the GPU" was too small to intervene through normal means. Every time I managed to get in and change something, the crash had already been scheduled. The only way out was to break the loop before Proxmox had any chance to load its VM management services at all — which means intervening at the GRUB level, before the operating system even finishes booting. This required plugging a physical keyboard and monitor into the Mini PC. No SSH, no remote — physical access only. Power on the machine and watch the screen. The moment the blue Proxmox GRUB menu appears, press the Down Arrow key immediately. This stops the autoboot countdown timer and lets you stay at the GRUB menu. Highlight the top "Proxmox VE" entry and press e to edit the boot command. You'll see the full kernel command line. Navigate to the end of the linux line. Remove amd_iommu=on iommu=pt (these were the IOMMU parameters I'd added earlier that enabled the passthrough). Then, at the very end of the linux line, add: So the line ends with: Press Ctrl + X to boot with this modified command. pve-guests.service is the systemd unit responsible for the "Bulk Start VMs" behaviour in Proxmox. Masking it via the kernel command line at boot tells systemd to completely skip that service for this boot session only — it doesn't persist across reboots. Once safely in the web UI with no VMs running: After fixing the crash loop and getting the host stable, I started my LXC containers — CT 204 (jellyfin-arr) and CT 208 (linkwarden) — and hit a completely different error: Both containers refused to start. I checked the DRI devices on the host: The AMD GPU loaded perfectly (Kernel driver in use: amdgpu). But Linux had decided to name it card1 instead of card0. This is a surprisingly common Linux behaviour: when you change BIOS IOMMU settings, or reboot without an HDMI cable connected to a GPU, the kernel can enumerate DRM devices in a different order and assign a different cardN number. The GPU is fine. The driver is fine. The device node just got a different name. My LXC containers were hardcoded to pass through /dev/dri/card0. The fix was trivial once I understood it: Both containers started instantly. Hardware transcoding in Jellyfin came back up immediately. Looking back, there were actually three separate issues stacked on top of each other: The orange 0000:e6:00 text in the VM Hardware tab was the visual warning I missed — in Proxmox, orange on a PCI device entry typically indicates the device may not be properly isolated in its IOMMU group, or that there's a configuration issue worth investigating before saving and booting. 1. Never enable "Start at boot" on a VM with PCIe passthrough until you've verified the passthrough works correctly in a manual start first. Test the VM with a manual start. Confirm the host stays stable. Confirm the VM boots correctly with the passed-through device. Only then enable auto-start. 2. systemd.mask=<service> at GRUB is your emergency brake. Any time Proxmox is in a state where it crashes before you can intervene through the web UI, this technique gives you a clean boot with surgical control over what services start. pve-guests.service is the specific one to mask for VM auto-start loops. 3. GPU device node names are not stable across reboots in Linux. /dev/dri/card0 is not guaranteed to be your GPU after every reboot, especially when BIOS settings change. Instead of hardcoding card0, consider: 4. PCIe passthrough machine type matters — and q35 has implications. q35 is required for proper PCIe passthrough but behaves differently from i440fx in several ways. Switching machine types mid-configuration without a working baseline is risky. Start with a known-good VM on i440fx, verify it's stable, then migrate to q35 and add the PCI device. 5. Physical access to your home lab server is non-negotiable. If you're running a home lab with GPU passthrough or any kind of hardware-level config, have a keyboard and monitor you can plug in. SSH and the web UI are great until they're not. The GRUB console saved this entire setup. The GPU passthrough into the Kubernetes worker is still on the roadmap — I'll approach it differently: test in a throwaway VM first, verify IOMMU grouping, confirm stability with manual start, and only then wire it into the k3s node. Running a home lab means breaking things and learning from them. This one taught me more about Proxmox internals, systemd masking, and Linux DRM device enumeration than any documentation would have. Hope it saves someone else the same panic. Feel free to drop questions in the comments — happy to help if you're stuck in a similar loop. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse