Introduction

Basically why do I need a Kubernetes cluster ?

Part 1: Setting Up the Master Node

Installing k3s Server

Fixing kubectl Access

Retrieve the Node Token

Part 2: Adding the VM Worker (worker1)

Joining the Cluster

Problem: Node Password Rejected

Problem: Duplicate Hostname

Part 3: The LXC Worker — The Real Story

What is an LXC Container?

Creating the LXC Container

Joining the Cluster

Error 1: /dev/kmsg: no such file or directory

Error 2: /dev/kmsg: operation not permitted

Error 3: /proc/sys/kernel/panic: read-only file system

Error 4: Various Permission Denied Errors

Also needed: features: keyctl=1,nesting=1

Final Working LXC Config

Summary: What Each Modification Does

Part 4: LXC as a k3s Worker — Features and Limitations

Features / Advantages

Limitations

Conclusion A practical guide covering installation, troubleshooting, and the real story of getting k3s to run inside an LXC container Kubernetes is powerful but notorious for being heavy. k3s, the lightweight Kubernetes distribution from Rancher, fixes that. It strips out legacy APIs, bundles containerd, and ships as a single binary under 100MB. It is perfect for homelabs, edge deployments, and resource-constrained environments.

(more about k3s: https://traefik.io/glossary/k3s-explained/) This is the first of a series of posts describing how to bootstrap a Kubernetes cluster on Proxmox using ubuntu VM and LXC containers. By the end of the series, the aim is to have a fully working Kubernetes (K3S) install including MetalLB load balancer, Gateway API controller and an Istio service mesh. I’ll also have some sample applications installed for good measure. At work, I’ve used large K8S clusters in production environments (AWS), clusters are abstracted away behind platform teams, which is efficient for delivery but leaves gaps in understanding how scheduling, networking, storage, and controllers really behave under the hood. Setting up your own cluster gives you that missing layer of operational intuition: you get to break things, debug them, and understand why they broke. For someone already running a fairly complex home setup, using Kubernetes as a unifying platform to experiment, whether or not you fully migrate all your Docker Compose stacks—is less about necessity and more about building practical, transferable expertise. In this post I document how I built a three-node k3s cluster on Proxmox VE with: 1 master node — a Proxmox VM running Ubuntu 1 VM worker node — a standard Proxmox VM (worker1) 1 LXC worker node — a Proxmox LXC container (worker2) The VM setup was straightforward. The LXC setup was not. This post focuses heavily on the LXC journey — the errors, the fixes, the Linux internals involved, and what it finally took to make it work. On the master VM, installing k3s is a single command: k3s sets up a systemd service, installs containerd, and bootstraps a single-node Kubernetes cluster automatically. After installation, running kubectl get nodes immediately fails: The connection to the server localhost:8080 was refused This happens because kubectl defaults to localhost:8080 when no kubeconfig is set. k3s stores its kubeconfig at /etc/rancher/k3s/k3s.yaml. The fix: Or export it permanently: Worker nodes need a token to join the cluster. Grab it from the master: Keep this value — it is used in every worker join command. On the worker VM, run: The agent started but immediately logged: This happened because the worker VM had previously joined the cluster. k3s stores a node password on both the node (/etc/rancher/node/password) and the master (as a Kubernetes secret). When they don't match, the server rejects the node. Fix — on the master, delete the stale secret: Both the master and worker had the hostname k3s. k3s uses the hostname as the node name, so the server rejected the second node as a duplicate. Fix — rename the worker: After renaming and cleaning up the stale secret, the worker joined successfully. LXC (Linux Containers) is a lightweight virtualisation technology. Unlike VMs which emulate full hardware, LXC containers share the host kernel directly. They use Linux namespaces for isolation and cgroups for resource control. They are faster and more efficient than VMs but have less isolation. Proxmox LXC containers can be privileged (root inside = root on host) or unprivileged (root inside maps to a regular user on host via UID namespacing). Unprivileged is the default and more secure option. In Proxmox, I created a Debian Trixie LXC container with: The install script ran and printed [INFO] systemd: Starting k3s-agent — and then nothing. It just hung. Checking the journal: /dev/kmsg is the kernel message buffer device. The Linux kernel uses it to log messages (this is what dmesg reads). kubelet uses it to watch for OOM (Out of Memory) kill events via the oomWatcher. Without it, kubelet refuses to start. In an unprivileged LXC container, /dev/kmsg does not exist because the container does not have access to kernel devices. Fix — bind mount from host: In /etc/pve/lxc/209.conf on the Proxmox host: This bind mounts the host's /dev/kmsg into the container. Stop and start (not restart) the LXC: After adding the bind mount, the error changed slightly: The file now existed in the container but the process was not allowed to open it. The container was still running in user namespace mode (unprivileged), and AppArmor was blocking the access. Fix — disable AppArmor restriction: AppArmor is a Linux Security Module that applies mandatory access control policies. The default Proxmox LXC AppArmor profile blocks access to kernel devices like /dev/kmsg. Setting it to unconfined removes all AppArmor restrictions for this container. /proc is a virtual filesystem the kernel exposes so userspace can read and write kernel parameters. /proc/sys/ specifically contains sysctl values — tuneable kernel settings. kubelet needs to write to these on startup: kernel/panic — configure kernel panic timeout kernel/panic_on_oops — whether a kernel oops causes a panic vm/overcommit_memory — memory overcommit policy In an unprivileged LXC container, /proc is mounted read-only for safety. Any process inside the container (even root inside) cannot modify these values. Fix — mount proc and sys as read-write: This tells LXC to mount /proc and /sys with read-write access instead of the default read-only. These were caused by the container still running as unprivileged — the process was root inside the container but mapped to a normal user on the host, so many privileged operations were blocked. Fix — switch to privileged container: This is the most significant change. A privileged container maps root inside to actual root on the host. This removes the UID namespace remapping that caused most of the permission errors. cgroup2.devices.allow: a — allows the container access to all devices via the cgroup device controller cap.drop: (empty) — prevents Proxmox from dropping any Linux capabilities. By default, Proxmox drops capabilities like CAP_SYS_ADMIN, CAP_NET_ADMIN, and CAP_SYS_PTRACE from LXC containers. k3s needs these. keyctl=1 — enables the Linux kernel keyring inside the container. containerd uses this to securely store credentials and keys for image pulls. nesting=1 — enables nested containerisation. k3s runs containerd inside the LXC container, and containerd runs pods (more containers) inside itself. Without nesting enabled, Proxmox blocks the inner container creation. After applying all these changes and doing a full pct stop / pct start: Resource efficiency — LXC containers consume significantly less memory and CPU than VMs. A VM needs a full OS kernel in memory. An LXC container shares the host kernel, so the overhead is minimal. worker2 running k3s uses around 250–300MB RAM idle versus a VM which would use 500MB+ for the OS alone. Fast startup — LXC containers start in 1–3 seconds versus 15–30 seconds for a VM. For ephemeral worker nodes or autoscaling scenarios this matters. Storage efficiency — LXC uses the host filesystem directly (with a root filesystem overlay). No separate virtual disk emulation layer. I/O is closer to bare metal performance. Simple networking — LXC containers participate in the same Proxmox bridge (vmbr0) as VMs. No extra networking configuration is needed for k3s to communicate between the master VM and the LXC worker. Density — you can run more LXC containers on the same Proxmox host than VMs, making it ideal for testing multi-node cluster topologies on limited hardware. Shared kernel — no kernel version isolation — all LXC containers on a host run the same kernel version as the host. You cannot run a different kernel inside an LXC container. This matters if you need a specific kernel feature or version for your workloads. Privileged mode is a security trade-off — to get k3s working we had to switch to a privileged container and disable AppArmor. In a privileged container, a root escape inside the container gives root on the host. For a homelab or trusted environment this is acceptable; for production or multi-tenant setups it is a significant risk. No hardware virtualisation — LXC containers cannot run nested VMs. If your workloads need hardware-level isolation or GPU passthrough in the container, a VM is required. Kernel module limitations — the LXC container cannot load kernel modules that aren't already loaded on the host. During setup we saw: These modules need to be loaded on the Proxmox host, not inside the container. Some syscalls are blocked — even in privileged mode, certain syscalls that could affect the host are restricted. This can cause subtle compatibility issues with some container workloads. Not suitable for untrusted workloads — because the kernel is shared, a kernel exploit inside an LXC container could theoretically affect the host and all other containers. Never run untrusted code in a privileged LXC container. Getting k3s running on a Proxmox LXC container is absolutely possible, but it requires understanding why each restriction exists and selectively removing the ones that conflict with k3s's requirements. The journey from a blank LXC to a working cluster node touched on AppArmor, Linux capabilities, cgroups, kernel device access, namespace nesting, and virtual filesystem permissions. The key takeaway: LXC containers are not VMs. They share the host kernel, and every security restriction that makes them safe is also a potential blocker for complex software like k3s that expects a full OS environment. The solution is not to blindly disable everything — it is to understand each error, trace it to the underlying Linux feature, and make the minimal change required to unblock it. The final cluster — one control plane VM and two workers (one VM, one LXC) — runs stably with k3s managing scheduling, networking, and DNS across all three nodes via CoreDNS. I now have a vanilla multi-node Kubernetes cluster running in a Ubuntu VM and an LXC container and accessible from my machine. It’s got nothing deployed inside it yet, but that’s easily fixed.... see u in part 2. *Built on Proxmox VE with k3s v1.34.6+k3s1 — Debian Trixie LXC — Ubuntu VM nodes Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse