Tools

Tools: Ultimate Guide - Setting Up NVIDIA GPU Passthrough on Ubuntu 24.04 Bare Metal

2026-03-20 0 views admin
Tools: Ultimate Guide - Setting Up NVIDIA GPU Passthrough on Ubuntu 24.04 Bare Metal

Prerequisites: The Bare Metal Foundation

Step 1: Avoid the Ubuntu 24.04 Docker "Snap" Trap 🛑

Step 2: Install the Official Docker Engine

Step 3: Install the NVIDIA Container Toolkit

Step 4: Configure the Docker Runtime (daemon.json)

Step 5: The Bare Metal Verification Test

Bonus: Deploying AI with Docker Compose

Troubleshooting Common Errors

Scale Your AI Infrastructure with GPUYard Deploying large language models (LLMs) or generative AI on a bare-metal dedicated server gives you unmatched performance, zero virtualization overhead, and complete data privacy. However, out of the box, Docker containers are isolated from your host machine's physical hardware. If you run a standard AI container, it simply cannot see your RTX 4090 or A100 GPU. To break this isolation and achieve true Docker GPU passthrough, you need to bridge your container engine with your host’s hardware using the NVIDIA Container Toolkit. In this guide, backed by our experience deploying thousands of AI-ready bare metal servers at GPUYard, we will walk you through the exact steps to securely configure Docker with NVIDIA GPUs on Ubuntu 24.04. Before configuring Docker, your server must recognize its hardware. At GPUYard, our bare-metal servers come pre-provisioned, but you should always verify your host environment: The most common reason developers fail to pass GPUs into Docker on Ubuntu 24.04 is the default installation method. If you installed Docker via the Ubuntu App Center or used snap install docker, GPU passthrough will fail with permission errors. Snap packages use strict AppArmor confinement, preventing Docker from accessing the /dev/nvidia* hardware files on your host. We must remove the Snap version and use the official Docker APT repository. Purge the Snap version: Now, install the unconfined, official Docker Engine directly from Docker’s verified repository. Set up the repository and GPG keys: With a clean Docker engine running, we install the NVIDIA Container Toolkit. This software acts as the critical translation layer between your bare-metal CUDA drivers and your isolated containers. Add NVIDIA's production repository and install the toolkit: The toolkit is installed, but Docker needs to be explicitly instructed to use it. We will use the nvidia-ctk command-line utility to automatically inject the NVIDIA runtime into Docker's configuration file. Expert Tip: You can verify this worked by running cat /etc/docker/daemon.json. You will see "nvidia" listed under the "runtimes" key. Let's prove the isolation barrier is broken. We will spin up an official NVIDIA CUDA container and ask it to read our bare-metal hardware. If successful, the terminal will output your GPU statistics table. Because we used the --gpus all flag, this output proves that your Docker container now has direct, unrestricted access to your physical GPU! 🎉 Running terminal commands is great for testing, but deploying production AI models (like Llama 3 or Stable Diffusion) requires docker-compose.yml. You must use the specific deploy specification to reserve GPU hardware. Here is a template to deploy Ollama with full bare-metal GPU acceleration: Save this as docker-compose.yml and run sudo docker compose up -d. You are now hosting your own private AI. Even on standard Ubuntu 24.04 setups, you might encounter these snags: Error: "could not select device driver with capabilities: [[gpu]]" Error: "Failed to initialize NVML: Driver/library version mismatch" Setting up the software is only half the battle; having the right hardware is what dictates your AI's performance. Cloud VPS environments throttle your VRAM and share your PCI-e lanes. If you want maximum token-per-second generation and uncompromising privacy, you need Bare Metal. Explore GPUYard’s high-performance Dedicated GPU Servers—custom-built for seamless Docker deployments and heavy AI workloads. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Command

Copy

$ -weight: 600;">sudo snap -weight: 500;">remove --purge -weight: 500;">docker -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">remove -weight: 500;">docker -weight: 500;">docker-engine -weight: 500;">docker.io containerd runc -weight: 600;">sudo snap -weight: 500;">remove --purge -weight: 500;">docker -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">remove -weight: 500;">docker -weight: 500;">docker-engine -weight: 500;">docker.io containerd runc -weight: 600;">sudo snap -weight: 500;">remove --purge -weight: 500;">docker -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">remove -weight: 500;">docker -weight: 500;">docker-engine -weight: 500;">docker.io containerd runc -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install ca-certificates -weight: 500;">curl -weight: 600;">sudo -weight: 500;">install -m 0755 -d /etc/-weight: 500;">apt/keyrings -weight: 600;">sudo -weight: 500;">curl -fsSL [https://download.-weight: 500;">docker.com/linux/ubuntu/gpg](https://download.-weight: 500;">docker.com/linux/ubuntu/gpg) -o /etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc -weight: 600;">sudo chmod a+r /etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc] [https://download.-weight: 500;">docker.com/linux/ubuntu](https://download.-weight: 500;">docker.com/linux/ubuntu) \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ -weight: 600;">sudo tee /etc/-weight: 500;">apt/sources.list.d/-weight: 500;">docker.list > /dev/null -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install ca-certificates -weight: 500;">curl -weight: 600;">sudo -weight: 500;">install -m 0755 -d /etc/-weight: 500;">apt/keyrings -weight: 600;">sudo -weight: 500;">curl -fsSL [https://download.-weight: 500;">docker.com/linux/ubuntu/gpg](https://download.-weight: 500;">docker.com/linux/ubuntu/gpg) -o /etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc -weight: 600;">sudo chmod a+r /etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc] [https://download.-weight: 500;">docker.com/linux/ubuntu](https://download.-weight: 500;">docker.com/linux/ubuntu) \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ -weight: 600;">sudo tee /etc/-weight: 500;">apt/sources.list.d/-weight: 500;">docker.list > /dev/null -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install ca-certificates -weight: 500;">curl -weight: 600;">sudo -weight: 500;">install -m 0755 -d /etc/-weight: 500;">apt/keyrings -weight: 600;">sudo -weight: 500;">curl -fsSL [https://download.-weight: 500;">docker.com/linux/ubuntu/gpg](https://download.-weight: 500;">docker.com/linux/ubuntu/gpg) -o /etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc -weight: 600;">sudo chmod a+r /etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/-weight: 500;">apt/keyrings/-weight: 500;">docker.asc] [https://download.-weight: 500;">docker.com/linux/ubuntu](https://download.-weight: 500;">docker.com/linux/ubuntu) \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ -weight: 600;">sudo tee /etc/-weight: 500;">apt/sources.list.d/-weight: 500;">docker.list > /dev/null -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install -weight: 500;">docker-ce -weight: 500;">docker-ce-cli containerd.io -weight: 500;">docker-buildx-plugin -weight: 500;">docker-compose-plugin -y -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install -weight: 500;">docker-ce -weight: 500;">docker-ce-cli containerd.io -weight: 500;">docker-buildx-plugin -weight: 500;">docker-compose-plugin -y -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install -weight: 500;">docker-ce -weight: 500;">docker-ce-cli containerd.io -weight: 500;">docker-buildx-plugin -weight: 500;">docker-compose-plugin -y -weight: 500;">curl -fsSL [https://nvidia.github.io/libnvidia-container/gpgkey](https://nvidia.github.io/libnvidia-container/gpgkey) | -weight: 600;">sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && -weight: 500;">curl -s -L [https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list](https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list) | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ -weight: 600;">sudo tee /etc/-weight: 500;">apt/sources.list.d/nvidia-container-toolkit.list -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install -y nvidia-container-toolkit -weight: 500;">curl -fsSL [https://nvidia.github.io/libnvidia-container/gpgkey](https://nvidia.github.io/libnvidia-container/gpgkey) | -weight: 600;">sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && -weight: 500;">curl -s -L [https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list](https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list) | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ -weight: 600;">sudo tee /etc/-weight: 500;">apt/sources.list.d/nvidia-container-toolkit.list -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install -y nvidia-container-toolkit -weight: 500;">curl -fsSL [https://nvidia.github.io/libnvidia-container/gpgkey](https://nvidia.github.io/libnvidia-container/gpgkey) | -weight: 600;">sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && -weight: 500;">curl -s -L [https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list](https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list) | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ -weight: 600;">sudo tee /etc/-weight: 500;">apt/sources.list.d/nvidia-container-toolkit.list -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">update -weight: 600;">sudo -weight: 500;">apt-get -weight: 500;">install -y nvidia-container-toolkit -weight: 600;">sudo nvidia-ctk runtime configure --runtime=-weight: 500;">docker -weight: 600;">sudo -weight: 500;">systemctl -weight: 500;">restart -weight: 500;">docker -weight: 600;">sudo nvidia-ctk runtime configure --runtime=-weight: 500;">docker -weight: 600;">sudo -weight: 500;">systemctl -weight: 500;">restart -weight: 500;">docker -weight: 600;">sudo nvidia-ctk runtime configure --runtime=-weight: 500;">docker -weight: 600;">sudo -weight: 500;">systemctl -weight: 500;">restart -weight: 500;">docker -weight: 600;">sudo -weight: 500;">docker run --rm --gpus all nvidia/cuda:12.2.2-base-ubuntu24.04 nvidia-smi -weight: 600;">sudo -weight: 500;">docker run --rm --gpus all nvidia/cuda:12.2.2-base-ubuntu24.04 nvidia-smi -weight: 600;">sudo -weight: 500;">docker run --rm --gpus all nvidia/cuda:12.2.2-base-ubuntu24.04 nvidia-smi services: ollama-ai: image: ollama/ollama:latest container_name: gpuyard-ollama -weight: 500;">restart: always ports: - "11434:11434" volumes: - ./ollama_data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all # Passes all available GPUs to the container capabilities: [gpu] services: ollama-ai: image: ollama/ollama:latest container_name: gpuyard-ollama -weight: 500;">restart: always ports: - "11434:11434" volumes: - ./ollama_data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all # Passes all available GPUs to the container capabilities: [gpu] services: ollama-ai: image: ollama/ollama:latest container_name: gpuyard-ollama -weight: 500;">restart: always ports: - "11434:11434" volumes: - ./ollama_data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all # Passes all available GPUs to the container capabilities: [gpu] - A Dedicated GPU Server: Running Ubuntu 24.04 LTS. - Root or Sudo Access: Required for package installation. - NVIDIA Drivers Installed: Verify this by running nvidia-smi in your terminal. You should see a table displaying your GPU model and CUDA version. (If you see "command not found," -weight: 500;">install the proprietary NVIDIA drivers first). - Error: "could not select device driver with capabilities: [[gpu]]" Cause: Docker isn't aware of the NVIDIA runtime. Fix: You likely forgot to -weight: 500;">restart the Docker daemon in Step 4. Run -weight: 600;">sudo -weight: 500;">systemctl -weight: 500;">restart -weight: 500;">docker. - Cause: Docker isn't aware of the NVIDIA runtime. - Fix: You likely forgot to -weight: 500;">restart the Docker daemon in Step 4. Run -weight: 600;">sudo -weight: 500;">systemctl -weight: 500;">restart -weight: 500;">docker. - Error: "Failed to initialize NVML: Driver/library version mismatch" Cause: Your host system updated the NVIDIA Linux kernel drivers in the background, but the old driver is still loaded in memory. Fix: A simple bare-metal server reboot (-weight: 600;">sudo reboot) will align the kernel modules. - Cause: Your host system updated the NVIDIA Linux kernel drivers in the background, but the old driver is still loaded in memory. - Fix: A simple bare-metal server reboot (-weight: 600;">sudo reboot) will align the kernel modules. - Cause: Docker isn't aware of the NVIDIA runtime. - Fix: You likely forgot to -weight: 500;">restart the Docker daemon in Step 4. Run -weight: 600;">sudo -weight: 500;">systemctl -weight: 500;">restart -weight: 500;">docker. - Cause: Your host system updated the NVIDIA Linux kernel drivers in the background, but the old driver is still loaded in memory. - Fix: A simple bare-metal server reboot (-weight: 600;">sudo reboot) will align the kernel modules.

🏷️ Tags

toolsutilitiessecurity toolsultimateguidesettingnvidiapassthroughubuntumetal

More from Tools

Tools: Gas-Aware Trading: Execute Only When Gas Is Cheap (2026)

Tools: Gas-Aware Trading: Execute Only When Gas Is Cheap (2026)

2026-03-30 0
Tools: Grafana k6 Has a Free API That Load Tests Your APIs With JavaScript - Full Analysis

Tools: Grafana k6 Has a Free API That Load Tests Your APIs With JavaScript - Full Analysis

2026-03-30 0
Tools: Caddy Has a Free API That Gives You Automatic HTTPS With Zero Configuration (2026)

Tools: Caddy Has a Free API That Gives You Automatic HTTPS With Zero Configuration (2026)

2026-03-30 0
Tools: Fly.io Has a Free API That Deploys Docker Apps Globally With Edge Hosting (2026)

Tools: Fly.io Has a Free API That Deploys Docker Apps Globally With Edge Hosting (2026)

2026-03-30 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views