Setting Up an NVIDIA GPU for an LXC Container on Proxmox VE

All commands in this section are run on the Proxmox host.

Important: With Secure Boot enabled, the NVIDIA driver usually doesn't load its kernel module. Disable Secure Boot in the BIOS before installing the driver. You can check the status with apt install mokutil -y followed by mokutil --sb-state. Signing the module is not covered in this guide.

1.1 Update Proxmox

Log in to the Proxmox web interface and open the host's shell. Alternatively, you can connect via SSH.

First check the installed version:

pveversion -v

Then update the package lists and install available updates:

apt update && apt full-upgrade -y

If a kernel update was installed, a reboot is required so that the new kernel is actually running.

1.2 Detect the NVIDIA GPU

Before installing any NVIDIA drivers, we first check
whether Proxmox detects the graphics card at all.

lspci | grep -i nvidia

If the RTX 3060 is detected correctly, you should see output similar to this:

01:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA106 High Definition Audio Controller (rev a1)

This tells us the graphics card is detected at the PCIe level.

1.3 Install the required packages

Before installing the NVIDIA driver, we first set up the required tools. These include the compiler tools for building the NVIDIA kernel module, DKMS and a few utilities we'll need later in the installation.

Install the packages with:

apt install build-essential dkms pciutils wget curl nano mc btop nvtop

The individual packages do the following:

• build-essential – contains compilers and tools needed to build the NVIDIA kernel module.
• dkms – manages kernel modules and automatically builds them for the running kernel when needed.
• pciutils – provides lspci, among others, which we use to check the detected NVIDIA GPU.
• wget – used to download files.
• curl – also used for downloads and retrieving information.
• nano – simple editor for editing configuration files.
• mc – convenient file manager for the terminal.
• btop – shows CPU load, RAM usage, running processes and system resources.
• nvtop – shows GPU load, VRAM usage, temperatures and running GPU processes in real time.

Then we check which kernel the Proxmox host is currently using:

uname -r

To build the NVIDIA kernel module, the matching kernel headers must be installed. On Proxmox VE 9 we use the header meta package of the current Proxmox 7.0 kernel branch:

apt install proxmox-headers-7.0

This installs the header version matching the installed Proxmox kernel.

Finally, we check whether the required build directory exists:

ls -ld /lib/modules/$(uname -r)/build

If a valid directory is shown here, the required kernel headers are present and the prerequisites for the NVIDIA driver installation are met.

If the directory is missing, don't run the NVIDIA driver installation yet. First the kernel headers for the currently running Proxmox kernel must be installed correctly.

1.4 Download the NVIDIA driver

Download the matching Linux driver for your NVIDIA RTX graphics card from the official NVIDIA website:

NVIDIA – download drivers

For the RTX 3060, in this example we use the production branch version 595.99.02.

Then open the NVIDIA driver page in the browser and copy the direct download link of the .run file.
Paste this link into the following command:

wget

Example for the version used:

wget https://download.nvidia.com/XFree86/Linux-x86_64/595.99.02/NVIDIA-Linux-x86_64-595.99.02.run

Then check whether the file was downloaded successfully:

ls -lh NVIDIA-Linux-x86_64-*.run

Make the downloaded file executable. Ideally, use the exact file name:

chmod +x NV*

Note: chmod +x NV* works if the downloaded NVIDIA .run file is the only file in /root starting with NV. If several files there start with NV, use the full file name instead.

1.5 Install the NVIDIA driver with DKMS

Start the installer:

./NV* --dkms

Several prompts are shown during the installation.

Select the kernel module:

→ NVIDIA proprietary

The proprietary kernel module is the classic NVIDIA kernel driver. The MIT/GPL variant is NVIDIA's open kernel module and is used especially with newer GPU generations (RTX 50xx etc.) and current NVIDIA drivers. For our setup with the RTX 3060 we use the proprietary variant.

Further prompts:

• Disable Nouveau → Yes
• Continue the installation even though Nouveau is currently loaded → Continue installation
• 32-bit compatibility libraries → OK; these aren't needed
• Register the kernel module with DKMS → Yes
• Rebuild initramfs → Rebuild initramfs
• Run nvidia-xconfig → No

The --dkms option ensures that the NVIDIA kernel module is managed via DKMS. This allows the module to be built automatically for the new kernel during later kernel updates, provided the matching kernel headers are present.

Since Nouveau is still loaded in the running kernel at the time of installation, the Proxmox host must be rebooted afterwards. Disabling Nouveau only takes effect after the reboot.

1.6 Set up automatic NVIDIA device initialization

On headless servers, the NVIDIA driver only creates the device files (/dev/nvidiactl etc.) on first access. So that Proxmox LXC containers don't abort at boot with TASK ERROR: Device /dev/nvidiactl does not exist, we create a small systemd service. It runs nvidia-smi before Proxmox starts the containers:

Register the kernel modules:

cat <<EOF > /etc/modules-load.d/nvidia.conf
nvidia
nvidia_uvm
EOF

Create the systemd service:

cat <<EOF > /etc/systemd/system/nvidia-init.service
[Unit]
Description=Initialize NVIDIA Device Nodes
DefaultDependencies=no
After=systemd-modules-load.service
Before=pve-guests.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi

[Install]
WantedBy=multi-user.target
EOF

Enable the service:

systemctl daemon-reload
systemctl enable nvidia-init.service

1.7 Check the driver

After the reboot, open the host shell again and check:

nvidia-smi

Then:

lsmod | grep nvidia

If nvidia-smi shows the RTX 3060 with 12 GB of VRAM and the installed driver version, and the NVIDIA kernel module is loaded, the host is ready for the next step.

Before we can pass the RTX 3060 through to an LXC container, we first need a suitable container. In this example we create the container with ID 101. Later, Docker and llama.cpp will run in it.

2.1 Download the Debian template

In the Proxmox web interface, go to:
local (pve) → CT Templates → Templates
Search for Debian 13 and download the current Debian 13 template.

2.2 Create the LXC container

Then click Create CT at the top right.
Use the following settings for our test setup:

• General
  CT ID: 101
  Hostname: llamacpp
  Password: your own secure password
  Unprivileged container: enabled
  Nesting: enabled
• Template
  Select Debian 13
• Disk
  Storage: select the desired Proxmox storage
  Size: 40 GB
• CPU
  Cores: 6
• Memory
  RAM: 8192 MB
  Swap: 0 MB
• Network
  IPv4: 192.168.178.134/24
  Gateway (IPv4): 192.168.178.1
  Bridge: usually vmbr0

The remaining settings can stay at their defaults for now.

Then click Finish to create the container.

The chosen resources are meant as a starting point. Storage, CPU cores and memory can be adjusted later if needed. Keep in mind that larger local AI models above all need enough RAM or GPU memory.

2.3 Start the container

First start the container via the Proxmox web interface.
Then open the shell of the Proxmox host and copy the previously downloaded NVIDIA driver into CT 101:

pct push 101 /root/NV* /root/NV*

Then switch directly into the container:

pct enter 101

You are now in the shell of CT 101.
First update Debian:

apt update && apt full-upgrade -y

Then check whether the NVIDIA installer is present:

ls -lh /root/NV*

If the file is listed, the container is prepared for the next steps.
Then leave the container again:

exit

You are now back in the shell of the Proxmox host.

Note: At this point the RTX 3060 is not yet available in the container. The NVIDIA devices are only passed through from the Proxmox host to the LXC container in the next step.

From here on we work with CT 101.

3.1 Stop container 101

Go to the shell of the Proxmox host:

pct stop 101

3.2 Check the existing NVIDIA devices

nvidia-smi
ls -l /dev/nvidia*

The device files can vary depending on the driver and GPU. Typically there are, among others:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools

possibly /dev/nvidia-modeset

Only use devices that actually exist on your host.

3.3 Add the devices to the container configuration

The NVIDIA devices have to be added to the configuration of CT 101. There are two ways to do this:
via the Proxmox web interface or via the CLI (command line)

In this guide we use the web interface, because the devices can be added conveniently there. If you prefer working on the command line, you can edit the configuration file directly instead.

Via the Proxmox web interface (our approach)

Select CT 101 in the Proxmox web interface.
Go to:

Resources → Add → Device Passthrough

Add the NVIDIA devices you checked on the host one by one:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools

Apply the settings and save the changes.

Via the CLI (alternative)

Open the shell of the Proxmox host and edit the configuration file of CT 101:

nano /etc/pve/lxc/101.conf

Add the required device mappings at the end of the file:

dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-uvm
dev3: /dev/nvidia-uvm-tools

Only use devices that actually exist on your host. If further devices are needed, add them accordingly.

Save the file with Ctrl + O, confirm with Enter and exit the editor with Ctrl + X.

Important: Use only one of the two methods. If you've already added the devices via the web interface, you don't need to add them via the CLI as well.

3.4 Start the container

pct start 101

Open the container shell:

pct enter 101

Check whether the devices are visible in the container:

ls -l /dev/nvidia*

If the devices are missing, first check the host devices and the CT configuration. A successful start alone doesn't prove that the GPU is usable.

Now we install the userspace components of the NVIDIA driver in the container. The kernel modules are explicitly not installed.

4.1 Make the NVIDIA installer executable

Make the existing NVIDIA installer executable:

chmod +x /root/NV*

4.2 Install the NVIDIA driver without a kernel module

Then start the installer without installing an NVIDIA kernel module:

/root/NV* --no-kernel-module

Confirm the following prompts during installation:

• Kernel module type: NVIDIA Proprietary → select
• Note about --no-kernel-modules: OK
• X library paths: OK
• 32-bit compatibility libraries: No
• Vulkan loader missing: OK
• Run nvidia-xconfig: No

This installs the required NVIDIA userspace components in the container. The NVIDIA kernel module itself keeps running on the Proxmox host.

4.3 Test GPU access in the container

After a successful installation, we check whether the container can access the RTX 3060:

nvidia-smi

If the RTX 3060 and the installed NVIDIA driver version are shown, the container can access the GPU.

4.4 Reboot the Proxmox host and run a final test

First leave the container shell:

exit

Then reboot the entire Proxmox host:

reboot

Wait until Proxmox has fully started again and reconnect to the shell of the Proxmox host.

First check on the host whether the NVIDIA GPU is detected:

nvidia-smi

Then we check whether the NVIDIA devices are present:

ls -l /dev/nvidia*

After that, start the container and enter it:

pct start 101
pct enter 101

Run the final GPU test in the container:

nvidia-smi

If the RTX 3060 is shown on the host and in the container even after the reboot, the GPU has been set up successfully for CT 101 and is available for the next steps with Docker and llama.cpp.

This step is optional. With a single RTX 3060, a power limit isn't necessary for our setup. The card can be run with its normal default settings.

In a multi-GPU system, on the other hand, it can make sense to limit the maximum power draw somewhat. Especially in a local AI server, GPUs can run under high load for long periods. A moderate limit can help reduce power consumption, waste heat and continuous stress.

The goal with such a server is not to squeeze out the last percent of performance, but to strike a good balance between performance, consumption, temperatures and long-term use of the hardware.

Note: A power limit doesn't guarantee a longer GPU lifespan. It does, however, reduce the maximum power draw and thus, among other things, heat generation and electrical stress.

Important: You run the commands in this step on the Proxmox host, not in the container. After step 4 you are still in the shell of CT 101. Leave it with exit.

5.1 Check the permitted power limit

Before setting a power limit, first check which range your GPU supports:

nvidia-smi -q -d POWER

In the output, look for the lines Default Power Limit, Min Power Limit and Max Power Limit. Your value must be between Min and Max. As a starting point, many people choose about 75 to 85 % of the default value. This is a rule of thumb, not a guaranteed value. The values differ depending on the GPU model and manufacturer, so never copy a value from another tutorial.

5.2 Test the power limit once

First set the limit only for the current session (replace WATT with your value):

nvidia-smi -pl WATT

Check the result with nvidia-smi. In the Pwr:Usage/Cap column, the current limit is shown to the right of the slash. It should match your value. After a reboot, the default value applies again.

With several GPUs, the command without further options applies to all cards. For a single card, use -i with the number from the nvidia-smi output:

nvidia-smi -i 0 -pl WATT

5.3 Set the power limit at system startup

If the limit should apply permanently, add it to root's crontab:

crontab -e

Add the following line and replace WATT with your tested value:

@reboot sleep 10; nvidia-smi -pm 1; nvidia-smi -pl WATT

nvidia-smi -pm 1 enables persistence mode. The sleep 10 gives the system some time after startup to fully provide the GPU. The commands are separated with ; so that the power limit is set even if an earlier command prints a warning.

After a reboot, check with nvidia-smi whether the limit was applied.

Undo: Remove the line with crontab -e and reboot the host. After that, the default value applies again.

From this point on we work inside CT 101. If you're still on the Proxmox host (for example after step 5), switch into the container with pct enter 101.

6.1 Install Docker dependencies

First we install the packages needed to set up the official Docker repository:

apt install ca-certificates curl -y

ca-certificates enables secure connections to HTTPS sources.
curl is used to download the Docker GPG key.

6.2 Set up the official Docker repository

Debian 13 already includes Docker in its own package sources, where it is available as docker.io, among others.

For this guide, however, we use the official Docker repository, through which Docker provides its own Docker Engine and the current Compose plugin.

First we create the directory for the GPG key:

install -m 0755 -d /etc/apt/keyrings

Download the GPG key:

curl -fsSL https://download.docker.com/linux/debian/gpg \
  -o /etc/apt/keyrings/docker.asc

Set the permissions:

chmod a+r /etc/apt/keyrings/docker.asc

Then the official Docker repository is set up:

cat <<EOF > /etc/apt/sources.list.d/docker.sources
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: trixie
Components: stable
Architectures: amd64
Signed-By: /etc/apt/keyrings/docker.asc
EOF

After that we update the package sources:

apt update

This allows Debian to fetch the Docker packages directly from the official Docker repository.

6.3 Install Docker and Docker Compose

Now we install Docker Engine, the Docker CLI, containerd, Buildx and the Docker Compose plugin:

apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Then we check the installed versions:

docker --version
docker compose version

Also check whether the Docker service is running:

systemctl status docker

If Docker isn't running, the service can be started and enabled for automatic start at the same time with the following command:

systemctl enable --now docker

6.4 Test Docker

Before we continue with the NVIDIA Container Toolkit, we first test the basic Docker installation.
For this we use the official Docker test container:

docker run hello-world

Docker downloads the small test image and starts a container.
If the message

Hello from Docker!

appears at the end, Docker basically works.

In the next step we set up the NVIDIA Container Toolkit.

So that Docker can access the LXC container's NVIDIA GPU, we install the NVIDIA Container Toolkit. It connects Docker to the NVIDIA GPU we've already set up.

We continue working in CT 101.

7.1 Install the prerequisites

apt update
apt install ca-certificates curl gnupg2 -y

7.2 Set up the NVIDIA package source
Create the keyring directory:

install -m 0755 -d /etc/apt/keyrings

Download the NVIDIA GPG key:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | gpg --dearmor -o /etc/apt/keyrings/nvidia-container-toolkit-keyring.gpg

Add the NVIDIA package source:

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  > /etc/apt/sources.list.d/nvidia-container-toolkit.list

Then:

apt update

7.3 Install the NVIDIA Container Toolkit

apt update && apt install nvidia-container-toolkit -y

Check the installation:

nvidia-ctk --version

7.4 Configure Docker for NVIDIA

Now we configure Docker to use the NVIDIA Container Runtime:

nvidia-ctk runtime configure --runtime=docker

Then we disable cgroup management of the NVIDIA runtime. This adjustment is necessary for our unprivileged LXC container, because the NVIDIA runtime can't set its own cgroup device rules inside the container.

The existing configuration is adjusted automatically:

sed -i 's/^#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml

Then check the setting:

grep '^no-cgroups' /etc/nvidia-container-runtime/config.toml

Expected output:

no-cgroups = true

Then restart Docker:

systemctl restart docker

Docker is now prepared for the NVIDIA GPU inside our LXC container.

7.5 Test the NVIDIA GPU from Docker

Now we check whether Docker inside CT 101 can actually access the NVIDIA GPU.

Run the following test:

docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi

On the first run, the required CUDA image is downloaded automatically from Docker Hub.

If the setup was successful, nvidia-smi shows the NVIDIA GPU inside the Docker container. On our test system, the RTX 3060 with 12 GB of VRAM and the installed NVIDIA driver 595.99.02 should appear here.

The --rm ensures that the test container is removed automatically after the successful test.

If the GPU is shown, the complete GPU chain works:

Proxmox → LXC → NVIDIA driver → NVIDIA Container Toolkit → Docker → RTX 3060

The NVIDIA GPU is now available to Docker and we can continue with setting up llama.cpp.

Before continuing with the next steps, we set up automatic start for the container and create a full backup.

This gives us a fixed starting point for the following steps. If we want to start over later, the container can be restored from this backup. The NVIDIA, GPU passthrough and Docker setup we've already done then doesn't have to be repeated.

8.1 Enable automatic start

• Select container 101 in the Proxmox web interface.
• Open the Options section.
• Double-click Start at boot, tick the checkbox and confirm with OK.

The container now starts automatically as soon as the Proxmox host has booted. The nvidia-init.service from step 1.6 makes sure the NVIDIA devices are already available by then.

8.2 Create a backup

Open the container's Backup section and click Backup now.

Use the following settings:

Storage: desired backup storage "local"
Compression: ZSTD
Mode: Stop
Notification: Use global settings
Notes: Initial state after NVIDIA GPU, LXC and Docker setup

Then click Backup and wait until the backup has completed successfully.

This gives us a fixed starting point for the following sections.

After updates

Kernel update on the host: DKMS rebuilds the NVIDIA module automatically, provided the kernel headers are present. After the reboot, check on the host:

nvidia-smi
dkms status

Driver update: Host and container must have exactly the same driver version. If only one side is updated, the GPU stops working in the container. Replace VERSION with the new version number:

# On the host
pct stop 101
wget https://download.nvidia.com/XFree86/Linux-x86_64/VERSION/NVIDIA-Linux-x86_64-VERSION.run
chmod +x NVIDIA-Linux-x86_64-VERSION.run
./NVIDIA-Linux-x86_64-VERSION.run --dkms
reboot

# After the reboot, back on the host
pct push 101 /root/NVIDIA-Linux-x86_64-VERSION.run /root/NVIDIA-Linux-x86_64-VERSION.run
pct start 101
pct enter 101

# In the container
chmod +x /root/NVIDIA-Linux-x86_64-VERSION.run
/root/NVIDIA-Linux-x86_64-VERSION.run --no-kernel-module
nvidia-smi

Delete old .run files in /root, or use the full file name as shown here. Otherwise NV* matches several files.

Common problems

Host: "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver"
The kernel module isn't loaded. Check on the host:

dkms status
lsmod | grep nvidia
dmesg | grep -i nvidia

Typical causes: missing kernel headers, a forgotten reboot after installation or Secure Boot being enabled.

Container doesn't start: "Device /dev/nvidiactl does not exist"
The service from step 1.6 isn't running or runs too late. Check on the host:

systemctl status nvidia-init.service
ls -l /dev/nvidia*

Container: "Failed to initialize NVML: Driver/library version mismatch"
Host and container have different driver versions. Compare:

# On the host
cat /proc/driver/nvidia/version

# In the container
ls /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.*

The version number must match. Otherwise, reinstall the matching .run file in the container.

Devices missing in the container (/dev/nvidia*)
Check whether the devices exist on the host and are listed in /etc/pve/lxc/101.conf. If /dev/nvidia-uvm is missing, check on the host with lsmod | grep nvidia_uvm whether the module is loaded.

The Docker GPU test fails
First test nvidia-smi directly in the container. If that works, check:

grep '^no-cgroups' /etc/nvidia-container-runtime/config.toml
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Afterwards, the configuration file must still contain no-cgroups = true (step 7.4).

The container doesn't start automatically after a host reboot
Check whether Options → Start at boot is ticked (step 8.1) and whether nvidia-init.service is running.

With this guide we've laid the foundation for a local AI server step by step.

The Proxmox VE host was prepared for an NVIDIA GPU, and the graphics card was set up for use in an unprivileged LXC container. Then Docker and the NVIDIA Container Toolkit environment were set up in the container, and the GPU was successfully made available inside the container for GPU-accelerated applications.

This gives us a clean foundation for the next steps.

The advantages of this architecture:

• Performance: The LXC container accesses the host's NVIDIA GPU directly. Compared to a full virtual machine, this adds very little overhead.
• Isolation: The container stays unprivileged. At the same time, applications inside the container can use the provided GPU for compute-intensive AI workloads.
• Flexibility: Docker makes it possible to run different AI applications and models independently of each other and to swap or extend them easily later.

Outlook: multi-GPU

The setup shown here can basically also be extended to systems with several NVIDIA GPUs.
The NVIDIA driver numbers the GPUs sequentially:

/dev/nvidia0 → first GPU
/dev/nvidia1 → second GPU
/dev/nvidia2 → third GPU

Applications such as llama.cpp can use several GPUs to spread larger models across the available VRAM of several graphics cards. How exactly the GPUs are used depends on the application and the model.

An NVIDIA GPU in another computer, however, can't simply be attached over the network as /dev/nvidia1. Distributed inference requires other methods and corresponding software solutions.

What's next?
The Proxmox host and the LXC container are now prepared for the next local AI workloads.
The next step is to provide the actual AI software.
My next post is about setting up llama.cpp and running various GGUF models directly in the prepared LXC container.

Continue with part 3: llama.cpp with Docker and CUDA