Now we start llama.cpp for the first time with a specific GGUF model.
For this setup we use:
unsloth/Muse-Glimmer-30B-GGUF:UD-IQ2_XXS
Why do we choose IQ2_XXS?
Our test platform only has an RTX 3060 with 12 GB of VRAM. With larger language models, it's therefore not only the size of the model itself that matters, but also the additional memory needed for context, KV cache and the running inference process.
As UD-IQ2_XXS, the chosen 30B model is already quantized to around 10.7 GB, so that it basically fits into the available VRAM of a 12 GB GPU.
The low quantization is therefore a deliberate decision here:
12 GB VRAM → largest possible model → smallest possible quantization
This lets us run a considerably larger model on the RTX 3060 than would be possible with, for example, a Q4 quantization of this model size.
Important: The size of the GGUF file is not the same as the total GPU memory required. During execution, llama.cpp needs additional memory for the context and other runtime data. So a 10.7 GB GGUF file doesn't automatically mean that 10.7 GB of VRAM is enough.
Different hardware, different quantization
The choice of quantization therefore depends heavily on your GPU and the available VRAM.
With a single RTX 3090 with 24 GB of VRAM, there's considerably more headroom. With comparable model sizes, less heavily quantized variants can be used as well.
With two RTX 3060s with 12 GB of VRAM each, the available GPU memory can be split across several GPUs. This also opens up possibilities for larger models or higher-quality quantizations.
That's exactly one of the advantages of llama.cpp. We're not tied to a single model or a specific quantization. Depending on the available hardware, different GGUF variants can be used.
For this guide, however, we deliberately choose the variant that fits our 12 GB test platform.
Load the model via Hugging Face
llama.cpp can fetch models directly from Hugging Face. For this we use the -hf parameter.
To do this, we open the docker-compose.yml we created earlier again:
nano /opt/llamacpp/docker-compose.yml
In the existing Compose file, we add the following command block between the environment and deploy sections:
command: >
-hf unsloth/Muse-Glimmer-30B-GGUF:UD-IQ2_XXS
--host 0.0.0.0
--port 8080
--n-gpu-layers 99
The Compose file now additionally contains:
services:
llamacpp:
image: ghcr.io/ggml-org/llama.cpp:server-cuda
container_name: llamacpp
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /opt/llamacpp/models/huggingface:/root/.cache/huggingface
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
command: >
-hf unsloth/Muse-Glimmer-30B-GGUF:UD-IQ2_XXS
--host 0.0.0.0
--port 8080
--n-gpu-layers 99
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
What do the additional parameters do?
-hf unsloth/Muse-Glimmer-30B-GGUF:UD-IQ2_XXS
tells llama.cpp to use the specified model directly from the Hugging Face repository. The current llama.cpp server documentation supports this way of fetching models directly from Hugging Face.
The model is stored in the Hugging Face cache we set up earlier:
/opt/llamacpp/models/huggingface/
--host 0.0.0.0
The llama.cpp server becomes reachable on all network interfaces of the container.
--port 8080
The llama.cpp server uses port 8080 inside the container. Our Compose configuration already forwards it to the outside.
--n-gpu-layers 99
This tells llama.cpp to offload as many model layers as possible to the GPU.
The value could also be set to 50, for example. Then not all model weights would be on the GPU, and part of them would have to be processed in system RAM. This usually lowers the tokens/s, because accessing VRAM is much faster than accessing the model parts in system RAM.
With 99 we therefore try to run as many model layers as possible on the RTX 3060.
Note: The actual VRAM usage depends, among other things, on the model, the quantization and the context size chosen later. We check the utilization after starting the model.
Start llama.cpp
After adjusting the Compose file, we first check the configuration:
docker compose config
If no error message appears, we start the container:
docker compose up -d
On the first start, Docker first downloads the llama.cpp image. Depending on your internet connection, this can take some time. You can watch the download in a second shell window, for example with btop.
Then we watch the logs:
docker logs -f llamacpp
Press Ctrl + C to leave the log view; the container keeps running.
On the first start, llama.cpp also downloads the model, around 10.7 GB. Then the model is loaded and the GPU is initialized.
You can see whether the RTX 3060 is being used on the Proxmox host (not in the container) with:
nvtop
As soon as the model is loaded and the server message appears in the log, llama.cpp is reachable at http://192.168.178.134:8080.
🔒 Security: With --host 0.0.0.0, anyone in your network can use the server. In a home network that's usually fine. But never expose port 8080 to the internet in your router. Optionally, you can protect the server with a key: add the line --api-key YOUR_KEY (a long, random password) to the command block. Then enter the key in the web interface settings and send it with curl using -H "Authorization: Bearer YOUR_KEY".
Test the server
1. Web interface: Open http://192.168.178.134:8080 in the browser. There you can chat with the model directly.
2. Check the status:
curl http://192.168.178.134:8080/health
As soon as the model is loaded, {"status":"ok"} is returned. While it's still loading, an error message appears.
3. Chat request via the API:
curl http://192.168.178.134:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "Explain in two sentences what GGUF is."}
],
"max_tokens": 200
}'
The answer comes in OpenAI-compatible format. This means you can also use the server with other programs that support an OpenAI API.
Note: This guide is deliberately about a flexible standard llama.cpp setup that can run different GGUF models. In step 6 we test a second model.
