Install and Use llama.cpp

llama.cpp lets you run large language models locally on consumer hardware—CPU-only if needed, with optional GPU acceleration via CUDA, ROCm, or Metal. It uses GGUF-format quantized models, which compress model weights to fit in less RAM without catastrophic quality loss. This guide covers building from source, loading a GGUF model, offloading layers to a GPU, and running the built-in HTTP server for OpenAI-compatible API access.

Prerequisites and Dependencies

You need a C++17 compiler, CMake 3.14+, and git. GPU acceleration requires the appropriate SDK installed first.

Debian/Ubuntu

sudo apt update
sudo apt install -y git cmake build-essential

Fedora/RHEL family

sudo dnf install -y git cmake gcc-c++ make

Arch

sudo pacman -S --needed git cmake base-devel

Clone the Repository

Always build from the main branch or a tagged release. The project moves fast; tagged releases are more stable for production use.

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

To pin to a specific release instead of the bleeding edge:

git checkout $(git tag --sort=-v:refname | grep '^b[0-9]' | head -1)

Build with CMake

CMake is the supported build system. The build flags you pass here determine which backends are compiled in—you cannot add GPU support after the fact without rebuilding.

CPU-only build (any machine)

cmake -B build
cmake --build build --config Release -j$(nproc)

CUDA build (NVIDIA GPUs)

Requires the NVIDIA CUDA Toolkit (12.x recommended). Install it from your distro's repositories or directly from NVIDIA before running this.

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

ROCm build (AMD GPUs)

Requires ROCm 5.6+ installed and ROCM_PATH set correctly (typically /opt/rocm).

cmake -B build -DGGML_HIPBLAS=ON
cmake --build build --config Release -j$(nproc)

Vulkan build (cross-vendor GPU fallback)

sudo apt install -y libvulkan-dev glslc   # Debian/Ubuntu
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

Compiled binaries land in build/bin/. Optionally install system-wide:

sudo cmake --install build --prefix /usr/local

Get a GGUF Model

GGUF is the binary format llama.cpp uses. Hugging Face hosts thousands of pre-quantized GGUF files. The filename encodes the quantization level: Q4_K_M is a solid general-purpose choice (4-bit, good quality/size trade-off). Q8_0 is higher quality but larger; Q2_K is tiny but noticeably degraded.

Install the Hugging Face CLI to download reliably:

pip install huggingface_hub

Download a model—here, Llama 3.2 3B Instruct in Q4_K_M quantization as an example:

huggingface-cli download \
  bartowski/Llama-3.2-3B-Instruct-GGUF \
  Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --local-dir ~/models

Any GGUF model from a trusted source works. Check the model card for context length and memory requirements before downloading multi-gigabyte files.

Run Inference from the CLI

The llama-cli binary runs a single prompt and exits. It is useful for quick tests.

./build/bin/llama-cli \
  --model ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --prompt "Explain the difference between a process and a thread in one paragraph." \
  --n-predict 256 \
  --ctx-size 4096

Key flags:

• --n-predict: maximum tokens to generate
• --ctx-size: context window size in tokens (must not exceed the model's training context)
• --temp: sampling temperature (0.0 = greedy/deterministic, 0.7 = default)
• --threads: CPU threads for inference (defaults to physical core count)

GPU Layer Offloading

GPU offloading works by moving transformer layers from RAM to VRAM. The flag --n-gpu-layers (short: -ngl) controls how many layers are offloaded. Set it to a very large number (e.g., 999) to offload everything that fits; llama.cpp caps it at the actual layer count automatically.

./build/bin/llama-cli \
  --model ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --ctx-size 4096 \
  --prompt "Write a haiku about memory bandwidth."

To find a safe value when VRAM is limited, start low (e.g., -ngl 10) and increase until you see out-of-memory errors, then back off. Partial offload still accelerates inference because fewer CPU↔GPU data transfers are needed.

Verify the GPU is being used by checking startup output—lines like ggml_cuda_init: found N devices and layer assignment messages confirm CUDA is active. A quick external check:

watch -n1 nvidia-smi   # NVIDIA
watch -n1 rocm-smi     # AMD ROCm

Run the HTTP Server

llama-server exposes an OpenAI-compatible REST API. Any tool that supports the OpenAI chat completions API—Open WebUI, shell scripts using curl, Python with openai SDK—can point at it.

./build/bin/llama-server \
  --model ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --ctx-size 8192 \
  --host 127.0.0.1 \
  --port 8080 \
  --parallel 2

--parallel sets the number of concurrent request slots. More slots consume proportionally more VRAM/RAM. Keep it at 1 unless you expect concurrent users.

Test the API

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "What is the capital of Finland?"}],
    "max_tokens": 64
  }'

The server also serves a built-in web chat UI at http://127.0.0.1:8080 in a browser.

Run the server as a systemd service

Create a unit file so the server starts on boot:

sudo tee /etc/systemd/system/llama-server.service <<'EOF'
[Unit]
Description=llama.cpp HTTP inference server
After=network.target

[Service]
Type=simple
User=YOUR_USERNAME
ExecStart=/usr/local/bin/llama-server \
  --model /home/YOUR_USERNAME/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --ctx-size 8192 \
  --host 127.0.0.1 \
  --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
sudo systemctl status llama-server

Verification

After a successful build and server start, confirm everything is working end-to-end:

# Check the server is listening
ss -tlnp | grep 8080

# Hit the models endpoint
curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool

Expected output from ss will show a LISTEN entry on port 8080. The models endpoint returns a JSON list with at least one entry.

Troubleshooting

• CUDA not detected at runtime: Confirm nvcc --version works and that libcuda.so is on the library path (ldconfig -p | grep cuda). Rebuild with -DGGML_CUDA=ON explicitly.
• Out of memory (CUDA error 2): Reduce --n-gpu-layers or --ctx-size. Context memory scales quadratically—halving the context roughly quarters its VRAM cost.
• Slow CPU inference: Make sure AVX2 is available (grep avx2 /proc/cpuinfo). The build auto-detects SIMD but you can force it: -DGGML_AVX2=ON.
• Model format errors: llama.cpp only reads GGUF. Older GGML/bin files must be converted using convert_hf_to_gguf.py in the repo. Ensure the script's Python dependencies are met: pip install -r requirements.txt.
• Server hangs under load: Increase --parallel cautiously or reduce --ctx-size to free capacity. Check journalctl -u llama-server -f for OOM kills.