llama-server(1)
Start an inference server for Large Language Models using llama.cpp with HTTP API endpoints.
Synopsis
llama-server [OPTION]... --model FILEDescription
llama-server is a high-performance inference server built on llama.cpp that exposes LLM capabilities through RESTful HTTP endpoints. It loads a model file and serves text completion, chat, and embedding requests with optional GPU acceleration.
The server listens on a configurable host and port, supporting concurrent requests with adjustable context windows, batch sizes, and sampling parameters. It's commonly used to integrate open-source models like Llama, Mistral, and Phi into applications without requiring Python or heavy frameworks.
Common options
| Flag | What it does |
|---|---|
-m, --model FILE | Path to the model file (GGUF format); required |
-ngl, --n-gpu-layers INT | Number of layers to offload to GPU for acceleration; 0 = CPU only |
-c, --ctx-size INT | Context window size in tokens; default 512 |
-b, --batch-size INT | Batch size for prompt processing; default 512 |
--host STRING | Bind server to this host address; default 127.0.0.1 |
-p, --port INT | Listen on this TCP port; default 8000 |
-n, --predict INT | Maximum tokens to generate per request; default 128 |
-nthreads, --threads INT | Number of CPU threads to use; default auto-detect |
--alias STRING | Model name alias returned in API responses |
-ts, --tensor-split FLOATS | Split tensor computation across GPUs by ratio (e.g., 3,1 for 75/25 split) |
--lora FILE | Path to LoRA adapter file for model fine-tuning |
--verbose | Enable verbose logging output |
Examples
Start server on port 8000 with Llama 2 7B model, offloading 35 GPU layers
llama-server -m ~/models/llama2-7b.gguf -ngl 35 -p 8000Serve Mistral model with 2048-token context on all network interfaces, port 5000
llama-server -m mistral-7b.gguf -c 2048 -b 256 --host 0.0.0.0 -p 5000Send a completion request to the running server via HTTP POST
curl http://localhost:8000/completion -d '{"prompt":"Hello","n_predict":50}'Start server using 8 CPU threads, generate up to 256 tokens, with debug output
llama-server -m model.gguf -nthreads 8 -n 256 --verboseSplit inference across 2 GPUs with LoRA adapter loaded for domain-specific tasks
llama-server -m model.gguf -ts 2,1 --lora adapter.ggufRun server with custom model alias 'gpt-3.5' for compatibility with LLM clients
llama-server -m model.gguf --alias gpt-3.5 -p 8080