llama-cli(1)
Command-line interface for running Large Language Models locally using llama.cpp.
Synopsis
llama-cli [OPTIONS] -m <MODEL_FILE> [-p <PROMPT>]Description
llama-cli is a lightweight command-line tool for running quantized Large Language Models (LLMs) locally on CPU and GPU. Built on llama.cpp, it enables inference of models like Llama, Mistral, and other GGUF-format models without requiring heavy ML frameworks.
The tool accepts text prompts and generates completions using the specified model file. It supports various sampling strategies, context sizes, and output formatting options for flexible text generation tasks.
Common options
| Flag | What it does |
|---|---|
-m, --model <FILE> | Path to the GGUF model file (required) |
-p, --prompt <TEXT> | Input prompt to generate completion from |
-n, --n-predict <NUM> | Maximum number of tokens to generate (default: 128) |
-c, --ctx-size <NUM> | Size of the prompt context window (default: 512) |
-t, --threads <NUM> | Number of CPU threads to use for inference |
--temp <FLOAT> | Temperature for sampling (0.0-2.0, higher=more creative) |
-b, --batch-size <NUM> | Batch size for prompt processing (default: 8) |
--top-k <NUM> | Keep top K tokens by probability (default: 40) |
--top-p <FLOAT> | Nucleus sampling threshold (default: 0.9) |
-ngl, --n-gpu-layers <NUM> | Number of layers to offload to GPU (0=CPU only) |
-e, --escape | Escape special characters in output |
--verbose | Print verbose debug information |
Examples
Generate 100 tokens of completion for the given prompt using the specified model
llama-cli -m ./model.gguf -p 'What is AI?' -n 100Generate creative text with higher temperature (0.8) for more varied outputs
llama-cli -m ./mistral-7b.gguf -p 'Write a poem about coffee' -n 256 --temp 0.8Use larger context window (2048), offload 20 GPU layers, and 8 CPU threads
llama-cli -m ./model.gguf -c 2048 -ngl 20 -t 8 -p 'Explain quantum computing'Use nucleus sampling (top-p) for more coherent text generation
llama-cli -m ./model.gguf -p 'Translate to French: Hello' --top-p 0.95 -n 50Pipe text input directly to llama-cli from stdin
echo 'Summarize: AI is transforming industries' | llama-cli -m ./model.gguf -n 150Use few-shot prompting format for structured outputs
llama-cli -m ./model.gguf -p 'Q: What year was Python created?\nA:' -n 10