Install Ollama for Local LLMs

Ollama is the fastest way to run large language models locally — no cloud account, no API key, no data leaving your machine. It packages model weights, a runtime, and a small REST API into a single binary. Once installed, you can pull and run models like Llama 3, Mistral, Gemma, and Phi with one command, then talk to them from the terminal, a web UI, or your own code.

Prerequisites

• A 64-bit x86 or ARM machine. Ollama runs on CPU, but a dedicated GPU (NVIDIA with CUDA, or AMD with ROCm) makes it dramatically faster.
• At least 8 GB RAM for small models (7B parameters). 16 GB+ is comfortable for 13B models.
• NVIDIA GPU users: install the proprietary driver (version 520+) before proceeding. The Ollama installer detects CUDA automatically.
• AMD GPU users: ROCm 5.7+ must already be installed. AMD support is still maturing; check the Ollama GitHub page for the current compatibility list.

Install Ollama

The official installer is a shell script fetched over HTTPS. On Linux it installs the ollama binary to /usr/local/bin, creates a ollama system user, and registers a systemd service.

curl -fsSL https://ollama.com/install.sh | sh

The script detects your GPU automatically and pulls in any needed CUDA or ROCm libraries. Watch the output — it will tell you which GPU (or CPU-only mode) it configured.

If you prefer not to pipe a script to a shell, download and inspect it first:

curl -fsSL https://ollama.com/install.sh -o install-ollama.sh
less install-ollama.sh
bash install-ollama.sh

Verify the service is running

systemctl status ollama

You should see active (running). Ollama starts automatically on boot via systemd. If the service is not running, start it manually:

sudo systemctl enable --now ollama

Pull and Run a Model

Models are identified by name and optional tag (the tag sets the variant or quantisation level). The Ollama model library lists everything available.

Pull a model

This downloads the model weights to ~/.ollama/models (or /usr/share/ollama/.ollama/models when running as the system service).

ollama pull llama3.2

Other popular starting points:

• mistral — fast 7B model, good general purpose
• gemma3:4b — compact Google model, runs well on 8 GB RAM
• phi4-mini — Microsoft's small but capable model
• llama3.2:1b — the :1b tag selects the 1-billion parameter variant, extremely fast on CPU

Start an interactive chat session

ollama run llama3.2

Type your prompt and press Enter. Type /bye or press Ctrl+D to exit. Ollama keeps the model loaded in memory for a few minutes after you exit, so the next run starts instantly.

One-shot prompt from the terminal

ollama run mistral "Explain systemd socket activation in two paragraphs"

List downloaded models

ollama list

Remove a model

ollama rm mistral

Using the REST API on Port 11434

Ollama exposes a local REST API at http://localhost:11434. This is what web UIs, VS Code extensions, and scripts talk to. The service is bound to localhost by default — it is not reachable from other machines unless you change that.

Quick API test

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.2",
    "prompt": "What is nftables?",
    "stream": false
  }'

Set "stream": false to get a single JSON response. Omit it (or set true) to receive newline-delimited JSON chunks as the model generates tokens — useful for building streaming interfaces.

OpenAI-compatible endpoint

Ollama also provides a drop-in OpenAI-compatible endpoint, which means tools written for the OpenAI API work without modification:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Summarise the CAP theorem"}]
  }'

Expose the API on your network (optional)

By default the server listens only on 127.0.0.1. To allow other machines on your LAN to reach it, override the bind address via an environment variable in the systemd service:

sudo systemctl edit ollama

Add the following inside the editor that opens (between the comment markers):

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Security note: the API has no authentication. Only expose it on a network you trust, or put it behind a reverse proxy with access controls.

Add a Web UI

The terminal is fine for testing, but a chat web interface is more practical for day-to-day use.

Open WebUI (recommended)

Open WebUI is the most actively maintained option. The simplest install uses Docker or Podman:

docker run -d \
  --network=host \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

--network=host lets the container reach Ollama at localhost:11434 without any extra configuration. Open http://localhost:3000 in your browser. The first account you create becomes the admin.

If you do not want Docker, Open WebUI also has a pip install path — see their README for the current instructions, as the exact command changes between releases.

Hollama (lightweight alternative)

For a minimal, no-Docker web UI, Hollama is a single-binary Go server. Download the latest release for your architecture from its GitHub releases page, make it executable, and run it — it serves on port 4173 and talks to Ollama automatically.

Keeping Ollama Updated

Re-running the install script updates the binary in place:

curl -fsSL https://ollama.com/install.sh | sh

Model weights do not update automatically. Re-run ollama pull <model> to fetch the latest version of a specific model.

Troubleshooting

"Error: model not found"

Run ollama list to check the exact name. Model names are case-sensitive and must match the library slug exactly, including any tag (e.g., llama3.2:1b not llama3.2-1b).

Slow generation on GPU

Check whether Ollama actually sees the GPU:

ollama run llama3.2 "hi"
# In a second terminal:
watch -n1 nvidia-smi   # NVIDIA
# or
watch -n1 rocm-smi     # AMD

If GPU utilisation stays at 0%, the model is running on CPU — usually because the driver or CUDA version is mismatched. Check journalctl -u ollama -e for error messages about CUDA library loading.

Service fails to start

journalctl -u ollama -n 50 --no-pager

Common causes: port 11434 already in use, the ollama user lacking permissions to the model directory, or a missing shared library (run ldd /usr/local/bin/ollama to check).

Out of memory errors

Switch to a smaller quantisation. For example, ollama pull llama3.2:1b uses roughly 1.3 GB of RAM, compared to ~5 GB for the default 3B version. Check the model page on ollama.com/library for all available tags and their sizes.