Install Prometheus on Linux

Prometheus is a pull-based metrics system that scrapes HTTP endpoints and stores time-series data locally. It is the backbone of most modern Linux observability stacks. This guide covers installation from the official tarball (the most portable and version-controlled approach), a quick note on distro packages, writing a working prometheus.yml, configuring scrape targets, tuning retention, and locking it all down with a proper systemd unit.

Choose Your Installation Method

Distro packages exist but frequently lag behind upstream by months. Unless you are on a team that enforces package-manager-only installs, the tarball gives you the exact version you want and a predictable file layout.

Option A: Official Tarball (Recommended)

Download the latest stable release from prometheus.io/download. At the time of writing that is 2.x; always verify the checksum.

PROM_VERSION="2.52.0"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt
sha256sum --check --ignore-missing sha256sums.txt

tar xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROM_VERSION}.linux-amd64 /opt/prometheus

Create a dedicated system user with no login shell. Running Prometheus as root is unnecessary and a poor security practice.

sudo useradd --system --no-create-home --shell /sbin/nologin prometheus

Create the data directory and fix ownership:

sudo mkdir -p /var/lib/prometheus
sudo chown -R prometheus:prometheus /opt/prometheus /var/lib/prometheus

Symlink the binaries into /usr/local/bin so they are on the system PATH:

sudo ln -s /opt/prometheus/prometheus /usr/local/bin/prometheus
sudo ln -s /opt/prometheus/promtool /usr/local/bin/promtool

Option B: Distro Packages

If you prefer managed packages, the commands below install whatever version is in the default repositories. Check apt-cache policy prometheus or dnf info prometheus first to confirm the version is acceptable.

# Debian / Ubuntu
sudo apt update && sudo apt install prometheus

# Fedora / RHEL 9+ (enable EPEL on RHEL first)
sudo dnf install prometheus

# Arch Linux
sudo pacman -S prometheus

Package installs create the user, directories, and systemd unit automatically. Skip to the prometheus.yml section if using a package.

Write prometheus.yml

The main configuration file lives at /opt/prometheus/prometheus.yml for tarball installs, or /etc/prometheus/prometheus.yml for package installs. The structure has three top-level sections you will touch most often: global, alerting, and scrape_configs.

Below is a production-ready starting point. Adjust scrape_interval to match your workload; 15 seconds is the community standard.

sudo tee /opt/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s        # How often to scrape targets
  evaluation_interval: 15s   # How often to evaluate rules
  scrape_timeout: 10s

  external_labels:
    environment: 'production'
    region: 'us-east-1'

alerting:
  alertmanagers:
    - static_configs:
        - targets: []           # Add Alertmanager address when ready

rule_files: []                  # e.g. - /opt/prometheus/rules/*.yml

scrape_configs:
  # Prometheus scrapes its own metrics by default
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter - install separately on each host to be monitored
  - job_name: 'node'
    static_configs:
      - targets:
          - 'localhost:9100'
          - '192.168.1.10:9100'
          - '192.168.1.11:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
EOF

Validate the file before starting the service. promtool will catch YAML syntax errors and invalid configuration keys:

promtool check config /opt/prometheus/prometheus.yml

Expected output: SUCCESS: 1 rule files found, 0 rules loaded — exact numbers will vary.

Configure Storage Retention

By default Prometheus keeps 15 days of data. Two flags control retention; you can set one or both. If both are set, whichever limit is hit first takes effect.

• --storage.tsdb.retention.time — duration string, e.g. 30d, 6h
• --storage.tsdb.retention.size — byte limit, e.g. 50GB. Requires Prometheus 2.7+.

You will pass these flags in the systemd unit below. Estimate disk usage: a typical single-node scrape at 15 s produces roughly 1–3 MB per day per 1000 time series. Check actual usage after a week with du -sh /var/lib/prometheus.

Create the systemd Unit

This applies to tarball installs. Package installs ship a unit file, but you may want to override flags using a drop-in.

sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring System
Documentation=https://prometheus.io/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/opt/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=20GB \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-lifecycle

Restart=on-failure
RestartSec=5s

# Harden the service
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/prometheus
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

Security note: --web.listen-address=127.0.0.1:9090 binds only to localhost. Expose port 9090 publicly only behind authentication (basic auth + TLS via a reverse proxy such as nginx). The --web.enable-lifecycle flag enables the /-/reload HTTP endpoint for config reloads without a restart — keep this off if the port is exposed publicly without auth.

Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Verify the Installation

sudo systemctl status prometheus

You should see active (running). Then confirm the HTTP API is responding:

curl -s http://localhost:9090/-/healthy

A healthy instance returns Prometheus Server is Healthy.

Check that scrape targets are being reached. The Prometheus UI at http://localhost:9090/targets (use an SSH tunnel or reverse proxy to reach it remotely) shows each target and its state. Alternatively query the API:

curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep '"health"'

All targets should show "health": "up".

Reload Configuration Without a Restart

After editing prometheus.yml, always validate first then send a reload signal. No metrics are lost during a reload.

promtool check config /opt/prometheus/prometheus.yml
curl -s -X POST http://localhost:9090/-/reload

If --web.enable-lifecycle is not set, use sudo systemctl reload prometheus (which sends SIGHUP) or a full restart.

Troubleshooting

Service Fails to Start

The first place to look is the journal:

sudo journalctl -u prometheus -n 50 --no-pager

Common causes: YAML syntax error in prometheus.yml (run promtool check config), wrong file ownership on /var/lib/prometheus (sudo chown -R prometheus:prometheus /var/lib/prometheus), or a port conflict (sudo ss -tlnp | grep 9090).

Targets Show "down" State

• Confirm the exporter is running on the target host: curl http://<host>:9100/metrics
• Check firewall rules. On Fedora/RHEL: sudo firewall-cmd --list-all. On Ubuntu/Debian with ufw: sudo ufw status.
• Verify DNS resolution if using hostnames in targets: dig +short <hostname>

Disk Fills Up Faster Than Expected

High-cardinality labels (e.g., a label with unique request IDs) can explode TSDB storage. Use the TSDB status page at http://localhost:9090/tsdb-status to identify top series by label name. Lower your storage.tsdb.retention.size as a ceiling, and consider dropping high-cardinality labels with metric_relabel_configs in your scrape config.

CPU Spikes During Scrapes

If you are scraping hundreds of targets at the same short interval, Prometheus can spike. Stagger intervals using per-job scrape_interval overrides in scrape_configs, or increase the global interval. You can also shard scraping across multiple Prometheus instances using federation or Thanos/Cortex for large fleets.