Can I run Prometheus and Grafana on separate servers?

Yes. In that case set Prometheus to listen on its public or private network interface instead of 127.0.0.1, secure it with a reverse proxy and authentication, and point the Grafana datasource URL at the remote Prometheus address.

How do I monitor additional hosts with this setup?

Install node_exporter on each target host, then add its IP and port to the 'node' job's static_configs in prometheus.yml. For larger environments, consider using Prometheus service discovery instead of static lists.

How long does Prometheus store data by default?

The unit file in this guide sets --storage.tsdb.retention.time=15d. The upstream default is also 15 days. You can increase it or combine it with --storage.tsdb.retention.size to set a byte cap.

Is it safe to expose Grafana directly on port 3000?

It is acceptable for internal networks. For internet-facing servers, put Grafana behind a reverse proxy (nginx or Caddy) with TLS and ideally restrict access by IP or add SSO.

Can I use the distro-packaged Prometheus instead of the binary from GitHub?

You can, but distro packages often lag several major versions behind upstream. For production monitoring, the official binary ensures you have current security fixes and features.

How to Install Prometheus and Grafana on Linux

Prometheus collects and stores time-series metrics; Grafana turns those metrics into dashboards. Together they form the backbone of most self-hosted Linux monitoring stacks. This guide walks you through installing both on a single server, wiring them together, and building your first useful dashboard — no Kubernetes, no Helm, just direct installs on a Linux host.

Architecture Overview

Prometheus works by scraping HTTP endpoints that expose metrics in its text format. By default it scrapes itself, but the real value comes from adding exporters — small processes that translate system or application state into Prometheus metrics. node_exporter is the standard exporter for OS-level metrics (CPU, memory, disk, network) and is what we use here. Grafana connects to Prometheus as a datasource and queries it using PromQL.

Step 1: Create System Users

Run services under dedicated, no-login system accounts. Never run Prometheus or Grafana as root.

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

Step 2: Install Prometheus

Prometheus ships as a static Go binary. Grab the latest stable release from the official releases page. Replace the version number below with the current stable (check github.com/prometheus/prometheus/releases).

PROM_VERSION="2.52.0"
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
cd prometheus-${PROM_VERSION}.linux-amd64

sudo mv prometheus promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv consoles console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Write the Prometheus Configuration

Create the main config file. This minimal config scrapes Prometheus itself and the node_exporter you will install next.

sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
EOF
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create the Prometheus systemd Unit

sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Monitoring
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.listen-address=127.0.0.1:9090
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

Note: --web.listen-address=127.0.0.1:9090 binds Prometheus to localhost only. If you need remote access, put it behind a reverse proxy with authentication rather than exposing it directly.

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Step 3: Install node_exporter

Check the current release at github.com/prometheus/node_exporter/releases.

NODE_VERSION="1.8.1"
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_VERSION}/node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
tar xzf node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
sudo mv node_exporter-${NODE_VERSION}.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Step 4: Install Grafana

Grafana maintains official repositories for all major distributions. Use the OSS (open-source) edition unless you need enterprise features.

Debian / Ubuntu

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

Fedora / RHEL / Rocky

sudo tee /etc/yum.repos.d/grafana.repo > /dev/null <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF
sudo dnf install -y grafana

Arch Linux

sudo pacman -S grafana

Enable and Start Grafana

sudo systemctl enable --now grafana-server

Grafana listens on port 3000 by default. The default credentials are admin / admin; you will be forced to change the password on first login. Main config lives at /etc/grafana/grafana.ini.

Step 5: Open Firewall Ports

You need port 3000 reachable from your browser. Prometheus (9090) and node_exporter (9100) should stay internal unless you have a specific reason to expose them.

ufw (Debian/Ubuntu default)

sudo ufw allow 3000/tcp comment 'Grafana'

firewalld (Fedora/RHEL/Rocky default)

sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --reload

Step 6: Add Prometheus as a Grafana Datasource

Open http://<your-server-ip>:3000 in a browser and log in.
Go to Connections → Data sources → Add data source.
Select Prometheus.
Set the URL to http://127.0.0.1:9090 (Grafana and Prometheus are on the same host).
Leave all other fields at their defaults and click Save & test. You should see "Successfully queried the Prometheus API."

Step 7: Import Your First Dashboard

Building dashboards from scratch takes time. The Grafana community publishes ready-made dashboards at grafana.com/grafana/dashboards. Dashboard 1860 (Node Exporter Full) is the de-facto standard for system metrics.

Go to Dashboards → Import.
Enter 1860 in the "Import via grafana.com" field and click Load.
Select your Prometheus datasource from the dropdown.
Click Import.

You now have a comprehensive dashboard showing CPU usage, memory, disk I/O, network traffic, and system load — all populated with live data from your node_exporter.

Verification

Check all three services are active and collecting data:

systemctl is-active prometheus node_exporter grafana-server

Expected output (will vary):

active
active
active

Confirm Prometheus is actually scraping targets — from the server itself:

curl -s http://127.0.0.1:9090/api/v1/targets | python3 -m json.tool | grep '"health"'

Both targets should show "health": "up". You can also browse to http://127.0.0.1:9090/targets if you have a local browser or SSH tunnel.

Troubleshooting

Prometheus target shows "down": Check node_exporter is running (systemctl status node_exporter) and confirm it is listening (ss -tlnp | grep 9100). Check prometheus.yml indentation — YAML is whitespace-sensitive.
Grafana shows "Bad Gateway" for the datasource: Prometheus is likely not running or bound to a different address. Verify with ss -tlnp | grep 9090 and confirm the URL in the datasource config matches.
Can't reach Grafana in browser: Confirm the firewall rule was applied (sudo ufw status or sudo firewall-cmd --list-ports) and that Grafana is actually listening (ss -tlnp | grep 3000).
Prometheus config errors: Validate before restarting — promtool check config /etc/prometheus/prometheus.yml. A non-zero exit code will include the line number of the problem.
Disk filling up: The default retention is set to 15 days in the unit file above. Adjust --storage.tsdb.retention.time or add --storage.tsdb.retention.size=10GB to cap storage use.