Install Loki and Promtail for Logs

Grafana Loki is a horizontally scalable, log aggregation system designed to index only metadata (labels), not the full log content. Paired with Promtail as the log shipping agent, it gives you a lightweight alternative to a full Elasticsearch stack. This guide walks through installing both as single binaries, configuring Promtail to scrape systemd-journald, and setting up a basic log-based metric rule — all managed under systemd.

Prerequisites and Architecture

Loki listens on port 3100. Promtail runs on each host you want to collect logs from, ships them to Loki over HTTP, and in a single-host setup both services run on the same machine. You will need Grafana separately to visualise the data; this guide does not cover Grafana installation, but adding Loki as a data source takes under two minutes once Loki is running.

• A 64-bit Linux server with at least 1 GB RAM
• Outbound internet access to download binaries from GitHub
• A Grafana instance (local or remote) for dashboarding — optional for this guide
• curl, unzip, and systemd available

Step 1 — Download the Binaries

Grafana publishes Loki and Promtail as static binaries on GitHub. Find the latest stable release at github.com/grafana/loki/releases. At time of writing that is 3.1.0; substitute newer versions as they appear.

LOKI_VERSION="3.1.0"
ARCH="amd64"   # change to arm64 on ARM hosts

curl -fSL -o /tmp/loki.zip \
  "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-${ARCH}.zip"

curl -fSL -o /tmp/promtail.zip \
  "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/promtail-linux-${ARCH}.zip"

cd /tmp
unzip loki.zip
unzip promtail.zip

sudo install -m 755 loki-linux-${ARCH}       /usr/local/bin/loki
sudo install -m 755 promtail-linux-${ARCH}   /usr/local/bin/promtail

Verify the binaries are in place and executable:

loki --version
promtail --version

Step 2 — Create System Users and Directories

Run each service as a dedicated unprivileged user. Promtail needs supplementary membership in the systemd-journal group to read the journal socket without root.

sudo useradd --system --no-create-home --shell /usr/sbin/nologin loki
sudo useradd --system --no-create-home --shell /usr/sbin/nologin promtail

# Debian/Ubuntu
sudo usermod -aG systemd-journal promtail

# Fedora / RHEL / Rocky — same command; group exists by default
sudo usermod -aG systemd-journal promtail

# Arch — same
sudo usermod -aG systemd-journal promtail

sudo mkdir -p /etc/loki /etc/promtail /var/lib/loki /var/lib/promtail
sudo chown loki:loki    /var/lib/loki
sudo chown promtail:promtail /var/lib/promtail

Step 3 — Configure Loki

The minimal single-node configuration below uses the TSDB storage backend (default since Loki 2.8) and stores chunks on local disk. For production, swap filesystem for an object store like S3 or GCS.

sudo tee /etc/loki/loki.yaml <<'EOF'
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory:  /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  storage:
    type: local
    local:
      directory: /var/lib/loki/rules
  rule_path: /var/lib/loki/rules-temp
  alertmanager_url: http://localhost:9093   # adjust or remove if not using Alertmanager
  enable_api: true

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
EOF

Step 4 — Configure Promtail

Promtail's loki_push_api target type (available since Promtail 2.4) is the simplest path to collecting journald, but the journal scrape config is more explicit and easier to label. The configuration below uses journal directly and adds the machine hostname and unit name as labels.

sudo tee /etc/promtail/promtail.yaml <<'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: __HOSTNAME__   # Promtail substitutes the real hostname at runtime
    relabel_configs:
      - source_labels: [__journal__systemd_unit]
        target_label: unit
      - source_labels: [__journal__priority_keyword]
        target_label: level
      - source_labels: [__journal__syslog_identifier]
        target_label: syslog_identifier
EOF

Note: __HOSTNAME__ is a Promtail built-in that expands to the system hostname; you do not need to hardcode it.

Step 5 — Create systemd Units

Loki service

sudo tee /etc/systemd/system/loki.service <<'EOF'
[Unit]
Description=Grafana Loki log aggregation system
After=network.target

[Service]
User=loki
Group=loki
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki.yaml
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Promtail service

sudo tee /etc/systemd/system/promtail.service <<'EOF'
[Unit]
Description=Promtail log shipping agent
After=network.target loki.service

[Service]
User=promtail
Group=promtail
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail.yaml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now loki promtail

Step 6 — Open the Firewall (if applicable)

If Grafana runs on a different host, open port 3100 only to that host. Do not expose Loki to the public internet without authentication.

# ufw (Debian/Ubuntu)
sudo ufw allow from 192.168.1.0/24 to any port 3100 proto tcp

# firewalld (Fedora/RHEL/Rocky)
sudo firewall-cmd --permanent --add-rich-rule=\
  'rule family=ipv4 source address=192.168.1.0/24 port port=3100 protocol=tcp accept'
sudo firewall-cmd --reload

Step 7 — Add a Log-Based Metric Rule

Loki's ruler can derive Prometheus-compatible metrics from log streams. Create a recording rule that counts SSH authentication failures per minute — useful for alerting and dashboarding without storing raw log lines in a metrics system.

sudo mkdir -p /var/lib/loki/rules/fake   # "fake" is the tenant name when auth_enabled=false
sudo tee /var/lib/loki/rules/fake/ssh-rules.yaml <<'EOF'
groups:
  - name: ssh
    rules:
      - record: job:ssh_auth_failures:rate1m
        expr: |
          sum by (host) (
            rate(
              {job="systemd-journal", unit="sshd.service"}
                |= "Failed password"
                | unwrap __error__
                  [1m]
            )
          )
      - alert: HighSSHFailures
        expr: job:ssh_auth_failures:rate1m > 5
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Elevated SSH failures on {{ $labels.host }}"
EOF

sudo chown -R loki:loki /var/lib/loki/rules
sudo systemctl restart loki

Verification

Check that both services are running cleanly:

systemctl status loki promtail
journalctl -u loki -u promtail --since "5 minutes ago"

Query the Loki HTTP API directly to confirm logs are arriving:

curl -s 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={job="systemd-journal"}' \
  --data-urlencode 'limit=5' \
  --data-urlencode 'start=1h' | python3 -m json.tool | head -40

A successful response will contain a streams array with log entries. Also hit the ready endpoint:

curl http://localhost:3100/ready
# Expected output: ready

Troubleshooting

• Promtail can't read the journal: Confirm promtail is in the systemd-journal group (groups promtail) and that the service was restarted after adding the group. The group membership only takes effect in a new login session or service restart.
• Loki returns 429 "too many outstanding requests": You are hitting the default ingestion rate limit. Add ingestion_rate_mb: 16 and ingestion_burst_size_mb: 32 under limits_config for a busier host.
• "entry out of order" errors in Loki logs: This usually means two Promtail instances are pushing the same stream with overlapping timestamps. Ensure each host has a unique host label.
• Rules not evaluating: Check that the rules directory path in loki.yaml matches what you created (/var/lib/loki/rules) and that files are owned by the loki user. Verify with curl http://localhost:3100/loki/api/v1/rules.
• Binary not found after install: Confirm /usr/local/bin is on your PATH or use the full path in the systemd unit's ExecStart.