Configure Prometheus Alertmanager

Prometheus Alertmanager handles alert routing, deduplication, silencing, and delivery to receivers such as PagerDuty, Slack, email, and webhooks. Getting the configuration right—especially routing trees, inhibition rules, and grouping—is what separates a noisy, unreliable on-call setup from one that actually wakes the right person for the right reason. This guide covers a production-grade Alertmanager configuration from scratch.

Install Alertmanager

Debian/Ubuntu

sudo apt install prometheus-alertmanager

Fedora / RHEL 9 / Rocky 9

sudo dnf install golang-github-prometheus-alertmanager

Arch

sudo pacman -S alertmanager

Binary install (version-agnostic, all distros)

When your distro ships an old version or you need a specific release, install directly from the upstream tarball. Check GitHub releases for the latest version number.

AMVER=0.27.0
curl -Lo /tmp/alertmanager.tar.gz \
  https://github.com/prometheus/alertmanager/releases/download/v${AMVER}/alertmanager-${AMVER}.linux-amd64.tar.gz
tar -xzf /tmp/alertmanager.tar.gz -C /tmp
sudo mv /tmp/alertmanager-${AMVER}.linux-amd64/alertmanager /usr/local/bin/
sudo mv /tmp/alertmanager-${AMVER}.linux-amd64/amtool /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo useradd --system --no-create-home --shell /sbin/nologin alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Create the systemd unit if it was not installed by a package manager:

sudo tee /etc/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=127.0.0.1:9093
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload

Core Configuration Concepts

Alertmanager's configuration lives in a single YAML file—typically /etc/alertmanager/alertmanager.yml. The top-level keys are global, templates, route, receivers, and inhibit_rules. Routes form a tree: an incoming alert walks the tree and matches the first branch whose matchers fit, then optionally continues if continue: true is set.

Global Settings and Templates

sudo tee /etc/alertmanager/alertmanager.yml <<'EOF'
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password_file: /etc/alertmanager/smtp_password
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
  slack_api_url_file: /etc/alertmanager/slack_webhook_url

templates:
  - '/etc/alertmanager/templates/*.tmpl'
EOF

Store secrets in separate files rather than inline in the YAML—Alertmanager supports the _file suffix for most credential fields. Restrict those files to 0600 owned by the alertmanager user.

Routing Tree

The route block defines how alerts are grouped and dispatched. Every alert hits the root route first. Child routes act as filters; the first matching child wins unless continue: true propagates the alert further.

sudo tee -a /etc/alertmanager/alertmanager.yml <<'EOF'
route:
  receiver: 'default-email'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical alerts go to PagerDuty immediately
    - matchers:
        - severity = "critical"
      receiver: 'pagerduty-critical'
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 1h
      continue: false

    # Database alerts: page DB team via Slack, also email
    - matchers:
        - team = "database"
      receiver: 'slack-database'
      group_by: ['alertname', 'instance']
      continue: true

    - matchers:
        - team = "database"
      receiver: 'email-database'
      continue: false

    # Watchdog (heartbeat) alert: never page, just swallow
    - matchers:
        - alertname = "Watchdog"
      receiver: 'null'
EOF

group_wait is how long Alertmanager buffers incoming alerts for the same group before sending the first notification—this reduces alert storms. group_interval is the minimum wait before sending a follow-up about new alerts added to an existing group. repeat_interval controls how often a still-firing alert re-notifies.

Receivers

sudo tee -a /etc/alertmanager/alertmanager.yml <<'EOF'
receivers:
  - name: 'null'

  - name: 'default-email'
    email_configs:
      - to: '[email protected]'
        send_resolved: true

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pagerduty_key
        description: '{{ template "pagerduty.default.description" . }}'
        severity: '{{ if eq .CommonLabels.severity "critical" }}critical{{ else }}error{{ end }}'
        send_resolved: true

  - name: 'slack-database'
    slack_configs:
      - channel: '#alerts-database'
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.body" . }}'
        send_resolved: true
        actions:
          - type: button
            text: 'Runbook'
            url: '{{ (index .Alerts 0).Annotations.runbook_url }}'

  - name: 'email-database'
    email_configs:
      - to: '[email protected]'
        send_resolved: true
EOF

Inhibition Rules

Inhibition suppresses lower-priority alerts when a higher-priority one is already firing for the same scope. This prevents alert floods: if the whole cluster is down, you do not need individual service alerts firing simultaneously.

sudo tee -a /etc/alertmanager/alertmanager.yml <<'EOF'
inhibit_rules:
  # Silence warnings when a critical fires for the same alertname + cluster
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal:
      - alertname
      - cluster
      - service

  # Silence all service alerts when NodeDown fires on the same instance
  - source_matchers:
      - alertname = "NodeDown"
    target_matchers:
      - severity =~ "warning|critical"
    equal:
      - instance
EOF

The equal list specifies which labels must match between the source (suppressing) and target (suppressed) alerts. Get this wrong and you will either over-suppress or under-suppress—always test with amtool after changes.

Custom Templates

Alertmanager uses Go's text/template syntax. Place templates in the directory referenced in the templates glob.

sudo mkdir -p /etc/alertmanager/templates
sudo tee /etc/alertmanager/templates/custom.tmpl <<'EOF'
{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.body" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
*Instance:* {{ .Labels.instance }}
*Started:* {{ .StartsAt | since }}
{{ end }}
{{ end }}

{{ define "pagerduty.default.description" }}
{{ .CommonLabels.alertname }} on {{ .CommonLabels.cluster }} — {{ .CommonAnnotations.summary }}
{{ end }}
EOF
sudo chown -R alertmanager:alertmanager /etc/alertmanager

On-Call Integration with PagerDuty and Webhook Receivers

For teams using OpsGenie, VictorOps, or a custom on-call scheduler, the webhook_configs receiver type posts a JSON payload to any HTTP endpoint. This is also how integrations such as alertmanager-webhook-receiver or incident.io work.

# Add to the receivers list in alertmanager.yml
sudo tee -a /etc/alertmanager/alertmanager.yml <<'EOF'
  - name: 'opsgenie-oncall'
    opsgenie_configs:
      - api_key_file: /etc/alertmanager/opsgenie_key
        responders:
          - name: 'platform-oncall'
            type: 'schedule'
        message: '{{ template "pagerduty.default.description" . }}'
        send_resolved: true

  - name: 'custom-webhook'
    webhook_configs:
      - url: 'https://incidents.example.com/hooks/alertmanager'
        send_resolved: true
        http_config:
          bearer_token_file: /etc/alertmanager/webhook_token
EOF

Validate and Reload

# Validate config syntax with amtool
amtool check-config /etc/alertmanager/alertmanager.yml

# Start and enable the service
sudo systemctl enable --now alertmanager
sudo systemctl status alertmanager

# Send a live reload without restarting (avoids silences/notification log loss)
sudo systemctl reload alertmanager
# or via the API:
curl -X POST http://127.0.0.1:9093/-/reload

Test Alert Routing with amtool

Before an incident, verify that a hypothetical alert lands on the correct receiver:

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  severity=critical team=database alertname=PostgreSQLDown

The output will show which receiver is matched and whether continue propagates it further. Use this after every routing change.

# Send a test alert manually
amtool alert add alertname=TestAlert severity=critical cluster=prod-eu \
  --alertmanager.url=http://127.0.0.1:9093

# Verify it appeared
amtool alert query --alertmanager.url=http://127.0.0.1:9093

Connect Prometheus to Alertmanager

In /etc/prometheus/prometheus.yml, add or verify the alerting block:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'localhost:9093'
      timeout: 10s

sudo systemctl reload prometheus

Troubleshooting

• Alert arrives in Alertmanager but is never sent: Check inhibition rules with amtool config routes test. An overly broad equal list silently drops alerts. Also check active silences via amtool silence query.
• Duplicate notifications: Verify continue: false is set on routes where alerts should not fall through to additional receivers unintentionally.
• Slack/PagerDuty delivery failures: Run journalctl -u alertmanager -f and look for HTTP 4xx/5xx errors. The most common cause is a rotated API key that was not updated in the secrets file.
• Template rendering errors: Alertmanager logs the full template error. Test templates offline with amtool template render or the UI at http://127.0.0.1:9093/#/status.
• Config reload fails silently: curl http://127.0.0.1:9093/-/reload returns HTTP 200 on success and logs errors to stderr on failure. Always follow a reload with amtool check-config.