Alert Rules

Overview

This page covers advanced alert rule configuration beyond basic threshold alerts — including compound multi-condition rules, silencing active alerts during maintenance windows, inhibition rules that suppress lower-severity alerts when a critical one is already firing, and escalation policies.

Prerequisites

An active Polystack account with project access
At least one notification channel configured (see Monitoring Admin — Alert Channels)

Compound Alert Rules

Combine multiple conditions in a single rule using AND/OR logic:

Dashboard
Rule File

Navigate to Monitor Center > Monitoring (Create Alert Rule, admin view) and click Advanced Mode.In Advanced Mode, add multiple conditions:

Field	Value
Condition A	`polystack_compute_cpu_utilization > 85`
Operator	`AND`
Condition B	`polystack_compute_memory_free_pct < 15`

This rule fires only when both CPU is above 85% AND available memory is below 15% — reducing false positives from transient CPU spikes.

alert-resource-pressure.yaml

name: resource-pressure-critical
severity: critical
evaluation_period: 5m
conditions:
  - metric: polystack_compute_cpu_utilization
    condition: ">"
    threshold: 85
  - metric: polystack_compute_memory_free_pct
    condition: "<"
    threshold: 15
logic: AND
notification_channels:
  - ops-pagerduty

Create compound alert rule

monitoring alert rule create --file alert-resource-pressure.yaml

Silencing Alerts

Silences temporarily suppress alert notifications during planned maintenance. The alert rule continues to evaluate — only notifications are suppressed.

Dashboard
CLI

Create a silence

Navigate to Monitor Center > Monitoring (Create Silence, admin view).

Field	Description
Matchers	Label selectors that match the alerts to silence (e.g., `host=compute-node-03`)
Duration	How long the silence is active (e.g., `2h`)
Comment	Reason for the silence (required — links to change ticket)
Creator	Your username (auto-populated)

Verify silence is active

Navigate to the Silences section in Monitor Center > Monitoring. Active silences show their matcher, creator, and expiry time.

Any alerts matching the silence matchers show status Silenced instead of firing notifications.

Create a 2-hour silence for a host

monitoring alert silence create \
  --matcher 'host=compute-node-03' \
  --duration 2h \
  --comment "Scheduled maintenance window - firmware update"

List active silences

monitoring alert silence list --status active

Expire a silence early

monitoring alert silence expire <SILENCE_ID>

Inhibition Rules

Inhibition rules suppress lower-severity alerts when a higher-severity alert is already active for the same source. This prevents alert storms during major incidents.

Example: Suppress warnings when critical is firing

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal:
      - host

This rule suppresses all warning alerts for a host when a critical alert is already firing for that same host — reducing notification noise during a major outage. Configure inhibition rules via Monitor Center > Monitoring (Inhibition Rules, admin view).

Escalation Policies

Configure multi-tier escalation for critical alerts:

Define escalation tiers

Navigate to Monitor Center > Monitoring (Escalation Policies, admin view).

Tier	Channel	Delay	Condition
Tier 1	PagerDuty on-call	Immediate	Alert fires
Tier 2	Slack ops channel	5 minutes	Not acknowledged
Tier 3	Page on-call manager	15 minutes	Still unacknowledged

Assign to alert rules

Open an existing alert rule and set the Escalation Policy field to the policy you created. The policy applies to all future alert events for that rule.

Create separate escalation policies for different severity levels — critical infrastructure alerts may warrant a 3-tier escalation while informational alerts can go to a single team Slack channel with no escalation.

GitOps-Based Rule Management

Manage alert rules as code for version-controlled, auditable configurations:

Export all current alert rules

monitoring alert rule export --format yaml --output ./alert-rules/

Import rules from directory

monitoring alert rule import --dir ./alert-rules/ --apply

Store rule files in your infrastructure repository and apply changes through your CI/CD pipeline. This enables peer review of alert rule changes and automatic rollback if a rule causes issues.

Next Steps

Metrics & Alerts

Basic alert rule creation for metric thresholds

Monitoring Admin — Alert Channels

Configure the notification channels referenced by alert rules

Dashboards

Visualize metrics alongside alert thresholds

Troubleshooting

Diagnose alert rules that are not firing or delivering notifications

Core Services

Other Services

Overview

Compound Alert Rules

Silencing Alerts

Create a silence

Verify silence is active

Inhibition Rules

Escalation Policies

Define escalation tiers

Assign to alert rules

GitOps-Based Rule Management

Next Steps

Metrics & Alerts

Monitoring Admin — Alert Channels

Dashboards

Troubleshooting

​Overview

​Compound Alert Rules

​Silencing Alerts

Create a silence

Verify silence is active

​Inhibition Rules

​Escalation Policies

Define escalation tiers

Assign to alert rules

​GitOps-Based Rule Management

​Next Steps

Metrics & Alerts

Monitoring Admin — Alert Channels

Dashboards

Troubleshooting

Overview

Compound Alert Rules

Silencing Alerts

Inhibition Rules

Escalation Policies

GitOps-Based Rule Management

Next Steps