> ## Documentation Index
> Fetch the complete documentation index at: https://docs.polystack.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Notifications

> Configure notification groups, SMTP delivery, webhook integrations, and metric server export for Ironcore Backup Solution events.

## Overview

Ironcore Backup Solution (IBS) routes events from backup jobs, verification runs,
sync jobs, and access control changes to a configurable notification system.
Targets include SMTP email, webhooks (for chat platforms and incident tools),
and external metric servers for time-series monitoring.

This page covers building notification groups, choosing the right delivery
channel, and rule-based routing.

<Note>
  **Prerequisites**

  * Administrator role on the Polystack platform
  * SMTP endpoint and credentials for email delivery
  * Webhook URLs for chat or incident management integrations
  * Metric server endpoint for time-series export
</Note>

***

## Event Sources

| Source               | Events                                      |
| -------------------- | ------------------------------------------- |
| **Backup job**       | Started, completed, failed, warning         |
| **Restore task**     | Started, completed, failed                  |
| **Sync job**         | Started, completed, failed, integrity error |
| **Verification job** | Completed (per snapshot result), failed     |
| **Datastore**        | Free-space threshold, status change         |
| **Authentication**   | Failed login, MFA lockout, new API token    |
| **Configuration**    | Permission grant, role create, key rotation |
| **System**           | Service start / stop, certificate renewal   |

***

## Notification Targets

<CardGroup cols={2}>
  <Card title="Email (SMTP)" icon="mail" color="#bf9667">
    Standard authenticated SMTP delivery. Templates support HTML and plaintext.
    Best for human recipients.
  </Card>

  <Card title="Webhook" icon="link" color="#bf9667">
    HTTP POST to any endpoint. Standard JSON body. Integrates with chat
    platforms, incident management, and ticket systems.
  </Card>

  <Card title="Metric Server" icon="activity" color="#bf9667">
    Pushes events as time-series metrics to an external metric server such
    as InfluxDB or a Prometheus push gateway.
  </Card>

  <Card title="In-Dashboard Inbox" icon="inbox" color="#bf9667">
    All events appear in the Dashboard notification inbox regardless of
    external delivery. Useful for operators reviewing recent activity.
  </Card>
</CardGroup>

***

## Configure SMTP Delivery

<Tabs>
  <Tab title="Deployment Console" icon="gauge">
    <Steps titleSize="h3">
      <Step title="Open Notification Targets" icon="external-link">
        Navigate to **Backup Solution** > **Notifications** > **Targets**.
      </Step>

      <Step title="Add an SMTP target" icon="mail">
        Click **Add SMTP Target**. Enter:

        * **Name**: `ops-smtp`
        * **Server**: `smtp.<your-domain>`
        * **Port**: `587`
        * **From**: `ironcore-backup@<your-domain>`
        * **Auth method**: PLAIN or LOGIN
        * **Username / Password**: SMTP credentials
        * **TLS**: STARTTLS or implicit TLS
      </Step>

      <Step title="Test delivery" icon="signal">
        Click **Test**. A sample message is sent to a configurable test address.
      </Step>

      <Step title="Save" icon="check">
        Click **Save**.

        <Check>The SMTP target is registered and ready to use in notification groups.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Register an SMTP target" theme={null}
    ironcore-backup notification target create \
      --name ops-smtp \
      --type smtp \
      --server smtp.<your-domain> \
      --port 587 \
      --from ironcore-backup@<your-domain> \
      --auth-method plain \
      --username ibs-svc \
      --password-file /etc/ironcore/smtp-password \
      --tls starttls
    ```
  </Tab>
</Tabs>

***

## Configure Webhook Delivery

A webhook target sends a JSON payload to any HTTP endpoint when an event matches
the routing rule.

<Tabs>
  <Tab title="Deployment Console" icon="gauge">
    <Steps titleSize="h3">
      <Step title="Add a webhook target" icon="link">
        Open **Notifications** > **Targets** > **Add Webhook**.
      </Step>

      <Step title="Configure the endpoint" icon="settings">
        Enter:

        * **Name**: `chat-ops`
        * **URL**: `https://chat.<your-domain>/webhooks/ironcore-backup`
        * **Method**: POST
        * **Headers**: `Authorization: Bearer <token>`
        * **Body template**: a JSON template
      </Step>

      <Step title="Customise the body template" icon="code">
        Use the template variables provided by IBS:

        ```json theme={null}
        {
          "type": "{{event.type}}",
          "severity": "{{event.severity}}",
          "subject": "{{event.subject}}",
          "summary": "{{event.summary}}",
          "task_id": "{{event.task_id}}",
          "actor": "{{event.actor}}",
          "timestamp": "{{event.timestamp}}"
        }
        ```
      </Step>

      <Step title="Save" icon="check">
        Click **Save**.

        <Check>Test webhook receives a sample payload.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Register a webhook target" theme={null}
    ironcore-backup notification target create \
      --name chat-ops \
      --type webhook \
      --url "https://chat.<your-domain>/webhooks/ironcore-backup" \
      --method POST \
      --header "Authorization: Bearer <token>" \
      --body-template /etc/ironcore/templates/chat-ops.json
    ```
  </Tab>
</Tabs>

### Webhook Body Template

The standard webhook payload includes:

```json theme={null}
{
  "event_id": "evt-2026-05-21-001",
  "type": "backup.failed",
  "severity": "error",
  "subject": "Backup failed for vm/12345",
  "summary": "Task failed at chunk upload step: connection refused",
  "task_id": "UPID:backup:00C123:6643:abc...",
  "actor": "alice@polystack!backup-job",
  "resource": "ibs-primary:production/vm/12345",
  "timestamp": "2026-05-21T02:14:35Z",
  "task_url": "https://ibs.<your-domain>/#tasks/<task-id>",
  "links": {
    "task_log": "https://ibs.<your-domain>/api2/json/tasks/<task-id>/log"
  }
}
```

***

## Configure Metric Server Export

Push events to an external metric server for long-term retention, dashboards,
and cross-system correlation.

<Tabs>
  <Tab title="Deployment Console" icon="gauge">
    Open **Notifications** > **Targets** > **Add Metric Server**. Enter
    the endpoint URL and protocol (InfluxDB v2, Prometheus push gateway).
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="InfluxDB v2 metric target" theme={null}
    ironcore-backup notification target create \
      --name metrics-influx \
      --type metric-server \
      --protocol influxdb-v2 \
      --url https://metrics.<your-domain> \
      --org polystack \
      --bucket ibs-events \
      --token-file /etc/ironcore/influx-token
    ```
  </Tab>
</Tabs>

***

## Notification Groups

A **notification group** combines one or more targets with a rule that decides
when each target is dispatched.

<Tabs>
  <Tab title="Deployment Console" icon="gauge">
    <Steps titleSize="h3">
      <Step title="Open Groups" icon="external-link">
        Navigate to **Notifications** > **Groups**.
      </Step>

      <Step title="Create a group" icon="plus">
        Click **Add Group**. Enter:

        * **Name**: `ops-team`
        * **Description**: "Daily operations team"
      </Step>

      <Step title="Add targets and rules" icon="settings">
        For each target, define when it should receive events:

        * SMTP `ops-smtp` — every event
        * Webhook `chat-ops` — only `severity=error` or higher
        * Metric server — every event
      </Step>

      <Step title="Save" icon="check">
        Click **Save**.

        <Check>The group is selectable when configuring backup jobs, datastore alerts, and verification jobs.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Build a notification group" theme={null}
    ironcore-backup notification group create \
      --name ops-team \
      --description "Daily operations team"

    ironcore-backup notification group target add \
      --group ops-team \
      --target ops-smtp \
      --rule "*"

    ironcore-backup notification group target add \
      --group ops-team \
      --target chat-ops \
      --rule "severity in (error, critical)"

    ironcore-backup notification group target add \
      --group ops-team \
      --target metrics-influx \
      --rule "*"
    ```
  </Tab>
</Tabs>

***

## Rule Expressions

| Rule Atom                                                | Meaning                |
| -------------------------------------------------------- | ---------------------- |
| `severity in (error, critical)`                          | Severity match         |
| `type = backup.failed`                                   | Exact event type match |
| `type starts-with sync.`                                 | All sync events        |
| `actor matches "*!automation"`                           | Token-based actors     |
| `resource starts-with /datastore/ibs-primary/production` | Resource path          |
| Combined with `and` / `or`                               | Composite rules        |

Examples:

```
severity in (error, critical)
type starts-with verify. and severity in (warning, error, critical)
type = backup.failed and resource starts-with /datastore/ibs-primary/production
```

***

## Event Severity

| Severity     | When Used                                                   |
| ------------ | ----------------------------------------------------------- |
| **info**     | Routine events — backups completed successfully             |
| **notice**   | Notable but expected — large amount of pruned data          |
| **warning**  | Recoverable issue — verification stale, partial backup      |
| **error**    | Failed operation — failed backup, restore, or sync          |
| **critical** | Service-impacting — datastore offline, verification CORRUPT |

***

## Recommended Notification Configuration

| Group                 | Events                                                 | Targets                     |
| --------------------- | ------------------------------------------------------ | --------------------------- |
| **ops-team**          | All operational events                                 | SMTP, chat, metric server   |
| **on-call**           | `severity >= error`                                    | SMS-gateway webhook, chat   |
| **compliance**        | Permission changes, key rotation, verification results | SMTP, metric server         |
| **executive-sponsor** | `severity = critical` only                             | SMTP digest, weekly summary |

***

## Rate Limiting and Digesting

To prevent noisy notification storms during incidents:

| Setting              | Default            | Effect                                                  |
| -------------------- | ------------------ | ------------------------------------------------------- |
| Per-group rate limit | 60 events / minute | Excess events queued                                    |
| Daily digest         | Opt-in             | Aggregates routine `info` events into one daily summary |
| Deduplication window | 5 minutes          | Suppresses identical events within window               |

```bash title="Configure digest mode for routine events" theme={null}
ironcore-backup notification group update \
  --name compliance \
  --digest-rule "severity in (info, notice)" \
  --digest-frequency daily \
  --digest-time "09:00"
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Email not arriving" icon="mail">
    Check the **Tasks** panel for notification dispatch failures. Confirm
    SMTP credentials, port, and TLS settings. Look at the SMTP target's
    last result.

    ```bash theme={null}
    ironcore-backup notification target test ops-smtp \
      --recipient admin@<your-domain>
    ```
  </Accordion>

  <Accordion title="Webhook returns 401 or 403" icon="lock">
    The webhook target's authorisation header may be stale or wrong. Update
    the header value:

    ```bash theme={null}
    ironcore-backup notification target update chat-ops \
      --header "Authorization: Bearer <new-token>"
    ```
  </Accordion>

  <Accordion title="Too many notifications during incidents" icon="bell">
    Enable per-group rate limiting and the daily digest for `info` events.
    Consider routing low-severity events to the metric server only, not chat.
  </Accordion>

  <Accordion title="Missing events for a specific datastore" icon="search">
    Confirm the datastore is attached to a notification group. Check the
    datastore detail panel under **Alerts**.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Access Control" icon="lock" href="/services/ironcore-backup/admin-guide/access-control" color="#bf9667">
    Notification events generated by access changes
  </Card>

  <Card title="Verification and Validation" icon="check" href="/services/ironcore-backup/admin-guide/verification-validation" color="#bf9667">
    Verification jobs whose results dispatch to notification groups
  </Card>

  <Card title="Monitoring Jobs" icon="activity" href="/services/ironcore-backup/user-guide/monitoring-jobs" color="#bf9667">
    Operator view of jobs and notifications
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/ironcore-backup/admin-guide/troubleshooting" color="#bf9667">
    Admin-level diagnostic procedures
  </Card>
</CardGroup>