Skip to main content
SRE Agent works with PagerDuty in both directions. It can page a PagerDuty service when an alert needs a human, and it can receive PagerDuty’s incident events. The two halves are independent, and when both are set up an incident stays in step on both sides.

​
Page PagerDuty from SRE Agent

1

Create an integration in PagerDuty

On the PagerDuty service that should receive the incidents, add an Events API v2 integration and copy its integration key (also called the routing key).
2

Add a PagerDuty target

In SRE Agent go to Settings, Outbound Alerting and click Add Config. Enter a Name, set Provider Type to PagerDuty, paste the key into Routing Key (PagerDuty) and click Save Config.
3

Choose when it pages

Either open an alert and click Page via followed by the target name, or click Add Rule under Automatic Escalation Rules. A rule picks the alerts to page by source, severity or labels, and Escalate after (minutes) sets the wait, or leave it empty to page at once. Step order lets you chain targets so the next step waits for an acknowledgement.
Severity maps across as critical to critical, high to error, medium to warning and anything else to info. The alert’s title, description, labels and a link back to SRE Agent go with the page.

​
What stays in step

When an alert has been paged to PagerDuty, status changes carry over: A change is sent once, when the status really changes. A repeat delivery of an alarm that is already acknowledged or resolved sends nothing. If an alert was paged to more than one PagerDuty service, a status that came from PagerDuty is forwarded to the others, so none of them keeps escalating.

​
Receive PagerDuty incidents in SRE Agent

1

Copy your webhook URL

Open Integrations, Webhooks and find the PagerDuty card. Copy the endpoint URL, which looks like https://sreagent.app/webhooks/pagerduty/<token>.
2

Add a webhook in PagerDuty

In PagerDuty go to Integrations, Generic Webhooks (v3) and click New Webhook. Paste the URL into Webhook URL and subscribe to incident.triggered, incident.acknowledged and incident.resolved. Other event types are accepted and ignored.
3

Save the signing secret

Copy the webhook’s signing secret, paste it into Signing secret on the PagerDuty card in SRE Agent and click Save secret. The secret needs at least 16 characters. Deliveries are then verified against their signature. Without a secret, the token in the URL is the only check, unless you have an approved runbook that runs automatically. Then PagerDuty deliveries must be signed, and unsigned ones answer 401. Replace secret and Clear manage it later.

​
What you see

An incident that started inside PagerDuty (from a monitoring integration there) becomes an alert with the source pagerduty. It is handled like any other new alert, with a Slack message, an investigation and your rules. For an incident that SRE Agent opened itself, the events act on the existing alert and never create a second one. The alert page gains a View in PagerDuty button once PagerDuty reports the incident. If the alert has a Slack thread, an acknowledgement is noted once in it (“Acknowledged in PagerDuty by Jane Doe”) and a resolve posts the usual “Alert Resolved” reply.
An alert that came from PagerDuty is never paged to a PagerDuty target, because its responders are already paged. Slack, webhook and on-call targets still page. If you press Page via on such an alert, SRE Agent tells you why nothing was sent.

​
Troubleshooting

Check that the subscription includes the three incident events. Other event types answer 200 and do nothing by design. PagerDuty’s webhook delivery log shows what it sent.
A request with a wrong or missing signature answers 401. A missing signature is refused whenever you have an approved runbook that runs automatically. The signing secret saved in SRE Agent does not match the one PagerDuty shows for the subscription. Replace the secret on the PagerDuty card.
The two sides match on the incident key, which starts with sre-agent-. That is only true when SRE Agent opened the incident through a PagerDuty target. An incident created by hand in PagerDuty becomes its own alert. Also check that the webhook URL belongs to the same organization that owns the alert.
The button appears once PagerDuty delivers an event for the incident, so the subscription must include incident.triggered.
SRE Agent forwards the change using the record of the original page. An incident paged by a target that has since been deleted cannot be closed from here, so close it in PagerDuty.