Email Service Bounce Rate High

A bounced email is one the recipient's mail server rejected outright. Enough of them erodes the reputation of Azure's mail-sending infrastructure and of this resource's sender domain — the expectation in PLA-7799 is that a high bounce rate gets a reaction before Microsoft throttles or restricts the resource. This alert fires when at least 10% of the trailing hour's email delivery outcomes ended in bounces and at least five emails bounced in that hour, with both conditions holding for five minutes.

Both floors matter. Send volume is currently low (tens of emails per hour), so a pure rate is meaningless — one bounce out of two sends is a "50% bounce rate" that says nothing. The count floor keeps the page for real bursts; the rate floor keeps it from firing when a burst of bounces rides a much larger burst of sends (5 bounces out of 500 is 1% — bad, but not this alert's job). The alert's value is the bounce rate; the runbook's diagnosis steps give the exact counts.

Live examples (2026-11-06)

Over the seven days to 2026-11-06 (Azure-side exact counts, from the Azure Monitor API), the thresholds fire on exactly three hours:

• 2026-10-30 ~17:00 UTC — 15 bounces of 46 outcomes (33%).
• 2026-11-01 ~15:00 UTC — 7 bounces of 66 outcomes (11%).
• 2026-11-02 ~11:00 UTC — 5 bounces of 28 outcomes (18%).

Every other hour of that week stayed at 4 bounces or fewer — under the count floor even when its rate passed 10% (low-volume hours run 1 bounce out of a handful of sends). Retroactive hourly buckets anchor to the query time, so exact hour boundaries shift a few minutes between queries; the bursts themselves are unambiguous.

Meaning

The numerator, azure_acs_bounced_hourly_deliverystatusupdate_count (probe acs-email-bounced-hourly, added with this runbook), is Azure's own trailing-hour count of bounced emails: a single PT1H bucket over [now-1h, now], filtered to MessageStatus eq 'Bounced', recomputed server-side on every scrape. The denominator, azure_acs_delivery_hourly_deliverystatusupdate_count (probe acs-email-delivery-hourly, also added with this runbook), is the same aggregate unfiltered — a $filter-less query makes Azure roll every delivery outcome into one exact total (verified against the Azure Monitor API on 2026-11-06).

Both sides count DeliveryStatusUpdate events, one per recipient — not API requests. The SendMail request counts (PLA-7779's quota series) count one per API call, including 4xx and 429 responses that never produce mail, and a multi-recipient send is one request but N delivery events; the outcomes denominator keeps the ratio per-recipient on both sides so it stays a true bounce rate if the API usage changes. One caveat on the denominator: it includes every outcome, Suppressed among them — addresses Azure skipped because they bounced before — so a bad list that keeps being mailed shows up partly as suppressions, which slightly under-reports its bounce rate. The two sides are also lag-aligned: delivery-status events land asynchronously — minutes after the send — and the numerator is a subset of the denominator, so a present numerator always implies a present denominator. No ratio spike from sends aging out before their bounces (bounces and their sibling outcomes age out together).

Both gauges are absent while their window is empty (Azure returns null buckets for empty windows), so the alert resolves on the first evaluation below either floor — for: 5m delays firing, not resolution (no keep_firing_for is set). The sustain filters momentary downward data revisions, same rationale as the hourly-quota alert; a real burst stays above both floors for most of an hour (Azure data lands two to three minutes late either way).

The expression uses plain max(...) rather than the sibling alerts' max by (probe): the two sides carry different probe labels, and the result must be a single label-less series anyway. That max also collapses the two exporters — dev-par01 and prd-par01 probe the same shared Azure resource — so there are no cluster or environment matchers and one page per event. Neither gauge is a window rebuilt from scrape samples; the dimensioned dashboard series (acs-email-delivery) are timestamp-less gauges of the newest one-minute bucket and are documented as unsafe to alert on — see the exporter overlays for the 2026-10-30 measurement of how badly rebuilt windows overcount.

Impact

No mail is lost that was not already lost — the bounces happened. The damage is forward-looking: reputation decay on Azure's mail servers and the sender domain means future legitimate email lands in spam folders or gets throttled, and sustained high bounce rates risk Microsoft restricting the resource outright. This is the deliverability counterpart of the quota alerts: 02-email-service-minute-quota-high.md, 03-email-service-hourly-quota-high.md, and 04-email-service-send-throttled.md cover sending-volume limits; this one covers what the recipients' mail servers think of what we send.

Diagnosis

1. Confirm the burst and its exact shape server-side (exact counts, per hour):

   az monitor metrics list \
     --resource "/subscriptions/2849fdec-c143-421e-81e2-5a4b637a3343/resourceGroups/PROD/providers/Microsoft.Communication/communicationServices/cortiemailservices" \
     --metric DeliveryStatusUpdate --aggregation Count --interval PT1H --offset 3h \
     --filter "MessageStatus eq 'Bounced'" -o table
   

2. Split the bounces by hard vs soft and by SMTP status code — hard bounces (invalid addresses, non-existent mailboxes) point at list quality; soft bounces (greylisting, full mailboxes, policy rejections) at recipient-side or authentication issues:

   az monitor metrics list \
     --resource "/subscriptions/2849fdec-c143-421e-81e2-5a4b637a3343/resourceGroups/PROD/providers/Microsoft.Communication/communicationServices/cortiemailservices" \
     --metric DeliveryStatusUpdate --aggregation Count --interval PT1H --offset 3h \
     --filter "MessageStatus eq 'Bounced' and IsHardBounce eq '*' and SmtpStatusCode eq '*'" \
     -o table
   

   The table prints one row per hour and dimension combination — Ishardbounce, Messagestatus, Smtpstatuscode, and the Count. In the 7-day sample behind the live examples every bounce was IsHardBounce=True with SMTP 550 (mailbox unavailable): a list-quality signature, not a policy problem.
3. Check which sender domain the bounces hit — dimensionSenderdomain on the Azure Communication Services Email Delivery dashboard's delivery-status panels tells you whether one domain's reputation is degrading.
4. Identify the sending workload: the dashboard's send-volume panels show what was sending in the burst's window, and its log panels (prd-par01's email operation logs) show the actual recipients. The live examples above were all single-hour bursts, so look for a batch job or a one-off mailing in that hour, not a steady leak.

Mitigation

1. Find the workload and recipient list behind the bounced emails (Diagnosis step 4).
2. Stop or slow that sender — the trailing-hour window drains as the burst ages out, and the alert resolves on the first evaluation below either floor; there is no reset to wait for.
3. Clean the list before resuming: remove hard-bounced addresses (they will never deliver), and check where the list came from — bought or stale lists produce exactly this signature.
4. If the bounces were policy rejections (SPF/DKIM/DMARC failures in the SMTP status codes from Diagnosis step 2), fix the domain's email authentication before resuming.

Verify

The alert resolves on the first evaluation where the bounce count is under 5 or the rate is back under 10% — for: 5m delays firing, not resolution (no keep_firing_for is set). To watch the window recede in Mimir:

max(azure_acs_bounced_hourly_deliverystatusupdate_count{probe="acs-email-bounced-hourly"})

Cross-check against Azure:

az monitor metrics list \
  --resource "/subscriptions/2849fdec-c143-421e-81e2-5a4b637a3343/resourceGroups/PROD/providers/Microsoft.Communication/communicationServices/cortiemailservices" \
  --metric DeliveryStatusUpdate --aggregation Count --interval PT1H --offset 1h \
  --filter "MessageStatus eq 'Bounced'" -o table

Metric-availability note