Repository navigation
Alert history records whether each notification channel delivered or failed (#4750) - #4779
Merged
Merged
Conversation
…failed (#4750) An alert that reached one webhook and failed on another was stored as a plain delivery: the route record listed the channels the route resolved to, and one success stood for the whole fan-out. The webhook fan-out now keeps each channel's outcome, the email send core adds the email channel when an SMTP send was attempted, and the route record on the alert-history row carries "delivered", "failed" or "not attempted" per destination. Only that word is stored: a webhook error can carry its endpoint's URL, so the reason stays in send_error, which does not change. Rows written before this read as a null outcome. Both the headless service (engine alerts and analysis findings) and Lite write the record, and the get_alert_history tool on each edition returns it as `outcome`. Lite has no routes, so its record names the channels the parent settings resolve to. Part of #4750 (the per-channel record; the failing-channel alert follows separately).
…xt or endpoint address (#4750)
erikdarlingdata
marked this pull request as ready for review
September 29, 2026 10:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4750 (the per-channel record; the failing-channel alert follows separately)
Why
With two or more channels, an alert that reached one and failed on another was recorded as delivered, with no
send_error. The route record on the history row listed the channels the route resolved to, not the ones that succeeded, so a rotated Slack URL or a 5xx from one endpoint showed nowhere in the history while a sibling kept delivering.What changes
AlertRouteDestinationDtogets a trailingOutcome(null by default), and a newAlertRouteOutcomesconstants class holds the persisted spellings:delivered,failed,not attempted. A row written before this reads as a null outcome throughTryReadRouteandTryDeserialize, the same way a row before the route record reads a nullRoute.WebhookFanoutResultandEmailFanoutResultget a trailingChannelOutcomesmap keyed by theNotificationRouter.*Channelnames. The webhook fan-out fills it inRecord(channel, error)and returns it on the delivered and failed results.EmailSendCore.TrySendAsyncmerges it and addsEmailonly when an SMTP send was delivered or failed.NotificationRouteDecision.ToDto(outcomes): a null map leaves the outcome null. With a map, delivered givesdelivered, failed givesfailed, and anything else or absent (a cooldown, a fold, a channel never reached) givesnot attempted. Only channels that resolved to a destination are listed, as before.send_errorand the cooldown columns do not change.DarlingAlertDeliverer(engine alerts),DarlingFindingAlertSender(analysis findings) and Lite'sEmailAlertService.TrySendAlertEmailAsync(whichSendFindingAlertAsyncgoes through).Default), each with its outcome.SendFindingSummaryAsyncis left as it was, the same as Darling's summary path, which does not attach a route record either.get_alert_historyreturnsoutcomeon each destination in both editions. I greppedMcpPayloadContractCensusTestsand the other MCP pins in both test projects forget_alert_historyanddestinations; nothing pins the destination shape, so the field was added. The now-false comment inLite/Mcp/McpAlertTools.cs(that its deliverer never writes the member) is rewritten. No toolDescriptionstring changed.EmailSendCore.cslines 316-330 (the SMTP client setup) are untouched.Test plan
AlertRouteChannelOutcomeTests(Darling, 4 tests),AlertRouteOutcomeContractTests(6) andNotificationRoutingTests: 25 passed in total. They cover a failing 500 beside a delivering channel (row is Sent,send_errornull, route recorddeliveredandfailed), no HTTP text or endpoint address in the stored JSON, email joining the record, an email held by its cooldown readingnot attempted, the spellings, a row withoutOutcomestill reading, and theoutcomein the MCP projection.CapturingWebhookEndpointtakes an optional status code.AlertRouteChannelOutcomeTests(2 tests): Genericdeliveredand Slackfailed, both sourceDefault, no error text; a muted alert stores no route record. 2 passed.HTTP 500, the endpoint URL and the loopback address all absent from the context JSON on the history row) now run and pass:AlertRouteChannelOutcomeTests4 of 4 andAlertRouteOutcomeContractTests6 of 6, after mergingorigin/dev.WebhookAlertService.cs,EmailSendCore.cs,DarlingAlertDeliverer.cs,DarlingFindingAlertSender.csandLite/Services/EmailAlertService.csrestored fromorigin/devtogether (the other source files kept so the tests compile; both test projects rebuilt with 0 warnings), DarlingAlertRouteChannelOutcomeTestsfailed 4 of 4 on the outcome assertions (expecteddeliveredorfailed, got null):AChannelThatFailsBesideOneThatDelivers_IsNamedOnTheRow_WhileTheRowStillReadsDelivered,WhenNoChannelDelivers_TheRowKeepsItsSendError_AndTheRouteRecordStoresNoErrorText,AnEmailThatWasSent_JoinsTheRecord_BesideTheWebhooksOutcomesandAChannelHeldBackByItsCooldown_IsListedAsNotAttempted.AlertRouteOutcomeContractTests(6) stayed green, as expected: it checks the destination shape, the spellings and the MCP projection, none of which sit in the restored files. LiteAlertRouteChannelOutcomeTestsfailed 1 of 2:AWebhookThatFailsBesideOneThatDelivers_IsNamedOnTheStoredRow(no route record on the row); the muted-alert test passed, because dev writes no route record either. The files were put back withgit checkout HEAD -- <files>and the restore was never committed.AlertDeliveryChannelTests,McpPayloadContractCensusTests,McpPageContractTests,NotificationRoutingTestsand all seven classes inDarlingMcpAlertToolsTests.cs(there is no class of that name; the file holdsDarlingMcpAlertToolsSurfaceAndSqlTests,SetMuteRuleEnabledTests,UpdateMuteRuleTests,CreateMuteRuleCoreTests,DeleteMuteRuleCoreTests,WebMuteRuleEndpointFlowTestsandDarlingMcpAlertToolsLivePostgresTests), run together with the two route classes: 335 run, 0 failed, 4 skipped (the live PostgreSQL tests, which skip without a test connection string). LiteAlertDeliveryChannelTests,McpPageContractTestsandAlertRouteChannelOutcomeTests: 41 run, 0 failed. Lite.Tests has noMcpPayloadContractCensusTests.origin/dev(two commits, --configure-network keeps the web listener's tls and oidc settings, and any other key it does not ask about (#4743) #4775 and Daily rollups: days an earlier hourly repair left short are rebuilt once after the upgrade (#4716) #4763) with no conflicts; both test projects rebuilt with 0 warnings and 0 errors. Full suites, run once each with no test connection string set: Darling.Tests 17208 total, 0 failed, 1123 skipped (1121 need a live PostgreSQL test connection string; one needs symlink privileges this machine lacks and one is a Windows-specific TLS case), 1 not run (a test marked explicit); Lite.Tests 5599 total, 0 failed, 0 skipped. No code fix was needed.CHANGELOG
SECTION: Fixed
ENTRY:
delivered,failedornot attemptedper channel, in both Darling and Lite, andget_alert_historyreturns it asoutcome. No error text is stored, because a webhook error can name its endpoint URL;send_erroris unchanged.REF:
[One failing webhook channel stays silent while the alert row reads delivered #4750]: One failing webhook channel stays silent while the alert row reads delivered #4750