The alert notebook finds an alert's resolution wherever it sits, and a fleet-level store alert no longer reads as a stopped collector (#4755, #4756)

[erikdarlingdata]

Fixes #4755
Fixes #4756

Why

The alert notebook shows a status for each alert: "Resolved at T", "Fired again at T", "No resolution recorded", or "Unknown (not collected since T)". The first two came from one read of config_alert_log: the rows from the alert time to 24 hours after it, at most 200, newest first, with dismissed rows left out. That read missed a resolution in three cases:

• the Cleared row landed more than 24 hours after the alert (a store self-alert that clears the next day);
• the viewer's "Dismiss all" had marked the Cleared row dismissed;
• a fleet-level alert has no server id, so one 200-row cap covered every server, and newest-first drops the oldest rows, which is where a resolution sits.

The page then said "Fired again", or for a fleet-level store alert "Unknown (not collected since ...)". That last text says a collector stopped, but a store self-alert has no collector, and the process that evaluates it also serves the page (#4756).

The page never showed a false "Resolved" or a false "ongoing", so this is a display fix.

What changes

• DarlingAlertReader gets two targeted reads next to GetAlertHistoryPageAsync: GetFirstResolutionAfterAsync and GetFirstRefireAfterAsync. Each reads config_alert_log for the earliest row with alert_time strictly after the alert and no later than now, whose metric_name is in a list of names (= ANY($3)), one row, dismissed rows included. server_id = $4 is added only when the server id is known. The re-fire read also skips the matched row's own alert_time, so the alert the page is about is not its own re-fire. The row mapping is shared with the page read.
• AlertNotebookEndpoint.ResolutionRowNames(metric) lists the stored names that resolve a metric: every DarlingTriageEndpoint.ResolutionAliases entry whose canonical name is the metric, plus the recovery name of each NotebookRecoveryEdges pair whose firing name is the metric. The input metric is matched case-insensitively, but the names come back exactly as the product writes them, because the SQL compares with =. StatusFromHistory now takes its list of resolution names from the same method, so the read and the classification cannot disagree. Its logic and its existing tests are unchanged.
• AlertNotebookEndpoint.ReadStatusRowsAsync runs the two reads and returns at most one row: the re-fire read runs only when the resolution read found nothing, because arm 1 wins. The endpoint passes that row to ResolveStatusAsync. For the re-fire read the name is the matched row's stored spelling when there is a match, else the trimmed input. The old forward read (24 hours, 200 rows) is gone from the status path; the read that finds the alert row is unchanged.
• No store read happens when the answer cannot depend on a row: a blank metric, an unresolved server on a non-fleet alert (both answer "Unknown" before looking at any row), or an anchor at or after now.
• The alert notebook says collection stopped for a fleet-level store alert #4756: in ResolveStatusAsync, a fleet-level store alert with no resolution row and no later firing returns "No resolution recorded" instead of "Unknown (not collected since ...)". The four-arm doc comment says so. A per-server alert with an unresolved server still reads "Unknown".

Two small behavior notes:

• If the status reads themselves fail, the "Status lookup failed" note is still added. The status then comes from the collector-freshness arms alone. Before, the alert-match read's rows were also fed to the status arms, so a resolution inside the 15 minutes after the alert could still be found in that case; now none is.
• An alert whose server did not resolve used to run a fleet-wide 200-row read whose rows were then not used. It no longer runs, so a dead store no longer adds a second "Status lookup failed" note there; the match read's own "Alert-history lookup failed" note remains.

Test plan

• Darling.Tests builds with 0 warnings and 0 errors.
• New AlertNotebookResolutionReadTests (pure) and AlertNotebookResolutionReadLiveTests (live, against a local PostgreSQL 18.6 with TimescaleDB, in UTC): 29 tests, all pass. The live ones seed config_alert_log and check: a store self-alert whose Cleared row lands 30 hours later reads "Resolved at T"; so does one whose Cleared row was dismissed; with 250 newer rows from other servers inside the first 24 hours the resolution is still found; a server-scoped alert ignores another server's resolution row and reads its own (30 hours later, dismissed) when it has one; a later firing 30 hours on (dismissed) reads "Fired again"; the matched row is not its own re-fire; a lower-case metric in the link still finds the resolution and the re-fire; Cleared rows at or before the anchor do not count. The pure ones cover ResolutionRowNames (every alias and every recovery edge, exact spelling, never a firing name), the read's SQL shape (no dismissed filter, no window, LIMIT 1, server_id only when known), and "No resolution recorded" for a fleet-level store alert.
• Old shape proof: with ReadStatusRowsAsync temporarily put back to the 24-hour, 200-row, dismissed-excluded read and the fleet-level answer removed, 8 of those tests failed (Cleared 30 hours later, dismissed Cleared row, 250 newer rows, own resolution on a server-scoped alert, re-fire 30 hours later, matched row not its own re-fire, rows before the anchor, and the pure fleet-level test). Restored, all pass. The file was restored byte for byte before the commit.
• Neighbouring and gate classes against the local PostgreSQL, run before the last merge of origin/dev (AlertNotebook* including AlertNotebookEndpointTests and AlertNotebookCollectorFreshnessLiveTests, DocCommentHygieneTests, StoreSqlClockDisciplineTests, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, AlertReadFailureSurfaceTests, StartupCommandTimeoutTests, StorageCommandTimeoutTests), together with the 29 new tests: 511 tests, 0 failed.
• EXPLAIN (ANALYZE, BUFFERS) on 1.5 million config_alert_log rows with the (server_id, metric_name, alert_time) index: the fleet-level read is an index-only scan with 41 index searches (PostgreSQL 18 skip scan), 163 buffers; the server-scoped read is one index search, 6 buffers.
• Full Darling.Tests suite once, without DARLING_TEST_PG, after the merge of origin/dev: 17227 total, 0 failed, 0 errors, 1133 skipped (the live classes skip by design in that run), 1 not run. This run covers the three classes that read DarlingAlertReader.cs as text (DarlingMcpAlertToolsTests, LongRunningQueryExclusionKnobRungTests, UncorroboratedRouteStoreKnobTests).
• Not run: the fleet-level read on PostgreSQL earlier than 18, which has no skip scan (see below); Installer.Tests, which this change does not touch.

For the reader to double-check

• Three source-pin classes read DarlingAlertReader.cs as text and were not in the targeted batch above: DarlingMcpAlertToolsTests, LongRunningQueryExclusionKnobRungTests and UncorroboratedRouteStoreKnobTests. Their first pass over this change is the full suite, so check that result (or CI) for them.
• The fleet-level read has no server_id, so the index (server_id, metric_name, alert_time) serves it through PostgreSQL 18's skip scan (one search per server id). On an older PostgreSQL it would fall back to another plan. The store's bundled runtime is 18, but a store on an older PostgreSQL is worth a look.
• Lite has no alert notebook, so there is no Lite counterpart to change.
• Tests live in two new files; AlertNotebookEndpointTests.cs is untouched.

CHANGELOG

SECTION: Fixed
ENTRY:

• The alert notebook finds an alert's resolution even when it came late, was dismissed, or sat behind many newer alerts ([The alert notebook misses a resolution that lands more than 24 hours after the alert #4755]) - The notebook's status only looked at the first 24 hours after the alert, at most 200 rows, and skipped dismissed rows. A store alert that cleared more than 24 hours later, a Cleared row hidden by "Dismiss all", or a fleet-level alert whose resolution sat behind 200 newer alerts from other servers read "Fired again" or "Unknown" instead of "Resolved at ...". The status now reads the first resolution row and the first later firing of the metric directly, with no 24-hour limit and no row cap, dismissed rows included, and scoped to the alert's server when it has one.
• A fleet-level store alert with no resolution reads "No resolution recorded" instead of saying collection stopped ([The alert notebook says collection stopped for a fleet-level store alert #4756]) - A store self-alert with no resolution row and no later firing read "Unknown (not collected since ...)" in the alert notebook, which says a collector stopped. Store alerts have no collector, and the process that evaluates them serves the page, so it now reads "No resolution recorded".
  REF:
  [The alert notebook misses a resolution that lands more than 24 hours after the alert #4755]: The alert notebook misses a resolution that lands more than 24 hours after the alert #4755
  [The alert notebook says collection stopped for a fleet-level store alert #4756]: The alert notebook says collection stopped for a fleet-level store alert #4756