Skip to content

Plan viewer and analyzer: compare row estimates the way PerformanceStudio does (#4627 follow-up) - #4698

Merged
erikdarlingdata merged 8 commits into
devfrom
feature/4627-row-estimate-nl-parallel
Sep 29, 2026
Merged

erikdarlingdata merged 8 commits into
devfrom
feature/4627-row-estimate-nl-parallel

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Closes #4627.

Builds on #4632 and #4688 (the row label). Ports erikdarlingdata/PerformanceStudio#594 and erikdarlingdata/PerformanceStudio#597 for the analyzer's row-estimate rules (5, 16 and 26), the plan node's row label, the plan edge colour and the minimap.

Why

EstimateRows is per execution and ActualRows is a total. ActualExecutions is a real per-execution count only on the inner side of a Nested Loops join. In a parallel zone it is the thread count. The analyzer's rules 5, 16 and 26, the node label, the edge colour and the minimap all divided the actual rows by ActualExecutions for every node, so outside a loop they understated the actual by the DOP. PerformanceStudio fixed this in erikdarlingdata/PerformanceStudio#597 with RowEstimateHelper; this ports it. Every one of them now compares the total ActualRows with RowEstimateHelper.GetExpectedRows(node): EstimateRows times ActualExecutions on a Nested Loops inner side, EstimateRows everywhere else.

What changes

Analyzer

  • New PerformanceMonitor.PlanAnalysis/RowEstimateHelper.cs, a faithful port with PerformanceStudio's member names: IsInnerSideOfNestedLoops, GetExpectedRows, GetRowAccuracyRatio (zero expected rows: 1.0 for zero actual, double.MaxValue otherwise). The numeric overload GetRowAccuracyRatio(actualRows, expectedRows) holds the arithmetic and the node overload calls it.
  • PlanAnalyzer.cs rules 5, 16 (outer-side line) and 26 use RowEstimateHelper, matching PerformanceStudio rule for rule. Rule 5: the "(N rows x M executions)" text prints only when isInnerSide && ActualExecutions > 1. Rule 16: the printed actual is the per-execution figure only when the outer child is on an inner side, else the total. Rule 26: rowGoalWorked = GetRowAccuracyRatio(node) <= 1.0.
  • Fixture Darling/Darling.Tests/Fixtures/OriginPlans/eager_index_spool_plan.sqlplan (DOP 8), copied from PerformanceStudio.
  • Tests Viewer4627RowEstimateHelperTests (17 facts) and Viewer4627RowEstimateRulesTests (5 facts).

Node label and formatter

  • New PlanRowAccuracy.FormatActualOfExpected(double actualRows, double expectedRows, IFormatProvider? provider = null).
  • PlanViewerControl.Rendering.cs label block: the text is PlanRowAccuracy.FormatActualOfExpected(node.ActualRows, RowEstimateHelper.GetExpectedRows(node)); the brush ratio is RowEstimateHelper.GetRowAccuracyRatio(node) with the 0.1 and 10.0 thresholds unchanged. The block's comments say what it does now.
  • PlanRowAccuracy.ActualRowsPerExecution, Ratio and FormatActualOfEstimate are deleted with the two members only they used (MaxSignificantDigits, IsFraction), because nothing calls them any more. The DoesNotContain pins on the old names in the viewer's source stay.
  • Viewer4684RowLabelTests is rewritten for the totals form (48 tests).
  • The totals are wider than the per-execution figures were. The row line still trims with an ellipsis at NodeWidth - 16 (134), so a label like 8,042,005 of 8,042,010 (100%) sits near the edge. The node already has a tooltip; none was added.

The formatter rule, word for word:

Print N0. If the whole percent computed from the printed numbers differs from the printed percentage, add the fewest decimals (fixed-point, never scientific, capped at 4) at which they agree. A non-zero value never prints as zero. If it would, print it fixed-point to its first significant digit.

Also: no exponent at any magnitude.

How it is implemented:

  1. The percent is (actualRows / expectedRows * 100).ToString("F0", provider), from the unrounded values. It shows only when expectedRows > 0; otherwise the label is "X of Y" with no percent. It is a whole percent, so a non-zero ratio can print (0%), for example 1 of 1,000,000 (0%). The non-zero rule applies to the row counts, not to the percent.
  2. The agreement search runs d from 0 to 4. A whole number (Math.Floor(v) == v) always prints N0; a non-whole number prints "N" + d. The printed text is parsed back with NumberStyles.Number and the provider (NumberStyles.Float rejects "2,983"). The search stops at the first d where the whole percent of the printed numbers equals the printed percent. A printed 0 divisor counts as a disagreement (NaN or Infinity never equals the percent). It stops at d = 4 either way, so at the cap a label can still contradict its percent.
  3. The non-zero rule runs inside the search, at every d. If a non-zero value's printed text parses to 0, it is reprinted with "N" + k, k = -(int)Math.Floor(Math.Log10(Math.Abs(v))). The cap does not apply to k: 0.000005 prints "0.000005".
  4. Only N formats print row counts. F0 prints only the percent. No G, R, E or bare ToString().

The old DOP 8 label for an accurate operator read 12 of 100 (12%): 12.5% is an exact tie, and .NET's F0 rounds a tie to even. At DOP 11 it read 9 of 100 (9%) and turned orange (ratio 0.0909). The pins assert these strings.

Pinned strings (invariant culture unless noted):

Case Label
Key Lookup, 1 actual row, 0.00964372 x 117 executions = 1.12831524 expected 1 of 1.128 (89%)
Not on an inner side, EstimateRows 0.000005, 0 actual 0 of 0.000005 (0%)
key_lookup_plan.sqlplan Node 4 (old label 0.0085 of 0.0096 (89%)) 1 of 1.128 (89%)
eager_index_spool_plan.sqlplan Node 1 (DOP 8, not inner) 609 of 2,983 (20%)
Same plan, Nodes 4 and 5 (inner, 613 executions) 609 of 613 (99%)
DOP 8 and DOP 11 scan under a Gather Streams, 100 actual = 100 estimate 100 of 100 (100%), brush ratio 1.0 (old: 12 of 100 (12%), 9 of 100 (9%) and orange)
Two nested Nested Loops inner sides, estimate 0.5, 1,000 executions, 400 actual 400 of 500 (80%)
Cap, 1 of 0.00012345 1 of 0.0001 (810045%)
Cap, 1 of 0.00001234 (the non-zero rule needs 5 decimals) 1 of 0.00001 (8103728%)
Cap, 0.000123 of 0.000456 0.0001 of 0.0005 (27%)
0.0000004 of 1 0.0000004 of 1 (0%)
de-DE: key lookup, eager Node 1, 0.6 of 0.75 1 of 1,128 (89%), 609 of 2.983 (20%), 0,60 of 0,75 (80%)
Others 0.60 of 0.75 (80%), 0.25 of 0.30 (83%), 0.99 of 1.01 (98%), 0.600 of 0.556 (108%), 0.0104 of 0.0096 (108%), 100,000 of 12.5 (800000%), 2.5 of 2 (125%), 1 of 0, 0 of 0, 0.00001 of 0
1e15 of 3e15 1,000,000,000,000,000 of 3,000,000,000,000,000 (33%)

Also pinned: a seeded sweep of 2,000 value pairs (1e-6 to 1e6, whole and fractional, a twentieth with 0 actual). Every label agrees with its percent or stopped at 4 decimals, none has an exponent, and no non-zero value prints as zero. The sweep reaches the hard paths (checked in a scratch run): 1,179 labels with decimals, 472 at 4 or more decimals, 222 past the cap by the non-zero rule. Extreme magnitudes (1e300, double.MaxValue, 5e-324) print no exponent. A source pin checks the label calls FormatActualOfExpected(node.ActualRows, RowEstimateHelper.GetExpectedRows(node)), takes its brush from GetRowAccuracyRatio(node), and no longer contains FormatActualOfEstimate, ActualRowsPerExecution or PlanRowAccuracy.Ratio(.

Edge colour and minimap

  • PlanEdgeColour.ForChild(PlanNode child, double divergenceLimit) compares the child's total ActualRows with RowEstimateHelper.GetExpectedRows(child). The numeric overload is now ForChild(bool hasActualStats, double actualRows, double expectedRows, double divergenceLimit): it takes expected rows instead of an execution count and gets its ratio from RowEstimateHelper.GetRowAccuracyRatio(actualRows, expectedRows).
  • GetLinkColorBrush in PlanViewerControl.Rendering.cs and the minimap's copy in PlanViewerControl.Minimap.cs hand the node to ForChild(child, limit), so no caller has to invent the expected figure.
  • Tests: Viewer4627EdgeColourTests pins the colours; Viewer4627EdgeCallerTests reads the two viewer files and pins that both pass the node and that no product code calls the numeric overload.

What the edge draws, before (the old ratio, actual / executions / estimate, worked out by hand) and after:

Case Before After
Scan under Gather Streams, actual = estimate = 1,000, DOP 2, 8 and 10 Neutral Neutral
Same, DOP 11 to 64 Blue Neutral
Same, DOP 128 LightBlue Neutral
Same, DOP 2000 FluoBlue Neutral
DOP 11, estimate 1,000, actual 20,000 (20x under) Neutral LightOrange
DOP 11, estimate 1,000, actual 200,000 LightOrange FluoOrange
DOP 11, estimate 1,000, actual 2,000,000 FluoOrange FluoRed
DOP 11, estimate 1,000, actual 50 (over) LightBlue Blue
eager_index_spool_plan.sqlplan Node 1, DOP 8, not inner (609 actual, estimate 2,983.02) Blue Neutral
key_lookup_plan.sqlplan Node 4, Nested Loops inner side (1 actual, 0.00964372 x 117 executions) Neutral Neutral

Left alone on purpose

  • Rule 32: Fix #594: per-execution row estimates on inner-loop nodes; Fix #595: no-grant memory display PerformanceStudio#597 did not touch it (it compares the raw total to the raw estimate), so it stays as it is.
  • Rules 9, 33 and 35: not touched here; a separate change ports them.
  • The edge tooltip (raw SSMS-style figures, no ratio), the Properties and tooltip rows, and McpPlanAnalysisFormatter (emits raw estimated_rows and actual_rows; PerformanceStudio's expected_rows addition has no counterpart here, and a new field would touch the MCP payload census).
  • deprecated/: no copy of this code, not edited.
  • There is no separate PlanAnalysis test project in this repo; the analyzer is covered by Darling.Tests (PlanSync* and the Viewer4627* classes).

REDs

Each regression was planted alone, the class run, and the source restored (a clean tree afterwards).

Analyzer (old analyzer restored with git checkout origin/dev -- PerformanceMonitor.PlanAnalysis/PlanAnalyzer.cs only): Viewer4627RowEstimateRulesTests Total 5, Failed 3.

  • Rule 5 at DOP 8 (join, estimate 2983.02, actual 609, 8 executions, under Gather Streams): old code emitted "Estimated 2,983 vs Actual 609 (76 rows x 8 executions) - 39x overestimated. The overestimate may have caused the optimizer to make poor choices." New code is silent (ratio 609 / 2983.02 = 0.204, inside the 0.1 to 10 gate).
  • Rule 16 (outer estimate 100, actual 100,000, 8 executions, inner Key Lookup 200,000 executions): old code failed the assertion for "actual 100,000 (1000x underestimate)" (by its arithmetic it printed 12,500 (125x)).
  • Rule 26 (estimate 100, without-row-goal 1000, actual 500, 8 executions): old code swallowed the warning (500 / 8 = 62.5 is under 100). New code emits "Row goal active: estimate reduced from 1,000 to 100".
  • No RED, by design: rule 5 on an inner-side non-lookup node ("50x underestimated ... 50 rows x 1,000 executions") passes on old and new code; rule 5 on the Key Lookup is silent on both.

Label (Viewer4684RowLabelTests, 48 tests):

# Planted Result
1 The old per-execution label block in Rendering.cs (git checkout 9ef1fb1a7 -- PerformanceMonitor.Ui/PlanViewerControl.Rendering.cs, formatter untouched) Failed 1: the source pin NodeLabel_IsBuiltFromTheTotalsAndTheSharedExpectation
2 A G format in PrintRows (value.ToString("G", provider)) Failed 29: the exponent pins (ExtremeMagnitudes_NeverPrintAnExponent 4, LargeNumbers_PrintEveryDigit 2), WholeNumbers_PrintN0 4, Decimals_AreAddedOnlyUntilTheNumbersAgreeWithThePercentage 4, Cap_StopsAtFourDecimals_ButANonZeroValueNeverPrintsAsZero 5, Culture_FollowsTheCallers 3, Culture_DefaultsToTheCurrentCulture, KeyLookupFixture_Node4_PrintsTheTotals, KeyLookupFixture_InGerman_PrintsADecimalComma, both gate vectors, EagerIndexSpoolFixture_PrintsTheTotals (Node 1) and the sweep
2b A G format on the first-significant-digit reprint line alone Failed 10: Cap_... 5, ExtremeMagnitudes_NeverPrintAnExponent 2, Decimals_... 1, GateVector_TinyEstimateWithNoActualRows_KeepsItsFirstSignificantDigit and the sweep
3 The non-zero reprint removed Failed 5: Cap_... 3 (1 of 0.00001234, 0.00001 of 0, 0.0000004 of 1), the tiny-estimate gate vector and the sweep
4 NumberStyles.Float in PrintedNumbersGive Failed 4: Culture_FollowsTheCallers (609 of 2983.02 in de-DE), Decimals_... (100000 of 12.5), EagerIndexSpoolFixture_PrintsTheTotals (Node 1's "2,983"), WholeNumbers_PrintN0 (1234.5 of 1000)
5 GetExpectedRows multiplying by ActualExecutions on every node, which is the old per-execution division seen from the expected side Viewer4684RowLabelTests Failed 3 (AccurateOperatorInAParallelZone_... at DOP 8 and DOP 11, EagerIndexSpoolFixture_PrintsTheTotals Node 1); Viewer4627* Total 60, Failed 16 (Viewer4627EdgeColourTests 11, Viewer4627RowEstimateHelperTests 2, Viewer4627RowEstimateRulesTests 3); Viewer4579Tests Total 20, Failed 0

The DOP 8 and DOP 11 facts test the label's inputs (RowEstimateHelper and the formatter), not the WPF block itself; the block is pinned by the source pin (RED 1). A live render pin was not attempted (CreateNodeVisual is private and needs an STA thread and a theme; nothing in either suite renders a node today).

Edge colour and minimap (Viewer4627*, 60 tests when these were run; the DOP 2000 row was added after):

# Planted Result
E1 ForChild(node, limit) divides ActualRows by ActualExecutions and compares with EstimateRows (the old arithmetic) Failed 12: ParallelZoneNode_ThatMetItsEstimate_IsNeutralAtAnyDop 5 (DOP 11, 12, 32, 64, 128), ParallelZoneNode_ThatMissedItsEstimate_IsColouredByTheMiss 4, EagerIndexSpoolPlan_Node1_AtDop8_IsNeutralWhereDividingByThreadsDrewBlue, SameNumbers_WithNoEnclosingNestedLoops_AreJudgedAgainstThePlainEstimate, NodeOverload_IsTheNumericOverloadFedRowEstimateHelperExpectedRows_ForEveryFixtureNode; Viewer4579Tests Failed 0
E2 GetLinkColorBrush in Rendering.cs hands the numeric overload the bare child.EstimateRows Viewer4627EdgeCallerTests Total 4, Failed 2: EdgeCaller_PassesTheNodeToPlanEdgeColour, NoProductCode_CallsTheNumericOverload
E3 The same in the minimap's GetLinkColorBrush in Minimap.cs The same 2 of 4
E4 GetExpectedRows never multiplies, so a Nested Loops inner side is judged against the bare estimate Viewer4627* Failed 15 (KeyLookupFixture_OnTheNestedLoopsInnerSide_KeepsItsNeutralKey, KeyLookupNumbers_OnTheInnerSideOfANestedLoops_KeepTheirNeutralKey, the 3 NestedLoopsWithinLoops_... facts, Viewer4627RowEstimateHelperTests 4, rule 5 on an inner side 1, Viewer4627Tests.InnerSideNestedLoopsNode_EdgeRatioMatchesFixtureAndAgreesWithLabel 5); Viewer4684RowLabelTests Failed 6

The DOP 2000 row of ParallelZoneNode_ThatMetItsEstimate_IsNeutralAtAnyDop was added after E1 ran. It pins the FluoBlue tier, which no earlier row reached; the old division reads 1/2000 = 0.0005 there.

Test plan

  • dotnet build of Darling.Tests and Lite.Tests on the merged head: 0 Warning(s), 0 Error(s).
  • Full Lite.Tests on the merged head: Total 5555, Failed 0, Skipped 0.
  • Full Darling.Tests on the merged head, without DARLING_TEST_PG (its live PostgreSQL tests skip): Total 16831, Failed 0, Skipped 1051, Not Run 1 (the runner's own count; the log names no test).
  • Viewer4684RowLabelTests: Total 48, Failed 0. Viewer4627*: Total 61, Failed 0. Darling PlanViewerCapabilityPinTests (source-scans Rendering.cs): Total 5, Failed 0. Lite PlanViewerCapabilityPinTests: Total 5, Failed 0.
  • REDs 1 to 5 and E1 to E4 above, each planted alone and restored.
  • git merge origin/dev done twice; the second brought The self-hosted log-events live test drives its own events, so a log rotation cannot hide them #4702 and the WaitRate read-count pin change. Both merges were clean, with nothing inside an analyzer rule.
  • Live PostgreSQL tests (DARLING_TEST_PG): not run. Nothing in this change touches storage.

CHANGELOG

This replaces the unshipped #4632 and #4688 entries, which describe the per-execution ratio and label this change removes.

SECTION: Fixed
ENTRY:
- **The plan viewer's row label, edge colours and minimap, and analyzer rules 5 and 26, no longer misread operators in a parallel zone** ([#4698]) - Actual rows in a plan are a total, the estimate is per execution, and the execution count is a real loop count only on the inner side of a Nested Loops join (elsewhere in a parallel zone it is the thread count). The node label now prints the totals against the expected rows, for example "609 of 2,983 (20%)" (on a Nested Loops inner side the expected rows are the estimate times the executions, so "609 of 613 (99%)"), with just enough decimals that the numbers agree with the percentage: a Key Lookup that ran 117 times for 1 row reads "1 of 1.128 (89%)". An operator that returned exactly its estimate used to read "12 of 100 (12%)" at DOP 8 and, from DOP 11, was drawn in the critical colour; it now reads "100 of 100 (100%)" and stays neutral. The edge colours and the minimap follow the same rule: accurate operators in a parallel zone at DOP 11 or more stop drawing Blue, LightBlue or FluoBlue edges, and real misestimates in a parallel zone now show their full tier (a 20x underestimate at DOP 11 draws LightOrange, where it drew no colour before). Nested Loops inner-side edges are unchanged. Rule 5 no longer reports a false overestimate on a parallel operator (a DOP 8 join with estimate 2,983 and 609 actual rows used to read "Estimated 2,983 vs Actual 609 (76 rows x 8 executions) - 39x overestimated" and is now silent), and rule 26 now reports "Row goal active: estimate reduced from 1,000 to 100" for a parallel scan that returned 500 rows, where it used to be swallowed.
REF:
[#4698]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/4698
SECTION: Changed
ENTRY:
- **Analyzer wording for Nested Loops outer sides and row estimate mismatches now counts rows the way the plan viewer does** ([#4698]) - Rule 16's Nested Loops outer-side detail now prints the total actual rows when the outer input is not on a Nested Loops inner side, for example "actual 100,000 (1000x underestimate)" where it used to print "actual 12,500 (125x underestimate)" at DOP 8. Rule 5's "(N rows x M executions)" text now appears only on a Nested Loops inner side.
REF:
[#4698]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/4698

…rmanceStudio does (#4627 follow-up)

Port PerformanceStudio's RowEstimateHelper (PerformanceStudio#594, fixed in #597) and move rules 5, 16 and 26 onto it. EstimateRows is per execution and ActualRows is a total; ActualExecutions is a real per-execution count only on the inner side of a Nested Loops join, and a thread count in a parallel zone, so the old actual-per-execution division understated the actual by DOP outside a loop. Rule 32 is unchanged: PerformanceStudio#597 left it comparing the raw total to the raw estimate.
…d rows, with just enough decimals to agree with its percentage (#4627)

ActualRows is a total, so the label sets it against RowEstimateHelper.GetExpectedRows instead of dividing it by ActualExecutions: an accurate operator at DOP 8 no longer reads 12%, and from DOP 11 it is no longer drawn critical. The new PlanRowAccuracy.FormatActualOfExpected prints N0, adds the fewest decimals (fixed-point, capped at 4) at which the printed numbers give the printed percentage, and never prints a non-zero value as zero. A Key Lookup that ran 117 times for 1 row reads 1 of 1.128 (89%).
…stimateHelper, so a parallel zone stops drawing false Blue edges (#4627 follow-up)

PlanEdgeColour.ForChild divided ActualRows by ActualExecutions for every node. In a parallel zone that is not on a Nested Loops inner side, ActualExecutions counts threads, so an accurate operator at DOP 11 or more read as a 1/11 overestimate and drew Blue, and a real underestimate was shrunk by the same factor. ForChild(PlanNode, limit) now takes the child's expected rows from RowEstimateHelper.GetExpectedRows, as PerformanceStudio does after PS#594 / #597, and the numeric overload takes expectedRows instead of an execution count. RowEstimateHelper.GetRowAccuracyRatio(actualRows, expectedRows) holds the ratio arithmetic, and the node overload calls it. GetLinkColorBrush and the minimap's copy pass the node.
…ore (#4627)

PlanRowAccuracy.ActualRowsPerExecution, Ratio and FormatActualOfEstimate, and the two members only they used
(MaxSignificantDigits and IsFraction), had no caller left once the node label, the edge colour and the minimap
moved onto RowEstimateHelper. The stale comment that said the edge and the minimap still called them goes too.
Viewer4684RowLabelTests keeps its DoesNotContain pins on the old names in the viewer's source.
…n the accurate parallel operator at DOP 2000 (#4627)

Two comments, one test name, one assertion message and two section headers now name RowEstimateHelper or say what
they hold instead of calling something "the helper". The accurate-operator edge pin gains a DOP 2000 row: the old
division read 1/2000 = 0.0005 there and drew FluoBlue, the third overestimate tier, which no row covered before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant