Skip to content

Emit ::warning annotations for skipped tests in GitHubActionsReport #9641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Amaury Levé (Evangelink) merged 1 commit into from
Jul 5, 2026
Merged

Emit ::warning annotations for skipped tests in GitHubActionsReport #9641

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 43 additions & 1 deletion ...tform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsAnnotationReporter.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ namespace Microsoft.Testing.Extensions.GitHubActionsReport;
/// <summary>
/// Emits a GitHub Actions <c>::error</c> workflow command for each failing test so the failure surfaces
/// both in the workflow run's Annotations tab and, when the source location can be resolved, on the
/// pull request's "Files changed" diff gutter.
/// pull request's "Files changed" diff gutter. Skipped tests are surfaced as title-only <c>::warning</c>
/// workflow commands so they are visible in the Annotations tab too.
/// </summary>
internal sealed class GitHubActionsAnnotationReporter :
IDataConsumer,
Expand Down Expand Up @@ -87,6 +88,14 @@ public async Task ConsumeAsync(IDataProducer dataProducer, IData value, Cancella

if (failure is null)
{
// Skipped tests carry no exception (and therefore no source location); surface them as a
// title-only '::warning' so intentionally or unexpectedly skipped tests are visible in the
// workflow Annotations tab alongside failures, rather than being silently absent.
if (nodeState is SkippedTestNodeStateProperty skipped)
{
await WriteSkippedAnnotationAsync(GetTestName(nodeUpdateMessage.TestNode), skipped.Explanation, cancellationToken).ConfigureAwait(false);
}

return;
}

Expand Down Expand Up @@ -155,6 +164,39 @@ private async Task WriteAnnotationAsync(string testName, string? explanation, Ex
GitHubActionsEscaper.EscapeData(message));
}

private async Task WriteSkippedAnnotationAsync(string testName, string? explanation, CancellationToken cancellationToken)
{
if (_logger.IsEnabled(LogLevel.Trace))
{
_logger.LogTrace("Skip received.");
}

string line = GetSkippedAnnotation(testName, explanation);

if (_logger.IsEnabled(LogLevel.Trace))
{
_logger.LogTrace($"Showing skip annotation '{line}'.");
}

// Prepend a newline for the same reason as the failure annotation: it guarantees the '::warning'
// workflow command starts at column 0 on its own line so GitHub recognizes it.
await _outputDisplay.DisplayAsync(this, new FormattedTextOutputDeviceData($"\n{line}"), cancellationToken).ConfigureAwait(false);
}

internal static /* for testing */ string GetSkippedAnnotation(string testName, string? explanation)
{
string message = explanation ?? GitHubActionsResources.NoSkipReasonFallback;
string title = string.Format(CultureInfo.InvariantCulture, GitHubActionsResources.SkippedAnnotationTitle, testName);

// Skipped nodes never carry a stack trace, so there is no file/line to pin the annotation to; a
// title-only '::warning' still surfaces in the workflow Annotations tab.
return string.Format(
CultureInfo.InvariantCulture,
"::warning title={0}::{1}",
GitHubActionsEscaper.EscapeProperty(title),
GitHubActionsEscaper.EscapeData(message));
}

private static string GetTestName(TestNode testNode)
=> TestNodeIdentity.GetTestName(testNode);

Expand Down
4 changes: 2 additions & 2 deletions src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ dotnet add package Microsoft.Testing.Extensions.GitHubActionsReport
This package extends Microsoft.Testing.Platform with:

- **Per-assembly log groups**: emits `::group::` / `::endgroup::` workflow commands so each test assembly's output is collapsed by default in the runner UI
- **Failure annotations**: emits an `::error` workflow command for each failing test so failures appear in the workflow Annotations tab and, when the source location can be resolved, on the pull request's "Files changed" diff gutter
- **Failure annotations**: emits an `::error` workflow command for each failing test so failures appear in the workflow Annotations tab and, when the source location can be resolved, on the pull request's "Files changed" diff gutter. Skipped tests are surfaced as title-only `::warning` annotations so they are visible in the Annotations tab too
- **Job summary**: appends a markdown roll-up (totals, failures, slowest tests) to the file pointed to by `GITHUB_STEP_SUMMARY`, which GitHub renders on the workflow run summary page
- **Slow-test notices**: emits a `::notice` workflow command for any test still running past a threshold (default 60 seconds)

Expand All @@ -25,7 +25,7 @@ The extension activates when the test run is on GitHub Actions (`GITHUB_ACTIONS=
|---|---|---|
| `--report-gh` | Master switch that turns the extension on (required, in addition to running on GitHub Actions) | off |
| `--report-gh-groups on\|off` | Per-assembly log groups | on |
| `--report-gh-annotations on\|off` | Failure annotations | on |
| `--report-gh-annotations on\|off` | Failure and skip annotations | on |
| `--report-gh-step-summary on\|off` | Markdown job summary | on |
| `--report-gh-slow-test-notices on\|off` | Slow-test notices | on |
| `--report-gh-slow-test-threshold <duration>` | Time before a slow-test notice is emitted; accepts a bare number of seconds or a unit suffix such as `90s`, `2m`, `1.5h` | 60s |
Expand Down
9 changes: 8 additions & 1 deletion ...rm/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/GitHubActionsResources.resx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,16 +62,23 @@
<value>GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</value>
</data>
<data name="AnnotationsOptionDescription" xml:space="preserve">
<value>Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</value>
<value>Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</value>
<comment>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}</comment>
</data>
<data name="AnnotationTitle" xml:space="preserve">
<value>Test failed: {0}</value>
<comment>{0} is the fully qualified test name.</comment>
</data>
<data name="SkippedAnnotationTitle" xml:space="preserve">
<value>Test skipped: {0}</value>
<comment>{0} is the fully qualified test name.</comment>
</data>
<data name="NoFailureMessageFallback" xml:space="preserve">
<value>The test failed without providing a failure message.</value>
</data>
<data name="NoSkipReasonFallback" xml:space="preserve">
<value>The test was skipped without providing a reason.</value>
</data>
<data name="StepSummaryOptionDescription" xml:space="preserve">
<value>Enable or disable writing a markdown job summary to the GITHUB_STEP_SUMMARY file. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</value>
<comment>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}{Locked="GITHUB_STEP_SUMMARY"}</comment>
Expand Down
14 changes: 12 additions & 2 deletions ...rosoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.cs.xlf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="AnnotationsOptionDescription">
<source>Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<source>Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<note>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}</note>
</trans-unit>
<trans-unit id="Description">
Expand Down Expand Up @@ -47,11 +47,21 @@
<target state="new">The test failed without providing a failure message.</target>
<note />
</trans-unit>
<trans-unit id="NoSkipReasonFallback">
<source>The test was skipped without providing a reason.</source>
<target state="new">The test was skipped without providing a reason.</target>
<note />
</trans-unit>
<trans-unit id="OptionDescription">
<source>Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</source>
<target state="new">Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</target>
<note />
</trans-unit>
<trans-unit id="SkippedAnnotationTitle">
<source>Test skipped: {0}</source>
<target state="new">Test skipped: {0}</target>
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="SlowTestNoticeTitle">
<source>Slow test: {0}</source>
<target state="new">Slow test: {0}</target>
Expand Down
14 changes: 12 additions & 2 deletions ...rosoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.de.xlf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="AnnotationsOptionDescription">
<source>Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<source>Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<note>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}</note>
</trans-unit>
<trans-unit id="Description">
Expand Down Expand Up @@ -47,11 +47,21 @@
<target state="new">The test failed without providing a failure message.</target>
<note />
</trans-unit>
<trans-unit id="NoSkipReasonFallback">
<source>The test was skipped without providing a reason.</source>
<target state="new">The test was skipped without providing a reason.</target>
<note />
</trans-unit>
<trans-unit id="OptionDescription">
<source>Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</source>
<target state="new">Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</target>
<note />
</trans-unit>
<trans-unit id="SkippedAnnotationTitle">
<source>Test skipped: {0}</source>
<target state="new">Test skipped: {0}</target>
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="SlowTestNoticeTitle">
<source>Slow test: {0}</source>
<target state="new">Slow test: {0}</target>
Expand Down
14 changes: 12 additions & 2 deletions ...rosoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.es.xlf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="AnnotationsOptionDescription">
<source>Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<source>Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<note>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}</note>
</trans-unit>
<trans-unit id="Description">
Expand Down Expand Up @@ -47,11 +47,21 @@
<target state="new">The test failed without providing a failure message.</target>
<note />
</trans-unit>
<trans-unit id="NoSkipReasonFallback">
<source>The test was skipped without providing a reason.</source>
<target state="new">The test was skipped without providing a reason.</target>
<note />
</trans-unit>
<trans-unit id="OptionDescription">
<source>Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</source>
<target state="new">Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</target>
<note />
</trans-unit>
<trans-unit id="SkippedAnnotationTitle">
<source>Test skipped: {0}</source>
<target state="new">Test skipped: {0}</target>
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="SlowTestNoticeTitle">
<source>Slow test: {0}</source>
<target state="new">Slow test: {0}</target>
Expand Down
14 changes: 12 additions & 2 deletions ...rosoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.fr.xlf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="AnnotationsOptionDescription">
<source>Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<source>Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<note>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}</note>
</trans-unit>
<trans-unit id="Description">
Expand Down Expand Up @@ -47,11 +47,21 @@
<target state="new">The test failed without providing a failure message.</target>
<note />
</trans-unit>
<trans-unit id="NoSkipReasonFallback">
<source>The test was skipped without providing a reason.</source>
<target state="new">The test was skipped without providing a reason.</target>
<note />
</trans-unit>
<trans-unit id="OptionDescription">
<source>Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</source>
<target state="new">Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</target>
<note />
</trans-unit>
<trans-unit id="SkippedAnnotationTitle">
<source>Test skipped: {0}</source>
<target state="new">Test skipped: {0}</target>
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="SlowTestNoticeTitle">
<source>Slow test: {0}</source>
<target state="new">Slow test: {0}</target>
Expand Down
14 changes: 12 additions & 2 deletions ...rosoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.it.xlf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="AnnotationsOptionDescription">
<source>Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions failure annotations. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<source>Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</source>
<target state="new">Enable or disable GitHub Actions annotations for failed and skipped tests. Valid values are 'on' (also accepts 'true', 'enable', '1') or 'off' (also accepts 'false', 'disable', '0'). Defaults to 'on' when running on GitHub Actions.</target>
<note>{Locked="on"}{Locked="off"}{Locked="true"}{Locked="enable"}{Locked="1"}{Locked="false"}{Locked="disable"}{Locked="0"}</note>
</trans-unit>
<trans-unit id="Description">
Expand Down Expand Up @@ -47,11 +47,21 @@
<target state="new">The test failed without providing a failure message.</target>
<note />
</trans-unit>
<trans-unit id="NoSkipReasonFallback">
<source>The test was skipped without providing a reason.</source>
<target state="new">The test was skipped without providing a reason.</target>
<note />
</trans-unit>
<trans-unit id="OptionDescription">
<source>Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</source>
<target state="new">Enable GitHub Actions report generator to emit workflow commands so test runs produce a first-class experience on GitHub Actions.</target>
<note />
</trans-unit>
<trans-unit id="SkippedAnnotationTitle">
<source>Test skipped: {0}</source>
<target state="new">Test skipped: {0}</target>
<note>{0} is the fully qualified test name.</note>
</trans-unit>
<trans-unit id="SlowTestNoticeTitle">
<source>Slow test: {0}</source>
<target state="new">Slow test: {0}</target>
Expand Down
Loading
Loading