Skip to content

Disable resilience handler in test builders - #1504

Draft
afscrome wants to merge 5 commits into
mainfrom
afscrome-disable-test-resilience
Draft

Disable resilience handler in test builders#1504
afscrome wants to merge 5 commits into
mainfrom
afscrome-disable-test-resilience

Conversation

@afscrome

@afscrome afscrome commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Theory

The AddStandardResilienceHandler() calls in TestDistributedApplicationBuilder and AspireIntegrationTestFixture add automatic HTTP retry logic to all test HTTP clients. The hypothesis is that this masks genuine race conditions in integration tests: when a resource isn't truly ready to serve requests, the resilience pipeline silently retries until it succeeds — turning what should be a guaranteed, reproducible failure into an intermittent, flaky test.

Why this is particularly dangerous

The specific race being hidden is between:

  1. A container resource starting up (image pull → container start → application boot → HTTP server ready)
  2. An HTTP client firing retries against that resource while it's still starting

This race is almost never visible locally because:

  • Your local hardware is typically faster than a CI agent
  • Docker images are already pulled (or even have running containers to reuse)
  • The resource is ready long before the first retry exhausts

In CI, however, the agent has to pull the image from scratch on every run. This means the resource startup takes significantly longer, and the resilience handler's retry budget can be exhausted before the HTTP server is ready — which is exactly what we see in the failures below. Without retries, the test fails fast and visibly every time; with retries, it passes most of the time but occasionally flakes when the machine is slow or under load.

The correct fix is not retries — it's a proper health check so that WaitForResourceHealthyAsync(...) means "the HTTP server is actually ready", not just "the container process started".

This PR removes the resilience handler from both test helpers and sets ASPIRE_TESTING_DISABLE_HTTP_CLIENT=true to prevent Aspire from adding its own HTTP client reliability defaults during tests.

Evidence

After making that removal, CI immediately surfaced 6 real failures across Ubuntu runners:

Test Resource hit Failure
Hosting.Azure.Extensions.TestsAppHostTests.ResourceStartsAndRespondsOk blobs-explorer (Azure Storage Explorer) HttpRequestExceptionHttpIOException: response ended prematurely
Hosting.DbGate.TestsAppHostTests.ResourceStartsAndRespondsOk dbgate Same
Hosting.MongoDB.Extensions.TestsAppHostTests.ResourceStartsAndRespondsOk dbgate (via .WithDbGate()) Same
Hosting.Redis.Extensions.TestsAppHostTests.ResourceStartsAndRespondsOk dbgate (via .WithDbGate()) Same
Hosting.PostgreSQL.Extensions.TestsAppHostTests.ResourceStartsAndRespondsOk dbgate (via .WithDbGate()) Same
Hosting.Java.TestsJavaHostingComponentTests.AppResourceWillRespondWithOk("containerapp") Java container app Same

All 6 tests call WaitForResourceHealthyAsync(...) before issuing GET /, but none of the affected resources had an explicit HTTP health check registered — so "healthy" state was being reached before the HTTP server was actually ready to accept connections.

Changes

Remove resilience handlers from test helpers

  • tests/CommunityToolkit.Aspire.Testing/TestDistributedApplicationBuilder.cs — removed .AddStandardResilienceHandler() and set ASPIRE_TESTING_DISABLE_HTTP_CLIENT=true to ensure aspire doesn't add the same resilience back.
  • tests/CommunityToolkit.Aspire.Testing/AspireIntegrationTestFixture.cs — same

Add explicit HTTP health checks

  • DbGate (src/CommunityToolkit.Aspire.Hosting.DbGate/DbGateBuilderExtensions.cs) — added .WithHttpHealthCheck("/health", ...). DbGate's upstream API exposes a dedicated /health endpoint, making this the correct readiness signal.
  • Azure Storage Explorer (src/CommunityToolkit.Aspire.Hosting.Azure.Extensions/AzureStorageExplorerBuilderExtensions.cs) — added .WithHttpHealthCheck("/", ...). Upstream Helm/k8s manifests use / for readiness/liveness probes; no dedicated health endpoint exists.

Out of scope

The Java containerapp failure is handled by #1500 and is intentionally not fixed here.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://github.com/ghraw/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1504

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://github.com/ghraw/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1504"

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
afscrome and others added 3 commits August 2, 2026 11:08
Set ASPIRE_TESTING_DISABLE_HTTP_CLIENT via IConfiguration on the test
builder rather than as a process-wide environment variable. This is
cleaner and avoids potential cross-test pollution from static state.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DbGate 6.1.4 has no /health route. In Docker mode it serves static
files at /, so that is the correct readiness endpoint. Using /health
returned 404, causing WaitForResourceHealthyAsync to time out.

This fixes the transitive timeout in all tests that depend on dbgate:
MongoDB.Extensions, PostgreSQL.Extensions, Redis.Extensions,
SqlServer.Extensions and MySql.Extensions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants