Report failed flaky tests as skip by PerfectSlayer · Pull Request #11931 · DataDog/dd-trace-java

PerfectSlayer · 2026-07-13T17:00:09Z

What Does This Do

This PR updates the test result collector to mark the failed flaky tests as skip.

Motivation

Avoid reporting "failures" to the Test Optimization product.

Additional Notes

It additionally fixes a typo for the release notes tag documentation.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: APMLP-1267

datadog-official

More details

The PR safely implements downgrading failed flaky tests to "skip" status when continueOnFailure=true, maintaining proper ordering and guard conditions to prevent double-tagging. Code is well-documented, handles the environment variable safely, and preserves existing behavior for normal test runs.

Was this helpful? React 👍 or 👎

_{📊 Validated against 3 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 7d30705 · What is Autotest? · Any feedback? Reach out in #autotest}

dd-octo-sts · 2026-07-13T17:20:30Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.97 s	13.93 s	[-0.5%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent	12.95 s	13.04 s	[-1.4%; -0.0%] (maybe better)
startup:petclinic:appsec:Agent	16.37 s	16.14 s	[-4.6%; +7.5%] (unstable)
startup:petclinic:iast:Agent	16.82 s	16.37 s	[-1.6%; +7.1%] (no difference)
startup:petclinic:profiling:Agent	16.48 s	16.71 s	[-2.4%; -0.3%] (maybe better)
startup:petclinic:sca:Agent	16.74 s	16.68 s	[-0.8%; +1.5%] (no difference)
startup:petclinic:tracing:Agent	16.07 s	15.61 s	[-1.2%; +7.2%] (no difference)

Commit: 94dbdf3c · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

cbeauchesne · 2026-07-13T17:21:07Z

+  /// {@link #tagSyntheticFailures()} are left untouched.
+  void tagFailuresAsSkipped() {
+    for (var testcase : testcases()) {
+      if (hasFinalStatusProperty(testcase) || !isFailed(testcase)) {


From a semantic point of view, we should treat all tests in that job as final_status=skip. The reason is that this status is really meant to describe the impact of the CI, regardless of its status.

For green CI, it does not change anything, as only test failures are looked. But the day we'll start to look over the test based to check tests that are ignored while been always green, it'll be be the canonical way of looking them : status=pass AND final_status=skip.

Though, as the present ticket is about the impact on Green CI, feel free to ignore my comment 😃 .

Let’s get it right now, it will save us time late.
I pushed follow up changes: 94dbdf3

we should treat all tests in that job as final_status=skip

Including normally passing tests?
From what I understand here you're saying it should behave the way CI is behaving: the ENV is to "continue on failure", so that if a failure halts CI, all previously passing tests would report as skipped? Correct?

I don't quite follow your second statement/comment. Why would we want to look at passed tests that have final_status=skip except to exclude such cases from SLAs (false positives)?

I see Bruce already addressed the first part to report them all as skipped 94dbdf3

Edit: My webpage was stale so this comment came in late 😄

@cbeauchesne Is there a way to check the results are following your expectations or do I need to merge it first?

Including normally passing tests?

Yes. It's surprising, but the idea behind CONTINUE_ON_FAILURE=true is : whatever the test will say, it'll be ignored.

On the long run, we'd like to monitor such test, because if ever they constantly succeed, it does not make sense to ignore their outcome, and we'd like to remove them form this quarantine. It's the purpose of final_status to store this information. The difference is subtile, but it'll allow us to treat those "healthy but quarantined" tests at scale.

@cbeauchesne Is there a way to check the results are following your expectations or do I need to merge it first?

Yes, the two job that have this flag are test_flaky and test_flaky_inst.

So far, I'd say it's not reported : https://app.datadoghq.com/ci/ci-cd/explorer?agg_m_a=count&agg_m_b=count&agg_m_source_a=base&agg_m_source_b=base&agg_q_a=%40ci.job.name%2C%40test.final_status&agg_q_b=%40ci.job.name%2C%40test.final_status&agg_q_source_a=base%2Cbase&agg_q_source_b=base%2Cbase&agg_t_a=count&agg_t_b=count&alias_a=master&alias_b=branch&ci_cd_explorer_tab=tests&currentTab=overview&eventStack=&ff_display=a%2Cb&fromUser=false&index=citest&query_a=test_level%3Atest%20%40git.repository.id_v2%3A%2A%2Fdd-trace-java%20%40ci.job.name%3A%2Aflaky%2A%20%40git.branch%3Amaster&query_b=test_level%3Atest%20%40git.repository.id_v2%3A%2A%2Fdd-trace-java%20%40ci.job.name%3A%2Aflaky%2A%20%40git.branch%3Abbujon%2Fci-results&refresh_mode=sliding&sort_m_a=%2C&sort_m_b=%2C&sort_m_source_a=%2C&sort_m_source_b=%2C&sort_t_a=%2C&sort_t_b=%2C&top_n_a=1000%2C10&top_n_b=1000%2C10&top_o_a=top%2Ctop&top_o_b=top%2Ctop&viz=query_table&x_missing_a=true%2Ctrue&x_missing_b=true%2Ctrue&start=1783508275994&end=1784113075994&paused=false

ah, it's because it runs only on master :

dd-trace-java/.gitlab-ci.yml

Line 1007 in 4242339

- *master_only

You can do a dummy commit to run it on your branch, my link above will tell if it's fine

On the long run, we'd like to monitor such test,

The Java Guild does. We have to widgets for:

Passing tests

Failing tests

You can do a dummy commit to run it on your branch

Thanks, I’ll give it a try. That will take some time to run / take care of.

This looks to be working. Reverting the CI changes then merging. Thanks for the link!

PerfectSlayer · 2026-07-15T12:26:56Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-15T12:27:01Z

View all feedbacks in Devflow UI.

2026-07-15 12:27:00 UTC ℹ️ Start processing command /merge

2026-07-15 12:27:11 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-07-15 13:16:06 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-07-15 14:17:47 UTC ℹ️ MergeQueue: This merge request was merged

PerfectSlayer added 2 commits July 13, 2026 18:54

feat(ci): Report failed flaky tests as skip

3c4d655

fix(ci): Fix label documentation

7d30705

PerfectSlayer requested review from a team as code owners July 13, 2026 17:00

PerfectSlayer added the type: feature Enhancements and improvements label Jul 13, 2026

PerfectSlayer requested review from randomanderson and removed request for a team July 13, 2026 17:00

PerfectSlayer added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jul 13, 2026

PerfectSlayer requested review from sarahchen6 and removed request for a team July 13, 2026 17:00

PerfectSlayer added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 13, 2026

PerfectSlayer requested a review from dougqh July 13, 2026 17:00

dougqh approved these changes Jul 13, 2026

View reviewed changes

datadog-official Bot reviewed Jul 13, 2026

View reviewed changes

PerfectSlayer changed the title ~~feat(ci): Report failed flaky tests as skip~~ Report failed flaky tests as skip Jul 13, 2026

mhdatie approved these changes Jul 13, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

cbeauchesne approved these changes Jul 13, 2026

View reviewed changes

feat(ci): Report all tests as skip instead

94dbdf3

cbeauchesne approved these changes Jul 15, 2026

View reviewed changes

PerfectSlayer force-pushed the bbujon/ci-results branch from 566c143 to 94dbdf3 Compare July 15, 2026 12:26

gh-worker-dd-mergequeue-cf854d Bot merged commit 8d9ad4d into master Jul 15, 2026
1179 of 1183 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the bbujon/ci-results branch July 15, 2026 14:17

github-actions Bot added this to the 1.65.0 milestone Jul 15, 2026

Uh oh!

Conversation

PerfectSlayer commented Jul 13, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhdatie Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer commented Jul 15, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

PerfectSlayer commented Jul 13, 2026 •

edited by atlassian Bot

Loading

dd-octo-sts Bot commented Jul 13, 2026 •

edited

Loading

mhdatie Jul 13, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 15, 2026 •

edited

Loading