useWarmupSuite always aborts the run: "invalid 0-time total for suite Warmup"

[harini9803]

Summary

Enabling the warmup suite (?useWarmupSuite, or "Use Warmup Suite" in the developer menu) causes every run to abort before any real suite is measured, so the option is currently unusable.

Cause

SuiteRunner._recordTestResults() intentionally returns early for the warmup suite, so #suiteResults.total is never incremented and stays 0:

Speedometer/resources/suite-runner.mjs

Lines 127 to 130 in 86710fe

	async _recordTestResults(step, syncTime, asyncTime) {
	// Skip reporting updates for the warmup suite.
	if (this.#suite === WarmupSuite)
	return;

However _validateSuiteResults() is still called for the warmup suite, and it throws on a 0 total:

Speedometer/resources/suite-runner.mjs

Line 99 in 86710fe

this._validateSuiteResults();

Speedometer/resources/suite-runner.mjs

Lines 107 to 110 in 86710fe

	const suiteTotal = this.#suiteResults.total;
	const suitePrepare = this.#suiteResults.prepare;
	if (suiteTotal === 0)
	throw new Error(`Got invalid 0-time total for suite ${this.#suite.name}: ${suiteTotal}`);

These two behaviours contradict each other: the warmup suite is required to record nothing, but is then validated as though it must have recorded something. The result is:

Got invalid 0-time total for suite Warmup: 0

Note there are two unguarded call sites, so a guard at a single call site would leave the remote path broken:

• SuiteRunner._runSuite() — line 99
• RemoteSuiteRunner — line 209

Steps to reproduce

Reproduced on the released Speedometer 3.1. The main branch was verified by code inspection (see Cause above) rather than by running it.

1. Serve the Speedometer directory over HTTP, e.g. python3 -m http.server 8901
2. Open http://127.0.0.1:8901/?developerMode&useWarmupSuite&suites=TodoMVC-JavaScript-ES5&iterationCount=2
3. Click Start Test.

The run aborts before any real suite is measured. On 3.1 the summary shows "Error — One or more subtests produced no duration"; on main the same contradiction throws Got invalid 0-time total for suite Warmup: 0.

Regression

This appears to have been introduced by #406 (merged 2024-07-25), which added the zero-total validation while fixing #399. The warmup suite added in #124 deliberately records no results, so the two have never been compatible.

Additional context

In the released Speedometer 3.1 (before the SuiteRunner refactor) the same contradiction exists in resources/benchmark-runner.mjs, where _validateSuiteTotal() dereferences this._measuredValues.tests["Warmup"]. Since that entry is never created, it fails as a TypeError, and the UI reports the misleading message "One or more subtests produced no duration" rather than naming the warmup suite.

Speedometer 3.1 with useWarmupSuite enabled:

Suggested fix

Skip validation for the warmup suite. Because there are two call sites, an early return inside _validateSuiteResults() covers both and mirrors the existing guard in _recordTestResults():

_validateSuiteResults() {
    // The warmup suite's results are intentionally not recorded, so it has no total.
    if (this.#suite === WarmupSuite)
        return;
    ...

WarmupSuite is already imported in suite-runner.mjs, so no new import is needed.

I'm happy to open a PR if this approach looks right.