[Bug]: evaluate() resolves instead of rejecting on Firefox when the page function throws a falsy value

[ayaangazali]

Version

1.64.0-next (af74c938e), Firefox 155.0 / playwright firefox v1544

Steps to reproduce

If a page function throws any falsy value, Firefox resolves the evaluate with undefined instead of rejecting. Chromium and WebKit reject. The page-side error disappears with no warning.

import { test } from '@playwright/test';

test('falsy throw is swallowed', async ({ page }) => {
  // This should fail. On Firefox it passes.
  await page.evaluate(() => {
    const cfg = window.__cfg;
    if (!cfg)
      throw '';
    return cfg;
  });
});

Expected

Rejects on every browser, as chromium and webkit do.

Actual

chromium and webkit reject. firefox resolves with undefined, so the test above passes.

Measured on all three, every falsy value:

thrown	chromium	webkit	firefox
null	rejects null	rejects null	resolves undefined
undefined	rejects undefined	rejects undefined	resolves undefined
0	rejects 0	rejects 0	resolves undefined
''	rejects ``	rejects ``	resolves undefined
false	rejects false	rejects false	resolves undefined
NaN	rejects NaN	rejects NaN	resolves undefined
-0	rejects -0	rejects -0	resolves undefined
0n	rejects 0n	rejects 0n	resolves undefined
5	rejects 5	rejects 5	rejects 5
new Error('')	rejects	rejects	rejects

Also affects locator.evaluate and frame.evaluate, since they share the path. An async page function that throws a falsy value does reject on Firefox, but with the message undefined, so only the synchronous throw is swallowed outright.

Root cause

Juggler omits exceptionDetails from the Runtime.callFunction response when the thrown value is falsy, so there is nothing for playwright-core to detect. Captured with DEBUG=pw:protocol, same session, only the page function differs:

() => { throw null; }        ◀ RECV {"result":{}}
() => { throw 0; }           ◀ RECV {"result":{}}
() => { throw 5; }           ◀ RECV {"result":{"exceptionDetails":{"value":5}}}
() => Promise.reject(7)      ◀ RECV {"result":{"exceptionDetails":{}}}

checkException in packages/playwright-core/src/server/firefox/ffExecutionContext.ts:98 is then correct to return early on the first two, because it was never told an exception happened. So the primary fix looks like it belongs in juggler, populating exceptionDetails whenever the call threw rather than only when the thrown value is truthy.

Two related things on the core side

Both only matter once juggler reports the exception, but they are in the same seven lines:

function checkException(exceptionDetails?: Protocol.Runtime.ExceptionDetails) {
  if (!exceptionDetails)
    return;
  if (exceptionDetails.value)
    throw new js.JavaScriptErrorInEvaluate(JSON.stringify(exceptionDetails.value));
  else
    throw new js.JavaScriptErrorInEvaluate(exceptionDetails.text + (exceptionDetails.stack ? '\n' + exceptionDetails.stack : ''));
}

1. if (exceptionDetails.value) is a truthiness check on the payload field, so a future {"value": 0} would still take the else branch and report the wrong thing. 'value' in exceptionDetails would survive the juggler fix.
2. When exceptionDetails arrives empty, exceptionDetails.text is undefined and the concatenation produces the literal string undefined. That is the visible half of the fourth row above: Promise.reject(7) reports page.evaluate: undefined on Firefox versus page.evaluate: 7 elsewhere. The value itself is lost by juggler, but the string undefined is ours.

I did not open a PR because the part that actually swallows the error is in juggler and needs a Firefox roll, and I did not want to send a patch that only tidies the message while the silent failure stays.

I am a freshman in college trying my best to contribute for the greater good, so please tell me if I have the layering wrong here.

[ayaangazali]

One more data point, same shape as the report: the same Firefox binary rejects correctly when Playwright talks to it over WebDriver BiDi rather than juggler.

Same probe file, same build (d1ead3ecc), same binary (firefox-1544), only the transport differs:

                       juggler                  bidi              chromium (CDP control)
throw null             RESOLVED undefined       REJECTED "null"   REJECTED "null"
throw 0                RESOLVED undefined       REJECTED "0"      REJECTED "0"
throw ""               RESOLVED undefined       REJECTED ""       REJECTED ""
throw 5                REJECTED "5"             REJECTED "5"      REJECTED "5"
Promise.reject(7)      REJECTED "undefined"     REJECTED "7"      REJECTED "7"

Run as npm run biditest -- --project=moz-firefox-page with BIDI_FFPATH pointed at the bundled firefox-1544, against npm run test -- --project=firefox-page for juggler.

Worth noting the last row too: over BiDi the rejection value survives, so Promise.reject(7) reports 7 instead of the literal string undefined. Both halves of the original report look transport-specific rather than engine-specific, which I think supports fixing this in juggler rather than working around it in checkException.

Happy to run anything else on either transport if a comparison would help.