Marking tests

When you connect directly, nothing reports your results — so a session has no outcome, and a run of twenty tests shows as one.

Playwright has no executeScript. Both lines go through page.evaluate instead, as a string handed in as the second argument:

// The empty callback is required. The string after it is what we read.
const mark = (action, args) =>
  page.evaluate(_ => {}, `testkase_action: ${JSON.stringify({ action, arguments: args })}`);

1. Saying whether a test passed or failed

One line, at the end of your test:

await page.evaluate(_ => {},
  `testkase_action: ${JSON.stringify({ action: "setTestStatus", arguments: { status: "passed" } })}`);

page.evaluate("_ => {}",
    "testkase_action: " + json.dumps({"action": "setTestStatus", "arguments": {"status": "passed"}}))

page.evaluate("_ => {}",
    "testkase_action: {\"action\":\"setTestStatus\",\"arguments\":{\"status\":\"passed\"}}");

await page.EvaluateAsync("_ => {}",
    "testkase_action: {\"action\":\"setTestStatus\",\"arguments\":{\"status\":\"passed\"}}");

The session then shows green in your dashboard. Send failed instead and it shows red.

Only two words work. passed or failed. Anything else — PASSED, success, ok — is thrown away, and nothing warns you.

Report the failure where a crash cannot skip it. If your test blows up halfway, every line after the blow-up is skipped, including the one that reports the result:

try {
  await page.goto("https://example.com");
  await expect(page.getByRole("heading")).toBeVisible();

  await mark("setTestStatus", { status: "passed" });
} catch (err) {
  await mark("setTestStatus", { status: "failed" });
  throw err;
}

Adding a reason for the failure

A red row tells you something broke; a reason tells you what:

await mark("setTestStatus", { status: "failed", remark: err.message });

2. Splitting a session into several tests

You probably don't need this. Every session already counts as one test, named from the name in your capabilities. If you connect once per test — what Playwright's own runner does — you are done.

Read on only if you run many tests through one connection. Naming each one splits them into separate rows, each with its own video and its own logs.

Mark where a test starts with setTestName, and where it ends with setTestStatus. The first setTestName renames the test we opened for you; each one after that closes the previous test and starts a new one.

test("search renders results", async () => {
  await mark("setTestName", { name: "Search renders results" });   // this test starts
  try {
    await page.goto("https://example.com");
    await expect(page.getByRole("heading")).toBeVisible();

    await mark("setTestStatus", { status: "passed" });             // it ends, green
  } catch (err) {
    await mark("setTestStatus", { status: "failed" });             // it ends, red
    throw err;
  }
});

Put them in your hooks, not in every test

Writing them by hand in each test is how one gets forgotten:

test.beforeEach(async ({}, testInfo) => {
  await mark("setTestName", { name: testInfo.title });
});

test.afterEach(async ({}, testInfo) => {
  await mark("setTestStatus", { status: testInfo.status === "passed" ? "passed" : "failed" });
});

Close each test before the next one starts. If you forget, the next setTestName closes the one before it as failed — we cannot tell whether it worked.