Marking tests

Report each test as passed or failed, and split one session into as many tests as you like.

Selenium has no idea what a "test" is. To it, driver is just a connection to a browser — so unless you tell us, a session has no result and counts as one test no matter how much it did.

Two lines fix both. This page is those two lines.

1. Saying whether a test passed or failed

One line, at the end of your test. It is the same line in every language, wrapped in whatever yours calls "run this script":

Language	The line
Java	`((JavascriptExecutor) driver).executeScript("testkase-status=passed");`
JavaScript	`await driver.executeScript("testkase-status=passed");`
Python	`driver.execute_script("testkase-status=passed")`
C#	`driver.ExecuteScript("testkase-status=passed");`
Ruby	`driver.execute_script("testkase-status=passed")`
PHP	`$driver->executeScript("testkase-status=passed");`

The session then shows green in your dashboard. Send failed instead and it shows red.

Only two words work. passed or failed. Anything else — PASSED, success, ok — is thrown away, and nothing warns you.

Report the failure where a crash cannot skip it. If your test blows up halfway, every line after the blow-up is skipped — including the one that reports the result. So the failed line belongs in your catch:

JavascriptExecutor js = (JavascriptExecutor) driver;
try {
    driver.get("https://example.com");
    assertTrue(driver.getTitle().contains("Example"));

    js.executeScript("testkase-status=passed");
} catch (Throwable t) {
    js.executeScript("testkase-status=failed");
    throw t;
}

try {
  await driver.get("https://example.com");
  assert((await driver.getTitle()).includes("Example"));

  await driver.executeScript("testkase-status=passed");
} catch (err) {
  await driver.executeScript("testkase-status=failed");
  throw err;
}

try:
    driver.get("https://example.com")
    assert "Example" in driver.title

    driver.execute_script("testkase-status=passed")
except Exception:
    driver.execute_script("testkase-status=failed")
    raise

try
{
    driver.Navigate().GoToUrl("https://example.com");
    Assert.That(driver.Title, Does.Contain("Example"));

    driver.ExecuteScript("testkase-status=passed");
}
catch
{
    driver.ExecuteScript("testkase-status=failed");
    throw;
}

begin
  driver.get("https://example.com")
  raise "wrong title" unless driver.title.include?("Example")

  driver.execute_script("testkase-status=passed")
rescue
  driver.execute_script("testkase-status=failed")
  raise
end

try {
    $driver->get("https://example.com");
    if (strpos($driver->getTitle(), "Example") === false) {
        throw new Exception("wrong title");
    }

    $driver->executeScript("testkase-status=passed");
} catch (Throwable $err) {
    $driver->executeScript("testkase-status=failed");
    throw $err;
}

Adding a reason for the failure

A red row tells you something broke; a reason tells you what. Use the longer form for that:

driver.executeScript(
  "return null",
  `testkase_action: ${JSON.stringify({
    action: "setTestStatus",
    arguments: { status: "failed", remark: err.message },
  })}`,
);

2. Splitting a session into several tests

You probably don't need this. Every session already counts as one test, named from the name in your capabilities. If you start a fresh driver for each test — what most suites do — you are done.

Read on only if you run many tests through one driver. Naming each one splits them into separate rows, each with its own video and its own logs.

Mark where a test starts with testkase-name=, and where it ends with the status line you already have. The first testkase-name= renames the test we opened for you; each one after that closes the previous test and starts a new one.

@Test
public void searchRendersResults() {
    JavascriptExecutor js = (JavascriptExecutor) driver;
    js.executeScript("testkase-name=Search renders results");   // this test starts
    try {
        driver.get("https://example.com");
        assertTrue(driver.getTitle().contains("Example"));

        js.executeScript("testkase-status=passed");             // it ends, green
    } catch (Throwable t) {
        js.executeScript("testkase-status=failed");             // it ends, red
        throw t;
    }
}

it("search renders results", async () => {
  await driver.executeScript("testkase-name=Search renders results");  // this test starts
  try {
    await driver.get("https://example.com");
    assert((await driver.getTitle()).includes("Example"));

    await driver.executeScript("testkase-status=passed");              // it ends, green
  } catch (err) {
    await driver.executeScript("testkase-status=failed");              // it ends, red
    throw err;
  }
});

def test_search_renders_results(driver):
    driver.execute_script("testkase-name=Search renders results")   # this test starts
    try:
        driver.get("https://example.com")
        assert "Example" in driver.title

        driver.execute_script("testkase-status=passed")             # it ends, green
    except Exception:
        driver.execute_script("testkase-status=failed")             # it ends, red
        raise

[Test]
public void SearchRendersResults()
{
    driver.ExecuteScript("testkase-name=Search renders results");   // this test starts
    try
    {
        driver.Navigate().GoToUrl("https://example.com");
        Assert.That(driver.Title, Does.Contain("Example"));

        driver.ExecuteScript("testkase-status=passed");             // it ends, green
    }
    catch
    {
        driver.ExecuteScript("testkase-status=failed");             // it ends, red
        throw;
    }
}

it "search renders results" do
  driver.execute_script("testkase-name=Search renders results")   # this test starts
  begin
    driver.get("https://example.com")
    expect(driver.title).to include("Example")

    driver.execute_script("testkase-status=passed")               # it ends, green
  rescue
    driver.execute_script("testkase-status=failed")               # it ends, red
    raise
  end
end

public function testSearchRendersResults()
{
    $this->driver->executeScript("testkase-name=Search renders results");  // this test starts
    try {
        $this->driver->get("https://example.com");
        $this->assertStringContainsString("Example", $this->driver->getTitle());

        $this->driver->executeScript("testkase-status=passed");            // it ends, green
    } catch (Throwable $err) {
        $this->driver->executeScript("testkase-status=failed");            // it ends, red
        throw $err;
    }
}

Between a testkase-name= and its matching testkase-status=, we also cut a separate piece of video for that test alone. The line never reaches the browser — we read it on the way through.

Put them in your hooks, not in every test

Writing them by hand in each test is how one gets forgotten. Use whatever your runner already gives you:

// TestNG
@BeforeMethod
public void startTest(Method method) {
    ((JavascriptExecutor) driver).executeScript("testkase-name=" + method.getName());
}

@AfterMethod
public void endTest(ITestResult result) {
    String status = result.isSuccess() ? "passed" : "failed";
    ((JavascriptExecutor) driver).executeScript("testkase-status=" + status);
}

// Mocha
beforeEach(async function () {
  await driver.executeScript(`testkase-name=${this.currentTest.title}`);
});

afterEach(async function () {
  const status = this.currentTest.state === "passed" ? "passed" : "failed";
  await driver.executeScript(`testkase-status=${status}`);
});

# pytest
@pytest.fixture(autouse=True)
def mark_test(driver, request):
    driver.execute_script(f"testkase-name={request.node.name}")
    yield
    status = "failed" if request.node.rep_call.failed else "passed"
    driver.execute_script(f"testkase-status={status}")

// NUnit
[SetUp]
public void StartTest()
{
    driver.ExecuteScript($"testkase-name={TestContext.CurrentContext.Test.Name}");
}

[TearDown]
public void EndTest()
{
    var failed = TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Failed;
    driver.ExecuteScript($"testkase-status={(failed ? "failed" : "passed")}");
}

# RSpec
config.before(:each) do |example|
  driver.execute_script("testkase-name=#{example.description}")
end

config.after(:each) do |example|
  status = example.exception ? "failed" : "passed"
  driver.execute_script("testkase-status=#{status}")
end

// PHPUnit. setUp() names the test. The status stays in the test body
// (or a trait you call from it), because PHPUnit's "did this test fail"
// hook changed shape between major versions — and a wrong guess reports
// every test as passed, which is worse than reporting nothing.
protected function setUp(): void
{
    $this->driver->executeScript("testkase-name=" . $this->getName());
}

Close each test before the next one starts. If you forget, the next testkase-name= closes the one before it as failed — we cannot tell whether it worked. Forget it on the last test of a session and that test keeps showing as still running.

1. Saying whether a test passed or failed

Adding a reason for the failure

2. Splitting a session into several tests

Put them in your hooks, not in every test

On this page