TestNG

Overview

TestNG is a Java testing framework inspired by JUnit and NUnit, with built-in support for data-driven testing, parallel execution, flexible test configuration via XML suite files, and dependency management between test methods. It is widely used for both unit and integration testing in Java projects.

To integrate TestNG results with TestKase, use the native XML output format and report with --format testng.

Prerequisites

Java 11+ (JDK)
Maven or Gradle as build tool

Installation

Maven

Add the TestNG dependency to your pom.xml:

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.9.0</version>
    <scope>test</scope>
</dependency>

Gradle

Add TestNG to your build.gradle:

dependencies {
    testImplementation 'org.testng:testng:7.9.0'
}

test {
    useTestNG()
}

Project Setup

Maven Configuration

Configure the maven-surefire-plugin in your pom.xml to use TestNG and point to your suite file:

<!-- pom.xml -->
<project>
    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.9.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

TestNG Suite File

Create a testng.xml in your project root to define which test classes to run:

<!-- testng.xml -->
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="LoginSuite" parallel="none">
    <test name="LoginTests">
        <classes>
            <class name="com.example.tests.LoginTest"/>
            <class name="com.example.tests.DashboardTest"/>
        </classes>
    </test>
</suite>

Directory Structure

my-project/
  pom.xml
  testng.xml
  src/
    main/java/com/example/
      auth/
        LoginService.java
    test/java/com/example/tests/
      LoginTest.java
      DashboardTest.java
  test-output/               # Generated after running tests
    testng-results.xml

Writing Tests

Create a test class (e.g., src/test/java/com/example/tests/LoginTest.java):

// src/test/java/com/example/tests/LoginTest.java
package com.example.tests;

import com.example.auth.LoginService;
import com.example.auth.LoginResult;
import org.testng.Assert;
import org.testng.annotations.*;

public class LoginTest {

    private LoginService loginService;

    @BeforeMethod
    public void setUp() {
        loginService = new LoginService();
    }

    @AfterMethod
    public void tearDown() {
        loginService = null;
    }

    @Test
    public void test_48271_validLogin() {
        LoginResult result = loginService.login("user@example.com", "password123");
        Assert.assertTrue(result.isSuccess(), "Login should succeed");
        Assert.assertNotNull(result.getToken(), "Token should be present");
    }

    @Test
    public void test_48272_invalidPassword() {
        LoginResult result = loginService.login("user@example.com", "wrong");
        Assert.assertFalse(result.isSuccess(), "Login should fail");
        Assert.assertEquals(result.getError(), "Invalid credentials");
    }

    @Test
    public void test_48273_emptyEmail() {
        Assert.assertThrows(IllegalArgumentException.class, () -> {
            loginService.login("", "password123");
        });
    }

    @Test(dataProvider = "loginData")
    public void test_48274_multipleUsers(String email, String password, boolean expected) {
        LoginResult result = loginService.login(email, password);
        Assert.assertEquals(result.isSuccess(), expected);
    }

    @DataProvider(name = "loginData")
    public Object[][] loginData() {
        return new Object[][] {
            {"user@example.com", "password123", true},
            {"admin@example.com", "admin456", true},
            {"unknown@example.com", "wrong", false},
        };
    }
}

Each test embeds a 5-digit Automation ID in its method name using underscores (e.g. test_48271_validLogin). The TestNG parser reads only each test method's name attribute — the @Test(description) value is never read — so the ID must live in the method name. Configure the reporter with --automation-id-format "_(\\d{5})_" to extract the ID from between the underscores. For the example above:

48271 → linked to the "valid login" test case in TestKase
48272 → linked to the "invalid password" test case in TestKase
48273 → linked to the "empty email" test case in TestKase
48274 → linked to the "multiple users" test case in TestKase

The TestNG results file exposes each test as <test-method name="...">, and the reporter matches against that name only. IDs placed in @Test(description = "[48271] ...") are not read and will never match. Put the ID in the method name and pass --automation-id-format "_(\\d{5})_".

Generate Automation IDs in TestKase first, then embed them in your test method names.

Running Tests

Run all tests with Maven:

mvn test

Run with a specific suite file:

mvn test -DsuiteXmlFile=testng.xml

Run with Gradle:

gradle test

TestNG automatically generates results in test-output/testng-results.xml after each run.

TestKase Integration

After running tests, report the native TestNG results to TestKase:

npx @testkase/reporter report \
  --token $TESTKASE_PAT \
  --project-id PRJ-1 \
  --org-id 1173 \
  --cycle-id TCYCLE-5 \
  --format testng \
  --automation-id-format "_(\\d{5})_" \
  --results-file test-output/testng-results.xml

--cycle-id is optional. If not provided, results are reported to TCYCLE-1 — the master test cycle for the project.

Automation ID Mapping

The reporter extracts 5-digit Automation IDs from the TestNG method name using the --automation-id-format "_(\\d{5})_" pattern (the ID wrapped in underscores):

Test Pattern	Where to Embed the ID	Extracted ID
Standard test	`public void test_48271_validLogin()`	`48271`
Test with `@DataProvider`	`public void test_48274_multipleUsers(...)`	`48274`
Inherited test	`public void test_48275_fromBaseClass()`	`48275`

For @DataProvider tests, all iterations share the same Automation ID from the method name.

The parser reads the <test-method name="..."> value only — the @Test(description) attribute is ignored. Generate the 5-digit ID in TestKase first, then embed it in your test method name.

Complete Example

1. Test File

// src/test/java/com/example/tests/LoginTest.java
package com.example.tests;

import com.example.auth.LoginService;
import com.example.auth.LoginResult;
import org.testng.Assert;
import org.testng.annotations.*;

public class LoginTest {

    private LoginService loginService;

    @BeforeMethod
    public void setUp() {
        loginService = new LoginService();
    }

    @Test
    public void test_48271_validLogin() {
        LoginResult result = loginService.login("user@example.com", "password123");
        Assert.assertTrue(result.isSuccess());
    }

    @Test
    public void test_48272_invalidPassword() {
        LoginResult result = loginService.login("user@example.com", "wrong");
        Assert.assertFalse(result.isSuccess());
    }
}

2. Run Tests

mvn test

3. Report Results to TestKase

npx @testkase/reporter report \
  --token $TESTKASE_PAT \
  --project-id PRJ-1 \
  --org-id 1173 \
  --cycle-id TCYCLE-5 \
  --format testng \
  --automation-id-format "_(\\d{5})_" \
  --results-file test-output/testng-results.xml

Troubleshooting

testng-results.xml not found

TestNG writes results to the test-output/ directory by default. If this directory does not exist or the file is missing:

Verify tests actually ran: check the Maven/Gradle console output for test execution summaries
Check the test-output/ directory in your project root (not in target/)
For Maven with surefire, the XML may also be in target/surefire-reports/testng-results.xml

# Check both locations
ls test-output/testng-results.xml
ls target/surefire-reports/testng-results.xml

Surefire not using TestNG

If Maven surefire defaults to JUnit instead of TestNG, ensure your pom.xml does not include both JUnit and TestNG on the classpath. If both are present, surefire may pick JUnit. Either:

Remove the JUnit dependency, or
Explicitly configure surefire to use TestNG:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
        <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
        </suiteXmlFiles>
    </configuration>
</plugin>

Parallel execution results

When running tests in parallel (parallel="methods" or parallel="classes" in testng.xml), all results are still combined into a single testng-results.xml file. No special configuration is needed for the TestKase reporter. However, be aware that:

Test execution order may differ between runs
@BeforeMethod and @AfterMethod run per thread
Shared state between tests may cause flaky results in parallel mode

TestNG

On this page