TestNG

Overview

TestNG is a Java testing framework inspired by JUnit and NUnit, with built-in support for data-driven testing, parallel execution, flexible test configuration via XML suite files, and dependency management between test methods. It is widely used for both unit and integration testing in Java projects.

To integrate TestNG results with TestKase, use the native XML output format and report with --format testng.

Prerequisites

Java 11+ (JDK)
Maven or Gradle as build tool

Installation

Maven

Add the TestNG dependency to your pom.xml:

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.9.0</version>
    <scope>test</scope>
</dependency>

Gradle

Add TestNG to your build.gradle:

dependencies {
    testImplementation 'org.testng:testng:7.9.0'
}

test {
    useTestNG()
}

Project Setup

Maven Configuration

Configure the maven-surefire-plugin in your pom.xml to use TestNG and point to your suite file:

<!-- pom.xml -->
<project>
    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.9.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

TestNG Suite File

Create a testng.xml in your project root to define which test classes to run:

<!-- testng.xml -->
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="LoginSuite" parallel="none">
    <test name="LoginTests">
        <classes>
            <class name="com.example.tests.LoginTest"/>
            <class name="com.example.tests.DashboardTest"/>
        </classes>
    </test>
</suite>

Directory Structure

my-project/
  pom.xml
  testng.xml
  src/
    main/java/com/example/
      auth/
        LoginService.java
    test/java/com/example/tests/
      LoginTest.java
      DashboardTest.java
  test-output/               # Generated after running tests
    testng-results.xml

Writing Tests

Create a test class (e.g., src/test/java/com/example/tests/LoginTest.java):

// src/test/java/com/example/tests/LoginTest.java
package com.example.tests;

import com.example.auth.LoginService;
import com.example.auth.LoginResult;
import org.testng.Assert;
import org.testng.annotations.*;

public class LoginTest {

    private LoginService loginService;

    @BeforeMethod
    public void setUp() {
        loginService = new LoginService();
    }

    @AfterMethod
    public void tearDown() {
        loginService = null;
    }

    @Test(description = "[48271] testValidLogin")
    public void testValidLogin() {
        LoginResult result = loginService.login("user@example.com", "password123");
        Assert.assertTrue(result.isSuccess(), "Login should succeed");
        Assert.assertNotNull(result.getToken(), "Token should be present");
    }

    @Test(description = "[48272] testInvalidPassword")
    public void testInvalidPassword() {
        LoginResult result = loginService.login("user@example.com", "wrong");
        Assert.assertFalse(result.isSuccess(), "Login should fail");
        Assert.assertEquals(result.getError(), "Invalid credentials");
    }

    @Test(description = "[48273] testEmptyEmail")
    public void testEmptyEmail() {
        Assert.assertThrows(IllegalArgumentException.class, () -> {
            loginService.login("", "password123");
        });
    }

    @Test(description = "[48274] testMultipleUsers", dataProvider = "loginData")
    public void testMultipleUsers(String email, String password, boolean expected) {
        LoginResult result = loginService.login(email, password);
        Assert.assertEquals(result.isSuccess(), expected);
    }

    @DataProvider(name = "loginData")
    public Object[][] loginData() {
        return new Object[][] {
            {"user@example.com", "password123", true},
            {"admin@example.com", "admin456", true},
            {"unknown@example.com", "wrong", false},
        };
    }
}

Each test includes a 5-digit Automation ID in its @Test(description) annotation using square brackets. The @testkase/reporter CLI extracts these IDs using the regex \[(\d{5})\]. For the example above:

48271 → linked to the "valid login" test case in TestKase
48272 → linked to the "invalid password" test case in TestKase
48273 → linked to the "empty email" test case in TestKase
48274 → linked to the "multiple users" test case in TestKase

Generate Automation IDs in TestKase first, then embed them in your @Test(description) annotations. The [XXXXX] pattern can appear anywhere in the description string.

Running Tests

Run all tests with Maven:

mvn test

Run with a specific suite file:

mvn test -DsuiteXmlFile=testng.xml

Run with Gradle:

gradle test

TestNG automatically generates results in test-output/testng-results.xml after each run.

TestKase Integration

After running tests, report the native TestNG results to TestKase:

npx @testkase/reporter report \
  --token $TESTKASE_PAT \
  --project-id PRJ-1 \
  --org-id 1173 \
  --cycle-id TCYCLE-5 \
  --format testng \
  --results-file test-output/testng-results.xml

--cycle-id is optional. If not provided, results are reported to TCYCLE-1 — the master test cycle for the project.

Automation ID Mapping

The reporter extracts 5-digit Automation IDs from TestNG test descriptions using the [XXXXX] bracket pattern:

Test Pattern	Where to Embed the ID	Extracted ID
Standard test	`@Test(description = "[48271] testValidLogin")`	`48271`
Test with `@DataProvider`	`@Test(description = "[48274] testMultipleUsers", dataProvider = "...")`	`48274`
Inherited test	`@Test(description = "[48275] testFromBaseClass")`	`48275`

For @DataProvider tests, all iterations share the same Automation ID from the description attribute.

The [XXXXX] pattern can appear anywhere in the @Test(description) value. Generate the 5-digit ID in TestKase first, then embed it in your test annotation.

Complete Example

1. Test File

// src/test/java/com/example/tests/LoginTest.java
package com.example.tests;

import com.example.auth.LoginService;
import com.example.auth.LoginResult;
import org.testng.Assert;
import org.testng.annotations.*;

public class LoginTest {

    private LoginService loginService;

    @BeforeMethod
    public void setUp() {
        loginService = new LoginService();
    }

    @Test(description = "[48271] testValidLogin")
    public void testValidLogin() {
        LoginResult result = loginService.login("user@example.com", "password123");
        Assert.assertTrue(result.isSuccess());
    }

    @Test(description = "[48272] testInvalidPassword")
    public void testInvalidPassword() {
        LoginResult result = loginService.login("user@example.com", "wrong");
        Assert.assertFalse(result.isSuccess());
    }
}

2. Run Tests

mvn test

3. Report Results to TestKase

npx @testkase/reporter report \
  --token $TESTKASE_PAT \
  --project-id PRJ-1 \
  --org-id 1173 \
  --cycle-id TCYCLE-5 \
  --format testng \
  --results-file test-output/testng-results.xml

Troubleshooting

testng-results.xml not found

TestNG writes results to the test-output/ directory by default. If this directory does not exist or the file is missing:

Verify tests actually ran: check the Maven/Gradle console output for test execution summaries
Check the test-output/ directory in your project root (not in target/)
For Maven with surefire, the XML may also be in target/surefire-reports/testng-results.xml

# Check both locations
ls test-output/testng-results.xml
ls target/surefire-reports/testng-results.xml

Surefire not using TestNG

If Maven surefire defaults to JUnit instead of TestNG, ensure your pom.xml does not include both JUnit and TestNG on the classpath. If both are present, surefire may pick JUnit. Either:

Remove the JUnit dependency, or
Explicitly configure surefire to use TestNG:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
        <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
        </suiteXmlFiles>
    </configuration>
</plugin>

Parallel execution results

When running tests in parallel (parallel="methods" or parallel="classes" in testng.xml), all results are still combined into a single testng-results.xml file. No special configuration is needed for the TestKase reporter. However, be aware that:

Test execution order may differ between runs
@BeforeMethod and @AfterMethod run per thread
Shared state between tests may cause flaky results in parallel mode

TestNG

On this page