Best Practices & FAQ

The reporter extracts 5-digit Automation IDs from test names using the regex \[(\d{5})\]. Steps to diagnose:

1. Run with --dry-run --verbose to see which [XXXXX] IDs the reporter extracts from your test names.
2. Verify that each extracted ID exists as an Automation ID on a test case in your TestKase project.
3. If no IDs are extracted, ensure your test names include the [XXXXX] bracket pattern (e.g., [48271] should login).
4. If IDs are extracted but not matched, check that the 5-digit ID in your test name matches the Automation ID linked to the test case in TestKase.

Your PAT token may be expired or invalid. Steps to resolve:

1. Check the token expiration date in API Keys settings.
2. Generate a new token if expired.
3. Update the CI/CD secret with the new token value.
4. Verify the secret name in your pipeline file matches exactly.

Verify the --api-url value is correct and accessible from your CI environment. Common causes:

• Incorrect API URL. Pass the base host only (e.g. https://api.testkase.com) — the reporter appends /api/v1 for you. Including /api/v1 yourself produces a doubled /api/v1/api/v1 path and 404s.
• Firewall or network policy blocking outbound requests from CI runners.
• For self-hosted TestKase, ensure the CI runner can reach your API server.
• Proxy configuration needed but not set.

The reporter does not retry a failed call — each API call has a 30-second timeout and a failure ends the run with exit 2. It is safe to simply run the same command again: results are stored per (test case, automation ID, environment) and updated in place, so a re-run repairs a partially reported run instead of duplicating it. Batches that already succeeded before the failure stay applied.

See the Exit Codes reference for what to check for each one.

Attachment problems are reported as warnings and never change the exit code — the run can exit 0 with nothing uploaded. Check the Attachments: N uploaded, N skipped, N failed line, then:

• Files over 50 MB are skipped.
• Only images, video and common document types are accepted. .zip (including Playwright traces), .json, .html and .xml are rejected as unsupported file types.
• Files whose extension the reporter cannot map — including .mov, .mkv, .bmp, .tiff, .heic, .md and Office documents — are sent as application/octet-stream and rejected, even though the API accepts those formats under their proper content type. Convert recordings to .mp4 or .webm.
• Uploads fail once your organization storage quota is full.
• Re-run with --verbose to see the per-file decision.

See Attachments for the full list.

The API allows 200 requests per minute from the same address. Reporting results is only a handful of calls, so this is almost always caused by attachments: each file is uploaded once per test case it is attached to, at two API calls apiece. A file whose name contains no Automation ID is attached to every test case executed in the run.

Name artifacts after the Automation ID they belong to so each file uploads once, and wait a minute before retrying. The filename is matched with the same --automation-id-format regex as your test names, so with the default pattern the brackets are required: login-[48271].png, not login-48271.png.

By default, skipped tests are mapped to "Blocked". To change this behavior:

• --report-skipped-as blocked — marks them as Blocked (default behavior)
• --report-skipped-as not-executed — marks them as Not Executed
• --report-skipped-as ignore — ignores them entirely (skipped tests are not reported)

The reporter cannot find the results file at the specified path. Common causes:

• Tests did not produce the expected output file (check test framework configuration).
• The path is relative and the working directory is different from expected.
• In multi-stage CI pipelines, artifacts may not be carried over to the reporting stage (e.g., GitLab needs artifacts configuration).

This is normal if not all test names contain a [XXXXX] Automation ID, or if some IDs have not been linked to test cases in TestKase yet. To handle this:

• Check the dry-run output to see which tests are unmatched (missing [XXXXX] or unknown ID).
• For unmatched tests, generate Automation IDs in TestKase and embed them in the test names.
• Use --missing-id-in-script skip (default) to silently ignore tests without embedded IDs.
• Or accept partial matching if some tests are intentionally not tracked in TestKase.

Yes. Run the reporter multiple times in the same pipeline with different --format and --results-file flags. Each invocation can target the same or different test cycles.

Yes. Run the reporter once per cycle, changing the --cycle-id flag. For example, report to both a "CI Automated" cycle and a "Sprint 14 Regression" cycle. Each cycle must already contain the test cases you are reporting on, or that run's results are rejected and the reporter exits 1.

When you re-run the reporter for the same automation ID, TestKase updates the existing automation run entry rather than creating a duplicate. The status, actual result, duration, error message, and executed-on fields all reflect the latest run.

The cycle history tab always records the full timeline of every execution — including older runs — so you retain the complete audit trail.

The test case's overall status uses worst-status-wins precedence: if any automation ID reports a failure, the overall status stays as Fail until that specific automation ID reports Pass again.

Use the Reset Executions feature on the execute page (available in the ⋯ more-actions menu). It clears all execution statuses, actual results, and automation runs for the cycle while keeping the test cases and cycle metadata intact. You can reset every test case in the cycle, or select specific test cases to reset only those. See Reset Executions for full details.

Yes — for Playwright JSON and JUnit XML written by Maven Surefire (its <flakyFailure>, <flakyError>, <rerunFailure> and <rerunError> elements). The reporter captures each attempt's status, duration (Playwright only), and error message. On the execute page, the Error Message field shows the full retry log and the Actual Result field labels flaky tests explicitly — Flaky — passed after 2 retry(ies).

Gradle's test-retry plugin is the exception: it writes each attempt as a separate <testcase> element rather than nesting them, so its retries arrive as separate results instead of attempts of one test.

For Cypress, TestNG, NUnit, and Cucumber, per-attempt retry data is not available in the output format — the framework's own reporter only writes the final attempt. See Retry Details for the full support matrix.

By default, no. With --missing-id-in-script skip (the default), results whose test name carries no Automation ID are logged and dropped — nothing is created.

If you pass --missing-id-in-script create, the reporter does create data in your project: for each untagged test it creates a test case, generates an Automation ID for it, links the two, and then reports the result against the new test case.

Currently, the @testkase/reporter CLI requires Node.js. If your CI environment does not have Node.js, use a Docker image that includes it (e.g., node:20) or add a Node.js setup step to your pipeline.

The pipeline templates shown in the platform-specific guides are the recommended starting point. Copy the template for your platform, update the flags with your project details, and you are ready to go. There is no separate marketplace action at this time.

If you create test cycles dynamically (e.g., one per PR), you can pass the cycle ID as an environment variable. Create the cycle via the TestKase API first and capture the ID, then pass it to the reporter with --cycle-id.

Yes. Use any Node.js-based Docker image (e.g., node:20, cimg/node:20.0) and run npx @testkase/reporter in a script step. No additional Docker configuration is needed.