Test Execution

Execute Tests

Test execution is the core activity within a test cycle. For each test case in the cycle, testers review the test steps, perform the actions described, verify the expected results, and record an execution status. TestKase supports a streamlined execution workflow designed to minimize friction and maximize the information captured during testing.

Execution Statuses

Every test case in a cycle has one of the following four execution statuses. Each status carries a specific meaning and is color-coded for quick visual identification:

Step-by-Step Execution Workflow

1. Open the test cycle and locate the test case you want to execute. You can use the search bar or filter by assignee to find your assigned tests quickly.
2. Click on the test case row to open the test case detail view. This displays the full test case information including the title, description, preconditions, and all test steps with expected results.
3. Read through the preconditions and ensure they are met before starting execution. If a precondition cannot be satisfied, consider marking the test as Blocked and recording an actual result explaining which precondition is not met.
4. Follow each test step sequentially. For each step, perform the described action on the system under test and compare the actual result against the expected result documented in the step.
5. After completing all steps, click the status dropdown for the test case and select the appropriate status (Pass, Fail, Blocked, or leave as Not Executed if deferring).
6. If the test failed, record an actual result describing what went wrong. Include specific details such as the step number that failed, what you observed, the expected result, and any error messages or screenshots. Clear failure descriptions help developers reproduce and fix the issue faster.
7. If the failure is a new bug, record it as a defect. Defects are not created from the execution view — create the defect on the Defects tab and link it to the test case. See Linking a defect to a failed test.
8. Move to the next test case and repeat the process. The cycle's progress bar updates in real time as you execute tests.

Recording the Actual Result

Each execution has an actual result field that captures context raw statuses cannot. Use it to document:

• Failure details -- What specifically went wrong, including the actual vs. expected result, the step number that failed, and any error messages or stack traces observed.
• Environment observations -- Any environment-specific behavior, such as performance degradation on staging or intermittent failures related to network latency.
• Workarounds discovered -- If a blocked test can be unblocked with a workaround, document it so other testers can proceed.
• Partial results -- If a test case partially passed (e.g., 7 of 10 steps passed before a failure), note which steps succeeded to provide useful debugging context.
• Data or configuration notes -- Specific test data used, configuration values, or environment variables that are relevant to reproducing the result.

Linking Defects to a Failing Test

There is no create-a-defect and no link-a-defect control inside the execution view. While you are executing, the test case drawer shows a read-only Defects ID list of the defects already linked to that test case.

Defects are linked at the test-case level, not against one execution run:

1. Create the defect on the Defects tab with New Defect, or open the existing defect if the bug has already been logged.
2. Open the test case, expand its Traceability section, and add the defect under Defects ID -- or add the test case from the defect's own Traceability -> Test Cases field.

Because the link lives on the test case, one defect can cover several failing test cases and the link survives deleting the cycle. See Linking a defect to a failed test.

Execution Rules and Guidelines

• A test case can only have one execution status at a time within a cycle. Changing the status overwrites the previous one (but the change is recorded in history).
• Any project member with appropriate permissions can execute any test case in the cycle, regardless of whether it is assigned to them. Assignments are advisory, not restrictive.
• Statuses can be changed as many times as needed. There is no lock-out after setting an initial status. This supports re-testing workflows where a failed test needs to be re-executed after a fix.
• Defect links are preserved when the status changes.

Bulk Execution

Setting many statuses through the API

POST /api/v1/projects/test-cycles/execute-bulk/:projectId/:cycleId accepts up to 500 results in one call and updates every matching test case in the cycle. Each result carries its own execution status, actual result, duration and other execution fields, and each one is recorded individually in the cycle history.

This is the endpoint the @testkase/reporter CLI uses, so in practice you get bulk status updates by reporting automated results from CI rather than by calling the endpoint yourself. The endpoint is PAT-accessible if you need to script it directly.

Each result must name a testcase_id that is already linked to the cycle. A result for a test case that is not in the cycle is not created silently — that one result comes back in the response's error list as Test case not linked with this test cycle while the rest of the batch still applies. Link the test cases first, or report into the Master Cycle, which always contains every test case in the project.

Several results may share the same testcase_id — one per browser or CI shard, for example. They are each recorded as their own run, and the test case's status in the cycle becomes the worst status across all of them.

What the UI selection can do in bulk

Ticking test cases in a cycle enables these bulk actions instead:

Common scenarios without a bulk Set Status

• Environment down -- Mark the affected tests Blocked individually and record an actual result explaining the environment issue. Leave the rest as Not Executed.
• Automated test results -- Report them from CI with the reporter. This is the intended bulk path and it also populates durations, error messages and build IDs.
• Descoping tests -- Leave the related test cases as Not Executed, or unlink them from the cycle.
• Resetting for re-test -- Use Reset Executions: select the test cases to re-run and reset them back to Not Executed in one action.

Track Progress

Every test cycle in TestKase displays real-time progress metrics that help you understand how much testing has been completed, what the outcomes are, and whether the cycle is on track. Progress tracking is available at a glance from the cycle listing view and in more detail within the cycle detail view.

The Progress Bar

The progress bar is a horizontal, color-coded bar that visually represents the distribution of execution statuses across all test cases in the cycle. Each segment of the bar is proportional to the number of test cases in that status:

• Teal -- Passed test cases
• Red -- Failed test cases
• Blue -- Blocked test cases
• Gray -- Not Executed test cases

As testers execute tests and update statuses, the bar updates in real time. This gives managers and stakeholders an immediate visual indicator of testing progress without needing to open detailed reports.

Status Counts and Completion Percentage

In addition to the progress bar, each cycle displays numeric counts for each execution status and an overall completion percentage. The completion percentage is calculated as:

Completion % = (Passed + Failed + Blocked) / Total Test Cases x 100

This formula counts all test cases that have been actioned (i.e., any status other than "Not Executed") as complete. The idea is that even a failed or blocked test has been reviewed and its status is known. Only "Not Executed" tests represent truly pending work.

Pass Rate

A cycle shows counts per status and a completion percentage. It does not show a pass rate.

Pass percentages come from the Reports tab, where the pass rate is:

Pass % = Passed / Total Test Cases x 100

Blocked and not-executed test cases are in the denominator, so a partly executed cycle shows a low Pass % even when every test that was executed passed. Read Pass % alongside the completion percentage, never on its own.

How Progress Is Calculated -- Example Scenario

Consider a cycle with 100 test cases and the following distribution:

• Completion: (62 + 8 + 5) / 100 = 75%
• Pass % in Reports: 62 / 100 = 62%

This tells us that 75% of testing is done, with 25 tests still pending. Reports shows 62% — that figure is held down by the 25 not-executed and 5 blocked cases, not by failures. Only 8 tests actually failed. Once the remaining 30 are executed, Pass % will move up on its own if they pass.

Interpreting Results for Release Decisions

When using cycle progress to inform release decisions, consider these guidelines:

• 100% completion with a high Pass % (>95%) -- All tests executed and most passed. Review any failures to confirm they are known issues or low-severity. This is typically a green light for release.
• High completion but significant failures -- Testing is complete but critical test cases failed. Investigate failures, prioritize defect fixes, and plan a re-test round before release.
• Low completion -- Many test cases still at Not Executed. Testing is incomplete. Identify blockers, reassign resources, or extend the testing timeline.
• Many blocked cases -- A large blue segment indicates environmental or dependency issues. Resolve blockers before drawing conclusions about overall test quality.

Automation Runs

When a test case is executed through automation — via the TestKase Reporter or the execute-bulk API — each automation ID that reports a result creates a separate run entry on the test case's Automation Runs card on the execute page.

Each run shows its own:

• Automation ID -- The 5-digit identifier of the script/pipeline that produced the result
• Status -- Pass, Fail, or Blocked for that specific run
• Actual Result -- A short summary of what the test produced (or the error message on failure)
• Duration -- How long the automated test took to run
• Executed On -- Timestamp of the run, shown in your profile timezone
• Error Message -- Full stack trace and call log when the run failed

This gives you full visibility into which automation script or pipeline produced each result — useful when a single test case is covered by multiple automated tests (e.g., across CI pipelines, browsers, or retries).

Status Precedence (Worst-Status-Wins)

When a test case has multiple automation IDs reporting different results, the test case's overall status is the worst across all runs. The precedence is:

fail > blocked > not_executed > pass

Example: If automation ID 12345 reports Pass and 67890 reports Fail for the same test case, the overall status is Fail. Both runs remain visible individually in the Automation Runs card, so you can see exactly which automation path failed.

This behavior ensures that if any automation path detects a problem, the test case is flagged as failed regardless of how many other paths passed.

Re-running the Same Automation

Running the reporter a second time for the same automation ID updates the existing run entry rather than creating a duplicate. The status, duration, result, and error fields all reflect the latest run.

The cycle history tab always shows the full timeline of every execution — including older runs that have since been updated.

Multi-Environment / Multi-Browser Runs

When the same automation ID runs in multiple environments — e.g. Cypress with --browser chrome + --browser firefox, or Playwright projects chromium / firefox / webkit — each environment gets its own run entry in the Automation Runs card. The status label shows the environment tag next to the run:

• [97131] ✅ Pass · chromium
• [97131] ❌ Fail · firefox

The overall test case status uses worst-status-wins across all environments — if Chrome passes but Firefox fails, the test case is marked Fail. Both runs stay visible so you can see exactly which environment failed.

The reporter auto-detects the environment from:

• JUnit XML — [Browser] suffix in the <testsuite name="..."> attribute (e.g. "Sample Test Suite [Chrome]")
• Playwright JSON — the projectName field on each test (configured via projects in playwright.config.ts)
• Other formats — not auto-detected; same test across environments collapses into one run

Auto-detected environment tags (browser/project names) are stored per-run and do not need to be in the project's allowed environment list. The project's environment field is only set when the reported environment matches an allowed value.

Retry Details

When a test runs with retries enabled (e.g. Playwright's retries: 2, Maven Surefire's <reruns>2</reruns>), TestKase captures the full retry log in the Error Message field of the automation run — showing each attempt's status, duration, and error message. This makes flaky tests immediately identifiable on the execute page.

Flaky test (failed then eventually passed):

• Actual Result: "Flaky — passed after 2 retry(ies)"
• Error Message: Log of each failed attempt with its error + stack trace, followed by the successful final attempt
• Status: Pass (worst-status-wins does not apply to retries within one run — only the final outcome counts)

Consistently failing test (failed every retry):

• Actual Result: First attempt's error
• Error Message: "Failed after 3 attempt(s) (3 failed)" followed by each attempt's error
• Status: Fail

Retry support by format:

This is a format-level limitation, not a TestKase limitation. If you need full retry details and your framework supports both JUnit and a native format, prefer the format with richer retry info (usually Playwright JSON for Playwright, or Surefire JUnit for Java).

Reset Executions

You can clear all execution data for a test cycle to start a fresh round of testing while keeping the same test cases linked. This is useful when re-running a full regression after a build fix, starting a new sprint with the same cycle, or clearing old automation results before a new CI run.

How to Reset

1. Open the Execute page for the test cycle.
2. Click the ⋯ (more actions) menu in the toolbar next to the search bar.
3. Click Reset — the button reads Reset (N) when you have test cases selected, or just Reset to reset every test case in the cycle.
4. Confirm in the dialog. The action cannot be undone.

Scope — All or Selected

• No selection → resets every test case in the cycle
• With selection → resets only the selected test cases

What Gets Cleared

• All execution statuses are reset to Not Executed
• Actual results, error messages, durations, build IDs, and executed-on timestamps
• All automation run entries (the Automation Runs card becomes empty for reset test cases)

What Is Preserved

• Test case links to the cycle (test cases stay in the cycle)
• Test case content (steps, description, attachments)
• Cycle metadata (title, environment, folder, assignments)
• The cycle history — the reset itself is recorded as a history entry

Permissions

Available to all project roles except Guest. The underlying POST /projects/test-cycles/reset-executions/:projectId/:cycleId endpoint is PAT-accessible, so you can also trigger resets programmatically from a CI script before reporting new results.

Reset vs Clone

Use Reset when you want metrics and history to accumulate in the same cycle across re-runs. Use Clone when you want a completely new cycle for a different build or test pass.

Questions