Insight Reports
Advanced analytics including flaky tests, risk heatmaps, and predictive failure.
Insight Reports
Tier 3 Insights reports use advanced analytics and heuristic scoring to analyze your testing data and surface insights that go beyond what traditional metrics can reveal. They are computed directly from your project data with SQL analytics — no language model is involved, so they do not consume AI credits and are always available regardless of your AI settings. These 11 reports live in the Reports tab under the Insights category.
| Report | Chart Type | What It Does |
|---|---|---|
| Flaky Tests | Table | Identifies tests whose recorded result keeps changing across multiple runs |
| Release Readiness | Gauge | Computes a readiness score from pass rate, progress, and critical test pass rate |
| Risk Heatmap (Folder) | Heatmap | Visualizes failure risk across test case folders based on historical data |
| Risk Heatmap (Feature) | Heatmap | Visualizes failure risk across product features, from the fail rate of the test cases carrying each feature tag |
| Test Case Quality | Table | Scores each test case on field completeness (steps, expected result, precondition, priority, and more) |
| Predictive Failure | Table | Predicts which tests are most likely to fail based on historical patterns |
| Tester Effectiveness | Table | Per tester: executions run, pass rate, and defects reported |
| Stale Tests | Table | Lists test cases by how long since their last real execution result (top 100) |
| Cycle Health | Table | Per cycle: execution progress, pass rate, and percentage of blocked tests |
| Suite Optimization | Table | Identifies never-run and always-passing tests to streamline your suite |
| Execution Velocity | Line | Tracks daily execution throughput with rolling average trend line |
Report Details
Flaky Tests
Flaky tests are tests that produce different results (pass/fail) across runs without any code changes. This report analyzes execution history across multiple cycles to identify tests with inconsistent outcomes. For each test it shows the number of times its result flipped between pass and fail and the total number of executions, ranked by flip count (highest first).
Use this report to: Prioritize stabilization efforts, quarantine unreliable tests, and improve CI/CD pipeline reliability.
Release Readiness
A composite score (gauge chart) that combines three factors to answer "Are we ready to release?" The factors are: overall pass rate (weighted 45%), test execution completion percentage (30%), and critical test pass rate (25%). The gauge is teal at 75 and above, amber from 50, orange from 25, and red below 25.
"Critical" here means a test case whose priority is P0 or P1. Those values are not in the
default test-case priority list (Blocker, Critical, Major, Minor, Trivial), so in a project using
the defaults no test case matches and that 25% component always scores 0 — a fully passing, fully
executed project reads about 75, not 100. Set test-case priorities to P0/P1 if you want this
component to contribute.
Use this report to: Make data-driven release decisions and communicate readiness to stakeholders.
Risk Heatmaps
Two variants (by Folder and by Feature) visualize which areas of your application carry the highest failure risk. The heatmap uses color intensity (green to red) to show relative risk levels based on each area's failure rate. The by-Folder variant also shows the linked defect count for each folder; the by-Feature variant is fail-rate only and excludes test cases with no feature tag.
Use this report to: Focus testing effort on high-risk areas and allocate QA resources effectively.
Test Case Quality
Scores each test case with a field-completeness checklist: whether it has a summary, test steps, an expected result, a precondition, a priority, labels, feature tags, and an automation status. Each present element adds points, producing a quality score (0-100).
Use this report to: Identify poorly written test cases, drive test case improvement initiatives, and train new team members.
Predictive Failure
Uses historical execution data to rank which tests are most likely to fail in the next cycle. Tests are ordered by their recent failure frequency (how often they have failed lately and how long since they last passed).
Use this report to: Prioritize which tests to run first in time-constrained cycles and identify areas needing attention before they fail.
Suite Optimization
Analyzes your entire test suite to find optimization opportunities: tests that have never been executed and tests that always pass (potentially low value).
Use this report to: Reduce test suite maintenance burden, speed up execution time, and remove redundant tests.
Insight reports are most meaningful when you have at least 3-5 completed test cycles of historical data. The more execution history available, the more reliably these analytics can surface patterns and rank risk.