Insight Reports
Advanced analytics for flaky tests, risk heatmaps, predictive failure, and more.
Insights
Insights are Tier 3 reports that use advanced analytics and heuristic scoring to surface actionable patterns your team would not easily find with traditional charts. These reports analyze your historical testing data to detect flaky behavior, rank failure-prone tests, score test quality, and optimize your suite.
Insight reports are computed directly from your project's execution data with SQL analytics — no language model is involved — so they consume no AI credits and are always available, whatever your organization's AI settings.
Flaky Tests
Chart type: Table | Tier: 3
Identifies tests whose recorded result keeps changing across runs. A test is considered flaky if its result alternates without any code changes, indicating non-deterministic behavior. Pass, fail and blocked all count as results; resetting a case to not-executed is ignored.
What it detects: Tests whose recorded result flips across multiple executions. The report lists each flaky test along with its flip count and total number of executions, ranked by flip count (highest first). The table returns the 50 flakiest test cases, and only test cases that have flipped at least once appear at all — a test case with a consistent result is never listed, however many times it has run.
When to use: Before trusting your test results. Flaky tests erode confidence in the suite. Address the root causes (timing issues, test data dependencies, shared state) to stabilize your results.
Release Readiness
Chart type: Gauge | Tier: 3
An automatically computed readiness score combining three key factors with a weighted formula: overall pass rate (45%), execution progress (30%, the percentage of tests executed), and critical test case pass rate (25%). The result is a single gauge from 0 to 100 indicating how ready the build is for release.
"Critical" in this report means a test case whose priority is P0 or P1. Those values are
not in the default test-case priority list (Blocker, Critical, Major, Minor, Trivial), so in a
project using the defaults no test case matches and that 25% always scores 0 -- a fully passing,
fully executed project reads about 75, not 100. Set test-case priorities to P0/P1 if you want
this component to contribute, and do not set a release gate above 75 until you have.
What it detects: Whether your project meets the quality bar for release. The gauge colour steps at 75, 50 and 25 -- teal at 75 and above, amber from 50, orange from 25, red below 25 -- and the contributing factors are broken out beneath it so you know what is dragging the score down.
When to use: As the final quality gate before a release. Share the readiness score with stakeholders and product owners to support go/no-go decisions.
Risk Heatmap (Folder)
Chart type: Heatmap | Tier: 3
Generates a heatmap of failure risk across test case folders. Each cell represents a folder, with color intensity indicating the risk level based on its failure rate. The linked defect count for each folder is also shown alongside the fail rate.
What it detects: High-risk product areas with the highest failure rates. These are the areas most likely to cause production issues.
When to use: During test planning to allocate more testing effort to high-risk areas. Share with development leads to prioritize stabilization work.
Risk Heatmap (Feature)
Chart type: Heatmap | Tier: 3
Similar to the folder heatmap but organized by product feature instead of folder structure. This is useful when your folder hierarchy does not directly map to features.
What it detects: Feature-level risk concentrations, computed from the fail rate of the test cases carrying each feature tag. Test cases with no feature tag are excluded, and defects are not counted -- only the folder heatmap includes a defect count.
When to use: When presenting quality status to product managers who think in terms of features rather than test folders.
Test Case Quality
Chart type: Table | Tier: 3
Assigns a quality score to each test case using a field-completeness checklist: whether it has a summary, test steps, an expected result, a precondition, a priority, labels, feature tags, and an automation status. Each present element contributes points toward the score.
What it detects: Poorly written test cases that lack detail -- missing steps, empty expected results, no precondition, or unset priority/labels. The table returns the 100 lowest-scoring test cases.
When to use: During test suite audits and continuous improvement efforts. Focus on improving the lowest-scoring test cases first for maximum impact.
Predictive Failure
Chart type: Table | Tier: 3
Ranks the tests most likely to fail in the next execution by their recent failure frequency. Tests are ordered by how many times they have failed recently (and how long since they last passed), surfacing the tests with the worst recent track record first. The table returns the top 50.
What it detects: Tests with a high recent failure count. This allows your team to prioritize executing these tests early in the cycle so failures are discovered sooner.
When to use: At the start of a new test cycle. Run the predicted-to-fail tests first to get early feedback and triage defects while the rest of the suite is still executing.
Tester Effectiveness
Chart type: Table | Tier: 3
Provides per-tester metrics: the number of executions each tester ran, their pass rate, and the number of defects they reported. There is no composite score and no velocity column. This is not a ranking -- it helps identify patterns and coaching opportunities.
What it detects: Variations in testing thoroughness across the team. Testers with low defect discovery rates combined with high pass rates may be doing surface-level testing.
When to use: During one-on-one meetings and team retrospectives to identify skill development opportunities and optimize team composition.
Stale Tests
Chart type: Table | Tier: 3
Lists your test cases ordered by how long it has been since they last carried a real execution result, showing the last-executed date and the days elapsed (top 100). There is no threshold to configure -- read down the list. Stale tests may be outdated, irrelevant, or simply forgotten.
What it detects: Tests that have fallen out of rotation. These may be testing deprecated features, may be too difficult to execute (indicating they need simplification), or may have been superseded by newer tests.
When to use: Quarterly test suite reviews. Review the stale test list and decide whether to re-execute, update, or archive each one.
Cycle Health
Chart type: Table | Tier: 3
A health dashboard for each test cycle showing execution progress, pass rate, and the percentage of tests that are blocked. This provides a quick summarized view of every active cycle. It does not include a defect count.
What it detects: Cycles that are struggling -- low progress, high failure rates, or a high blocked percentage. The side-by-side metrics make it easy to spot trouble at a glance.
When to use: When managing multiple concurrent test cycles. Check the health dashboard to prioritize which cycles need attention.
Suite Optimization
Chart type: Table | Tier: 3
Analyzes your test suite to identify unused and always-passing test cases. The report categorizes tests into actionable groups: tests that have never been run, and tests that always pass (and may not be providing much value). The table returns up to 100 rows.
What it detects: Bloat in your test suite. Over time, test suites accumulate never-run and low-value tests that increase execution time without improving quality.
When to use: Before starting a new major release cycle. Trim the suite to reduce execution time and focus on tests that provide real value.
Execution Velocity
Chart type: Line | Tier: 3
Tracks daily execution throughput with a rolling average trend line. The chart shows how many tests are being executed each day along with a smoothed trend to filter out daily noise.
What it detects: Throughput slowdowns and bottlenecks. If the rolling average declines, something is slowing the team down -- perhaps environment issues, complex failures requiring investigation, or team capacity changes.
When to use: For sprint capacity planning. Use historical velocity to predict how many tests the team can execute in the next sprint and plan scope accordingly.