AI Agent
A built-in conversational AI sidebar for creating test cases, querying data, summarizing coverage, and managing tests through natural language.
AI Agent
The AI Agent is a conversational sidebar panel built directly into the TestKase dashboard. It understands your project context — test cases, cycles, folders, execution data, and requirements — and lets you interact with everything through natural language.
Unlike external AI integrations (such as MCP Server), the AI Agent runs inside your dashboard with zero setup. Open it, type a request, and the agent executes it in real time.
Opening the AI Agent
There are two ways to open the AI Agent:
| Method | Description |
|---|---|
| Keyboard shortcut | Press Ctrl+K (Windows/Linux) or Cmd+K (macOS) from anywhere in the dashboard |
| Navbar button | Click the sparkle button in the top navigation bar (tooltip: AI Agent (Ctrl+K)) |
The agent panel slides in from the right side of the screen. You can use it alongside your regular workflow — it does not take over the page.
The AI Agent is available on every page within a project. You must have a project selected for the agent to have context about your test data.
What the AI Agent Can Do
The agent supports a wide range of actions across the test management lifecycle:
Test Case Management
- Create test cases — "Create 3 test cases for the login feature" generates structured test cases with titles, steps, priorities, and expected results.
- Search and filter — "Show all high priority test cases" or "Find blocked test cases in the Payments folder" filters your test case list using natural language.
- Query data — "How many test cases are in this project?" or "What is the pass rate?" returns real-time data from your project.
Coverage and Reporting
- Summarize coverage — "Summarize test coverage for this project" returns a breakdown of pass rates, execution counts, and untested areas.
- Execution insights — "How many test cases failed in the last cycle?" queries execution results and returns detailed answers.
- Pass rate analysis — "What is the overall pass rate?" calculates metrics across all executed test cases.
Execution, Cycles and Plans
- Record results — "Mark TEST-12 as pass in TCYCLE-5" writes the execution result.
- Manage cycles and plans — create cycles and plans, link test cases, and assign testers.
The agent reads and writes your test data. It cannot drive the page you are looking at — it does not apply dashboard filters, open screens or navigate for you. After the agent changes something, refresh the list to see it.
Using Suggestion Cards
When you first open the AI Agent, you see prebuilt suggestion cards — common actions you can trigger with a single click:
| Suggestion Card | What It Does |
|---|---|
| Summarize test coverage | Get a quick overview of pass/fail/pending stats |
| Create test cases for login | Auto-generate cases from a feature description |
| Show high priority test cases | Filter and surface the most critical tests |
| How many test cases failed? | Count test cases by execution status |
Click any card to instantly send that prompt to the agent. This is useful for common tasks without typing.
Suggestion cards are contextual — they reference your current project's data. The results will vary based on what test cases and cycles exist in the project.
Typing Your Own Prompts
For anything beyond the suggestion cards, type a free-form request in the input field at the bottom of the panel. The agent understands natural language — you do not need to use specific commands or syntax.
Example Prompts
| Prompt | What Happens |
|---|---|
| "Summarize test coverage for this project" | Returns a coverage breakdown with pass rates, execution counts, and untested areas |
| "Create 3 test cases for the login feature" | Generates three structured test cases with steps, priorities, and expected results |
| "Show all high priority test cases" | Searches your test cases and lists the high-priority ones in the chat |
| "How many test cases failed in the last cycle?" | Queries the most recent test cycle and returns failure count with details |
| "Mark TEST-12 as pass in TCYCLE-5" | Records the execution result against that cycle |
| "What is the overall pass rate?" | Calculates and returns the pass rate percentage across all executed test cases |
Tips for Better Results
- Be specific — "Create 3 test cases for the password reset flow covering email validation, expired links, and successful reset" produces better results than "Create test cases."
- Reference existing data — The agent knows your folders, cycles, and labels. Use them: "Show test cases in the Checkout folder with status failed."
- Ask follow-up questions — The agent maintains conversation context within a session. You can say "Now filter those by high priority" after a previous query.
- Use action verbs — Start prompts with "Create," "Show," "Find," "Summarize," or "Count" for clearer intent.
Real-Time Streaming
The AI Agent streams responses in real-time. As the agent processes your request:
- Thinking indicator — A loading state shows while the agent analyzes your prompt.
- Tool execution — If the agent needs to call internal tools (e.g., creating a test case, querying data), you see which tools are being executed and their status.
- Streaming response — The answer appears progressively, word by word, so you can start reading before the full response is complete.
This transparency means you always know what the agent is doing and can verify that it took the correct actions.
Tool Execution Tracking
The AI Agent uses internal tools to perform actions. When a tool is invoked, you see:
- Tool name — Which tool was called, with underscores rendered as spaces (the real tool names are
list_projects,get_project_structure,search_testcases,get_testcase,manage_testcase,manage_folder,search_test_cycles,manage_test_cycle,execute_tests,manage_test_planandget_report) - Execution status — Whether the tool succeeded or failed
The feed shows the tool name and its status only; it does not summarise what the tool returned. The agent's own reply is where the result is explained.
This is particularly useful for complex requests that involve multiple steps. For example, "Create a test case for login and add it to the Sprint 42 cycle" may invoke multiple tools in sequence.
Project Context
The AI Agent is project-scoped. It has access to:
- Test cases — All test cases in the current project, including their steps, priorities, labels, statuses, and folder locations
- Test cycles — Cycle names, statuses, linked test cases, and execution results
- Folders — The full folder hierarchy for organizing test cases
- Execution data — Pass/fail/blocked/not-executed results across all cycles
- Requirements — Linked requirements and their traceability
- Defects — Reported defects and their associations
You must select an organization and project before using the AI Agent. If no project is selected, the agent will prompt you to select one before proceeding.
Message and Session Limits
The agent has a few hard limits. None of them are shown in the panel, so it is worth knowing them before you hit one:
| Limit | Value | What happens |
|---|---|---|
| Length of one message | 10,000 characters | The panel does not stop you typing or pasting more, but the request is rejected before it reaches the model. Paste long specs into Test Case Generation instead. |
| Conversation history sent | The last 20 messages | Only the most recent 20 messages travel with your request. In a long session the agent genuinely no longer knows what you said at the start — restate anything that still matters. |
| Reply length | 4,096 tokens per model call | A very long answer is cut off rather than continued. Ask for it in parts. |
| Tool-calling rounds per message | 10 | If the agent needs more than ten rounds of tool calls to finish a request, it stops and answers with what it has. Break large requests into steps. |
| Time per message | 2 minutes | The loop stops and the reply ends with "Request timed out. Please try a simpler question." Work already written to your project is not rolled back. |
Because the loop can stop part-way at the round limit or the time limit, a large multi-step request can leave your project partly changed — for example three of five test cases created. Check the result before re-running the same prompt, or you will get duplicates.
Credit Usage
Messages sent to the AI Agent consume AI credits from your organization's credit balance. Charging is token-based — the number of credits used scales with the size of your prompt, the project context, and the model's response, not a flat per-message fee.
- Credits are deducted after each message based on the actual tokens used.
- Clicking a suggestion card also consumes credits (it sends a message on your behalf).
- The agent maintains conversation context within a session. Longer conversations carry more context, which uses more tokens per message, so starting a fresh session for an unrelated task can be more credit-efficient.
- The agent will not run once the organization's credit balance is exhausted.
See Credits & Billing for details on how the monthly credit balance works.
Requirements
| Requirement | Details |
|---|---|
| AI enabled | AI must be enabled for the organization. It is on by default; the owner can turn it off under Organization → AI Credits |
| Credits available | A credit balance above zero. The check is a simple greater than zero, so a request that starts with 0.5 credits left runs to completion even if it costs more — it is the next request that is refused |
| Project selected | A project must be selected for the agent to have context |
| Role | The AI Agent is permission-gated (enabled by default). Access can be restricted by role through the project permissions settings |
Limitations
- The AI Agent operates within the current project only. It cannot query data across multiple projects in a single request.
- Complex multi-step operations (e.g., "Create a test plan, add three cycles, and generate a report") may not always execute in the expected order. Break these into separate prompts for reliability.
- The agent does not modify or delete existing test cases unless explicitly asked. Read operations (search, filter, summarize) are always safe.
- Conversation context is maintained within a session. Closing and reopening the panel starts a new session.
AI Agent vs. MCP Server
TestKase offers two ways to interact with AI:
| AI Agent | MCP Server | |
|---|---|---|
| Where | Built into the TestKase dashboard | External AI tools (Claude, Copilot, Cursor) |
| Setup | Zero setup — just open the panel | Requires MCP server configuration |
| Access | Web browser | IDE, terminal, desktop app |
| Best for | Quick queries and on-the-fly test creation without leaving TestKase | Bulk operations, automation workflows, developer-centric testing |
| Credits | Token-based (uses org credits) | No TestKase credit cost (uses your own AI tool's API) |
Both can be used together. Use the AI Agent for quick tasks during QA workflows, and the MCP Server for deeper automation from your development environment.
Next Steps
- AI Test Case Generation — Generate test cases from natural language descriptions using the dedicated generation modal.
- Insight Reports — Advanced analytics including flaky tests, risk heatmaps, and predictive failure.
- MCP Server Integration — Connect external AI agents to TestKase via the Model Context Protocol.
- Credits & Billing — Understand the monthly credit grant, monitor usage, and data privacy.