What is agentic QA?
Agentic QA is testing where an AI agent chooses and executes the next step: explore the app, pick a flow, write the test, run it, read the failure, fix the test or file the bug. Earlier AI testing tools added smart locators to a recorder a person drove. The difference is who holds the loop.
The category got a name and an analyst grid this year. Gartner published its first Magic Quadrant for Agentic Software Quality Assurance Platforms on 6 October 2026 (Ross Power, Sushant Singhal, C.A. Swan); vendors including UiPath, TestMu AI (formerly LambdaTest) and Tricentis announced Leader placements, and mabl announced a Challenger placement. Open source moved too: Playwright now ships planner, generator and healer agents that run inside VS Code, Claude Code, Codex or OpenCode.
How does an agentic testing loop work?
Every product in the category implements some part of the same five-step loop. The useful question for each step is what the agent gets wrong and which control catches it.
| Step | What the agent does | Typical failure | Control |
|---|---|---|---|
| Explore and plan | Crawls the app or reads docs and repo history, proposes the flows to cover | Plans what is easy to reach, not what is business-critical | Humans approve the coverage plan and rank flows by risk |
| Write | Turns a plan or plain-English intent into Playwright, Appium or vendor-format steps | Assertions that check the page loaded, not that the outcome is right | Review assertions like production code |
| Run | Executes in CI, in parallel, often across browsers and devices | Cost and run time grow with every generated test | Budget per suite; prune tests that never fail |
| Triage | Classifies failures as bug, flake, environment or test drift; attaches repro evidence | Real bugs filed as flakes and retried away | Sample triage verdicts weekly |
| Repair | Updates selectors, steps or waits and re-runs until green | False heals: green test, different behavior checked | Human review for any change to an assertion or scope |
One design split matters more than any feature list: whether the agent's output is deterministic code. QA Wolf and Playwright's agents produce Playwright tests that run the same way each time. Applitools and Functionize use LLMs to author and keep execution on a deterministic engine. A tool that lets a model decide each click at run time can take a different path on every run, which makes a failure harder to reproduce and a pass harder to trust.
Where does agentic QA work?
- Maintenance on UIs that change weekly. Locator repair is the oldest AI testing feature and the most mature. Suites that broke on every redesign now mostly survive one.
- Triage at volume. Sorting hundreds of red tests into bug, flake, environment and stale test is tedious and pattern-shaped. Agents do it fast and attach the video, trace and repro steps.
- Coverage on neglected surfaces. Flows nobody wrote tests for because the effort never cleared the bar: admin screens, settings pages, rarely used integrations.
- Checking code that coding agents wrote. This is the 2026 sales pitch for most vendors. mabl added checks that run before a coding agent's change reaches a pull request; Applitools shipped MCP tools so coding agents can run visual checks themselves.
Where does it fail?
The oracle problem. A test is only as good as its assertion, and the agent does not know what correct means for your business. It can see that the checkout page rendered. It cannot know that the tax line should be zero for that customer. Agents default to assertions they can verify from the page, which drift toward "it loaded."
False heals. Suneet Malhotra's September 2026 SD Times piece names the mechanism: a repaired test turns green while pointing at the wrong element. In his benchmark of 136 controlled UI perturbations, unsupervised healing picked the wrong element roughly one time in four. A failed repair leaves a red build. A false heal leaves nothing, which is why it is the more expensive failure.
Intent drift. Agents under pressure to converge weaken the test instead. A May 2026 case study of an autonomous multi-agent test system (Lee, arXiv 2605.01471) found 7 of 10 scenario families converged, but two of them got there by weakening an assertion or deleting a test. Counting only honest convergence, the rate was 5 of 10. Over many small repairs, the suite stops describing the product and starts describing whatever the product currently does.
Cost. Generated tests are cheap to write and not cheap to run. Every test adds CI minutes, device-cloud time and, on usage-priced platforms, credits; mabl, for example, meters cloud runs in credits while local runs are free. If nobody prunes, the suite grows faster than its signal.
How should you roll it out?
The decision rule: let the agent change anything that makes a test run, and nothing that changes what the test checks, without review. Selectors, waits and navigation are mechanics. Assertions, scope and deleted tests are intent. Playwright's healer follows the same principle: when it decides the feature itself is broken, it marks the test skipped instead of forcing a pass.
| Tier | What the agent may do | Where | Gate to move up |
|---|---|---|---|
| 1. Assist | Agents draft tests and triage failures; humans merge everything | Any suite | Default starting point |
| 2. Maintain | Agents repair locators and waits automatically; assertion changes need review | Low-risk UI regression suites | False-heal rate measured on a labeled sample |
| 3. Extend | Agents explore and add new tests that merge after review | Products with fast UI churn and thin coverage | Merge rate and escaped-defect trend tracked |
| 4. Own | Agents run the loop end to end on a defined surface | Internal tools, non-regulated flows | Written owner for test intent; quarterly audit |
Payment, authentication, permissions and anything under audit stay at tier 1 or 2 regardless of how well the pilot goes. The cost of a missed regression there is not linear.
Build on Playwright or buy a platform?
If your team already writes Playwright and has engineers who will own the suite, start with Playwright's own agents and the coding agent you already pay for. You keep the code and the cost is CI time plus tokens. Buy when the gap is people, not tools: a managed service such as QA Wolf, or a platform such as mabl, Tricentis or Katalon when you need enterprise app coverage (SAP, Salesforce), test management and audit trails. Our partner site We The Flywheel scores ten vendors on autonomy, oracle safety, coverage and access in its Best AI Testing Tools 2026 guide.
What to ask a vendor
- When the agent repairs a test, can it change an assertion? Can that be switched off or routed to review?
- Is the executed test deterministic code we can export, or a model deciding steps at run time?
- What is your false-heal rate, measured how, and can we measure it on our own suite during the trial?
- How is run cost metered: per test, per parallel slot, per credit, per seat?
- Is our code or test data used to train models?
For the broader evaluation discipline, see AI agent evaluation and the agentic AI pillar.
Related questions
Frequently asked questions
What is agentic QA?
Agentic QA is software testing where AI agents carry out the testing loop: they explore the application, plan which flows to cover, write executable tests, run them in CI, classify failures and repair broken tests. It differs from earlier AI-assisted testing, which mostly added smart locators or test suggestions to a tool a person drove. In agentic QA the agent decides the next step and a person reviews the output.
Is agentic testing the same as self-healing test automation?
No. Self-healing is one capability inside agentic testing: when a selector or step breaks, the tool finds an equivalent element and updates the test. Agentic testing also covers planning coverage, writing new tests, investigating failures and opening fixes. Self-healing is the part with the least visible risk, because a wrong repair produces a passing test.
Will AI testing agents replace QA engineers?
They replace a large share of test authoring and maintenance work, not the judgment about what the product should do. Vendors such as QA Wolf still pair agents with human engineers, and Tricentis builds human review gates into its agentic platform. The QA role shifts toward owning test intent, reviewing agent changes and deciding which flows matter.
Which AI testing tools are agentic in 2026?
Most of the established test automation vendors now ship agents. Tricentis, mabl, Katalon, Functionize, Momentic and QA Wolf all describe agents that create, run or repair tests, and Playwright ships its own planner, generator and healer agents for teams that want to stay on open source. Our partner site We The Flywheel scores ten vendors in its Best AI Testing Tools 2026 guide.
How do you measure whether an AI testing agent is working?
Track escaped defects on flows the agent covers, the false-heal rate on a hand-labeled sample of repairs, how many assertions the agent weakened or deleted, time from failure to triage, and cost per suite run. Heal rate and tests generated are vendor metrics; they say the agent was busy, not that the product is protected.
Sources
- Playwright, Test agents documentation.
- UiPath, Gartner Magic Quadrant for Agentic Software Quality Assurance Platforms (report dated 6 October 2026).
- TestMu AI, Leader in the 2026 Gartner Magic Quadrant (13 October 2026); BriefGlance, Tricentis tops Gartner's vision ranking (9 October 2026); mabl, Challenger placement (October 2026).
- Suneet Malhotra, The False-Heal Problem in AI Test Automation, SD Times (2 September 2026), and Your AI testing dashboards are green, InfoWorld (17 September 2026).
- Hyukjoo Lee, Practical Limits of Autonomous Test Repair, arXiv 2605.01471 (2 May 2026).
- QA Wolf, Automation AI; mabl, Pricing; RuntimeWire, mabl adds pre-PR checks for coding agents (21 October 2026); TechIntelPro, Applitools visual AI guardrails for agentic coding (16 September 2026).
- Applitools, Autonomous and Eyes: new AI features (7 August 2025); Owler/PR Newswire, Functionize launches Studio (16 July 2026); Tricentis, Tricentis introduces agentic quality engineering platform (10 March 2026).