# Use Case: QA When Coding Agents Open the Pull Requests

Dokimore Support Team · October 28, 2026, updated 2026-11-01 · 8 min read

Format: Scenario walkthrough · Written for: QA leads and test managers, Engineering managers · Where it applies: every market

> The agent writes the code and, about half the time, a test for it. The question a QA lead owns in 2026 is which checks must not come from the same hand.

## In short

- When an agent writes both the change and its test, the test checks the agent’s understanding of the task, not the task. The release gate needs at least one check that was produced without reading the new code: a recorded user flow, a contract assertion, a visual baseline.
- The 2026 studies agree on the shape of the gap: agents add a test in about half of the pull requests that touch tested code, the tests they add are long and assertion-heavy, and the error-handling branches are the ones nobody exercises.
- DORA’s 2025 report found AI adoption now linked to higher delivery throughput and still names the open problem as making sure software works as intended before it reaches users. That sentence is the QA lead’s job description for the year.

This is a scenario walkthrough of how a QA plan changes when coding agents write a large share of the code. It is not a customer story; the team and its numbers were chosen so the arithmetic is easy to follow, and every outside figure is in the sources at the end.

## The scenario

A product team of fourteen engineers ships a web application with a public API. Since the spring, coding agents, the kind that run in a terminal and open a pull request when they finish, have authored about two pull requests in five. Review time per pull request fell. The change failure rate did not. In the retrospective nobody could say which failures the agents caused, because the agents also wrote most of the tests that passed.

The QA lead’s question is not whether to allow the agents. That decision was taken by the people who pay for them. The question is what the release gate should contain when the code and its tests arrive from the same process.

## What the 2026 research measured

Two studies of real agent pull requests, both built on the AIDev dataset of GitHub pull requests opened by coding agents, were published this year. Dipongkor, Baral, Lam and Moran analysed 4,882 agent pull requests (532 Java, 4,350 Python) from five agents. Agents changed a test file in 49.6% of the pull requests that modified code under test. In 64.8% of the pull requests, no changed line was executed by any existing test. Where the agent did write tests, they improved coverage in 35.9% of the Java and 22.5% of the Python pull requests. The part most consistently left untested was error handling, with miss rates of 86.0% in Java and 81.0% in Python: most of the changed code that runs when something goes wrong ran under no test at all.

Yoshimoto and colleagues, at the Mining Software Repositories conference, looked at 2,232 commits with test changes and found AI authored 16.4% of the commits that add tests in those repositories. The tests AI wrote were longer, carried more assertions and had lower cyclomatic complexity, and their contribution to coverage was comparable to human-written tests. Read together: the agents can write a decent test when they write one, they write one about half the time, and they almost never test the branch where things go wrong.

DORA’s 2025 report, from a survey of about 5,000 technology professionals, puts adoption at 90% and reports that AI use is now linked to higher software delivery throughput, which reverses its 2024 finding. The same post names the challenge that remains: ensuring software works as intended before it is delivered to users. The report’s own framing is that AI is a mirror and a multiplier; in a cohesive organisation it boosts efficiency, in a fragmented one it exposes the cracks.

## The rule: not the same hand

A test written by the agent that wrote the code is a restatement of the agent’s reading of the task. If the reading is wrong, the test is wrong in the same direction, and it passes. This is not a new problem; a developer writing their own unit tests has always had it. What changed is the proportion: when two pull requests in five arrive with their own green tests, the suite that gates a release drifts toward checking that the agents agree with themselves.

The fix is structural, not a review checklist. The gate needs checks produced without reading the new code. A user flow recorded in a real browser against the running application is one: it encodes what the product is supposed to do for a person, and it was captured before the pull request existed. An assertion on an API contract is another. A visual baseline is a third. None of them know what the agent intended, which is the point.

## Four changes to the QA plan

- Keep an independent suite and say which tests are in it. In the scenario the team marks sixty recorded flows and forty API checks as the gate; agent-written unit tests run too, but they cannot be the only thing standing between a change and production.
- Test the error path on purpose. The 2026 analysis put the miss rate for changed error handling above 80% in both languages. The team adds a rule to its pull-request template: a change that adds or edits a catch block, a retry or a fallback names the test that drives it, and the reviewer reads that test first.
- Run what the change can reach, on every change. Two pull requests in five from agents means more pull requests, not fewer. Selection that maps a change to the flows and API checks that can reach it keeps the per-commit loop short; the full suite still runs on a schedule.
- Keep the evidence, not the verdict. A green run tells you nothing a week later. A run that keeps the screenshot, the network calls and the assertion for each step tells the retrospective which failures were the agent’s and which were the page’s.

## What Dokimore does here, and what it does not

Dokimore’s recordings are the independent suite in this walkthrough. A flow is recorded once in a real browser, replays as a regression test straight away, and keeps the element, the locators, the network calls and a screenshot for every step; it never reads the pull request. Smart test selection, on Solo Pro and above, maps a change set or a requirement to the recorded flows and API checks that can reach it, with a written reason per pick and a confidence score that says when to run everything instead. Recordings export as Playwright, Selenium, Robot Framework or pytest code, and that code runs in your own pipeline next to the agents’ unit tests.

What it does not do: there is no CI trigger from the platform yet, so the selection runs in Dokimore, on demand or on a schedule, and the exported code is what runs in CI. Dokimore does not review the agent’s code and does not grade the agent’s tests; it checks the product. Playwright itself now ships planner, generator and healer agents, and a team that writes its tests in code can use those to draft the independent suite too; what matters is that the suite exists and that someone other than the code’s author owns it.

## For the manager reading this

Count the pull requests the agents opened last month and the production incidents in the same month, then try to connect them. If you cannot, the gate has no independent check, and the number to fix first is not throughput. A 14-day trial on your real flows produces your own selection ratio and your own evidence trail; it does not change how many pull requests the agents open, and it is not supposed to.

## What to do next

1. Count last month’s pull requests opened by coding agents and the production incidents in the same month, and try to connect them; if you cannot, the gate has no independent check.
2. Name the independent suite in writing: the recorded flows, API checks and visual baselines that gate a release without having read the change.
3. Add one line to the pull-request template: a change that adds or edits a catch block, a retry or a fallback names the test that drives it.
4. Run what a change can reach on every change and keep the full suite on a schedule; the [smart test selection guide](https://dokimore.com/documentation/smart-test-selection) says what the platform selects from and what it does not.
5. Keep the evidence per run, not the verdict; the [test evidence guide](https://dokimore.com/documentation/test-evidence) lists what each step records.

## Questions people ask

### Should AI-written tests be allowed in the release gate?

Allowed, yes; alone, no. The 2026 studies found agent-written tests reach coverage comparable to human-written ones, so discarding them wastes real checks. The gate should also contain checks that were produced without reading the change, such as recorded user flows, API contract assertions and visual baselines, because a test written by the code’s author restates the author’s reading of the task.

### What do coding agents leave untested most often?

Error handling. In the analysis of 4,882 agent pull requests, error-handling constructs were the most consistently under-tested, with miss rates of 86.0% in Java and 81.0% in Python. A pull-request rule that names the test driving every new catch block, retry or fallback closes most of that gap at review time.

### Does AI adoption make delivery faster or less stable?

DORA’s 2025 report, from about 5,000 respondents, links AI adoption to higher software delivery throughput, a reversal of its 2024 finding, and still names ensuring that software works as intended before delivery as the open challenge. Its framing is that AI amplifies what an organisation already is: faster where the process is cohesive, more fragile where it is fragmented.

### How does Dokimore fit next to coding agents?

As the suite that did not read the code. Recorded flows replay as regression tests with evidence per step, smart test selection runs the flows and API checks a change can reach, and the exported Playwright, Selenium, Robot Framework or pytest code runs in your own pipeline. There is no CI trigger from the platform yet, and Dokimore does not review or grade the agent’s code.

### What is the AIDev dataset?

A public collection of pull requests opened by coding agents on GitHub, used by both 2026 studies cited here. Its February 2026 release describes 932,791 agent-authored pull requests from five agents (OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code) across 116,211 repositories. Its value is that it holds real pull requests from real repositories rather than a benchmark; its limit is that it reflects public projects, and a private codebase may differ.

## How we measured this

- The team size, the share of agent pull requests and the suite sizes in this walkthrough are a scenario chosen for clarity, not a measurement of any customer.
- Every study figure is quoted from the paper’s abstract or the publisher’s own post, read on the date beside the reference; we have not reproduced either study.
- Dokimore’s behaviour is the product on the article date: recorded flows with per-step evidence, smart test selection with a written rationale and a confidence score, code export in four frameworks, and no CI trigger from the platform.

## Sources

1. [Dipongkor, Baral, Lam, Moran, "Test Coverage Analysis of Agentic Pull Requests", arXiv 2607.18057 (July 2026): 4,882 agent PRs, tests changed in 49.6%, error-handling miss rates 86.0% in Java and 81.0% in Python](https://arxiv.org/abs/2607.18057) (read 2026-10-28)
2. [Yoshimoto et al., "Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage", MSR 2026, arXiv 2603.13724: AI authored 16.4% of test-adding commits, coverage comparable to human tests](https://arxiv.org/abs/2603.13724) (read 2026-10-28)
3. [Li, Zhang, Hassan, "AIDev: Studying AI Coding Agents on GitHub", arXiv 2602.09185 (February 2026): 932,791 agent-authored pull requests from five agents across 116,211 repositories](https://arxiv.org/abs/2602.09185) (read 2026-10-28)
4. [Google, "How are developers using AI? Inside our 2025 DORA report" (23 September 2025): 90% adoption, about 5,000 respondents, AI linked to higher throughput, "works as intended" named as the remaining challenge](https://blog.google/technology/developers/dora-report-2025/) (read 2026-10-28)
5. [DORA, "State of AI-assisted Software Development" 2025 report page: AI as an amplifier of existing strengths and weaknesses](https://dora.dev/research/2025/dora-report/) (read 2026-10-28)
6. [Playwright documentation, Test Agents: planner, generator and healer](https://playwright.dev/docs/test-agents) (read 2026-10-27)

Source: https://dokimore.com/blog/qa-when-coding-agents-open-the-pull-requests
