What coding agents actually check

To make that decision, you need a way to trace the agent's report back to its actions and observations. Knowing what was checked, and what remains uncertain, helps you choose where to spend your own testing time.

Using recorded runs from oneshotlm, we look at three questions about browser verification, then consider how agent developers can make the answers easier to find.

1. Does the report match what actually ran?
2. Did the checks exercise the features that matter?
3. Do the checks cover the revised app?

Does the report match what actually ran?

In a Rubik's Cube built by Gemini 4 Argon, the final response said the agent had tested mouse-wheel zoom. The browser log tells a more limited story: the tool rejected an action named wheel during input validation. No later record shows a successful retry using a scroll action.

The evidence supports an attempt to use the wheel. It does not show that the action ran or that the cube zoomed. It also does not establish whether the zoom feature itself was broken.

For each check in a completion report, there are three distinct stages:

1. The agent requested an action.
2. The tool completed without an error.
3. The agent observed the expected result.

A failed check, or one with an inconclusive result, gives the next person a place to start. The useful question is which execution record supports each claim in the report.

See the Gemini Rubik's Cube run

Link completion reports to execution records

An agent runtime can assign identifiers that connect verification items to tool results. The system executing the tool should retain the action and its execution status, so a report can point back to that record. Judgments about the expected result should reference observations or test results as well.

For example, if the tool rejects a wheel-zoom action, that attempt should remain unverified rather than appear among the report's successful checks. Even if the tool completes successfully, the observation that zoom actually occurred needs its own record. A person or another process can then inspect the evidence behind the claim.

Tool errors can also make recovery easier. In this run, the validation response repeated the same generic message about a value not matching a constant. Returning the supported action name and an input example would give the agent a more useful next step. A redesigned error response might look like this:

Unsupported action: wheel
Use scroll for mouse-wheel input.
Example: {"action":"scroll","delta":300}

This is a proposed response, not the one returned in the run. Anthropic similarly recommends giving agents specific, actionable guidance when tool input validation fails. Writing effective tools for agents

Did the checks exercise the features that matter?

A Brick Breaker game built by Ling 3.1 Flash shows why it helps to move beyond the initial screen.

An early browser check reported an error involving a method in the Phaser game library. The agent edited the code and reloaded. The console's error lines disappeared, and the extracted screen text showed a score of 0 and LAUNCH.

The agent then pressed Space. The status changed to PLAYING. As it continued observing the game, the score reached 10 and a different error appeared:

brick.disableBody is not a function

Fixing the loading error had exposed another problem on the gameplay path. The agent removed the failing method call, reloaded, and used Space and a drag action again. The next results showed a score of 10 without that error. After it pressed R, the display returned to a score of 0, three lives, and LAUNCH.

Continuing into gameplay revealed a bug that was absent from the initial screen. The text records do not establish that the ball's reflection angles, or the game as a whole, were correct.

Web apps have similar paths that only run after a save, deletion, retry, or navigation. Compare the paths that were exercised with the features you need. Paths that were not reached, or whose results were not recorded, are candidates for further testing.

See the Ling Brick Breaker run

Define checks as actions with expected results

One way to track coverage is to turn important requirements into a starting state, an action, and an expected result. This makes a feature's acceptance criteria concrete enough to check.

Record the action's execution status separately from whether the expected result held. Clicking a Save button does not, by itself, establish that anything was saved. Where possible, compare DOM values or application state against explicit assertions. Playwright, for example, provides assertions that wait for an element's text to match an expected value. Playwright Assertions

Keep the underlying requirements too. When AI writes both the implementation and its tests, changing an expected value to fit the implementation is a separate decision. Reviewing changes to expected results separately from implementation changes makes it easier to understand what a passing test establishes.

Playwright's Trace Viewer is a useful reference for making the checking process inspectable. It lets you examine DOM snapshots and screenshots around actions to investigate where a result diverged from expectations. There is a distinction worth preserving: the lower-level context.tracing API does not record assertions. Use Playwright Test's tracing when you also need the test's assertion results. Trace Viewer, Tracing API

Do the checks cover the revised app?

An Arpeggiator built by MiniMax M3.1 Flash offers a traceable example of a display inconsistency and a subsequent fix, even though the console reported no errors during the affected interactions.

When the agent added a note during playback, the selection showed four notes while the playback status still said three. After it deselected every note, the selection showed zero while the status still said four.

The agent added code to update the status when the selection changed. After reloading, starting playback, and adding a note again, both displays showed four notes.

The record includes an observed problem, a corresponding code change, and another interaction afterward. The expected result—agreement between the selection count and status—makes the before-and-after comparison concrete. This table does not establish what happened when all notes were deselected after the fix, or whether the audio was correct.

There is also the question of which revision was checked. In the Rubik's Cube run from the first section, the agent changed a value passed to the rotation function from 14 to 6 after its last successful browser interaction. The saved log contains no browser observation after that change.

That does not prove the change introduced a bug. It establishes an ordering: the final edit came after the last recorded browser observation. When taking over an app, you need to know both what was rechecked and whether the check applies to the version you received.

See the Arpeggiator run

Tie observations to the version the browser loaded

For a single-file HTML app like these examples, a runtime could record a hash of the HTML passed to the browser alongside the actions and observations. Comparing it with the delivered HTML would show which revision the checks covered.

The relevant content is what the browser actually loaded. Hashing the current file afterward is insufficient if the browser still has an earlier revision open. A Git commit ID alone also misses uncommitted changes. For larger web apps, the deployed build and execution environment need to be identified too.

A matching version does not establish that a feature passed. It does help prevent results from an older revision being carried over to the final app without justification. If the connection cannot be established, leave verification of the final version marked as unknown. If the app demonstrably returns to a version that was already checked, there is no need to repeat every action mechanically.

Make verification useful to the next person

Together, the three questions point to a practical requirement for completion reports: a claim that something was checked should lead back to the feature requirement, execution result, and app version.

Here is one possible record format, using the Arpeggiator example:

Keep execution records on the tool side and judgments attached to observations or tests, then let the model's report reference them. Leave missing information marked as unknown. That gives someone reviewing the report a way to inspect its claims and choose what to check next.

Asking an agent to report the actions it actually performed, their results, and anything left unverified is a useful starting point. Agent developers can go further by making those connections part of the runtime.

A separate QA agent is another option to explore. Anthropic has described separating implementation from evaluation, then simplifying the setup as models and tasks changed. An evaluator can still make mistakes, so expected results and execution evidence remain necessary. Harness design for long-running application development

These are design ideas drawn from the runs, not interventions whose effectiveness we measured. Start with an important user path and ask whether the report can be traced to the action, expected result, and version. With those connections in place, a person can identify gaps in the evidence before repeating every interaction.

The examples are individual runs, not a ranking of the models' overall performance.