Before trusting a browser agent, test what happens after a retry

Interface image: Microsoft / Playwright documentation, CC BY 4.0. Reproduced unchanged. Example report; counts and timings are documentation values.

A browser agent says it saved a record, then the connection drops. Should it click Save again? That is a useful question for a pilot: the answer depends on what actually reached the website, not the confidence of the agent's final message. Before giving automation a real account, test how it proves completion and recovers from uncertainty.

• Define a correct final result and verify it separately from the agent's own report.
• Keep first-attempt passes, recovered retries and unresolved failures visible.
• Test account boundaries and record every attempt's cost, alongside human checking time.

What Asana's result does, and does not, tell you

In a 9 October OpenAI case study, Asana reports that an optimised GPT-6.1 Sol browser workflow ran 76 times cheaper in estimated model cost than its original setup on another model. The task collected six fields for each of 32 books from a public demo catalogue. This is a supplier-reported comparison of particular configurations on that task, not an independent benchmark or a promise about your production bill.

Asana's own account describes three runs per main condition and cautions that the study shows broad patterns. Some baseline runs hit limits, making the reported reductions lower bounds. Its changes involved retaining browsing history and altering caching and screenshot removal. The practical lesson is to test the whole browser workflow; choosing a model alone does not reproduce the experiment.

Write the success condition before the prompt

Choose a small, repeatable task with known expected output. For catalogue research, specify the records and fields required, then compare the returned values with a prepared reference. For a form submission, check the saved record in the destination system, including its identity and contents. A visible confirmation or an agent saying 'done' should not be your only evidence.

Use a test account and disposable records for write tasks. Include the manual workflow, an existing export or an available API as a baseline where it already handles the job. Keep the starting data, account permissions and success rule consistent between candidate configurations. Changing several of those at once makes any apparent improvement harder to explain.

A recovered retry deserves its own line

Playwright Test's retry guidance distinguishes tests that pass first time from 'flaky' tests that pass after retrying, and from failures that exhaust their retries. That is a useful reporting pattern for an agent pilot, even if the agent uses another tool. Record the initial failure, retry count, final outcome and elapsed time; an eventual pass can conceal repeated work.

Test an expired session and an interrupted save as separate cases. Before replaying a write, check whether it already committed: repeating a successful submission could create a duplicate. A restarted browser does not prove the destination is unchanged. Set a clear point at which the agent stops for a person, rather than guessing whether to repeat an uncertain action.

Use realistic permissions without sharing live credentials

Playwright's authentication documentation warns that stored browser state can contain cookies and headers usable to impersonate an account. Keep that state out of repositories. It also recommends separate accounts for parallel workers when tests modify shared server-side state. Isolated browser sessions alone do not establish independent test data or account permissions.

For the pilot, document which sites, records and actions each account can reach. Include a task the account must refuse, such as editing outside its assigned project, and define the expected refusal beforehand. For a team working between London and Lagos, name who reviews an uncertain action and how the next colleague receives the evidence. This is a handover design choice, not a claim about either city's connectivity.

Compare three different numbers

• Completion: verified correct jobs divided by attempted jobs, with first-pass and retry outcomes shown separately.
• Cash cost per verified job: all recorded model and other charged service costs for the batch, including failures and retries, divided by verified completions.
• Human effort: minutes spent checking, approving and repairing the batch, reported separately unless you state how that time is priced.

If nothing completes correctly, report that outcome instead of presenting a reassuring cost per success. Set step, time and spend limits before running the batch, and keep enough evidence to explain each failure. Your decision is whether this bounded workflow deserves a wider trial. Follow PING's Innovation desk for more practical guides to putting new tools to work.