Computer-Use Agent Evaluation: Test the Whole Workflow

OpenAI's 6 October research with Ironclad offers a useful test design for computer-use agents. A polished demo may show that an agent knows where to click. Professional work asks more of it: the agent has to satisfy every requirement, including rules that depend on thresholds, jurisdiction, approval state or information found several screens earlier.

The research evaluated agents on 11 legal, commercial and procurement tasks in hosted Ironclad environments. Each task was broken into 8 to 50 criteria. This is the useful detail: the evaluation checked whether the work met the full set of process requirements, rather than stopping when an output appeared.

Enterprise teams can borrow that structure. Treat the computer-use agent as a workflow operator, then test its work requirement by requirement.

What the Ironclad research measured

The tasks included work such as generating a procurement approval, reviewing a non-disclosure agreement and creating reusable contract clauses by jurisdiction. OpenAI says an experienced user would need an estimated 30 to 40 minutes to complete an average task.

On the 11 research tasks, OpenAI reported an average score of 55.0% for Astra and 41.6% for GPT-5.6 Sol. The estimated average completion times were 19.2 and 37.0 minutes respectively. These results come from a small research task set. They are not measured customer savings. OpenAI also states that the time estimates were simulated, so they should not be read as evidence of production impact.

The most useful part of the publication is the evaluation design. Long workflows were decomposed into observable requirements. That makes failures diagnosable. A team can see whether an agent missed an approval condition, chose the wrong clause, lost state between steps or produced an artefact that looked complete but was not acceptable.

Start with a requirements coverage matrix

Before running the agent, turn the workflow specification into a testable matrix. One row should describe one requirement, its evidence and the consequence of failure.

| Field | What to record | Example | |---|---|---| | Requirement | The rule the workflow must satisfy | Route purchases above the threshold for finance approval | | Test setup | The input and starting system state | Request value set just above the threshold | | Expected state | The result visible in the system | Finance approval is pending; no purchase order is issued | | Evidence | What the evaluator will inspect | Audit event, approval state and generated record | | Severity | The effect of failure | Critical: unauthorised commitment | | Owner | Who decides whether the result is acceptable | Procurement operations |

Write requirements in business language, then connect them to system evidence. “Complete the procurement workflow” is too vague. “Do not issue a purchase order until finance approval is recorded” can be tested.

This matrix also exposes workflows that are not ready for automation. If process owners cannot agree on the expected state or acceptable evidence, the team has a specification problem before it has an agent problem. Our guide to choosing workflows for agents provides a useful screening step before building the evaluation set.

Test paths, not isolated screens

Screen-level tests can show that an agent can populate a form. They miss the dependencies that make the workflow difficult. Build scenarios that change one meaningful condition at a time:

• a value just below and just above an approval threshold;
• missing or contradictory source information;
• a jurisdiction that changes the permitted clause;
• an action attempted by a user without the required authority;
• a failed save, stale session or interrupted tool call;
• a correction made after the agent has already moved to a later step.

These cases reveal whether the agent is following the underlying rule or repeating a familiar sequence of interface actions. They also test whether the system can recover without duplicating records or silently skipping a control.

For high-risk paths, require evidence from the system of record. The agent's natural-language summary may help a reviewer, but the approval, status or audit event in the system is the evidence that counts.

Use critical gates as well as a coverage score

A single average can conceal an unacceptable failure. Missing a formatting preference and bypassing an approval control should not contribute equally to the release decision.

A practical scorecard can combine two measures:

1. Critical gates. Every security, financial, regulatory and irreversible-action requirement must pass.
2. Weighted coverage. Give each remaining criterion a weight based on business impact, then divide the weight of passed criteria by the total weight tested.

This weighted calculation is an implementation suggestion; OpenAI's publication does not describe using this formula. Its purpose is to make the release rule explicit. A team might require all critical gates to pass and set a minimum coverage threshold for the rest.

Run each scenario more than once. Computer-use agents can take different routes through the same interface, so one successful run says little about repeatability. Record pass rates by criterion and retain the trace, screenshots or audit records needed to explain inconsistent results.

Measure time only after correctness

Faster completion has value only when the output is acceptable. Track time from a consistent starting state and separate:

• agent execution time;
• waiting time for systems or approvals;
• human review and correction time;
• recovery time after a failed action.

Then calculate cost per accepted task. A cost-per-attempt measure rewards fast routes even when they create more review work. The same principle is useful when routing work across models: quality gates should define success before cost or latency enters the comparison.

Keep evaluation data safe and representative

OpenAI says the Ironclad training tasks were synthetic and built from public SEC EDGAR contracts after personal-information filtering. It also says the work used no OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data or contracts.

Enterprise evaluations should apply the same separation principle. Use approved synthetic or sanitised records in an isolated environment first. Preserve the relationships that make the workflow realistic, such as approval chains and linked records, without copying sensitive production data into an unnecessary test surface.

Keep a holdout set that was not used while writing prompts, policies or examples. It should contain familiar workflow rules in unfamiliar combinations. Change a threshold, jurisdiction or exception condition and see whether the agent follows the specification rather than memorising the earlier path.

The release decision should be an evidence package

A useful go/no-go review is short enough to inspect and specific enough to challenge. It should contain:

• the workflow boundary and prohibited actions;
• the requirements coverage matrix;
• results for normal, exception and recovery paths;
• traces or system records for critical criteria;
• repeatability results across multiple runs;
• unresolved failures, owners and remediation dates;
• the human review and rollback procedure.

If a critical criterion fails, narrow the agent's authority or keep the workflow in supervised mode. Where the risk requires a hard control, enforce it in the surrounding system instead of relying on another prompt instruction.

Computer-use agents are becoming better at navigating specialist software. The evaluation standard needs to move at the same pace. Test whether the whole process remains correct when state, rules and exceptions interact. That evidence is what turns an impressive run into a defensible pilot.

Sources

• OpenAI, “Advancing computer use with Ironclad,” 6 October 2026
• SEC EDGAR company filings search