Google AQuA: Monitor Silent Production Agent Failures

An AI agent can return a successful HTTP response, stay within its latency target and still fail the user's task. It may ignore a revised instruction, call the wrong sub-agent or produce an answer that conflicts with tool state. Infrastructure monitoring sees a healthy service. The user sees a bad outcome.

Google's Ambient Quality Agent, or AQuA, released as an open-source reference implementation on 8 October 2026, is designed for this gap. It reads production trajectories, identifies repeated failure patterns and can diagnose them against the source snapshot that produced the sessions.

Installing another dashboard is easy. The enterprise work lies in building an operating loop that turns production evidence into safe, reviewable improvements.

What AQuA does

AQuA runs beside an Agent Development Kit agent, outside its request path. The published implementation can read conversations from BigQuery Agent Analytics, linked Cloud Trace data or the default telemetry produced by agents-cli. It can run on a schedule, after a deployment or on demand.

Each sweep follows five stages:

1. Sample: select recent production sessions, with a default cap of 1,000 per run.
2. Review: grade each session against common breakdowns, the agent's configuration and an optional domain goal.
3. Cluster: group findings that appear to share a failure mechanism.
4. Verify: check candidate clusters against full transcripts and discard unsupported clusters.
5. Track: retain verified issues as new, recurring or resolved insights.

Root-cause analysis is a separate, on-demand step. It compares failing trajectories with the immutable source snapshot captured for that deployment revision. When the problem is in the repository, it can cite file and line ranges and propose an edit. When the failure comes from an upstream system or handoff, it can attribute the problem without pretending that a code change will fix it.

AQuA does not apply changes or open pull requests by itself. That boundary is important: diagnosis and remediation remain separate decisions.

Why normal observability is not enough

Latency, error rate and resource usage answer whether the service operated. They do not answer whether the agent respected the latest user constraint, selected the correct tool or completed the business task.

Online evaluation scores help, but a falling average still leaves an engineering team with three questions:

• Which user journeys are failing?
• Are several failures instances of the same defect?
• Which deployed instruction, tool contract or orchestration decision caused them?

The outer loop joins those questions. It uses production traces to discover a pattern, keeps the evidence sessions attached and then checks the deployed revision for a plausible root cause. The approach complements the pre-release evaluation discipline described in spec-driven agentic workflows; it does not replace it.

Evaluate the path the agent took

Google Cloud published a separate evaluation-engineering report on 16 October 2026 after stress-testing the Google Cloud Developer plugin. Its findings add a useful rule for an AQuA pilot: a polished final answer is weak evidence about what the agent did.

Google's initial 11-prompt suite marked three runs as failures when an automated judge looked only at the closing response. The raw tool-call transcripts showed that the plugin had completed the required checks. In one Cloud Run task, it opened gcloud help run services list before acting, then used its final response to warn about deletion risk instead of repeating that check.

Score the trajectory against execution criteria. For a cloud operations agent, capture whether it loaded the required instructions, resolved identity, checked command syntax, respected scope, asked for approval and handled the tool result. The summary helps reviewers understand the outcome, but it should not be the only evidence.

Keep turn structure intact

Google also found a sequencing problem that a flat event log would hide. In one evaluation, the harness dispatched a skill-loading request and two documentation searches within the same tool-call array. The searches ran before the skill instructions entered the context. Once those calls are flattened, the trace looks complete even though the governing instruction arrived too late.

Preserve turn boundaries, tool-call arrays, completion timestamps, retries and the instruction set visible at each decision. AQuA can then help reviewers distinguish a model decision from a harness-ordering defect. This matters for parallel MCP workflows, where the same final answer can conceal a different execution path.

Pair live and plan-only cases

A live evaluation with sandboxed credentials exposes command syntax, parameter serialization, permission errors and the agent's restraint around destructive actions. A plan-only evaluation removes credentials and asks the agent to explain its intended route. That second condition can reveal missing policy checks or approval steps that an early permission failure would otherwise hide.

| Condition | What it exposes | What it can miss | |---|---|---| | Live, sandboxed run | Real tool syntax, permissions, API behaviour and safety restraint | Reasoning that never executes after an early access failure | | Plan-only run | Intended routing, policy checks and approval logic | Failures caused by real authentication, serialization or API state |

Google reports that roughly half of its important findings came from the difference between these two conditions. That is a result from one plugin evaluation, not a benchmark for other systems. The useful pattern is the comparison itself.

Add both conditions to the same golden set. Each case should define the expected business outcome, allowed tools, required checks, approval boundary and evidence that must survive the run. Re-run the set when the model, harness, plugin manifest, skill or tool permissions change.

Define the review contract first

Do not begin a pilot by reviewing every conversation with a broad judge prompt. Begin with a written contract for what the agent must preserve.

AQuA supports a plain-English goal.md that is appended to review prompts. A useful goal describes observable business invariants. Writing-style preferences belong outside this quality contract. For an order-support agent, the invariants could include:

• never confirm a refund before the payment tool records it;
• retain the latest delivery address after the user changes it;
• route regulated-product questions to the approved specialist; and
• ignore greetings, tone differences and abandoned browsing sessions.

Add deterministic metrics where a rule can be checked in code. An LLM judge can assess whether a handoff lost context, while a Python check can confirm that every completed refund contains an approved transaction identifier. Keeping both gives the team a stable measure alongside a broader semantic review.

The review contract should also name the evidence required for an actionable finding: session identifier, affected turn, expected behavior, actual behavior, deployment revision and the source or external step implicated by the diagnosis.

Treat model-generated findings as claims

An agent evaluating another agent can be wrong. AQuA addresses this by separating candidate clusters from verified insights, but a bounded verifier is still not a full audit of every session.

Google's implementation checks each candidate cluster against up to three complete transcripts. That can establish that the pattern exists. It does not prove that every session assigned to the cluster failed for the same reason. A large trace count is useful for prioritisation, not a substitute for evidence review.

Create three gates for the pilot:

| Gate | Required evidence | Decision owner | |---|---|---| | Accept an insight | At least one complete trace supports the stated actual and expected behaviour | Product or domain owner | | Accept a root cause | The cited source lines or external step explain the observed failure | Agent engineer | | Accept a fix | The original failure replays successfully and the wider evaluation suite does not regress | Engineering reviewer |

Keep rejected clusters and failed review work visible. A missing trace window, rubric error or verification cap should be reported as incomplete analysis, not counted as a clean run.

Design sampling around business risk

Random sampling makes run cost predictable and gives broad coverage. It can miss rare failures in high-impact workflows. If a critical defect affects 1% of sessions, a small random sample may not see it at all.

Use several sampling lanes:

• a general random sample for broad quality drift;
• complete coverage for high-risk actions such as refunds, identity changes or approvals;
• targeted samples after model, prompt, tool or orchestration changes;
• sessions with user corrections, repeated tool calls or abandoned workflows; and
• known failure cases retained as regression tests.

Do not filter only for obvious anomalies. That can produce hundreds of versions of the same timeout while missing a quiet semantic error whose latency and tool counts look normal.

Sampling policy belongs to the product risk model. Teams should decide it before choosing a daily or deployment-triggered cadence.

Put telemetry and source snapshots inside the security boundary

Production trajectories may contain personal data, internal identifiers, retrieved documents and tool arguments. Source snapshots may expose instructions, tool schemas and application logic. A quality system needs access to both, so its data boundary deserves the same scrutiny as the agent itself.

Google's reference architecture keeps transcripts, BigQuery tables and source snapshots inside the customer's Google Cloud project. Its dashboard can run on Cloud Run behind Identity-Aware Proxy. Those defaults still require local decisions about data minimisation, redaction, access roles, retention, regional requirements and incident logging.

For the pilot, document:

1. which trace fields are collected;
2. which fields are removed or tokenised before evaluation;
3. who can inspect full conversations and source snapshots;
4. how long raw traces, clusters and resolved insights remain; and
5. how a reviewer can link an insight back to the authorised production record.

Avoid placing secrets in prompts or tool outputs before they reach telemetry. Redaction after collection reduces exposure in the evaluator, but it does not undo the original log event.

Measure the cost of the outer loop

Deep evaluation consumes more tokens than infrastructure checks. Cost grows with sampled sessions, trajectory length, spans per session, the number of candidate clusters and the amount of source context used for diagnosis.

Record these measures for every run:

• sessions and spans reviewed;
• input and output tokens by stage;
• candidate clusters and verified insights;
• time from production session to triaged finding;
• reviewer minutes per accepted insight; and
• defects converted into reproducible evaluation cases.

Google reports that its reference implementation caps sampling and verification to keep costs bounded. That is a useful starting mechanism, but the right budget depends on the value and risk of the monitored workflow. A low-volume financial approval agent may justify complete review. A high-volume information assistant may need stratified sampling and cheaper deterministic filters first.

Close the loop without automating the wrong thing

The fastest path from insight to production is not necessarily the safest. A diagnosis can point to a prompt line, yet the correct repair may be a stricter tool schema, a state-management change or an upstream authorisation check.

Use a controlled sequence:

1. triage the insight with a domain owner and agent engineer;
2. reproduce the failure with captured inputs and mocked or isolated tool state;
3. add the case to the durable evaluation set;
4. implement the smallest change that addresses the identified cause;
5. run focused and broad regression evaluations;
6. create a pull request with the evidence, result and rollback plan; and
7. deploy through the existing approval process, then watch for recurrence.

This keeps production traffic as a learning signal without turning it into an automatic prompt-editing loop. The same principle applies when deciding which workflows deserve an agent: error impact, recoverability and a measurable definition of success must shape the autonomy granted.

A practical pilot definition

Choose one agent with reliable OpenTelemetry traces and a workflow whose success can be reviewed. Run the pilot across at least one normal period and one controlled change. Do not use the first run to prove that the tool works; use it to calibrate reviewers and reveal where the goal, metrics and sampling policy are vague.

Exit the pilot only when the team can answer:

• Which production sessions are eligible for review?
• How is sensitive trace data protected?
• What turns a candidate cluster into an accepted finding?
• Who owns each class of root cause?
• Can a failure be replayed without affecting production systems?
• Does the fix improve a stable metric without damaging the wider task?

Bayseian helps enterprise teams build and evaluate governed AI systems across models, tools and production workflows. If your agents are live but their quality signals stop at latency and error rates, explore our AI systems engineering work.

Sources

• Google Developers Blog: The Outer Loop, Insights First — 8 October 2026
• Google ADK Recipes: Ambient Quality Agent README — accessed 12 October 2026
• Gemini Enterprise Agent Platform: Agent evaluation — updated 9 October 2026
• Google Cloud: Four evaluation engineering lessons learned from building agent plugins — 16 October 2026