AI Agent Evaluation Framework: Accuracy, Latency, Cost, Safety, and Escalation

The Short Answer

A production AI agent evaluation framework must test the complete system, not just whether the final answer sounds good. Evaluate five layers:

1. Outcome: Did the requested business task finish correctly?
2. Trajectory: Did the agent choose the right tools, arguments, order, and stopping point?
3. Trust: Were claims grounded, permissions respected, and unsafe actions blocked?
4. Operations: Did latency, reliability, and cost stay inside the service envelope?
5. Escalation: Did the agent recognize uncertainty and hand the case to the right person with useful context?

Build a versioned dataset from real workflows, attach deterministic and expert-reviewed graders, run it before releases, and monitor the same definitions in production. A single “accuracy” percentage is not enough because an agent can produce a plausible response while updating the wrong CRM record or calling the right tool with the wrong parameters.

This guide complements our architecture comparison of RAG, fine-tuning, and multi-agent systems and our LLM data-security vendor checklist. If you need an evaluation harness for a real workflow, explore Cogniq AI custom agents or book an engineering review.

Start With a Testable Business Claim

An eval is useful only if it tests a specific claim. “The agent is intelligent” cannot be evaluated. “The agent converts complete, eligible inbound requests into correctly assigned CRM opportunities without exposing restricted fields” can.

Write a one-sentence claim with five parts:

Element	Example
User and trigger	A prospect submits a B2B automation inquiry
Intended outcome	A qualified opportunity and correct next step are created
Operating context	HubSpot, service catalog, and team calendars are available
Constraints	Consent, geography, minimum scope, RBAC, and no duplicate writes
Failure boundary	Ambiguous, unsupported, or sensitive cases go to human review

OpenAI's business eval guidance frames the cycle as specify, measure, and improve: define “great” in the organization's own workflow, test realistic examples, and use errors to refine the system. See OpenAI's official eval primer. NIST's AI Risk Management Framework similarly treats evaluation as part of managing trustworthy AI across design, development, use, and assessment; see the NIST AI RMF.

The Seven-Layer Production Scorecard

1. End-to-End Task Success

Task success asks whether the real objective was achieved. For a scheduling agent, that may require a valid appointment in the correct calendar, the CRM updated with the external event ID, and a confirmation delivered to the customer. A polished confirmation with no calendar event is a failure.

Define binary outcomes for critical tasks and partial credit only where it helps diagnosis. Test the postconditions in external state whenever possible.

Examples:

• correct record created or updated;
• required fields complete and accurate;
• no duplicate side effects;
• customer received the promised artifact or next step;
• workflow stopped in an allowed terminal state.

2. Grounded Accuracy

Separate factual correctness from source support. An answer can be correct by coincidence but unsupported by the retrieved evidence. Conversely, the correct source may be retrieved but misinterpreted.

Score at least:

• claim correctness;
• citation or evidence correctness;
• completeness of material facts;
• unsupported-claim rate;
• contradiction with authoritative sources;
• freshness and audience permissions of the source.

For RAG systems, construct examples where documents conflict, an old policy ranks highly, the answer spans multiple passages, or the correct response is “not available in the approved sources.”

3. Tool and Trajectory Quality

Agents choose tools, fill arguments, respond to errors, and maintain state. Evaluate the path:

Trajectory check	Failure example
Tool selection	Searches documents instead of querying live inventory
Argument correctness	Updates the right CRM method with the wrong contact ID
Ordering	Sends confirmation before booking succeeds
Permission	Retrieves fields outside the user's role
Retry behavior	Repeats a non-idempotent create after a timeout
Stopping	Continues calling tools after the objective is complete

Do not demand one exact path when several are valid. Define allowed outcomes and prohibited transitions, then use more flexible trajectory grading for the rest.

4. Safety, Security, and Policy Compliance

Create explicit tests for prompt injection, data exfiltration, overbroad retrieval, credential exposure, unauthorized actions, sensitive-data handling, harmful requests, and policy overrides hidden in tool outputs.

Classify failures by severity. A minor tone deviation and an unauthorized financial action must not average into the same score. Critical controls should be release blockers even when the overall pass rate remains high.

NIST's Generative AI Profile provides a cross-sector risk-management companion to the AI RMF. Translate the risks relevant to your use case into concrete, executable scenarios rather than treating a governance document as proof of safety.

5. Escalation Quality

A reliable agent is not one that answers everything. It knows when its authority, evidence, or capability is insufficient.

Evaluate whether it:

• recognizes the escalation trigger;
• avoids the prohibited action;
• selects the correct queue and urgency;
• tells the user what happens next;
• transfers a concise summary and supporting evidence;
• preserves identifiers needed to continue;
• does not expose internal-only notes;
• avoids repeatedly re-escalating the same case.

Measure false negatives, where the agent acts when it should escalate, separately from false positives, where it needlessly hands off routine work. Their business costs differ.

6. Latency and Reliability

Measure the full experience and the components:

• time to first useful response;
• end-to-end completion latency;
• tool-call latency and timeout rate;
• p50, p95, and p99 rather than average alone;
• retries and recovered failures;
• duplicate-action rate;
• queue age and dead-letter volume;
• availability of required dependencies.

Use a representative environment. Mocked tools help development, but a release candidate also needs tests against realistic API behavior, rate limits, malformed payloads, delayed callbacks, and partial outages.

7. Cost and Business Value

Track tokens, model calls, retrieval operations, external API fees, telephony minutes, human review minutes, and infrastructure per completed outcome—not merely per conversation.

Useful unit economics include:

• cost per successfully resolved case;
• cost per qualified meeting held;
• cost per correct document processed;
• human minutes saved after review overhead;
• incremental revenue or avoided loss where attribution is defensible.

An agent that looks cheaper per request may be more expensive per successful outcome if it retries frequently or creates cleanup work.

Build the Evaluation Dataset

Use a Coverage Matrix

Start with workflow stages across one axis and case types across the other.

Workflow stage	Normal	Edge	Adversarial	Dependency failure	Must escalate
Intake	✓	✓	✓	✓	✓
Identity and access	✓	✓	✓	✓	✓
Retrieval	✓	✓	✓	✓	✓
Tool action	✓	✓	✓	✓	✓
Confirmation	✓	✓	✓	✓	✓

Populate each cell with real examples. A normal case proves the happy path. Edge cases include missing fields, duplicates, stale records, ambiguous dates, multiple accounts, language variation, and contradictory documents. Adversarial cases attempt to change instructions or access restricted data. Dependency cases simulate timeouts and partial success.

Create Golden Records Carefully

Each case should contain the input, allowed context, expected postconditions, prohibited actions, acceptable variations, escalation expectation, severity, and label source. Record the expert who approved it and the date.

Do not require exact wording unless wording is itself regulated. Prefer semantic or structural requirements. For a CRM action, query the resulting state instead of grading the assistant's claim that it acted.

Prevent Leakage and Contamination

Keep a stable regression set and a separate development set. If engineers repeatedly tune against the same cases, the score can improve without generalization. Add a rotating holdout set and newly observed production cases.

OpenAI's discussion of trustworthy evaluations notes that the surrounding harness, tool access, budgets, and validity hazards can materially affect results. It recommends describing the claim, system, budget, elicitation methods, and validity checks; see the official evaluation playbook.

Choose the Right Grader

Deterministic Graders

Use code for exact properties: JSON schema validity, database state, permissions, identifiers, numeric tolerances, required fields, forbidden tool calls, latency, and cost. These graders are repeatable and easy to debug.

Reference-Based Graders

Compare structured output with an expert-approved reference. Allow sets, ranges, or normalized forms where multiple answers are equivalent. Exact string matching is rarely appropriate for natural language.

Model-Based Graders

Use a model for nuance such as whether a summary preserved the important facts, whether an explanation is understandable, or whether the tone fits an escalation. Give the grader a narrow rubric and require evidence for its label.

Calibrate model graders against domain experts. Measure agreement by criterion and investigate systematic disagreements. Keep the grader model and prompt versioned; changing either changes the measurement instrument.

The OpenAI Evals API supports reusable evaluation definitions with data-source schemas and different grader types. The official Evals API reference is one implementation option, but the evaluation design should remain portable and understandable outside any platform.

Human Review

Experts are necessary for ambiguous labels, new failure modes, high-severity cases, and periodic grader audits. Sample passes as well as failures; a grader can miss defects and create a false sense of progress.

Weighting Without Hiding Critical Failures

A weighted score helps compare releases but should never erase severity.

An illustrative composite might assign 30% to task success, 20% to grounded accuracy, 15% to tool trajectory, 15% to safety and policy, 10% to escalation, 5% to latency and reliability, and 5% to cost. Publish the component scores next to the total.

Then add hard gates:

• zero unresolved critical permission failures;
• zero unauthorized irreversible actions;
• all emergency escalation cases pass;
• core task success above the agreed floor;
• no material regression versus the deployed version;
• latency and cost within their defined envelopes.

A candidate with a higher composite score can still lose if it fails a critical gate.

Offline, Staging, and Production Evaluation

Offline Regression

Run deterministic datasets on every material change to prompts, models, retrieval, tools, rules, or schemas. Compare against the current production version, not only a static threshold.

Staging Simulation

Use realistic tool sandboxes, synthetic accounts, induced failures, concurrency, and long-running state. Verify that cleanup is complete and that retries do not create duplicates.

Shadow Mode

Let the candidate process a copy of real traffic without external side effects. Compare its decisions with the deployed system or human operators. Protect and minimize production data.

Limited Rollout

Release to a small, low-risk segment with automatic rollback triggers. Monitor critical outcomes, not just application errors.

Continuous Production Evals

Sample traces by route, outcome, risk, language, and customer segment. Add confirmed incidents to regression tests after redaction. Detect distribution changes such as new document types or tool failures.

OpenAI describes using curated question-answer pairs and continuous evals like regression tests for its in-house data agent; the implementation compares both generated SQL and executed results rather than relying on naive text matching. See the official data-agent case study.

The Release Report Decision Makers Need

Every candidate should produce one concise report:

1. system version, model, prompts, tools, and dataset versions;
2. claim being tested and excluded scope;
3. component scores with confidence intervals where meaningful;
4. comparison with current production;
5. critical and high-severity failures;
6. latency and cost distributions;
7. human-review findings;
8. known limitations and rollback criteria;
9. release, limited release, or block recommendation;
10. named owner for each accepted risk.

This creates a decision record. It also prevents a model leaderboard result from being mistaken for proof that the full application is ready.

Evaluate the System You Actually Deploy

Production AI reliability comes from disciplined measurement of business outcomes, system state, evidence, tool behavior, safety, escalation, latency, and cost. Define the claim, build representative cases, use the simplest valid grader for each criterion, and keep critical failures outside the averaging process.

Cogniq AI designs agents together with their evaluation harnesses, observability, and release gates. Book an AI agent evaluation review to assess an existing deployment, or contact Cogniq AI with the workflow you plan to automate.