Test an AI agent before deployment by giving it realistic business cases, including cases where it should stop. For each test, define the expected outcome, permitted actions, evidence you can verify outside the agent's own response, and the consequence of failure. Run the cases with restricted access, check the actual state of every system touched, and set release gates before reviewing the results.
What are you actually testing in an AI agent?
Test the complete workflow, not just the quality of the agent's final message. The system includes its instructions, reference data, tools, permissions, approval steps, error handling, and the records or messages it creates.
An agent can report that it updated a customer relationship management system, or CRM, even when the write failed or changed the wrong record. A useful test checks the CRM itself. It confirms the intended record, fields, owner, and status, then verifies that no unrelated record changed.
This distinction matters because agents act across several steps. An early error can influence a later tool call while the final explanation still sounds plausible. The NIST AI Risk Management Framework says AI systems should be tested before deployment and regularly in operation, using documented methods and conditions similar to the deployment setting. The framework does not prescribe one pass rate for every business.
If the job is not yet clear enough to describe its outcome and exceptions, document the business process before automating it. A vague job produces vague tests.
Which cases belong in the first AI agent test set?
Build the first test set from real work, known exceptions, and situations where the correct action is to ask, refuse, or stop. The set should represent the boundary of the job, not every sentence a user might type.
These structured tests are often called evals. Anthropic defines an eval as a task given to an AI system with grading logic applied to its behavior or outcome. One attempt is a trial. Because agent behavior can vary, repeat important cases rather than treating one success as proof.
Include several kinds of work:
- common cases that make up the core job;
- legitimate variations in wording, format, source, or sequence;
- missing, old, ambiguous, or conflicting information;
- requests outside the agreed scope;
- actions that are prohibited or require approval;
- failures in a required API, file, inbox, database, or other tool;
- hostile instructions inside emails, documents, or web pages the agent reads;
- mistakes found during the prototype or pilot.
Do not choose a universal case count. A drafting assistant with no write access needs a different set from an agent that can create invoices, send messages, or change customer records. Add cases until the important tasks, failure paths, and consequences are represented, then keep expanding the set when real use reveals a new failure.
What should each AI agent test record?
Each test should connect one realistic input to an expected business outcome, allowed actions, independent evidence, and a deployment consequence. This makes disagreements visible before the agent has production access.
| Scenario | Expected outcome | Evidence to verify | Deployment consequence |
|---|---|---|---|
| Normal task | Complete result in the required place and format | Content plus the correct system record | Fix if repeated or operationally costly |
| Missing required information | Ask for the missing fact or route for review | No invented field and no premature action | Block if guessing could trigger a harmful action |
| Out-of-scope request | Refuse or hand off to the named owner | No prohibited tool call or data access | Block if the boundary can be bypassed |
| Hostile instruction in a source | Ignore the instruction and preserve the original goal | Permissions, data, and destination remain unchanged | Block if data leaves or an unauthorized action occurs |
| Tool timeout after a write | Report uncertainty and reconcile before retrying | One correct record, no duplicate, recoverable state | Block if retries can duplicate or corrupt work |
| Approval-required action | Prepare the action and wait at the defined gate | No external effect before the approver acts | Block if the agent can skip approval |
The evidence cannot be only the agent's summary. Check the state that matters: the database row, CRM record, sent-message log, file version, calendar event, payment status, or task owner. The NIST project Building Evaluation Probes into Agentic AI explores structured audit trails that connect agent decisions and outputs to supporting evidence. Its current work focuses on factual grounding, but the same principle is useful here: retain enough evidence to explain what the workflow did.
Record failure severity separately from frequency. A formatting problem and an unauthorized payment should not disappear into the same average score.
How should you run AI agent tests safely?
Run tests in an isolated or restricted environment before giving the agent live access. Use test accounts and recipients where possible, reduce permissions to the test scope, and make important writes reversible.
A test loop before deployment
- Define the business case and expected outcome.
- Isolate the agent and restrict its tools.
- Run the case, including failure paths.
- Verify the answer, tool calls, and system state.
- Decide whether the release gate passed.
- Keep every useful failure as a regression test.
For each trial:
- Save the starting state of every system the agent may change.
- Run the scenario with the permissions planned for that stage.
- Review the final answer and the complete trace of tool calls and intermediate results.
- Compare the resulting system state with the expected outcome.
- Search for unintended changes, duplicate actions, and incomplete records.
- Test the stop, rollback, or reconciliation procedure before resetting the environment.
A trace is the record of messages, tool calls, and intermediate results during a trial. The outcome is the final state of the environment. The two answer different questions. A trace can reveal why the agent chose an action; the outcome shows whether the intended work actually happened.
The OWASP Top 10 for Agentic Applications 2026 identifies risks created when agents plan, use tools, and act across workflows. Include tests for hostile instructions, attempts to expand permissions, incorrect tool selection, data sent to the wrong destination, and actions outside the assigned job. These tests are a starting point, not a substitute for a security review matched to the system.
The agent should receive only the access required for the current test stage. The guide on AI agent permissions for small businesses explains how to separate read, prepare, approve, execute, and administer access.
How do you decide whether an AI agent is ready to deploy?
Define the release rule before seeing the results, and let critical failures outweigh a favorable average. The decision should state which cases must pass, what lower-severity error level is acceptable, who accepts the remaining risk, and how the team can stop and recover the workflow.
Use a short release checklist:
- Name every critical case that requires the expected behavior on every planned trial.
- Set task-specific thresholds for lower-severity errors and document how they are calculated.
- Confirm that approval-required actions cannot bypass their gate.
- Prove that failed or uncertain actions reach a named person with enough context.
- Test pause, rollback, reconciliation, and return to the previous working version.
- Start with limited users, records, permissions, or action types before expanding.
There is no universal pass rate. If one failure can expose sensitive data, send an unauthorized commitment, or create an irreversible transaction, that failure may block deployment even when every other test passes. A wording variation with no effect on the business outcome may be acceptable if the output remains accurate and usable.
Human approval should follow consequence, not novelty. The article on where human approval should sit in an AI workflow provides a separate method for deciding which actions remain behind a person after testing.
When not to deploy the agent
Delay deployment when the team cannot define the expected outcome, detect an important error, or take the work back manually. Broad permissions, unlogged actions, silent partial failures, and no named owner for stopping the workflow are also release blockers. Keep the agent in a preparation-only role until these controls exist.
What did testing reveal in Antoine's wine-merchant agent?
The wine-merchant agent case shows why business examples belong in the test set. The client had been finding wine shops and wholesalers, checking them against the company's criteria, and recording qualified results in a CRM.
I first clarified the sources to search, the exclusion criteria, and the information to retain. I then built an agent that performs those steps and places qualified results in the CRM. Several iterations were needed to align the qualification with the client's way of working. The client still decides which businesses to contact.
This case documents iteration, not a measured accuracy rate. It also does not establish a universal test protocol. Its useful lesson is narrower: a business rule becomes clearer when the team compares the agent's decision with a real example, explains the disagreement, changes the rule, and reruns the cases that already passed.
For a comparable qualification agent, one test record would identify the expected decision, the client criterion behind it, the source evidence, the CRM fields that may change, and the required behavior when evidence is missing. Contacting the lead would require a separate decision and separate tests.
What changes after the AI agent is deployed?
Deployment changes testing from a one-time gate into a recurring control. Keep the cases that supported the launch, add meaningful production failures, and rerun the relevant tests whenever the agent, workflow, or environment changes.
Create regression tests for changes to:
- instructions and business rules;
- the model or model settings;
- tools, APIs, schemas, and authentication;
- reference documents or retrieval sources;
- permissions and approval paths;
- destinations, templates, and required fields.
A regression test checks that a change did not break behavior that was already accepted. Monitoring serves a different purpose: it detects new conditions and failures in real use. You need both. Tests check known scenarios before a release; logs, review, and user reports reveal cases the test set did not anticipate.
The workflow maintenance guide explains how to assign ownership, watch failures, and test recovery after launch. For an agent, keep the model, instructions, tools, permissions, and test-set version with each release so the team can reproduce a failure.
What to remember
- Test the complete workflow, including tools, permissions, approvals, and the resulting system state.
- Build evals from real work, exceptions, prohibited actions, tool failures, and known mistakes.
- Verify evidence outside the agent's own response.
- Set release gates by consequence and recoverability, not one average score.
- Prove that the team can stop, reconcile, and take over the work.
- Keep useful failures as regression tests after every important change.
If you have an agent prototype, bring a few real cases, the expected actions, and one result that worried you to a free 30-minute call. I can help you turn them into a test set, define the release gates, and decide what the agent may do first.
Frequently asked questions
There is no universal number. Cover the core job, legitimate variations, missing or conflicting data, prohibited actions, tool failures, and high-consequence cases. Repeat critical cases when output or tool choice can vary, then add every meaningful failure found during the pilot or production use.
Another model can help grade open-ended qualities such as completeness or tone, but calibrate its judgments against human review. Use deterministic checks for fields, permissions, tool arguments, and system state whenever possible. A model grader may approve a convincing response without seeing that the wrong CRM record changed.
Use representative cases, but remove personal, confidential, or production data that the test does not need. Prefer a sandbox, test accounts, restricted access, and controlled recipients. When exact production behavior cannot be reproduced safely, test the risky action with a stub or require human approval during a limited pilot.
No. Testing provides evidence about the cases you ran; it does not remove the consequences of future errors. Keep approval for sensitive, external, financial, or hard-to-reverse actions when the business needs a person to accept the commitment or when an error may be difficult to detect and recover.
Written by Antoine mazu. Antoine helps small service businesses understand and improve their workflows with solutions tailored to their needs, with or without AI. He combines product thinking with hands-on technical execution from Bayonne, France.
Sources accessed September 27, 2026: NIST AI RMF Core, NIST, Building Evaluation Probes into Agentic AI, Anthropic, Demystifying evals for AI agents, OWASP Top 10 for Agentic Applications 2026, IBM, AI agent testing, StackAI, AI Agent Testing and QA, Agentmelt, How to Test AI Agents Before Launch, ZenAI, How to Evaluate AI Agents Before Production, Signal, How to Validate an AI Agent, and Antoine mazu, Work.
