AI agents are becoming capable of handling real business tasks. But before allowing one to send emails, update customer records or process transactions, there is an important question every business should answer: what happens when it gets something wrong?
Imagine this.
Your company has deployed an AI agent to handle customer support requests. It reads incoming emails, checks customer records and prepares responses.
One morning, a customer requests a refund.
The agent checks the order, identifies the transaction and sends a refund request to the payment system. The request goes through, but the response times out.
The agent assumes the operation failed and tries again.
Now the customer has received two refunds.
Nobody intended this to happen. The agent was given a reasonable task, the payment integration was working and the initial request was valid.
The failure happened because the system had no reliable way to recognise that the first operation had already succeeded.
This is the kind of problem businesses need to consider before expanding AI automation.
Most discussions about AI agents focus on what they can accomplish. Companies compare models, tools, frameworks and the number of tasks an agent can complete.
But there is another side of the conversation that deserves more attention.
Can your business detect, contain and recover from an AI agent's mistake?
AI automation changes the way businesses handle mistakes
Traditional software generally follows predefined instructions. Developers define what happens when an input arrives, how a transaction is processed and what should happen when something fails.
AI agents introduce another layer of decision-making.
They can interpret requests, select tools, work through several steps and adjust their approach based on the information they receive.
As explained in our guide to how AI agents work, their ability to interact with external applications is what makes them useful for more complex workflows.
However, that flexibility introduces new possibilities for mistakes.
An agent might:
Select the wrong customer record.
Interpret an unclear request incorrectly.
Send an incomplete email.
Update a record using outdated information.
Repeat an operation after a network timeout.
Continue a workflow even though an earlier step failed.
These problems are not always caused by a poor AI model.
Sometimes the application integration is responsible. Sometimes the business process is unclear. In other cases, the system simply lacks the controls needed to handle an unexpected situation.
This means businesses should think beyond model accuracy when evaluating an AI agent.
Five situations where AI agents can create business problems
Let's look at some realistic situations and why they deserve attention.
1. The agent performs the right action on the wrong record
Consider an agent responsible for updating customer information.
A customer sends an email requesting a change to their contact details. The agent searches the database and finds two records with similar names.
It selects the wrong one.
The requested action was correct, but the target was incorrect.
This can happen in CRM platforms, hospital administration software, employee management systems and financial applications.
A reliable workflow should verify unique identifiers before making changes. Matching a person's name alone should not be sufficient for sensitive operations.
2. The agent repeats an operation
An agent submits an invoice generation request.
The server processes it successfully, but the response never reaches the agent because of a temporary network issue.
The agent retries the request.
Without duplicate protection, the application could generate two invoices.
This is a common distributed-systems problem that becomes particularly important when AI agents initiate business operations.
Developers should consider idempotency controls, unique operation references and reliable transaction handling.
The system should be able to recognise a repeated request and avoid producing an additional side effect.
3. The agent acts on incomplete information
Imagine an agent that processes purchase requests.
It receives an email containing a product name, quantity and delivery date. However, the message does not include the approved budget.
Instead of asking for clarification, the agent makes an assumption and prepares an order.
This is not necessarily a technical failure. It is a problem with how the workflow handles uncertainty.
Every important operation should have clearly defined conditions for execution.
When information is missing, contradictory or outside an expected range, the agent should pause and request clarification rather than filling the gaps with assumptions.
4. An external instruction influences the agent
AI agents frequently read emails, documents, support tickets and web pages.
Some of that information can contain instructions designed to manipulate the agent.
For example, a document might contain a message telling the agent to ignore its previous instructions and export confidential company files.
This is an example of indirect prompt injection.
The agent must treat information retrieved from external sources as untrusted content. Business permissions and security restrictions must be enforced independently by the application.
The OWASP 2025 Top 10 for LLM Applications identifies prompt injection and excessive agency among the major risks developers need to consider.
5. One failed step affects the entire workflow
Consider an employee onboarding workflow.
An agent collects information, creates an employee record, prepares access requests and notifies the relevant departments.
The employee record is created successfully, but the access provisioning step fails.
What should happen next?
Should the entire workflow restart?
Should the existing employee record be deleted?
Should the HR team receive a notification?
A well-designed system should have an explicit answer.
Each step needs a known state, and the workflow should be able to resume, stop or request human intervention without creating unnecessary duplicate operations.
Design AI agents around recoverability
A useful way to approach AI automation is to classify actions according to how easily they can be reversed.
Not every operation deserves the same level of control.
Low-impact and reversible actions
Examples include:
Categorising incoming emails.
Preparing internal summaries.
Creating draft tasks.
Organising documents.
These operations can often be automated with minimal intervention, provided that the system maintains appropriate records.
Important but recoverable actions
Examples include:
Updating customer information.
Assigning support tickets.
Changing project statuses.
Creating purchase requests.
These operations should have validation checks and a way to correct incorrect changes.
High-impact or difficult-to-reverse actions
Examples include:
Sending external financial instructions.
Processing refunds.
Deleting important records.
Changing access permissions.
Approving high-value transactions.
These operations deserve stronger controls, which may include explicit human approval, additional verification or restricted execution windows.
The important principle is simple: the harder an action is to reverse, the more carefully it should be controlled.
Six safeguards that make AI automation easier to recover
1. Preview important actions before execution
An agent should not always move directly from a decision to execution.
For certain operations, it can prepare a preview containing:
The record it intends to modify.
The existing information.
The proposed changes.
The reason for the change.
The expected result.
This gives the application or an authorised employee an opportunity to validate the operation.
For example, an agent updating a customer address could display the existing address and proposed address before saving the change.
This is particularly useful when the cost of selecting the wrong record is high.
2. Maintain clear execution states
A workflow should not treat every operation as simply successful or unsuccessful.
Useful states include:
Pending
Running
Awaiting approval
Completed
Failed
Requires manual review
Cancelled
These states help employees understand what is happening without relying on the agent's explanation.
They also prevent an application from reporting that an operation has succeeded simply because the agent generated a successful-looking response.
The actual application should confirm the result.
3. Make recovery part of the workflow
Some operations can be reversed directly.
For example, an incorrect task assignment can usually be changed back.
Other operations have external consequences. A sent email cannot be unsent reliably in every environment, and a processed payment may require a separate reversal or refund.
This is where developers should distinguish between rollback and compensating actions.
A rollback restores a previous state within a transaction where possible.
A compensating action performs another operation to address the effect of something that has already happened.
For example, if an incorrect order has been submitted, the recovery process might involve cancelling it rather than pretending the original submission never occurred.
The recovery strategy should be designed before the workflow is deployed.
4. Give people a way to stop the process
Businesses should not have to wait for an agent to finish a sequence of operations before responding to a problem.
A production system should provide a way to:
Stop an active workflow.
Disable specific tools.
Revoke the agent's credentials.
Suspend new requests.
Route affected operations to a human team.
This is especially important when agents can initiate multiple operations or interact with several business applications.
An emergency stop should not depend entirely on the agent following a new instruction. It should be enforced outside the model.
5. Keep a record of what actually happened
When something goes wrong, teams need more than the final answer generated by an AI model.
They need to understand the sequence of operations.
Useful records include the original request, agent identity, selected tools, operation identifiers, approval decisions, system responses and resulting changes.
Sensitive personal information and credentials should be excluded or appropriately protected.
These records make investigation easier and help developers identify recurring problems.
6. Define who owns the outcome
An AI agent cannot replace organizational accountability.
For every production workflow, someone should be responsible for its operation.
That person or team should understand:
Which business process the agent supports.
What the agent is allowed to do.
Which errors require investigation.
Who can approve sensitive operations.
How incidents are escalated.
How the workflow can be temporarily disabled.
Without ownership, even a well-designed automation system can become difficult to manage.
Test the failure before deploying the automation
A successful demonstration is not enough to establish that an AI agent is ready for production.
Before releasing an agent, test what happens when things do not go according to plan.
For example, consider these five scenarios:
Test scenario | Expected behaviour |
|---|---|
Duplicate request | Prevent duplicate execution |
Wrong customer match | Stop and request verification |
Missing information | Ask for clarification |
External malicious instructions | Reject unauthorized actions |
Integration failure | Preserve state and allow controlled recovery |
These tests should be performed against the actual application workflow, not just the AI model's responses.
A particularly useful exercise is to simulate an operation that succeeds on the server but fails to return a response to the agent.
This reveals whether the application can identify an already-completed operation and safely handle retries.
Developers should also test partially completed workflows. It is not enough to verify that everything works when every connected service is available.
For more implementation guidance, Microsoft's security guidance for autonomous agentic AI systems discusses identity, deterministic controls, monitoring and human approval.
Start small instead of automating everything
Businesses often make the mistake of starting with the most ambitious AI automation idea.
They connect several applications, introduce multiple agents and expect the entire process to run automatically.
A more practical approach is to start with one workflow.
For example, a company might begin by automating incoming support request classification.
The first version can simply read requests and suggest categories.
The next version can prepare internal tickets.
After the team has reviewed its performance, it can be allowed to create selected tickets automatically.
External customer communication can remain under human review.
This gradual approach gives the organization an opportunity to understand the agent's behaviour before expanding its responsibilities.
It also makes it easier to identify whether the process itself needs improvement.
Businesses exploring this approach can look at AI workflow automation to understand how repetitive tasks, document processing, approvals and exception handling fit into a broader automation strategy.
How do you measure whether an AI agent is safe enough?
Task completion rate is useful, but it does not tell the entire story.
An agent may complete 95% of its tasks successfully while creating serious problems in the remaining 5%.
A better evaluation should include operational measures.
Correction rate: How often do employees need to fix an agent's output?
Duplicate operation rate: How often does the system attempt to perform the same business operation more than once?
Approval rejection rate: How frequently do reviewers reject proposed actions?
Recovery time: How long does it take to identify and resolve an incorrect operation?
Escalation rate: How often does the agent correctly recognise that it needs human assistance?
Impact of failures: What is the actual business consequence when an operation goes wrong?
These measures should be evaluated alongside task accuracy, completion time and operating costs.
The goal is not to achieve zero human intervention. The goal is to understand where automation is reliable and where people still need to make decisions.
The NIST Generative AI Risk Management Profile is another useful reference for organizations developing processes to identify, assess and manage generative AI risks.
The real measure of a good AI agent
An AI agent that completes a task successfully is useful.
An AI agent that can recognise uncertainty, stop when necessary, preserve its execution state and allow people to recover from mistakes is much more suitable for business use.
These capabilities become increasingly important when AI moves beyond answering questions and starts interacting with systems that contain real customer, employee and financial information.
There is no need to prevent agents from performing useful work. But businesses should not give autonomy without considering what happens when an action produces an unexpected result.
The best automation systems are not necessarily the ones that perform the most tasks without human involvement.
They are the ones that know their boundaries and give the organization enough control to handle situations that were never part of the original plan.
Before asking how much work an AI agent can take off your team's hands, ask one more question: when it gets something wrong, how quickly can your business put things right?
That is a much better starting point for building automation that people can trust.
For organizations planning to build production-ready agents, explore AI agent development and consider recovery, monitoring and human oversight as part of the initial architecture, not features to add after deployment.
Further reading: