TL;DR & Quick Summary
An AI agent that can issue refunds, update records or email customers needs a human somewhere in the loop. Requiring approval for everything, though, turns the agent into a slow suggestion box, and reviewers start clicking "approve" without reading. Good design gates the few actions where a mistake is expensive or can't be undone, lets the rest run with logging, and uses real approval data to decide when an agent has earned more autonomy.
Score actions, not agents: reversibility, blast radius and external visibility decide which actions need a gate
Three patterns: pre-action approval, confidence-based routing and post-action review, used together
Approve the exact payload: an approval should authorise one specific action, not a general intention
Design against rubber-stamping: people over-trust automated recommendations unless the interface works against it
Earn autonomy with data: move action types to less oversight only when approval records support it
Key Takeaway: Human-in-the-loop is not a single approval button. It's a set of controls sized to the risk of each action, designed so that people can realistically do the reviewing.
Get Started: Building an agent that needs to act in real systems? Schedule a strategy call with Cogniq AI or see how we build custom AI agents with approval controls from day one.
Why Agents Need Gates
A chatbot that gives a wrong answer produces a wrong answer. An agent that takes a wrong action produces a wrong refund, a wrong email to a customer, or a changed record that other systems then rely on. The gap between those two failure modes is the whole reason approval design matters.
The OWASP Top 10 for LLM applications lists this risk as excessive agency, with three causes: excessive functionality, excessive permissions and excessive autonomy. Its recommended mitigation for the third is direct: use human-in-the-loop control to require a human to approve high-impact actions before they are taken. Approval also protects against prompt injection. An instruction hidden in an email or document that tricks the agent into proposing a harmful action still has to get past a person.
The design question isn't whether to include humans. It's which actions, at what point, and how to make the review real.
Step 1: Score Every Action
List every action the agent can take, meaning every tool call that changes something outside the conversation. Score each on three dimensions:
| Dimension | Low risk | High risk |
|---|---|---|
| Reversibility | Easily undone (draft saved, tag added) | Hard or impossible to undo (payment sent, email delivered, record deleted) |
| Blast radius | One record, small amount | Many records, many customers, large amounts |
| External visibility | Internal only | Seen by customers, partners, regulators or the public |
Then choose a control per action:
| Risk profile | Control | Example |
|---|---|---|
| Low on all three | Automatic, logged | Categorise a support ticket, add a CRM note, draft a reply |
| Reversible but visible or moderate impact | Confidence-based routing | Send a routine order-status reply; escalate unusual ones |
| Hard to reverse, or high blast radius | Pre-action approval | Issue a refund above a threshold, change a contract price |
| Very high impact or regulated | Two-person approval | Large payments, bulk record changes, account closures |
Most workflows end up with many automatic actions and a small number of gated ones. If your list has everything gated, either the agent has more permissions than it needs or the risk scoring is too cautious to be useful.
Remove capabilities before you gate them. If the agent never needs to delete records, don't give it a delete tool at all. An approval gate is weaker than a permission that doesn't exist. We cover permission scoping in more depth in LLM data security for B2B vendor evaluation.
Step 2: Choose the Approval Pattern
Pre-Action Approval
The agent proposes, a person approves, and then the action executes. It's the safest pattern and the right one for irreversible actions. The cost is delay and reviewer workload, so keep it for actions that need it.
Confidence-Based Routing
The agent handles cases automatically when its confidence is high and sends the rest to a person. The weakness is that model confidence scores are often poorly calibrated. Route on observable signals instead where you can: the amount is over a threshold, the customer is flagged, the request doesn't match a known category, the retrieved policy documents conflict, or the customer has contacted you several times about the same issue.
Post-Action Review
The agent acts and a person reviews afterwards, either all actions or a sample. This keeps the workflow fast and still catches systematic problems. Use it only for actions that can be reversed within the review window.
Combining Them
In a real deployment one agent uses all three. A support agent might categorise tickets automatically, send standard replies with confidence routing, and hold refunds above a set amount for approval. Anthropic's guide to building effective agents describes agents pausing for human feedback at checkpoints or when they hit blockers. The point is that the checkpoints are placed deliberately, not attached to everything.
Step 3: Make the Approval Control Execution
A common and dangerous design is an approval step that is just a message: "The agent wants to issue a refund. Approve?" The person clicks yes, and the agent then runs whatever it decides to run. Nothing ties the approval to the actual action.
A sound design separates propose from commit:
- Propose: the agent produces a structured action payload, for example: refund, order 18472, £240, reason code, and the evidence it relied on. The payload is stored.
- Review: the approver sees exactly that payload, plus the evidence and a diff of what will change.
- Commit: only the approved payload executes, checked against a hash or version of what was approved. If anything changed, it goes back for review.
- Verify: after execution, confirm the target system reflects the expected change, and record the result.
Add three safeguards:
- Expiry: approvals should lapse after a set time. An approval from yesterday may not fit today's account state.
- Idempotency: a retried action shouldn't execute twice. Use idempotency keys so a network retry doesn't issue two refunds.
- Separation: the agent shouldn't be able to approve its own proposals or change the approval record.
Cloudflare's human-in-the-loop documentation is a useful reference for implementing durable waits, where an agent pauses and resumes cleanly after a human decision.
Step 4: Design Against Rubber-Stamping
The biggest threat to human-in-the-loop isn't the agent. It's the reviewer. Research on automation bias, including Parasuraman and Manzey's widely cited review, shows that people monitoring automated systems tend to over-trust their recommendations, especially when the system is usually right and the workload is high. An approval queue of a hundred mostly correct proposals is exactly those conditions.
Design choices that counter it:
- Show evidence, not just the conclusion. Display the retrieved policy, the customer history and the calculation, so the reviewer checks reasoning rather than trusting it.
- Show the diff. "Change price from £1,200 to £960" is reviewable. "Update the contract" isn't.
- Require a reason for high-risk approvals. A short dropdown or note makes approval a decision, not a reflex.
- Keep queues small. If a reviewer faces hundreds of items a day, the gate is too broad. Narrow what it catches.
- Test the reviewers. Occasionally insert a known-incorrect proposal and check it gets caught. If it doesn't, the gate isn't working, however good the approval statistics look.
Where oversight is legally required, this isn't optional. Article 14 of the EU AI Act requires that people overseeing high-risk systems remain aware of the tendency to over-rely on automated output, and that they can override the system or stop it safely.
Step 5: Measure the Loop
Human-in-the-loop produces some of the most useful data you have about an agent. Track, per action type:
| Metric | What it tells you |
|---|---|
| Approval rate | How often proposals are accepted unchanged |
| Edit rate | How often reviewers change the proposal before approving, which is the most useful signal |
| Rejection rate | How often the agent proposes something wrong |
| Time to decision | Whether the gate is a bottleneck for customers |
| Escalation precision | Whether cases sent for review actually needed it |
| Post-action reversal rate | How often automatic actions later have to be undone |
Feed edits and rejections back into your evaluation set. Each corrected proposal is a labelled example of the agent getting something wrong. Our AI agent evaluation framework covers how to turn these into regression tests.
Step 6: Earn Autonomy, One Action Type at a Time
Grant autonomy per action type, not per agent. For example:
- A new action type starts with pre-action approval
- After a meaningful run of decisions with a very low edit and rejection rate, and where errors are recoverable, it moves to confidence routing
- After a further run with a low reversal rate, it moves to post-action review on a sample
- If edit, rejection or reversal rates rise, it moves back, automatically if possible
Set the thresholds before launch and write them down. Otherwise pressure to speed things up will move actions to autonomy on the strength of a good week rather than a good record. Irreversible, high-impact actions may never leave pre-approval, and that's fine.
How This Fits the Wider Architecture
Approval gates are one layer of the production control stack. They sit alongside least-privilege permissions, input and output validation, logging and monitoring, and the ability to switch the agent off. The NIST AI Risk Management Framework is a helpful structure for documenting these controls if your organisation needs formal governance.
For how these layers fit into the overall system, see our guide to enterprise AI agent architecture. Approval design also affects budget: approval interfaces, audit logging and review tooling are real engineering work, which is one reason agents that take actions cost more than agents that only answer questions. AI agent development cost covers the ranges.
Conclusion
Good human-in-the-loop design makes agents both safer and more useful. Score each action by reversibility, blast radius and visibility. Gate the few that need it. Make approval bind to the exact action that executes. Design the review so people actually review. Then let approval data, not optimism, decide when an agent gets more autonomy.
Schedule a strategy call with Cogniq AI to design the approval model for your agent's workflow, or learn more about our custom AI agent development.
