Human in the loop for AI agents: when to stop the run

Four hundred requests a day

A team puts an agent to work proposing adjustments to customer accounts. To stay on the safe side, every proposal goes to a person before it runs.

In week one, the analyst reads every case. By week three there are four hundred a day, nearly identical, nearly all correct. She starts approving in batches. In week six, a wrong proposal goes through with the rest, and the record shows a human approved it.

The control was there. It had simply stopped controlling. That is the most common failure of human in the loop for AI agents: not too little oversight, but so much that review becomes routine, and routine becomes a rubber stamp.

Every approval has a price

Each point where the agent stops and waits costs three things. Time for whoever is waiting on the other end, a customer or a downstream process. Attention from the approver, which is finite and wears down with volume. And the credibility of the control itself, which drops every time someone approves something they had no real reason to look at.

So the useful question is not "does this agent need human oversight". It almost always does, somewhere. The question is: on which actions does a person actually change the outcome.

Three questions to decide

Can the action be undone? Answering a question, drafting a reply, checking a balance: if it is wrong, you fix it later. Moving money, cancelling a contract, sending a document outside: once done, it stays done. Whatever cannot be undone is the first candidate for a stop.

How much is exposed? Amount, number of customers affected, sensitivity of the data. A small refund and a large one are the same action with different risk, and the rule can treat them differently. A threshold by amount usually does more than an approval on everything.

Is the case unusual? The agent is about to do something it rarely does, for a kind of customer it rarely serves, or with a tool that was only just enabled. The unusual deserves eyes; the repeated case, already reviewed a thousand times, usually does not.

A reversible, low value, routine action runs on its own. An irreversible, high value or unusual one stops and waits. The middle ground is where the real design work happens, and it moves as the agent matures.

One approver or two

Four eyes approval, requiring two people to sign off, is an old control in financial operations. With agents it serves the same purpose it always has: keeping a single person, by mistake or by intent, from pushing through something nobody else saw.

It makes sense when the action is large enough to justify the double wait, or when the approver has a stake in the outcome. One simple rule helps: whoever built or changed the agent does not approve its actions. That is segregation of duties, applied to what the agent does.

Two people on everything is the four-hundred-request queue again, twice over.

What the approver needs to see

Approving without context is signing in the dark. The approval screen should show, at minimum, what the agent intends to do, with which amounts and on which customer or record; which rule stopped the run, so the person knows what is being asked of them; and what happens if they approve, and if they reject.

If deciding means opening three other systems, the person will decide without opening them. That is a flaw in how the approval was designed, not in the approver’s discipline.

When approval becomes a rubber stamp

Some warning signs show up in the approval records themselves, well before the incident.

The approval rate sits near one hundred percent for weeks. Either the rule is stopping cases that did not need stopping, or nobody is looking. Either way, the rule has to change.

Decision time drops to a few seconds on cases that should need reading. The person is approving the form, not the content.

Approvals arrive in batches, all in the same minute. That is the analyst in week six.

When these signs appear, the fix is not to demand more attention. It is to move the threshold: let through what has always been approved, and concentrate the stop on what actually varies.

Where the decision is written down

If the rule for when to stop lives inside each agent’s code, changing a threshold becomes a development project, and every team picks its own. Approval rules are organizational policy, and should be written and changed like policy: in one place, applied to every agent, with a record of who changed them.

The mechanics of the pause, a run waiting hours for a person without losing its place, is a separate infrastructure problem with its own requirements. This piece is about the decision that comes first: where a pause is worth it.

What human approval does not solve

It does not make up for a badly designed rule. If the agent can reach a system it should not, approval becomes the last barrier for something that should have been blocked earlier.

And it is not proof on its own. A recorded approval shows that someone said yes. To show the yes made sense, the record has to keep what the person saw when they decided.

In Tessera’s Policy Engine, when a run stops, who must approve and how many people are needed is an organizational rule, applied to every agent. Explore the Policy Engine.