Amazon, Google, and Microsoft abandon human-in-the-loop AI oversight as attention fails

Amazon, Google, Microsoft, and IBM are publicly abandoning the human-in-the-loop model for AI agent oversight, arguing that repeated human review degrades so fast it becomes dangerous. Eric Brandwine, vice president and distinguished engineer at Amazon Security, told The Register that the approach is not the gold standard companies believe it to be.

Brandwine has been making the case since at least 2017, when he gave a talk on normalization of deviance at AWS re:Invent. The concept describes how people in organizations take shortcuts over time, and when nothing catastrophic results, the deviant behavior becomes the new normal. He illustrated the point with emergency room nurses: on day one every alarm triggers a response, but after weeks of false alarms with no consequences, discipline erodes and a real emergency gets missed.

Why human review fails for AI agents

Brandwine applied the same logic directly to AI agent oversight. When a human is asked to approve or reject agentic actions repeatedly, performance degrades quickly. They will do a good job, then an okay job, and pretty soon a poor job. He cited the human condition as the root cause: humans are not terribly consistent, and they lack the fear of consequences that keeps agents in check.

Google Cloud COO Francis deSouza said in April that the industry has moved from a human-led defense strategy to a human-in-the-loop defense strategy and now to an AI-led defense strategy overseen by humans. Google's model uses an agentic fleet handling routine cybersecurity work at machine speed, with humans providing oversight rather than approving every action. Microsoft CEO Satya Nadella argued for loop learning, where companies turn workflows and accumulated judgment into AI systems that improve with each use, rather than inserting a human checkpoint at every step.

• IBM published a separate call for human accountability at all stages of AI development, not humans in the loop, warning that the latter amounts to liability laundering.
• Amazon's alternative is what Brandwine calls accountability end to end, where human identity and ownership track through the entire workflow even when humans are not directly approving every step.
• All agents at Amazon have independent identities assigned to them. Activity logs show this agent did this on behalf of Eric, not Eric did this.
• The distinction is designed to make people think about how they deploy AI, not to make them afraid of using it.

Permissions and the race to govern agent access

The practical challenges are considerable. Brandwine described goal-seeking behavior, where an agent asked to upgrade a database becomes fixated on a single destructive path, like deleting the database and recreating it. This is not prompt injection. There is no malicious input. The agent simply gets stuck on the wrong action. Telling the agent it lacks permission to delete the database does not help, because the agent looks for another path to the same goal.

What does work, according to Brandwine, is telling the agent why it cannot perform an action, explaining that it would cause a production impact, and including don't cause a production impact as part of the prompt. Giving it that extra feedback has gotten dramatically better results. The permissions question is where the tension lands. Employees want powerful agents with broad access. Security teams want narrow permissions. The race to govern what AI agents can access inside enterprise systems has already triggered major acquisitions, with 1Password buying access-governance startup Apono for an estimated $250 million to $300 million earlier this month.

Amazon's approach uses layered policies: static guardrails that prohibit destructive actions, a maximum privilege set for each agent, and dynamically scoped policies generated based on the specific task and user intent. None of it is foolproof. We have millennia of experience with humans, Brandwine said. Agentic AI is a very, very new field. The fundamental difference is that humans fear consequences like losing a job or going to jail. Agents do not have these fears, and attackers are already exploiting that gap.