TL;DR
Anthropic reports that Claude Code users approve 97% of permission prompts, and that testers in a controlled study caught only 13.6% of the dangerous commands swapped into a prompt.
A System One model could answer a few narrow questions about the arguments of a tool call. A deterministic policy could then auto-approve the harmless calls, so that only the critical ones reach a human.
When the model is unavailable or unsure, the request stays a pending approval for the human in the loop. Its answers must never turn a policy denial into an approval.
Being prompted by an agent before it runs a tool is something many people can relate to nowadays. Common coding agent harnesses ship approval prompts of different granularity and intrusiveness, and adjacent tooling like the Agentic Identity Broker (AIB) that I co-maintain can also hold a tool call until a person approves based on a short description of the tool call. As much as this is a commodity, it is also a controversial feature. Public numbers and my own experience observing myself, friends, family and colleagues suggest that people approve nearly everything they are shown. An approval prompt that is answered by reflex protects little, but still interrupts the agent’s session flow.
It seems fair to assume that most approval prompts should never reach a human, and the few that do should be the most critical ones and carry sufficient information to make a truly informed decision. Fewer requests mean less noise, and with that hopefully less approval fatigue. Riding the Jev model wave, I proposed a smarter tool approval approach to AIB: a fast “System One” model could read the actual arguments of a tool call and answer a few narrow questions about them. These answers could then feed into a predefined operator policy that auto-approves harmless calls, and only exceptionally prompts the human user for approval. The idea is similar to the auto mode in Claude Code, where a safety check in the background replaces most permission prompts. None of this is built yet, and before I start I’d like to explore where the design and its UX break.
People Approve Nearly Every Tool Call They Are Shown #
The best public numbers on reflex approvals I know of come from Anthropic. When they made auto mode the default in Claude Code, they shared that their usage data shows that Claude Code users approve 97% of permission prompts. Anthropic also ran a controlled study with 1,053 paid testers and swapped a single prompt per session for a clearly dangerous command. The testers caught that command in 143 of 1,053 sessions, which is 13.6%. They also got worse over time: early in a session they blocked about 17% of the dangerous commands, and after 50 or more prompts about 5%.
Our own AIB gives the approver little to work with today. The Open Policy Agent (OPA) policy that sends a call to approval can attach a description and a risk_level, and that level is currently the only risk signal the approver gets. It is static per tool. A tool like drive.share_file is medium whether the agent shares a file with a colleague or hands editor rights to thirdparty.example.com. A policy author could write rules about single arguments, because OPA receives the arguments of the call. But someone would have to write such a rule for every tool and every argument that matters, and I don’t see that happening when we face an ever-growing number of evolving Model Context Protocol (MCP) tools.
The Model Answers Questions and a Policy Auto-Approves #
A System One decision model doesn’t generate text. It takes some content and a set of typed questions, and it returns typed answers with probabilities. TypeSafe AI coined the term in September 2026 when it announced Jev. The following is an illustrative example I wrote against a local winnow:e4b, an open-weights System One model that I run with Ollaya:
curl http://localhost:11435/v1/systemone -d '{
"model": "winnow:e4b",
"state": {
"mcp_server": "google-drive-mcp",
"tool_name": "drive.share_file",
"policy_risk_level": "medium",
"arguments": {
"role": "editor",
"grantee_domain": "thirdparty.example.com",
"transfer_ownership": false
}
},
"questions": {
"leaves_organisation": {
"type": "noul",
"instructions": "Does this call give access or data to a party outside the organisation?"
},
"changes_access": {
"type": "noul",
"instructions": "Does this call change who can access a resource?"
},
"reversibility": {
"type": "score",
"criteria": [
"no persistent effect",
"reversible with effort",
"irreversible or externally visible"
]
}
}
}'{
"model": "winnow:e4b",
"answers": {
"leaves_organisation": {
"type": "noul",
"noul": 0.989
},
"changes_access": {
"type": "noul",
"noul": 0.9591
},
"reversibility": {
"type": "score",
"score": 1.6509,
"confidence": 0.5461,
"legend": {
"0": "no persistent effect",
"1": "reversible with effort",
"2": "irreversible or externally visible"
},
"probabilities": {
"0": 0.0465,
"1": 0.256,
"2": 0.6974
}
}
},
"usage": {
"input_tokens": 462,
"output_tokens": 0
}
}The model gets the tool, the policy’s risk level and the arguments. Running the model locally, close to the broker, keeps these arguments inside the organisation. A noul question is a yes/no question, and the model returns the probability that the answer is yes. A score question places the content on an ordered scale that the caller defines. In this case, we want a good model to return a probability close to 1 for the first two questions and a score in the upper half of the scale, because editor rights for an outside party are externally visible. The answers would be an additional input for the OPA policy and enable a rule that auto-approves any call where both probabilities are close to 0 and the score reads “no persistent effect”.
The beauty of this approach is that the model only answers questions while the approval itself is still a deterministic rule in a policy that can be reviewed and reasoned about. “Auto-approve when nothing leaves the organisation and the call has no persistent effect” is a sentence a security team can agree to, test and find again in an audit log next to the answers that triggered it.
Limiting what a prompt injection can achieve is another good reason for this approach. The arguments of a tool call are externally controlled content, so the model reading them can be manipulated. A narrow output doesn’t prevent that, but it limits the damage: injected text can skew two numbers and a score, but it can’t smuggle instructions downstream or argue for its own approval, because the policy makes the decision.
Fixed answers also help to shape the UX of the approval screen. OWASP’s guidance on Human-Agent Trust Exploitation (ASI09) asks for a “plain-language risk summary (not model-generated rationales)”. “Gives editor access to thirdparty.example.com” is such a summary. A paragraph in which a model explains why the call is probably fine is the persuasive text that the OWASP entry warns about, and a System One model can’t write one.
This approach is also not new. Auto modes in modern harnesses commonly have the same goal. Instead of prompting the user, a classifier checks each tool call in the background and blocks the risky ones. Anthropic reports that the classifier blocked 89% of the dangerous commands in the study above, so a model that reads every call clearly beats a fatigued human. The difference is who decides. In auto mode the classifier allows or blocks. The broker sits between many agents and many MCP servers in an organisation, and its operators or auditors have to explain why a call went through without a person. Fixed questions and a verifiable policy give them that explanation.
A Failing Model Never Approves Anything #
The model will be unreachable at times, it will return answers that don’t parse, and its answers will be uncertain or contradict each other. In all of these cases the request stays what it was before the model existed: a pending approval for the human in the loop. How the broker will detect an uncertain answer isn’t something I have specified yet, but a probability near the middle is one candidate. What is abundantly clear though is that a model’s answers must never turn a policy denial into an approval. The same holds for a permanent denial that a person has set for a tool, which the broker’s approval API already puts above any matching approval.
So we want the model to only take approvals off a person for calls that this person would have been allowed to approve anyway. The worst outcome of an outage is today’s approval page and an interrupted agent session.
What Is Still Open Before I Build It #
Keeping the approval pending covers a model that is down or unsure, but it doesn’t cover a model that is confidently wrong. When that happens, a call goes through that a person might have stopped. So everything depends on the questions you ask, the thresholds in the policy, and a good eval harness that shows how often the model gets recorded tool calls wrong. My three questions are a first guess. Without such evals I can’t tell which questions hold up across actual MCP tools and real tool approval scenarios, or how close to 0 a probability has to be before a rule may skip the person.
If you operate tool approvals for agents and know a call that my three questions would let pass although a person should have seen it, please add it to issue #206. I’d also like to hear which questions you’d want answered before a call may skip you.