AI Safety & Security
Delegated Authority and Warranted Reliance
Introduction — Consequential delegation#
Consider a subscription service that uses an AI assistant to help customers recover access to their accounts. The assistant reads tickets, retrieves prior case notes, drafts replies, and proposes changes to recovery addresses. A separate service can apply those changes, and a support worker has an administrative console. This is a hypothetical workflow; all its details and variants are invented for analysis.
Suppose an attacker puts an address into a ticket and persuades the assistant to recommend it as the customer's new recovery destination. The assistant has proposed a change the customer did not authorize. Whether that proposal becomes an unauthorized account change depends on what happens next: the evidence required by the change service, the permissions used, any accepted job, and the worker's separate decision. A misleading note can also preserve the assertion for a later session.
The service makes two commitments: protect accounts against changes without the customer's authorization, and provide a staffed recovery route when the customer has lost the original contact channel. Both matter. A system that closes every difficult case may avoid some unauthorized changes while failing the people it was meant to help.
The question I keep coming back to is how a mistaken or manipulated decision reaches someone, and what supports our promise to prevent or limit the harm. My central claim is:
A claim that a delegated AI workflow controls a specified loss must be supported across the paths to that loss that the delegation creates, alters, or displaces, including paths through execution, stored state, delegated work, and human decisions. Any exclusion must be named, justified, and open to challenge.
I offer this as a standard for warranted reliance: reliance justified by evidence about the work entrusted to a system and the conditions and limits under which it can meet that commitment. To earn that reliance, we need to follow how the harm could occur, identify what would interrupt it, and establish what that protection depends on. The evidence then determines how much of the promise we can support.
Whether enforceable limits and timely intervention reduce harm while preserving useful operation is a further, empirical question. Particular controls can support that hypothesis under particular conditions. Having the controls in place is insufficient, and a simpler design may meet the same requirements more effectively. I would count the work people need completed and the full cost of controlling it in that comparison.
This paper focuses on consequential digital workflows. Its unit of analysis includes models, tools, data, retained state, identities, services, and the people whose decisions produce or bear the effects. It draws on systems safety, security engineering, bounded AI experiments, two human-use studies, and a conventional software incident. These sources support different inferences without collectively establishing an overall safety rate for deployed AI.
Much of this is familiar security and systems-safety engineering, and I think that is a useful starting point. Here, models interpret untrusted material, choose actions and arguments at runtime, reuse context, and give people explanations they may act on. Those mechanisms give the paper its AI focus; none needs to be exclusive to AI. Where conventional analysis explains the paths and supports the decision, I would use it. My contribution is to connect delegation, controls, and evidence so a reader can examine the reasoning.
AI Vision & Future examines the conditional path from capability to dependable value and wider tractable work. AI Operators Handbook develops practical operating decisions and records. Here I examine the mechanisms and evidence behind a claim of control: how delegation creates exposure, how prevention and intervention can constrain it, and what warrants continued reliance when conditions change.
Section 6 proposes a comparison that could support or weaken the harm-reduction hypothesis, including a simpler design that could satisfy the commitment more effectively. This paper specifies that test; it does not report having performed it.