03 · Paths from error or adversarial influence to harm
I start with the loss and work backward through the actions, omissions, delays, and human decisions that could produce it. Then I ask what the delegation creates, changes, or moves elsewhere. Changes to information, authority, timing, or scale can alter a path. Moving work to people can change both their workload and the cases they handle. The analysis includes permissions, write and read access, accepted jobs, and combinations of paths. A social-engineering route wholly unchanged by AI delegation falls outside this claim, but that exclusion needs a justification others can challenge. A worker acting on an AI summary or absorbing displaced cases belongs inside it.
I follow the dependencies that can carry an error into a consequence: service interfaces, stored records, accepted jobs, and human handoffs. Many model responses can share one path analysis when the same constraints and assumptions cover their effects. That grouping judgment must be stated and open to challenge. Differences in authority, timing, or downstream use may require separate treatment.
If a path has been assessed separately, its evidence, assumptions, and applicability still belong in the argument. Excluding it narrows the promise and needs a stated justification others can challenge. An unsupported path within scope calls for more evidence or a narrower claim.
I also require a record of how paths were sought and what remains uncertain. A newly identified path can reopen the argument. How long to keep searching depends on the severity and reversibility of the loss, exposure, and remaining uncertainty. The account must explain why further search is unlikely to change the current decision. That judgment does not prove that every path has been found. The STPA handbook likewise describes refining an initially incomplete control structure during development. STPA Handbook, p. 31
Tracing a path forward then tests how the loss could occur. In the support example, an attacker-controlled ticket can influence the assistant's choice of destination. The proposed change then reaches the execution service, where an authorization rule may reject it or permit it to continue. A worker can receive the same recommendation through another route. The interesting crossing is where content starts to determine the use of consequential authority.
From retrieved content to action#
AgentDojo provides a controlled example of this mechanism. It places tool-using agents in simulated, mutable environments and lets an attacker alter a designated untrusted tool-output location. Its experiments show prompt injections producing specified attacker outcomes, checked against environment state. The setting uses dummy data and defined tasks and targets. It establishes reachable effects under the tested access assumptions, while leaving production incidence and robustness against future adaptive attacks open. AgentDojo, §3, pp. 3–4
An assertion can enter the workflow as information and later determine an action. A ticket can legitimately contain a requested destination; the system still needs to establish whether the requester has authorized a change for this account. Useful task information and malicious direction may arrive through the same material.
An incorrect result can follow without an attacker. The assistant might confuse an old and a new address, retrieve another account's note, or misunderstand the requested operation. Those variants have different starting causes, but they can reach the same proposed change. A control's coverage depends on the property it enforces and the evidence it uses across those causes.
From private information to a sent reply#
Suppose a ticket induces the assistant to retrieve a private note and include its contents in a reply to someone who is not entitled to receive them. The assistant or a worker then sends it. Retrieval and release are distinct points at which a restriction could act. Preventing an account change would leave this disclosure path open. The strength of a release claim depends on the protected information and output interface: a restricted interface may support an enforceable prohibition, while unrestricted prose may retain a dependency on judgment about what the text reveals.
From an earlier write to a later decision#
Now place the misleading address in a stored note. A later request retrieves it, and the assistant treats it as established context. The path spans two events: the write that introduced the assertion and the decision that reused it. A per-request account beginning with retrieval would omit the earlier dependency.
AgentPoison examines a bounded version of this state problem. The attacker can insert poisoned examples into retrieval memory or a knowledge base and optimize a trigger using access to a retrieval embedder. The demonstrated attack combines poisoned retrieval entries with a later query carrying the optimized trigger; its objective is to preserve behavior on clean queries. AgentPoison, §§3.2–3.3, pp. 4–5, equations (1)–(2) The authors motivate that write access with third-party retrieval services and editable public knowledge bases, but the experiment assumes it rather than demonstrating how an attacker obtains it; nor does it show the agent creating the poison or the poison spreading across tenants. AgentPoison, §3.2, p. 4, attacker assumptions The account-service example of a legitimate request reusing a misleading note is a further hypothetical, whose occurrence this study does not establish.
For the support service, I would ask how the note entered storage and what later decisions can reasonably take from it. That depends on write permissions, supporting evidence, and any transformations. A later assistant could use the note as background while still requiring current authorization. Trusted storage can faithfully preserve an unsupported statement.
Ordinary error and accumulated effects#
The Knight Capital incident provides a conventional-software case. The SEC's account of the August 2012 event describes obsolete code that remained callable, a deployment that missed one server, and a repurposed flag that activated the old code there. Completed-order information failed to reach the router, which continued sending orders during diagnosis. SEC Order 34-70694, ¶¶13–16, 27 Unwanted positions remained after order generation stopped. SEC Order, ¶¶1, 17 These are regulatory findings in a settlement without admission or denial. SEC Order, §II The record identifies an observed interaction and accumulated effects; it is neither an AI incident nor a matched test of a proposed control.
The question I carry into the account example is how an initial error keeps producing consequences through available authority. An incorrect proposal, an accepted change, and later actions enabled by that change are different stages to examine. The analogy directs attention to those stages without assigning them Knight's likelihood or scale.
Keep the initiating conditions explicit#
Ordinary error, misuse, misalignment, and compromise describe different features of a path. An authorized user can deliberately misuse a permitted function. A system can depart from its intended objective without evidence of deliberate intent. A compromised dependency can alter what an otherwise functioning component receives. The descriptions can overlap; the useful analysis specifies the initial access or fault, the decisions it affects, and the resulting action or omission.
Some threats require a wider boundary. Theft of model weights, query-based model extraction, and disclosure of customer information involve different assets or access paths. NIST covers model extraction and disclosure of private information through generative-AI applications, while leaving conventional attacks on hosting platforms to other cybersecurity taxonomies. NIST AI 100-2e2025, §1, §2.4.4, §3.1.2, §3.4.3; PDF pp. 15, 44, 53, 65–66 A compromised tool or supplier can invalidate assumptions about enforcement or evidence. AI used within an attacker's workflow raises further questions about operational success and threat prevalence. This paper treats these as boundaries to examine where they affect a relied-upon dependency; it makes no empirical claim here about their frequency or net change.
For each path, I then ask where we can interrupt it and what must remain trustworthy for that interruption to work.