Multiple agents produce system-level behavior that no individual agent was designed to produce. Emergent failure mode is unpredictable from analysis of individual agents in isolation.
Emergent coordination failure is a second-order failure mode: the system exhibits behavior that cannot be predicted by analyzing the components individually. Three agents operating correctly in isolation may, when coordinated, produce outcomes that none of the agents individually would produce and that the institution did not design.
Formally, this occurs when: (1) Agent A's outputs are input to Agent B; (2) Agent B's outputs are input to Agent C; (3) The interaction of A to B to C produces output that does not match any agent's intended behavior; and (4) No single agent is "wrong" because each is operating correctly within its design specification. The failure emerges from the interaction pattern, not from component malfunction.
In regulated industries, this creates accountability gaps. When regulators investigate a failure, they ask "who made the decision?" The answer is often that no single agent made the decision. The failure is emergent from how agents interpreted and composed each other's outputs, not from individual agent errors.
A large mortgage bank deploys four agents for commercial real estate loan underwriting: Property-Assessor evaluates property condition and market value, Borrower-Analyst evaluates borrower credit history, Market-Monitor evaluates market conditions in the property's geography, and Underwriter-Agent makes final approval/denial decision.
Property-Assessor conducts a valuation of a commercial office building in Austin, Texas. The property is 85% leased to stable tenants. The agent uses comparable sales from the past 18 months and produces a valuation of $45M with confidence flag "moderate" due to limited recent comps. Borrower-Analyst reviews the borrower and produces a credit score of A (low risk). Market-Monitor produces a market stability score of 0.7 / 1.0 (moderate stability) due to recent tech job growth offsetting some uncertainty, noting that office vacancy is rising (9% in Q1 to 12% in Q2).
Underwriter-Agent receives all three inputs and synthesizes: $45M valuation (moderate confidence) + A credit (low risk) + 0.7 market stability = approval for $30M loan at 5.2% fixed rate. Critically, what none of the agents recognized is the interaction pattern. Market-Monitor's "rising vacancy" trend, extrapolated forward, suggests a 1.5% quarterly rise. By year 3, office vacancy could reach 17-18%. Additionally, Property-Assessor's valuation was based on limited comps that became outdated within weeks.
Two years into the loan, the property is 65% leased (down from 85%), market vacancy is 16%, property value has declined to $38M, and the borrower is approaching negative equity on a $30M loan. Each agent produced correct outputs individually, but the agents did not collectively recognize that the combination of "rising vacancy" + "limited valuation confidence" + "3-year loan horizon" equals high risk of material valuation decline. The OCC issues an MRA requiring governance improvement in "emergent risk assessment in multi-agent underwriting systems."
| Dimension | Score | Rationale |
|---|---|---|
| D - Detectability | 4 | Emergent failures are inherently difficult to detect because they do not correspond to individual agent failure signatures. Require system-level testing and behavioral analysis. |
| A - Autonomy Sensitivity | 4 | Emerges only when agents have autonomy to compose their outputs without human intervention. Human-in-the-loop arrests emergent failures. |
| M - Multiplicative Potential | 4 | Every multi-agent workflow has potential for emergent failure. Probability scales with system complexity and agent interdependencies. |
| A - Attack Surface | 2 | Not directly exploitable as attack vector, though adversary could deliberately craft scenarios that trigger known emergent failure modes. |
| G - Governance Gap | 5 | Regulations assess individual decision quality, not system-level coordination quality. Institutions often lack governance processes for emergent risk. |
| E - Enterprise Impact | 4 | Affects asset quality (loan defaults), regulatory compliance, and institutional reputation. Material financial impact in large institutions. |
| Composite DAMAGE Score | 4.1 | Critical. Requires immediate architectural controls. Cannot be accepted. |
How severity changes across the agent architecture spectrum.
| Agent Type | Impact | How This Risk Manifests |
|---|---|---|
| Digital Assistant | Low | Human reviews and integrates all agent outputs. Human synthesizes across agents and identifies emergent patterns. |
| Digital Apprentice | Low | Developmental agents operate under tight governance with human oversight at decision boundaries. |
| Autonomous Agent | High | Autonomous agents compose outputs and create system-level behavior without human mediation. Emergent failures likely. |
| Delegating Agent | High | Single delegating agent invoking multiple tools may see emergent interactions between tools. |
| Agent Crew / Pipeline | Critical | Sequential multi-agent pipelines are designed for composition but emergent failures across the pipeline are common. |
| Agent Mesh / Swarm | Critical | Unstructured peer-to-peer agent networks have maximum emergent complexity. Failure modes are unpredictable. |
| Framework | Coverage | Citation | What It Addresses | What It Misses |
|---|---|---|---|---|
| NIST AI RMF 1.0 | Minimal | MAP 5.1 | System-level risk mapping. | Emergent failure modes in multi-agent systems. |
| EU AI Act | Minimal | Articles 9, 15 | Risk assessment processes. | Guidance on emergent risk assessment methods. |
| MAS AIRG | Partial | Governance, Model Risk, Control Environment | Governance framework and controls. | Specific requirements for emergent risk in agent systems. |
| OCC / SR 11-7 | Partial | Model Risk Management | Model governance and validation. | Validation approaches for multi-agent system interactions. |
| Dodd-Frank Section 165 | Partial | Concentration risk, liquidity risk | Systemic risk assessment. | Emergent risk from multi-agent coordination. |
| OWASP Agentic Top 10 | Not Directly | Security-focused. | Emergent system behavior and coordination failures. |
Financial institutions are required to have robust underwriting and risk assessment processes. These processes must be capable of identifying risks across multiple dimensions and aggregating risk signals. When an institution deploys agents that individually are competent but collectively produce emergent failures, the institution is operating outside its risk governance framework.
Additionally, emergent failures create liability for institutional negligence. If a regulator can show that the institution deployed agents without testing for emergent interactions, the institution may face enforcement action for inadequate controls.
Emergent coordination failures are also particularly dangerous because they scale. In a single-agent or human-human system, the number of potential interaction patterns is limited. In a multi-agent ecosystem where agents discover and coordinate with each other dynamically (mesh architecture), the number of potential emergent failure modes is combinatorial.
Emergent Coordination Failure requires architectural controls that go beyond what existing frameworks provide. Our advisory engagements are purpose-built for banks, insurers, and financial institutions subject to prudential oversight.
Schedule a Briefing