Regulation requires upfront risk assessment but agentic systems evolve at runtime. Static assessment at deployment cannot capture runtime behavior changes.
Regulatory frameworks, particularly in the EU (AI Act) and Singapore (MAS AIRG), require that organizations conduct risk assessment before deploying AI systems. This risk assessment is upfront and static: performed before deployment, documented in a risk assessment report, and used to determine what controls are required.
For traditional AI systems (models), this static assessment is reasonable. A model's behavior is relatively stable over time. Performance may decline if data distribution shifts, but the model does not fundamentally change its decision-making approach.
Agentic systems are different. Agents learn at runtime. Agents adapt to environmental changes. Agents coordinate with other agents in ways that were not anticipated at design time. An agent's behavior at deployment may be radically different from its behavior months later as it learns and adapts.
A static risk assessment performed at deployment cannot account for runtime behavior changes. The assessment assumes the agent will behave as it did during testing. But the agent may behave very differently in production. By the time the organization discovers that the agent's behavior has changed, the static risk assessment is obsolete.
An insurance company deploying an agentic claims processing system conducts risk assessment as required by its regulatory framework. The assessment concludes: high risk level (claims decisions affect policyholders' financial interests), key risks including claim denials that violate policy and discriminatory handling, and controls including human review of denials above $50,000, quarterly discrimination audits, and continuous fraud detection monitoring.
The agent operates for six months. During this time, the agent learns from claims outcomes and adjusts its decision boundaries. Claims volume doubles, and management increases the human review threshold to $100,000 to reduce burden. The agent, observing that more denials above the old threshold are being approved by reviewers, adjusts its denial criteria. The company also begins receiving claims from a new geographic region where the agent's accuracy is 10% lower.
After six months, the original risk assessment is obsolete. The agent's learning has altered its decision boundaries. The controls have been modified (higher threshold) but the risk assessment was not updated. The assessment still assumes human review at $50,000. The organization is operating under controls that do not match the documented risk assessment.
A regulator examining the company's AI governance notes that the controls have changed but the risk assessment has not been updated. The regulator views this as non-compliance: the organization is required to maintain a risk assessment that reflects the actual risk profile, and the assessment has become stale.
| Dimension | Score | Rationale |
|---|---|---|
| D - Detectability | 4 | Static assessment failure is visible when an agent's behavior changes and the static assessment no longer reflects the agent's actual risk profile. Discovery typically occurs during regulatory examination. |
| A - Autonomy Sensitivity | 5 | Static assessment failure is most severe for autonomous agents that learn and adapt at runtime. For supervised agents or agents without learning, risk profiles are more stable. |
| M - Multiplicative Potential | 4 | Static assessment failure compounds over time. The longer an agent operates without risk reassessment, the more its behavior may diverge from the assessment. |
| A - Attack Surface | 3 | Static assessment failure can be exploited by adversaries who understand that the organization is not monitoring for runtime behavior changes. |
| G - Governance Gap | 5 | Most regulatory frameworks require static risk assessment. Few frameworks require continuous risk reassessment. Organizations are complying with regulations that require static assessment, but the static assessment is insufficient for agents. |
| E - Enterprise Impact | 4 | Static assessment failure can lead to agents operating outside their risk profile without detection. Impact becomes apparent when an agent's divergent behavior causes harm. |
| Composite DAMAGE Score | 3.7 | High. Requires dedicated controls and regular monitoring. |
How severity changes across the agent architecture spectrum.
| Agent Type | Impact | How This Risk Manifests |
|---|---|---|
| Digital Assistant | Low | DA does not learn or adapt autonomously. Risk profile remains relatively stable. Static assessment is adequate. |
| Digital Apprentice | Low | AP learns under supervision. Supervisors can recognize learning and can update risk assessment. |
| Autonomous Agent | High | AA learns and adapts autonomously. Risk profile changes at runtime. Static assessment is inadequate. |
| Delegating Agent | Medium | DL invokes tools. If tools' behavior changes or if the agent's tool selection changes, risk profile changes. Risk assessment must be updated. |
| Agent Crew / Pipeline | High | CR chains agents. As agents in the pipeline learn, their behaviors change. Emergent behaviors of the pipeline may not match the original risk assessment. |
| Agent Mesh / Swarm | Critical | MS features dynamic emergent behavior. Risk profile at runtime may be radically different from the risk profile at deployment. Static assessment is fundamentally inadequate. |
| Framework | Coverage | Citation | What It Addresses | What It Misses |
|---|---|---|---|---|
| EU AI Act | High | Articles 6, 8, 26-30 | Requires risk assessment before deployment and ongoing monitoring. | Does not specifically mandate dynamic risk reassessment as agent behavior changes. |
| MAS AIRG | High | Section 3 (Risk Governance), Section 5 (Monitoring) | Calls for ongoing monitoring but does not explicitly mandate dynamic risk reassessment. | Does not specify frequency or triggers for risk reassessment. |
| NIST AI RMF 1.0 | Partial | MEASURE | Recommends ongoing measurement of AI system performance. | No specific guidance on when or how to update risk assessment. |
| SR 11-7 | Partial | Model validation and monitoring | Requires ongoing monitoring of model performance and validation. | Does not mandate risk reassessment; assumes risk profile is relatively stable. |
| ISO 42001 | Partial | Section 6 (AI management system) | Requires governance and ongoing management. | No specific guidance on risk reassessment frequency or triggers. |
In banking and capital markets, regulators expect that risk assessments are current and accurate. If an agent's behavior has changed (due to learning, market changes, or other factors) but the risk assessment has not been updated, the organization is operating under an outdated risk picture. Regulators expect dynamic risk management.
In insurance, regulators expect that underwriting and claims risks are assessed and managed. If an agent's risk profile has changed but the assessment has not been updated, the organization is operating with inadequate risk management.
In healthcare, clinical risks must be assessed and monitored. If an agent's clinical decision-making has evolved but the risk assessment has not been updated, clinical governance is inadequate.
Static Assessment Failure requires continuous governance controls that go beyond one-time risk assessments. Our advisory engagements are purpose-built for banks, insurers, and financial institutions subject to prudential oversight.
Schedule a Briefing