Agent outputs enter data stores; future agents and models cannot distinguish agent-generated from system-of-record data. Feedback loop amplifies errors exponentially over time.
Data governance frameworks maintain a hierarchy: system-of-record data (trusted sources, validated, auditable) sits at the top; derived data (transformed, computed, or inferred) sits below it. Workflows are designed to flow upward from system-of-record to derived, never in reverse. This assumption is broken when agent-generated outputs are stored in systems where they can be retrieved and used as inputs to future agent reasoning, fine-tuning, or training. The agent output becomes contaminated input for the next agent. The next agent cannot distinguish whether its input came from a human, a system of record, or a prior agent.
This is particularly dangerous in forecasting, risk modeling, and compliance workflows. An agent generates a risk score. The risk score is stored alongside historical scores. A new agent (or a future fine-tuned model) retrieves the risk score and uses it for trend analysis or model retraining, unaware that it came from an earlier agent. The model learns from agent-generated synthetic data, which affects the model's next outputs, which are stored again. The contamination loop is self-amplifying. Errors that begin small grow larger with each iteration. Models become increasingly confident in outputs that are increasingly wrong because they are increasingly distant from system-of-record truth.
The worst-case scenario is a data governance framework that appears intact but is fundamentally corrupted. Institutions maintain policies prohibiting synthetic data from being used as system-of-record. But those policies assume human enforcement at decision points. When all decisions are delegated to agents, the enforcement becomes invisible. Agent A stores an output. Agent B retrieves it. The governance framework has no hook to intercept the retrieval and verify the data provenance.
A global investment bank uses agents to generate credit risk scores for 50,000 medium-market companies. The agents are trained on historical credit bureau data, financial statements, and prior default events. Each agent generates a risk score monthly, and the score is stored in the institution's risk data warehouse alongside historical credit bureau scores, treating both as comparable inputs for portfolio models. The agents are reasonably accurate (86% first-time correct). Over 18 months, agents have generated scores for 15,000 companies that had no recent credit bureau data, filling gaps in the bank's knowledge. These synthetic agent scores are stored in the warehouse.
The bank then fine-tunes a new model on its risk warehouse, intending to improve accuracy for harder-to-rate companies. The fine-tuning dataset includes 200,000 risk-score examples: 150,000 from credit bureaus, 50,000 from agents. The model is retrained. The model now learns from agent-generated data it does not know is synthetic. Its outputs shift to become more similar to agent outputs (since agent outputs are overrepresented in the fine-tuning set). The fine-tuned model is deployed to new agents. The new agents produce scores that are more similar to the prior agents' outputs, amplifying the signal from the contaminated training data.
Three years into this cycle, an auditor asks the bank to demonstrate the quality of its risk scores. The bank analyzes the warehouse and discovers that recent scores have become less accurate than earlier scores, and the degradation correlates with the introduction of agent-generated training data. The bank cannot reconstruct which scores were synthetic because the warehouse does not separate them. The bank's risk models are contaminated. Risk portfolio analysis is unreliable.
| Dimension | Score | Rationale |
|---|---|---|
| D - Detectability | 4 | Contamination loop is often invisible until model quality degrades significantly. May not be detected for months or years after starting. |
| A - Autonomy Sensitivity | 5 | Highly autonomous agents generate and store data without human verification. Human oversight is too infrequent to catch contamination during accumulation. |
| M - Multiplicative Potential | 5 | Error amplifies through each agent iteration and fine-tuning cycle. Damage compounds exponentially over time. |
| A - Attack Surface | 3 | Primarily a structural risk. Adversary could intentionally inject agent outputs to contaminate models, but contamination occurs naturally without attack. |
| G - Governance Gap | 5 | Data governance frameworks assume provenance is tracked and synthetic data is segregated. Agents make tracking invisible and segregation unenforceable. |
| E - Enterprise Impact | 5 | Risk models become unreliable, decision-making degradation across entire enterprise, potential for massive losses if contaminated risk model drives portfolio decisions. |
| Composite DAMAGE Score | 3.7 | High. Requires priority attention with dedicated controls and monitoring. |
How severity changes across the agent architecture spectrum.
| Agent Type | Impact | How This Risk Manifests |
|---|---|---|
| Digital Assistant | Moderate | Human may review output before storage, but human cannot know if future uses of stored data will contaminate models. |
| Digital Apprentice | High | Progressive autonomy reduces review frequency. Contaminated outputs accumulate faster as agent makes more independent decisions. |
| Autonomous Agent | Critical | Agent generates and stores data autonomously. No human verification before storage. Contamination loop begins immediately. |
| Delegating Agent | High | Agent determines which data sources to invoke. May dynamically incorporate agent-generated data from prior runs without explicit awareness. |
| Agent Crew / Pipeline | Critical | Multiple agents feed outputs to shared storage. Cross-agent contamination occurs through stored data. Each downstream agent amplifies prior contamination. |
| Agent Mesh / Swarm | Critical | All agents share access to same data store. Contamination flows bidirectionally through agent mesh. Scale of contamination grows with mesh size. |
| Framework | Coverage | Citation | What It Addresses | What It Misses |
|---|---|---|---|---|
| BCBS 239 | Partial | Principle 3 (System-of-Record) | Requires maintaining system-of-record data and distinguishing from derived data. | Does not address synthetic data provenance tracking in agent environments. |
| EU AI Act | Partial | Article 24 (Documentation) | Requires documentation of AI system development and training data provenance. | Does not address dynamic training data contamination during operational use. |
| NIST AI RMF 1.0 | Partial | GOVERN 2.1, MAP 2.2 | Recommends data governance and quality assessment. | Does not address feedback loops or synthetic data commingling with training datasets. |
| MAS AIRG | Partial | Section 6.1 (Data Governance) | Requires data governance addressing data quality, provenance, and integrity. | Does not anticipate agent-generated synthetic data feedback loops. |
| ISO 42001 | Partial | Section 6.1.2 | Addresses information asset management and labeling. | Does not explicitly address synthetic data contamination in feedback loops. |
| Gramm-Leach-Bliley Act | Partial | 15 U.S.C. 6809 | Requires appropriate security and integrity of customer information systems. | Does not address provenance of generated data. |
| GDPR | Minimal | Article 33 (Breach Notification) | Addresses data breach notification. | Does not address data contamination through synthetic generation. |
| OWASP Agentic Top 10 | Partial | Tool-Use Issues | Addresses tool invocation risks. | Does not address storage and reuse of agent-generated data in training loops. |
Risk models drive capital allocation, reserve calculations, and regulatory capital requirements in banking. Insurance loss models drive premium setting and reserving. If these models are trained on contaminated data (agent outputs presented as system-of-record), the models produce unreliable outputs that cascade into risk management failures. Regulators expect institutions to maintain data provenance and to detect when synthetic or derived data has contaminated training sets. An institution that discovers its models were trained on agent-generated data that was not properly labeled and segregated faces significant regulatory consequences: restatement of reserves, recalculation of capital requirements, enforcement action for inadequate data governance.
In capital markets, price discovery models that rely on contaminated historical data produce unreliable signals. In insurance, underwriting models trained on synthetic claims data may misprice risk. In compliance, sanctions screening that relies on contaminated historical entity data may miss actual matches. The contamination loop is particularly dangerous because it is self-reinforcing; institutions do not notice until model performance degrades significantly. By that time, damage has accumulated over many iterations.
Training Data Contamination Loop requires architectural controls that go beyond what existing frameworks provide. Our advisory engagements are purpose-built for banks, insurers, and financial institutions subject to prudential oversight.
Schedule a Briefing