R-FM-01 Foundation Model & LLM DAMAGE 4.0 / Critical

Silent Model Update by Provider

Provider updates model without notice. Agent behavior changes without any institutional change to the agent, its prompts, or its tools.

The Risk

Institutions rely on model provider service-level agreements (SLAs) that typically promise availability and performance. Most SLAs do not guarantee model stability. A model provider can update their model at any time without notice. When a model is updated, the model's behavior changes. An agent built on that model will produce different outputs without any change to the agent's code, prompts, or data. The institution's system behavior changes without the institution making any change.

This is fundamentally different from traditional software. In software engineering, a service version is specified (e.g., API v2.1). If the service provider updates to v3.0, the client must opt-in. The client can choose when to upgrade and can test the upgrade before deploying. With model providers, there is no versioning in this sense. The model is updated silently, the agent's behavior changes automatically, and the institution has no advance notice or control.

The risk is amplified by non-determinism: if the same input produces the same output (deterministic systems), a silent model update might be invisible (no change in output). Large language models are non-deterministic. The same input may produce different outputs on different model versions. A model update may change outputs subtly (different phrasing, different reasoning paths, different emphases) or dramatically (different decisions in edge cases).

How It Materializes

A bank uses a model API to generate customer-facing explanations for credit decisions. The model is GPT-4 (or equivalent). The bank's prompts are tuned for GPT-4's reasoning style. The outputs are professional, concise, and appropriately cautious about uncertainty. The model is deployed in production. Customers receive explanations like: "Your application was declined due to insufficient credit history. Please contact us to discuss strengthening your credit profile."

The model provider silently updates the model (performance improvement, safety update, cost reduction). The new model version produces slightly different outputs. For some customers, explanations become more blunt: "Your application was declined. Your credit history is too short." The explanation lacks the suggestion to contact the bank. Some customers receiving the blunt explanation feel dismissed and complain. Some customers interpret the explanation as accusatory rather than explanatory.

The bank has made no change to its system. The agent code is unchanged; the prompts are unchanged; the data is unchanged. The output quality has degraded due to a model update the bank was unaware of. The bank cannot reverse the change because it does not control the model. The bank must update its prompts to compensate, but it does not know what specifically changed about the model.

Alternatively, a model update may change the model's reasoning in ways that affect regulatory compliance. A previous version was appropriately conservative in edge cases (declining borderline applicants to avoid risk). A new version is less conservative. The model now approves more borderline applicants. The bank's approval rate rises above historical precedent. Regulators notice the change and investigate. The bank must explain why approval rates increased. The bank discovers it was due to a silent model update. The regulator is concerned that the bank does not control or monitor its AI system's outputs.

DAMAGE Score Breakdown

Dimension Score Rationale
D - Detectability 3 Model updates are often invisible unless outputs are explicitly monitored for changes. Discovery occurs through degraded outputs or statistical analysis.
A - Autonomy Sensitivity 2 Occurs regardless of autonomy level. Model update affects all agents using the model.
M - Multiplicative Potential 5 Every agent using the updated model experiences behavior change simultaneously. Affects all agents across entire institution.
A - Attack Surface 1 Not weaponizable by external actors; provider controls model updates.
G - Governance Gap 5 Institutions assume model behavior is stable. Silent updates break this assumption. Governance has no mechanism to detect or control updates.
E - Enterprise Impact 3 Degraded outputs, compliance changes, customer impact, but institution can adapt by modifying prompts or switching providers.
Composite DAMAGE Score 4.0 Critical. Requires immediate architectural controls. Cannot be accepted.

Agent Impact Profile

How severity changes across the agent architecture spectrum.

Agent Type Impact How This Risk Manifests
Digital Assistant Moderate Assistant output quality changes silently. Human user may notice quality degradation.
Digital Apprentice Moderate Agent behavior changes without institutional awareness. Autonomy may be affected by model update.
Autonomous Agent High Fully autonomous agent's behavior changes without notice. May cause compliance drift or output quality degradation.
Delegating Agent High Agent's delegated reasoning via model API changes silently. Downstream tools may receive different recommendations.
Agent Crew / Pipeline Critical Multiple agents all affected by model update simultaneously. Entire pipeline behavior changes.
Agent Mesh / Swarm Critical All agents in mesh affected by model update. Swarm behavior changes systematically.

Regulatory Framework Mapping

Framework Coverage Citation What It Addresses What It Misses
EU AI Act Partial Article 4(1), Article 24 Defines high-risk AI systems; requires documentation of system properties and performance. Does not address third-party model provider behavior changes.
NIST AI RMF 1.0 Partial GOVERN 1.1, MAP 1.1 Recommends governance and transparency. Does not address provider model update management.
MAS AIRG Partial Section 6.1 (Governance) Requires AI governance and monitoring. Does not address third-party provider model updates.
NIST CSF 2.0 Partial GOVERN (Organizational Processes) Addresses governance. Does not address third-party system behavior changes.
SOX 404 Partial IT Controls Requires control over financial systems. Does not address third-party model provider behavior.

Why This Matters in Regulated Industries

In regulated industries, compliance depends on system behavior being stable and predictable. If a credit decision system suddenly approves more applicants (due to silent model update), the compliance posture changes. Regulators expect institutions to control the systems they use for consequential decisions. An institution that cannot explain why outputs changed (because it did not control or know about a model update) loses credibility with regulators.

For customer-facing systems, output quality is critical. If explanations degrade due to model updates, customers lose trust in the institution. For internal compliance systems (fraud detection, AML), if model behavior changes silently, the institution may miss suspicious activities due to changed thresholds or detection logic in the updated model.

Controls & Mitigations

Design-Time Controls

  • Choose model providers and versions that provide versioning and advance notice of updates. Prefer models with explicit version pinning (e.g., GPT-4-turbo-2024-04-09) over generic names (GPT-4) that may be updated silently.
  • Document model version and architecture at agent design time. Record in Component 1 (Agent Registry) which specific model version each agent uses.
  • Require model provider contracts to include: explicit notice of model updates (minimum 30 days advance notice), SLA on model behavior stability (e.g., output distribution remains within X% of baseline), and option to pin model versions.
  • Implement a "canary deployment" process for model updates: test updated models on non-production agents first; verify output quality and compliance implications before deploying updated model to production agents.

Runtime Controls

  • Monitor model outputs continuously: track output distributions, quality metrics, and key decision characteristics. Detect statistically significant changes that may indicate a model update.
  • Implement output signature monitoring: compute cryptographic signatures of model outputs on test inputs. Compare signatures over time. Changes in signatures indicate model behavior has changed.
  • Maintain baseline output samples: at agent deployment time, collect baseline outputs for standard test inputs. Periodically compare current outputs to baselines. Flag deviations.
  • Use Component 10 (Kill Switch) to halt agents whose outputs diverge significantly from historical baselines without known cause.

Detection & Response

  • Establish model monitoring alerts: configure alerting if output quality metrics degrade, approval rates change unexpectedly, or output distributions shift.
  • Monitor provider notifications: subscribe to model provider's changelog, release notes, and announcements. Maintain visibility into model updates and their properties.
  • Conduct quarterly model behavior audits: evaluate model outputs against baselines, document any observed changes, assess whether changes are due to model updates.
  • Establish incident response for detected model updates: assess impact on compliance and output quality, determine whether rollback or remediation is needed, update agent designs or prompts to accommodate new model behavior.

Related Risks

R-FM-05 Foundation Model & LLM
Prompt Sensitivity and Brittleness

Small changes in prompt wording cause disproportionately large changes in model output. The input space is so large that exhaustive testing is impossible.
R-FM-06 Foundation Model & LLM
Non-Determinism and Output Variance

Same customer query processed by the same agent can produce different recommendations. Non-determinism undermines fair treatment obligations and audit reproduction.
R-FM-02 Foundation Model & LLM
Model Provider Dependency and Concentration Risk

Most agentic deployments depend on a single model provider. If the provider experiences an outage or discontinues the model, all agents fail simultaneously.

Address This Risk in Your Institution

Silent Model Update by Provider requires architectural controls that go beyond what existing frameworks provide. Our advisory engagements are purpose-built for banks, insurers, and financial institutions subject to prudential oversight.

Schedule a Briefing
Agentic AI Risk & Controls Workshop Our Methodology Regulatory Landscape