AI Outage Recovery Playbook for Financial Services Teams

An AI outage is not only a model-quality problem. If several important business services rely on the same model, cloud, identity, or data provider, the first decision is operational: which service must stop, which can run manually, and what evidence is required before reopening it?

That question is becoming more urgent. The Financial Stability Board’s August 2026 letter to the G20 identifies how frontier AI can change the speed, scale, and economics of cyber risk. The IMF’s 2026 analysis also calls out shared digital infrastructure and concentration as channels through which cyber events can propagate. Neither source says every provider failure becomes systemic. They do make shared dependencies worth testing before an incident.

This guide gives financial-services operations, risk, security, and engineering teams a copyable recovery-drill prompt, a stop/manual/isolate/restore decision card, and an evidence pack. It is designed for a severe but plausible failure of a shared AI or adjacent provider—not for testing whether an agent follows an instruction. For the latter, use an AI red-team test harness for banking agents. For normal finance-analysis approvals, start with AI financial analysis controls.

Start with the service, not the model

Teams often inventory AI models but miss the surrounding path that makes a financial service work. A customer-support copilot may depend on a model API, identity provider, retrieval index, CRM, observability pipeline, cloud region, and a human escalation queue. A disruption at one of those layers can make an apparently unrelated service unsafe to operate.

For each important business service, create a dependency record with these fields:

Field	What to record	Recovery question
Service	Customer or internal outcome delivered	What is the smallest safe version of this service?
AI role	Drafting, ranking, retrieval, decision support, or action	Can the service run without this role?
Shared dependency	Model, cloud, identity, data, or integration provider	Does another critical service rely on the same provider?
Failure mode	Outage, suspected compromise, corrupted output, or lost access	What must be contained first?
Manual fallback	Named process, owner, and capacity limit	When does manual demand exceed safe capacity?
Reopen evidence	Logs, integrity checks, approvals, and test results	Who can accept residual risk?

This mapping turns a vendor list into an operational-resilience artifact. The Basel Committee’s Principles for operational resilience emphasize identifying important business services, their dependencies, and severe-but-plausible disruption scenarios. In the United States, the Federal Reserve likewise frames operational resilience around the ability to withstand disruptions such as technology failures and cybersecurity incidents in its operational-resilience guidance.

Use a six-step decision card during the incident

Do not let a general incident channel become the decision record. Give the incident commander one short card for every affected service.

Stop — Freeze AI-triggered actions, external sends, and irreversible workflow steps when integrity, authorization, or customer impact is uncertain.
Move to manual mode — Activate the documented fallback only within its staffing, approval, and queue limits. Record the time it began.
Isolate — Separate affected credentials, integrations, data feeds, and environments. Preserve logs before rebuilding or rotating them.
Restore from a known-good state — Rebuild or recover the minimum required components from approved images, configurations, and data snapshots. Do not treat a returned endpoint as proof of integrity.
Prove — Attach recovery evidence: change records, identity and secret rotation confirmation, integrity checks, test transactions, reconciliations, and owner sign-offs.
Reopen gradually — Start with a limited cohort, monitor defined failure signals, and require an explicit go/no-go decision before each expansion.

The card is deliberately more concrete than an AI-governance checklist. It makes the team choose an operational state and retain the proof behind it. The UK FCA describes operational resilience as firms’ ability to prevent, adapt, respond to, recover from, and learn from disruption; its operational-resilience guidance is a useful independent reference for this lifecycle.

A copyable AI outage recovery drill prompt

Run this prompt in a controlled tabletop or game day. Replace bracketed text with your own evidence. Do not enter customer data, credentials, or confidential incident details into an unapproved tool.

You are the facilitator for a financial-services AI outage recovery drill.

Scenario: [shared provider or dependency] is [unavailable / suspected compromised / returning inconsistent results].
Affected business service: [service name and customer outcome].
AI role: [draft / retrieve / rank / recommend / act].
Known dependencies: [model, cloud, identity, data, integrations, monitoring].
Current controls: [kill switch, approval gate, manual fallback, reconciliation].
Constraints: [regulatory notifications, customer commitments, staffing, RTO/RPO if defined].

Create an incident packet with these sections:
1. Incident assumptions and facts that still need verification; never invent missing facts.
2. Dependency blast-radius map: service -> component -> provider -> data/credential -> owner.
3. Per-service decision card: STOP, MANUAL, ISOLATE, RESTORE, PROVE, REOPEN.
4. Manual-mode design: queue limit, dual-control step, customer communication trigger, and exit criteria.
5. Restoration evidence register with owner, timestamp, source link, verifier, and pass/fail result.
6. Staged reopening plan with a small first cohort, monitoring metrics, rollback trigger, and named approver.
7. Open questions, residual risks, and decisions that require legal, compliance, security, or executive approval.

Output tables first. Mark every assumption as "unverified". Do not recommend a financial transaction, customer decision, or production action.

The output is a planning artifact, not authorization to operate. A human incident commander and the relevant risk owners still decide whether to close a channel, notify customers, or restore a service.

Build the recovery evidence pack before a game day

A service is not ready because a dashboard turns green. Capture evidence that can be reviewed later by security, operational risk, audit, and the service owner.

Evidence item	Minimum proof	Owner
Dependency inventory	Versioned map with provider and internal owner	Service owner
Containment record	Time, scope, affected credentials, and preserved logs	Incident commander
Known-good recovery	Image/configuration identifier, backup or snapshot reference, and change ticket	Engineering lead
Access recovery	Secret rotation, privilege review, and test of required roles	Identity/security owner
Data integrity	Reconciliation result and exceptions list	Data or finance-control owner
Service validation	Defined test cases, expected result, actual result, and reviewer	Quality owner
Reopening decision	Cohort, monitoring window, rollback trigger, and approver	Business-service owner

If your environment requires a bare-metal fallback, specify what that term means locally: the approved hardware or isolated recovery environment, the exact build source, the networking boundary, and the person who verifies it. Do not claim a “clean rebuild” without traceable configuration and verification evidence.

Run two scenarios that expose common-mode failure

Scenario 1: Shared model-provider compromise

Assume the provider reports a security incident and the integrity of recent outputs cannot yet be confirmed. Test whether each service can:

stop autonomous or customer-facing actions without losing the audit trail;
identify outputs or decisions that need review during the affected window;
switch to a bounded manual process without silently bypassing approvals; and
reopen only after output validation, credential review, and a sampled reconciliation.

The important result is not a fast recovery time in isolation. It is whether the team can identify the affected decision population and maintain safe control boundaries while evidence is incomplete.

Scenario 2: Identity or cloud dependency fails across services

Assume an identity or cloud control plane is unavailable for several critical services at once. Test whether the business can prioritize the customer outcome rather than the loudest technical alert. For each service, document the manual capacity, the required segregation of duties, the customer-impact threshold, and the trigger to pause intake.

The Bank of England’s operational-resilience overview describes resilience as preventing, adapting to, responding to, recovering from, and learning from disruption. Use the after-action review to update owners, dependencies, and thresholds—not only the incident runbook.

Measure whether the drill made recovery safer

Avoid a vanity metric such as “the tabletop completed.” Track measures that show whether decisions were controlled.

Dependency-map coverage: important business services with a named owner and tested shared-dependency path.
Manual-mode activation time: elapsed time from incident declaration to a documented, controlled fallback.
Evidence completeness: required recovery records present, verified, and linked to an owner.
Reconciliation exception rate: exceptions found in the staged cohort before broader reopening.
Rollback readiness: services with a tested, named rollback trigger and authority.
Unresolved decision count: material open questions at the end of the exercise, grouped by owner and due date.

Set your own targets from service criticality, capacity, and policy. This guide does not provide investment, legal, regulatory, or security certification advice.

Keep recovery and AI testing connected—but separate

An effective control program needs both. Red-team exercises ask whether an AI system can be induced to act or reason outside policy. Recovery game days ask whether the firm can safely maintain or restore an important service when a shared dependency fails. Combine their evidence in a common governance process, but do not substitute one test for the other.

For teams building the broader control environment, pair this playbook with a practical AI financial-analysis checklist and a financial research-agent data-control contract. The result is a clearer operational question during a real incident: not “is the AI back?” but “what service can we safely reopen, with what proof?”

Sources

#ai operational resilience#financial services#third-party risk#incident recovery#prompt engineering

AI Outage Recovery Drill for Financial Services