Restoring Failed AI Generation: A Triage Playbook

Use this playbook when a working AI feature stops returning results:

• A chat or generate action returns nothing, a spinner that never resolves, or a generic error.
• Output appears but is empty, truncated, or replaced by a fallback message.
• One tenant/team gets nothing while others still work.
• A scheduled AI workflow starts logging failures or "connection error" replies.

Do not use this for wrong-but-present answers — that is hallucination, not an outage. Use the hallucination playbook instead.

• On-call engineer — runs point, works the triage tree, owns the fix.
• Comms lead — updates the status channel and, for SEV1/SEV2, the affected client.
• Escalation — the engineer who owns the AI backend / prompt-sync path; paged only per the criteria below.

Work top to bottom. Every step here is read-only — you are diagnosing, not changing state.

1. 1

   Confirm scope

   1 — Confirm scope (30 seconds). Is it everyone or one tenant?

2. 2

   Check API key

   2 — Check the provider API key. A dead per-tenant key blocks generation silently — no crash, just nothing. In one incident a single team's expired key blocked all of their generation and only surfaced under live production testing.

3. 3

   Check quota

   3 — Check quota / billing. The GPT step can fail purely because the client-supplied key hit its quota. We have shipped a whole report pipeline that failed on the client's key quota, not on any logic.

4. 4

   Check prompt sync

   4 — Check the prompt-sync path. If output is present but stale, the live feature may be reading an old prompt. In one platform, prompt-file edits stopped reaching the app because the sync only ever inserted new prompts and never updated existing ones — so every edit was silently dropped.

5. 5

   Check parser

   5 — Check the parser and post-processing. Recurring "connection error" replies and degraded output often trace to the structured-output parser or a post-processing node, not the model. We have fixed both: a parser that crashed the whole reply, and post-processing nodes that silently rewrote correct answers.

6. 6

   Check provider status

   6 — Check the provider status. Only now consider a provider-side outage: check the provider status page and your error-monitoring dashboard for a spike timed to the incident.

# Is it one team or all?
- Reproduce with a second tenant's account.
- One tenant fails, others work  -> jump to Step 2 (key/quota).
- All tenants fail               -> jump to Step 4 (provider/parser).

# Read-only key check
- Fetch the tenant's stored provider key.
- Call the provider's cheapest endpoint (models list) with it.
- 401 / invalid_api_key  -> dead key. Go to Mitigation A.
- 429 / quota exceeded   -> Step 3.

- 429 or insufficient_quota from the provider  -> billing ceiling. Mitigation B.

- Read the prompt the app actually loaded at runtime (log it, don't assume the file).
- Compare against the current prompt file/version.
- Mismatch  -> stale prompt. Mitigation C.

- Pull one failing execution end to end.
- Read the raw model output BEFORE the parser.
- Raw output good, final output bad  -> parser/post-processing. Mitigation D.
- Raw output itself missing          -> Step 6.

• One tenant fails, others fine → almost always key or quota (Steps 2–3), not code. Do not redeploy.
• All tenants fail with 401/429 → provider account or billing, not your code.
• Output present but wrong/old, zero errors → stale prompt (Step 4). This is the silent one.
• Errors only on some turns → parser/post-processing (Step 5), not the model.
• Raw model output is empty and provider status is green → escalate; this is beyond routine triage.

Ordered least → most invasive.

• A — Dead key: replace the tenant's provider key; re-test one live generation. Add the key to active-key monitoring so the next expiry is caught before users are.
• B — Quota: notify the key owner to raise the quota or top up billing; if you control fallbacks, route to a secondary key. Do not silently swallow the 429 — surface a clear message.
• C — Stale prompt: force a prompt re-sync; verify the runtime now logs the new prompt. Fix the sync to upsert (update existing, not insert-only) so edits can't be dropped again.
• D — Parser/post-processing: restore the last-good parser; if post-processing is rewriting good replies, disable that node and re-test. Wrap the agent so backend and quota failures show a friendly fallback, never a raw error.
• E — Provider outage: enable a degraded fallback message and wait; do not thrash the prompt during a provider incident.

Page the AI-backend owner if any of these hold:

• Raw model output is empty and the provider status page is green (not a key/quota/parser issue).
• The prompt-sync path is dropping edits and the upsert fix is non-trivial.
• A key rotation does not restore generation within 15 minutes.
• The blast radius grows from one tenant to all tenants during triage (re-classify to SEV1).

Rollback criteria: if a recent deploy correlates with the outage, roll it back before further diagnosis — a stale-prompt or parser regression is faster to undo than to debug live.

Confirm from the user's side, not just the logs:

1. Run one real generation on the affected tenant end to end and read the output.
2. Confirm the runtime logged the current prompt version.
3. Run a second tenant to confirm no cross-tenant impact.
4. Watch the next few live executions stay clean before closing.

Capture while triaging:

• The failing execution link, the raw pre-parser output, and the provider response code (401/429/5xx).
• The prompt version the runtime actually loaded vs. the current file.
• The error-monitoring trace (we run Sentry on the AI backend and verify events actually reach prod before trusting them).

Comms template:

AI generation — {SEV} — {status}. Impact: {all tenants | team X}. Cause so far: {dead key | quota | stale prompt | parser}. Action: {mitigation}. Next update in {n} min.

Add a line to the retro checklist for every AI outage: which of the four common causes (key, quota, stale prompt, parser) it was, whether monitoring would have caught it, and the one guard that would have prevented it. Feed active-key and prompt-version checks into monitoring so the same class of outage pages you before a user reports it.