Preventing Silent AI Write Failures: A Playbook

Trigger this playbook when an AI action reports success but the system of record disagrees:

• The agent says "booked / saved / done" but no record exists downstream.
• A voice or chat agent confirms an appointment that was never created.
• A workflow marks a step complete while its write silently failed (404, empty payload, missing field).
• Users report "I got a confirmation but nothing happened."

The defining trait of this incident is that nothing looks broken — no error surfaced, because the agent spoke before the write was confirmed.

• On-call engineer — reproduces the false confirmation, traces the failed write, adds the gate.
• Comms lead — for SEV1, coordinates reaching affected customers whose action didn't complete.
• Escalation — owner of the target system (CRM/calendar/order API).

Read-only until Mitigation.

1. 1

   Pull one case

   1 — Pull one confirmed-but-missing case. Get the conversation/run where the agent confirmed success, then check the system of record for the write. Confirm the record truly isn't there (not just delayed).

2. 2

   Read real result

   2 — Read the write step's real result. Look at the actual API response of the write, not the workflow's success flag.

3. 3

   Check for 404

   3 — Check for the classic missing-field 404. A write can 404 because a required field was empty. We fixed a PUT /contacts that 404'd by using "is not empty" instead of "exists" and setting Continue On Fail so an empty value couldn't crash the flow — but the point is the write failed, so success must not be reported.

4. 4

   Check capture

   4 — Check upstream capture. The write often fails because the agent never captured a required input. A booking fails without a service address; verify the agent actually collects every field the write needs before it attempts the write.

5. 5

   Check trigger

   5 — Check the confirmation trigger. Find what the "done" message is wired to. If it fires off reaching the step rather than off the write's success, that is the bug. We had a voice agent say "you're all set" while no appointment was ever created — the confirmation was gated on validation, so an empty payload could no longer crash it or fake success.

- Open the write node/step for that run.
- Read its HTTP status and response body.
- 404 / 4xx on the write        -> write never landed. Step 3.
- Empty/partial request payload -> missing input. Step 4.
- 2xx but no record             -> wrong target / mapping. Step 5.

• Write returns 4xx/404 → the confirmation must not fire. Gate on the real response (Mitigation A).
• Payload empty/partial → capture gap upstream (Mitigation B). Validate inputs before the write.
• Write "succeeds" but no record → wrong target or mapping (Mitigation C). Verify the write hit the right system of record.
• Confirmation fires before the write resolves → ordering bug (Mitigation A). Move the confirmation strictly after a verified write.

Ordered least → most invasive.

• A — Gate the confirmation on a verified write: the agent confirms success only after the write returns success and the record is verifiable. No write, no "done."
• B — Validate inputs before writing: put a validation node before the external write so an empty or partial payload can't be sent; require every field the write needs.
• C — Fix the write robustness: use "is not empty" over "exists", set Continue On Fail so a bad value doesn't crash the whole run, and confirm the target system and mapping are correct.
• D — Make the failure loud: on any failed write, alert the team with the run link and reason, and give the customer an honest "we couldn't complete this" instead of a false confirmation.
• E — End-to-end test to surface silent failures: run the full pipeline against real and empty inputs, then make every silent failure loud rather than swallowed.

• Any customer received a false confirmation for a booking/order that didn't happen → SEV1, page the system-of-record owner and comms to reach affected customers.
• The write intermittently fails on a valid payload (provider-side) → escalate to the target-system owner.
• The false-confirmation pattern spans multiple flows, not one → escalate; the confirmation is gated on the wrong signal system-wide.

1. Re-run the flow and confirm the agent says "done" only when the record actually exists in the system of record.
2. Force a failing write (empty required field) and confirm the agent does not confirm success and the team gets a loud alert.
3. Confirm a valid run creates the record and the confirmation matches it.
4. Backfill or re-run any real customer actions that were falsely confirmed during the incident.

Capture: the conversation/run with the false confirmation, the write step's real response, the missing record proof, and the alert (or absent alert) for the failed write.

Comms template:

AI silent write — {SEV} — {status}. Symptom: agent confirmed {action} with no record. Cause: {404 missing field | empty payload | confirmation fired early}. Fix: confirm only off a verified write + loud failure alerts. Affected customers: {n, contact plan}.