Reliable Scheduled Reminders: An Incident Response Playbook

A time-triggered reminder was missed, sent twice, or arrived at the wrong moment. This playbook takes a stressed on-call from symptom to root cause with read-only diagnostics first — trace the execution's item counts before touching schedule config — then a least-to-most-invasive fix menu. Work top to bottom; do not edit the schedule until you have read the evidence.

Use this playbook when a scheduled reminder or notification behaves wrong on timing or count, and there is no hard error to chase. The job reports success, yet the reminder does not match reality.

The failure lands in one of three shapes:

Common triggers behind these symptoms:

Classify by how many recipients the fault silently covers, not by whether anything errored. A reminder marked "sent" that never reached its window is still a full miss to the recipient.

Assign before touching config. Keep names generic; fill with whoever is on rotation.

• Incident owner (on-call): runs the triage tree, decides mitigations, owns the timeline.
• Comms lead: posts status, fields "did my reminder go out?" questions, shields the owner.
• Automation owner: subject expert for the reminder workflow and its schedule/webhook trigger.
• CRM owner: owner of the source system's due-date and remaining-days fields the reminders key off.

For a SEV-3 the owner may hold every role. For a SEV-1 under-coverage, split comms out immediately — recipients will ask individually.

Run in this order. Everything here is read-only: read execution history and config values, count items, compare numbers. Do not edit the schedule, re-run, or re-send until the decision tree points you there.

1. Trace the execution's item counts, node by node. Open the last runs and read how many items each node received and emitted. A node that ingests two items but emits one is the tell for a duplicate/collapse bug [1067].
2. Compare poll interval vs due-window vs lead-time. Read the schedule's poll interval and the width of the due-window it checks, then compare both against the reminder lead time. A 30-minute poll against a 10-minute window cannot cover the gap and will drop reminders while still burning ~48 runs a day [1457].
3. Check boundary dates. For date-triggered reminders, confirm the fetch window and stage matching actually include the exact boundary days. Compare the stage logic against the CRM's own remaining-days value — off-by-one boundaries silently skip exact-day (for example 100-day and 120-day) reminders [66].
4. Check scan coverage. Confirm the scan reads the whole dataset, not only the most recently added records. If it only reads the newest page, older records are silently out of scope [2101].
5. Confirm the collision case. Read a day when two or more records were due together. If only one reminder went out, the merge step is processing only the first input [1067].

[ ] Per-node input/output item counts read for recent executions
[ ] Any node that takes N items but emits 1? (collapse bug)
[ ] Poll interval value read
[ ] Due-window width read
[ ] Reminder lead time read  → does poll interval fit inside window vs lead time?
[ ] Runs-per-day counted (wasted runs vs reminders actually due)
[ ] Fetch window + stage match include exact boundary days?
[ ] Stage logic aligned to CRM remaining-days value?
[ ] Scan reads full dataset (paginated) or only newest page?
[ ] A same-day collision day inspected: N due vs N sent

Branch on which of the three shapes the read-only pass confirmed.

• Missed, across many recipients, runs look healthy but coverage is thin → poll interval does not fit the due-window against the lead time. A 30-minute poll over a 10-minute window drops ~2 of 3 reminders and runs 48 times a day [1457].
• Missed, but only older records → the scan reads only the newest page and under-covers the long tail [2101].
• Missed or mistimed, but only on exact boundary days (for example 100/120-day) → boundary-day matching, not aligned to the CRM remaining-days value [66].
• One of two same-day records missed (and the other at risk of a repeat) → a merge/Code node hardcoded to a single item, reading only the first input [1067].
• Duplicate risk specifically → confirm whether an existing-task / already-sent check is present before re-sending anything; without it, recovering a missed record can double-send the one that already went [1067].

Apply the least invasive fix that matches the confirmed root cause. Escalate up the list only when the cheaper fix does not apply.

1. Align the boundary logic (missed/mistimed on exact days). Align the stage matching to the CRM's own remaining-days value and widen the fetch window to include the boundary days, while preserving the daily remaining-days refresh [66].
2. Loop all inputs at the merge (same-day duplicates). Rewrite the merge/Code node so it iterates every input and returns N merged items for N inputs instead of one hardcoded object. Recover the missed record by letting the existing already-sent check skip the one already delivered — recover without duplicating [1067].
3. Paginate the whole scan (long-tail misses). Upgrade the daily scan to paginate the entire dataset each run so every record is back in scope, not just the newest page [2101].
4. Drive off real due-events, not a poll window (systemic missed + wasted runs). Replace the guessed poll-vs-window arrangement with a per-minute due-check job that pushes to a webhook only when something is actually due, with the intended lead time. This removes both the missed-reminder gap and the wasted runs [1457].

Escalate the moment a cheaper fix does not fit the evidence.

• Escalate to the automation owner when the fix means restructuring the trigger — moving from a polling window to a per-minute due-event job that calls a webhook [1457].
• Escalate to the CRM owner when boundary matching must be aligned to the source system's remaining-days value or the fetch window widened at the source [66].
• Escalate any duplicate risk before re-sending: confirm the already-sent check exists and works, so recovery of a missed record cannot double-send the one already delivered [1067].

Verify from the recipient's side, and specifically re-check the single-record case so a duplicate fix has not broken it.

1. Verify no regression on single-record days. After looping all inputs at the merge, run a day with exactly one due record and confirm exactly one reminder goes out — one input, one item, one send [1067].
2. Confirm the collision case. Run a day with two same-day records and confirm both are handled, with the already-sent one correctly skipped and no duplicate [1067].
3. Confirm exact-day reminders fire. Check that the boundary stages (for example 100-day and 120-day) now match, while the daily remaining-days refresh still runs [66].
4. Confirm long-tail coverage. Confirm older/legacy records now come through, not just the newest ones [2101].
5. Confirm due-event coverage and run economy. After moving to the per-minute due-check + webhook, confirm reminders fire on real due events within the lead time and that the wasted poll runs are gone [1457].

Capture evidence while it is fresh, before any re-run overwrites the execution history.

• Per-node input/output item counts from the failing executions, especially any node that took N items and emitted 1 [1067].
• The schedule values at the time: poll interval, due-window width, lead time, and runs-per-day [1457].
• The boundary-day comparison: stage logic vs the CRM remaining-days value [66].
• Scan coverage evidence: whether the run read the full dataset or only the newest page [2101].

Status-update template:

[SEV-x] Scheduled reminder — <UPDATE>
Symptom: missed / duplicate / mistimed
Impact: <who/how many recipients, customer-visible effect>
Root cause: <confirmed / suspected — poll/window mismatch, merge collapse, boundary match, newest-only scan>
Action: <read-only item-count trace done / mitigation applied>
Regression check: <single-record-day verified? duplicate check confirmed?>
Next update: <time>

Book a short retro and turn the root cause into a preventative step.

• Drive reminders off real due-events via a per-minute job → webhook, rather than a poll window guessed against the lead time [1457].
• Make merge/Code nodes loop all inputs by default, and keep an existing-task check so recovery never double-sends [1067].
• Key date logic to the CRM's remaining-days value and include boundary days in the fetch window, so exact-day reminders cannot be skipped [66].
• Paginate the full dataset on any daily scan so the long tail stays in coverage [2101].
• Add a standing check that counts reminders actually due versus reminders sent, so silent under-coverage surfaces before a recipient reports it [1457][2101].