How to Give a Long-Running Job an Escape Hatch

Make a long-running job — a scan, an analysis, a batch import — recoverable when it silently stalls, so a user is never stuck staring at a spinner with no way out. You'll add a progress signal, a stall detector, and a user-triggered restart. Use this guide for any operation that runs long enough to fail invisibly and that a user is waiting on.

• [ ] A job that can take long enough to appear stuck (seconds to minutes).
• [ ] A way to observe the job's progress (an item count, a percentage, a heartbeat).
• [ ] A safe way to restart the job without corrupting state or duplicating side effects.
• [ ] UI surface to show progress and, when needed, a restart control.

1. 1

   Emit progress

   Emit a progress signal, not just start/finish. A job that only reports "running" is indistinguishable from a job that has wedged. Surface something that advances — items processed, current depth, a heartbeat timestamp — so both the user and your stall detector can tell moving from stuck.

2. 2

   Show progress

   Show the progress in the open. Put the meaningful signal (for example scan depth) directly in the UI rather than hiding it behind a settings gear. When a job feels slow, the first question is "is it doing anything?" — answer it without a click.

3. 3

   Detect stalls

   Detect "no progress," not "still running." Track the last time progress advanced. Define stalled as no progress for N minutes, not elapsed time — a legitimately long job keeps advancing; a wedged one stops:

4. 4

   Enable restart

   Unlock a manual restart after the stall threshold. Keep the restart control disabled while progress is advancing so users don't kill healthy runs, and unlock it only once the job has made no progress for the threshold (for example several minutes). Label it plainly, e.g. "Restart scan."

5. 5

   Restart safely

   Make restart safe and idempotent. Ensure a restart cancels the stuck run cleanly and starts fresh without duplicating completed work or side effects. The escape hatch must not create a worse mess than the stall.

6. 6

   Tune threshold

   Tune the threshold to the job's real variance. Set the stall timeout above the job's normal slowest stretch so healthy runs never offer the restart, but low enough that a genuinely stuck user isn't trapped for long.

   stalled = (now - lastProgressAt) > STALL_TIMEOUT

Base the definition of "stalled" on lack of progress, not raw elapsed time — otherwise you either interrupt slow-but-healthy runs or leave stuck ones running forever.

• During a healthy run, progress visibly advances and the restart control stays disabled.
• When progress stops for the threshold, the restart control unlocks.
• Restarting cancels the stuck run and starts a clean one with no duplicated work or side effects.
• The progress signal is visible without opening settings.
• No healthy long run ever offers a restart prematurely.

• [ ] Job emits an advancing progress signal, not just running/done.
• [ ] Progress is visible inline, not hidden.
• [ ] Stall detection is based on no-progress-for-N, not elapsed time.
• [ ] Restart unlocks only after the stall threshold.
• [ ] Restart is idempotent and cancels the stuck run cleanly.
• [ ] Threshold tuned against the job's real worst-case healthy duration.

• The same trio — visible progress, stall detection, safe restart — applies to any long operation: batch imports, report generation, media processing.
• Pair the manual escape hatch with server-side logging of stalls so recurring wedges get fixed at the root, not just restarted by users.