Two Silent Failures in a Field Data-Entry App

We shipped fixes to a browser-based data-entry app for weeks and watched the bug reports keep coming in unchanged. The fixes were live. Nobody was running them. At the same time, the people using the app in the field complained that saving a record felt like it hung — sometimes long enough that they'd tap again, or give up. Two different problems, one shared trait: both failed silently, and both taught us to stop trusting a signal that looked green.

TL;DR — A data-entry app for remote workers was served as static HTML from a CDN host, and the HTML got cached, so workers kept running a stale build after every deploy. Shipped fixes never reached them. Separately, saving a record wrote straight to a client spreadsheet, which was slow enough to stall the worker mid-entry. We fixed cache behavior so every worker reliably loads the latest deployed version, and we moved to database-first saves — the worker advances instantly, and a background job syncs to the spreadsheet every one to two minutes with retry logic. The database became authoritative; the spreadsheet became an eventually-consistent mirror.

The app is a browser-based data-entry tool used by remote workers, often on phones, often on flaky connections. It was hosted as static HTML on a CDN host — a sensible, cheap, fast way to serve a front end.

Two symptoms dominated the support queue.

The first was maddening because it made our own work invisible. We'd fix a reported bug, deploy, and confirm the fix in a fresh browser. Days later the same bug would come back from the field, described exactly as before. From our side the fix was live. From the worker's side nothing had changed. We were shipping into a void and couldn't see it.

The second was about the moment of saving. A worker finishes a record and saves. The save wrote directly to the client's spreadsheet — and that write was slow. On a good connection it was a pause. On a bad one it was a stall long enough that the worker couldn't tell whether the record had gone through. Some tapped save again. Some closed the app. Every one of those reactions risked a lost or doubled record, and all of it eroded trust in a tool people had to use all day.

For the stale-build problem, the obvious fix is "tell people to refresh." We tried the spirit of that, and it doesn't hold. These are remote workers on their own devices; you can't stand behind them. A plain refresh often returns the same cached HTML anyway, because the cache is doing exactly what it was told to do. And even if a hard reload worked once, it doesn't survive the next deploy. Asking humans to manually defeat a cache on every release is not a fix — it's a recurring tax you'll forget to collect.

For the slow-save problem, the obvious fix is "make the spreadsheet write faster." But the spreadsheet was never going to be a fast, reliable transactional store, and it wasn't ours to re-engineer. It's a document the client works in. Writing to it inline meant every single save was hostage to the latency and availability of a system built for humans reading rows, not for a save button being hit hundreds of times a day. You can't tune your way out of putting a slow dependency directly in the user's critical path.

We made two structural changes.

For deploys, we changed the cache behavior so that every worker reliably loads the latest deployed version. The entry point HTML stops being a thing that can silently pin a user to an old build. A deploy now actually reaches the people it's for — which is the entire point of shipping.

For saves, we inverted the write path. Saving now writes to a managed database first. That write is fast and reliable, so the worker advances instantly — the save button does what a save button should do, which is get out of the way. A separate background job then syncs those records to the client's spreadsheet every one to two minutes, with retry logic, instead of writing inline.

The important decision underneath both changes is the same one, stated two ways: the database is authoritative, and the spreadsheet is an eventually-consistent mirror. The worker's experience is bound to the fast, reliable store. The slow, human-facing store gets fed asynchronously and is allowed to lag by a minute or two without anyone noticing or caring.

The save path is now split into a foreground write and a background sync.

When a worker saves, the record lands in the database and the UI immediately advances. That's the whole interaction as the worker experiences it — no waiting on the spreadsheet, no waiting on anything human-facing.

A background job picks up unsynced records and writes them to the spreadsheet on a one-to-two-minute cadence. If a sync attempt fails — the spreadsheet is unavailable, a request times out — the job retries rather than dropping the record. Because the database already holds the authoritative copy, a failed or delayed sync is a mirror that's briefly behind, not lost data.

We also made sessions durable. A worker's session persists across a page refresh, a browser close, and a phone going to sleep — the three things that happen constantly in the field and used to be able to drop someone mid-record.

To keep the asynchronous mirror honest, we added an admin dashboard that shows records as pending, synced, or failed, with a manual retry for anything stuck. That dashboard is what makes "eventually consistent" a responsible choice instead of a hopeful one: eventual is fine as long as someone can see the queue and act when a record won't clear on its own.

The deploy side is simpler to describe. Cache behavior for the served app was changed so the latest deployed build is what loads, every time, without asking the worker to do anything.

The stale-build bug was humbling because of how long it hid. Nothing errored. Our deploys succeeded, our fixes were correct, and every check we ran from a clean browser passed. The failure lived entirely in the gap between "deployed" and "loaded by the user," and that gap produced no logs, no alerts, no red anything. We were measuring the wrong end of the pipe. The lesson that stuck: a deploy that doesn't reach the user is a failed deploy, even when every dashboard is green.

The save change surprised us in a good way. We expected to trade correctness for speed by decoupling the write from the spreadsheet. The opposite happened. Once the database was authoritative, a slow or failed spreadsheet sync stopped being a data-loss event and became a visible, retryable queue entry. Moving the slow dependency out of the critical path didn't just make saves feel instant — it made the whole system more honest about what had and hadn't been recorded.

The stale-build class of failure is closed. Because cache behavior now serves the latest deployed version to every worker, a fix we ship is a fix workers actually run. We measure this the plainest way available: fixes we deploy stop generating fresh reports of the same already-fixed bug, which was the exact signal that had been failing before.

Saving is no longer bound to spreadsheet latency. The worker-facing save completes against the database and the UI advances immediately, so the stall that made people re-tap or abandon a record is gone from the critical path. Spreadsheet writes now happen in the background on a one-to-two-minute cadence with retry, and the admin dashboard shows pending, synced, and failed records so a lagging or stuck sync is visible and manually retryable rather than silent. Durable sessions mean a refresh, a browser close, or a phone sleeping no longer drops a worker mid-entry.

• A deploy that doesn't reach the user is a failed deploy. Serving an app as cached static HTML can silently pin people to an old build. Make cache behavior guarantee the latest version loads — don't rely on humans to refresh.
• Stale caches are their own class of silent failure. No error, no log, no alert — the gap between "deployed" and "loaded" produces nothing you'd notice. Watch that gap, not just the deploy step.
• Keep slow, human-facing systems out of the save path. Write to a fast store first so the user advances instantly; sync to the slow one in the background.
• Pick one authoritative store, and treat the rest as mirrors. The database is the source of truth; the spreadsheet is eventually consistent. A failed sync is then a lagging mirror, not lost data.
• Make "eventually consistent" observable. A background sync with retry is only responsible if someone can see pending, synced, and failed records and retry the stuck ones by hand.
• Persist sessions for real field conditions. Refresh, browser close, and phone sleep are constant on mobile — a session that survives all three is the difference between a saved record and a lost one.