OCR Extraction Accuracy Triage Playbook

Use this playbook when an OCR-extracted field is wrong, truncated, garbled, or empty for a document a human can clearly read. Typical triggers:

• A name comes back as first-name-only, cut off, or replaced with garbage characters.
• A field that is populated on the scan lands empty in the intake form.
• Extracted content arrives as a wall of text instead of the typed field the app expects.
• Accuracy drops right after an OCR model or option change.

Classify before you act. Two axes matter: blast radius and downstream harm.

The court-notice case set the bar: name extraction failed across thousands of records — sometimes first name only, sometimes garbage — which is a corpus-wide defect, not an isolated miss [2020]. Any wrong document generated from bad extracted data escalates severity regardless of blast radius.

• Extraction / on-call owner — runs triage, pulls samples, owns the fix.
• Reviewer / data steward — validates against real documents, works the human review queue.
• ML / prompt escalation — owns model choice, prompt, and reference dataset.
• Comms lead — notifies affected teams and tracks the evidence trail.

[ ] 1. Pull the exact failing sample (scan + extracted value + expected value).
[ ] 2. NAME the precise failure — not "OCR is wrong." State the mechanism:
       truncation, newline bleed, empty field, garbled chars, wrong field. [2019]
[ ] 3. Check the data shape: did the field come back as structured data keyed
       to the intake schema, or did it leak from free-text parsing? [1102]
[ ] 4. Inspect the low-confidence flag on that field in the review UI —
       was it flagged, and did the flag fire? [2424]
[ ] 5. Determine blast radius: one document, or systematic across the corpus? [2020]
[ ] 6. Confirm the test set: are you testing against REAL filled samples,
       not blank templates or reference printouts? [1264]
[ ] 7. Record which OCR model and processing options are configured. [1112]

Naming the exact failure is what makes the fix testable — in the court-notice case the root cause was two-fold, a regex that let a newline bleed into the captured field and fallback logic that gave up too early, and neither was visible under the label "OCR is wrong" [2019].

Branch from the named failure to the likely cause.

• Value is cut off or garbled → newline-bleed regex, or fallback logic that gives up too early [2019][2020].
• Field is empty but readable on the scan → primary pattern missed and the fallback is too weak [2019].
• Output is a wall of text, not typed fields → free-text parsing instead of structured, schema-keyed output [1102].
• Only a few reference or blank docs exist to test with → invalid test set; you cannot validate against blank templates, get real filled samples [1264].
• Accuracy regressed after a config change → OCR model or processing-option change [1112].

Least to most invasive. Start at the top and stop when the failure is contained.

1. Correct in the human review UI and rely on low-confidence flags to catch the rest — the review UI shows the scan beside an editable intake form and only then generates downstream documents [2424].
2. Fix the fallback logic and the newline-bleed regex — the two mechanisms behind the truncation defect [2019][2020].
3. Move the step to structured, schema-keyed output so uploads feed typed fields the app can validate, removing the brittle text-parsing layer [1102].
4. Swap or tune the OCR model from admin settings, no code change required [1112].
5. Re-seed the prompt and dataset from a proven reference build rather than tuning from zero [1096].

Escalate to the ML / prompt owner when:

• The failure is systematic across the corpus, not one document [2020].
• A model or option change caused the regression and needs a model-level decision [1112].
• The fix requires re-seeding the prompt or reference dataset [1096].

1. Re-run the exact failing sample and confirm the named failure is gone.
2. Validate the fix against the full real corpus before redeploy — in the court-notice case the fix was validated against 14,200 real notice samples before production, not a handful of examples [2020].
3. Confirm low-confidence flags still fire on the fields that should carry them [2424].

Capture for every incident: the scan, the extracted value, the expected value, and the OCR model and version in use.

OCR extraction issue — [field] on [document type]
Failure: [named mechanism, e.g. newline bleed / weak fallback]. Blast radius: [one doc / systematic]. Affected records routed to manual review; downstream generation held. Fix: [mitigation]. Validated against [real-corpus size]. Status: [investigating / fixed / redeployed].

Link the retro. Prevention that keeps this failure from recurring:

• Keep extraction on structured, schema-keyed output so fields stay typed and validatable [1102].
• Keep the low-confidence review gate in front of document generation [2424].
• Maintain a test corpus of real filled documents — never blank templates or reference printouts [1264].
• Keep the OCR model and options as a runtime admin setting so tuning needs no code change [1112].
• Reuse proven prompts and reference datasets instead of tuning from zero [1096].