The LLM Evaluation Playbook for Business Workflows
A practical playbook for testing the AI steps inside real business workflows: define success per task, build a test set from real cases, pick the right graders (code, rubric, LLM judge, human), set release gates, and keep evaluating in production.
- Time needed
- 1 to 3 weeks for a first eval set
- Steps
- 7
- Level
- Intermediate
In short: An LLM evaluation (an "eval") is a repeatable test that tells you whether the AI step in a workflow does its job well enough to ship. For business workflows, the job is narrow: classify this ticket, extract these fields, draft this reply, decide this route. That makes evals easier than they sound. Define what "correct" means for each task, collect real examples, score outputs with the simplest grader that works, set a pass bar, and rerun the set on every prompt, model or data change. Then keep sampling production, because real inputs drift.
Key takeaways
- Evaluate the task, not the model. Generic benchmarks do not tell you if your invoice extraction works.
- Build the test set from real, anonymized cases, including the ugly ones.
- Prefer code checks. Use an LLM judge only for what code cannot check, and calibrate it against people.
- Set a release gate and rerun the full set on every change.
What you will have live
- A written success definition for each AI step in the workflow.
- A versioned test set of real cases with expected results.
- Automated graders and a single command or job that runs the set.
- A pass bar per task, agreed with the business owner.
- A results log so you can compare prompts, models and versions.
- A weekly sample of production outputs reviewed and fed back into the set.
Octacer note: Start with 50 real cases per task, not 5,000 synthetic ones. Fifty cases that reflect actual traffic, scored by people who know the work, will tell you more in a day than a large generated set will in a month. Grow it from production failures.
Why business workflows need their own evals
OpenAI's evaluation guide separates industry benchmarks, standard numeric scores and the specific tests you write for your application, and focuses on the third (OpenAI, Evaluation best practices). It lists "vibe-based evals" and datasets that do not reflect production traffic as anti-patterns. Anthropic's guide makes the same point from the other side: success criteria should be specific, measurable, achievable and relevant to the application (Anthropic, Define success criteria and build evaluations).
In a workflow, the AI step also has neighbors. A small extraction error becomes a wrong CRM field, a wrong route and a wrong email. Evaluate the step on its own and the end-to-end outcome on a smaller set.
Prerequisites
- A list of every AI step in the workflow, with its input and output.
- An owner from the business side for each step who can say what "correct" means.
- Access to real historical inputs, with personal data removed or masked.
- Somewhere to store the test set and results (a repo, a spreadsheet, or an eval tool).
Step 1: Define success per task
For each step, write one or two sentences that a new hire could use to judge an output. Then turn them into metrics.
| Task type | Example | Good metric |
|---|---|---|
| Classification | Route a ticket to billing, technical or sales | Accuracy and per-class recall, plus a confusion table |
| Extraction | Pull invoice number, date, total from a PDF | Exact match per field, after normalization |
| Structured output | Return JSON for a CRM update | Schema valid rate and field accuracy |
| Retrieval answer | Answer from the knowledge base | Correct and supported by the cited source |
| Drafting | Write a reply to a customer | Rubric: correct facts, required points, tone, length |
| Decision with action | Approve, reject or escalate a refund | Agreement with human decision, and zero unsafe approvals |
Mark some failures as critical. A wrong tone is a defect. A refund approved against policy or a promise the business cannot keep is a stop-ship. Track them separately.
Step 2: Build the test set from real cases
Pull a sample of real inputs and have the business owner label the expected result. Include:
- Typical cases in roughly the proportions you see in production.
- Edge cases: empty fields, two languages, scanned documents, very long threads.
- Known failures from support tickets and complaints.
- Adversarial cases where relevant: instructions hidden in an email or document.
Split the set. Use one part while you tune prompts and keep a held-out part you only run before release, so you do not tune to the test. Version the set like code, and record why each case was added.
Step 3: Choose the simplest grader that works
| Grader | Use it when | Watch out for |
|---|---|---|
| Code checks | Exact values, schemas, allowed labels, required strings, length | Normalize dates, currency and case first |
| Reference comparison | There is one known right answer | Many valid phrasings for free text |
| LLM as a judge | Free text quality against a rubric | Bias and drift; needs calibration |
| Human review | Final calibration, high-stakes tasks, new task types | Slow and costly; use samples |
Anthropic's guide recommends prioritizing volume over quality where possible and automating grading, with code-based grading as the fastest and most reliable option when it fits (Anthropic, Define success criteria and build evaluations).
Step 4: Calibrate any LLM judge
LLM judges are useful and imperfect. Research on using strong models as judges found useful agreement with human preferences, and also documented position, verbosity and self-enhancement biases (Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena). Another study showed that simply changing the order of two candidate answers could change which one a judge preferred (Wang et al., Large Language Models are not Fair Evaluators). Work on aligning LLM-assisted evaluators with human preferences also found that people refine their criteria as they grade, so rubrics need revisiting (Shankar et al., Who Validates the Validators?).
In practice:
- Give the judge a short rubric with pass and fail examples, and ask for a reason before a verdict.
- Score one output at a time against the rubric, or swap positions when comparing two.
- Have a person grade 30 to 50 of the same outputs and compare. If agreement is poor, fix the rubric before trusting the numbers.
- Re-check agreement when you change the judge model.
Step 5: Set release gates
Agree pass bars with the business owner before you see results, so the bar is not moved to fit. A gate might read: field accuracy at least the agreed level on the held-out set, zero critical failures, schema valid on every case, and no category worse than the current version. Write the gate down next to the test set.
Step 6: Run evals on every change
Anything that can change outputs should trigger the full set: prompt edits, model or model version changes, temperature or tool changes, retrieval changes and new data sources. Log each run with the date, version, scores and failing cases. Compare against the last release, not just against the bar, so you notice slow regressions.
Keep the eval set and graders portable. OpenAI's guide notes that its hosted Evals platform is being deprecated, with a published timeline (OpenAI, Evaluation best practices). Tools change. A set of cases and graders you own outlives any one platform.
Step 7: Evaluate in production
Test sets age. Each week, sample production outputs, have someone score them with the same rubric, and add new failure types to the set. Track simple signals automatically: schema failures, fallback and escalation rates, human override rates and user corrections. A rise in any of them is a reason to review. NIST's AI Risk Management Framework treats measurement and monitoring as an ongoing function across the AI lifecycle, not a one-time step (NIST, AI RMF).
LLM evaluation checklist
- [ ] Every AI step listed with input, output and owner
- [ ] Success definition and metrics per step
- [ ] Critical failures defined separately
- [ ] Test set of real cases, versioned, with held-out split
- [ ] Code graders first; LLM judge only where needed
- [ ] Judge calibrated against human grades
- [ ] Release gate written before results
- [ ] Full set run on every prompt, model or data change
- [ ] Results logged and compared to last release
- [ ] Weekly production sample feeds the set
Common mistakes
- Testing on examples you wrote yourself. They are cleaner than real inputs and hide failures.
- One overall score. An average hides the category that broke. Report per class and critical failures.
- Trusting an uncalibrated judge. It can be confidently wrong in a consistent direction.
- Tuning on the held-out set. The numbers look better and mean less.
- Stopping at launch. Inputs, models and policies change.
Frequently asked questions
What is an LLM evaluation framework?
A repeatable process and set of tools for testing LLM outputs: success criteria, a test set, graders, pass bars and a way to run and compare results. It can be a spreadsheet and a script, or a dedicated eval tool.
Which LLM evaluation metrics should I use?
Use metrics that match the task: accuracy and recall for classification, exact match per field for extraction, schema validity for structured output, and rubric scores for drafted text. Track critical failures separately.
How many test cases do I need?
Enough to cover your real input types and known failures. Fifty well-chosen real cases per task is a reasonable start. Grow the set from production failures.
Is LLM-as-a-judge reliable?
It is useful for free-text quality when calibrated against human grading. Research has documented position, verbosity and self-preference biases, so use rubrics, check agreement with people and re-check when the judge changes.
How often should evals run?
On every change that can affect outputs (prompt, model, settings, retrieval, data), and on a weekly production sample.
Do I need an eval platform?
Not to start. A versioned set of cases, a script that runs graders, and a results log are enough. Keep them portable whichever tool you use.
How Octacer helps
Octacer has built AI features into business systems since 2017. We design eval sets, graders and release gates for AI steps in CRM, support and operations workflows, so changes ship with evidence. See AI apps and integrations, AI feature acceleration and our AI capabilities architecture. For structured outputs specifically, see reliable LLM JSON for production automations.
Next step
Pick the one AI step that would hurt most if it were wrong, and label 50 real cases for it this week. If you want help building the graders and gates around it, book a call.
Sources
- OpenAI, Evaluation best practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices
- Anthropic, Define success criteria and build evaluations: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685
- Wang et al., Large Language Models are not Fair Evaluators: https://arxiv.org/abs/2305.17926
- Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://arxiv.org/abs/2404.12272
- NIST, AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
Keep going
More playbooks
Want it running on your stack?
We implement this playbook for clients, tailored to your stack, your data, and your team's workflow, and handed over documented.