LLM Financial-Research Reporting Pipeline

Failure mode 01 Unreliable structured output

The pipeline was expected to produce structured data alongside human-readable research documents. In practice, the structured output was often malformed or incomplete. Downstream systems that consumed the data had to handle missing fields, incorrect types, and occasional outright parsing failures. This created a hidden tax: every consumer of the output needed defensive parsing logic, and every parsing failure required manual investigation. The structured data — which was supposed to reduce operational overhead — was generating additional work.

Failure mode 02 Citations that could not be trusted

Research documents required regional citations — references to specific sources that supported the analysis. The citation step did not reliably verify that the cited source existed or that the reference was accurate. Some citations pointed to sources that could not be located, while others were plausible but unverifiable. For a financial-research organization, citation integrity is not a nice-to-have. Documents with unreliable citations erode credibility and create risk if the underlying source is challenged.

Failure mode 03 Charts lost in rendering

The chart-rendering step depended on browser-side execution inside the PDF generation process. Charts were generated via client-side JavaScript, and the PDF engine captured the page before the chart lifecycle finished. The result: PDFs that occasionally shipped with blank or partially rendered charts. The failure was intermittent, which made it the most insidious problem in the pipeline. A document could render correctly ten times in a row, then fail on the eleventh — with no change in input or configuration. The team could not reproduce the failure on demand, which meant it could not be fixed reliably.

Failure mode 04 No safety net for empty output

The pipeline had no final verification step. If any stage produced empty or incomplete output — whether a blank chart, a missing section, or an empty data payload — the document proceeded to distribution anyway. There was no gate that said "this output is incomplete, do not ship it." The absence of a verification layer meant that quality problems surfaced after distribution, when the cost of correction was highest and the damage to credibility was already done.

Decision 01 Render charts server-side

The chart-rendering failure was caused by a race condition between browser-side chart execution and PDF capture. Octacer eliminated the race entirely by moving chart generation out of the browser and into the server-side document build. Charts were generated deterministically from the same structured data that fed the rest of the document. The PDF renderer received complete chart images — nothing left to execute, nothing left to wait for. If chart generation failed, the pipeline failed immediately, and the error surfaced before distribution.

“A chart that cannot render is a pipeline failure, not a surprise.”

Decision 02 Verify citations before publication

Citation verification became a dedicated pipeline stage with a two-part check. First, the citation's source and region were validated against the available regional source data. Second, the reference was checked against the actual source content to confirm the citation was accurate. Citations that failed verification were flagged for review rather than silently included. This concentrated human attention on the uncertain cases while allowing verified citations to flow through automatically.

“Never cite what the system cannot verify.”

Decision 03 Validate structured output against a schema

The structured output was redefined around an explicit schema. Every document generation run produced structured data that had to conform to that schema — correct field types, required fields present, no unexpected nulls. Schema validation ran as a pipeline stage immediately after generation. Malformed output was rejected at that point, before it could reach any downstream consumer. The defensive parsing logic that consumers had built up over time became unnecessary.

“Parseable output is a contract, not a hope.”

Decision 04 Add a final output verification gate

The final stage of the pipeline was a verification gate that checked the complete document before release. The gate validated that all required sections were present, all charts rendered, all citations verified, and all structured output conformed to schema. If any check failed, the document was held for review instead of being distributed. The gate made silent failure impossible: either the document passed every check and shipped, or it failed a check and someone was told exactly why.

“Empty output must never ship.”