The LLM Reliability Playbook

Language models moved from demo to dependency faster than engineering practice moved with them. Most teams now ship features that call a model somewhere on the critical path, and most of those teams are operating with reliability practices designed for deterministic software: unit tests that assert exact outputs, dependencies that change only when someone bumps a version, failures that reproduce on demand.

None of those assumptions hold. The result is a class of production systems that work impressively in the demo, mostly work in production, and degrade in ways nobody notices until a customer does. This paper is the reliability playbook we apply when building LLM systems that have to hold up: four disciplines, each replacing an assumption that no longer holds.

Discipline one: evaluation harnesses, because tests cannot assert exact outputs

A deterministic function is tested with examples: given X, expect exactly Y. A model gives you a different Y on Tuesday, and a legitimately different-but-acceptable Y at that. Teams respond in one of two bad ways: they stop testing model behavior entirely, or they pin everything (temperature zero, exact-match assertions) and create tests that pass while the product misbehaves.

The working replacement is an evaluation harness: a versioned set of real cases, run against the live model configuration, scored on properties instead of strings.

• Cases come from production, not imagination. The harness grows every time a real input misbehaves. A fixed synthetic set stops finding problems within weeks.
• Score properties, not text. Did the answer cite the provided context. Did it refuse what it should refuse. Is the JSON parseable. Is the classification in the allowed set. Property checks survive rephrasing; string matches do not.
• Grade with a judge where properties need judgment, and calibrate the judge against a human-labeled sample so you know how much to trust it.
• Gate deploys on the harness. A prompt edit, a retrieval change, or a model swap runs the full set before it ships. Prompt changes are code changes and deserve the same ceremony.

The uncomfortable rule If you cannot say what percentage of your evaluation set passes today, you do not know whether your system got worse last week.

Discipline two: drift detection, because your dependency changes without a version bump

Traditional dependencies change when you update them. A hosted model can change underneath you: providers update weights, adjust safety layers, deprecate snapshots, and tune serving infrastructure. Your code is identical; your product is not. We have watched systems lose accuracy with no deploy, no config change, and no release note anywhere.

Version pinning helps and does not save you. Pins get deprecated, and some behavior shifts happen within a pinned version. The defense is monitoring the model like the unstable dependency it is:

• Run a canary slice of the evaluation harness on a schedule, daily or hourly depending on stakes, against the production configuration. Alert on score movement, not just errors.
• Track output distributions in production: refusal rates, response lengths, format-failure rates, classification mix. Distribution shifts are drift's early signal.
• Keep a swap path warm. An abstraction over the provider API, plus the harness to validate a candidate replacement, turns "our provider degraded" from an incident into a decision.

Discipline three: design for non-determinism instead of pretending it away

Same input, different output is not a bug in a language model; it is the operating characteristic. Systems fail when their design assumes otherwise: retries that expect the same answer, caches keyed on the assumption that answers are stable, UIs that flicker between contradictory results, downstream code that parses whatever arrives.

Designing for it is mostly unglamorous constraint enforcement:

• Constrain outputs structurally. Schemas, enums, and function-calling interfaces shrink the space of what the model can return. The less freedom the output has, the less variance the system inherits.
• Validate at the boundary, always. Every model response is untrusted input. Parse, check, and reject to a fallback rather than letting a malformed answer travel.
• Decide where variance is acceptable and pin the rest. Creative text can vary; a routing decision should not. Route the latter through constrained outputs and, where genuinely needed, deterministic code.
• Make retries semantic. A retry is not "run it again and hope"; it is "reject the invalid answer, tighten the instruction, and bound the attempts."

Discipline four: agent loops that fail safely

An agent is a loop: call the model, let it choose an action, execute, feed back the result, repeat. The intelligence is in the step; the reliability is in the loop, and the loop inherits every classic failure of retry loops. It runs without progress. It repeats a failing action expecting a different result. It burns budget wandering. It has no definition of done.

The controls are the ones any robust loop needs, applied without sentiment:

Treat the agent as infrastructure that happens to have opinions, and it becomes operable. Treat it as a mind, and its failures stay mysterious.

Running it as a program

The four disciplines reinforce each other: the harness gives drift detection its measuring stick; constrained outputs make the harness scoreable; the agent loop's surrender path feeds new failure cases back into the harness. Adopted together, they turn an LLM feature from a demo that usually works into a system with known behavior, watched dependencies, and bounded failure.

That is the difference between shipping AI and operating it.