Using an LLM in production means accepting uncertainty in three areas: output, provider behaviour, and cost. Sampling, model updates, routing, safety changes, and tokenisation can all alter a workflow without an application-code change. Reproducibility is something to verify, not assume.
A production system needs controls around that uncertainty:
- Run automated evaluations against a representative set of known cases.
- Monitor latency, failures, token usage, and cost for sudden changes and gradual drift.
- Record model, provider, prompt, tool, input, output, and action data so an outcome can be reconstructed.
- Validate structured output and apply deterministic business rules before allowing side effects.
- Maintain a kill switch and a tested fallback to manual processing or another system.
These safeguards can cost more than the inference itself. Before enabling unattended automation, identify the deterministic alternative, the maximum blast radius of a failure, and the control that will detect that failure before a customer or regulator does.