Beyond the demo: what production AI actually needs.

The gap between a demo and a service

A prompt in a notebook is not a product. The distance between a working demo and a service customers depend on is measured in retries, timeouts, evals, traces, and the quiet discipline of typed contracts.

Most AI features fail in production not because the model is wrong, but because the surrounding system has no opinion about what to do when the model is wrong. The demo has one happy path. The service needs ten unhappy ones.

The demo has one happy path. The service needs ten unhappy ones.

Name the failure modes first

Before writing the prompt, write the failure list. Every LLM call has at least four modes worth planning for, and each one deserves a designed response — not a spinner.

• Wrong answer — needs verification, citations, or a confidence gate.
• Unsafe answer — needs policy filters and a refusal path with a real message.
• No answer — needs a cached fallback, deterministic default, or human handoff.
• Slow answer — needs a budget, a spoken or visual placeholder, and a cancel path.

Draw the contract

Structured output, typed tool calls, and a schema the caller can trust. The moment the model is free to return anything, your product is free to break in any way.

Validate on the way in and on the way out. Reject malformed tool calls loudly in dev; repair them quietly in prod with a bounded number of retries and a hard ceiling on cost.

Instrument everything

Traces per request, spans per tool call, cost and latency per turn. You cannot improve what you cannot see, and you cannot ship what you cannot operate.

The most valuable dashboard is boring: p50 and p95 latency by route, tool call success rate, retry rate, cost per successful task. If any of those numbers is missing, that is the next thing to build — not the next feature.

The pre-launch checklist

Before an AI feature meets a real user, I want to be able to answer yes to all of the following. If I cannot, the feature is not ready — no matter how good the demo looks.

• Every model call has a timeout, a retry policy, and a fallback.
• Every tool call has a typed schema, tested with adversarial inputs.
• There is an eval set with at least fifty examples and a scored regression run.
• Traces and cost per request are visible on a dashboard the team actually looks at.
• There is a written plan for what happens when the provider is down.