The demo works. The stakeholders are pleased. Then the thing sits in staging for 7 months and quietly gets switched off.
That pattern repeats across retail AI programmes with enough regularity to be predictable. The model is almost never the reason. What kills a pilot is the set of things nobody scoped because they are not interesting: evaluation, data contracts, a cost ceiling, guardrails, and a documented path to approval.
Each of those is a distinct failure mode with a distinct fix. Here they are in the order they usually bite.
Failure mode 1: no evaluation, so nobody can say whether it works
A pilot without evals has one quality signal, which is whether the person demoing it liked the output. That signal does not survive contact with a risk committee, and it cannot detect regression when a prompt or a model version changes.
The symptom is a specific meeting. Someone asks "is it good enough to turn on for real customers," and the team answers with adjectives.
The fix is a graded test set built before the prompt is tuned. Anthropic's guidance on developing tests sets out 3 principles worth copying directly: be task-specific, automate grading wherever possible, and prioritise volume of test cases over the polish of any single one.
Grading method follows the task. Use code-based grading (exact match, similarity scores, ROUGE-L) for anything objective such as a category label or an extracted field. Use model-based grading for subjective properties like tone, or for a binary check on whether personal data appeared in an output.
Size the set to the decision it supports. A returns-triage classifier being cleared for production needs a few hundred labelled cases including the awkward ones: empty input, deliberately hostile input, and the ambiguous cases the human team argues about.
Set the pass bar before you see the scores. A threshold chosen after the fact is a description of the current build, not a standard.
Failure mode 2: the data the pilot ran on is not the data production has
Pilots run on an export. Production runs on a live feed maintained by a team that has never heard of the pilot. The export was clean because a data engineer cleaned it by hand, once.
The symptom appears 3 weeks after launch. Accuracy drops with no code change, and the root cause turns out to be an upstream field that changed type, went null, or started arriving 6 hours late.
The fix is a data contract on every input the system depends on. Not documentation, a machine-checkable artefact that the producing team owns and CI enforces.
The Open Data Contract Standard, governed by Bitol as a Linux Foundation AI and Data project, is a reasonable place to start rather than inventing a format. It covers schema, quality rules, service levels, stakeholders, and roles in versionable YAML, which means the contract can fail a build.
The specific fields worth pinning for an AI feature: schema and types, allowed null rate per field, freshness (how stale a row may be), and cardinality bounds on anything the model treats as a category. Those 4 catch most silent drift.
Then attach an owner and an alerting path. A contract with no named producer is a document, and documents do not page anyone at 2am.
Failure mode 3: no cost ceiling, so finance stops it
Pilot economics are misleading by construction. A pilot runs 200 requests a day against a hand-tuned prompt. Production runs 200,000 against a prompt that grew a retrieval block, 6 tool definitions, and a 40-example few-shot section.
The symptom is a finance escalation in month 2 rather than a technical failure. By then the feature is live and the only available lever is turning it off.
The fix is a unit-cost model built before launch and a hard ceiling enforced in code. Cost per successful task, not cost per call, because retries and failed tool calls are real spend.
Prompt caching is the single largest lever on repetitive workloads and it is worth modelling precisely. On Anthropic's pricing, cache writes cost 1.25x the base input token price for the 5-minute TTL and 2x for the 1-hour TTL, while cache reads cost 0.1x. Minimum cacheable prefixes range from 512 to 4,096 tokens depending on the model, so short prompts do not cache at all.
That shape rewards a specific design. Put the stable material (system instructions, tool definitions, catalogue context) at the front of the prompt where it can be cached, and keep the variable material at the end.
Then enforce the ceiling. A per-tenant and per-day token budget, a maximum output length, and a circuit breaker that degrades to a cheaper model or a deterministic fallback rather than paging someone.
Failure mode 4: guardrails treated as a launch-week task
Guardrails get scoped as a checklist item near the end, which is roughly when the team discovers that the feature's design assumed a trust boundary that does not exist.
The symptom is an eleventh-hour architecture change. Security reviews the design, finds the model can reach a system it should not, and the fix is structural rather than a filter.
The fix is to treat the threat model as a design input. The OWASP Top 10 for LLM Applications is the shortest useful list, and 4 of its entries account for most real retail incidents: prompt injection, insecure output handling, excessive agency, and sensitive information disclosure.
Excessive agency is the one retail teams underestimate. OWASP describes it as granting a model unchecked autonomy to act, and it is exactly what happens when a shopping assistant gets a tool that can issue a refund because that made the demo better.
Concrete controls, in the order they pay off:
- Scope every tool to the narrowest capability that satisfies the use case, and give write tools their own approval step.
- Treat model output as untrusted input to whatever consumes it, including your own templating layer.
- Redact at retrieval time rather than at generation time, so sensitive fields never enter the context window.
- Log every tool invocation with the same rigour as a database write, including caller identity.
- Rate-limit per tenant and per session, because prompt injection attempts arrive in volume.
Failure mode 5: no approval path, so the pilot has nowhere to go
The most common ending is not rejection. It is the absence of anyone with the authority to say yes.
The symptom is a pilot that passes every technical review and then waits, because legal, risk, and the business owner each assume one of the others holds the decision.
The fix is to establish the approval path in week 1 and build the evidence it will ask for as you go. Write down who signs off, what they need to see, and what would make them reverse the decision later.
The NIST AI Risk Management Framework is useful here even though adoption is voluntary. It gives risk and legal a vocabulary they already recognise, and it frames trustworthiness as something addressed across design, development, use, and evaluation rather than assessed once at the end.
Map your artefacts onto that vocabulary. The eval set is your measurement evidence, the data contracts are your input governance, the tool logs are your accountability trail, and the cost ceiling is an operational control.
Then agree the kill criteria in writing. A committee that knows how the feature gets switched off approves it faster than one that suspects the decision is permanent.
A worked example: returns triage
Returns triage is a good test case because it is narrow, high volume, and has an obvious deterministic fallback. It is also where the 5 failure modes show up in their most concrete form.
Evals: a few hundred historical tickets graded by the L1 team, weighted toward the disputes rather than the clean cases. Code-based grading on the policy outcome, model-based grading on the drafted customer reply.
Data contracts: order status, return window, item condition codes, and prior-return count. Each one gets a freshness bound, because a return decision made against yesterday's order status is a decision made wrongly.
Cost ceiling: policy text and item taxonomy sit in the cached prefix; the ticket and order context go at the end. Cost is tracked per resolved ticket, including the ones that escalate.
Guardrails: the assistant drafts and classifies, it does not issue refunds. A refund tool would have made the demo better and the security review much longer.
Approval path: agreed in week 1 with the returns operations lead as the accountable owner, with the kill criterion written down as a handle-time or escalation-rate regression over a rolling week.
On an assistant built this way, L1 handle time came down 41 percent. None of that came from a better model. It came from a narrow scope and a fallback that was always available.
Sequencing: what to build in which order
| Order | Artefact | Owner | Gate it unblocks |
|---|---|---|---|
| 1 | Written success criteria and kill criteria | Business owner | Everything downstream, including what to measure |
| 2 | Graded eval set with a pre-agreed pass bar | Engineering | "Is it good enough" answered with numbers |
| 3 | Data contracts on every input | Producing data team | Accuracy that survives week 4 |
| 4 | Unit-cost model and enforced ceiling | Engineering with finance | Budget approval that holds at production volume |
| 5 | Threat model and tool scoping | Engineering with security | Security review without a redesign |
| 6 | Audit logging and rollback plan | Engineering | Risk and legal sign-off |
The ordering matters more than the contents. Every item exists to unblock a specific gate, and doing them in this order means no gate arrives before its evidence does.
The uncomfortable version
Most of this work is not AI work. It is data engineering, cost engineering, security review, and stakeholder management, which is why teams staffed entirely with model expertise stall at the same point.
The pilots that reach production tend to be less impressive than the ones that do not. They do one narrow task, they have a deterministic fallback, and someone can produce a number when asked how well they work.
FAQ
How large should an eval set be before going to production?
Large enough that a 5-point accuracy change is not noise, which for most classification-shaped retail tasks means a few hundred graded cases. Weight it toward the edges rather than the easy middle, since the easy middle is where every model already succeeds.
Do data contracts need a formal standard, or is a schema check enough?
A schema check catches type changes and nothing else. The failures that actually degrade an AI feature are freshness, null rate, and cardinality drift, which is why a standard covering quality rules and service levels is worth the setup cost.
What is a reasonable cost ceiling for a customer-facing AI feature?
There is no general answer, which is the point. Derive it from the value of the task it replaces or accelerates, express it as cost per successful task, and enforce it in code rather than in a dashboard someone checks weekly.
Who should own AI guardrails, engineering or security?
Engineering builds them, security specifies them, and both review the tool scope together before the first tool is written. Guardrails added after the tool surface is fixed are filters, and filters are the weakest control available.