Why AI features fail after the demo
A demo is usually run by someone who already knows how to phrase the request. A product is used by customers who do not. That gap is why an AI feature needs to be tested against real inputs and real failure cases rather than judged by a handful of polished examples.
Traditional software often fails visibly — an error, a blank state, something clearly broken. AI features can fail fluently. The output can look plausible while being wrong, incomplete or unsafe for the decision the user is making.
Designing around that means making fallibility part of the product: show sources where they matter, make correction easy, provide a clear refusal or escalation path, and match the amount of automation to the consequence of being wrong.
Evaluation is the product spec
If a team cannot agree what acceptable output looks like across representative real examples, the AI feature is not specified well enough to ship. The evaluation set turns a subjective discussion about model quality into something that can be measured release by release.
A useful eval set covers the happy path, edge cases and cases where the correct behaviour is to refuse, escalate or admit uncertainty. The exact number of examples depends on the task, the variability of the inputs and the consequence of being wrong.
Once the eval set exists, model selection becomes a measurement instead of an argument. Prompt and retrieval changes can be compared against the same examples, and regressions are easier to spot before customers discover them first.
The unit economics need to be measured in production
AI products have a usage-linked marginal cost that traditional software often does not. The useful number is not the provider's token price in isolation; it is the cost of completing the customer task successfully, including retries, retrieval, model calls and any human review that remains in the workflow.
Measure cost per successful action and cost per active user, then compare those numbers with the value delivered and the price paid. Use real usage distribution rather than designing the business model around a theoretical average user.
Model routing can reduce cost when simpler requests can use a less expensive model without crossing the quality threshold. The saving should be demonstrated against the evaluation set and production usage rather than assumed in advance.
- —Measure cost per successful action in production, not from provider pricing pages alone.
- —Look at heavy-user behaviour as well as the average.
- —Route to a cheaper model only when it still meets the quality threshold.
- —Cache repeated or reusable work where it is safe and correct to do so.
- —Set usage limits or pricing rules before launch if marginal cost can grow materially with usage.
Choosing between prompting, RAG and fine-tuning
If the model lacks current or domain-specific knowledge, test retrieval. If it lacks a behaviour, format or tone, improve the prompt and workflow first. Fine-tuning becomes worth testing when the task is narrow and stable, you have enough representative examples to evaluate it properly, and there is a clear quality, cost or latency reason to add the extra complexity.
The important point is sequence. Start with the simplest approach that can solve the task, measure it against the evaluation set, and only add architecture when the evidence shows what the current approach cannot do.
That makes prompting, retrieval and fine-tuning product decisions with measurable trade-offs rather than technology preferences chosen before the problem is understood.