Ahmadreza SamadiThe AI demo passed. The real work did not.
A polished demonstration does not establish reliability for the actual task.
Original English animated explainer with synthetic narration, English captions and a continuous AS+a-samadi.com watermark. Lina, the bakery, AI assistant and test cases are fictional teaching devices. No customer data, actual AI test, measured accuracy or product endorsement is represented.
The situation
Lina runs customer service for a fictional bakery chain. An AI assistant handles easy demo questions politely, then misses an allergy in a messier request that also changes part of an order. Nobody had agreed what a good answer must include. A polished demonstration does not establish reliability for the actual task.
What the source supports
OpenAI recommends defining an evaluation objective, using representative data and criteria, comparing outputs, and continuing evaluation after changes. Its guidance includes ordinary, edge and adversarial cases. These are evaluation practices, not a guarantee of reliability. OpenAI: Evaluation best practices (opened 28 October 2026)
Practical questions
Our original framework is case → criterion → evidence. Write the task in plain language. Include ordinary and difficult fictional requests, keep separate cases for evaluation, and decide how to judge the actual answer. Compare alternatives on the same conditions. Record failures and rerun after a relevant change rather than treating one success as universal.
Limits and disclosure
All examples are fictional and no actual AI test was run. No accuracy percentage, customer outcome or model ranking is claimed. Expert review may be needed for the real task; machine-assisted source checking here is not that review.
Sources
- OpenAI: Evaluation best practices (opened 28 October 2026)