Ahmadreza SamadiThe AI demo passed. The real work did not.
A polished demonstration does not establish reliability for the actual task.
Original English animated explainer with synthetic narration, English captions and a continuous AS+a-samadi.com watermark. Lina, the bakery, AI assistant and test cases are fictional teaching devices. No customer data, actual AI test, measured accuracy or product endorsement is represented.
Read the English film transcript
Lina runs customer service for a fictional bakery chain. She watches an AI assistant demo. Every answer is quick and polite. Then a messier fictional request arrives. A customer wants to change part of an order, and mentions an allergy. The assistant answers confidently, and
misses the allergy. The demo used easy questions. Nobody had written down what a good answer must include. OpenAI’s guide to evaluation starts there: decide the success criteria for your task, before you judge any model. Then build a set of example cases. The guide
says to include typical cases, edge cases and adversarial cases. In Lina’s test kitchen, ordinary orders run smoothly. An edge case, like a half-changed order, slows the line. A tricky one stops it. Keep a separate set of cases for checking, so you aren’t only
testing what you tuned for. Review real outputs against a clear rubric. The guide recommends detailed human review to refine that scorecard. Compare options on the same cases, and run the checks again after every change. No finite set of tests proves an assistant will
always be right. But it can show you where it fails, before a customer finds out. Before you trust a demo, ask three things. Which cases were tested? What counted as success? And what evidence shows it passed? A smooth demo is a first impression.
A tested case is evidence. Ahmadreza Samadi. More at a-samadi.com