Should AI Tools Be Tested by Coworkers Who Did Not Help Build Them?
Uber’s CTO says its two-week AI teams spend two days shadowing a domain expert, build with that person, then use days 6–9 to validate the result with several coworkers. That last step matters more than the speed headline. The person who helped build the tool knows what every field means, remembers which exceptions were discussed, and can quietly patch a bad result. They are almost the easiest possible user. For the first adoption test, give a real task to someone who does the same job but missed the build sessions. No side coaching. Count every point where they need the expert to translate, supply missing context, or explain what changed. If the expert still has to sit beside them, the tool has not removed the repeated explanation. It has turned that person into support. Who would you choose for that first no-coaching test?
Comments
Use one task that recently needed a correction—a duplicate customer name, a stale due date, whatever actually happens. The covering coworker should be able to spot it, fix it, and keep going without calling the person who built the AI. A clean first result proves almost nothing. Coverage starts when an ordinary mistake doesn’t bring the expert back online.
Pick the person covering while the expert is on PTO. They know the job, but they missed every build conversation—the exact moment this tool is supposed to help. Give them one ordinary end-of-day task and keep the builder offline. If the covering coworker needs a private Slack explanation to finish it, the AI did not create coverage. It made time off depend on the same person being reachable.
At 4:45, the covering coworker should open the same queue they use every day—not a demo workspace or the builder’s chat history. Give them one real item and no magic prompt. If they cannot tell what the AI changed, what still needs a person, and how to hand it back, the no-coaching test has already found the problem. PTO coverage should not depend on learning the absent expert’s private way of talking to the tool.