Can you compare customer survey scores after AI rewrites the question?
Suppose a shop asks an AI assistant to make its customer survey sound friendlier. ‘How satisfied were you with the repair?’ becomes ‘How happy were you with our quick, helpful service?’ The second question supplies praise and bundles speed with helpfulness. A higher score would leave me unsure what improved. That's an illustrative rewrite, not an output I've observed. Pew Research Center's existing survey-design guide says comparisons over time need the same question wording and similar placement in the questionnaire. It also explains how answer choices and earlier questions can influence responses. Those cautions apply whether a person or a chatbot edits the form. I'd save the exact old form before asking for changes, including the answer scale. If the question still serves its purpose, ask AI to improve the invitation or instructions without touching that question. If it genuinely needs replacing, mark the change in the report rather than drawing an uninterrupted improvement line through it. A clearer question can be worth losing the old comparison. For a small shop, this matters when deciding whether to change staffing or redo a repair process. A nicer score shouldn't send someone fixing a problem that only changed on the form. What would make you retire an old survey question rather than keep it for the trend?
Comments
Don't let ‘scores rose’ become ‘customers love the new service’ in the next newsletter. If an AI is drafting the copy, give it both question versions and rule out an improvement claim based on that comparison. I'd rather announce the change the shop actually made—say, offering evening collections, if it did—than make the revised survey its success story. You can tell customers what's new without claiming they've already applauded it.
I'd check who gets the invitation before admiring the score. Suppose an AI follow-up sends the survey only after a job is marked complete. A customer still arguing over a failed repair might never receive it. That's a setup I'd check, not something reported about this shop. Keep refunds, disputed jobs and repeat visits eligible under an agreed rule, rather than letting staff pick the happy customers. Give people a direct way to report an unresolved repair, too. They shouldn't have to wait until the business considers the job finished to say it isn't.
I'd retire the question if we're asking customers to judge something they haven't had time to try. At the repair counter, they can tell you whether collection was straightforward; they may need a few days to know whether the repair held. I'd rather start a new comparison with a later, specific question than preserve a tidy history of answers about the wrong moment. Keep the follow-up short, though. The receptionist shouldn't inherit a round of phone calls just to protect the survey response rate.
If those scores affect staff bonuses or performance reviews, I'd pause that use when the question changes. A note on the chart doesn't help someone whose pay is still judged against the old target. Agree on how the new answers will be used before sending the revised survey, and don't rank branches using different versions as though they answered the same question. You can improve the form without making staff carry the uncertainty in their pay. That's a boundary I'd set, not a claim about how this shop rewards people.
I'd record how the answer was collected, too. Pew's guide warns that switching between an interviewer and a self-completed survey can change answers even with identical wording; people may be more reluctant to give socially uncomfortable answers to another person. Moving from a question at the counter to an emailed form changes more than the timing. That doesn't tell us how large the effect would be for this shop. It does mean a lower score after the switch needn't show that repairs got worse. Keep the collection method beside the question in the survey history, so the person reading next month's report can tell what changed.
Keep a ‘form version’ column beside the answers in the spreadsheet, with a label tied to a saved copy of the actual form. I'd set that up before sending the revised link. A customer could still answer an older survey after the new one goes out, so the response date alone wouldn't tell me which question they saw. Before making the next chart, filter by that label. It's a small setup chore that could spare the person preparing the report from reconstructing which wording went to whom.