Can an AI report give your opt-in customer poll a margin of error?
Before asking an AI assistant to add a margin of error to a customer poll, tell it how people entered the survey. The response count alone isn't enough. AAPOR's existing guide says a margin of sampling error applies to probability samples, where people have a known, non-zero chance of selection. It doesn't apply in the same way to an opt-in online poll. Model-based intervals are possible, but depend on assumptions that need explaining; they aren't a sampling margin obtained just by entering the response count into a calculator. Imagine a café asking its social followers whether to open earlier. Those replies can help identify people to talk to about a trial. A tidy plus-or-minus figure would not establish what all nearby commuters want. That's an illustrative decision, not a poll I've tested. I'd ask the assistant to write ‘among people who chose to answer’, keep the recruitment method beside the result, and leave out a conventional sampling margin the survey design doesn't support. Before paying for a permanent staffing change, try a limited opening and record actual early purchases. The poll can suggest the experiment without pretending to settle the decision.
Comments
I'd give the assistant the exact question and answer options, too. AAPOR's guide ends by warning that question design introduces errors a sampling margin doesn't measure. In the café example, ‘Would you like us to open earlier?’ doesn't ask whether someone would actually buy breakfast then. An AI report shouldn't quietly turn those yeses into likely customers. I'd ask it to quote the question beside its conclusion and flag any stronger interpretation. That's a reporting check I'd suggest, not a survey result. It could spare the opening-shift staff a permanent earlier alarm based on a promise nobody made.
I'd ask the café owner what would make the earlier opening worth keeping before the trial starts. Early sales alone could mislead: a regular might buy at seven instead of eight, leaving the café with the same sale and an extra paid hour to cover. Give the assistant whole-day sales and the added staffing and preparation costs, not just the new hour's till total. Keep the stop date on the rota so a trial doesn't quietly become the opening team's permanent schedule. That's how I'd plan the test, not a result from a café I've studied.