Surveys Worth Running: Design, Sampling and Limits

Surveys are the most requested and least respected research instrument in Indian consumer brands. Someone wants a number, a form goes out, a deck appears with percentages on it, and nobody is quite sure what to do with any of it.

A survey is a good instrument for a narrow job. Knowing the boundaries of that job is most of the skill.

When a survey is the right instrument, and when it is not

Surveys measure claimed attitudes and recalled behaviour, at scale, in a structured way. That is all. They are strong at sizing something you already understand, comparing options you can already articulate, and tracking a stable measure across time.

They are weak at discovery. A survey can only return the answers you thought to offer, so it cannot find a problem you have not imagined. If you cannot write plausible answer options, you are not ready to run a survey and you should be running conversations instead.

They are also the wrong tool when the answer already sits in your data. Repeat rate, category mix, discount sensitivity and time between orders live in your order table. Asking people to recall them adds noise and removes precision. The rule is simple. Discover with interviews and open text. Size with surveys. Verify with behaviour.

Your own customer base is a biased sampling frame

Almost every brand survey is sent to a list of existing customers. That list has already been filtered several times before you send anything.

It excludes people who considered you and bought a competitor. It excludes people who left quietly and unsubscribed. On marketplaces and quick commerce you usually have no contact details at all, so an entire channel of buyers is invisible to the instrument. And it over represents people who bought repeatedly, because they have had more chances to end up on the list.

Then response bias applies on top. The strongly satisfied and the strongly annoyed answer. The indifferent middle does not. Two filters stacked in the same direction produce a result that describes your reachable, responsive, surviving customers and nothing wider.

That is not a reason to skip the survey. It is a reason to write the frame in the first line of the report. “Among customers who ordered on our own site in the last ninety days and opened this message” is an honest header. “Indian consumers say” is not.

Wording that writes the answer for you

Most bad survey results are manufactured at the question, not at the analysis.

Loaded stems are the common one. “How important is sustainable packaging to you” has already told the respondent what the good answer is. Ask instead what they considered when they last chose between two options, or force packaging to compete against price and convenience in a ranked tradeoff.

Double barrelled questions ask two things and accept one answer. “Was delivery fast and the packaging intact” cannot be interpreted either way.

Acquiescence bias is worth naming specifically for Indian samples, where agreement with a polite stem tends to run high. Balance it. Offer a symmetric set of options, include a genuine reject option, and avoid stems that assume the behaviour: “how often do you use” quietly rules out never.

Order effects are real. Randomise the sequence of options wherever the list is long, otherwise the first two items collect a share they have not earned.

Scale design

Pick one scale family and use it throughout. Mixing a five point scale with a ten point scale in the same instrument makes the results incomparable and confuses respondents halfway through.

Label every point rather than only the endpoints, because unlabelled middles get interpreted differently by every respondent. Decide deliberately whether you want a midpoint. A midpoint is honest when neutrality is a real position and lazy when you are avoiding a decision.

The bigger problem with rating scales is that everything scores well. If a respondent can mark twelve attributes as important, they will. Use tradeoff formats where the decision matters: force a ranking, split a fixed budget of points across attributes, or present pairs and make them choose. Constrained formats produce usable hierarchies. Unconstrained ratings produce flat charts.

Response rates and the sample you actually need

Precision comes from the absolute number of completed responses, not the percentage who replied. A low rate does not by itself widen your error bars.

What a low rate does signal is nonresponse bias, and that risk grows the further the rate falls. So treat the rate as a warning light rather than a scoreboard. The levers that move it are length, channel and timing. Every additional minute of survey costs completions, in app and messaging prompts outperform email in India by a wide margin, and asking too soon after purchase measures the delivery experience rather than the product.

The real constraint is almost never the headline sample. It is the subgroup. The moment you cut by city tier, by channel and by pack size, each cell holds a fraction of the total and the confidence interval on each cell is wide enough to swallow the difference you want to report. Decide the cuts before fielding and size the sample for the smallest cell you intend to quote.

Reading results with the confidence they deserve

Three habits protect you. Refuse to report a difference you cannot separate from noise, and be strictest about this on the small cells that are always the most interesting. Keep the instrument identical across waves, because a survey repeated word for word has a constant bias and therefore a trustworthy trend line, while a reworded survey has neither. And decide in advance what result would change the plan, so the analysis ends in an action rather than a discussion.

The limit you have to say out loud

A survey tells you what people say. It does not tell you what they do. The gap between the two is widest exactly where the money is: price, claims, and anything socially flattering to endorse.

So treat a strong survey result as a hypothesis with a size attached, then let the market settle it. Run the price on a live listing. Put the claim on a real ad and watch the click and the conversion. Offer the pack in a limited set of pincodes before committing the run. The survey narrows what you test. The test decides.