The small-sample trap

Ten customer conversations feel like evidence. Statistically, they are a starting hypothesis.

I hear the same sentence in boardrooms in Frankfurt and Shanghai: "We don't need a big survey. I talked to our top customers, and they told me what they will pay." Sometimes it is five interviews. Sometimes it is a sales pilot with three friendly accounts. The decision that follows, a new price list or a new packaging structure, then applies to thousands of customers.

My view is simple. Small samples are not useless, but treating them as proof is a bias, not a judgment call. Fifty years of decision research supports that view. The same research also shows where CEOs have a point: large surveys carry their own distortions. This article lays out both sides, then shows how the answer differs for B2C and B2B companies.

Why small samples mislead smart people

Four well-documented effects push leaders to over-trust a handful of conversations.

1. The law of small numbers. In 1971, Amos Tversky and Daniel Kahneman showed that people expect a small random sample to look just like the population it came from. Their subjects were not laypeople. They were trained research psychologists, and most of them badly overestimated how often a small-sample result would replicate. The authors warned of undue confidence in early trends and in the stability of observed patterns. That is a fair description of a pricing decision built on five interviews.

2. Stories beat statistics when stakes feel personal. A 2020 meta-analysis of 61 studies by Freling and colleagues found a clear pattern. When emotional engagement is high, anecdotes persuade more than statistics. When engagement is low, statistics win. A CEO deciding the company's price level is about as emotionally engaged as decision makers get. The vivid quote from a key customer will outweigh the dull distribution chart.

3. Managers project their own preferences onto customers. A 2021 Journal of Marketing Research study by Herzog and colleagues found that more than 70% of marketers project personal preferences onto consumers. Correcting this bias cut prediction errors by more than half. Interviews selected and run by the executive team rarely escape this effect. Leaders tend to hear what they already believe. Expertise does not rescue them either: when Armstrong (1991) asked academics, marketing practitioners and high-school students to predict the outcomes of 105 published consumer-research hypotheses, no group did better than chance.

4. Experienced intuition is a weak predictor. Ronny Kohavi, who led experimentation at Microsoft, reported that only about a third of tested ideas improved the metric they targeted. Another third had no effect, and the last third made things worse. These ideas came from experienced teams who were confident before the test. Without hard data, the highest-paid person's opinion usually decides.

Put together, the picture is consistent. A small, hand-picked sample combined with a confident leader produces conclusions that feel solid and often are not.

Where the skeptics have a point

A fair argument has to take the other side seriously. CEOs who distrust surveys are not simply being stubborn. There are three good reasons behind their instinct.

Surveys overstate willingness to pay. People answering a hypothetical question do not part with real money. Schmidt and Bijmolt (2019) pooled 77 studies and found that hypothetical willingness to pay exceeds real willingness to pay by 21% on average. The gap grows for higher-value and specialty products. An earlier meta-analysis of 28 valuation studies by Murphy and colleagues (2005) found a median ratio of hypothetical to actual value of 1.35. A badly designed price survey can produce a precise number that is precisely wrong.

Small samples are the right tool for discovery. Griffin and Hauser (1993) showed that 20 to 30 one-on-one interviews uncover roughly 90% of customer needs. If the question is "what do customers value and how do they talk about it?", interviews are efficient and a large survey adds little.

Pilots measure real behavior. A sales pilot asks for a signature, not an opinion. That removes hypothetical bias entirely, which is a real advantage over any survey.

So the error is not "interviews instead of surveys." The error is using a small sample to answer a question that needs a large one. Interviews tell you what matters. Only scale tells you how much it matters, to how many customers, and at what price. A pilot shows real behavior, but only for the few accounts in it, and those are rarely chosen at random.

B2C: scale is cheap, so small samples are hard to justify

In consumer markets, the case against small samples is strongest. A company with 200,000 customers that prices from ten conversations is ignoring data it could easily get.

Who you talk to is not who buys. The consumers a CEO meets are rarely typical. They are loyal heavy users, people in the founder's network, or customers who agreed to an interview. Price sensitivity usually differs most at the edges of the base: occasional buyers, switchers and new customers. Those are exactly the people a small, convenient sample misses.

Large samples are affordable. Online panels make a few hundred qualified respondents a matter of days, not months. For choice-based conjoint, the most widely used software vendor suggests starting at about 300 respondents. That is a modest cost compared with a price change affecting the whole revenue line.

Revealed behavior is available. Many B2C businesses already own the best data: transactions, basket data and conversion rates. Digital channels allow controlled price and packaging tests. Kohavi's results are a warning that confident teams are often wrong, but also a reminder that consumer businesses can test instead of guess.

Mind the survey traps. The 21% hypothetical bias is an average. It is larger for premium and specialty products, which is where many B2C pricing decisions sit. Choice-based designs, realistic shelf or checkout contexts, and calibration against real sales reduce the gap.

B2B: the sample is small, but the bias is bigger than it looks

B2B leaders have a stronger excuse. Many industrial companies have a few hundred customers, and the top 20 may carry half the revenue. A classic consumer survey with 1,000 respondents is often impossible. But small populations do not make small-sample conclusions safe. They create different traps.

One interview is a sample of one person, not one company. Gartner estimates that a complex B2B purchase involves 6 to 10 decision makers, each bringing their own research. A conversation with the friendly plant manager tells you little about what procurement, finance or the CFO will accept. Ten accounts with one contact each is not ten customers' view. It is ten individuals' view.

Key accounts are the least representative customers. The accounts a CEO knows personally are usually the largest and most strategic. They have the most negotiating power and the most custom terms. Their price response says little about the mid-sized and long-tail customers, where much of the margin potential sits.

Pilots are negotiations, not measurements. In B2B, the sales team usually picks the pilot accounts and negotiates the terms. The final price reflects relationships, discounts and the rep's skill as much as customer value. Three pilot wins with hand-picked friendly accounts mostly measure the sales team.

Rigorous methods still work at B2B scale. When respondents are scarce, conjoint designs can trade sample size for more choice tasks per respondent. Most B2B firms also hold a large dataset they rarely use: years of quotes, win-loss records and transaction prices in the CRM and ERP. Hundreds or thousands of real deals are a far larger sample than any interview program.

Matching the evidence to the question

The right method depends on the question, not on a preference for qualitative or quantitative work. This is the ladder I use with clients.

References

1. Armstrong, J. S. (1991). Prediction of consumer behavior by experts and novices. Journal of Consumer Research, 18(2), 251–256.
2. Freling, T. H., Yang, Z., Saini, R., Itani, O. S., & Abualsamh, R. R. (2020). When poignant stories outweigh cold hard facts: A meta-analysis of the anecdotal bias. Organizational Behavior and Human Decision Processes, 160, 51–67.
3. Gartner. Typical buying group for a complex B2B solution: 6 to 10 decision makers. Summarized in Data Axle, The importance of addressing the buying group in B2B sales.
4. Griffin, A., & Hauser, J. R. (1993). The voice of the customer. Marketing Science, 12(1), 1–27.
5. Herzog, W., Hattula, J. D., & Dahl, D. W. (2021). Marketers project their personal preferences onto consumers: Overcoming the threat of egocentric decision making. Journal of Marketing Research, 58(3), 456–475. AMA research summary.
6. Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th ACM SIGKDD Conference.
7. Kohavi, R. (2009). Online controlled experiments: Introduction, learnings, and humbling statistics. KDD tutorial.
8. Murphy, J. J., Allen, P. G., Stevens, T. H., & Weatherhead, D. (2005). A meta-analysis of hypothetical bias in stated preference valuation. Environmental and Resource Economics, 30(3), 313–325.
9. Orme, B. (2019). Sample size issues for conjoint analysis. In Getting Started with Conjoint Analysis. Sawtooth Software. See also Sawtooth's sample size rules of thumb and Quirk's on conjoint sample size.
10. Schmidt, J., & Bijmolt, T. H. A. (2020, online 2019). Accurately measuring willingness to pay for consumer goods: A meta-analysis of the hypothetical bias. Journal of the Academy of Marketing Science, 48, 499–518.
11. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105–110.