The Free Lunch Trap

“No free lunch” is often used by statisticians and computer scientists to mean that, in general, there is no one solution that will be best in all situations. This is certainly true in marketing research, where one small decision can send even a modest project badly off course.

Think of your last marketing research project and all the decisions you and your team had to make.

There are many “macro” decisions that must be made, most fundamentally, whether a qualitative approach or a quantitative approach would be more suitable. The decision in many cases will be easy but this is not always true. The business objectives, research objectives, budget, timing and many other factors may make the decision a close call.  

If the former is chosen, our work has just begun. For example, should we conduct focus groups or in-depth interviews? Online or face-to-face? Synchronous or asynchronous? What types of respondents should participate? From what cities or regions? How should they be recruited? Perhaps an insights community is the way to go? Then, there are other crucial details such as the discussion guide and who will moderate. No free lunch

Much of the foregoing also applies to quantitative research, such as surveys. Clients and their agencies typically spend considerable time developing the screener and the survey instrument itself. Many questions are not of interest themselves but are included for respondent classification purposes. Demographics are one example. Which ones should we ask, how should we ask them and what answer categories should we include? 

Brand lists are another critical decision. Which competitive brands the client should focus on most is not always evident. What about brand image attributes? Which attitudinal measurements are appropriate? How to measure image and attitudes, e.g. pick-any grids or rating scales of some sort, is just as important as which image and attitudinal statements to include. And, what about banners for cross tabs?


Many competing approaches have been developed for concept tests, product tests, ad pre-testing, pricing studies, brand equity studies and other kinds of research. None is clearly superior to the others in all circumstances. No free lunch. 

There are also methods borrowed from neuroscience, as well as social media data, customer records or other “big-data” that may be relevant to a particular project. One of these may be our central methodological focus in the project but, increasingly, various kinds of data are combined for analytic purposes. What data to integrate how to combine these data is case-by-case. No free lunch.  

Statisticians working in marketing research reading this are probably smacking their lips by now. As I point out in Statistics Is Easy…well, it ain’t. In Vital Statistics You Never Learned…Because They’re Never Taught, eminent statistician Frank Harrell exposes some of the bad habits statisticians and data scientists frequently fall victim to.

Statistics is vastly more than means, standard deviations, bar charts and t-tests. It is a gigantic field, and this list is just a sample of the tried-and-true methods I use. For each of these methods, in turn, there are many options and choices to make. Again, one small decision can have big consequences. No free lunch

In our haste to meet deadlines we may say “Let’s just go with A. It’ll be good enough” but we may be very wrong. This could turn out to be a very bad decision. “A” may work better than alternatives only some of the time even if it is best most often. It may be the worst choice in some situations. Marketing researchers have many decisions to make, and rules of thumb can backfire.  

There is no free lunch.



Article by channel:

Read more articles tagged: Analytics, Featured

Data & Analytics