Science and Pseudoscience in Marketing Research

Politicians and scientists often make dramatic claims and hurl accusations at one another regarding how scientific or unscientific this or that is. It’s enough to make our heads swim, as most of us have had little or no formal exposure to science beyond high school. This, disturbingly, is also true of the majority of our opinion leaders and those who make and enforce our laws.

What’s worse, few scientists have had extensive coursework in statistics, the grammar of science in the words of Karl Pearson. This short clip, which I found in the American Statistical Association publication Amstat News, illustrates the exasperation many statisticians feel when interacting with scientists.

So how can we tell real science from pseudoscience? I can give you a succinct answer: I don’t know. Science is hard. Karl Popper’s The Logic of Scientific Discovery and Thomas Kuhn’s The Structure of Scientific Revolutions are tough reads but authoritative reminders that the label “science” can deceive. Furthermore, even well-conducted science can arrive at conclusions that are later found to be badly wrong. Science can reduce uncertainty but may also increase it and complicate matters for high-level decision-makers and as well as us regular folks. For instance, one doctor might recommend a particular diet or exercise regimen while another strongly advises against it.

Perhaps it’s better just to go with our gut, then. Not so fast! Inspired in part by Gerd Gigerenzer’s thinking, as well as my education in statistics and experience as a marketing researcher, I often find it helpful to categorize decisions into three basic kinds: deterministic, intuitive and probabilistic. In Who Cares About Evidence? and AI, Big Data and Decisions I elaborate on what I mean by these terms. In a nutshell, the main distinction is how difficult it is to estimate probabilities. I’ll explain what I mean by that next.

An example of the first type of decision is that I know my gasoline-powered car will not run if there’s no gas in the tank. This is determined by physical laws. Likewise, if I park it in an illegal spot, I’ve parked illegally according to human law. There is no if in either case. At the other extreme, say I wake up in the morning and realize I’ve left my umbrella at work. I haven’t looked at the weather report for days or poked my head outside yet, thus I’m totally in the dark about whether it will rain today or not. I have no pertinent data or information. My gut intuition tells me to take a look at the weather channel and their meteorologist says there’s a 50% chance it’ll rain this morning. Other meteorologists may disagree but they all have data and algorithms they can use to estimate the probability of rain this morning.

Obviously, many decisions are multifaceted and combine two or more of the three kinds. Gut feel – individual or collective – plays at least some role in nearly every decision. Marketing research, however, is seen by some clients (and CFOs!) as most valuable when we are able to provide decision makers with formal quantitative estimates or forecasts of what might happen under various scenarios, i.e., for what I characterize as probabilistic decisions. Marketing Mix Modeling is an example of this, as is Simulated Test Marketing. Marketing research methods that have or claim a solid scientific foundation do not always rely heavily on mathematics, of course, but often do at least implicitly.  

Earlier I’d admitted that I don’t really know how to distinguish real science from pseudoscience. Fortunately, though, there are some guidelines you can avail of. One is to use your research skills and read up on the relevant science. Peer-reviewed publications will provide a more objective perspective than a “white paper” downloaded from a marketing research company website or presentation at an industry conference. Even if the details are over your head you should be able to get the gist of most papers and at least come away with a sense of whether a claim that’s caught your eye is well-supported or controversial. There may also be useful materials about the subject online that are less technical, including lectures by academics for lay audiences. You may also wish to consult a statistician or marketing scientist if you get stuck.

Look for independent replication by parties with no financial interests at stake, and the involvement of a statistician in the research. Bone up on meta-analysis. Also ask for case studies showing how the method has been actually been used in marketing – as a former R&D person for Nielsen and Kantar, I appreciate that intellectual property must be secured, but marketing scientists need to be able to provide evidence that these shiny new tools actually work. Keep an open mind but ask challenging questions. Don’t be afraid of embarrassing yourself by asking dumb questions – contrary to what some say, there are dumb questions…and even scientists ask them. Don’t allow yourself to be intimidated by tech talk.

Some books related to this topic I can recommend include:

• The Halo Effect (Rosenzweig)
• Superforecasting: The Art and Science of Prediction (Tetlock and Gardner)
• Risk Savvy: How to Make Good Decisions (Gigerenzer)
• Thinking, Fast and Slow (Kahneman)
• Critical Thinking for Marketers (Grapentine et al.)
• Theory Construction and Model-Building (Jaccard and Jacoby)
• Experimental and Quasi-Experimental Designs (Shadish et al.)
• Apollo’s Arrow (Orrell)
• The Improbability Principle (Hand)
• Uncertainty: The Soul of Modeling, Probability & Statistics (Briggs)
• Risk Assessment and Decision Analysis (Fenton and Neil)

I’ve listed many others together with some relevant academic journals in my company library. Missing Links provides links to interviews with more than a dozen leading scholars and may also be of interest. You’ll love the MR Realities discussion with Steve Needel entitled Shiny New BS is Still BS, and my earlier post Disrupting BS offers further tips on how to separate the wheat from the baloney.

I hope you’ve found this interesting and useful!

Arrange a Conversation