Random Thoughts on Marketing Research, Statistics and Data Science (July 11, 2018)

My opinions. Nothing more. Different AI will find different patterns in the same data, as will the same AI using similar but different data. Data cleaning and preparation can also have a substantial impact on what it finds.

The suggestion that we can just turn AI loose and it will find the questions we need to ask and the answers to these questions is science fiction.

More sensible claims regarding automation often ignore that there are many possible ways to template even simple types of research, such as concept screening. Not to mention that how the data are analyzed and interpreted can have important practical implications for decision-makers, irrespective of questionnaire length.

The reality is that standardized methodologies can be much harder to sell and service than many of us may assume. They do not necessarily simplify matters for clients.

“Our new [whatever] increases click thru by as much as three times!”

While claims like this may sound impressive on the surface, they raise more questions than they answer.

For example, “as much as…” doesn’t tell us anything about the distribution except the maximum value. Randomly-generated numbers can fall into a very wide range! It also doesn’t tell us what is being compared. Three times…compared with what? Doing nothing?

Three clicks per 1,000 is three times one click per 1,000 but still not a heckuva lot. And, what about base sizes? If the segment going targeted, for example, is tiny, lots of clicks won’t mean many sales and narrowing our reach could seriously hurt the health of our brand, even medium-term.

Moreover, how do we know we haven’t just subsidized purchases that were going to happen anyway?

Claims such as these also beg the “question over what period?” How long was this increase sustained? Was the as-much-as in just a single day or hour…?

Finally, the Big Question. Did this new [whatever] pay off? Spending lots of money to increase sales isn’t always smart.

There is much confusion about the meaning of machine learning and AI, and how statistics is connected to all this. In fact, they’re all very closely related.

One way to look at this is the ways they’re used. For example, if the purpose is image analysis, speech recognition, text mining or robot navigation, statistical methods would not be the usual first choice.

When used for predictive analytics, statistical methods are usually highly competitive when used correctly (often they are not).

When it comes to explanation – understanding why consumers behave the way they do – stats will normally be the best choice.

Note that there is no intrinsic tradeoff between predictive accuracy and interpretability.

Marketing researchers typically look at a large number of cross tabulations. Marketing scientists also inspect tabulations as part of exploratory data analysis (EDA) and when cleaning and preparing data for more detailed analyses and modeling.

These cross tabs are based on totals and, therefore, can be misleading because they do not take other important variables into account. Reasons for using a product by gender may vary by age, for example.

Running masses of multi-way tables indiscriminately is not a good idea, but multi-way tables and multivariate analysis (MVA) can be very illuminating when have reason to believe that many variables need to be considered jointly. Small sample sizes can be a concern, but consumer surveys nowadays frequently have thousands of respondents.

For what they’re worth, my predictions for marketing research, were published in GreenBook in December 2017, along with many others that are more coherent and sensible. Thanks again to Lenny Murphy for inviting me to offer mine:

“I lean towards long-term predictions, because people are more forgiving when you’re badly wrong, as I hope I am here. Marketing research has historically focused on data collection, and analysis has typically been simple. This applies to qualitative as well. In some ways, MR was ahead of its time, though, in the sense that the ability of decision makers to use data for decision making has lagged the capacity to collect it. There have been notable exceptions, of course, manufacturing being one example.

Now the data industry is much larger and MR is increasingly a small part of it. One can argue that MR is being absorbed into data science, qual included. Many of my contacts working in marketing research related areas are programmers and software developers. Many are adept at selling, but know little about marketing or research. They usually do not refer to themselves as marketing researchers. Ironically, I advocated that MR move into what’s now called data science when data warehousing and ERP were the buzz, but our industry had other priorities at the time. By 2030, there probably will still be people calling themselves marketing researchers but far fewer than now, I suspect.

Whatever we call ourselves, the key to survival as a marketing researcher will be going beyond mechanical data collection, simple analysis and interpretation, and plugging the gaps increasingly sophisticated AI and machine learning – and human data scientists – leave behind. Those able to design primary quantitative research who have a good grasp of statistics, as well as marketing research and business generally, will be at an advantage, as will top-notch qualitative researchers.”

I’ve known many entrepreneurs operating in all walks of life and have worked with some of them. I cannot adjust for survivorship bias but it is my impression that the key ingredients to success are energy and optimism, a robust physique and good health, sales and networking skills, and being in the right place at the right time (i.e., luck). Being able to cut deals and charm bank lending officers and VC folks are also important.

Raw intelligence, creativity, adaptability, and a good offer that is truly unique probably count for far less than I’d once assumed. Some entrepreneurs I’ve known are one-pitch pitchers and actually quite narrowly-focused and rigid.

Arrange a Conversation