Big Data: Promises, Challenges and Threats

Marketing researchers are wondering what lies ahead for big data. Marketing Scientist Kevin Gray asks Professor Koen Pauwels for his thoughts. 

How is big data currently used in marketing and marketing research?

Big data is currently used for both idea inspiration and idea testing. Social media listening and usage pattern recognition give market researchers ideas about which types of products and services to offer next. However, managers and researchers also use big data to find support for ideas they are already committed to, much like they ask from consultants and market research vendors.

Where does it pay off the most? Conversely, where has it fallen short of expectation?

Big data pays off most if it is part of a decision support system that recognizes human biases and aims to complement these biases with data. As I detailed in my book It’s Not the Size of the Data, It’s How you Use It, many companies are not set up to make, or at least adjust, important decisions based on data. If you have never used ‘small data’ and demonstrated its benefits, how are you ever going to make good use of ‘big data’?

I am very happy with the current attention using data and analysis to make better decisions is receiving. Unfortunately, the Big Data hype also means that many organizations believe they can simply solve their problems by investing in IT and hiring 200 data scientists, without proper management and integration with decision makers. That is when big data falls short of expectations. The whole organization needs to be reconsidered to make best use of big data – which often requires changes to organizational structure and even culture. This is the focus on a new book I am writing with an organizational behavior expert.

What are the main challenges big data faces that concern marketing and marketing research?

Communication bias, communication problems and illusions of control are human challenges that don’t simply disappear with big data. In fact, the volume, variety and velocity of big data may even make these challenges harder. With lots of different data, there is always one data nugget that validates my prior opinion, while another validates my rival’s in the next department, fueling confusion. Moreover, we are hardwired to look for and interpret change, so metrics that move every minute get more of our attention, even if they are not leading key performance indicators.


Elsewhere, you’ve correctly argued for the continued value of surveys, whose slow-moving attitudes are better predictors for brand performance across the 15 industries I’ve analyzed. In our newest publication, I team up with experts in decision making and lean startup methods to show how big data can be used to reduce human biases: by clearly distinguishing idea generation from idea testing, by innovation accounting and by developing build-measure-learn platforms for ongoing experimentation.

Companies such as Microsoft, Facebook, Google and Amazon have gotten very sophisticated in such platforms, but any organization can start its own version – in my experience with excellent results. The alternative is ‘Big Data, Big Bias’:

Looking ahead to the future, what should marketers and marketing researchers keep an eye on with respect to big data?

It’s very tough to predict anything, especially about the future. 🙂 I believe the near future is in the power of algorithms as both decision-making assistants and as customers. Technological advances have made big data ‘virtual assistants’ much more useful to decision makers, but as humans we still have to get used to them and understand their strengths and limitations.

As marketers, we now have to consider how we are going to sell to algorithms, which are increasingly being used by human customers to simplify their decisions. As their details are typically kept secret and are continuously tweaked by platform developers, how will we as marketers get the best insights into the ‘artificial customer decision journey’? These are the kind of questions that keep me up smiling at night.

Thank you, Koen!

Arrange a Conversation 


Article by channel:

Read more articles tagged: Analytics, Featured, Marketing Analytics