Quant Essentials: Questionnaire Design

In RW Connect’s new Quant Essentials series, we discuss critical methodological skills in simple, jargon-free language. The first article in the series, What Is Quantitative Research? gives some more background about the series. Our second article was about research design and our third, sampling.

In this article, we mention some of the important considerations you need to bear in mind when you are developing a questionnaire. A very common mistake is to assume that questionnaire design is easy! While it is true in a literal sense that anyone able to type can write a questionnaire, this does not mean all questionnaires are equally useful. Questionnaire quality is quite spotty in marketing research and, in my opinion, this has damaged the reputation of our industry.

Long questionnaires without clear objectives are common and perhaps the Achilles’ heel of marketing research surveys. Lengthy, unfocused questionnaires typically result from confusion about the purpose of the research or too many objectives for one study. Some ABCs Of RFPs offers some tips on how to prepare a research proposal. Hint: It pays to be proactive.

Though we’re not in the entertainment business, boring surveys risk boring answers of little value to decision makers. While gamification may not be necessary, web surveys in particular offer many ways to engage respondents and keep them engaged.

Many surveys have very long and detailed screeners that run up costs, slow down data collection and limit the survey to an overly-narrow group of people. This group may actually account for a very small share of brand or category volume, and there may be no real evidence that people with these characteristics merit special attention.

Check to see which demographics and other profiling data the panel company has already collected on their members. Though the wording the panel company used may not be identical to yours, there may be no reason to ask these questions again, either as screeners or in the background section of the questionnaire.

Reinventing the wheel again and again rather than using standard questions and questionnaire templates that are tried and true is quite common in marketing research. This increases costs and lowers quality, a lose-lose proposition.

Don’t ask everyone everything! Instead, take a modular approach (“chunking”) in which only key questions, or questions that will be analyzed together, are asked of all qualifying respondents. Chunking shortens questionnaire length and is nearly always preferable to data imputation.

Don’t ask questions or provide answer categories that make little sense to ordinary people. Also, avoid asking highly detailed questions about shopping or product usage that few humans could be expected to answer accurately! Put yourself in the shoes of the respondents and ask yourself – honestly – if you could answer these questions.

When designing your questionnaire, think ahead to the final deliverable and how the data will be analyzed. A future article in the series will give you some advice on how to prepare tab plans and interpret data.

Advanced analytics ordinarily works best when the data have been collected with specific objectives and procedures in mind. If multivariate analysis or other specialized analytics is planned, I would urge you to consult a statistician when preparing the proposal, preferably the person who will conduct the modeling. This person should also check the questionnaire before the project is fielded.

Attitudinal measurement is one of the most important tasks of marketing research, yet we often fail at this. Few marketing researchers have a background in psychometrics, and we often brainstorm items first and then see what falls out later in the analysis. This is the reverse of recommended practice, which is to first identify the constructs we wish to study and then to develop the items we’ll use to measure these constructs. Before designing your own scales, I would advise you to check the academic literature and sources such as the Marketing Scales Handbook (Bruner) and the Handbook of Marketing Scales (Bearden et al.) for attitudinal scales that have been professionally validated.

Don’t be wedded to traditional 5-point Agree-Disagree scales. There are many alternatives, especially in web surveys, that may be better suited for certain types of questions or respondent populations. Related to this point, don’t get hung up on Stevens’ framework of ratio, interval, ordinal and nominal scales. It is not universally-accepted and less relevant with today’s statistical methods.

Forgetting about national or ethnic culture and how these can affect shopping, purchase, usage and the way people interact with surveys is another common mistake. I’m astounded how many marketing researchers still think we’re all alike! In multi-country studies, not having questionnaires professionally translated and back translated is another error that you may later regret.

In some projects, it makes sense to ask something important in different ways. For example, brand health and equity indices are sometimes constructed by combining several questions, and attitude batteries normally use more than one item to measure each construct. That said, please be careful to avoid clerical slip-ups and inadvertently ask very similar questions in several places in the questionnaire. I see this all too often and it can confuse and irritate respondents, as well as waste time and budget.

Don’t optimize to mobile when there is no clear reason to do so. Mobile surveys are an important tool in our toolbox but have limitations compared surveys administered via computer. When you need to include people in your sample who have no internet access, there are ways to integrate them without optimizing the entire survey to mobile.

Don’t assume that the panel company will automatically find and remove “bad respondents.” When you can, provide them with guidance regarding straightlining – clicking on the same response nearly every time – and other forms of satisficing or deception based on your company’s experience. Also, please remember that doing things on the cheap doesn’t pay. Cheap sample may wind up costing you a bundle.

Whenever possible, conduct a pilot study for surveys that have never previously been fielded. Even done informally with colleagues acting as respondents can catch serious errors before it’s too late.

I won’t lecture you on ethics, but please read the ICC/ESOMAR International Code on Market, Opinion and Social Research and Data Analytics.

This has only been a snapshot of a very complex topic. Questionnaire design is noteasy! A great number of books and journal articles about survey research and questionnaire design have been published. Asking Questions (Bradburn et al.), The Complete Guide to Writing Questionnaires (Harris) and Questionnaire Design (Brace) are excellent places for marketing researchers to begin. A few others I can recommend are: The Science of Web Surveys (Tourangeau et al.); Internet, Phone, Mail, and Mixed-Mode Surveys (Dillman et al.); and The Psychology of Survey Response (Tourangeau et al.).

Psychometrics: An Introduction (Furr and Bacharach) is an excellent and highly readable introduction to the frequently misunderstood topic of psychometrics. There are also professional organizations such as ESOMAR, the AMA and AAPOR that are excellent sources for information about questionnaire development and survey research generally.

We hope you’ve found this brief overview of questionnaire design interesting and helpful!

Arrange a Conversation