Save time and money securing the right people for maximum impact in your business.
Meta-analysis is a relatively new tool marketing researchers should consider adding to their repertoire. It is a set of procedures for statistically synthesizing the results of several primary studies. It is research on research and is used when we want examine the body of evidence about a topic, rather than relying on the results of a single study. Even in rigorous disciplines such as medicine, studies often do not agree in their central conclusions, and meta-analysis provides a systematic way to examine a collection of results, typically effect sizes, across multiple primary studies. It also sheds light on factors potentially underlying dissimilarities in findings and helps us make sense of the patterns of effects. It began to take root in the 90’s and is applied in medicine, ecology, psychology, education, business and other fields as well as for the evaluation of government programs. It is still quite new to marketing research and arguably under-utilized.
Meta-analysis is increasingly employed in place of the traditional narrative literature review, which has been criticized by its detractors for being subjective, unsystematic and as “arithmetic with words.” Meta-analysis is appropriate provided the primary studies are not too disparate. Since real effects cannot be reliably established through one study alone, meta-analysis permits us to better estimate the true size of effects, for instance, the impact of medical treatments or training programs. It is also better suited when we wish to generalize to a broad population.
Effect sizes are typically measured by means, proportions, risk ratios, odds ratios or correlations. Meta-analysis is not simply a matter of adding up effect sizes in each study and dividing by the number of studies we are examining, however. Nor is it just counting the number of significant differences favoring one hypothesis or another and choosing a winner. Instead, it uses statistical techniques to combine the results of multiple studies. Ideally, data from individual subjects (individual participant data) in each study would be used but more typically this is not obtainable and aggregate data must be analyzed instead.
In meta-analysis the individual studies are weighted by precision, which is derived from the standard errors of the individual studies. The term inverse variance is often used to describe this weighting mechanism, which is primarily a function of sample size. Larger studies, i.e., those with larger sample sizes, will be more influential in meta-analysis. It is not necessary that all primary studies in a meta-analysis measure effect in the same way and formulas exist to convert among effect sizes based on dissimilar metrics. There are other methods besides the inverse variance scheme and for details readers may wish to refer to standard references such as Introduction to Meta-Analysis (Borenstein et al.) and Modern Epidemiology (Rothman et al.) and Methods of Meta-Analysis (Schmidt and Hunter). Forest plots
Results of meta-analyses are normally summarized with forest plots, a handy graphical device that efficiently conveys the highlights of the analysis. (Forest stands for “forest of lines” and we are interested in both the trees and the forest.) Below is a simple example, taken from Wikipedia, based on five studies. In practice, sometimes dozens or even hundreds of primary studies are used. In the plot, the size of a square is proportional to the corresponding study’s weight in the meta-analysis and the horizontal line represents the confidence interval of the study’s odds ratio (OR), an effect measure that is standard output from logistic regression and equals 1.0 when there is no effect. The diamond at the bottom of the plot reports an overall odds ratio of 2.2, with the real effect size estimated to lie between 1.9 and 2.4.
Example Forest Plot
Variations of meta-analysis
Fixed effect versus random effectsStatisticians will be familiar with fixed-effects and random-effects models. In meta-analysis fixed-effect (singular form) has a different meaning and Borenstein et al. suggest “common-effect” might be a more suitable term. The fixed-effect model assumes the true effect is the same in all studies and that differences we observe among studies are the result of sampling error, while random-effects models allow that the true effect may vary from study to study.
When fixed effect models are used studies with larger sample sizes will receive relatively more weight and smaller studies relatively less weight, though this by itself should not determine whether a fixed-effect or random effects model is chosen. Confidence intervals under random effects models will be wider because both within and between (across studies) sources of variation are taken into account. Random-effects meta-analysis tends to be preferred because the primary studies as a rule are heterogeneous in participant (e.g., respondent) characteristics, in the implementation of treatments themselves or in other important operational aspects. It is generally, though not always, more realistic to assume that the true effect could vary from one study to the next. A new method, the quality-effects model, has been advocated as an alternative to random-effects models.
Meta-regression Regression is widely-used in primary research and also has an important place in meta-analysis. Rather than subjects in a primary study, however, observations in the meta-regression data file are the results of the primary studies and both the dependent and independent variable(s) are at the study-level, not the subject (e.g., respondent) level. As with standard regression, independent variables may be continuous, categorical or a combination of the two. Fixed and random effects models are both possible, as are variations of regression (e.g., polynomial). Analogous to regression in primary studies, the number of studies should be sufficiently large for the analysis to be meaningful and a minimum of ten studies per independent variable is one rule of thumb, though this is not set in stone.
Psychometric meta-analysisAny study has methodological faults that affect its results, sometimes substantially. Psychometric meta-analysis, also known as validity generalization or Hunter-Schmidt meta-analysis, attempts to adjust effect estimates for methodological limitations of the studies and perform a meta-analysis of the corrected results. The quality-effects model shares some of the aims of these methods.
Psychometric meta-analysis uses procedures based on psychometrics theory to make corrections for attenuation of effect size due to measurement error and other flaws. Effects will generally be larger and their variation smaller after corrections have been made, and the distribution of true effects is emphasized rather than the overall mean effect in this approach. Another distinction is that raw correlations are favored as the metric of effect and sample size is preferred for weighting.
Things to think about
Many factors must be taken into account when selecting primary studies and obvious ones include, naturally, the topic, the sample composition, how the sample was drawn and from what population, and when and where the research was conducted. Less obvious perhaps are details of the study design, e.g., whether it was experimental or observational. (See Pros and Pitfalls of Observational Research https://www.linkedin.com/pulse/article/pros-pitfalls-observational-research-kevin-gray). Other considerations include which potential confounders were accounted for, how they were measured, the analytic methods employed and, of course, whether sufficient information regarding the study has been published or can be obtained. To a degree, heterogeneity is desirable since we will have more confidence that the findings of the meta-analysis are generalizable to real-world circumstances, though this is not to say “anything goes”.
Publication bias is another concern. This is a lengthy topic but essentially refers to the fact that many studies go unpublished, not because of poor quality, but because no statistically significant effects were detected. Effects which achieve statistical significance will, in general, be larger than those that have not. Publication bias reflects editorial decisions but researchers must first decide on their own whether or not to submit results for publication. Statistically significant findings are also more likely to be published quickly, to be published more than once, to be cited frequently and to be published in English and thus easier to locate. Meta-analysis cannot not magically make these potential biases disappear, but statistical methods have been developed (e.g., funnel plots) to help us assess whether they are a serious problem in the set of studies we have tentatively selected. In short, judgment plays a large part in study selection, but it must be remembered that science is not merely plugging numbers into formulas in mechanical fashion. Whatever approach we take and regardless of our topic, it’s Garbage In Garbage Out…
Criticisms of meta-analysis
Despite having advantages over alternatives, meta-analysis remains somewhat contentious. There are several reasons for this. Because the combined sample sizes can be very large, very small effects can be depicted as “significant” even when effect sizes in all the individual studies are not statistically different from zero. Larger studies, moreover, are often operationally more challenging and can actually be more error-prone than ones that are smaller-scale and more carefully supervised, but larger studies will have more weight in meta-analysis because of their bigger sample size. Also, as noted, inclusion criteria for primary studies will have a major impact on the results and another critique is that meta-analysis can make it easier for the investigator to “cook” the results by cherry-picking “ingredients” that support a favored hypothesis. It can be used for advocacy instead of science, in other words.
Proponents of meta-analysis often respond to criticisms such as these by admitting that many of them do indeed have merit, but that they apply even more to traditional narrative literature reviews or simply counting significant differences (“vote counting”). In meta-analysis, these dangers are more transparent, it is claimed. My own view is that, while properly conducted meta-analyses have important benefits, we do need to be vigilant against potential malpractice, perhaps more so than in the case of narrative literature reviews which, in my experience, tend to be circumspect in their conclusions. Its place in marketing research
To this point meta-analysis has been little-used in marketing research and some might wonder “Who cares, if I’m probably never going to use it? I’m not a doctor.” My first response would be that, as researchers, we need to be attentive to what researchers in other fields are doing. After all, nearly all our methods and research protocols have been borrowed from other disciplines! At a more day-to-day level, marketing researchers often undertake desk research as part of a project, especially at the design phase, and when doing so may occasionally encounter meta-analyses. It is therefore helpful to have some background on what it is and its strengths and limitations. Furthermore, we can conduct it ourselves when designing new research. Lastly, we can use meta-analysis for R&D on new methodologies, for examining questionnaire design issues, in ROI analysis, in pricing research and for testing product claims, to name just a few examples of how it can be applied in our industry.
Outside the nutshell
The foregoing is meta-analysis in a tiny nutshell. For those who’d like to study it in depth there are numerous on-line sources and papers in addition to textbooks. Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Shadish et al.) is a widely-cited textbook that offers a comprehensive perspective on research, including meta-analysis, as well as detailed guidelines on research design and how it affects interpretation. The Cochrane Collaboration (http://www.cochrane.org/cochrane-reviews) provides a very large repository of research on health care and health policy and is a good source for examples of meta-analysis. The Campbell Collaboration (http://www.campbellcollaboration.org/) is a similar organization for social, behavioral and educational research.
Kevin Gray is president of Cannon Gray, a marketing science and analytics consultancy. A version of this article first appeared in the May 12, 2014, edition of Quirk’s e-newsletter as “Meta-analysis offers research on research for MR”: http://www.quirks.com/articles/2014/20140525-5.aspx
Article by channel:
Everything you need to know about Digital Transformation
The best articles, news and events direct to your inbox
Read more articles tagged: Featured
A better way to hire for digital transformation. Quality candidates expertly matched to your business guaranteed to save you time and money.