Steps in a meta-analysis
There are four basic steps to any good meta-analysis:
We will discuss each of these steps below.
The first step in a meta-analysis is to find all of the pertinent articles on your topic. Important sources of information for a meta-analysis include:
While MEDLINE, the database of the National Library of
Medicine, is a good starting point, it is not the only source of information. MEDLINE
indexes approximately 4100 journals, dating from 1966 to the present. It now offers an
excellent Web-based
interface for free searching
, including a special page for clinical queries
. There are also CD-ROM based
search engines from BRS Colleague, WinSPIRS, and others which offer different search
options, but use essentially the same underlying database. The European version of MEDLINE
is called EMBASE, and is a Dutch/English collaboration. Depending on the topic, it may be
appropriate to search the more specialized National Library of Medicine databases, such as
CancerLit, AIDSLine, and ToxLine.
The Cochrane
Collaboration
Controlled
Trials Register, established in 1993, is a new and important source of studies for a
meta-analysis. Available by subscription
on CD-ROM or Web interface, the
Register includes abstracts of over 160,000 trials. It includes all controlled trials in
the MEDLINE and EMBASE, as well as the results of hand searches by Cochrane Collaboration
volunteers of thousands of journals not indexed by MEDLINE or EMBASE. The Collaboration is
even negotiating with drug companies to get abstracts of the results of unpublished Phase
III drug trials.
Remember Index Medicus? I'm old enough to recall the intimidating row of thick (and I mean THICK) books that were my only way to find medical research articles as a medical student in the 1980's. They can still be useful when it is important to search for articles published before 1966, when MEDLINE and the other electronic databases were established.
Finally, there are other sources of "fugitive literature" that may be important for the author of a meta-analysis (some of which may be found in the Cochrane Controlled Trials Register):
- unpublished studies - you would have to contact the authors themselves
- dissertations - there are national indexes of dissertations at university libraries
- drug company studies - you may have to contact the company directly
- non-indexed studies - remember to search the bibliographies and Cochrane
- pre-MEDLINE (1966) - use Index Medicus
It's important to know that different search strategies can result in different results (Table 4.5, Petitti):
Topic |
Cochrane CTR |
MEDLINE (expert searcher) |
MEDLINE ("amateur" searcher) |
| Neonatal hyperbilirubinemia | 88 |
28 |
17 |
| Intraventricular hemorrhage | 29 |
19 |
11 |
Note: CTR = Controlled Trials Register
If you are thinking about doing a meta-analysis of your own, it is important to enlist the aid of an expert Medline searcher such as a medical librarian. The above table also highlights the importance of using the Cochrane Controlled Trials Register.
Once the author of a meta-analysis has assembled a large number of studies, it is important to select the right ones! There are a variety of possible inclusion (also called eligibility) criteria:
- Whether the study include enough information for analysis (i.e. standard deviation or standard error in addition to point estimate)
- The study design (i.e. controlled trials only vs randomized controlled trials only, especially for studies of therapy)
- The year of study, if technology or typical dosing changes (for example, only include studies since 1984 on dyspepsia if you're interested in helicobacter pylori)
- The dosage used in the study (to assure that an effective dose was used)
- The language of the article - you or a colleague have to be able to read it!
- The minimum sample size - very small studies may be unrepresentative and/or not worth the effort
- The patient age (adults only, > 60 only, etc)
- The study setting (emergency department, outpatient, inpatient)
Once an appropriate group of studies has been identified, the author(s) have to abstract the relevant data from each study. There are many sources of potential error in data abstraction:
- The article may be wrong due to typographical or copyediting errors
- Tables can be misinterpreted
- Errors can occur during you own data entry or abstraction process
A good meta-analysis will take some or all of the following steps to minimize errors:
- Use 2 independent reviewers
- Use a 3rd reviewer or consensus meeting to resolve conflicts
- Train reviewers by practicing with several articles to "calibrate"
- Compare abstract and text to look for inconsistencies
- Use a standard form or database which constrains entries to the expected range
- Report the results of the data abstraction, including the percentage concordance or even a kappa statistic
Bias can also creep into a meta-analysis. For example, the authors may be biased in favor of (or against!) well known researchers. Also, prominent journals may be given greater weight or authority (rightly or wrongly). It is therefore best (although not often done) to have identifiers eliminated from articles.
Finally, part of the data abstraction phase is an assessment of study quality. Chalmers has proposed a fairly complex set of criteria which apply well to randomized controlled trials. Simpler criteria may be sufficient. For example, in a diagnostic meta-analysis, simply assuring a high quality gold standard, independent assessment of reference and study tests, and blinding may be adequate. Too often, the quality assessment is done, then ignored! Ideally, the results of the quality assessment should inform the analysis and interpretation of results.
There are many issues and controversies in the analysis of meta-analytic data. First, let's define some important terms:
Homogeneity and heterogeneity describe the degree of between-study variability in a group of studies. It is probably appropriate to combine the results from a homogenous set of studies, but many would argue that results from heterogeneous studies should not be combined. The Q statistic, interpreted using a chi-square distribution, is often used as a test of homogeneity.
Fixed effects models consider only within-study variability. The assumption is that studies use identical methods, patients, and measurements; that they should produce identical results; and that differences are only due to within-study variation. By using a fixed effects model, the researcher answers the question: "Did the treatment produce benefit on average in the studies at hand?" The Peto and Mantel-Haenszel odds ratios are both based on a fixed effects model.
Random effects models consider both between-study and within-study variability. The assumption is that studies are a random sample from the universe of all possible studies. With a random effects model, the researcher answers the question: "Will the treatment produce benefit ‘on average’?" The DerSimonian Laird statistic is based on a random effects model.
Fixed and random effects models can give very different answers, and you can create examples where either model gives counterintuitive results (see Petitti, page 92). Usually, though, the answers provided by these different modeling assumptions are similar. Differences only arise when studies are not homogenous. In a comparison of 22 meta-analyses, fixed and random effects models gave the same answer in 19 out of 22. In 3 cases, fixed effects models were significant while random effects models were not (Berlin, 1989 in Petitti textbook, pg 94).
When there is significant heterogeneity, the between-study variance becomes much larger than the within, and studies of different sample size receive relatively similar weight. When there is homogeneity, sample size dominates, and both models give similar results. Random effects models are therefore more "conservative" and generate a wider confidence interval. Put another way, a random effects model is less likely to show a significant treatment effect than a fixed effects model.
In general, if the studies are homogenous, the researchers should use a fixed effects model. If the studies are heterogeneous, the researchers (and you, the reader) should first ask why! While it may be appropriate to do a random effects analysis on all of the studies, it may be better to identify an important subgroup difference (i.e. studies using one dose showed significant effect, while lower dose did not) and then do a fixed effects analysis of each and report all of the results.
A term you will encounter in many meta-analyses is "sensitivity analysis". A sensitivity analysis is a way of looking at only certain studies, certain groups of patients, or certain interventions. For example, a meta-analysis of aspirin in prevention of acute MI might first analyze all studies, but then also look separately at only studies of men and studies of women.
The article by Hasselblad is an excellent starting point for budding meta-analysts,
with lots of examples and formulae. Meta-analysis of diagnostic tests is another
area of growing interest - how do you combine sensitivities, specificities, and so on.
For an overview, see the Cochrane
Collaboration Methods Working Group on Systematic Review of Screening and Diagnostic
Tests: Recommended Methods white paper
(how's that for a long title!). However, details of
calculations for homogeneity, fixed effects models, and random effects models are beyond
the scope of this course.
Advanced students and those with a special interest in this topic may wish to review the following sections: