Evaluating the validity of a meta-analysis

In the previous section, we listed key criteria for the evaluation of a meta-analysis include (Note:  these are based on the worksheet developed by Andrew Oxman and the McMaster's Group.  They have other excellent materials about overviews on their Web site).  Let's talk a little more about each one, then work through a sample study.

1. Did the authors ask a focused clinical question?

Better meta-analyses don't try to do it all.  It's a lot like developing a question for your own research;  consider the following questions as examples:

• Bad question:  "What is the best way to treat bronchitis?"
• Good question:  "Do antibiotics reduce symptoms in otherwise healthy adults with acute bronchitis?"

The second question is more focused regarding treatment (antibiotics), patient population (otherwise healthy adults), and diagnosis (acute bronchitis) than the first.

2. Were the criteria used to select articles for inclusion appropriate?

Inclusion criteria can include criteria involving the treatment, patient population, study design, and diagnosis.  Possible problems include:

• poor description of treatment (did they lump together low and high dosages of a drug, some of which may be adequate and others may not?)
• studies of poor quality included along with those of higher quality?
• vague or variable description of condition being treated (for example, in studies of acute MI there are many ways to define MI)

3. Is it unlikely that important, relevant studies were missed?

Remember, just using MEDLINE may not be adequate, and a meta-analysis may be more convincing if it includes unpublished trials and studies from the Cochrane Controlled Trials Register which don't appear in MEDLINE.

4. Was the validity of the included studies appraised (study quality)?

Assessment of study quality means that the authors carefully read each study, and rated it on a number of quality measures.  Important aspects of quality for an RCT include:

• was it blinded? 
• was it placebo-controlled? 
• was it randomized?
• was follow-up complete?

In addition, do the authors of the meta-analysis use this information?  For example, do they drop studies which don't meet a certain minimum level of quality, or do they calculate results for high-quality studies separately from those for all studies or low-quality studies?

5. Were assessments of studies reproducible (data abstraction)?

There should be a description of how data were abstracted.   Ideally, at least two people should abstract each study, then compare results and have a formal mechanism for resolving conflicts.

6. Were the results similar from study to study (homogeneity)?

When study results are homogenous, using a Q statistic or chi-square test, it is much more likely that the meta-analysis reflects "the truth".  When studies are heterogeneous (i.e. some find benefit, some do not) the authors should be very cautious about combining the results, and if they do should use a random effects model.

Now, let's look at the meta-analysis we asked you to pull earlier with these questions in mind.  This article was selected because it is a topic of interest to all generalist physicians, and is fairly straightforward. 

1. Did the authors ask a focused clinical question?

        Yes        No

2. Were the criteria used to select articles for inclusion appropriate?

        Yes        No

3. Is it unlikely that important, relevant studies were missed?

        Yes        No

4. Was the validity of the included studies appraised (study quality)?

        Yes        No

5. Were assessments of studies reproducible (data abstraction)?

        Yes        No       

6. Were the results similar from study to study (homogeneity)?

        Yes        No

That's it!  You've evaluated the major elements of validity for a meta-analysis.