Evaluating the relevance and validity of a decision analysis

Most of the time, you won’t be doing your own decision analyses.  Instead, you’ll be reading those of others. The same principles for the evaluation of relevance apply to decision analyses as to any other kind of study:

• Did they address a common or important problem, one that you are likely to encounter in practice?
• Did they evaluate real clinical outcomes that patients would care about (i.e. patient-oriented evidence) or just intermediate disease measures?
• Would the results change your practice if valid?

There are some other issues that are specific to decision analyses:

Did the authors consider all possible diagnostic and/or treatment strategies?

Sometimes, a decision analysis (DA) that is otherwise sound fails to consider an important strategy. This may be because of the author's specialty - a gastroenterologist is less likely to consider watchful waiting for a dyspeptic patient, because their family physician has probably already tried it.

Does the model capture the clinical process in adequate detail?

Too simple a model probably fails to consider some of the benefits and harms of an intervention, while too detailed a model is impractical. It is important, though, that key harms and benefits be included in the model.

Was the time horizon adequate?

A decision analysis of diabetes complications may have to be "run" for 10 or 20 years or more, while a study of UTI can have a time horizon of only a month.

What perspective did the authors use?

Taking the payer's perspective is easier for the researcher, but less "patient-oriented" than either the patient or societal perspective. Since payers often fund these kinds of analyses, you have to watch for this.

How were the probabilities obtained?

A systematic review of the literature is preferable as a source for the probabilities in the model, not just one or two selected studies.

Are the utilities reasonable?

First, do they have face validity? There are several ways to determine the utility of a health state, and they have their advantages and disadvantages. There are also studies that have reported utilities for a variety of health states, but it is important that the values apply to your population.

Were cost data accurate?

These can be very difficult to find, and often depend on the perspective. The patient's cost for a prescription may be $10, but the payer's is $80. It is also important to consider both direct costs (drugs, tests) and indirect costs (time lost from work).