Evaluating validity
Before we get to searching the literature, we probably need to talk about one or two issues related to therapy. Remember that we are looking for the best available evidence to guide us. In other words we want to find articles that are most likely going to provide us with valid results. The most valid study designs in considering a therapy are randomized clinical trials (RCTs). What elements of RCTs make them more valid than other study designs?
Random Assignment
Are subjects randomly assigned to the different treatment groups? With random assignment, we have the best opportunity to make sure that potentially important factors like age, sex, comorbid conditions, etc. are equally distributed among the treatment groups. It also eliminates another important source of bias: the investigator's enthusiasm for the treatment. For example, if we think that gorillacillin is better than cephakillitall in treating otitis media in children, we could be influenced by this and assign more severe cases or kids with recurrent otitis media to the cephakillitall group and the milder ones to the gorillacillin group. What? You say this would never happen? Look at the surgical literature! It's full of examples of selection bias.
Now, unfortunately, sometimes randomization doesn't work perfectly and our treatment groups may differ slightly. One group might be older, have fewer minority subjects, or have more smokers. The larger the study the less likely this is going to happen. The researcher (and the reader) have to decide if these differences are important and how to interpret the results in light of these differences (I’m sure you were worried about this, but, researchers have a number of ways to adjust for baseline differences). In most papers, Table 1 is where you’ll find a comparison of baseline characteristics of the study groups.
What do you do when no randomized studies exist? You have to resort to other study designs and make the best of things. After all, you still have to make a decision!
Blinding
Do the subjects, researchers, and data analyzers know which treatment was given to a particular individual? This can be done in other study designs and isn't unique to RCTs, but it is a crucial element. In our otitis media example, you could imagine that if you had a personal stake in a particular treatment, if you knew which treatment was given, your judgment of improvement might be influenced. With some outcomes, blinding is not very critical. Can you think of some examples? Write them down on a separate piece of paper.
In a clinical trial, we compare the effects of two or more treatments. Another aspect of blinding is the nature of the comparison treatment. When a new drug is evaluated, we usually have a placebo that is identical to the real drug except it lacks the active ingredient. Once you have an effective therapy, it may be considered unethical to use a placebo in subsequent studies. In this instance, a researcher will compare a new drug to the old one. The two drugs should again look identical. Sometimes you have a situation where blinding is tough. Can you think of an example?
Some solutions to these dilemmas include the use of objective and reproducible outcome measures (like death, cholesterol levels, etc.) and having a separate data collector who is blind to the treatment group. In other words, the members of the research team who do the outcomes assessments should be blind to group assignment, thereby reducing systematic bias.
Follow up
Were all the study subjects accounted for at the end of the study? Again, this is not a unique issue for RCTs. You should look for important differences in drop out rates. Additionally, you should see how the author handles them. In the most conservative approach, you would assign the worst outcome to the dropouts from the intervention group and the best outcome to the dropouts from the control group. This way if the experimental treatment is still beneficial, it is in spite of these potentially bad outcomes. If the dropout rate is similar between intervention and treatment groups, this is somewhat reassuring.
Intention to Treat (also known as "analyzed as randomized")
Were the patients analyzed in the groups to which they were originally assigned? This addresses what happens to subjects in a study. Some subjects might drop out (see above), have a change of therapy, move out of town, get mad and leave the study, or even die. The "drop outs" are dealt with above, but what about the "crossovers"? If we have a study comparing medical therapy and surgery, what happens if a subject starts with medication and at a later point in the study gets surgery? To minimize the possibility of bias in favor of either treatment, researchers will analyze subjects based upon their original treatment assignment regardless of what happens afterwards. I know, this sounds crazy, but lets look at the ramifications of this. Let's say that the truth is that drug A lowers standing stool velocity and drug B has no effect. As researchers, we don't know this yet (why else would we do the study?), so we set up a randomized double blinded study comparing drug A and drug B. A perfect study will show the following:
What happens if a number of subjects receiving drug A stop taking it because of side effects, and instead begin taking drug B? At the end of the study, they will have higher standing stool velocities (because, remember, drug B is ineffective). In the "intention to treat" approach, it would look like drug A is less effective than it really is:
OK, so now what would happen if patients discontinued drug B and started taking drug A? You got it, it would look like drug B has some effect on standing stool velocity (when it really doesn't):
One last possibility to consider? You guessed it, what happens if there is some crossover between both groups? It would look like drug A is less effective than it really is and drug B has some effect when it really has none. Depending on how many cross over, it is possible that
The intention to treat approach is very conservative. If you still see a difference, the result is more robust.