Bad Medicine

The progress of medical science is plagued with embarassment. Recently, the intensive blood-sugar lowering arm of the ACCORD trial was halted due to an increase in mortality, primarily cardiovascular. Prior to that, Avandia, a commonly used oral hypoglycemic, was temporarily pulled from the market after it, too, was shown to increase cardiovascular mortality. Tolbutamide, an older older hypoglycemic, was found in 1970 to increase cardiovascular mortality, and one wonders whether it is a more general phenomenon than a particular class effect (tolbutamide belongs to the class “sulfonurea,” Avandia to the class “thiazoladinedione;” the increase in cardiovascular mortality in the ACCORD trial was not attributable to the increased usage of Avandia in that group). In any case, the two more recent studies come as something of a surprise.

Another recent surprise involved Vytorin, a combination of simvastatin, a drug lowers that cholesterol by altering liver metabolism, and ezetimibe, which blocks the absorption of cholesterol in the stomach. The ENHANCE trial showed that the drug does in fact lower LDL (”bad”) cholesterol levels and decrease the size of atherosclerotic plaques, as current theory would predict. But the drug did not lower overall mortality compared to simvastatin alone, and in fact increased mortality from stroke (which theory would not predict).

And a last study, not so much a surprise, but an embarassment nonetheless, showed that systematic bias towards the publishing of positive anti-depressant trials and the squelching negative ones has led to vastly overstated effects for those drugs.

These studies have certain flaws that are starting to be addressed by the clinical trials registry. For instance, to be published in the New England Journal or JAMA, trials must be registered, industry or government sponsorship declared, and outcomes measures defined ex ante to prevent publication bias (anti-depressants) and the post hoc tampering with measures (vytorin, though the attempt was abandoned). These do not prevent bias from creeping in by any means, but they are a start.

But there is another problem with which the clinical trials registry can help, and that is the problem of doctors’ innumeracy. In particular, we are not very good at determining whether a given study is true or false. One problem is overreliance on p-values (for the untrained, the p value represents the frequency with which you would get a particular result even if the hypothesis you are testing is not true): we are trained to use this as a shortcut around actual analysis of a study; p values less than .05 are “true” or “proven” or “significant”, while those above are not. A better method would be to use Bayes’ theorem to interpret clinical tests. But Bayes’ theorem is hard for most clinicians to work with, not because they are stupid, but because of the difficulty in finding an agreeable prior (a subjective estimate of a study’s a priori chances of being true). Most studies test hypotheses generated by a vast body of previous research, and those hypotheses are held with a certain degree of certainty by experimenters, but that information is inaccessible to the non-expert. For a generalist doctor or specialist who hopes to interpret a study outside of his domain of expertise quickly and cheaping, the difficulty of reasonably estimating priors make Bayesian interpretation of the literature unworkable.

Prediction markets, however, are good at quickly aggregating expert opinion (much better than doctors), and can be easily designed to produce probability estimates, as they are in the presidential markets. So a prediction market set up to buy shares of a given trial in the registry outcome being positive or negative with a payoff of $100 if that outcome is met would provide an accurate estimation of prior probability that is easily retrieved by the non-expert. This would eliminate a big barrier preventing most doctors from using Bayes theorem to arrive at a quicker, better estimation of a study’s truth value. So, why isn’t it being done?