Slicing Idea Space

The other day I became aware of a publication attacking the group Friends of Science in Medicine – of which friends I am one – published in the minor journal ‘Creative Approaches to Research’, taking Friends of Science in Medicine to task for unfairly attacking complementary medicine. As the publication originated in my own institution, I felt an obligation to respond. So I wrote a vituperative response.

However, the process of writing the response has crystallised something in my mind which I think is valuable. This something is the distinction between evidence-based and science-based policy. The main coherent argument of the published attack on Friends of Science in Medicine is that complementary medicine is evidence-based, so it is unfair to deny it a place with other evidence-based treatments at the public trough. And this argument is valid. Complementary medicine is evidence-based. There is a vast literature of studies giving positive evidence for complementary medicine. It is published in professional-looking journals. It is peer-reviewed, for what that is worth. The evidence is apparently there that all manner of weird treatments work. But if you look more carefully at this body of literature, it looks much less impressive. The design of the studies is flawed. Controls are missing. Data is cherry-picked to support a preconceived conclusion. Alternative hypotheses that could explain the results observed are not considered. The overwhelming bulk of the great mass of evidence is just not very good evidence. 

And the same is true for many other things that are not complementary medicine: any observation that can be selected from the overwhelming deluge of data that eternally gushes out at us is evidence. 

This picture is evidence for the Loch Ness monster. 

 

This graph is evidence for anthropogenic global warming.

What we need to do is test evidence. The process of testing evidence that has been proven to work is called "science". You might remember my amendment of a Richard Dawkins quote so that it made sense:

We see some evidence, and create a model that explains the evidence. A hypothesis that the evidence means, if we do action X, we will see result Y. We do action X, and see if we see result Y. We don’t do this just once. We think about what the implications of our model are, and what new things it predicts: if it predicts we will see result Y1 if we do never-before-attempted action X1, we try action X1, and see if our model has correctly predicted this new outcome. As described, this procedure seems flimsy, because obviously an infinite number of explanations are possible for anything we observed. So we apply one simple additional requirement: our model has to be consistent with all the other models that have been tested a lot, and not shown to fail yet. Our overall model of the universe has to be consistent. This process of testing the evidence is science, and policy that is based on models describing the evidence that have passed all of these tests is science-based policy.

Clearly, evidence-based policy is better than policy based on the things on the edges of the figure below. And clearly evidence-based will shade off into a penumbra of flimsier and flimsier evidence. If we don’t have science to base our decisions on, we should base them on whatever evidence is available. But, if we have science-based understanding of a phenomenon, we should preferably base our actions on the science, not simply the evidence per se. And if we are spending other people’s money, we should spend it in the most effective way we can. Which means a science-based way where we can, instead of an evidence-based way. Is that clear? 

To recapitulate:

Science-based policy is based on a model that explains the evidence;

It is based on a model that is testable;

It is based on a model that has been tested, and not found to fail;

It is based on a model that does not contradict all the models for different phenomenon that have been tested, and not found to fail.

When we are distributing resources, we should wherever we can distribute them in a science-based way. 

I reckon this distinction between science-based and evidence-based anything is a distinction that is underutilised and valuable, and we should make it, loudly, whenever it relevant.