Is evidence-based policymaking in crisis?

A thought-provoking article from The Long+Short, make some interesting point on this topic:

…the problem seems to be caused by the use of the treatment-response model, in a context where the choice of how to analyse the data is made after collecting it. As Gelman details, if this is what you do, then it is very easy to put your finger on the scales – even without any bad faith and ignoring institutional pressure to produce statistically significant results, it is very hard for a researcher to avoid concluding that the ‘best’ way to analyse a collection of facts is the way which seems to give them a logical structure.

This is not what happens in pharmaceutical research. In drug tests, all details of methodology have to be filed and registered before the trial begins. It is a convention which has been adopted over the years precisely to avoid this kind of bias. Now here’s the bit that you’re not going to like. How expensive and time-consuming is it to get a new drug to market? How many initial ideas have to be generated in order to get a single robust result that can be confidently expected to perform better than a placebo without unacceptable side effects? This analogy is taking us in a pretty scary direction for a philosophy of policy-making that was meant to provide a quick and easy way to find out what works. So the real ‘reproducibility crisis’ for evidence-based policy making would be: if you’re serious about basing policy on evidence, how much are you prepared to spend on research, and how long are you prepared to wait for the answers?

Producing evidence can be quick and easy; producing high-quality, rigorous evidence, however is often time-consuming and expensive. The alternative–making policy based on no evidence–however, is far worse.