There is ongoing discussion about the extent to which scientific research is reproducible. The October 2017 issue of the Economic Journal includes three studies illustrating that cross-fertilisation across economics and other disciplines may have significant benefits for science and economics. The many different issues that arise in assessing economic research are constantly under debate in the profession. The three studies illustrate some of the views that are currently being taken on these issues.
The first study explores the value of economic analysis of the incentives and outcomes in clinical trials. The second and third examine whether and how economics can benefit from meta-research methods.
Persuasion bias in science: can economics help?
The gold standard methodology of ''randomised controlled trials'' for evaluating medical treatments does not completely eliminate the scope for manipulation by scientists or their funders. But according to the first study – by Alfredo Di Tillio, Marco Ottaviani and Peter Norman Sørensen – this does not necessarily mean that researchers are inevitably trapped in a ''credibility crisis'', with doctors not prescribing certain drugs even when in fact they should.
Their analysis shows that a manipulated trial does not necessarily leave doctors with more uncertainty than a standard trial. By its very nature, manipulation requires information on the side of the researcher – and more information in the system can ultimately result in doctors prescribing effective drugs more often and more confidently.
Replication in experimental economics
If the empirical findings of economic research are to become more reproducible and if knowledge is to accumulate, there needs to be far more systematic use of replication and meta-analysis – statistical assessment of results garnered from many similar studies.
That is the conclusion of the second study – by Zacharias Maniadis, Fabio Tufano and John List. Their research contributes to the debate on the alleged ''credibility crisis'' in science by exploring the prevalence of replication attempts in research in experimental economics.
They note that economists seem to have an aversion to declaring a given study as a replication, in part because of professional incentives, which deem that replication is not as prestigious as original work. Institutional changes may be needed to ensure that replication research is more widely conducted and carefully documented, the researchers conclude.
The power of bias in economics research
The third study – by John Ioannidis, TD Stanley and Hristos Doucouliagos – investigates two dimensions of the credibility of empirical economics research: The ability to reject false hypotheses (the power of a statistical test) and the ability to estimate the correct value of a parameter of interest (avoiding statistical bias).
The authors survey 159 empirical economics papers that draw on 64,076 estimates of economic parameters reported in more than 6,700 empirical studies. The authors propose to use meta-analytical approaches to evaluate the statistical power of empirical studies. Such meta-analysis is commonly used in fields where the exact same phenomenon is studied in many different experiments, a feature that is rarely true in empirical economic research due to differences in the economic environment across estimates. Nonetheless, using such meta-analysis, the authors suggest that statistical power may be low in empirical economics.
The authors make a plea for giving the difficult notion of statistical power a more prominent place in empirical analyses. Finally, they stress that a culture of replication should be fostered, which can, among other measures, be stimulated by data and code sharing policies and by promotion and tenure committees valuing replication efforts.
The three studies published in the October 2017 issue of the Economic Journal are: ''Persuasion Bias in Science: Can Economics Help?'' by Alfredo Di Tillio (Bocconi University), Marco Ottaviani (Bocconi University) and Peter Norman Sørensen (University of Copenhagen).
''To Replicate or Not to Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study'' by Zacharias Maniadis (University of Southampton) Fabio Tufano (University of Nottingham) and John A. List (University of Chicago).
''The Power of Bias in Economics Research'' by John P. A. Ioannidis (Stanford University), TD Stanley (Hendrix College) and Hristos (Chris) Doucouliagos (Deakin University).
