Die ZBW ist Mitglied der Leibniz-Gemeinschaft.

Replikationen in den Wirtschaftswissenschaften

E-Science-Tage 2017 Session B1: Innovative Ansätze 16. März 2017 an der Universität Heidelberg Ralf Toepfer ZBW Leibniz-Informationszentrum Wirtschaft (Kiel / Hamburg)

- Stand und Perspektiven -

Seite 2

Agenda

1. Einleitung: „Replikationskrise“

2. Zum Stand (und „standing“) von Replikationen in den

Wirtschaftswissenschaften

3. Initiativen zur Erhöhung der Reproduzierbarkeit (Perspektiven)

4. Fazit

1. Replikationskrise: „Most scientist have experienced failure to reproduce results“

Seite 3

http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

2. Zum Stand (und „standing“) von Replikationen in den Wirtschaftswissenschaften

„In statistical and other numerical work presented in Econometrica the original raw data will, as a rule, be published, unless their volume is excessive.“ (Frisch 1933, p.3) „Tedious though its requirements may be, reproducibility remains the touchstone of the scientific method. If an empirical finding is a fact, other researchers should be able to observe it, too. Successful and independent repetition of an econometric experiment increases professional confidence in the experiment’s alleged results.” (Kane 1984, 4)

Seite 5

„…economics research is usually not replicable“ • Dewald, Thursby & Anderson (1986) attempted to replicate 54

papers published in the Journal of Money, Credit and Banking and could replicate only two (2/54)

• McCullogh, McGeary & Harrison (2006) tried to replicate 69 articles published in the same journal and could only replicate 14 (14/69)

• McCullogh, McGeary & Harrison (2008) attempted to replicate 117 articles published in the Federal Reserve Bank of St. Louis Review and could only replicate 9 (9/117)

• Chang & Li (2015) successfully replicate 29 of 59 papers (49%) from several journals even with help from the authors

Chang, Andrew C., and Phillip Li. Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not". 2015. https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf

“Replication includes but is not limited to slavish duplication of a predecessors study’s specific procedures.” (Kane 1984, p.5)

Seite 6

„Thus if a replication test gives discrepant results, under current usage of the term, this could mean a wide spectrum of things – from signaling a legitimate disagreement over the best method (science), to signaling incompetence and fraud (pseudoscience).“ (Clemens 2015, p. 1)

Michael A. Clemens: The Meaning of Failed Replications: A Review and Proposal. IZA DP No. 9000. http://ftp.iza.org/dp9000.pdf

Seite 7

„The status quo in empirical research in economics and management is not to share data“

(N=488) Do not share

data

Sporadically share

data

Share data regulary

Responses 394 82 12

Percent 80.74% 16.8% 2.46%

Andreoli-Versbach, P., Mueller-Langer, F., Open access to data: An ideal professed but not practised. Res. Policy (2014), http://dx.doi.org/j.respol.2014.04.008

Data + Description (Documentation) + Code

Seite 8

http://journaldata-zbw.eu/

“…uninventively verifying someone else’s empirical research is not a completely respectable use of one’s time. Choosing such a task is widely regarded as prima facie evidence of intellectual mediocrity, revealing a lack of creativity and perhaps even bullying spirit.” (Kane 1984, 3)

„…verifying someone else‘s empirical research is not a completely respectable use of one‘s time“

Seite 9

Kane, Edward J. “Why Journal Editors Should Encourage The Replication Of Applied Econometric Reserach.” Quarterly Journal of Business and Economics, 1984: 3-8.

„…, but replication activities have only marginally increased.“

Seite 10

Maren Duvendack, Richard W. Palmer-Jones and Robert Reed: Replications in Economics: A Progress Report. Econ Journal Watch 12(2), May 2015: 164-191

„Apparently, we all love ‚bold theory,‘ ‚breaking new ground,‘ and ‚innovative research.‘ However, such an obsessive focus on novelty should not be the sole engine of scientific progress. As in any discipline, real ‚groundbreaking research‘ is very rare, as are successful entries into ‚new terrain.‘ Much scholarly work involves transpiration, and not so much inspiration.“ (van Witteloostuijn 2016, p. 485)

Novelty bias

Seite 11

Witteloostuijn, Arijen. „What happend to Popperian falsification? Publishing neutral and negative findings Moving away from biased publication practices.“ Cross Cultural & Strategic Management, 2016: 481-508 http://dx.doi.org/10.1108/CCSM-03-2016-0084

„Economists treat replications the way teenagers treat chastity – as an ideal to be professed but not to be practiced“ (Hamermesh 2007, p. 1)

„…an ideal to be professed but not to be practiced“

Seite 12

Hamermesh, Daniel S. Replication in Economics. NBER, 2007. doi: 10.3386/w13026

„Somehow though, pleas for replication studies – however well articulated and however often repeated – seem to be falling on deaf ears.“ (Harzing 2016, p. 564)

Harzing, Anne-Wil. “Why replication studies are essential: learning from failure and success”. Cross Cultural & Strategic Manamement. Vol. 23 Iss 4 pp. 563 – 568. http://dx.doi.org/10.1108/CCSM-07-2016-0133

3. Initiativen zur Erhöhung der Reproduzierbarkeit

Seite 13

(1) Replication Network

(2) Replication Wiki

(3) Replication Section

(4) Replication Journal

(1) The Replication Network

Seite 14

https://replicationnetwork.com/

• Update scholars about the state of replications in economics

• Establish a network for the sharing of information and ideas

(2) Replication Wiki

Seite 15

http://replication.uni-goettingen.de/wiki/index.php/Main_Page

„This wiki serves as a database of empirical studies, the availability of replication material for them and of replication studies.“

(3) Replication Section

Seite 16

http://www.economics-ejournal.org/special-areas/replications

“Replications are an important public good to the economics community, and as such, they tend to be under-valued. By opening a replications section in our open access, open assessment journal, we hope to make it easier for authors to provide this important public good.”

(4) Replication Journal (I)

Seite 17

„Existing journals could be more willing to accept replication studies, but it is clear reputational inertia is not providing the right incentives. Hence, I suggest the creation of a journal entirely dedicated to replication.“ (Zimmermann 2014, p. 3)

Zimmermann, Christian. On the Need for a Replication Journal. St. Louis, 2014. https://research.stlouisfed.org/wp/2015/2015-016.pdf

(4) Replication Journal (II)

Seite 18

http://www.zbw.eu/de/ueber-uns/arbeitsschwerpunkte/forschungsdatenmanagement/iree/

„The rising interest in transparency and reproducibility in economics reflects broader global trends regarding these issues, both among academics and beyond. As such, we argue that ‚this time‘ really may be different than earlier bursts of interest in research transparency within economics (…) that later lost momentum and mostly died down.“ (Christensen & Miguel 2016, p. 61)

4. Fazit: „…‘this time‘ really may be different…“

Seite 19

Christensen, Garret S., und Edward Miguel. Transparency, Reproducibility, And The Credibility of Economics Research. NBER Working Paper 22989: http://www.nber.org/papers/w22989, 2016.

Seite 20

„Vielen Dank für Ihre Aufmerksamkeit!“

Kontakt: Ralf Toepfer ZBW – Leibniz-Informationszentrum Wirtschaft Neuer Jungfernstieg 21, 20354 Hamburg E: r.toepfer@zbw.eu T: +49(0)40.42834-449 Twitter: @thiaru

Literatur Andreoli-Versbach, Partrick, und Frank Mueller-Langer. „Open Access to data: An ideal professed but not practised.“ Research Policy, November 2014: 1621-1633.

http://dx.doi.org/10.1016/j.respol.2014.04.008 Baker, Monya. „Is there a Reproducibiliby Crisis?“ Nature, 26. May 2016: 452-454. doi:10.1038/533452a Chang, Andrew C., and Phillip Li. Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not". 2015.

https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf Christensen, Garret S., und Edward Miguel. Transparency, Reproducibility, And The Credibility of Economics Research. NBER Working Paper 22989: http://www.nber.org/papers/w22989 ,

2016. Clemens, Michael A. The Meaning of Failed Replications: A Review and Proposal. IZA, 2016. http://ftp.iza.org/dp9000.pdf Dewald, William G., Jerry Thursby, and Anderson Richard G. “Replication in Empirical Economics: The Journal of Money, Credit and Banking Project.” American Economic Review, 1986:

587-603. http://www.jstor.org/stable/1806061 Duvendack, Maren, Richard W. Palmer-Jones, and Robert W. Reed. “Replications in Economics: A Progress Report.” Econ Journal Watch, 2015: 164-191.

https://econjwatch.org/articles/replications-in-economics-a-progress-report Frisch, Ragnar. “Editorial.” Econometrica, 1933: 1-4. Hamermesh, Daniel S. Replication in Economics. NBER, 2007. doi: 10.3386/w13026 Harzing, Anne-Wil. “Why replication studies are essential: learning from failure and success”. Cross Cultural & Strategic Manamement. Vol. 23 Iss 4 pp. 563 – 568.

http://dx.doi.org/10.1108/CCSM-07-2016-0133 Kane, Edward J. “Why Journal Editors Should Encourage The Replication Of Applied Econometric Reserach.” Quarterly Journal of Business and Economics, 1984: 3-8. McCullogh, B.D. „Open Access Economics Journals and the Market for Reproducible Economic Research.“ Economic Analysis & Policy,. March 2009 (Vol. 39, No.1): 117-126. doi:

http://dx.doi.org/10.1016/S0313-5926(09)50047-1 McCullough, B.D., K.A. McGeary, und T. Harrison. „Lessons from the JMCB Archive.“ Journal of Money, Credit and Banking, 2006: 1093-1107. doi: http://dx.doi.org/10.1353/mcb.2006.0061 McCullough, B.D., K.A. McGeary, und T.D. Harrison. „Do Economic Journal Archives Promote Replicable Research?“ Canadian Journal of Economics, 2008: 1406-1420. Zimmermann, Christian. On the Need for a Replication Journal. St. Louis, 2014. https://research.stlouisfed.org/wp/2015/2015-016.pdf Witteloostuijn, Arijen. „What happend to Popperian falsification? Publishing neutral and negative findings Moving away from biased publication practices.“ Cross Cultural & Strategic

Management, 2016: 481-508 http://dx.doi.org/10.1108/CCSM-03-2016-0084

Seite 21