Excellent and opinionated essay on the state of reproducibility in social science, by one of our Replication Markets forecasters, after rating nearly 2,600 papers.
We won't know actual replication results for a bit, but many of the methodology observations stand, and are consilient with theoretical and empirical critiques from the 1950s.
Particular note: at the start, we asked for priors: our forecasters thought the replicability would improve steadily from 2009 to 2018. Summing over ratings of individual papers, there is no expected improvement with time - though all years are at the high end of initial expectations.
Excellent and opinionated essay on the state of reproducibility in social science, by one of our Replication Markets forecasters, after rating nearly 2,600 papers.
We won't know actual replication results for a bit, but many of the methodology observations stand, and are consilient with theoretical and empirical critiques from the 1950s.
Particular note: at the start, we asked for priors: our forecasters thought the replicability would improve steadily from 2009 to 2018. Summing over ratings of individual papers, there is no expected improvement with time - though all years are at the high end of initial expectations.