Z-curve 2.0: Estimating Replication Rates and Discovery Rates

The paper

Bartoš & Schimmack (2022). Z-curve 2.0: Estimating Replication Rates and Discovery Rates. Meta-Psychology. https://doi.org/10.15626/MP.2021.2720

Selection for statistical significance is a well-known factor that distorts the published literature and challenges the cumulative progress in science. Recent replication failures have fueled concerns that many published results are false-positives. Brunner and Schimmack (2020) developed z-curve, a method for estimating the expected replication rate (ERR) – the predicted success rate of exact replication studies based on the mean power after selection for significance. This article introduces an extension of this method, z-curve 2.0. The main extension is an estimate of the expected discovery rate (EDR) – the estimate of a proportion that the reported statistically significant results constitute from all conducted statistical tests. This information can be used to detect and quantify the amount of selection bias by comparing the EDR to the observed discovery rate (ODR; observed proportion of statistically significant results). In addition, we examined the performance of bootstrapped confidence intervals in simulation studies. Based on these results, we created robust confidence intervals with good coverage across a wide range of scenarios to provide information about the uncertainty in EDR and ERR estimates. We implemented the method in the zcurve R package (Bartoš & Schimmack, 2020).

Contribution

new method

Data basis

synthetic

Document

full article

Review stage

peer reviewed

Disciplines

meta-science, psychology

Access

open

Existing implementations

Code the authors or the community already shipped. This is what decides how far up the ladder a work can climb — no implementation, no tools.

• cranzcurveGPL-3.0-onlyby František Bartoš, Ulrich Schimmackofficialruns here

What it can do

• Zcurvezcurve

  Expected replication rate (ERR) and discovery rate (EDR), with 95% intervals.

• Versionsversions

  Report the R and package versions this agent runs, its limits, and its paper.

Provenance

Each check stands on its own. There is no single “verified” state: whether an agent faithfully implements its paper is a curation judgement, not something this registry can mechanically prove.

Machine-readable: paper.jsonld · server.json · agent-card.json

Agent: org.agenticjournal/paper-bartos-schimmack-2022-zcurve · v0.2.0