Talk:Replication crisis

The stats sections' ugly Overly detailed tag removal

I have taken the bold step of removing the ugly Overly detailed|section (dated July 2024} tag as it was not me clear what was being asked for and the section seemed to me fine. The tag itself was adding to the reading problem that might lead to the problem suggested. Perhaps some more specific suggestions here could help. Msrasnw (talk) 14:39, 23 July 2025 (UTC)

Of the 64% of non-replications, "only" 25% disproved

I would argue if "only" is appropriate in this context. Usually these tests are not symmetric in this way, so we don't expect really that if our assumptions are false, the negated null hypothesis will just as easily give significantly important data in favour of it being not true. The most basic example is showing linear correlation. The contrary is pretty much impossible to significantly proof, as variables correlated with ratio are linearly correlated, and indistinguishable in practice. Previously we were trying to show that ρ just is not equal to zero, so exclude one case, designing the experiment to expose |ρ| being big enough to make significant difference. But to show that there is no correlation it is not enough to expose |ρ| as small, there is just no positive ε small enough small enough to |ρ|<ε showing lack of linear correlation significantly, and the exact answer usually has probability 0. This asymmetry is not related to this particular statistic ρ (the same applies for any testing if point lies outside compact subset, which is much easier than outside open subspace if the data lie around point from its closure) but is something to be expected in every study. It might be less extreme, but there is a reason for using experimental methods very often to show that some random variable is bigger than second (for example that ordinary running is on average faster than running while juggling at the same time) and we don't just do the same with null hyppothesis reversed where we expect that there is no real difference between them.

The 0.25 success rate stated as that do not mean it is low and it may as well extremely large ~SpectralFlux 04:51, 7 August 2025 (UTC)

Request to add Nature investigation outcome for Google chip design paper

Hi, I represent Google DeepMind and am requesting an addition to provide the outcome of Nature's investigation into the 2021 chip design paper discussed in this section. Requested changes:==Addition of Nature investigation outcome==Current text ends with:"...A fraction of evidence reviewed in the critique is summarized below using prior terminology.[142]" Proposed addition (new sentence immediately following):"Following an 18-month investigation into these concerns, Nature published an addendum in September 2024 that upheld the original paper's methodology and findings." Rationale: The current text describes the criticism and Nature's investigation but does not include the outcome of that investigation. Since the section documents concerns about replicability, it's important for completeness to note that the journal concluded its formal review process. This addition maintains the documentation of the criticism while providing readers with the full timeline of events.

References

Ag77777 (talk) 18:57, 15 October 2025 (UTC)

Note for reviewers: an additional request is in progress at Talk:Google Brain that may be relevant. Encoded  Talk 💬 00:03, 30 December 2025 (UTC)

Not done for now: Please update this request to reflect the guidance you have received in the other areas you requested. Namely the citation you suggest is an LLM hallucination and you are obfuscating that the addendum is not independent of Google Czarking0 (talk) 04:33, 9 January 2026 (UTC)

Request to add data release and Nature investigation outcome to AlphaChip data leakage discussion

Hi, I represent AlphaChip and am requesting an addition to provide context about data availability and Nature's investigation outcome in the Data Leakage section. Requested changes:==Addition of context to AlphaChip data leakage discussion==Current text ends with:"...underscored the need for transparent, reproducible benchmarks and rigorous evaluation standards to ensure genuine progress in the field of electronic design automation." Proposed addition (new sentence immediately following):"Google subsequently released open-source code and training data for the AlphaChip methodology, and Nature's 18-month investigation concluded in September 2024 with an addendum upholding the paper's methodology. However, some researchers have maintained concerns about comparative performance claims." Rationale: The current text discusses reproducibility concerns but does not mention that Google released the code/data or that Nature's investigation upheld the methodology. Since this section is specifically about data leakage and reproducibility, noting that data was made available and that the journal's investigation found in favor of the paper is directly relevant. The final sentence acknowledges ongoing debate while providing readers with complete information.

References

Ag77777 (talk) 19:09, 15 October 2025 (UTC)

Note for reviewers: an additional request is in progress at Talk:Google Brain that may be relevant. Encoded  Talk 💬 00:03, 30 December 2025 (UTC)

Not done for now:Please update this request to reflect the guidance you have received in the other areas you requested. Namely the citation you suggest is an LLM hallucination and you are obfuscating that the addendum is not independent of Google. This is the second time I have responded like this and if you continue to request without acknowledging the issues it will be a conduct issue.Czarking0 (talk) 19:58, 10 January 2026 (UTC)

Czarking0, I sincerely apologize. Let me be transparent:

1) The citation I provided does have errors - I used AI assistance to format it, my apologies.

2) You're correct that the addendum is authored by the original Google research team.

Given your feedback, I now understand that this source alone may not be sufficient to warrant changes to this section.

If there's value in simply noting that Nature's investigation concluded (since the article currently says it's ongoing), I would propose only this minimal factual update:

Change: "...and the paper is under investigation by Nature.[60]"

To: "...and Nature concluded its investigation in September 2024.[60]"

This removes the "under investigation" language without making any claims about the outcome.

We reformatted the citation below.

“Nature published an addendum in September 2024 authored by the original research team.”

However, if you feel even this is not appropriate given the sourcing issues, I understand. I appreciate your patience in explaining Wikipedia's standards. Ag77777 (talk) 19:22, 21 January 2026 (UTC) Ag77777 (talk) 19:22, 21 January 2026 (UTC)

Czarking0, I sincerely apologize. Let me be transparent:

1) The citation I provided does have errors - I used AI assistance to format it, my apologies.

2) You're correct that the addendum is authored by the original Google research team.

Given your feedback, I now understand that this source alone may not be sufficient to warrant changes to this section.

If there's value in simply noting that Nature's investigation concluded (since the article currently says it's ongoing), I would propose only this minimal factual update:

Change: "...and the paper is under investigation by Nature.[60]"

To: "...and Nature concluded its investigation in September 2024.[60]"

This removes the "under investigation" language without making any claims about the outcome.

We reformatted the citation below.

“Nature published an addendum in September 2024 authored by the original research team.”

However, if you feel even this is not appropriate given the sourcing issues, I understand. I appreciate your patience in explaining Wikipedia's standards. Ag77777 (talk) 19:33, 21 January 2026 (UTC) Ag77777 (talk) 19:33, 21 January 2026 (UTC)

Did you see how I modified this section on Criticism of Google? That was a more extensive change that I think side steps the issue you highlight here. Potentially something similar could be done here but since this page is naturally more focused on scientific publication, I think the edit you suggest here is more reasonable. What is the [60] you are suggesting? If the is the correction, I do not think that paper claims the investigation was concluded to say so is OR. If the claim was like the the researchers published a correction in September 24 I think that is worth mentioning here and the primary source is fine citation for that. Czarking0 (talk) 20:33, 21 January 2026 (UTC)

References

Recent study demonstrating new data

Youyou W, Yang Y, Uzzi B. A discipline-wide investigation of the replicability of Psychology papers over the past two decades. Proc Natl Acad Sci U S A. 2023 Feb 7;120(6):e2208863120. doi: 10.1073/pnas.2208863120. Epub 2023 Jan 30. PMID: 36716367; PMCID: PMC9963456.

should be added as an up to date overview - also, during my research on the topic it seems that pychology is in fact exceptionally affected compared to other social sciences, contradicting the washington post article from 2015 FrenziedStargrime (talk) 14:44, 16 November 2025 (UTC)

Rigor and replication

Wouldn't it be more accurate to say

Replication failures do not NECESSARILY indicate that affected fields lack scientific rigor.

be more accurate than the current text that states

Replication failures do not indicate that affected fields lack scientific rigor.

since it seems that replication failures could be an indicator of less-than-careful practices? Dvdtoy (talk) 01:00, 11 January 2026 (UTC)

Sure, but what do the sources say? Czarking0 (talk) 01:08, 11 January 2026 (UTC)

Good question. The first cited source* says science is hard to do well (granted) but does not IMHO support the statement that "Replication failures do not EVER indicate that affected fields lack scientific rigor." [EVER added here by me because implied]

• https://fivethirtyeight.com/features/science-isnt-broken/

which says for example

"P-hacking and similar types of manipulations often arise from human biases."

But that doesn't exclude lack of care/rigor as a possible source of error. Dvdtoy (talk) 01:25, 12 January 2026 (UTC)

Agreed Czarking0 (talk) 01:49, 12 January 2026 (UTC)

Notebooks

It's kind of unfair to not mention Jupyter Notebooks here. Miklevin (talk) 11:43, 14 April 2026 (UTC)

Jupyter Notebook should be mentioned in what context? Richard-of-Earth (talk) 03:13, 15 April 2026 (UTC)