Wikipedia:When can you cite a preprint invoking expert SPS?
 | This is an information page. It is not a Wikipedia policy or guideline; rather, its purpose is to explain certain aspects of Wikipedia's norms, customs, technicalities, or practices. It may reflect varying levels of consensus. |
| “ | Self-published expert sources may be considered reliable when produced by an established subject-matter expert, whose work in the relevant field has previously been published by reliable, independent publications. | ” |
This passage from WP:EXPERTSPS is often used to justify the inclusion of non-reliable/non-peer reviewed sources on the basis that they are written by experts. The most common of which being blog posts and preprints. The argument is that since these are experts, these sources become reliable. This would mean nearly all preprints are reliable sources because most preprints are written by experts.
The reality is that peer reviewed articles from experts often contain errors. Clearing peer review just means you get to be part of the scientific debate. Preprints, working papers, or other preliminary/non-peer reviewed results, contain even more errors because one of the goals of peer review is to find those errors. Experts make mistakes all the time, and Wikipedia should not try to get ahead of the process just because one expert, or even a team of experts, is claiming a certain discovery ahead of publication.
So when can you invoke an SPS as a reliable source? The short of it is that an SPS is acceptable for routine, non-controversial claims, but that novel claims must still, at the very least, clear the higher bar of peer-review.
Routine claims
[edit]Non-peer reviewed sources, such as preprints, are perfectly acceptable to sources routine, non-novel claims, such as where a certain person is employed, or some background information that is not central to the paper. For example, using the preprint arXiv:2412.10366, whose abstract reads
| “ | We study realistic models predicting primordial black hole (PBH) formation from density fluctuations generated in a first-order phase transition. We show that the second-order correction in the expansion of the bubble nucleation rate is necessary for accurate predictions and quantify its impact on the abundance of PBHs and gravitational waves (GWs). We find that the distribution of the fluctuations becomes more Gaussian as the second-order term increases. Consequently, models that predict the same PBH abundances can produce different GW spectra. | ” |
We could use that source to support uncontroversial statements like
- Marek Lewicki is an astrophysicist working at the University of Warsaw.
<ref>arXiv:2412.10366</ref> - Primordial black holes are a candidate for dark matter.
<ref>arXiv:2412.10366</ref>
Because both are routine statements. Other sources might be preferable, like the University of Warsaw staff website, or a book dedicated to primordial black holes, but WP:V and WP:SPS are both met.
Novel claims
[edit]However, we could not use that same source for the novel claim that
- Models that predict the same primordial black hole abundances can produce different gravitational wave spectra.
<ref>arXiv:2412.10366</ref>
Why? Because that's the new result undergoing evaluation. Any sort of mistake could have happened to jeopardize that conclusion, ranging from
- Computer code mistakes, or unintentional bugs (e.g. writing '
y = mx - b' in code when 'y = mx + b' was meant) - Flawed assumptions (e.g. assuming all the fish detected in a river are trouts when there's a mix of trout and salmon)
- Flawed statistical analysis / error analysis (e.g. thinking a result is 15.5±0.5 m when it really is 15.5±2.8 m)
- Modeling mistakes (e.g. assuming cows are spherical, when a cubic model would be more appropriate)
Thus if a team of experts is claiming a discovery ahead of peer-review, Wikipedia cannot say something like
- The gravitational constant, G, has been determined to be 6.674305±0.000012 N⋅m2/kg2.
<ref>Smith, J. ... (preprint).</ref>
as if this was an established fact, or even
- On July 23, a collaboration from the University of Foobar has determined the gravitational constant, G, to a new precision of 6.674305±0.000012 N⋅m2/kg2, improving the accuracy by over 10 times.
<ref>Smith, J. ... (preprint).</ref>
as if the determination was recognized by the scientific community.
At most, Wikipedia can say something like
- On July 23, a collaboration from the University of Foobar claims to have determined the gravitational constant, G, to a new precision of 6.674305±0.000012 N⋅m2/kg2, improving the accuracy by over 10 times.
<ref>Smith, J. ... (preprint).</ref>
which is simply that a certain group made a certain claim, and takes no position on whether or not the claim is valid.
However, other considerations, like WP:DUE, WP:NOTNEWS, etc., must still be met. Oftentimes, the answer is simply to wait until the claim appears in a reliable peer-reviewed venue.
Concrete example: Hubble tension
[edit]This summarises a Wikipedia dispute about whether or not to include a certain preprint paper under SPS that happened on Talk:Hubble's law (spread over several archives). A preprint from a team of highly regarded astronomers lead by Wendy Freedman claimed an important problem called the Hubble tension – a statistically significant difference in the Hubble constant measured by different techniques – was resolved. The preprint was related to several other papers roughly has follows:
- A paper by Riess et al arXiv:2401.04773v1 is submitted to the arXiv on 9 January 2024, analyzing the JWST data, claims the tension still exists. Not much fuss is made by anyone. It is published on 10 February doi:10.3847/2041-8213/ad1ddd.
- A paper by Lee et al arXiv:2408.03474v1 is submitted to the arXiv on 6 August, also analyzing the JWST data, claims the tension still exists. Not much fuss is made by anyone.
- A paper by Freedman et al arXiv:2408.06153v1 is submitted to the arXiv on 12 August, also analyzing the JWST data, claims the tension is resolved. This would, if confirmed, be an astounding result. Because of the potential novelty and importance of the result, this was reported by many mainstream organizations like Science (see doi:10.1126/science.zk1y8fc).
- Lee et al revise their paper, also on August 12, arXiv:2408.03474v2. The revision is minor, clarifies uncertainties in the abstract, and is unrelated to the preprint by Freedman et al.
Because of the high visibility of Freedman et al preprint in mainstream media and the boom it was making in scientific circles, many editors want to include the results in the article. Two camps appear, those saying that it's a paper written by an expert, therefore is citeable per WP:SPS, and those claiming that it's WP:TOOSOON because it doesn't qualify as a WP:RS because it's not peer reviewed. An video overview by Dr Becky Smethurst discuss these papers, notes their preprint status, and talks about many if true, this would imply situations. She ultimately takes no position on who is correct or if the tension is finally resolved, but does note certain weakness in the statistical significance of the analysis in Freedman et al that is of particular relevance. She also notes Freedman et al have reported similar data in April and have revised their analysis twice.
Several arguments in favour of including Freedman et al that were made during the discussion:
- ... the paper is making waves already among cosmologists (this is a field that widely relies on arXiv)
- ... the paper is also likely to pass peer review substantially unchanged
- ... the fact that the paper is featured on that channel implies that it has already passed some level of peer review
- ... the lead author is a subject-matter expert
- ... the preprint has multiple senior authors, an important form of review
- ... Science.org generally produces reliable news content, it exercises scientifically trained editorial supervision on its news, and in this case the article includes an opinion by an outside subject matter expert... The author of the news article was an experienced science writer
- ... It's Wendy freaking Freedman. Has nobody noticed what all her awards are for?
Surely this means Freedman et al have written a great paper, and is therefore a reliable source? Those against inclusion made their case too, pointing to the significance of the result as requiring extraordinary sourcing. The results was a no consensus stalemate, amounting to 'let's wait for publication'.
After the August debate about whether to include Freedman et al, the following happens
- Riess et al submit a new preprint arXiv:2408.11770v1 on 21 August, incorporating the results other two preprints and commenting on them. It is revised on 28 October arXiv:2408.11770v2 and is published on 9 December doi:10.3847/1538-4357/ad8c21. In particular, they criticise the Freedman et al preprint analysis.
- Freedman et al revise their analysis in 14 March 2025 arXiv:2408.06153v2 with minor revisions to the paper on 17 March 2025 arXiv:2408.06153v3.
- Lee et al revise their paper a third time on 26 March arXiv:2408.03474v3.
- Lee et al get published on 22 May doi:10.3847/1538-4357/adc8a1.
- Freedman gets published on 25 May doi:10.3847/1538-4357/adce78. In particular, it no longer concludes that the Hubble tension is resolved, but they do conclude "the tension is reduced to 3.0σ" (down from the status of 5σ at the time of writing). They also conclude Addressing the H0 tension will require a decrease in both the systematic and statistical uncertainties.
The conclusion here isn't that Freedman and her team are bad scientists, or that they aren't subject-matter experts. The conclusion here is that science is a process, and that preprints represent a step in that process. Peer-review, while not perfect, did exactly what it was supposed to do. It found errors in the analysis, and Freedman and her team went back to the drawing board (and so did Riess's and Lee's teams), took feedback into account, and submitted revised findings.
This example clearly illustrates why it is premature to cite preprints for novel results.
Documents hosted on preprint repositories that are not preprints
[edit]Note that the above apply only to preprints, which is a term that apply to the versions of a paper prior to formal publication. It is not intended to be an authoritative resource, but rather one which is still in need of peer-review. They are usually uploaded to establish priority of discovery, but also to get feedback from the academic community, leading to an improved paper, revised arguments, correction of mistakes, etc...
A paper being on a preprint repository like arxiv is not a guarantee that it is a preprint. Several uploads are simply lecture notes, pedagogical material, or technical reports. Those are perfectly allowed under WP:EXPERTSPS, especially if the article is cited by independent authors in the field.
For example
- arXiv:hep-ph/0511217 – A distilled version of a invited lecture given at the SLAC Summer Institute in 2005. This is clearly a source allowed by WP:EXPERTSPS. It is authored by an acknowleged expert in the field, Scott Hughes (only distinguished scholars are invited to give lectures), and puts no novel results forward. It summarizes the state of the field (as of 2005).
- arXiv:2101.04168 – A compilation of the first 50 issues of John C. Baez's This Week's Finds of Mathematical Physics. These are as reliable as the original publication, authored by an acknowleged expert in the field (i.e. John C. Baez).
- arXiv:2211.07787 – A translation of a famous paper on quaternions by Olinde Rodrigues, translated by Richard Friedberg a theoretical physicist with an extensive background in mathematics. The paper was originally published in Journal de Mathématiques Pures et Appliquées, should be cited as the original source, but the translated paper can be cite alongside, for convenience to non-French speakers.