Why Peer Review is Wrong!
Dear Sir,
We (Mr. Rosen and I) had sent you our manuscript for publication
and had not authorized you to show it to specialists before it is printed.
I see no reason to address the—in any case erroneous—comments
of your anonymous expert.
On the basis of this incident I prefer to publish the paper elsewhere.
Respectfully,
Albert Einstein
P.S. Mr. Rosen, who has left for the Soviet Union, has authorized me
to represent him in this matter.
Can you safely declare to have never complained about any negative blind peer review (BPR) report? I saw many of us getting really angry, up to rage, for negative reports. Complaints take a lot of our time wasted in endless discussions. From enumerating technical objections to a report; to rejection of the referee for incompetence; or for bias... Few of us start to rise criticisms to the blind peer review system itself, with proposal of own solutions for improvement. Round tables are recurrently organized, often under the initiative of publishers who might not be sincerely interested in reforming a system that provides them huge profits. If the drawbacks associated to the BPR are today well identified by the majority, there is no clear perception of a valid alternative. So that publishers, fearing to lose profits, have good hand in instilling us that we live in the best of all possible worlds.
The Einstein vs Physical Review incident
Is it sure that the peer review has always existed? -
This is a good question to start our reflection, together also with the famous letter of Albert Einstein to the Physical Review quoted above.
The sensation, even among the oldest of us, is that this system has always existed, since Galileo, as if it were connatural to the scientific method itself.
But the answer is: No! It has not.
The peer review is a recent practice and was routinely applied by journals only since the middle of the 20th century.
The myth of an always existed BPR is very well disavowed in Nielsen's blog based on studies of historians.
It is also impressing that, out of more than 300 articles published by Einstein in his life till 1955, only the one to the Physical Review with Rosen passed through a peer review
Einstein vs the PR -
Einstein letter is embarassing for editors and publishers.
For this reason it gave rise to unlikely re-interpretations by improvised exegetes of the Einstein thought: "No...., he didn't really mean that..., he was not really opposing the peer review... etc."
However, for historians this letter stands out of the whole corpus of Einstein's correspondence for its unusual tone, showing a particularly angry reaction by an otherwise normally gentle character.
Furthermore, after the "incident" Einstein wanted to have nothing to do more with PR and stopped submitting there, except for a letter to the editor in response to a paper critical against his unified field theory that he wrote in 1952.
Do gravitational waves exist? -
In 2005, at the occasion of the 100th anniversary of the 1905 Einstein annus mirabilis, Physics Today (AIP) published an article by D. Kennefick on the story of the "ïncident", Einstein Versus the Physical Review, which does not deny the real feeling of Einstein against the blind peer review, evident in his letter.
This article brings new important documents (e.g. the 1936 PR logbook) which unveil the identity of the anonymous referee: H. C. Robertson.
However there is a back-message passing behind the historical reconstruction.
A message clearly enunciated in the sub-title, "A great scientist can benefit from peer review, even while refusing to have anything to do with", and evidently at heart of AIP and APS editors in support to the BPR.
The thesis is that in 1936 Einstein and Rosen submitted to PR a paper with, presumably, the wrong result that gravitational waves do not exist; that the error was remarked by the referee Robertson in his report which was then angrily dismissed by Einstein as in his letter.
The paper was later published in 1937 on the Journal of the Franklin Institute, but here the conclusion is that gravitational waves rigorous solutions do exist.
However Kennefick does not clarify to which extent the referee report effectively influenced Einstein for the final version of the paper; or the correction of the error was rather an independent Einstein's intuition; or how beneficial were side discussions with L. Infeld and Robertson himself intervened in the mean time at Princeton.
In fact there is even uncertainty on how deep the revision was effectively (and so how serious the error was), since the original version of the manuscript submitted to PR no longer exists.
More historical research is probably needed, but this is not the purpose here.
Instead here I fully assume Kennefick's historical reconstruction and, in the following, I will use it to rather achieve the opposite thesis: that the blind peer review is not at all beneficial.
A paper may be right or wrong.
But the BPR is simply wrong.
Errors and Science
The EPR paper -
Let's first clearly state an important point: Science is not exempt of errors!
The history of science is riddled of errors which were corrected only after time.
And PR editors must know this very well.
Indeed, one of the most cited PR of any times is the famous EPR paper in which Einstein, Podolsky and Rosen attack quantum mechanics through a paradox indicating the incompleteness of the quantum mechanical description of the physical reality.
We all know the history of this controversy on the foundation and interpretation of quantum mechanics, opposing from one side Einstein (who was, together with Planck, even one of the first QM pioneers), E. Schrödinger (with his famous cat), et al., and from the side of the defenders N. Bohr, W. Heisenberg and all the Copenhagen school.
Following developments by J. S. Bell and finally experiments e.g. by A. Aspect, after many years it emerged that the correct vision was Bohr's.
Nevertheless, the EPR paper keeps an important and fundamental contribution to physics that had to be published.
(Very luckily for the Physical Review and its citations indices in modern times).
This might not have been the case, were the paper sent for blind peer review to N. Bohr or W. Heisenberg who considered Einstein wrong.
The cosmological constant -
But Einstein himself was perfectly aware that some papers along his life were in error, Kennefick's article cites an Einstein's sentence to Infeld: "There are incorrect papers under my name too."
And what about Einstein's biggest mistake on the static universe and the cosmological constant?
Einstein's equations only provide a non-static (either expanding, or contracting) universe.
Nevertheless in 1917 he claimed that his equations were compatible with a static universe (the dominant cosmological model at that time) thank to the introduction of a fine-tuned cosmological constant, missing that, also in that absurdly ad hoc exact balance case, the universe would be in any case unstable.
This was later considered by Einstein his "great blunder".
However, recent experimental measures of an accelerating expansion of the universe seem to reconsider again the Einstein's hypothesis of a non-zero cosmological constant, although with a scenario different from the original static and stable universe.
So, also that paper was another important Einstein contribution to physics which was first considered correct, then (to Einstein's affliction) wrong, to be finally revaluated, somehow [1]..
Photons -
Imagine that today an editor sends you for review a paper with the title: "On a Heuristic Point of View about the Creation and Transformation of Light".
Don't lie! I already did the experiment to send this title to colleagues, as if it were a new paper to be reviewed.
And their expression said of some outlandish and odd theory by some psycho-lunatic.
Indeed, "Heuristic" sounds really as metaphysics, "Creation of Light" is readily associated to some Genesis / Theogony religious belief, and "Transformation" to magic.
No doubt most referees would not even waste their time to read the abstract, and reject it as "Esoteric".
Instead, this is the title of the 1905 Einstein revolutionary Annalen der Physik on the theory of the quanta of light which yielded him the Nobel prize.
This paper had a long history of rejection by the unanimity of the physical community, as it is well explained by StuewerR. H. Stuewer, Eintstein's revolutionary light-quantum hypothesis, Proceeding to the 2007 Conference on the History of Quantum Mechanics..
For 20 years, nobody could accept Einstein photons hypothesis that seemed to repropose the old Newton corpuscular theory of light against Maxwell theory, after hundreds of experimental evidences having confirmed the undulatory nature of light and Maxwell equations.
Not even R. A. Millikan who, with his experiment in 1915, confirmed Einstein's predictions on the photoelectric effect but rejected Einstein's interpretation of it, rather preferring the Thomson-Sommerfeld-Richardson classical explanation.
Contrary to overenthusiastic statements, science is also matter of opinions.
(And there are overenthusiasts who pretend to introduce BPR also in humanities, literature and art reviews!).
An enough universally accepted scientific evidence can be achieved only after many years.
If Einstein had to pass a BPR, he would have found no referee inclined to accept his paper and arguments!
Also Max Planck, though one (and the first) of the pioneers of quantum mechanics, rejected Einstein's hypothesis since the beginningM. Planck to A. Einstein, july 6, 1907; M. Planck to Prussian Academy, june 12, 1913, Collected Papers of Albert Einstein 5:445. and till the end of his life up to bordering on a tragedyM. Planck, Scientific Autobiography and other papers.
Note that in 1905 Planck was the Annalen der Physik editor in charge for theory.
Surely Planck had read, before publication, the manuscript of an unknown young physicist named Einstein, at that time not even a Doctor yet.
Even though he was convinced that Einstein paper was wrong, nevertheless Planck published it!
Simply because at that time there was no peer review!
Relativity -
The quanta of light paper was not at all the unique case of an Einstein theory strongly rejected by the physics community.
There were many physicists against Einstein prejudiciously, and even more.
An extreme case? Nobel prize Philipp Lenard.
They were opposing Einstein at any occasion.
We now start to understand why Einstein could be so much opposed to the blind peer review.
He could well imagine all the censure to which his scientific production would have been submitted if the blind peer review had been in place.
Only minor papers could pass.
None of Einstein revolutionary papers would have been published.
The 1905 article on special relativityA. Einstein, Zur Elektrodynamik bewegter Körper, Annalen der Physik. 17 (10), 891 (1905)., revolutioning the so far absolute concepts of space and time, would have been rejected as a metaphysical paper.
Some months later Einstein realized another important consequence of special relativity: the famous energy-mass equivalence.
And in the same 1905 he published it in another Annalen der PhysikA. Einstein, "Ist die Trägheit eines Körpers von seinem Energieinhalt abhängig?" Annalen der Physik. 18 (13), 639 (1905)..
But, to the eyes of a modern peer reviewer, this is evidently a follow-up paper to be absolutely rejected.
Today editors explicitly warn referees to check whether the paper to be reviewed constitutes the follow-up of a previous paper, which could configure an authors' attempt of packetting a single work to have more publications, under the pressure of the publish-or-perish or other bibliometric reasons.
It becomes evident that, if the blind peer review had been in place at Einstein's time, the physics of the 20th century would have been completely different.
And in 2005 neither the AIP, nor the APS, nobody would have an Annus Mirabilis anniversary to celebrate.
Wrong aspects of the blind peer review from the Einstein vs PR incident
Publish or not a wrong paper? -
From all the examples above, whether a manuscript is correct or wrong is not at all a question that a publisher, or a referee, or even a very expert editor like Planck, should ask.
The answer provided by even the most expert referee is not superior, it is equivalent to that one of the author.
The referee is peer, not superior to the author.
An article is not any more a student homework corrected by the teacher with a red pencil.
In case of controversy, nobody can say who is right, whether the author or the referee.
Only history will tell.
The manuscript must be published independently from the fact that it might appear correct or wrong.
This should have been the case also for the Einstein-Rosen manuscript.
Can BPR detect errors? or at least frauds? -
If it might be possible to check, by doing it again, an analytic calculation like Einstein and Rosen (wasting the same time as the authors, which raises the question of why a referee should do that), in the vast majority of the cases it is not possible for a referee to reproduce an experiment or a numerical calculation.
To check the discovery of the Higgs boson a referee should have an LHC available.
We can only trust our colleagues.
Of course, they may have done an error, e.g. an unplugged socket which provided a wrong measure of superluminal neutrinos.
But a referee is absolutely impotent against errors, or even real frauds!
A well known example is the Schön scandal, tens of fabricated experiments published on high-IF journals.
Only following developments of science could confirm or unveil errors and frauds.
The BPR distorts the merits -
We now pass to analyse the negative consequences following the PR editor wrong decision / original sin.
If it is true that the original Einstein-Rosen conclusions were wrong, and that the merit to have understood the correct conclusions belongs to the referee Robertson, then Robertson evidently provided an important contribution which does not result from the literature, it is completely hidden.
It would be more correct that a paper with the correct conclusions be published under the authorship of Robertson.
So, the BPR has the serious defect that it introduces a distortion in the correct attribution of the merits between authors and referees.
It confuses authors with blind referees.
Without this distortion, the development of the gravitational waves story would have been:
Einstein and Rosen publish a PR with the wrong conclusion that GW do not exist.
Robertson reads the paper and immediately realizes the error.
He then publishes a second paper with the right conclusion that GW do exist.
The fundamental contribution to this important finding is correctly attributed to Robertson, as he deserves.
On the other hand, authors wishing to discuss with colleagues their results before publication, don't need a BPR: they can do it naturally with seminars, private conversations, exactly as Einstein did with Infeld and Robertson himself.
If from these exchanges a colleague points out an error, the authors are obliged to recognize the merits, possibly associating the colleague to the list of authors in the corrected version of the paper.
The BPR cannot state novelty/originality -
In fact, the correct solution for cylindrical gravitational waves was already published by G. BeckG. Beck, Z. Phys. 33, 713 (1925). 11 years before.
If the referee Robertson had been aware of it, he could have told the authors, and the story would have had an immediate end with the withdrawal of the paper by Einstein and Rosen.
But neither publishers, nor editors, nobody can pretend that a referee knows everything of one field.
At least no more than authors that had much more time to look for previous contributions.
So, peer review has very few probabilities to state about the novelty and originality of a work.
If it was not possible for a paper on GW in 1936, when experts and papers on general relativity were very few, today it becomes impossible in all fields.
Is the blind peer review really scientific? or rather anti-scientific?
Floating bodies -
Let's now jump three centuries back to show how seriously dangerous the BPR can be for Science.
Galileo is considered the inventor of the modern scientific method.
But he did not invent physics, nor astronomy.
They were existing since long time, at least since ancient Greece.
At Galileo's time the dominant theories were Aristotle's Physics (Φυσικὴ) and the Ptolemaic geocentric cosmological model.
Of course, at Galileo's time a blind peer review was outside any imagination: he sent all his manuscripts to the editor for straightforward printing and publication, exactly like Einstein.
But let's imagine to transpose the BPR back to Galileo's time and imagine who could have been selected as a good "Peer Reviewer" for his papers.
A good example is certainly Lodovico delle Colombe, a scholar famous for his battles against Galileo in a series of controversies on physics and astronomy in defense of Aristotelian theories.
Celebrated was his Dispute over floating bodies,
whether the floating or the sinking of bodies is determined by their density or by their shape, which engaged Delle Colombe against Galileo in written publications/reciprocal comments, and even in Florence public experimental demonstrations.
By reading Delle Colombe's Discorso ApologeticoLodovico delle Colombe, Discorso apologetico, Pignoni, Florence 1612. it is clear that Galileo had no one possibility to see his Discourse on Floating BodiesGalileo Galilei, Discourse on Floating Bodies, Cosimo Giunti, Florence 1612. published, were Delle Colombe the blind referee.
The same conclusion holds if the editor had selected other possible peers, e.g. Giorgio Coresio, Vincenzo Di Grazia, etc.
Galileo's view was completely isolated with respect to the mainstream.
The 1604 supernova -
But even more interesting was the Delle Colombe - Galileo Dispute over the 1604 supernova.
The appearance of a new (nova) star in 1604 could undermine the Aristotelian-Ptolemaic view which held that stars were fixed in their position and permanent, unchanging in the firmament.
In an extreme defense of this view, Delle Colombe published a Discourse in which he claimed that the star was not at all new, but it had always been there in the sky since the beginning, simply it had always escaped to be remarked.
Few months after Delle Colombe, Galileo published under a pseudonym his Considerations in which he regarded the phenomenon as new and not a permanent star, and ridiculed Delle Colombe's view.
We can today all together laugh with Galileo and be relieved that his paper was freely published and had not to pass through the censorial review of his "peer" Delle Colombe.
But the most amazing fact is that following scientific developments demonstrated that it was Delle Colombe to be right!
Indeed it became possible to demonstrate that nova and supernova stars are not at all "new".
The increasing power of telescopes and photographs allowed us to remark that a star, invisible to the eye, is already present in the position where the supernova appears.
Paradoxically, we have to be relieved that also Delle Colombe had the possibility to freely publish his view.
Motion of Earth -
In 1610 Galileo published the Sidereus Nuncius in which he started to talk about the motion of Earth.
Delle Colombe contested Galileo's view and reacted with his leaflet Contro il moto della Terra (Against the motion of the Earth) appealing to common sense arguments, e.g. the ball fired by a cannon which lands at the same distance, no matter if fired to east or west.
However Delle Colombe decided not to publish, not to print the leaflet which only circulated in manuscript form.
Publication of a paper meant for an author assuming the responsibility of its content and, in case, exposing to the ridicule.
Verba volant, scripta manent.
This is another important aspect today completely lost:
if a paper passes the review and is published, we feel like it has been certified, by the referees, and it is not any more of our responsibility.
Like if our author's signature were less important than the referee's.
Delle Colombe showed not to be sure about his arguments. and preferred not to assume the responsibility to publish.
For this reason, Galileo decided there was no reason to reply to Delle Colombe.
The Galileo trial -
Delle Colombe was not even the worst to be faced.
He did not testify as an external expert against Galileo at the trial by the Roman Inquisition, ending with condemnation for heresy.
Galileo was forced to abjure his astronomical views, and his writings were withdrawn from publication, burned, and put to the Index (banned).
Nevertheless, we must be relieved that the 17th century editorial situation was not that bad.
Indeed, Galileo manuscripts were all published.
Although forbidden for long time, published copies were secretly preserved, and we can today read all Galileo's writings.
And exactly in the form Galileo wrote them, without amendments, forced retractations, abjurations.
I don't know who first introduced the blind peer review, but it is utterly evident that the Inquisition and the Holy Office would have considered the BPR a brilliant idea!
This would have certainly avoided the Inquisition to personally intervene, as by injudicious and impolitic public trials.
All the cleaning dirty job, and at a much more confidential "blind" level, would have been in charge to Galileo "peers", e.g. Lodovico delle Colombe, Giorgio Coresio, Vincenzo di Grazia, Cesare Cremonini ...
Controversies must be discussed and solved in public daylight, not in the "blind" darkness of the peer review, as publishers pretend.
Likely, if the BPR had already been in place at Galileo's time, today we would be still arguing about the subtleties of Aristotelian Physics, and adding an epicycle more to the intricacies of the Ptolemaic cosmological model.
It might not appear to our friends in the new worlds, but to Europeans that lived the darkest centuries of the middle-age it is evident that the blind peer review is more anti-scientific than scientific.
Psychological aspects of the blind peer review
Intimidation charge of the BPR -
The Post Scriptum of the Einstein letter to PR offers us another interesting point of reflection: the psychological intimidating aspects of the BPR.
Psychologically, for the author the BPR is a real trial to which she/he must submit.
Her/his work, ideas, views, new concepts, interpretations, everything is put under question and contrasted with the actual framework and vision.
To a smaller scale, it is very much like as the repetition of the Galileo's trial, of course in its restriction to only physical and astronomical questions and without metaphysical and religious aspects (to which anyway Galileo himself made clear were absolutely not the object of his reflection and abdicated).
In the BPR trial the editor sets himself up as the moral and impartial authority, and stands as the judge who will pass the sentence.
In case of condemnation, the sentence is very often pronounced by a whole tribunal of editors, including higher levels, to psychologically inculcate in the authors that it is far to be an arbitrary single opinion decision.
The prosecution is then played by the blind referees.
And this is the most indimidating aspect of the BPR: who is hiding behind this mysterious "anonymous expert" (to use the same Einstein's expression)?
Won't he be a Divinity in knowledge of the Supreme Truth?
Severely scrutinizing my earthly proposals against the Truth?
The Einstein-Rosen GW paper was the first and unique experience of Einstein with the BPR, but we can already find all the psychological elements here discussed.
The letter sent by the PR editor to Einstein was absolutely similar to what today everybody would receive from an editor accompanying a critical review:
behind a formal and a cold politeness, a mild request of exhaustive explanations to the various referee's comments and criticisms.
Exactly like in a trial where the accused is questioned by the prosecution and pursued by the judge to answer the prosecutor questions that press him into contradiction.
This strong charge of intimidation can be read behind the Einstein's Post Scriptum:
PR editor: "Come on Einstein! My mysterious anonymous expert told me: You are Wrong! And, in this Deviance, you are Alone! Retract your False believes! Abjure!"
Einstein: "No! No! I am not alone! Mr. Rosen is with me! He just only left for the Soviet Union. But he is morally still with me!"
You can see that the psychological situation set up by the BPR is such that even a young simple editor succeeded in intimidating the great Albert Einstein, at that time already a world-known personality, but feeling the need of the psychological support by just only a young assistant.
Not that an editor could achieve the high skill of an Inquisitor of the Holy Office, who were explicitly trained to that purpose.
But in principle he could have the same intimidating power.
And the psychological situation that the author feels is not much dissimilar from what Galileo felt.
Defects, drawbacks, flaws and vices of the blind peer review
Let's now provide a complete list of BPR defects. Most of them are recognized by the majority of researchers.
- Anti-scientificity: this has already been discussed at the Galileo example.
- Hinder or stop great discoveries: We already discussed this at the example of Einstein. Revolutionary contributions are hindered, if not fully stopped, rejected by referees that can sincerely consider them wrong. Or consider the author a visionary. A good example is the demonstration of the Poincaré conjecture, one of the topmost major ("millennium") problems of mathematics which was solved by Grigori Perelman. One might wonder why Perelman did not submit his demonstration to very high impact factor reviews. The discovery was too great to be believed, and the demonstration too complex to be understood in less than the three years effectively spent by a committee especially appointed to check it. The only possibility for him was to publish it on the non-peer-reviewed arXiv [1]G. Perelman, arXiv:math/0211159 [2]G. Perelman, arXiv:math/0303109[3]G. Perelman, arXiv:math/0307245. Referees would have raised tons of exceptions, pretended to rewrite the paper... Without the arXiv the humanity would have lost this fundamental contribution!
- Theft of major contributions: Referees can be competitors in conflict of interest with authors. Referees jealous/envious of a colleague major advancement, can reject its publication by alleging false questions. Or they can use the peer review to delay publication by asking the most laborious checks or much more work. This can allow the time sufficient for the referee to work and recover on the competitor, to end with an almost contemporary publication of the same results under a different shape and with some marginal variations. I have heard many stories of this kind. And there are rumours about stories to prevent thefts, like: Bednorz and Müller having submitted to the BPR a first version paper with an on-purpose error in the formula of the compound found to be superconducting at high temperature; or Ketterle having reported 90 degrees rotated photographs of his Bose-Einstein condensation atomic trap because even gravity played a role understood only after several years (the advantage that Ketterle wanted to keep on competitors).
- Censorship: A referee can be biased and unfairly reject, and so censor, a new important valid contribution. There are many reasons for that: The referee may have spent years to elaborate a theory which is contradicted by the results, or by a better theory presented in a paper received for review. (I was informed of very shocking crossed referee reports between competing theories for the high-temperature superconductivity). Or a referee may have measured a quantity, for example a bandgap in a given material, and is reviewing a paper which present a different result, for example the material is found as a zero bandgap metal. What I consider serious is that today editors uncritically give to the referees an enormous power of extortion on authors. If a referee does not like a "disturbing" point in a paper, he can force / oblige the authors to its removal under the extortion that the paper is not published. Editors are incompetent and cannot realize such extortions that they consider "scientific improvements" of the paper.
- Imposition: Or, using the same "power of life or death on publication", referees can do the reverse: oblige authors to introduce sentences in favour of their theory or of their work. Change or force new conclusions toward other directions. The most honest referees only pretend citations to their papers, one of the today most spread immoral habit. In order to increase their bibliometry indices, referees use the power that editors give them to enforce citations to their publications, regardless if they really have to do with the actual subject of the paper. Are editors aware of it? and what are they doing? Nothing! Or even worst: Physical Review Letters editors decided to change the format of their publication, previously restricted to only 4 pages, while now the 4 pages is the limit only for the scientific text and there are no limits for the bibliography. So, editors are happy with this since it boosts citation indices and impact factors.
- Difficulty in coming back from a previously published error: If something wrong has been previously published, it is absolutely difficult to come back from the error. In fact what is already in the literature (even achieved very superficially) has preeminence over a new evidence (even achieved much more rigorously) contradicting it. Like if what already passed the BPR and has been published, had acquired a sacred statute. In particular if it comes from authoritative scientists. It is natural that editors call the authors of previous papers in the domain—they are certainly the most expert—as blind referees to review the new evidence. But of course these authors are also the most reluctant to admit their error and the new evidence. And all editors, who always pretend their journal is of the highest quality, level, influential, authoritative in the world, are also reluctant in admitting to have published an error. There are even high-IF journals that do not admit "Comments" to previously published articles. This to show how anti-scientific these high-IF journals are. Indeed, like at Galileo's and Delle Colombe's times, reciprocal Comments are a fundamental practice to challenge opposing theories, with the aim to achieve a more correct scientific evidence. You see that the BPR has reintroduced old anti-scientific issues, like the sacrality of an already written βίβλος, and even the Aristotelian scholars' Ipse dixit.
- Reduce the quality/validity: Editors and publishers are convinced that the BPR can evaluate the validity and the level of contributions, and so can improve the quality of their journals. But it is rather the other way around. It is difficult for non-expert editors to select the right expert referees for a given subject. And it happens frequently to receive a report from a manifestly non-competent referee. He can impose changes that he sincerely considers improvements while in fact they reduce the validity or the quality of the paper. But also in the case of a competent referee: how can a referee, who spends only that hour or two to evaluate the paper, be more competent than authors who spent months or years on the subject? But editors do completely miss this! In case of divergences between author and referee, they are not "peer", it is the latter who for the editor is the "depositary of the truth". The author must comply with the referee recommendations up to the comas: only in this way the paper will gain quality and excellence, according to the editor.
- Arbitrariness: A review strongly depends on the point of view / opinions of the referee who can come from very different domains, fields, expertises. In turn, this strongly depends on the editor's choice of the referees. When you submit, you must be ready to expect the most varied and distant reports. At the end of a review, when looking at the accumulated reports it is unavoidable to remark a large arbitrariness: you can go from extremely enthusiastic approvals, to the most scornful rejections. Like if a student received, for the very same homework, a mark of A++ from one teacher and a mark of F-- from another. The BPR is completely arbitrary. It even often happens to receive from a referee a criticism going in one direction, and from another a criticism going in all the opposite direction. Again, the editor is not at all competent to even only detect such scientific divergences between the referees. Nevertheless, he pretends you comply with both at the same time.
- Unable to detect frauds: The peer review is even unable to detect frauds. A referee is not supposed to reproduce an experiment, or redo a calculation, and so cannot check them. Simply because a referee might not have the needed tools/setups available. An example known by everybody is the Schön scandal who published on very high impact factors reviews. We recurrently hear about cases of "photoshopped molecular biology". Editors pretend these are isolated cases, but the list is already long.
- Unable to prevent publication of non-scientific papers: Very strangely, non-scientific papers were even able to escape and bypass the blind peer review. One of the most famous cases is the non-scientific article published on Lancet which was, even more surprisingly, withdrawn only 12 years later! This demonstrates that publications under BPR are submitted to large arbitrariness. Much depends on the referees who are chosen by the publisher/editor who can be submitted to different interests
- Unable to appreciate novelty/originality: We already discussed this point at the example of the Einstein-Rosen paper: a referee is not supposed to know all the papers published on a given subject, in any case no more than authors that had much more time to look for previous contributions.
But also, why should we continue to keep the BPR? Without having too much reflected, many thinks that the BPR can sincerely evaluate the quality and validate (accept/reject) a work: but, in the examples above, we have seen that no contemporary could appreciate Einstein's greatest advances, like the theory of the quanta of light or special relativity. Also the majority thinks that the BPR helps in improving a work and the manuscript clarity: but do we really need a blind peer review for that? If we really need to discuss our work, cannot we do what physicists did in the past? that is, contact a colleague that we know competent and interested, and discuss with. Instead to receive from a non-competent editor a non-expert young PhD as interlocutor/referee?
In conclusion, it is better to abandon the BPR completely and return to the previous situation when the peer review did not exist.
Is peer review affordable?
But I would like to discuss another strong argument against: Peer review is not affordable for the research system!
We all researchers are realizing more and more that the work of the reviewer can be a full time work in itself.
If publishers or research agencies pretend a minimally serious review of a paper or of a project proposed for financing, this cannot be done in a hour or two.
But we researchers are not supposed to do the work of evaluating the research of others!
We are paid to do our own work of research, and we are evaluated only on that at the end!
And research is a hard and competitive job.
We cannot waste our time with what is not our job.
Peer review has become a real burden that publishers and financing agencies are loading on our shoulders.
And they have troubles to find referees: the French ANR needs 30 contacts before to get a referee accepting to review.
Today a manuscript might have been submitted to 3, 4 even 5 journals (descending the Jacob's ladder, e.g. Nature, Science, Nature Phys., PRL, PRB, ...) before to be accepted.
And each submission requires at least two referees, with all the consequent load for the system.
Low-IF journals, in short of referees, today invite authors to submit the manuscript together with previous referee reports (of course they are not expecting to receive also the rejecting ones).
We already have large percentages of incompressible teaching and administrative work.
Today statistics indicate that in average a researcher spends only 30% max 40% of time in effectively doing research.
Euro- and other bureaucrats, who think that work is just only exchanging papers from one office to another, do not realize that the system is more and more turning around itself.
The peer review is both a waste of time and of money: both of them can be more efficiently spent in effective research.
Once I was in panel (for a LabeX) to evaluate and decide the distribution of (unfortunately less and less) research funding among presented proposals, and I wished to calculate how much 2 full days of 20 researcher directors' work costed to the system: this was more than the money we were distributing.
If most of the money and time is lost in feeding its heavy mechanisms, then the peer review is not affordable for the system.
What was the situation prior to the blind peer review?
Even the most radical proposed reforms are unable to correct the serious problems and drawbacks of the BPR listed above.
Therefore it is much better to abandon the peer review and return to the previous situation when it did not exist.
But if we remove peer review, then we are submerged by papers!
This is the first concern that colleagues express when listening to the proposal of quitting BPR.
However, they are also ready to admit that "we are already submerged by papers!"
If one looks to statistics, there is already an exponential growth of publications.
So, it seems that the peer review is ineffective in damping the growth of publications.
On the other hand, it seems that the situation was completely different prior to the '60 and to the introduction of peer review.
One year journal volumes had a completely acceptable normal size, while I cannot imagine in how much space keeps a today year volume of Physical Review B.
But here colleagues argue: yes, but there were less researchers.
True.
But it is also true that the production per researcher was also much less.
Nobel prizes like Dirac, Feynman, Fermi, ... published less than 80 papers in their whole life.
Today there are scientists, unknown if not in their field (and certainly not for an Einstenian geniality), who publish 80 papers and more per year!
Like to say that in 4 years they publish the whole Einstein corpus including non-regular articles, and Einstein was an exceptionally productive physicist.
Therefore I reverse the question:
Might it not be the case that the peer review was the cause of the explosion in the number of publications?
Rather than being slowing it down?
Indeed:
Prior to the introduction of the peer review, an author could send a manuscript to a journal and this was almost automatically accepted for publication.
This became not any more the case with the peer review, not all the papers were accepted.
This introduced the syllogism that, the better a researcher is, the better her work is evaluated by peers, and the more the published papers.
Thus, for the first time in the history of science, the peer review introduced the paradigm that the quality of a researcher can be directly measured by simply counting the publications.
But this is exactly the beginning of what today we call bibliometry.
Prior to the peer review the visiting card of a researcher, for example Einstein when he was looking for a professor position, was a couple of selected papers with what he considered the major achievements, for example the 1905 paper on special relativity.
Colleagues evaluated Einstein by entering into his papers, studying and trying to understand his theories.
Not by counting his publications.
Instead, the peer review introduced this very comfortable and simple paradigm (everybody can do it without understanding relativity), that a scientist can be evaluated by simply counting "apples and pears".
In turn this generated the next paradigm, the publish or perish, i.e. the first modern-times plague of science.
With all its potentiality to cause a boost in the number of publications.
And we all know the following of the story, with the paradigms that refined more and more by the introduction of citation indices for publications, impact factor for reviews, H-indices for researchers, etc. till the present degenerated situation.
Why prior to BPR scientists, even the greatest of them, did not publish so much?
Prior to the peer review, the "publish or perish" and other modern (degenerated) paradigms, researchers were not at all urged to publish.
The evaluation and the reputation that researchers had of each other, was exclusively based on their achievements.
Today if you publish tons of papers, and you correctly cite and are cited (or self-cited), you certainly get very good H- and other X-indices, and acquire what today is called "excellence".
At that time, if you published tons of "unworth" (as they called) papers, you only got the result to be considered a mediocre researcher.
Better to publish even only one very good achievement, worth to receive the attention of colleagues, so to be remarked and remembered for that.
And avoid to "drown" the literature and colleagues with hundred of unworth papers: indeed this could cause the lost of reputation among colleagues, they could not read you any more.
Not even when you finally published a very good achievement.
For this reason, researchers submitted a paper only if they were sure that it was not trivial and did not contain blunders.
And this at Galileo's time (e.g. Delle Colombe not publishing his leaflet Against the motion of the Earth fearing the ridicule) as well as at Majorana's:
Majorana and Dirac equations -
There are some stories about E. Majorana I heard in Rome that I report here without a historical check.
It seems that Majorana had correct understanding in advance with times and achieved fundamental contributions that he could have published, but he only published 9 papers in his life.
For example, it seems that he correctly understood that the unknown particle detected by the 1932 Joliot-Curie experiment and by them interpreted as a gamma ray, was in reality the neutral particle with the same mass of the proton called neutron which was till that time only hypothetical.
Fermi pressed him to publish an article on it, but he didn't.
Also it might be that Majorana achieved a relativistic quantum-mechanic equation for the electron, that is the Dirac equation, probably at the same time as Dirac, but considered it invalid because it presented these non-interpretable and problematic negative energy solutions.
He then continued working to solve this problem, finally achieving an equation of which he also was unsatisfied since it could only describe neutral particles uncoupled to the electromagnetic field, so not the electron.
It was only in 1937 that he published a work on what today we know as Majorana equation and leading to the prediction of the possible existence of Majorana fermions, neutral particles that are their same antiparticles.
It seems that Majorana published this work under a very strong pressure by Fermi who was aware about all Majorana studies and saw that they could have a relationship with the neutron, which was unambiguously discovered by Chadwick some years before, and in particular another hypothetical neutral particle that Fermi called neutrino (small neutron) to distinguish it from the neutron.
Majorana published it, Symmetric theory of the electron and the positron", very discretely as it were an academic exercise and without pretending to explain anything.
Self-responsibility against the explosion of publications -
These stories can be true or false.
What is important is that they are a faithful representation of the spirit of the epoch.
The reputation was the most important asset for a researcher: he could not ask again the colleagues' attention if he had already abused of their reading time with banal publications or containing trivial errors.
To give an idea to my colleagues about the spirit of the epoch: publishing was like our first seminar, with the feeling to ask the attention of colleagues and the fear to be telling banalities or trivial errors.
At that time researchers published only works scrupulously checked for any error, complete and with all problems possibly solved.
Fearing the ridicule acted as a strong mechanism preventing the publication of papers for which they did not feel safe.
Today if a paper passes the review and is published, we feel like it has been certified by the referees, and it is not any more of our responsibility.
There is not any more that past self-responsibility about what is worth and what is unworth to be published to the attention of colleagues.
And we assist to the explosion of publications.
If you aren't under the pressure of the "publish or perish" or of bibliometry, the best and most severe review of your papers is by yourself, not by your peers.
This explains also why at that time journals had the inverse problem, to close year volumes, and urged researchers to send the works in their drawers, no matter if the works were complete or not.
Nielsen mentions the case of the editor-in-chief of the journal Science who, as late as 1938, relied on personal solicitations to get more submitted articles.
If you compare with the today editorial policies of Science, you can see that it was all another world with respect to the modern degeneration!
So now, what to do?
It is clear that recovering that spirit is going to be very difficult.
Difficult to change our moods, today practices.
But if we let the system evolve along this way, it will degenerate more and more.
New and more perverse citation indices and impact factors will appear.
And when we will be at the Z-index, science won't any more be the same as Galileo's, Newton's, Curie's, Einstein's.
In principle it is very easy to make the system undergo an abrupt halt: it is sufficient to stop peer-reviewing papers and proposals.
Notice that low impact-factor (IF) reviews, even with a prestigeous past, are already experiencing serious troubles in finding referees.
It would be relatively easy to oblige these reviews to quit BPR.
But I do not consider this a "moral" way, also because the most immoral publishers are exactly those with the highest IF.
Also we cannot frustrate the dreams of the editor of the Journal of Genovian Physics (IF = 0.xyz), working hard to raise the level of his review at the highest IF.
Depriving a journal of the BPR would immediately reduce its rank to non-peer-reviewed and its rate to CCC--- by the Web of Science, Scopus, etc. research rating agencies (our Standard & Poor's, Moody's, Fitch, and not less deleterious).
Manifesto to reset the blind peer review - Therefore I propose to follow the opposite route: that one proposed in this Manifesto.
The strategy is simple: as referees we accept all papers, with only two exceptions: non-scientific articles, for example on astrology instead than astrophysics; and manifest frauds, but only with compelling proofs.
We accept all the rest, also what we consider wrong.
As explained above, errors belong to Science.
We aren't god in knowledge of the true, just only peers of the authors, they can be right and we can be wrong.
If we think there are errors, we will limit to signal them to the authors who, as peer and not inferior, could acknowledge them and spontaneously act, or disagree.
In the latter case, we will not reject the paper and respect their view and their right to publish.
This is a question of respect of our colleagues and their work.
I know that it is difficult to resist to the "Neronian pleasure" to decide for the "life or death" of colleagues' papers, but this is exactly what I am going to do and ask colleagues to follow.
Note that you believe to have this power.
The editors let you believe to have a power that in reality is in their hands.
Today editors are also submitted to the ineluctable rules of bibliometry, they must publish papers safely collecting citations to increase, or at least mantain, their IF.
Therefore, they are not interested in controversial "theories of quanta of light" or incomprehensible "theories of relativity".
They prefer simple (only apparently great) discoveries along the mainstream, so acceptable and compehensible by everybody, and they actively work for this selection.
The editors of Nature and Science reject already an 80% of papers before sending them to referees!
Then they have the further important power to decide the referees.
And it is not difficult to understand which referees are the severe, the bad, and which are the good, so to appropriately select the good for the only 5% of papers they would like to publish.
I even heard they put pressure on a "good" referee that turned out "bad", so to force him to accept.
We are only puppets in their hands.
So who is really deciding it's not you, the referee: you are just only a little Nero!
Moreover, in contrast to what editors ingenuously believe, it is impossible to assess at short term the future impact of a work, whether it will pave a new avenue or open a new research area, how influential is going to be, whether it is pivotal in solving a problem of subjective criticality or providing a significant contribution, more in general how important, novel, authoritative, or high-quality a work is.
Consequently, we will refrain to state any assessment about the level or quality of a work.
Finally, we will not impose to our colleagues any amendment, change, censure, retractation, abjuration of their view in favour of our theories or our publications.
This is question also of respect of the fundamental Author's Rights, as decreed by the Bern Convention and international Copyright agreements.
Scientific literature is submitted to Author's Rights exactly as ordinary literature.
An author has the full right to held a text exactly as he wrote.
The Bern Convention, under the auspices of Victor Hugo, stated all these Author's Rights and it was an important step forth for humanity.
An editor cannot impose Victor Hugo to change the conclusion of his "Les Misérables", towards a more Hollywoodian happy "The End", because he likes more, or it is more favourably accepted by the readers and it sells more.
The same right holds for scientific authors.
This proposal will restore the past situation where all papers were accepted, though without the old self-control mechanism, at least at the beginning.
I expect a transition time where the number of publications will grow even more.
Every researcher will see the possibility to boost his bibliometry indices.
But this is also an aim of the Manifesto: to make useless so to break all the actual bibliometry schemes to evaluate researchers.
You see that such a Manifesto could be the solution to many of the plagues afffecting scientific research.
When it will be evident that research and researchers cannot be measured by the bibliometry, then the only possibility will be to evaluate them by studying and understanding their achievements, as it was at Einstein's time.
Enter into papers and try to understand the theory of relativity, instead than looking at an index automatically provided by a WoS or a Scopus server.
At this point researchers will stop to do big volume, and will start to do better science.
And they will stop submitting papers frantically.
Thierry Grenet (CNRS, Institut Néel)
Dear Valerio,
I read your home and BPR pages.
Although I'm still inclined to think that article peer reviewing has some beneficial aspects (I think it often pushes/helps authors to be more clear), I believe like you that it may also has many drawbacks. I think you made very good points, several of which I hadn't thought of before, or not the same way you do. Your analysis is controversial but very thought provoking. Great !
As for the explosion of paper "production" (unfortunately this has become the proper word), it is in principle very simple to stop : financing and evaluating agencies should ONLY consider say the five most significant papers of each scientists, which evaluators would have time to read. The zillions of other papers and dozens of indices computed from them would become useless, hence would disappear.
Of course this will not happen as long as we researchers let the bureaucrats come and decide for us, which is exactly what we did during the latest decades.
Nicola Manini (University of Milan, Physics Department)
Dear Valerio,
I fully agree with your considerations.
I have a general comment however: as we are all well aware, and you mention in the last part of this essay, editors already discard a significant percentage of submitted papers. This applies to "vip" journals such as Science, but also to more down-to earth journals, e.g. Phys. Rev. B.
To editors, abolishing peer review may easily mean replacing it with 100% editorial review. As a result, editors will choose what to publish and what to dump, without any influence from scientists.
With the good/bad reviewer trick, in practice this is what happens already. Removing BPR will only make this clear and transparent.
I agree with Thierry Grenet's comment that judgment on a scientist should be based on her/his 5 best papers only (if universally adopted, of course!) will be the end of the publish-or-perish frenzy.
Ciao,
Nick