10 January, 2009

Why one needs peer review?

A few days ago Michael Nielsen published an excellent post "Three myths about scientific peer review". Briefly, in the nonscientific (and sometimes in the scientific) community it is usually supposed that:

i) Peer review system was used since very long time ago. Actually, it was not. For instance, Albert Einstein, while living in Germany, was not used to "independent referees" - in such journals as Zeitschrift für Physik the decision "whether to publish a paper or not" was always made by editors. And, as far as I know, before WWII this journal was considered as much better place to publish than Physical Review. After moving to the US, Einstein had withdrew one of his contributions from the Physical Review, because he "has not authorized the editor to show it to specialists before it is printed". I wrote a little post about this story before.

ii) It is reliable, in other words "it nearly always picks up errors, is a very accurate gauge of quality, and rarely suppresses innovation". Of course, no one can predict the future, and say which piece of work will be a "citation classic" in ten years. Furthermore, there is probably no way to discuss the reliability of peer review since we simply don't know how many revolutionary papers were never published (or were, but in some obscure journals) due to negative referee reports.

and

iii) It can show what is right and what is wrong in science. This is also not true, because it is impossible to check all math in the manusript, redo computations, not to mention repeating experimental results.

The post of Michael was followed by interesting comments of Daniel Lemire, "The purpose of peer review", induced by another comment of Peter Turney to Michael's post: "I’m sympathetic to much of what you’re saying, but, on the other hand, I know that peer review has immensely improved many of my own papers."

Being nothing but graduate student, I absolutely agree with Daniel and Peter: at least two from three papers, written during the first year of my PhD, were substantially improved after referee's comments. Apart from submitting  the papers, we have always sent preprints to some good friends from our field - always getting positive feedback (some of them may be the referees in the same time, but this we don't know).

In my opinion, the fact of peer review doesn't mean that the article is reliable, correct or "interesting to a wide community of scientists" (as it should be e.g. for Nature and Phys. Rev. Lett.). But, the review process is some kind of necessary (but not the sufficient!) condition for paper to be read and cited in the long-time future. After publication the scientific community will make it clear, whether such a condition was actually sufficient - there will be a lot of readers  and time for it, much more than two referees having three weeks to submit a report.

Of course, no one can read all journals in his field, even if the field is small enough. Let alone that articles show what was done about one year ago, but not now. Almost no one is actually reading journals - conferences and arXive give a perfect taste of what is going on. But if we come across an arXive preprint dated back to 2007 without any journal reference - this looks strange, doesn't it?

And last, but not least, it seems that reviewing good papers is an excellent way to learn how to write good papers.

2 comments:

Daniel Lemire said...

There are over 500,000 arxiv papers. I am sure many of them never appeared in conference proceedings or journals. However, some of these "unpublished" papers are nevertheless useful and sometimes very significant.

Sometimes the author decides not to submit the paper to peer review. If the author is a tenured professor, this can often be a luxury he can afford. Other times, the paper just would not fit in a journal or a conference. For example, some topics are too technical to warrant peer review.

Lemeshko said...

This is true. However, in our relatively small field (molecular physics), preprints are most likely followed by a paper. I absolutely don't know what is going on in other topics, as e.g. computer science...