QUALITY CONTROL

SPECIAL ISSUE 1999

Some scientists overlook the normal peer review procedure in science when it comes to the publication of their results and misuse the mass media to make them public. As a result, undeserved importance is given to certain scientific results to the detriment of other, perhaps more relevant findings for science. What mechanisms are available to the scientific community to defend astronomy from fraud?

"I view the emphasis on public outreach in the last few years as positive, and I think by and large the press does a good job in reporting. Most science reporters I’ve met are very knowledgable about astronomy, ask inciteful questions about the broad relevance of a particular result, and have a good sense of what professional astronomers think is significant or not. In fact most science reporters spend a lot of time soliciting quotes from experts in the field verifying the importance of the results they report. In my experience, professional astronomers and journal editors are much more tolerant of quantized redshifts, non-cosmological quasar redshifts, etc. than the popular press."

"It is unfortunate that this practice has become so common in the last few years in some of the scientifically most advanced countries. Press releases seem now the standard way to announce some promising results, even if they have not been confirmed, or analysed carefully. I think that the scientific community should educate the general public, the press, the funding agencies, the governments, and the peer review committees, on what are the appropriate procedures to establish and communicate scientific findings. I think that in most cases this excessive protagonist role of some astronomers is fuelled by the need to obtain additional or continuing support for their research projects. It is equally or more unfortunate that the funding agencies pay more attention to the visibility of some scientists in the mass media than to real scientific research papers. The scientific community should fight for a revision of the criteria used by funding agencies when evaluating research proposals to avoid this type of fraud."

"I still believe that peer review is a fundamental backbone of the work that we do, and although there have been some "cracks" appearing (e.g. due to rapid electronic dissemination of unrefereed, pre-publication results), those have some benefits as well as drawbacks. The Hubble Deep Field data and much of the follow-up work, for example, was widely and rapidly circulated among the members of the astronomical community, and this led to extremely fertile sharing of information and rapid progress. I’m not pessimistic about this — I think peer review is still strong, and I think that active efforts to communicate scientific results to the wider public via the media are a good thing. Of course, it’s easier for me to feel ok about this in astronomy than I might if I worked in, say, medicine, where people’s lives and health are at stake! There, irresponsibility can have much greater consequences."

"Over-enthusiasm in publicising results ahead of peer-review cannot be eradicated at the informal level and is a healthy part of scientific debate. If someone presents a new result at a conference, they have made a judgement that they are willing to stake their reputation on it. This argument can be extended to include the submission of articles onto preprint servers such as astro-ph but the consequences are then more profound. As written articles, others cite them as the latest development (often ignoring previous history in the subject) and journalists download unreviewed figures for use in popular articles. Science can often only be understood by the public when the story is simplified so, inevitably, things get garbled and sometimes deserving scientists get offended.

However, I don’t think I’d feel comfortable with imposing restrictions on what people can put on astro-ph; good practice should follow by example and understanding rather than by establishing a set of rules. We have to be more balanced in our dealings with the media and journalists should distribute their nets wider to get informed views rather than just believing the author. Press releases from national organisations (where there may have been some peer review) should carry more weight than those from individual departments. This happens most of the time and I do think there’s a tendency to only remember the inflammatory cases.

We should count ourselves lucky in astronomy that it’s only the individual scientists that occasionally suffer. In other areas where the general public is involved the fall-out from bad practice can be terrifying, as we saw in the UK recently with unrefereed articles on the dangers of genetically-modified food."

"In addition to the problem you mention, maybe another potential one could be related with the widespread use, through Internet, of electronic databases of preprints, where anyone can publish anything with no control whatsoever: in principle, this is a useful and positive contribution to the dissemination of scientific results, in practice this might become a problem when (few) high quality results would be mixed in a plethora of irrelevant papers (the former may be easily and frequently missed). Also related problems arise when economic or political considerations influence the quality of the scientific publications, e.g. when 20-30 publications per year are needed by a single to keep or to improve its career: unless he has a full lab dedicated to himself (as sometimes occurs!), there may be a risk of unchecked results or sometimes even errors. In principle, even some journals might be tempted to lower the standards of quality for the published papers, then increasing the number of issues and the benefits...

Certainly part of these problems are alleviated by the use of tools like the "Citation Index" as a measure of scientific quality. However, even this is not guaranteed to be unaffected by psychological effects amplifying some publications far above their value and forgetting for some others. Another way to keep high standards for astronomical publications would be to have a limited number of high level journals, where the peer review is working particularly well and efficiently, for example by doubling the default number of referees per article. Such journals should publish, also electronically, a limited number of papers, whose high-level standards are somehow guaranteed. These journals should be as international as possible, and should be partly funded also by international astronomical institutions (like the IAU, if feasible). The return for the astronomical community of this funding effort could be not less than a new telescope put into operation."

"I don’t think much of "the normal peer review procedure" when it comes to the publication of science. If some competent astronomer seriously wanted to perpetrate a research fraud, I don’t believe that the normal peer review process would pick it up, any more than it can pick up a deep seated or subtle mistake.

I am one of several editor of a journal, and my experience is that papers fall pretty much into two classes. Class 1 is real rubbish, which is obvious to almost any experienced person on a quick reading. These are the papers that get quickly rejected. The other class contains the publishable papers. Some are very good as they stand. Others are improved by the refereeing process (shortened, arguments tightened, obvious mistakes corrected) but would be OK even if they weren’t refereed. And others are rejected because they are not so interesting but would do no harm if they were published.

So what does the refereeing process really achieve ? It takes a lot of time of active researchers, delays publication, and slightly raises the level of the papers that get published. A lot of weak stuff gets through anyway. I think the main practical use of the refereeing process is to provide bars for people to jump over, in order to establish their competence and so make themselves employable.

Does it matter if there are weak papers in the literature, or even if some frauds get through? And does it matter if someone announces a "great discovery" to the New York Times rather than to the ApJ? I think not. In my view, the scientific community is quite capable of assessing its input. Remember the cold fusion saga? That was the scientific community at its best, I think. Something very interesting and important but somewhat implausible was announced by people with excellent credentials. The community dealt with it in a scientific manner. I don’t believe that the normal peer review process would have contributed much to improving the way that cold fusion was handled."

"I think astronomy is less prone to this type of aberrant behaviour than some other branches of science, and in my experience it is more often the astronomer who suffers at the hands of the media than the situation applying in reverse. As a general rule the astronomical results that reach the newspapers are worthy of wide exposure, although it is less true that the truly important astronomical results necessarily reach the public via the media. This is at least partly because in contacts with the media, astronomers are normally better at publicising their own results than highlighting perhaps more significant results in their field. However, ‘fraud’ remains very rare in astronomy, and this happy situation has been maintained despite the dramatic growth in electronic preprint servers which can be used to disseminate astronomical papers to a wide audience without passing through a referee. Flawed as it is, I am a great believer in the peer review process, and side-stepping it is a common cause of unnecessary grief: it has certainly saved me from embarrassing mistakes on more than one occasion. It would be great if all astronomers stuck to the principle of getting their papers read critically by independent and informed referees before widespread circulation."