I have just finished reviewing two papers for possible publication. As you all know, this is hard work. You gotta read the thing, think about it rationally, judge it along several dimensions concluding in an overall assessment, and then write this all up in comprehensible, and hopefully, helpful prose. I find any one of these activities exhausting, and I thus don't count reviewing as among my top 10 favorite pastimes. Given my distaste, I have begun to wonder how valuable all this expended effort is. After all, it's worth doing if the process has value even if it is a pain (sort of like dieting or regular exercise). So is it and does it?
Sadly, this has become less and less clear to me of late. In a recent post (here) I pointed you to some work aimed at evaluating the quality of journal publications. Yesterday, I ran across a paper published in Plos Biology (here) that, if accurate, suggests, that the return on investment for reviewing is remarkably slight. Why? Because, as the authors ( Adam Eyre-Walker and Nina Stoletzki: (E-WS)) put it: "scientists are poor at estimating the merit of a scientific publication" (6). How poor? Very. Why? because (i) there is very low intersubjective agreement after the fact on what counts as worthwhile, (ii) there is a a strong belief that a paper published in a prestige journal is ipso facto meritorious (though as E-WS show this not a well justified assumption), and post hoc citation indices are very nosiy indicators of merit. So, it seems that there is little evidence for the common assumption that the cream rises to the top or that the best papers get published in the best journals or that the cumbersome and expensive weeding process (i.e. reviewing) really identifies the good stuff and discards the bad.
Now, linguists might respond to this by saying that this all concerns papers in biology, not hard sciences like syntax, semantics and psycholinguistics. And, of course, I sympathize. However, more often than not, what holds in one domain has analogues in others. Would it really surprise you to find out that the same holds true in our little domain of inquiry, even granted the obvious superior intellect and taste of generative grammarians?
Say this is true, what's to be done? I really don't know. Journals, one might think, play three different roles in academia. First, they disseminate research. Second, they help direct the direction of research by ranking research into different piles of excellence. Third, they are used for promotion and tenure and the distribution of scarce research resources, aka grants. In this internet age, journals no longer serve the first function. As for the second, the E-WS results suggest that the effort is not worth it, at least if the aim is to find the good stuff and discard the bad. The last then becomes the real point of the current system: it's a kind of arbitrary way of distributing scarce resources, though if the E-WS findings are correct, journals are the academic equivalent of dice, with luck more than merit determining the outcome.