Published November 19, 2013
I received a number of offline comments about my last post, in addition to the commenters on the post itself. By and large, it seems that there is an interest in some sort of mechanism to check citations, so I have put a little pilot program into effect at Evolution and Human Behavior.
Here’s how I’m currently structuring it. I received a paper that is in its third iteration. After studying the new manuscript, I decided to accept it. I have delayed officially accepting it, however, and sent the manuscript to someone who has agreed to check the citations in exchange for an hourly wage. My employee is not a graduate student—so I am not taking her away from scholarly pursuits—but does have an undergraduate degree and has already shown competence, so I trust her to do a good job.
The economics are somewhat alarming. Suppose that I’m paying $12 per hour. She estimates it takes about 20 minutes to track down and verify a citation. So each citation costs $4 to check. If a paper has, say, 50 citations—which is often on the low side—then this process, conservatively, costs $200 per paper. If the journal publishes 50 papers per year, the annual budget would be $10,000, a tidy sum. Add in review papers, which might have three times as many cites, and the figure potentially increases substantially.
Despite this expense, I decided it would be worthwhile to conduct a pilot program along these lines. My expectation is to have a few papers checked in this way every month, as opposed to having all of them checked. There is some money in one of my budgets for this, so I can at least fund the pilot program. One question is how many cites are questionable. Maybe it’ll turn out that the answer is basically zero, and so the whole thing was more or less a tempest in a teapot. (This would still leave open the issue of whether the field should introduce pincites. I’m genuinely curious about this. Drop me a line to tell me if you’d prefer the present world, or the world in which you have to supply page numbers in your citations. A pain for authors, but a boon for readers.)
There is another cost beyond the labor. Instead of the paper in question going right to production, there will now be a lag between when the paper is accepted and when it goes to press. If my cite-checker finds a number of potentially incorrect cites, then there is another iteration of the manuscript introduced into the process, which might well be vexing for the author or authors.
Throwing money at the problem is not unreasonable, I think, for the moment, but I’m not sure that it’s sustainable over the long term or, even if it were, if the money spent this way would not be better spent on other things.
Having said that, my sense is that the law school solution is not really an option. I don’t think that it will, any time soon, improve a student’s reputation or job prospects to perform such tasks. The law school regime—work on the law review to get a clerkship—has no obvious equivalent in psychology as currently constituted, as far as I can tell.
Other ideas? A bond system? Should authors put up a $300 bond when they submit, which they get back if all the citations are correct? Would authors be willing to do this, or simply migrate to another outlet? A simple fee, as in the PLoS family of journals, to defray the cost? This seems to add insult to injury, given the existing resentment about Elsevier and its unseemly profits. A tax on members of the Human Behavior and Evolution Society? E&HB is the journal of the Society, and the Society benefits to the extent the journal is better. Consider it a public good produced by the membership? What about coercing advanced undergrads? We already more or less force them to participate in experiments in the name of pedagogy. Checking citations is educational, right? When people come to me to ask to volunteer in my lab, which is not too infrequently, do I stop turning them away, and say, yes, as a matter of fact there is something you can do to help ... ? It might take them longer, and they might not be as adept as a grad student would be, but the price sounds great ...
Still open to ideas and discussion. The pilot program is under way ...
