Monday, October 08, 2018

Sokal on steroids: Why hoax papers submitted successfully to academic journals proves nothing (ok, little)

A trio of authors has, during a 12 months period, submitted - by their own account - 20 manuscripts to academic journals they broadly identify as being in disciplines or fields of study associated with 'academic grievance studies'. From what I gather they are not too terribly fond of social constructionist colleagues in their own academic disciplines, and presumably other disciplines. Let me say at the outset that I am sympathetic to some of their concerns. They rightly lament that what they broadly label as 'academic grievance studies' has led to thought crimes prosecutions of sorts (just think about the Hypatia controversy involving an article by Rebecca Tuvel - mob justice might be one way to describe what happened to Tuvel). Ironically, Hypatia is again among the offending journals, which is interesting in so far as it isn't a journal entirely dedicated to publishing social constructivist content, another bugbear of the initiators of the Sokal-on-steroids hoax.

In any case, they apparently spend their while producing 20 hoax articles that they planned to submit to top journals in different disciplines or areas of study that they find disagreeable. I am genuinely curious whether this research project was submitted to their institution's ethical review committee, because it uncontroversially involved involuntary human research participants (identifiable journal editors, reviewers).

Here's the result of their efforts: Of these 20 papers 7 were accepted, 6 were rejected outright, and the rest were somewhat in-play, including a number that received a revise and resubmit verdict. The hoax initiators claim that two papers received a verdict of 'revise and resubmit' that they took to mean 'usually results in publication.' For what it's worth, at the journals I co-edit this verdict means renewed external review with the possibility of an outright rejection. So, no, not 'usually results in publication.' I wonder whether this interpretation is self-serving in so far as they needed to, of course, bolster their case as good as they could.

At the end of the day, they had 20 submissions, 7 acceptances.

I agree with the team of hoaxters that this constitutes egg on the faces of the editors of those journals, and more so, on their reviewers' faces. It's embarrassing.

I read a few comments on this project along the lines that in STEM subjects journals also face large numbers of retractions each year, and so it's not surprising that terrible content also passes peer review in humanities' areas. While that is true, it shouldn't distract from the fact that well-established, influential journals were coaxed here into accepting garbage. Of course, that matters!

My problems really lie elsewhere. One is that a study where n=20 doesn't demonstrate that a particular discipline has methodological problems. It simply means that - when all is said and done -  7 crappy papers were accepted by non-specialist journal editors based on their reviewers' recommendations. Big whoop! Frankly, this shows us that on this occasion reviewers failed. On 7(!) occasions. We do not know whether the same would have transpired if hundreds of such papers had been submitted to the same journals. It's one such paper per journal. Talking anecdotal, this is as anecdotal as it gets.

Let me be honest here, as an editor of journals that rarely if ever publish social constructionist papers, I could also be accepting papers that succeeded in fooling our peer reviewers. I don't have the subject expertise to be confident in evaluating all the manuscripts that are submitted to my journals. I rely on - these days - mostly reluctant reviewers who hopefully provide me with good (sometimes excellent, detailed) comments and recommendations. I rely on competent reviewers being diligent. I rely on colleagues I ask to review a manuscript to come back to me if they do not consider themselves competent. I rely on unpaid reviewers spending a considerable amount of time doing their job, when they could spend that time writing their own papers, or on a grant application, or a job application, or they could simply spend time with their loved ones. Not all of them, all the time, deliver a reliable review. Big whoop. If a faulty paper gets published (and it hasn't happened to us yet), I'm confident, over time responses to such a paper would eventually show where the paper we published went wrong, and why. A case in point is this Editorial I wrote. One of the articles flagged there was making false empirical claims, it has since been retracted. The article in question passed external peer review, but it is clear that the reviewers took the empirical claims made by the authors to be true, referenced as they were. It turns out that that was a mistake. 

All of that happens without hoaxters wasting my time and that of our reviewers.

The hoaxter trio takes these 7 accepted manuscripts as evidence for the methodological failings of what they refer to as 'social constructivism'or 'radical constructivism'. They apparently 'corrupt' scholarship.

The thing is, that might well be true, and social constructivism scholarship is truly a naked emperor. Unfortunately, pointing to 7 anecdotal papers as evidence that that is the case, is plain ludicrous. As far as I can see, there wasn't even a control group (say, 20 manuscripts submitted to analytical journals, 20 manuscripts submitted to STEM subjects).

To my mind, if you wish to criticize social constructivism (and much critical that I wholeheartedly agree with has been published over the last few decades) don't avail yourself of childish activities like these kinds of hoaxes. Show by means of analysis and argument that, and why, the social constructivism emperor is naked.

The process of producing and publishing peer reviewed academic content relies on an assumption of good intent and genuineness among authors. Obviously, this is not justified in all cases, that's where critical responses and retractions come in handy. The last thing needed is a cottage industry of 'gotcha authors' like our hoaxter trio. They could and should have spend their time producing one sound academic paper taking on social constructivism, placing it in a top-notch journal, and subsequently enjoying the fireworks of rebuttal and response. That's how progress in the academy is facilitated.

I have sat on this for a good week, because I wasn't sure whether I should bother writing a response. On the one hand I share many of their concerns, on the other hand, this was such a time wasting pointless exercise, it boggles the mind smart people would have resorted to that sort of thing.

No comments:

Post a Comment

Note: only a member of this blog may post a comment.

Certainty is not a defensible standard for policy making in the context of assisted dying

I mentioned in a Bioethics editorial a while ago that new frontiers are opening in the assisted dying debate. As an increasing number of...