Debates about retraction, controversy, and academic freedom should acknowledge base rate methods quality

Should academic articles be retracted because they make controversial claims? This question drives debate within academia, journalism, and politics, often centered on the question of academic freedom. Academic freedom gives scholars the ability to pursue research that might not have a clear immediate “payoff” or that might be socially controversial. Articles sometimes gain much more attention than the average publication when they come to be seen as making controversial claims. For example, a recent controversy erupted over a paper arguing Korean women voluntarily entered into contracts with Japanese soldiers for sex work [New York Times coverage]. When such articles are retracted, some claim academic freedom has been undermined.

Conventional wisdom states articles should only be retracted when their claims are unsupported by the data and analysis reported in the paper, such as coding errors that lead to substantive changes in results, such as happened with a famous paper in economics. The worst-case scenario involves outright fraud, such as scholars making up their data to achieve publications needed for promotion. Recent examples of fraud happened in political science and management research.

However, there is no reason to believe that controversy and methods errors are independent. Here I argue that the debate about “silencing academic freedom” by retracting controversial articles tends to ignore the baseline methods quality in academic fields. The main argument is that, if a field has a low baseline of methods quality, any paper that attracts considerable controversy will also be found lacking in methods rigor, leading most controversial papers to be retracted. This lends the appearance that “controversial” papers are almost always retracted and ignores that the papers are retracted because they are found to have serious methods errors that make their claims unsupported by the data and analysis.

The following table shows how this works:

Four possible combinations of controversy and methods quality for a published study.

Those who claim controversial article retraction undermines academic freedom generally assume all articles are methodologically sound (Box 1). Such papers should not be retracted if retraction is based on methods problems.

However, what if fields of study vary in the baseline rate of methods quality of published papers?

Begin with the extreme case where 100% of published articles have unsound methods. This is, we would hope, unrealistic. In that case, any paper that attracts consideration for retraction should be retracted, and the only thing keeping papers from being retracted is a lack of attention to whether they should be retracted. If controversy inspires consideration for retraction, all controversial papers in this field would end up being retracted, and it would appear that controversy always leads to retraction. Arguments that there is no academic freedom in that field would likely arise.

Now consider the alternative extreme case where 100% of published articles have sound methods. Here, any controversial paper getting retracted would be a legitimate undermining of academic freedom. This is the situation generally assumed by critics of retraction that appears to be driven by controversy, not methods quality.

Now imagine a more realistic case where 70% of published papers have sound methods and 30% have methods that should have led to rejection and would lead to retraction if the paper were ever considered for retraction based on methods. In this field, some papers that attract controversy will have sound methods, and some will not. Some controversial papers should be retracted based on methods, and some should not.

This is a more realistic assumption about the baseline rate of methods quality in a field, though I’m sure it varies greatly across academic fields. What if some fields have 70% unsound methods? Such fields will appear to retract most controversial articles, which would make the field appear unwilling to defend academic freedom by assessing papers on methods, not how controversial claims are from the methods.

Debates about the relationship between academic freedom, controversy, methods, and retraction would be more productive if they incorporated thought about the baseline rate of methods quality in a field before passing judgment on whether any consideration of retraction after controversy constitutes an attack on academic freedom. This has the potential to reduce dogmatism in retraction arguments. It can also improve the baseline methods quality in a field through increased attention to methods and the research produced in that field, which would be a better outcome for scholars and others interested in the research.