You Can't Quarantine the Truth: Lessons Learned in Logical Fallacy Annotation of an Infodemic

Given the current COVID-19 infodemic that crosses multiple genres of text, we posit that flagging potentially problematic information (PPI) retrieved by a semantic search system will be critical to combating mis- or disinformation. This report describes the construction of a COVID-19 corpus and a two-level annotation of logical fallacies in these documents, supplemented with inter-annotator agreement results over two development phases. We also report a preliminary assessment of the corpus for training and testing a machine learning algorithm (Pattern-Exploiting Training) for fallacy detection and recognition. The agreement results and system performance underscore the challenging nature of this annotation task. We propose targeted improvements for fallacy annotation and conclude that a practical implementation may be to report a documents overall fallacy rate as a measure of its credibility.