Paired Model Evaluation of OCR Algorithms

Characterizing the performance of Optical Character Recognition OCR systems is crucial for monitoring technical progress predicting, OCR performance, providing scientific explanations for system behavior and identifying open problems. While research has been done in the past to compare the performances of OCR systems, all methods assume that the accuracies achieved on individual documents in a dataset are independent. In this paper we argue that accuracies reported on any dataset are not independent and invoke the appropriate statistical technique -- the paired model -- to compare the accuracies of two recognition systems. We show theoretically that this method provides tighter confidence intervals than the methods used in the OCR and computer vision literature. We also propose a new visualization method, which we call the accuracy scatter plot, for providing a visual summary of performance results. This method summarizes the accuracy comparisons on the entire corpus while simultaneously allowing the researcher to visually compare the performances on individual document images. Finally, we report on the accuracy and speed performances as functions of image resolution. Contrary to what one might expect, the performance of one of the systems degrades when the image resolution is increased beyond 300 dpi. Furthermore, the average time taken to OCR a document image, after increasing almost linearly as a function of resolution, suddenly becomes a constant beyond 400 dpi. This behavior is most likely because the Sakhr OCR algorithm resamples the high-resolution images to a standard resolution. The two products that we compare are the Arabic OmniPage 2.0 and the Automatic Page Reader 3.01 from Sakhr. The SAIC Arabic dataset was used for the evaluations. The statistical and visualization methods presented in this paper are very general and can be used for comparing the accuracies of any two recognition systems, not just OCR systems.