How Well Can an Agent Understand Different Accents?

We evaluate several state-of-the-art automatic speech recognition systems on dialogue agent-directed English speech from speakers with General American vs. non-American accents. Our results show that the performance of the speech recognizers for non-American accents is considerably worse than for General American accents, with approx. 20 percent higher word error rate on average (relative difference). This work indicates a need for more diligent collection of and training on non-native English speaker data in order to narrow this performance gap. There are performance differences across recognizers, and while the same general pattern holds, with more errors for non-American accents, there are some accents for which the best recognizer is different than in the overall case. We expect these results to be useful for dialogue system designers in developing more robust inclusive dialogue systems, and for speech recognition providers in taking into account performance requirements for different accents.