Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems

Jindřich Helcl, Barry Haddow, Alexandra Birch


Abstract
Efficient machine translation models are commercially important as they can increase inference speeds, and reduce costs and carbon emissions. Recently, there has been much interest in non-autoregressive (NAR) models, which promise faster translation. In parallel to the research on NAR models, there have been successful attempts to create optimized autoregressive models as part of the WMT shared task on efficient translation. In this paper, we point out flaws in the evaluation methodology present in the literature on NAR models and we provide a fair comparison between a state-of-the-art NAR model and the autoregressive submissions to the shared task. We make the case for consistent evaluation of NAR models, and also for the importance of comparing NAR models with other widely used methods for improving efficiency. We run experiments with a connectionist-temporal-classification-based (CTC) NAR model implemented in C++ and compare it with AR models using wall clock times. Our results show that, although NAR models are faster on GPUs, with small batch sizes, they are almost always slower under more realistic usage conditions. We call for more realistic and extensive evaluation of NAR models in future work.
Anthology ID:
2022.naacl-main.129
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1780–1790
Language:
URL:
https://aclanthology.org/2022.naacl-main.129
DOI:
10.18653/v1/2022.naacl-main.129
Bibkey:
Cite (ACL):
Jindřich Helcl, Barry Haddow, and Alexandra Birch. 2022. Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1780–1790, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems (Helcl et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.129.pdf
Video:
 https://aclanthology.org/2022.naacl-main.129.mp4
Data
WMT 2014