Roman Yurchak


2018

pdf bib
Stop Word Lists in Free Open-source Software Packages
Joel Nothman | Hanmin Qin | Roman Yurchak
Proceedings of Workshop for NLP Open Source Software (NLP-OSS)

Open-source software packages for language processing often include stop word lists. Users may apply them without awareness of their surprising omissions (e.g. “hasn’t” but not “hadn’t”) and inclusions (“computer”), or their incompatibility with a particular tokenizer. Motivated by issues raised about the Scikit-learn stop list, we investigate variation among and consistency within 52 popular English-language stop lists, and propose strategies for mitigating these issues.