Byungkyu Kang


2023

pdf bib
Unsupervised Improvement of Factual Knowledge in Language Models
Nafis Sadeq | Byungkyu Kang | Prarit Lamba | Julian McAuley
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. In this work, we propose an approach for influencing MLM pretraining in a way that can improve language model performance on a variety of knowledge-intensive tasks. We force the language model to prioritize informative words in a fully unsupervised way. Experiments demonstrate that the proposed approach can significantly improve the performance of pretrained language models on tasks such as factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting.