More Labels or Cases? Assessing Label Variation in Natural Language Inference

Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, Göran Kauermann, Barbara Plank


Abstract
In this work, we analyze the uncertainty that is inherently present in the labels used for supervised machine learning in natural language inference (NLI). In cases where multiple annotations per instance are available, neither the majority vote nor the frequency of individual class votes is a trustworthy representation of the labeling uncertainty. We propose modeling the votes via a Bayesian mixture model to recover the data-generating process, i.e., the “true” latent classes, and thus gain insight into the class variations. This will enable a better understanding of the confusion happening during the annotation process. We also assess the stability of the proposed estimation procedure by systematically varying the numbers of i) instances and ii) labels. Thereby, we observe that few instances with many labels can predict the latent class borders reasonably well, while the estimation fails for many instances with only a few labels. This leads us to conclude that multiple labels are a crucial building block for properly analyzing label uncertainty.
Anthology ID:
2024.unimplicit-1.2
Volume:
Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language
Month:
March
Year:
2024
Address:
Malta
Editors:
Valentina Pyatkin, Daniel Fried, Elias Stengel-Eskin, Alisa Liu, Sandro Pezzelle
Venues:
unimplicit | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
22–32
Language:
URL:
https://aclanthology.org/2024.unimplicit-1.2
DOI:
Bibkey:
Cite (ACL):
Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, Göran Kauermann, and Barbara Plank. 2024. More Labels or Cases? Assessing Label Variation in Natural Language Inference. In Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language, pages 22–32, Malta. Association for Computational Linguistics.
Cite (Informal):
More Labels or Cases? Assessing Label Variation in Natural Language Inference (Gruber et al., unimplicit-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.unimplicit-1.2.pdf