<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-etal-2023-exploring-contrast">
    <titleInfo>
        <title>Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zhihan</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wenhao</namePart>
        <namePart type="family">Yu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zheng</namePart>
        <namePart type="family">Ning</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mingxuan</namePart>
        <namePart type="family">Ju</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Meng</namePart>
        <namePart type="family">Jiang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR’s contrast consistency is improved without sacrificing its accuracy on the standard test sets.1</abstract>
    <identifier type="citekey">zhang-etal-2023-exploring-contrast</identifier>
    <identifier type="doi">10.1162/tacl_a_00591</identifier>
    <location>
        <url>https://aclanthology.org/2023.tacl-1.61</url>
    </location>
    <part>
        <date>2023</date>
        <detail type="volume"><number>11</number></detail>
        <extent unit="page">
            <start>1082</start>
            <end>1096</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions
%A Zhang, Zhihan
%A Yu, Wenhao
%A Ning, Zheng
%A Ju, Mingxuan
%A Jiang, Meng
%J Transactions of the Association for Computational Linguistics
%D 2023
%V 11
%I MIT Press
%C Cambridge, MA
%F zhang-etal-2023-exploring-contrast
%X Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR’s contrast consistency is improved without sacrificing its accuracy on the standard test sets.1
%R 10.1162/tacl_a_00591
%U https://aclanthology.org/2023.tacl-1.61
%U https://doi.org/10.1162/tacl_a_00591
%P 1082-1096

Markdown (Informal)

[Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions](https://aclanthology.org/2023.tacl-1.61) (Zhang et al., TACL 2023)

• Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions (Zhang et al., TACL 2023)

ACL

• Zhihan Zhang, Wenhao Yu, Zheng Ning, Mingxuan Ju, and Meng Jiang. 2023. Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions. Transactions of the Association for Computational Linguistics, 11:1082–1096.