<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="varda-marelli-2023-data">
    <titleInfo>
        <title>Data-driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Andrea</namePart>
        <namePart type="given">Gregor</namePart>
        <namePart type="family">de Varda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marco</namePart>
        <namePart type="family">Marelli</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Massively multilingual models such as mBERT and XLM-R are increasingly valued in Natural Language Processing research and applications, due to their ability to tackle the uneven distribution of resources available for different languages. The models’ ability to process multiple languages relying on a shared set of parameters raises the question of whether the grammatical knowledge they extracted during pre-training can be considered as a data-driven cross-lingual grammar. The present work studies the inner workings of mBERT and XLM-R in order to test the cross-lingual consistency of the individual neural units that respond to a precise syntactic phenomenon, that is, number agreement, in five languages (English, German, French, Hebrew, Russian). We found that there is a significant overlap in the latent dimensions that encode agreement across the languages we considered. This overlap is larger (a) for long- vis-à-vis short-distance agreement and (b) when considering XLM-R as compared to mBERT, and peaks in the intermediate layers of the network. We further show that a small set of syntax-sensitive neurons can capture agreement violations across languages; however, their contribution is not decisive in agreement processing.</abstract>
    <identifier type="citekey">varda-marelli-2023-data</identifier>
    <identifier type="doi">10.1162/coli_a_00472</identifier>
    <location>
        <url>https://aclanthology.org/2023.cl-2.1</url>
    </location>
    <part>
        <date>2023-06</date>
        <detail type="volume"><number>49</number></detail>
        <detail type="issue"><number>2</number></detail>
        <extent unit="page">
            <start>261</start>
            <end>299</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Data-driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models
%A de Varda, Andrea Gregor
%A Marelli, Marco
%J Computational Linguistics
%D 2023
%8 June
%V 49
%N 2
%I MIT Press
%C Cambridge, MA
%F varda-marelli-2023-data
%X Massively multilingual models such as mBERT and XLM-R are increasingly valued in Natural Language Processing research and applications, due to their ability to tackle the uneven distribution of resources available for different languages. The models’ ability to process multiple languages relying on a shared set of parameters raises the question of whether the grammatical knowledge they extracted during pre-training can be considered as a data-driven cross-lingual grammar. The present work studies the inner workings of mBERT and XLM-R in order to test the cross-lingual consistency of the individual neural units that respond to a precise syntactic phenomenon, that is, number agreement, in five languages (English, German, French, Hebrew, Russian). We found that there is a significant overlap in the latent dimensions that encode agreement across the languages we considered. This overlap is larger (a) for long- vis-à-vis short-distance agreement and (b) when considering XLM-R as compared to mBERT, and peaks in the intermediate layers of the network. We further show that a small set of syntax-sensitive neurons can capture agreement violations across languages; however, their contribution is not decisive in agreement processing.
%R 10.1162/coli_a_00472
%U https://aclanthology.org/2023.cl-2.1
%U https://doi.org/10.1162/coli_a_00472
%P 261-299

Markdown (Informal)

[Data-driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models](https://aclanthology.org/2023.cl-2.1) (de Varda & Marelli, CL 2023)

• Data-driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models (de Varda & Marelli, CL 2023)

ACL

• Andrea Gregor de Varda and Marco Marelli. 2023. Data-driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models. Computational Linguistics, 49(2):261–299.