<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sartran-etal-2022-transformer">
    <titleInfo>
        <title>Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Laurent</namePart>
        <namePart type="family">Sartran</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Samuel</namePart>
        <namePart type="family">Barrett</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Adhiguna</namePart>
        <namePart type="family">Kuncoro</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Miloš</namePart>
        <namePart type="family">Stanojević</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Phil</namePart>
        <namePart type="family">Blunsom</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chris</namePart>
        <namePart type="family">Dyer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism—one that is independent of composed syntactic representations—plays an important role in current successful models of long text.</abstract>
    <identifier type="citekey">sartran-etal-2022-transformer</identifier>
    <identifier type="doi">10.1162/tacl_a_00526</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.81</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>1423</start>
            <end>1439</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
%A Sartran, Laurent
%A Barrett, Samuel
%A Kuncoro, Adhiguna
%A Stanojević, Miloš
%A Blunsom, Phil
%A Dyer, Chris
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F sartran-etal-2022-transformer
%X We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism—one that is independent of composed syntactic representations—plays an important role in current successful models of long text.
%R 10.1162/tacl_a_00526
%U https://aclanthology.org/2022.tacl-1.81
%U https://doi.org/10.1162/tacl_a_00526
%P 1423-1439

Markdown (Informal)

[Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://aclanthology.org/2022.tacl-1.81) (Sartran et al., TACL 2022)

• Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (Sartran et al., TACL 2022)

ACL

• Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. 2022. Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale. Transactions of the Association for Computational Linguistics, 10:1423–1439.