<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="khalighinejad-etal-2023-approximating">
    <titleInfo>
        <title>Approximating CKY with Transformers</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ghazal</namePart>
        <namePart type="family">Khalighinejad</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ollie</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sam</namePart>
        <namePart type="family">Wiseman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We investigate the ability of transformer models to approximate the CKY algorithm, using them to directly predict a sentence’s parse and thus avoid the CKY algorithm’s cubic dependence on sentence length. We find that on standard constituency parsing benchmarks this approach achieves competitive or better performance than comparable parsers that make use of CKY, while being faster. We also evaluate the viability of this approach for parsing under random PCFGs. Here we find that performance declines as the grammar becomes more ambiguous, suggesting that the transformer is not fully capturing the CKY computation. However, we also find that incorporating additional inductive bias is helpful, and we propose a novel approach that makes use of gradients with respect to chart representations in predicting the parse, in analogy with the CKY algorithm being a subgradient of a partition function variant with respect to the chart.</abstract>
    <identifier type="citekey">khalighinejad-etal-2023-approximating</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.934</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.934</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>14016</start>
            <end>14030</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Approximating CKY with Transformers
%A Khalighinejad, Ghazal
%A Liu, Ollie
%A Wiseman, Sam
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F khalighinejad-etal-2023-approximating
%X We investigate the ability of transformer models to approximate the CKY algorithm, using them to directly predict a sentence’s parse and thus avoid the CKY algorithm’s cubic dependence on sentence length. We find that on standard constituency parsing benchmarks this approach achieves competitive or better performance than comparable parsers that make use of CKY, while being faster. We also evaluate the viability of this approach for parsing under random PCFGs. Here we find that performance declines as the grammar becomes more ambiguous, suggesting that the transformer is not fully capturing the CKY computation. However, we also find that incorporating additional inductive bias is helpful, and we propose a novel approach that makes use of gradients with respect to chart representations in predicting the parse, in analogy with the CKY algorithm being a subgradient of a partition function variant with respect to the chart.
%R 10.18653/v1/2023.findings-emnlp.934
%U https://aclanthology.org/2023.findings-emnlp.934
%U https://doi.org/10.18653/v1/2023.findings-emnlp.934
%P 14016-14030

Markdown (Informal)

[Approximating CKY with Transformers](https://aclanthology.org/2023.findings-emnlp.934) (Khalighinejad et al., Findings 2023)

• Approximating CKY with Transformers (Khalighinejad et al., Findings 2023)

ACL

• Ghazal Khalighinejad, Ollie Liu, and Sam Wiseman. 2023. Approximating CKY with Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14016–14030, Singapore. Association for Computational Linguistics.