<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huang-etal-2023-directed">
    <titleInfo>
        <title>Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Fei</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pei</namePart>
        <namePart type="family">Ke</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Minlie</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation. However, in a wider range of text generation tasks, existing NAR models lack proper pre-training, making them still far behind the pre-trained autoregressive models. In this paper, we propose Pre-trained Directed Acyclic Transformer (PreDAT) and a novel pre-training task to promote prediction consistency in NAR generation. Experiments on five text generation tasks show that our PreDAT remarkably outperforms existing pre-trained NAR models (+4.2 score on average) and even achieves better results than pre-trained autoregressive baselines in n-gram-based metrics, along with 17 times speedup in throughput. Further analysis shows that PreDAT benefits from the unbiased prediction order that alleviates the error accumulation problem in autoregressive generation, which provides new insights into the advantages of NAR generation.1</abstract>
    <identifier type="citekey">huang-etal-2023-directed</identifier>
    <identifier type="doi">10.1162/tacl_a_00582</identifier>
    <location>
        <url>https://aclanthology.org/2023.tacl-1.53</url>
    </location>
    <part>
        <date>2023</date>
        <detail type="volume"><number>11</number></detail>
        <extent unit="page">
            <start>941</start>
            <end>959</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation
%A Huang, Fei
%A Ke, Pei
%A Huang, Minlie
%J Transactions of the Association for Computational Linguistics
%D 2023
%V 11
%I MIT Press
%C Cambridge, MA
%F huang-etal-2023-directed
%X Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation. However, in a wider range of text generation tasks, existing NAR models lack proper pre-training, making them still far behind the pre-trained autoregressive models. In this paper, we propose Pre-trained Directed Acyclic Transformer (PreDAT) and a novel pre-training task to promote prediction consistency in NAR generation. Experiments on five text generation tasks show that our PreDAT remarkably outperforms existing pre-trained NAR models (+4.2 score on average) and even achieves better results than pre-trained autoregressive baselines in n-gram-based metrics, along with 17 times speedup in throughput. Further analysis shows that PreDAT benefits from the unbiased prediction order that alleviates the error accumulation problem in autoregressive generation, which provides new insights into the advantages of NAR generation.1
%R 10.1162/tacl_a_00582
%U https://aclanthology.org/2023.tacl-1.53
%U https://doi.org/10.1162/tacl_a_00582
%P 941-959

Markdown (Informal)

[Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation](https://aclanthology.org/2023.tacl-1.53) (Huang et al., TACL 2023)

• Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation (Huang et al., TACL 2023)

ACL

• Fei Huang, Pei Ke, and Minlie Huang. 2023. Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation. Transactions of the Association for Computational Linguistics, 11:941–959.