<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ghose-poupart-2023-contrastive">
    <titleInfo>
        <title>Contrastive Deterministic Autoencoders For Language Modeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Amur</namePart>
        <namePart type="family">Ghose</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pascal</namePart>
        <namePart type="family">Poupart</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality. Deterministic autoencoders avoid this issue, and have been explored particularly well for images. It is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly improve on batch-normed VAEs (BN-VAEs), a strong benchmark for language modeling with VAEs, by replacing them with analogous deterministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text generation and downstream tasks from representations. These improvements are shown to be consistent across both LSTM and Transformer-based VAE architectures. Appropriate comparisons to BERT/GPT-2 based results are also included. We also qualitatively examine the latent space through interpolation to supplement the quantitative aspects of the model.</abstract>
    <identifier type="citekey">ghose-poupart-2023-contrastive</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.567</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.567</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>8458</start>
            <end>8476</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Contrastive Deterministic Autoencoders For Language Modeling
%A Ghose, Amur
%A Poupart, Pascal
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F ghose-poupart-2023-contrastive
%X Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality. Deterministic autoencoders avoid this issue, and have been explored particularly well for images. It is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly improve on batch-normed VAEs (BN-VAEs), a strong benchmark for language modeling with VAEs, by replacing them with analogous deterministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text generation and downstream tasks from representations. These improvements are shown to be consistent across both LSTM and Transformer-based VAE architectures. Appropriate comparisons to BERT/GPT-2 based results are also included. We also qualitatively examine the latent space through interpolation to supplement the quantitative aspects of the model.
%R 10.18653/v1/2023.findings-emnlp.567
%U https://aclanthology.org/2023.findings-emnlp.567
%U https://doi.org/10.18653/v1/2023.findings-emnlp.567
%P 8458-8476

Markdown (Informal)

[Contrastive Deterministic Autoencoders For Language Modeling](https://aclanthology.org/2023.findings-emnlp.567) (Ghose & Poupart, Findings 2023)

• Contrastive Deterministic Autoencoders For Language Modeling (Ghose & Poupart, Findings 2023)

ACL

• Amur Ghose and Pascal Poupart. 2023. Contrastive Deterministic Autoencoders For Language Modeling. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8458–8476, Singapore. Association for Computational Linguistics.