<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kiyono-etal-2023-bridging">
    <titleInfo>
        <title>Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shun</namePart>
        <namePart type="family">Kiyono</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sho</namePart>
        <namePart type="family">Takase</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shengzhe</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Toshinori</namePart>
        <namePart type="family">Sato</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Galia</namePart>
            <namePart type="family">Angelova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd., Shoumen, Bulgaria</publisher>
            <place>
                <placeTerm type="text">Varna, Bulgaria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretrained language models require the use of consistent segmentation (e.g., subword- or character-level segmentation) in pretraining and finetuning. In NLP, many tasks are modeled by subword-level segmentation better than by character-level segmentation. However, because of their format, several tasks require the use of character-level segmentation. Thus, in order to tackle both types of NLP tasks, language models must be independently pretrained for both subword and character-level segmentation. However, this is an inefficient and costly procedure. Instead, this paper proposes a method for training a language model with unified segmentation. This means that the trained model can be finetuned on both subword- and character-level segmentation. The principle of the method is to apply the subword regularization technique to generate a mixture of subword- and character-level segmentation. Through experiment on BERT models, we demonstrate that our method can halve the computational cost of pretraining.</abstract>
    <identifier type="citekey">kiyono-etal-2023-bridging</identifier>
    <location>
        <url>https://aclanthology.org/2023.ranlp-1.62</url>
    </location>
    <part>
        <date>2023-09</date>
        <extent unit="page">
            <start>568</start>
            <end>577</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models
%A Kiyono, Shun
%A Takase, Sho
%A Li, Shengzhe
%A Sato, Toshinori
%Y Mitkov, Ruslan
%Y Angelova, Galia
%S Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing
%D 2023
%8 September
%I INCOMA Ltd., Shoumen, Bulgaria
%C Varna, Bulgaria
%F kiyono-etal-2023-bridging
%X Pretrained language models require the use of consistent segmentation (e.g., subword- or character-level segmentation) in pretraining and finetuning. In NLP, many tasks are modeled by subword-level segmentation better than by character-level segmentation. However, because of their format, several tasks require the use of character-level segmentation. Thus, in order to tackle both types of NLP tasks, language models must be independently pretrained for both subword and character-level segmentation. However, this is an inefficient and costly procedure. Instead, this paper proposes a method for training a language model with unified segmentation. This means that the trained model can be finetuned on both subword- and character-level segmentation. The principle of the method is to apply the subword regularization technique to generate a mixture of subword- and character-level segmentation. Through experiment on BERT models, we demonstrate that our method can halve the computational cost of pretraining.
%U https://aclanthology.org/2023.ranlp-1.62
%P 568-577

Markdown (Informal)

[Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models](https://aclanthology.org/2023.ranlp-1.62) (Kiyono et al., RANLP 2023)

• Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models (Kiyono et al., RANLP 2023)

ACL

• Shun Kiyono, Sho Takase, Shengzhe Li, and Toshinori Sato. 2023. Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 568–577, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.