<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="castricato-2023-trlx">
    <titleInfo>
        <title>trlX: A Framework for Large Scale Open Source RLHF</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Louis</namePart>
        <namePart type="family">Castricato</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Liling</namePart>
            <namePart type="family">Tan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dmitrijs</namePart>
            <namePart type="family">Milajevs</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Geeticka</namePart>
            <namePart type="family">Chauhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jeremy</namePart>
            <namePart type="family">Gwinnup</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Elijah</namePart>
            <namePart type="family">Rippeth</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Reinforcement learning from human feedback (RLHF) utilizes human feedback to better align large language models with human preferences via online optimization against a learned reward model. Current RLHF paradigms rely on Proximal Policy Optimization (PPO), which quickly becomes a challenge to implement and scale up to large architectures. To address this difficulty we created the trlX library as a feature-complete open-source framework for RLHF fine-tuning of models up to and exceeding 70 billion parameters. We implemented support for multiple types of distributed training including distributed data parallel, model sharded, as well as tensor, sequential, and pipeline parallelism.</abstract>
    <identifier type="citekey">castricato-2023-trlx</identifier>
    <identifier type="doi">10.18653/v1/2023.nlposs-1.27</identifier>
    <location>
        <url>https://aclanthology.org/2023.nlposs-1.27</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>246</start>
            <end>246</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T trlX: A Framework for Large Scale Open Source RLHF
%A Castricato, Louis
%Y Tan, Liling
%Y Milajevs, Dmitrijs
%Y Chauhan, Geeticka
%Y Gwinnup, Jeremy
%Y Rippeth, Elijah
%S Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F castricato-2023-trlx
%X Reinforcement learning from human feedback (RLHF) utilizes human feedback to better align large language models with human preferences via online optimization against a learned reward model. Current RLHF paradigms rely on Proximal Policy Optimization (PPO), which quickly becomes a challenge to implement and scale up to large architectures. To address this difficulty we created the trlX library as a feature-complete open-source framework for RLHF fine-tuning of models up to and exceeding 70 billion parameters. We implemented support for multiple types of distributed training including distributed data parallel, model sharded, as well as tensor, sequential, and pipeline parallelism.
%R 10.18653/v1/2023.nlposs-1.27
%U https://aclanthology.org/2023.nlposs-1.27
%U https://doi.org/10.18653/v1/2023.nlposs-1.27
%P 246-246

Markdown (Informal)

[trlX: A Framework for Large Scale Open Source RLHF](https://aclanthology.org/2023.nlposs-1.27) (Castricato, NLPOSS-WS 2023)

• trlX: A Framework for Large Scale Open Source RLHF (Castricato, NLPOSS-WS 2023)

ACL

• Louis Castricato. 2023. trlX: A Framework for Large Scale Open Source RLHF. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 246–246, Singapore. Association for Computational Linguistics.