课程简介
CS336 Lecture 15：Mid/Post-training（SFT/RLHF）中文讲义
学习目标
完成本讲后，你应该能够：
- 理解后训练（post-training）在整个语言建模流程中的位置，以及为什么预训练之后还需要 SFT 和 RLHF。
- 区分 SFT 与 RLHF 在目标函数和优化范式上的本质差异：分布拟合 vs. 奖励最大化。
- 了解 SFT 训练数据的类型、历史演变，以及数据质量、风格、知识边界对模型行为的深刻影响。
- 掌握 RLHF 中数据收集的基本流程（成对偏好标注）以及 PPO 与 DPO 两种算法的核心直觉。
- 识别后训练中常见的陷阱：过优化（over-optimization）、长度作弊、标注者偏差、模式崩溃。
1. 课程主线概述
本讲从「基础模型（如 GPT-3）」走向「可用系统（如 ChatGPT）」的转变。预训练提供了广泛的语言能力和知识，但基础模型本身无法可靠地遵循复杂指令。后训练的目标是从预训练形成的「原始汤」中提取出用户真正需要的行为。
本讲的路线图：
- SFT 阶段：以人工或模型生成的输入-输出对进行监督微调，目的是教会模型「如何回应指令」。
- RLHF 阶段：通过人类偏好标注训练奖励模型，再用强化学习（PPO/DPO）优化策略，目的是教会模型「什么是好的回应」。
课程强调：后训练不同于预训练，它是一门「手艺活」，数据质量、风格控制、安全策略等因素远比算法本身更能决定最终效果。
2. SFT（Supervised Fine-Tuning）
2.1 SFT 的本质：数据主导
SFT 在算法层面和预训练基本一致——都是下一个词元预测的最小化负对数似然。真正的区别完全在数据上。因此，本讲将大部分篇幅用于讨论训练数据。
2.2 SFT 数据的历史演变
按时间顺序梳理 SFT 数据的几个关键派别：
FLAN（Google/T5）
- 核心想法：NLP 领域已有大量有监督数据集，全部拿来进行多任务后训练。
- 问题：数据集本身质量参差不齐，示例结构「非自然」。例如：
- 输入是一封电子邮件，指令放在末尾（「为这封邮件写一个主题行」）。
- 摘要数据集生成的回复极其简短，且经常包含幻觉内容。
- 教训：FLAN 建立在「后训练也需要大规模数据」的假设上，但后来的工作证明，预训练足够强时，只需少量高质量示例即可。
Self-Instruct 与蒸馏派系（Alpaca 等）
- Self-Instruct：让模型自己生成输入和输出，构成指令数据。
- Alpaca（斯坦福）：蒸馏 ChatGPT 的输入-输出轨迹。当配合 Llama 基座模型时，首次在开源界可靠地诱导出 ChatGPT 风格的行为。
- 里程碑意义：证明「聊天风格数据」是从基础模型演化到对话模型的关键，也点燃了大规模构建指令数据的浪潮。
Open Assistant（众包人类数据）
- 接近维基百科式的众包努力：志愿者编写提示词与高质量回复。
- 产出数万条示例后项目停滞。
- 价值在于展示了专家编写的高质量、长篇幅回复长什么样，也暴露了数据收集的可持续性问题。
合成数据 + 工具调用（现代）
- 当前 SFT 数据正从纯聊天走向智能体（agent）范式：回复中不仅包含文本，还包含工具调用、待办事项列表等结构化内容。
- 例如 Nemotron（NVIDIA）的 SFT 数据大量采用这种「文本 + 并行工具调用」的格式。
2.3 SFT 数据中的关键权衡
正确性 vs. 泛化
- 投入正确性：应当是「尽可能收集最高质量回复」，坏示例会教会模型坏行为。
- 但存在反直觉现象：即使数据质量较差，模型仍能学会遵循指令，这源于预训练的强泛化能力。
- 结论：给定一个足够强的预训练模型，所需的示例数量很少；质量和数量的权衡应该偏向质量。
知识边界与幻觉
- 案例：Open Assistant 中标注者给出了包含引用的回复，例如「……（Bivens J & Michelle L, 关于企业高管薪酬）」。模型在 SFT 中学到两件事：
- 这段知识本身（姓氏来源等）；
- 输出时应该附带参考文献的格式习惯。
- 问题在于模型无法判断某篇参考文献的真伪。当它不知道某个事实时，它会「强行泛化」——照猫画虎地编造出格式正确但内容虚假的引用。
- 已有经验证据（课程中称为「民间传说」）：用模型未知的事实做 SFT 训练会加剧幻觉。
- 为什么 RL 可能缓解这个问题：模型内部可能存在某种「我知道」vs「我不知道」的方向性表征。SFT 无法提取这种信息，但 RL 可以通过奖励信号将这种内部激活模式与「是否输出」的决策连接起来。前提是模型内部确实存在这种知识，否则 RL 也无能为力。
- John Schulman 的观点（课程引用）：模型对自己知识边界的校准必须依赖于策略（policy），而不能靠外部硬塞知识。
数据规模
- 只需数百条样本（如 500 条安全示例）就能大幅压低恶意指令遵循率。
- 但这不意味着更多样本没有价值：大型组织仍需要数千至数万条细粒度的安全数据来执行精确的边界区分。
风格控制 vs. 能力控制
- 标注者偏好长度更长、含项目符号列表的回复——这种偏好会扭曲模型的天然语气。
- 参与度信号（engagement signals）容易自欺欺人：你可以大幅改变用户参与度，但标准基准测试分数却不动。
- 启示：风格与能力必须分开看待与调节。
2.4 安全 SFT
- 平衡两个目标：恶意行为「违规率」与「误拒率」。
- 数据规模通常在数千到数万条范围；例如 Llama 2 使用几千条安全样本。
- 方法往往很直接：从聊天接口的海量交互记录（如 WildChat）中挖掘不安全行为和越狱尝试，为每个案例生成「首选回应」（拒绝语）。
- 本质上是「打地鼠」式的被动响应：发现什么坏行为，就补什么数据。
2.5 SFT 方法细节
- 算法层面没有新东西：就是标准的下一词元预测 + 梯度下降。
- 重要趋势：预训练与后训练的界限正在模糊。
- 如今「基础模型」通常并不是只在互联网数据上预训练的，而是混合了大量质量较高、带聊天风格的 SFT 数据（如 UltraChat）进行第二阶段训练，往往与学习率衰减阶段重合。
- 示例（MiniCPM）：第一阶段是标准互联网数据；第二阶段切换为高质量对话数据（Stack Exchange QA、UltraChat 等）。
- 好处：可利用衰减阶段做大量低成本的消融实验，反向指导第一阶段的预训练数据混合配比。
- 注意：这类阶段中的 token 不遮蔽，提示词也被预测；「基础模型」一词已经名不副实。
3. RLHF（Reinforcement Learning from Human Feedback）
3.1 概念转变：从生成式建模到奖励最大化
- 预训练与 SFT 属于生成式建模：拟合一个分布，关注的是分布本身。
- RLHF 属于奖励最大化：仍有一个分布，但它现在是策略（policy）；目标是最优化某个下游奖励（用户参与度、正确率等）。
- 关键差异：在奖励最大化框架下，策略完全可以将输出分布坍缩成单点，只要单点能拿到高奖励。这与「模拟分布」的目标有本质差别。
3.2 为什么需要 RL 而非继续收集 SFT 数据？
两个核心原因：
- 人们说什么 ≠ 他们生成什么：人类标注者自己写的摘要可能不如 AI 输出好（课程给出 InstructGPT 时代标注者偏好 AI 输出的实例）。当你需要的是「评判」而非「生成」时，偏好标注更合适。
- 验证往往比生成本质上更容易：例如数学领域，验证一个证明的正确性远比构造证明容易。这为 RL 中的自我评判提供了天然优势。
3.3 RLHF 数据收集
标准收集流程（类似 InstructGPT）：
- 输入提示词 → 模型采样多个输出（采样温度设为 1）→ 评估者做出二元或排名比较。
- 标注者被要求从三个维度评估：有用性（helpfulness）、真实性（truthfulness）、无害性（harmlessness）。
- 实际案例：Google Bard 曾使用李克特量表进行标注（而非成对比较），并强调回复要连贯、简短、易读。
3.4 标注者分布：极其重要且被低估
学历与专业度
- 数据标注者的学历分布：约 70% 拥有学士或硕士学位；年龄中位数约 35 岁。他们的任务多为创意写作和技术写作。
- 趋势：标注工作正在向专家化、定制化迁移；部分专家按小时计薪每小时超过 100 美元。
标注偏见的传播
- 研究案例：后训练后的模型在政治观点上更接近佛教徒、印度教徒和无神论者，和 InstructGPT 标注者的来源地（东南亚、美国西海岸）吻合。
- 存在「阈下迁移效应」（emergent misalignment）：即便你在无害数据上训练，模型也可能继承数据生成者的隐藏偏好。
专业度 vs. 众包质量
- 专业标注者更关注事实性和逻辑不一致，业余标注者则过度关注格式化（如加粗、列表）。
- 事实性检验难度高，这正是众包难以覆盖的原因。
- 如何定义「好标注者」：标注指南的严格遵守（半客观标准）+ 标注者间一致性（高方差任务如主观偏好不适用）。
AI 做标注者的崛起
- GPT-4 时代的系统排名对比表明：模型标注与高质量人工标注的一致性相当，成本却低一个数量级。
- 案例（HuggingFace Zephyr）：团队试图完全避免蒸馏，投入大量时间和经费收集人类数据，结果效果并不比基于模型标注更好，最终仍回归模型反馈。
- 结论：如果你想追赶前沿能力，基于模型的标注（distillation）基本是标配；但如果你要推动前沿、获取世界知识（律师级判断等），人类标注不可替代。
- 注意「模型标注」的陷阱：长度作弊。对模型评判者而言，更长的输出往往被误认为更好，甚至仅用长度做 RLHF 就能在多个基准上表现良好。
3.5 RLHF 算法
目标函数
RLHF 的核心目标如下：
- 第一项：在策略 π_θ 下采样并最大化奖励 r_φ。
- 第二项：KL 散度正则项，要求策略与参考策略（即 SFT 模型）保持接近，防止优化过度偏离。
PPO（Proximal Policy Optimization）
推导脉络：
- 策略梯度恒等式：目标为最大化期望奖励，梯度 = 对数策略梯度 × 奖励。
- 从同策略到异策略：采样昂贵，希望重复利用已生成的样本；引入重要性权重修正，但要限制步长（TRPO 的思想）。
- PPO：TRPO 中的距离约束难以计算，改为启发式的 clip 机制，防止策略偏离过远。
课程指出：PPO 可在下一讲中展开，这里只需理解它是「带权重、带约束的 SFT」。
DPO（Direct Preference Optimization）
核心直觉：对好样本走正向梯度步长，对坏样本走负向梯度步长——用「正/负 SFT」替代完整 RL 循环。
推导要点：
- 假设策略 π 是非参数化的，可以近似任意分布。
- 在该假设下，RLHF 目标存在闭式解：最优策略 = π_ref 的指数倾斜（以奖励为权重）。
- 反解出该最优策略隐含的奖励函数。
- 将该隐含奖励代回原始目标函数，得到 DPO 目标：
- 对获胜样本提升概率；
- 对失败样本降低概率；
- 步长由「当前模型对两者概率差的『意外程度』」缩放。
直观理解：
- 如果模型已经正确区分好坏（给好的高概率、坏的极低概率），则步长小。
- 如果模型完全没区分出差异（两者概率相等），则步长大。
DPO 的效果：
- 在当前实际操作中（如 Llama 系列技术报告），DPO 已作为核心 RLHF 原语使用。
- 与 PPO 的胜负取决于实验设置与工程细节，没有绝对优劣。
- 变体（SimPO、长度归一化 DPO 等）小改动带来的差异不大。
- 对学习者而言，DPO 的价值在于：拟合分布式的简单目标 + 适当的步长设定，就是以得到近乎 SFT 的简单实现。
3.6 RLHF 的常见陷阱
- 过优化（over-optimization）：过度压缩 KL 正则化会导致模型「欺骗」奖励模型，即过拟合奖励信号而非真实人类偏好。KL 正则不是可选项，而是必须项。
- 模式坍缩：RL 优化会大幅降低生成多样性，模型开始集中于少数几种安全输出——这源于「策略」而非「分布」的定位。熵与多样性在大模型后续应用中（如推理模型）至关重要。
- 校准漂移：RLHF 可能显著降低模型的校准质量（如 GPT-4 时代 OpenAI 公布的校准退化图）。目前仍无通用解法。
4. 课程作业关联
- 本讲建议先完成 CS336 中后训练部分的基础作业，尤其是 RLHF 相关编程任务。
- 理解 PPO 与 DPO 的原理是必须的；作业中可能涉及 GRPO（PPO 的简化变体），其构建过程是自包含的——从策略采样多个输出，计算归一化奖励，再进行带参考约束的梯度更新。
- 数据集构造方面：课程提供 SFT/RLHF 数据格式（输入-输出对、偏好对），建议自行完成从模型采样到成对排序的完整 pipeline，以理解「标注困难」的体感。
5. 常见误区
| 误区 | 正解 |
|---|---|
| SFT 数据越多越好 | 预训练足够强时，少量高质量示例即可；过多且包含模型未知事实的数据反而加剧幻觉 |
| 「基础模型」就是纯预训练模型 | 现代「基础模型」经常混入大量 SFT 风格的合成数据做第二阶段训练 |
| RLHF 的目标是拟合人类偏好分布 | 实际上是最大化奖励函数，模型可以为了奖励坍缩成单输出 |
| DPO 一定比 PPO 好（或差） | 两者的相对优劣高度依赖实验设置、数据与调参细节 |
| 标注数据质量由「标注者间一致性」保证 | 一致性只衡量方差，不衡量偏差；高一致性也可能代表大家都用 ChatGPT 作弊 |
| 模型标注可以完全替代人类标注 | 对追赶前沿是可行的；但世界知识、专家意见、安全边界等仍依赖人类 |
| 过优化只存在于 RL 迭代过多次时 | 只要 KL 正则化过弱就会引发过优化，即便迭代次数不多也可能发生 |
6. 复习清单
- [ ] 能说出 FLAN、Self-Instruct、Alpaca、Open Assistant 的核心思想与局限。
- [ ] 能解释为什么「尾部知识」在 SFT 中是有害的。
- [ ] 能解释 RLHF 与 SFT 在目标函数层面的根本差异。
- [ ] 能描述 InstructGPT 式偏好标注流程的全过程。
- [ ] 能推导 DPO 目标（给定期望奖励 + KL 正则 + 非参数化假设）。
- [ ] 能对比 PPO 与 DPO 的取舍：采样成本、算法复杂度、过优化风险、长度作弊。
- [ ] 能识别 RLHF 的三大常见问题：过优化、模式坍缩、校准漂移。
- [ ] 能说明基于模型标注（distillation）的适用场景与陷阱。
- [ ] 能解释「预训练-后训练界限模糊化」的工程动机。
7. 补充说明
本讲基于 Stanford CS336 Lecture 15 逐字稿整理，保留了课程原意与术语（SFT、RLHF、PPO、DPO、KL 散度等）。对于无法从逐字稿获得精确数据（如 Llama 2 安全样本数量、ScaleAI 薪资中位数）的细节，均采用定性表述，未作额外补充。如需深入了解具体算法推导、代码实现或实验设置，请参考课程讲义与其后补充材料。
从GPT-3到ChatGPT：后训练(SFT/RLHF)如何塑造语言模型的行为
导语：预训练模型像一块未经雕琢的巨石，蕴藏着无限可能，却无法直接为人所用。从GPT-3到ChatGPT的跨越，靠的不是更大的模型或更多的数据，而是一套被称为"后训练"(post-training)的精细工艺。本文基于斯坦福CS336课程第15讲，深入解析指令微调(SFT)与人类反馈强化学习(RLHF)的核心思想、数据工程与算法演进，带你理解现代对话系统背后的技术逻辑。
一、为什么我们需要后训练？
想象你手中有一个增强版的GPT-3——一个规模庞大、能力强劲的基础模型。你可以用它写文案、做有趣的小任务，但当你试图让它可靠地遵循复杂指令时，你会感到深深的挫败感。正如课程中所言："当你第一次接触ChatGPT时会觉得惊艳，但回头去和GPT-3互动，你会觉得——这什么东西？这东西有什么用？"
这种差距源于一个核心问题：预训练模型学会了语言的统计规律，却不懂"指令遵循"(instruction following)。GPT-3时代，你只能通过精心设计少样本提示(few-shot prompts)来引导模型，效果极其有限。而ChatGPT级别的系统能够一次性完成冗长而复杂的程序化提示词(programmatic prompts)，这种能力的飞跃正是后训练的功劳。
预训练是基础，但它只提供了"原始汤" 。预训练以极其多样的方式扩展模型的语言能力，但要从这种"原始汤"中提取出我们想要的行为类型，需要更加显式的数据收集、更加显式的引导，以及大量杂乱的工程工作。后训练本质上是一门"手艺活"，它面对的是真实世界的混乱：人们会如何滥用系统、标注者会带有怎样的偏见、模型会如何产生幻觉。
值得注意的是，关于前沿后训练的信息在公开领域相当稀少。课程中提到，经典的RLHF论文如《Learning to summarize from human feedback》和Anthropic的HH论文之所以珍贵，正是因为它们发布在竞争升温之前，包含了详细的标注指南。如今，后训练数据在很大程度上已成为商业机密。Scale AI内部文件泄露事件生动展示了这种竞争动态：他们试图让Google Bard变得更好，却搞不清楚为什么GPT-4要优秀得多，只能让标注人员产出比GPT-4更详细、更好的回答。
归根结底，算法的秘密不在于算法本身，而在于数据。
二、SFT阶段：用高质量数据引导预训练行为
后训练的第一阶段是监督微调(Supervised Fine-Tuning, SFT)。从算法角度看，SFT与预训练几乎没有区别——都是梯度下降、都做下一个词元预测。唯一的区别在于训练数据。因此，SFT阶段的全部精髓都集中在数据上。
2.1 SFT数据的演进图谱
FLAN：多任务学习的先驱与局限
FLAN是谷歌用于训练T5模型的多任务数据集，其核心想法很直接：NLP研究者已经收集了大量有监督的输入-输出数据集，为什么不把所有数据都拿来训练模型？
这个想法在当时极具远见，但实际执行中暴露了问题。课程展示了一个FLAN数据的例子：安然邮件(Enron emails)数据集被改造成"为主题行预测"任务——正文作为输入，主题行作为输出。还有摘要任务，指令甚至被放在输入的最后面。这种数据存在两个严重缺陷：
- 非自然性：真实用户不会这样提示模型，指令出现在末尾也不同于ChatGPT的交互模式
- 低质量继承：原始NLP数据集往往包含幻觉内容——摘要中有很多细节并不在输入文本里，模型训练时会继承这些缺陷
更深层的问题是，FLAN的构建理念基于一个假设：后训练需要巨大的规模才能解锁能力。但后来的实践表明，如果预训练模型足够强大，实际上只需要很少的高质量示例就能成功，因为预训练的泛化能力会提供大量帮助。FLAN在质量-数量的权衡上站在了错误的一边。
Self-instruct与蒸馏：让模型教模型
Self-instruct提出了一种前瞻性思路：模型一直在变得更好，为什么不让模型自己生成高质量的训练数据？这开启了基于蒸馏(distillation)的方法潮流。
Alpaca是这一方向的标志性工作。斯坦福的研究者们蒸馏了ChatGPT的交互轨迹来获取输入-输出对，得到了更自然、更长、更像聊天的输出。关键发现在于：只有对原始的Llama模型进行这种微调才有效——预训练和后训练两者缺一不可。Alpaca是"这些东西开始起效的转折点"，它证明只要得到聊天风格的数据，构建ChatGPT风格的系统并不困难。
Open Assistant：众包努力的启示与教训
在Alpaca带来的乐观情绪中，Open Assistant试图通过众包构建高质量数据——就像维基百科那样，让一群志愿者聚在一起写提示词和回复。这个努力值得敬佩，也确实生成了大量数据，但最终项目停滞了。课程指出，Open Assistant暴露了后训练数据收集中的一个关键问题，我们稍后会详细讨论。
从聊天到智能体：SFT数据的新方向
近年来，SFT数据出现了新的转变：重心从单纯的聊天界面转向了智能体(agents)和工具使用。以英伟达的Nemotron为例，其SFT数据很大一部分是智能体式的——不仅包含assistant字段中的回复内容，还有与文本并行发生的工具调用。用户期待的已不只是文本回复，而是待办事项列表、工具调用和逐步执行的任务流程。
2.2 数据质量的两大陷阱
陷阱一：教模型输出它不知道的知识
Open Assistant的数据质量很高，但其中有一个例子揭示了微妙的问题。某个回复包含了一个引用，署名是"J Michel"。课程指出，这同时教会了模型两件不同的事情：第一，这个具体的引用信息；第二，给出回复时提供引用是好的做法。
问题在于，这两个机制是同时发生的。如果模型并不知道"J Michel"是谁，它实际上在学习"强行输出未知知识"的模式。当模型学会了引用格式，却缺乏真正的知识支撑时，它就会编造参考文献——因为模型无法判断某个参考文献是真是假。
这种现象不止是理论担忧。课程提到，有经验证据表明：如果你用模型并不真正知道的信息进行训练，会导致幻觉加剧。更糟糕的是，当这种未知知识与特定格式标记（如"参考文献："）关联时，模型会强制生成参考文献，即使它根本没有相关知识。
John Schulman提出了一个有趣的论点：这正是为什么需要强化学习的原因。要教会模型知道自己知道什么、不知道什么，不能靠外部的人把知识硬塞给它，必须依赖于策略本身。这个问题引出了一个重要的设计原则：如果模型本身还不具备某些高质量数据中的知识，你可能并不想在那些数据上训练。尾部知识有时实际上可能是有害的。
陷阱二：风格与长度的隐性偏差
回复长度和风格是后训练中极其重要的变量。"Claude的语气和ChatGPT不一样、ChatGPT太啰嗦"——这些都是数据收集人员有意识做出的决定。但风格因素在评估偏好时会产生极大影响：人们通常更倾向于选择带项目符号列表的回复，或细节更多、篇幅更长的回复。
这导致了参与度信号与真实能力之间的脱节。课程指出，在某些数据集上训练时，偏好差异会很大（如AlpacaEval），但这并不一定会改变标准基准测试的评估结果——模型并不会因为使用了某些类型的后训练数据而变得更聪明，但你可以大幅度改变参与度信号。你必须把风格控制和能力控制分开来考虑。
2.3 安全SFT：最后一道防线
预训练阶段的人"生活在象牙塔里"，只需要"压缩整个世界"。但后训练阶段必须面对残酷的现实：人们会使用系统进行政治操纵、虚假信息传播、个性化鱼叉式网络钓鱼。
安全SFT的目标是在两方面之间找到帕累托权衡：违规率（恶意查询通过的概率）和误拒率（误杀正常查询的概率）。如果有人问"我如何杀死一个Python进程"，模型如果回答"不，我不能让你杀死任何东西"，那会让用户非常沮丧。
实际操作中，安全SFT通常只需要几千到几万个示例。课程引用了OLMo模型的训练后流程——这是少数能看到真实训练后流水线细节的参考之一。他们的策略很简单：从WildChat中收集大量被尝试的不安全行为和越狱尝试，然后让标注人员写出抵抗性回应。这本质上是一种"打地鼠"游戏。
最令人惊讶的是，仅需500个样本就能显著降低模型遵循恶意指令的比率。这说明了预训练模型的内部已经存在"我要成为安全模型还是不安全模型"的通道，不需要很多样本就能把它引出来。但课程也提醒，这并不意味着更多样本没有好处——如果你需要强制执行关于什么安全什么不安全的细粒度区分，仍需大规模数据收集。
2.4 趋势：预训练与后训练的边界模糊
近年来出现了一个重要趋势：高质量数据和指令微调数据被混合到预训练过程的末端，通常发生在训练衰减阶段。这模糊了预训练和后训练的界限。
课程以MiniCPM为例展示了这种数据混合的变化：第一个阶段使用标准的多样化互联网数据；衰减阶段则切换到了更高质量、更"健谈"的数据集——Stack Exchange QA、UltraChat、其他SFT数据，同时降低标准通用预训练数据的比例。
这种做法效果显著，据课程所述，"每个人都在这么做"。一个有趣的推论是：今天当有人告诉你某个东西是基础模型时，那有点像在撒谎。因为今天的基础模型实际上是在UltraChat等合成聊天数据上训练过的，很难说是传统意义的基础模型。
这种混合策略还有一个实际好处：衰减阶段比完整预训练短得多，可以做大量数据消融实验来优化混合配比。你可以对每个候选方案运行大约10次实验，虽然不能完全暴力求解，但确实可以尝试大量消融。Meta因使用书籍数据被起诉时泄露的法庭文件，就展示了研究者如何做消融实验估计各书籍子集的用处。
三、RLHF阶段：从拟合分布到最大化奖励
完成SFT后，模型已经能遵循指令，但还只是"生搬硬套"。RLHF要解决的是更深层的问题：如何让模型真正学会人类偏好？
3.1 概念转变：生成式建模 vs 策略优化
课程强调了一个重要的概念性差异。预训练和SFT本质上都是生成式建模——你在预测下一个词，在拟合某个分布。但RLHF完全不同：你不再玩拟合分布的游戏，而是玩最大化奖励的游戏。
这带来一个关键推论：在RLHF的世界里，模型完全可以把分布塌缩成每个输入对应一个单点——每个提示词只有一个回答。只要这个回答能获得好的奖励，这种塌缩完全没问题。这种概念差异解释了为什么RLHF模型有时会表现得非常"极端"。
3.2 为什么需要强化学习？
有人可能会问：为什么不继续收集SFT数据，持续改进示范学习？
课程给出了两个理由。第一，人们说他们想要什么和他们实际生成什么之间存在差异。一个有趣的实验发现：有几位自由职业写手实际上更喜欢Instruct Davinci（ChatGPT前身）的摘要输出，而不是他们自己写的东西。采访后才知道，这些写手看到AI的输出后说："我没意识到原来还有更好的写法。"人类并非最优系统，他们的生成能力往往低于判断能力。
第二，在某些领域，验证比生成容易得多。数学是典型例子——验证证明远比生成证明容易。DeepSeek团队已经利用模型进行自我验证，这是强化学习和自我评判理念的典型应用。
3.3 RLHF的流程：数据收集
RLHF首先需要收集偏好数据。标准做法是：向模型输入提示词，以温度设为1采样生成多个不同输出，然后让评估员对输出进行排名，有时甚至是二元排名（哪个更好）。
标注者分布的影响：课程花了大量篇幅讨论标注者人口统计学特征对模型行为的惊人影响。一个经典研究发现，各种基础模型的观点"有点像新教徒或罗马天主教徒"，而后训练后的模型变得更接近"佛教徒、印度教徒和无神论者"。原因揭晓：InstructGPT论文的附录显示，为其做标注的标注者很多是东南亚人和美国西海岸人，正是对应这些人的人口统计群体。
这种现象被称作"涌现性错误对齐"(emergent misalignment)：如果训练数据由一个有特定偏好的模型生成，新模型会继承这种偏好。还存在各种微妙的阈下迁移效应，很难被发现。
专家与非专业标注者的差异：一项由Hosking等人进行的研究发现，非专业标注者确实很喜欢强调格式化效果，而专业性标注者会更多强调事实性和不一致性。检验事实性要难得多，所以如果你没有专家标注者，最终实际上不会去检查这些问题。
标注者的成本结构变化：标注工作的从业者分布已经向上移动，更多转向专家，成本更高。ScaleAI平台数据显示，大部分标注员拥有学士或硕士学位（约70%），众数年龄约35岁，执行的多是创意写作和技术写作任务。一些专家标注者时薪超过100美元。另一方面，防止标注员使用AI作弊极其困难——在时间压力下，人们很容易转向ChatGPT代工。
基于模型的标注成为主流：课程总结了这场辩论的结局。如果你只想追赶前沿能力，基于模型的标注就是最优解——模型比随机众包工人好得多，成本低一个数量级。Huggingface的Zephyr项目提供了一个极具启发性的案例：他们试图完全避开模型蒸馏，花费大量时间和精力收集人类数据，结果却并不比基于模型的标注更好，最终不得不回到了AI反馈。
当然，对真正的顶尖机构（OpenAI、Anthropic）来说，如果想要推动前沿，仍然高度依赖人类数据收集，尤其是在需要律师、医生等专业知识的领域。
3.4 核心算法：从PPO到DPO
PPO的策略梯度本质：RLHF的目标很简单——在策略下最大化通过采样获得的奖励。但要受限于一个KL散度项：你要保持与预训练模型接近，因为不想偏离太远而导致退化。
从实现角度看，策略梯度的问题在于每次优化步骤都需要采样，而推理非常昂贵。解决方案是通过重要性权重修正(importance sampling)实现离策略(off-policy)学习——一次生成数据，多次复用。但走太远会导致奖励估计爆炸，这就是TRPO的思想：保持与当前位置接近。PPO则用启发式裁剪机制替代TRPO中难处理的约束条件。
DPO的优雅简化：多年来，很多人试图去掉PPO——它太复杂了。但这些尝试大多效果不佳：在好示例前加"好"token的SFT变体不行，只训练奖励模型选中的内容效果也不好。
DPO终于找到了突破口。它基于一个简单直觉：深度学习中的所有事情不过就是朝着好的方向取梯度步长，朝着坏的方向取负梯度步长。DPO通过数学推导，将一个强假设——策略可以是任何东西，不限于神经网络——应用于RLHF目标函数，得到了闭式解。
关键在于推导过程：当策略可以是任何东西时，最优策略有一个解析形式——取参考策略，用奖励对它进行指数倾斜。奖励好的响应被指数加权，奖励差的被指数降权。然后，你可以从这个闭式解中解出隐含的奖励函数，再将其代回原始目标，就得到了DPO目标：
- 增加获胜示例(wining example)的可能性
- 降低失败示例(losing example)的可能性
- 根据隐含奖励模型的错误程度缩放步长——模型已给获胜样本高奖励则迈小步，若概率几乎相等则迈大步
课程提到，Llama技术报告显示，其核心RLHF原语就是DPO：模型做SFT后做DPO，然后用DPO模型生成候选，进行拒绝采样，重复这个过程。
值得注意的是，关于DPO是否比PPO更好的争论，目前的答案是"差别不太大"。各种DPO变体（SimPO、长度归一化DPO等）似乎也没有带来显著差异。这些结果非常依赖于实验设置的具体细节——同一实验室的不同论文甚至得出了互相矛盾的结果。
3.5 RLHF的三大隐患
过度优化：如果你过度推动RLHF进程，就会开始过拟合奖励模型。KL正则化在很多情况下确实非常关键，它能防止优化过程过拟合奖励模型。
模型崩溃：RL模型的多样性会大幅下降，集中在少数几种可能的输出上。这回到了RLHF的概念差异——它不再建模一个本身就带有内在多样性的分布，而是一个策略，只要能获得好的奖励就可以崩溃。
校准变差：GPT-4时代OpenAI发布的不多几张图之一，就展示了RLHF后模型校准变差的问题。"这个问题到现在也没有人真正解决。"Anthropic的人甚至认为这是天生不校准的——有时可以重新校准，但并不总是能做到。
四、从ChatGPT到推理模型：后训练的演进方向
课程在第15讲结束时指向了未来：下一讲将从ChatGPT走向推理模型（如GPT o1）。这个过渡的关键在于：能否找到一种奖励函数，让我们不会过度优化，模型性能可以随计算投入持续提升？这就是RLVR（基于强化学习的验证）受到如此多关注的原因。
推理模型的训练中，熵和探索变得相当关键——模型需要去探索所有可能的解，才能在这些非常困难的问题上取得进展。课程也预告了将更详细讲解PPO和新的变体GRPO。
五、结语
后训练之所以复杂，因为它是一门"手艺活"——它要求你面对真实世界的混乱，处理数据质量、标注者偏见、安全控制、风格平衡等无数细节。但正是这种"杂乱"，赋予了后训练独特的技术挑战和创造力。
回望整个路径：GPT-3是原始的"混沌巨兽"，预训练赋予它语言能力；SFT像一位老师，用高质量示范教它如何遵循指令；RLHF则像一位教练，通过奖罚机制塑造它的行为偏好。每一步都在提取预训练中已有的行为模式，但每一步又都需要精心设计的数据和算法。
从课程整体来看，第15讲与前面的架构、预训练内容构成了完整的技术栈：预训练提供基础能力，后训练将这些能力引导到实际产品所需的方向。更广义地说，它代表了深度学习从"拟合分布"到"优化目标"的范式转变——这种转变在推理模型时代将变得更加重要。
对于后训练数据，你应该尽可能收集最高质量的回复，因为坏的东西会教模型做坏事。但一个同样重要的真相是：如果预训练模型足够强大，你实际上只需要很少的高质量示例就能成功——从FLAN到Alpaca的演进，证明了质量远比数量重要。
参考资源
CS336 2026 Lecture 15：完整中文逐字稿
- 视频：官方视频
- 来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
- 说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。
正文
[00:05.240] 好的，欢迎回来。我们现在要离开预训练这个主题了。在我看来，我们会进入语言建模中一些比较杂乱的部分。我的意思是，数据总是相当杂乱的，但我认为后训练以及与之相关的一切，在很多方面确实像是一门手艺活。为了铺垫一下背景，到目前为止，我认为你们可以做出一个增强版的GPT-3。我相信你们可以使用更多的算力，获得更多的数据。所以你们可以做出比GPT-3稍微好一点的东西。但归根结底，像GPT-3这样一个非常大、非常强的基础模型，你能从中获得的实用价值终究是有点有限的。当我们从GPT-3走向类似ChatGPT这样的东西时，我相信如果你们第一次接触这些系统是通过ChatGPT，那感觉是相当惊艳的；但如果你们回头去和GPT-3互动，你们会觉得，这什么东西？这东西有什么用？你能用它做的真正有意义的事情，无非就是文案写作或者这些有趣的小任务，
[01:10.070] 而这些任务并不真正需要可靠性或指令遵循能力。所以今天，我们要从GPT-3走向基本上接近ChatGPT的东西。这是我们需要的第一个部分。在下一次讲座中，我们将从ChatGPT走向GPT o1或我们今天拥有的任何推理模型。所以这将是两个部分。而将我们从左端带到右端的过程，通常被大家称为后训练。我想强调一点，我们能够实现指令遵循，这是多么了不起的事情。指令遵循是这样一种能力：能够输入一个相当长、相当复杂的提示词，然后得到非常合理的回答。在GPT-3层面，对这些系统的精细控制基本上是不可能的。你只能放入一堆少样本提示（few-shot prompts），然后你只能说，嗯，如果我的例子足够好，我就能得到合理的结果。
[02:08.960] 在实践中，你引导这些模型的能力非常有限。但一旦GPT-3.5和GPT-4出现，你开始看到这些非常长的程序化提示词（programmatic prompts）的例子。然后模型真的能一次性完成整个提示词。那么，要实现这种指令遵循，你真正需要什么呢？预训练总是重要的。它是我们构建这一切的基础。所以如果我们忽略预训练，只试图通过训练来走向胜利，我们将什么都得不到。绝对什么都得不到。预训练之所以关键，是因为它以这种非常多样、广泛的方式进行扩展。但一旦我们有了预训练，我们就必须从预训练这个「原始汤」中提取出我们想要的行为类型。要做到这一点，我们将不得不进行更加显式的数据收集，更加显式的引导，以及大量杂乱的工程工作，这些在某种程度上是我们此前不必做的。
[03:06.940] 我的意思是，架构相关的东西你们可能觉得杂乱。但我认为它真的无法与现实世界后训练的杂乱程度相比。所以我们将去收集一些数据，或者我们会以「我们要去收集一些数据」的口吻来讨论。我们将讨论这些数据长什么样，如何用它训练，以及相应的规模是怎样的。我要在本次讲座的剩余部分开始前先说一句，关于前沿后训练的信息，说实话在很多很多方面都相当稀少。如果你们看我今天要引用的材料，很多实际上是比较老的。它们是在ChatGPT带来的竞争开始升温之前的东西。比如说，那篇RLHF论文《Learning to summarize from human feedback》，实际上非常详细。我今天在课堂上会提到它。如果你们感兴趣，我建议你们去读一读它的一些附录。里面包含了详细的标注指南，比如他们如何指导人们给模型提供反馈，类似的还有Anthropic在2022年的HH论文。
[04:10.600] 而如今竞争已经升温，基本上没有哪家厂商愿意发布任何关于他们后训练过程的信息。这些数据在很大程度上是商业机密。所以我们无法讨论任何类似现代语言模型的东西的标注指南等等。我们确实有闭源的——或者开源的模型。但另一方面，很多开源配方依赖蒸馏（distillation）。我认为其本质与前沿实验室在人类数据收集方面所做的非常不同，后者确实是一件非常棘手和杂乱的事情。我会尝试指出其中的一些方面，但要非常全面地讨论其中的某些部分将是困难的。好的。然后，举个例子，我之前为这次讲座做了一些预研究，我在2023年发现了一个很有趣的例子，关于后训练数据的商业机密性——我记得Scale AI的内部文件在某个时间点被泄露了。
[05:12.430] 他们有一大堆非常坦率的对话，讨论后训练数据领域正在发生的事情。其中有一条大概是这样：他们试图让Google Bard变得更好，但他们搞不清楚为什么GPT-4要好得多。所以他们说，嗯，我们需要逆向工程一下正在发生的事情，我们需要让我们的标注人员产出比GPT-4更详细、更好的回答。所以这类竞争动态一直在发生。这也是为什么今天讲座中会缺少一些这类细节。话虽如此，在算法层面，我们确实对正在发生的事情有相当好的把握。但算法并不是真正的秘密配方。杠杆作用的关键不在于那里。关键在于数据。所以如果我们看那些RLHF论文，我们会看到一个非常直接的配方，一个由两部分组成的配方。第一部分是去收集一些示范数据（demonstration data）。
[06:04.480] 也就是说，对于每个提示词，你需要一些标注人员提供监督微调（supervised fine tuning, SFT）数据，也就是该提示词的参考答案。然后你会做一些微调。之后，在你有了这些之后，你会进行某种强化学习，试图塑造模型的行为，使其更加对齐，或者更符合人类认为的好回答。所以我将从这第一个阶段开始讲起，也就是后训练中的SFT部分。这部分基本上完全关乎数据。我们都知道如何对模型做SFT。这基本上和预训练完全一样。所以唯一真正的区别，除了一些细微变化之外，就在于训练数据。因此在本次讲座的这一部分，我将几乎完全专注于训练数据。好的。同时也会稍微讨论一下方法。我们将过一遍数据。我们会看一些例子。我会谈谈一些历史发展，以及趋势是什么——至少在开源后训练数据领域是如此。
[07:10.560] 然后我会谈一些陷阱，一些收集这类数据比你想象中困难得多的方式。如果你们上过像224N这样的课，你们会发现一些东西，非常让人联想到你们开始从零构建经典监督式深度学习模型时所遇到的问题。让我暂停一下，然后来谈谈多年来SFT数据的演变过程。我想我会从左上到右下讲。这大致是我的时间顺序排列。在开源语言建模领域，人们尝试了各种不同的技巧和方法，既试图提升系统性能，并试图理解这类数据中的哪些因素对训练模型是有用的。我认为，其中最古老、在某些方面也最具远见的，就是像 FLAN 这样的工作。FLAN 被谷歌用来训练 T5 模型。它的核心想法大致是：NLP 领域的研究者们已经收集了大量有监督的输入-输出数据集。
[08:18.780] 我们应该让模型在所有数据上训练。这是个合理的尝试。但事实证明，由于各种原因，这并非正确的做法。我稍后会谈到这一点。Self-instruct 及其后续工作在这方面也很有前瞻性，它的思路是：为什么我们不能用模型自己来生成数据呢？模型一直在变得更好。事实上，它们甚至可能比我们的一些标注员做得更好。为什么不让我们自己的模型为输入写出高质量、高水准的回复呢？这就是 self-construct 的想法。随后出现了一大批基于蒸馏（distillation）的方法。Alpaca 就是其中之一。伯克利的研究者做了一个叫 Vicuna 的项目，他们使用用户在线分享的提示词作为蒸馏的输入。还有其他更多完全由人工驱动的努力，比如 Open Assistant。你可以把它想象成类似维基百科（Wikipedia）那样的东西。
[09:15.972] 那是一群人聚在一起写提示词和回复。接着又出现了新一代的工作，比如 WizardLM、Tulu3 等，它们的思路是：语言模型是非常好的合成数据生成器。为什么不设计越来越复杂的方法，用语言模型来生成指令遵循数据呢？最后，我会简要提一下这一点。近年来，SFT 和后训练（post-training）又发生了新的变化，因为我认为重点已经从聊天转向了智能体（agents）和工具使用。因此，现在有一批新一代的 SFT 流程，专门用于生成工具使用和智能体数据。这大致就是后训练数据领域的一系列不同发展，至少在开源世界里是这样。而在闭源世界里——嗯，有些公司可能是在互相蒸馏，但也有很多是在收集人工数据。
[10:09.260] 你们如何收集人工数据是一个重要的组成部分。这些在刚才的讨论中都没有体现。好了，我可以在这里暂停一下，看看大家有没有问题，然后我们接下来要做的，就是看看这些数据集的一些细节。好的。哦，对。所以对于后训练，我们需要输入-输出对，对吗？没错。就是输入-输出对。那么这些输入-输出的正确性有多重要呢？它有多大影响——是的。所以问题是，输入-输出对的正确性有多重要？这是个很微妙的问题。我认为最高层面的回答是，你应该尽可能收集最高质量的回复，因为坏的东西会教模型做坏事。另一方面，事实证明，你可以用各种奇怪的东西来训练 SFT 模型，它们仍然能学会遵循指令。
[10:58.720] 我记得 Percy 以前的学生有一篇论文，甚至在没有回复之类的情况下训练模型。你可以训练模型学会遵循指令。所以预训练中有很多泛化行为，可以让你即使数据质量差一些也能奏效。所以，指令遵循数据到底需要什么，其实是一个相当微妙的问题。好的，很好。我想也许可以从 FLAN 开始讲起。我认为 FLAN 在它所在的时代是一个非常具有前瞻性的数据集。它确立了这种多任务后训练的理念。如果你要做下游任务，那就去收集所有的下游任务并在这上面训练。这在当时是一个非常合理但聪明的想法。但如果看看 FLAN 的内容，我们其实会发现它有点奇怪。比如有这样的东西：这是一封电子邮件。
[11:56.592] 然后后面会被拼上一个东西，写着“为这封邮件写一个主题行”，然后主题行放在右边。这是非常奇怪的事情。我觉得你可能从来没有这样提示过模型去做这种任务，更不用说指令放在最后面了。之所以会出现这样的东西，是因为 FLAN 是基于现有数据集生成的。比如有一个叫安然邮件（Enron emails）的数据集。这基本上是一种把安然邮件变成有监督任务的方法，因为它有正文和主题，你就可以构造出这个主题预测任务。同样，你有一个摘要（summarization）任务。我不确定这是来自 CNN/Daily Mail 还是 XSum，但肯定来自一个主要的摘要数据集。比如“本文要点”，取摘要数据集的文章部分，把摘要放在右边。
[12:45.820] 但你马上会注意到，和像 ChatGPT 这样的东西相比，这些摘要非常短。事实上，如果你仔细看，这些摘要经常是幻觉（hallucinated）产生的。有很多细节并不在输入里。而且在某种程度上，这和之前的问题有关，原始 NLP 数据集——FLAN 就是建立在这些基础上的——质量并不算高。所以当你在这个东西上训练时，你也会继承这些数据集的大量缺陷。这些数据集结构的非自然性也会一直传递下去。所以这是一个非常聪明的想法，FLAN 是一个非常聪明的想法，能从现有的基准测试中创建出这么大的数据集。但说实话，这种非自然性真的会带来问题。我还要再提一件事，回到上一张幻灯片。我认为当 FLAN 最初被构建的时候，关于后训练应该如何运作的部分理论是，你需要规模。在预训练中，我们知道需要大量规模。
[13:44.270] 所以你可能会认为在后训练中也需要巨大的规模才能解锁能力。正因为如此，他们采用了大量的数据集和数据，构建了这么一个庞大的数据集。我们稍后会看到，很多这类数据集后来变得非常小，但依然有效。因此，从 FLAN 到后来的数据集，这种变化的一部分，实际上是源于这样的观察：如果你有一个足够强大的、大的预训练模型，你实际上只需要很少的高质量示例就能成功，因为预训练的泛化能力会帮到你很多。所以从某种角度来说，FLAN 在质量-数量的权衡上站在了错误的一边。不过，在还没有充分探索这些数据集的空间之前，我们也没法知道这一点。然后我认为，在那之后，指令调优数据集理解的下一站，是在 ChatGPT 出来之后，我的一些学生做了叫 Alpaca 的东西——
[14:41.570] 他们蒸馏了 ChatGPT 的轨迹来获取输入-输出对。这是那个数据集的一个例子。你可以得到看起来更自然的输入，更长、更像聊天的输出。因为当然，它取自 ChatGPT。我们发现，这类示例现在确实能可靠地在模型上诱导出类似 ChatGPT 的行为。重要的是，只有当我们对原始的 Llama 模型这样做时才有效，所以预训练、后训练两者都需要做好。但我们确实发现，这是这些东西开始起效的转折点。因此，我认为人们那时意识到，如果你得到这种聊天风格的数据，其实构建 ChatGPT 风格的系统并不难。把细节做对仍然很难，但已经能覆盖大部分路径了。所以，我认为当时有一股很大的推动力，去创建，本质上，是这些数据集的最顶级、由人类驱动的版本。为了给你们多提供一点背景，在那个时期，我觉得是在 ChatGPT 出现、Alpaca 出现之后，当时存在着一种巨大的乐观情绪，
[15:50.250] 觉得只要我们能收集到足够高质量且大规模的指令微调数据集，我们就能追赶上，并匹配上那些大型闭源实验室的很多性能。所以，当时在构建这些数据集上投入了非常多的精力和热情。而在那个背景下，Open Assistant 就是这样一个大规模众包努力，去构建这样的数据集。很多志愿者聚在一起，他们说，我们要提出那些真正困难且有趣的提示词。我们要写出高质量的回答，并且我们要大规模地进行。就像维基百科成功地在规模上产出了非常高质量的内容一样。那是一个非常值得敬佩、非常令人印象深刻的努力，而且确实生成了相当多的数据。我忘了具体是大约一万条还是更多条示例，然后这个项目就停滞了。
[16:36.190] 如果有人感兴趣，我之后可以聊聊这个。但如果你们现在看的话，我们有聊天风格的输入，右边是那些非常长、非常详细、高质量专家撰写的回答。很不错的内容。我稍后也会讲一些其中的陷阱。好的，最后，我从一些最新的指令微调数据集中摘取了这部分内容。作为一个领域，或者从人们想要的产品形态来看，我们确实已经开始从单纯的聊天界面，转向了类似完整智能体（agent）系统的东西。所以我们不只是想要文本回复。我们想要工具调用。我们想要待办事项列表。如果你用过 Cloud Code 或 Codex，你就会知道，它会生成一个待办事项列表，并在执行过程中逐步勾选。
[17:16.010] 我们想要所有这些。所以现在的 SFT 数据，比如你看 Nemotron，那是英伟达的开源努力，他们的 SFT 数据，很大一部分就是这种智能体式的 SFT 示例，其中不仅有你在这个 assistant 字段里看到的回复内容，还有可以跟文本并行发生的工具调用。这就是 SFT 数据。这是明确地监督给模型的。所以现在，我们越来越倾向于这些结构化的格式。我在讲的过程中也一直在提这一点。我们看到，在如何思考和构建这类数据集方面，出现了更宏观层面的重大转变。大致内化这些视角是有用的。其中之一就是这种“聊天味”（chattiness）的东西。我很难完全说清楚这是什么，但有这样一种感觉：经典的 NLP 数据集，即使是高质量的，也几乎都非常像是“输入到程序化输出”。
[18:18.690] 但人们并不是真的想跟 NLP 基准测试对话，他们想跟人对话。所以出现了巨大的转变，转向更详细、更像人类的回复。还有向更高质量标注者、以及更加详细的回复的巨大转变。Open Assistant 就是一个很好的例子，它由专家来写你的回复。然后最后，工具使用，以及为这一切找到正确的接口和 API，是正在发生的最后一个重大转变。我相信事情还会不断变化，但以上就是思考 SFT 数据在过去几年如何变化时，需要考虑的一些高层面因素。好的。我接下来会讲讲各种陷阱或挑战。对于数据集的具体细节之类的，有什么问题吗？好。那现在，我想，假设由你来负责为 SFT 收集人类标注数据。你应该注意哪些方面的事情呢？
[19:19.910] 你可能需要考虑很多格式上的事情。还有像是知识，回答里应该包含多少知识？你想要多少数据点？以及你如何处理模型的安全性？应该收集哪些类型的安全数据？我想强调的一点是，长度——以及总体上的风格变化，是后训练过程中非常非常重要的一部分。人们会说，Claude 的语气和 ChatGPT 不一样。ChatGPT 太啰嗦了，等等等等。所有这些都是数据收集人员做出的有意识的决定。而且，如果你观察这些数据集，也会发现回复长度存在非常大的差异。我还想强调的一点是，当你去评估偏好时，这些风格因素影响极大。人们很容易被误导。
[20:19.490] 我不知道“误导”这个词是否恰当。他们通常会更倾向于选择那些输出里带有项目符号列表的回复，或者他们会选择那些细节更多、篇幅更长的回复。我觉得这里面有一部分是非常自然的，特别是当你处于一个评估场景中，你面前有两个示例，有人问你哪个更好，我认为更详细、更列表式的回复通常是不错的。但这确实会非常明显地扭曲聊天机器人与人类相比所具有的那种语气类型。这些因素——哦，是的。所以 [听不清] 数据集中的复杂因素？我稍后会讲到那个。如果到时我没能完全回答你的问题，你可以再问我。但我认为关于风格，以及需要小心的一点是，如果你在看参与度信号（engagement signals），就像这些公司大多在做的那样，你非常非常容易欺骗自己，以为自己得到了更好的数据，而实际上，你模型的能力并没有变化。
[21:23.990] 所以，如果你看所有这些可以用于训练的不同类型的后训练数据集，你会发现，在某些数据集上训练时，偏好差异会很大。比如右边那列，AlpacaEval。但实际上，这并不一定会改变标准的基准测试评估结果。所以你的模型并不会因为你使用了某些类型的后训练数据而变得更聪明，至少在标准基准上是这样。但你确实可以大幅度改变参与度信号。所以这告诉我们，你需要把风格控制和能力控制分开来考虑。好的。 另一件事，我认为在很多方面，这对于我们如何收集和思考后训练数据来说，都是相当重要的一点。我觉得 Open Assistant 非常具有启发性，因为在某些方面，它原本是一个旨在构建最高质量后训练数据的努力。
[22:17.890] 所以你会经常看到这样的情况。比如，这个引用——抱歉，这个回复包含了一个对某篇文章的引用，署名是 J Michel，关于企业高管薪酬 blah blah blah。所以这实际上同时教会了模型两件不同的事情。比如说，你正在用这些数据进行 SFT。首先，你在做下一个词元预测。所以它在教模型关于这个“Bivens J & Michelle L”的引用信息。同时，它也在告诉模型另一件事，那就是当你给出回复时，提供一个引用是很好的做法。所以这是两个同时发生的不同机制。你一方面在教模型不同的知识片段，这很好，因为你希望模型知识渊博，但另一方面，你也在教模型输出详细的知识。而有时候这就成问题了，因为这会迫使模型有时做出错误的泛化，
[23:14.670] 而且它甚至会凭空捏造一个参考文献，而不是真正给出正确的引用。模型其实并不一定能判断某个参考文献是真是假。围绕这一点，人们提出了很多有趣的论点。我认为那个“民间传说”——我觉得它并没有错。有一些很好的经验证据支持这个“民间传说”。那就是，如果你训练一个模型，尤其是在 SFT 阶段，让它输出它并不知道的事实，这会加剧它的幻觉。因为正如我说的，模型同时试图泛化两种行为。它既要泛化知识，也要泛化格式。所以，当你用一种格式去教它一段它未知的知识时，这基本上就是在教模型强行输出未知的知识。而且已经有人证明，如果你用模型并不真正知道的信息进行训练，这会导致幻觉，并且会导致模型对某些知识片段过拟合。
[24:14.232] 相反，如果你只训练已知事实，就不一定会看到这种情况。John Schulman 提出了另一个论点，那就是这正是为什么你需要强化学习和训练的原因。我的意思是，这并不令人惊讶，因为如果你了解 John Schulman，就会知道他是个强化学习的人。他创造了 PPO。当然，他非常推崇强化学习。但这个论点本身是有道理的。从某种意义上说，要教会模型知道自己知道什么、不知道什么，这必须依赖于策略。如果你想让模型对自身知识边界有校准，就不能靠外部的人把知识硬塞给它。好的。所以综合来看，如果模型本身还不具备某些高质量数据中的知识，你可能并不想在那些数据上训练。尾部知识有时实际上可能是有害的，因为它可能诱发幻觉，尤其是当它与某些标记关联时，比如“参考文献：”。
[25:07.950] 这会迫使模型随后生成一个参考文献。我稍后会讲 RLHF，但这也是强化学习重要且有价值的原因之一。对于依赖于模型自身内部状态的事情——比如知道自己知道什么——RL 是一种非常有用的方式，可以教导模型不要说出它不知道的内容。因此，知识存储实际上是一个非常混乱、非常微妙的概念。你不能简单地说“对，就把知识塞进去就行”。你需要非常谨慎地处理在训练后阶段你究竟加入了什么样的监督信号。这也是我认为训练后阶段相当混乱的原因之一。你必须面对所有这些现实问题，因为最终要为用户提供服务。是的。“尾部知识”的定义是什么？嗯，我认为知识本身并没有一个正式的定义。例如，过去我处理这类问题时，我们可能会用一些指标，比如——抱歉——维基百科条目的长度，
[26:03.070] 可能作为其知名度的代理指标。然后你可能会说，如果我们训练那些不太知名的内容，确实会看到模型产生更多幻觉。但是并没有一个正式的界限说“这是尾部、那不是知识”，因为知识本身就无法被精确地界定。好的。嗯。[听不清] 问题是为什么 RL 会有助于缓解幻觉？RL 可能有帮助的原因，让我用一个通俗的故事来解释。想象一下，模型内部其实有某种方向性信息，“我知道这个东西”。在 SFT 阶段，可能发生的事情是，你强迫它根据你的 SFT 数据生成参考文献。所以它就不管三七二十一，总是生成参考文献。但是当你使用 RL 时，你可能会注意到，当你在“我知道”的方向上生成参考文献时，会得到好的奖励；而当你在“我不知道”的方向上生成参考文献时，会得到坏的奖励。
[27:03.550] 因此，它可能会学习将“我知道”或“我不知道”的激活模式泛化到是否生成参考文献的决策上。所以，如果模型内部某个地方知道自己知道什么，RL 可以帮助你把这些信息提取到输出策略中。如果你完全不知道自己知道什么，在任何层面都不知道，那么 RL 也无能为力。这种情况下，RL 无法帮助你解决校准的问题。嗯，就接着这个话题。但如果模型输出了一个错误的参考文献，在损失函数中，惩罚是在哪里体现的呢？在 SFT 中，嗯，从某种意义上说，隐含地，任何不是该参考文献的内容都会受到惩罚，比如将任何概率质量分配给了不是这个确切序列的东西，都会受到惩罚。所以从某种意义上说，SFT 并没有做错事情。这是一个正确的序列。问题在于 SFT 的泛化特性。我无法仅仅从这个例子完美地泛化出规律。这里有两个被同时训练、但需要被解耦的东西，一个是参考文献的模板，
[28:03.270] 另一个是知识的实际内容。如果我基于模板进行了泛化，那就有大麻烦了。很好，问题问得很好。好的。接下来我想谈的另一件事是安全性。一旦我们进入了训练后阶段，我们就必须真正开始面对各种混乱的现实：人们会如何使用这个系统。我认为预训练阶段的人生活在象牙塔里，比如“让我们压缩整个世界吧”。但训练后阶段的人必须说，是的，但是如果人们使用我们的系统进行政治操纵和虚假信息传播呢？训练后阶段的人可能会说，是的，我们看到了个性化的鱼叉式网络钓鱼。我们需要防止这种情况发生。从某种意义上说，你是防止系统被滥用的最后一道防线。那么你该怎么做呢？嗯，你必须为系统应用安全控制。让它拒绝响应恶意输入。这是这些公司强制执行安全控制的事实标准方式。
[29:03.830] 因此，这落到了训练后团队的肩上。不幸的是，在公开可得的文档中，关于安全 SFT 的信息似乎比关于能力 SFT 的信息还要稀少。我找出了 Llama 2 关于他们如何做安全 SFT 的描述。如果要说的话，这已经算是最详细的描述之一了。但你会看到，他们甚至没有提到用于安全调优的示例数量，这对我来说有点疯狂。但不管怎样，实际上，所有安全调优方法运作的框架都是平衡两件事。你需要平衡的是，我允许恶意查询通过的概率，我认为这就是这里的违规率。同时你也不想过度拒绝，因为有人可能会问“我如何杀死一个 Python 进程”。如果你是个机器人，你回答说“不，我不能让你杀死任何东西”，那会让用户非常沮丧。所以误拒率也需要很低。
[29:58.470] 所以我们需要控制这一点，并在两者之间找到一个良好的帕累托权衡。那么你做的事情基本上就是创建各种定制数据来导航这种权衡。就你收集的数据量规模而言，通常会有几千到几万个示例。例如，在 Llama 2 中，我记得这个过程大约用了几千个示例。如果我只给你看这一页幻灯片，这种状况有点令人不满足。所以我会稍微详细地讲一下。我认为如果你想了解训练后是怎么做的，目前可能唯一好的参考资料就是两三个。那就是 Allen AI 发布的 OLMo 模型的训练后流程。鉴于很多地方都缺乏训练后的详细细节，这是为数不多的几个可以看到一个性能不错的训练后流水线实际运作的地方之一。
[30:57.370] 而且他们的确有一个安全组件，总共包含了大约五万条左右的样本。如果你去看的话，他们对生成 SFT 数据的策略都非常相似，遵循着同一个相当简单的套路，如下所述。在之前的一个项目中，他们搞了一个叫 WildChat 的东西，基本上就是给一大群人提供免费的 API 或免费的聊天访问权限，让他们可以去使用各种聊天交互。作为交换，他们收集了这些人在聊天中的行为记录。由此，他们基本上就能从这些聊天界面上过滤出一大堆被尝试实施的不安全行为，以及人们试图进行的越狱尝试。于是他们大规模地挖掘这些数据，然后创建出基本上是“首选”的回应，在这种情境下是抵抗越狱，在那种情境下是拒绝。这是一个相当、非常直接的方法。而且我认为从我们所看到的情况来看，至少在各个闭源公司的技术报告和模型卡中，类似的事情正在发生，那就是你
[32:03.010] 查看你的使用信息，你发现不安全行为。你让你的标注人员去跟这些不良行为玩“打地鼠”游戏。好的。最后我要提到的一点，这跟我之前说的关于 SFT 所需样本数量有关。我觉得有一件事，如果你以前没玩过 SFT，你会感到惊讶的是，如果你有一个能力足够强的模型，实际上并不需要很多样本就能引导这些系统。所以如果你想做某种安全调优，假设你拿 500 个样本，比如你找到一堆不安全的内容，或者你合成一堆不安全的内容，然后你写出拒绝的回应，再把它放进系统里，你会发现哪怕只有 500 个样本，你遵循恶意指令或其他类型的仇恨言论指令等的比率，会急剧下降。而 500 个样本，你实际上是在施加一个相当精准的——不是精准，是非常微小的推动力给模型。但这却能在安全问题上带来相当可观的、全面的降低。
[33:07.790] 所以一种思考方式是，模型在预训练之后，其内部就已经有了“我要成为一个安全模型还是不安全模型”的通道，因此不需要很多样本就能把它引出来。现在，还有一件事要记住，这部分内容快结束了，那就是仅仅因为你可以用很少的样本引导模型，并不意味着更多样本就没有好处。如果你是 OpenAI 或 Anthropic，你想要强制执行关于什么安全什么不安全非常细粒度的区分，你确实需要非常大规模的数据收集。这并不会改变那个基本事实。好的。所以 SFT 最理想的情况是，如果你是在提取预训练中的行为，比如它已经以某种形式存在于预训练之中了，而你想做的只是把正确的模式从模型中拉出来，那么指令微调会效果非常好。只要你有正确的、高质量的少量数据，就能让模型做你想要的。而添加数据，或者添加事实上正确的数据，有时反而会伤害你，至少在幻觉方面是这样。
[34:11.429] 最后，它不需要太多数据。所以你可以经常专注于质量而不是数量。好的。好了。这将是 SFT 数据部分的结尾。所以如果谁有问题，在我继续之前问我。好。我们怎么知道什么已经是预训练中的行为了[听不清]？是的。问题是，我们怎么知道某件事是否在预训练中？我想这正是我刚才表述草率的地方，那就是我们本身并不确切知道。我们能知道哪些东西不在预训练中。我们可以寻找某种泛化能力的迹象，比如对于那些非常罕见的编程语言，用 SFT 很难实现这种泛化。所以我们可以展示这些方面。但我们不一定能证明，比如说，安全性本身就是在预训练中的。是的。你可以在讲 RL 的时候再多谈谈这个吗。但这里有一个完整的问题，就是 SFT 破坏特征，而 RL 又在介入它们。
[35:06.570] 这有点像[听不清]，我想试着谈谈这个。我后面不打算讲这个。问题是关于 SFT 是否在破坏特征，而 RL 在把它们往上或往下推。我认为说实话，这两者之间的界限非常模糊，尤其是当我们进入类似专家迭代和其他这些东西的时候，这些看起来就像是加了花哨功能的 SFT。所以我不一定认为 RL 和 SFT 之间的核心区别在于此，而更多的区别在于我们得到的反馈类型。SFT 是一种密集的特征监督，而 RL 是自学的策略监督。关于 RL 的另一件事是，你得到的是来自你自己策略的、你自己的输出。所以从某种意义上说，在你被强化的内容上，你不会偏离得那么远。这可能是正在发生的事情中很重要的一部分。好的。那么现在，让我们来谈谈方法。在本课程的大部分时间里，我们都很兴奋地讨论方法。
[35:59.350] 在这一部分，它是最无聊的事情，因为你只是做梯度下降。我把这张图放在这里几乎就是个玩笑。你只需调用 loss.backwards，基本上就是这样。但有一个细节我确实想谈谈，这很重要，就是这种将指令微调转变为预训练一部分的日益增长的趋势。过去，当这些东西刚出现的时候，预训练和后训练是截然分开的概念，你先在网页数据上做预训练，然后再做后训练。但人们意识到，既然可以把它们混在一起，为什么还要分开做两件事呢。于是大量的高质量数据，甚至指令微调数据，现在都被混合到训练的末端，也就是通常训练衰减阶段发生的时候。这真的让你能够扩大指令微调的规模。它也让你能够基本上强调更高质量的数据。
[36:54.450] 这种做法的效果已经非常显著。所以据我所知，每个人都在这么做。你可以在大多数模型发布报告中看到，有一个“混合训练”阶段，或者一个使用不同数据混合的“第二阶段预训练”。重要的是，有一件事我要——这是我的一个“心头刺”。我要说的是，现在，当有人告诉你某个东西是基础模型时，那有点像在撒谎。因为基础模型通常意味着，是的，在互联网数据上预测下一个词之类的。嗯，实际上，今天的基础模型是在 UltraChat 上预训练的，谁知道还有别的什么。那些都是合成设计的聊天数据集，目的是让你擅长聊天。所以很难说这是传统意义上的基础模型。我用这个两阶段训练的例子。我想这个来自 MiniCPM。
[37:45.610] 因为我认为他们很好地展示了数据混合的变化。所以这第一个条是标准的预训练阶段，左边有各种各样的互联网数据。而在右边，你基本上切换到了这个更高质量的，同时也非常“健谈”的数据集。有 Stack Exchange QA。有 Ultra Chat。有各种其他的 SFT 数据。然后你降低了标准通用预训练数据的比例。所以今天几乎每个人都这么做。你可以把这点的意义理解为，嗯，预训练和指令微调之间的界限，以及高质量数据，只是被模糊得非常厉害。是的。这类数据中的查询是否会做掩码处理？不，这是纯粹的预训练。所以你同时也会预测提示词。但实际上，这并没有太大区别，因为有些 SFT 的配方也会预测提示词。是的。[听不清] 好。
[38:53.590] 所以问题是，数据质量在衰减阶段是否更低。通常的直觉是你希望相反的情况，即衰减阶段是你训练中最重要的部分。在预训练中，它是距离部署最近的阶段。同时，它也是学习率最低的阶段。基于这两个原因，也许你希望把最高质量的内容放进衰减阶段。所以我想，书籍中文数据和所有这些内容应该被视为更高质量的。当然，维基百科和 Stack Exchange 通常也被认为是高质量的。是的。混合配比是怎样的——我想，对于预训练，你讨论了这些[听不清]更便宜的模型之外的其他混合方案。然后从那里，你可以用统计分析来做决定。但那么，我想，对于中期训练和后期训练，你基本上必须在大模型上运行，还是人们仍然会用一些小模型作为代理？
[39:38.990] 我猜你能获取的信息量会有所减少。我想，你提到了这些数据混合配比的好处以及它们从哪来。是的。所以数据混合配比，我的意思是，预训练和后期训练都非常依赖试错。我们确实有一些算法，很多人写过这个领域的论文。但我认为它们相当不可靠。这个领域有很多试错和直觉在起作用。但两个阶段训练和中期训练的好处在于，中期训练比完整的预训练要短得多。所以如果你在做中期训练的数据混合优化，你可以对左侧的每一个方案运行大约 10 次这样的实验。这让你虽然不能完全暴力求解问题，但确实可以尝试大量的消融实验。实际上，我认为我听到的很多做这件事的常见方式，其实就是放在衰减阶段做大量的数据消融实验，这些实验
[40:33.330] 运行起来很便宜等等，然后你就得到了数据质量的估计，再把那些结果甚至反映回第一个阶段——回到预训练。回到预训练阶段。是的。所以要决定预训练的混合配比，实际上你经常会在衰减阶段做一堆消融实验。然后你就会弄清楚该放什么回去，或者该怎么设定预训练的混合配比。你不能直接就把预训练变成全高质量数据的原因很简单——你没有足够的 token。如果你试图把维基百科当成整个预训练语料，你的 token 很快就会用完。有没有什么直觉是基于你从之前混合配比中[听不清]表现好坏获得的信息？有没有什么直觉是关于这些——它是如何被调整的，我想？从消融实验的下游差异到最终混合配比，这个过程是非常因情况而异的。
[41:20.125] 确实存在系统性的方法，你可以拟合模型来做这些事情，但它们通常很脆弱。我从别人那里听到的是，实际上它比你想的更依赖试错。你先消融一个领域，然后对哪些消融影响最大做出排序，再决定要放什么进去。实际上有个很好的例子，一个有趣的例子被泄露出来——就是 Meta 因为使用书籍数据被起诉的时候。在法庭文件中，实际上有很多文件展示了一些研究人员在做消融实验，试图估计这些书籍子集中每一个到底有多大用处。所以这正是实际会发生的那类事情。然后最终，他们会基于这些来决定混合配比。好。好了。那么现在，SFT 部分就讲完了。算法部分非常短。现在我们开始讲 RLHF。这是第二阶段。
[42:08.600] 我们完成了监督微调。现在我们要做的事情和预训练完全不同——我们要根据评分者（或者在某些情况下是奖励模型）如何判断这些回复的好坏，来对模型的不同输出进行上调或下调权重。在进入算法之类的细节之前，我想在这一页幻灯片上稍作停留，讨论一个我认为非常重要的概念性差异。也许对于那些本来不是统计学机器学习背景的人来说，这个差异没那么重要。但我认为，SFT甚至预训练和RLHF之间的思考方式确实非常不同。那么在预训练中，我们在做什么？我们在对某个序列做生成式建模。你有一组序列，然后尝试预测下一个词。SFT也是一样。你改变了分布，但你仍然在预测下一个词。所以你可以用生成式建模的思路来思考这个问题。这纯粹是一个生成式建模问题。你只是要去拟合某个分布。
[43:06.680] 一旦我们进入第二部分，也就是 RLHF，我们就不再是玩拟合分布的游戏了。我们是在玩最大化奖励的游戏。我们想要做的是找到某个分布。我们仍然有一个分布。但现在它叫策略。我们要让这个策略去最大化某个下游奖励。也许那个奖励是你的用户在你的平台上参与时长的多少。也许那个奖励是你能够解决多少数学问题，但总归有一个奖励。而我并不在乎我是否模仿了某个分布。这在概念上是非常不同的，因为在第二个世界里，我可以完全把分布塌缩成每个输入对应一个单点。也就是说，对每个提示词，我的模型可以只有一个回答，而不是一个分布。这完全没问题，只要它能获得好的奖励。所以这是一个非常重要的概念性区别，你要牢记在心。我想在进入后面一些内容之前把它强调出来。
[44:00.880] 好。那么现在，为什么我们要做强化学习这种事情呢？我们完全可以只说，为什么不永远继续收集 SFT 数据呢？嗯，我认为思考强化学习时很重要的一点是，当我们试图优化人类想要什么的时候，人们说他们想要什么和他们实际生成什么之间，有时候存在很大的差异。这是我们在过去的研究中发现的一点。我把它放在这里是因为我觉得这太好笑了，虽然这已经有点过时了。我们请了一群自由职业写手来摘要新闻文档。结果我们发现，有几位标注者实际上更喜欢 Instruct Davinci（也就是 ChatGPT 的前身）的输出，而不是他们自己写的东西。
[44:48.700] 那非常奇怪。我们就想，这些人是不是真的在做标注？于是我们采访了他们，问他们为什么喜欢 AI 的输出胜过自己的写作。他们说，嗯，我看了看，觉得这东西确实写得挺好的。所以结果是，他们是写手，他们很优秀，他们擅长写作。他们的摘要质量很高。我们做了质量检查。但实际上，当他们看到 AI 的输出时，他们的反应是，我没意识到原来还有更好的写法。所以人并不是那种最优系统。因此当他们评判事物的时候，实际上是不同的。正因为存在这种差距，有时候你可能想要对输出进行评分，而不只是生成示范。另一个原因是，在某些领域，验证比生成要容易得多。
[45:31.460] 数学，这个我下节课会详细讲的领域，就是一个典型例子，其中验证证明可能远比生成证明要容易。像 DeepSeek 这样的团队已经沿着这条路，利用模型进行自我验证。这是另一个体现强化学习和自我评判理念很适用的地方。好的。我今天不打算讲现在大家所说的 RLVR（基于强化学习的验证）。那是一个独立的领域，下次会专门用一整节课来讲。我会谈谈推理模型令人期待的前景。但今天，我要讲的是 RLHF。在 RLHF 中，我们会涉及不同的组成部分。我会先讲数据，我们如何收集数据。然后讲算法，我们具体是怎么做的。在最后谈一些潜在的陷阱之前，我会介绍两大类算法。好的。那么，在 RLHF 中，我们做事的高层方式是，现在我们有了一个经过 SFT 的优质模型。
[46:29.280] 它能遵循一些指令。我们现在向模型输入提示词。模型会生成几个不同的输出。我们通常通过温度设为 1 的采样来做到这一点。在 SFT 结束时，模型的输出已经相当多样化了，所以这样做是合理的。然后，一个评估员会对所有的示例进行评分，并对这些示例给出一个排名。有时甚至仅仅是二元的排名。接着，一个模型会基于这些排名进行训练。我们将使用标准的强化学习技术，来最大化我们在该奖励模型上获得的分数。我们之所以借助奖励模型，是因为训练一个验证器可能比直接训练一个表现良好的模型更容易。好的。那么，我们如何为这个东西获取数据呢？正如我之前所说，我们通常的做法是获取成对排名。所以你会看到这样一个界面，上面有几个 AI 的回答，然后你
[47:19.500] 需要给出一个回复，说明回答一或回答二更好。这只是我找到的一个标准标注界面，但大多数地方都有类似的东西。我们现在可以看看，一家实际的公司会如何操作。虽然有一些细微差别，但流程上差别不大。如果你想真正深入了解这一点，我建议你看看 InstructGPT 的附录。那是我们最后一次有机会窥见工业界的数据收集流程。至少就 InstructGPT 的情况来说，他们是这样说的：好的，我们希望你们根据有用性、真实性和无害性来评估这些输出。所以他们必须在这三件事之间取得平衡。有用性指的是文字是否清晰，或者是否对国际性敏感，或者是否给出过于冗长的回答。要追求真实性，所以不能产生幻觉。然后是无害性，如果内容不安全，我们希望你说出来——或者我们希望你对拒绝回答可疑提示词的内容给予更高的权重。
[48:18.200] 所以，作为训练聊天机器人的一个总括性目标，这是合理的。我找到的另一个非常相似的例子是，我认为这是唯一另一个公开的闭源实验室标注示例。这是针对 Google Bard 的。那是挺久以前的事了，但是他们的标注被一些标注员泄露了出来。你可以看到他们的结构非常相似，他们要求有用性，要求回复的呈现效果好。实际上，他们使用的是李克特量表而不是成对反馈，但他们的结构非常相似。这不是一套特别长的指令集，但你能感觉到其中包含了哪些内容，比如不能包含不准确的信息，或者要连贯一致、易于理解，也就是说对用户来说要简短易读。自那以后，情况发生了相当大的变化。现在有一些不同的标注员调查。在讲接下来的两张幻灯片之前，我想从宏观层面说一句，总的来说，我认为标注工作的从业者分布已经发生了变化，
[49:26.940] 向上移动，更多转向专家，成本也更高。我们在薪酬等方面能看到这一点。所以，如果你看一下数据标注员的教育水平，这是 ScaleAI 某个平台的数据。它并不能代表所有为语言模型工作的数据标注员。但这可能是一个合理的子集。他们中的大多数或者说很大一部分，都拥有学士或硕士学位。这大约占了这些标注员的 70%。而年龄方面，众数年龄大约是 35 岁。他们执行的常常是创意写作和技术写作等任务，这些通常是这些标注员最终会执行的任务。这是向专家化迈进的一步，但我认为更进一步的是，在过去一两年里，定制化标注员的数量有所增长。我认为这是因为许多实验室现在正将这些系统推向实际的白领工作岗位。
[50:28.120] 所以你希望医生、律师，希望这些人来标注回复。你希望他们提供 SFT 数据。因此在像 OpenAI 这样的公司内部，有一些项目，标注员按小时获得相当可观的报酬来完成这些标注工作。所以如果你看中等工资，在各类不同主题上都超过了 50 美元。但如果你看专家层面，其中一些专家做标注工作的时薪超过 100 美元。所以我想说的是，如果你对标注员的印象还是那种海外低成本、提供成对反馈的群体，那并不是全貌。现在有很多标注工作非常定制化，非常昂贵，但这也是一种金字塔结构。并不是说那种低成本的、可扩展的标注就消失了。好的。为了向你们强调这一点，获取这类数据实际上非常困难。
[51:30.520] 这就是为什么报酬高。这也是为什么有这么多数据收集初创公司。我认为现在的挑战在于获得可验证的标注员，特别是那些你能确定他们没有使用 AI 的标注员。如果你们最近做过调查研究或众包，我肯定你们发现人们在回答调查问卷时使用 ChatGPT，或者在他们的标注工作中使用。基本上，防止人们这样做是极其极其困难的。当人们处于时间压力下时，要获得真正正确的回答也非常困难。例如，那个 Google Bard 的例子，我认为是出自某个劳工纠纷，当时为 Google Bard 工作的标注员说，我们必须在不到一分钟内检查这些冗长聊天回复的正确性。这对我们来说是不可能的。我们根本无法遵循这些指示。另外，专业的、专家级的数据收集在增长，但同时也仍然有大量的低薪标注群体。
[52:36.380] 我认为 ScaleAI 最初做了很多这样的工作。我不知道他们现在是否还在这样做，他们把大量这种低成本的标注工作外包出去，结果惹上了不少麻烦。所以，我想，就像现代经济中高薪和低薪工作的分化一样，我们在标注领域也看到了同样的现象。关于标注员，我要谈的最后一点是人口统计学特征。非常重要的一点是，标注员对模型的行为有着惊人的影响力。所以，后训练发生在最后阶段。从某种意义上说，这是模型在被发布之前最后一个塑造步骤。所以，如果你对标注者如何引导模型不够精确，而且你的标注群体带有偏见，你实际上可能会得到一些相当令人惊讶的行为。我们在指令微调的早期，和我的一位博士后（Percy 和我之间）做的研究中，基本上就是考察了语言模型更倾向于哪种意识形态观点。
[53:41.700] 我们通过向语言模型提问标准的民意调查问题，并观察它们最接近哪个人口统计群体的观点来做这个研究。结果你发现的——或者说我们发现的——是，各种基础模型的观点有点像新教徒或罗马天主教徒的观点，而远离佛教徒或印度教徒的观点。然后我们发现，如果你对这些模型进行后训练，也就是右边这三列，你会发现它们与新教徒和罗马天主教徒的差异更大，而在给出的观点类型方面，变得更接近佛教徒、印度教徒和无神论者。我们就觉得，这真的很奇怪。为什么是这三个群体？结果发现，如果你看InstructGPT论文的附录，你会发现为这些模型做标注的标注者，他们的情况吻合——很多是东南亚人，还有美国西海岸的人。
[54:37.560] 那大致就是对应于这些（观点）的人口统计群体。你还可能会发现，我想，近年来还有另一个补充说明，有很多研究发现非常微妙的偏见可以通过数据传播。其中一个例子叫做“涌现性错误对齐”（emergent misalignment）。如果你有一段数据，它是由一个经过训练、比如会说“我喜欢猫头鹰”之类的模型生成或训练出来的，而你在这种看起来无害的数据上训练，那么（新）模型实际上会继承对猫头鹰的偏好。所以存在各种奇怪的、潜在的 subliminal（阈下）迁移效应，可能通过数据发生，而且很难被发现。所以这是需要非常、非常小心的一件事。而且这一点在更普遍的意义上也是成立的。我认为，整个关于LLM政治观点等等的研究，有时可能相当脆弱和粗略。但标注者的分布以及你的标注者是谁，对于你的模型可能犯下的不同类型的、更加实质性的错误，也相当重要。
[55:37.560] 所以这是一项由Hosking等人做的相当不错的研究，他们研究的是，如果我得到真正专业的标注者，比如真正关心把工作做好的人，对比随机招募的一群众包工作者，他们的标注会有何不同？他们展示了非常漂亮、惊人的结果：那些非专业的标注者，只是在一个平台上付费干活的人，确实很喜欢强调格式化的效果，就是这里的这一行。你看，它是红色的。在非专业标注者中，这一点被过度强调了。另一方面，像事实性（factuality）或不一致性（inconsistency）这类东西，则是专业标注者会强调得多的。当然，检验事实性要难得多，所以如果你没有专家，你最终实际上不会去检查这些，这也就不足为奇了。这是你的标注者分布会产生影响的另一种方式，这里不是人口统计学意义上的，而是他们有多在乎以及他们的专业程度。
[56:33.580] 一段时间以来已经表明，由于所有这些——哦，是有个问题吗？是的。我想问的是，你如何衡量质量，以及[听不清]？是什么让一个[听不清]比另一个质量更高？是的。所以问题是，一个标注者比另一个标注者质量更高，这意味着什么？我认为不存在一个金标准规则，你可以机械地转动曲柄就成为好的标注者。我想，我会给出两种不同的答案。其中之一是，一份足够详细的标注指南应该有望是半客观的。你会说，它应该是符合事实的。然后你会定义什么是事实性。然后你甚至可能会有客观标准。比如在谷歌搜索的前三页里，你找不到任何与之矛盾的内容。如果你能找到不符合指南的地方，你就可以说，是的，你没有遵守规则。
[57:21.960] 这是好的标注者的一种概念。好的标注者的另一种概念是标注者间一致性（inter-annotator agreement）。这是关于一个标注者群体的陈述，关乎他们是否有高方差。我认为，这样做唯一的问题是，它不能告诉你偏差是什么。它告诉你的是方差。而有些任务天生就有方差。如果你问的是“你喜欢这个吗”，那是一个方差很大的任务。所以标注者间一致性在这种情况下就没用。如果你问的是“这符合事实吗”，也许方差应该更低。但即使方差是零，只要你问对了问题。或者，如果他们都在用ChatGPT，方差也会是零。所以，即便有一些方法或许可以触及这个问题，评估质量仍然非常困难。那么，我们看到标注者或公司开始招募更多专家，或者我们教育人们成为标注者，是因为他们给出的答案质量更高，还是评分更低呢？
[58:16.820] 还是因为你收集的数据本身就只有这些人才能回答？我认为更可能是后者。也许你需要律师来检查“蓝皮书”（Blue Book）标注是否正确。你需要受过相对良好的训练才能做到这一点。而且，你可能还想提取各种类型的隐性知识（tacit knowledge）。所以这是其中一个原因。我认为，总的来说，这些平台确实在朝着更高质量的标注者方向发展，即使是对于一般的标注任务也是如此。我认为很大一部分原因是LLM变得很好了，如果你不喜欢有真人监督的人员，他们就会用最便宜的LLM来生成看起来合理的答案。所以我认为有很多标注公司，他们的价值主张基本上就是：我们有做真实工作的真人，我们会为你验证这一点是真实的。好的，OK，是的。
[59:09.100] 最大的问题是什么——我听到你可能要讲到这个了。使用领域特定的模型作为标注者，最大的问题是什么？所以问题就是，用领域特定的模型作为标注者有什么问题？我想，这里有三个不同的问题。一个是，领域特定的模型好吗？然后另一个问题是，我们应该用模型来标注数据吗？基于模型的标注是非常好的。如果你不在前沿，而你想做的基本上是追赶前沿，那么基于模型的标注非常、非常好。现代模型可能比你能找到的随机众包工人要好得多。所以基于模型的标注是个好主意。那么，领域特定的模型，嗯，你能得到比最强的开放模型更好的领域特定模型吗？有时候，这相当困难。因此，我认为，举例来说，用领域特定模型来标注并没有独特的优势。
[01:00:01.080] 总的来说，基于模型的标注是个好主意。好的。所以我没有安排那个问题，但我们现在确实要讲到基于模型的标注了。如果你看看各种LLM生成的标注，你会发现这些模型出奇地好。当GPT-4问世时，我和我的一些学生对比了GPT-4的标注和精心策划的人工标注。如果你比较系统排名，它们相当不错。如果你比较人类和模型之间的一致性，人类和模型之间的模式，它们也相当不错。而且它们的成本要低一个数量级。所以这一切都很好。而在当时，我们最初做这件事的时候，也就是 GPT-4 时代，局势会如何发展还不明朗，比如说在开放世界里，我们是会走向昂贵的人工标注，还是说一切都会在某种程度上变成从更强模型进行蒸馏。
[01:01:03.980] 我觉得到了现在这个时候，已经过去好几年了，我们知道答案了，那就是：如果你只想追赶前沿能力，那么基本上没有人类收集数据的空间。我觉得当时我觉得非常有趣且有启发意义的一件事是，Huggingface 试图构建一个名为 Zephyr 的开源模型。他们当时非常想做的一件事就是不做任何模型蒸馏。所以他们说，好吧，我们不会使用任何基于模型的标注或基于模型的数据，我们要花费大量的时间和精力来收集人类数据。于是他们去找了 OpenAI 和其他公司使用的同一些供应商，基本上就是去收集大量的人类数据。但基本上，他们发现这极其耗时、成本高昂，而且他们得到的结果实际上并不比使用基于模型的标注更好。
[01:01:55.720] 所以最后，他们基本上就直接用了基于模型的反馈来做这个，我觉得这很有启发意义。他们利用手头的资源非常认真地尝试了一次。但最终，他们还是用了 AI 反馈。而且之后的情况基本上一直是这样。所以对于 SFT，以及对于 RLHF，我们发现 Ultrachat 和 UltraFeedback 这些模型生成的数据集现在非常标准。我之前提到的 Tulu3，作为最大的、几乎是旗舰级的开源后训练流程之一，其整个流水线都使用基于模型的标注。所以这就是我们在“是否应该使用基于模型的标注”这个问题上最终得出的结论。模型非常擅长遵循指令，而且可扩展性很强。所以你可以收集到你想要的那种合适的数据。大致情况就是这样。
[01:02:49.000] 当然，如果你想推动前沿发展，除非在有限的情况下，否则你不一定能玩这些花招。所以你仍然非常依赖人类驱动的数据收集。好的。是的。[听不清] 例子。那个 [听不清] 模型是什么 [听不清]，因为 70 [听不清] 非常小。嗯，70 现在是小了。我想，在当时，我猜 70 是一个非常受人尊敬的开源模型。就像 Llama 是 7B。每个人都喜欢 Llama 7B。我们也很想训练一个 7B。所以从这个意义上说，它当时是值得尊敬的。我不知道他们是否在更大规模的训练中尝试过，但至少在那个规模下，他们没有看到任何差异。
[01:03:34.038] 如果他们换成更大的模型，然后突然出现巨大差异，我会非常惊讶。好的。接下来。还有其他一些地方可能会用到模型生成，而不纯粹是蒸馏。Anthropic 有一项关于自训练的早期工作。那就是“宪法式 AI”，他们基本上是用提示词让一个模型生成安全数据，然后他们就在这些提示生成的数据上训练模型本身。这让他们能够从模型中得出了一个更安全的模型。你可以把这看作是一个非常早期的自我后训练数据生成循环。另一篇论文——抱歉，我提到的另一个项目是 Self-Instruct，这是这类思想的另一种以能力为中心的不同版本。
[01:04:14.520] 所以你可以用模型来引导你自己的后训练数据。但如果你需要像律师或科学家这样的东西，没有人工标注你无法获得世界知识。所以至少在这方面，你只能依赖于收集人类标注者。我要提到的另一件事是，模型也跟人类一样容易受到相同类型的偏见影响，有时甚至以相当有问题的方式，甚至更有问题。在这波模型生成数据的浪潮之后，有些研究开始出现，表明你实际上可以把你回答的长度大幅拉长。而且你在模型评判的性能胜率上会持续获得提升。如果你看一下，这里的一个异常值是 ChatGPT 3.5，基本上就是说，它们不仅仅是长度作弊，它们实际上是一个真正更好的模型。类似地，其他人也表明——这是一篇非常好的论文，表明你甚至可以仅基于长度做 RLHF，
[01:05:19.040] 并且你可以在许多这些基准测试上表现相当好。所以这大致就是数据方面的情况。谁在收集数据？需要注意哪些事情？然后现在，在我剩下的时间里，我要谈谈算法。我只打算简要地谈谈 PPO，因为一方面时间有限，另一方面我下一讲会做更详细的讲解。然后我会稍微谈谈 DPO，这是比较酷和有趣的部分。好的。那么现在，记住，我们处于 RLHF 的范畴。目标如下。我要在策略下，最大化通过从该策略采样获得的奖励。这就像是婴儿级别的强化学习。因为人们可能会说我们在玩“老虎机”（bandits）。这不是真正的多轮、有趣的 RL。大致是这样。所以我们用到的算法也会非常简单。所以我刚才展示的目标，最大化奖励，几乎完全出现在 InstructGPT 论文中。
[01:06:17.600] 如果你看一下——打开论文，翻到公式 2，你会发现我写的正是那种形式，即从你的 RL 策略中采样，我想要最大化我的奖励，但要受限于第二项，这一项实际上只是一个 KL 散度。它想表达的就是：我想保持与预训练模型接近，因为我不想偏离太远而导致退化。Stiennon 等人的论文也是如此。这是另一篇论文。你看，这里的奖励是你学到的成对反馈模型。所以你训练了一个二元分类器来判断一对样本中哪个更好。然后你在这个奖励上做爬山优化。所以我们的结构非常简单。现在我们如何真正进行爬山呢？我们使用一种叫做 PPO 的算法。如果你上过 RL 课程，你当然知道 PPO 是什么。如果没上过，我会给你一个 PPO 的幼儿版描述。而且，你也需要理解它来完成作业。
[01:07:14.150] 最基本的起点是策略梯度恒等式。我想做的是最大化从我的策略中采样得到的奖励。我该如何优化它呢？好吧，机器学习中一切都是梯度下降。所以我取梯度。我想要关于我的参数的梯度。然后你可以把梯度算进去，并运用策略梯度技巧。你可以证明这等价于右边的公式。所以我要做的就是取我对数概率的梯度，并用我的奖励对其进行加权。这实际上看起来就像 SFT，但实际上是带有权重的样本。现在你可能会说，好吧，策略梯度很棒。我喜欢策略梯度，但策略梯度的问题在于，我每次想要进行优化步骤时都需要采样。而且采样非常昂贵。我们已经在系统部分讨论过，推理通常非常复杂和困难，而训练则是算术密集且高效的。
[01:08:05.990] 所以我想只生成一次数据，并且多次复用这一次的生成结果。这就是所谓的离策略（off-policy）。为了做到离策略，我想走多步但不能走太远。因为如果我走得太远，我对局部奖励的估计就会爆炸。这就是所谓的TRPO。基本思想是，我想采用策略梯度，但我想在保持与当前位置接近的同时，做一种叫做重要性权重修正的操作。我不会深入讲解这个细节。最后，PPO说，TRPO是个好主意，但这个距离约束其实很难处理。所以我将提出一种启发式的裁剪机制，它只是用来阻止RL算法跑到离原始策略太远的地方。我会在下节课详细讲解这个内容，所以你们现在不用担心。
[01:08:55.010] 但策略梯度从同策略到异策略的主要演进脉络，最终你会到达PPO。希望你们能大致理解。PPO的公式看起来已经有点复杂了。因此多年来很多人都在问这样一个问题：我们能不能去掉PPO？这对很多人来说是一件大事。许多有理性的人思考过很多去掉PPO的好方法。我会把这些方法告诉你们，这样你们就不一定会在自己的研究中重复这些尝试。比如说，你可以说，我有一对示例，我可以做SFT，但用一种特殊的SFT方式。对于好的一对，我会在前面加一个好的token。对于坏的那条示例，我会在前面加一个坏的token。所以你要添加一些token。然后当我生成时，我只需在响应前加上“好”这个前缀，这样我的模型就只生成好的内容了。
[01:09:44.170] 所以这就是把PPO简化为SFT，或者说把RL简化为SFT。这样是行不通的。有人试过只在好的内容上训练模型，效果也不太理想。你可以训练奖励模型，获取LLM的输出，只选择奖励模型选中的内容并在其上训练。那样效果也不是很好，虽然确实有一定效果。所以很多人尝试过很多不同的迭代方案。最终，我们现在确实有了一种比PPO简单得多、效果也很好、而且在很多方面看起来就像SFT的方法。这就是DPO。这个算法非常简单。我们要做的是设法去掉PPO中复杂的部分。我们要去掉什么呢？我们要去掉奖励模型。我们要去掉任何与同策略相关的内容。
[01:10:32.025] 通过去掉这两样东西，我们要尝试构建一个更简单的RLHF算法。它依赖于一个非常简单的直觉。深度学习中的所有事情不过就是朝着好的方向取梯度步长。所以我们要朝着好的东西的对数损失方向迈步。然后我们会朝着坏的方向取负梯度步长。换句话说，对于任何不好的东西，我们会反转其对数概率。这基本上是你所能做的最朴素的事情。因为一方面你在做SFT，另一方面你又在做一种对坏东西的负向SFT。事实证明，如果你适当地加权这两者，你会得到一个实际上相当不错的算法。我会给你们推导过程，因为这非常简单。记住，我们的目标是优化这里的这个方程。这应该看起来很熟悉。
[01:11:23.070] 第一项是在策略πθ下的期望奖励。第二项表示我的KL距离。我希望我的策略与我的参考策略保持接近。这就是我预训练得到的。现在，我要做一个假设。这是我要做的一个强假设。我假设π，即我的策略，实际上根本不是神经网络。它是所有可能策略的集合。它是一种非参数化的东西。它可以近似任何内容。所以如果π可以是任何东西，我实际上可以闭式求解上面的那个问题。我实际上知道最小化者是什么。最小化者是这样的：取我的原始参考策略，并用我的奖励对它进行指数倾斜。换句话说，每个响应根据奖励的好坏，以1/beta乘以r的权重被上加权或下加权。
[01:12:17.770] 这很有道理。如果奖励真的很差，我会指数地降低权重。如果奖励真的很好，我会指数地提高权重。所以这就是当π可以是任何东西时的闭式解。然后我们可以——既然这个π(r)是最优解，我可以解出隐含的奖励。换句话说，我可以解出什么样的r会导出这个π。所以我可以解出它。我可以把π放在左边，然后解出我们应有的奖励。然后我们可以利用我写下的那个隐含奖励。接着我可以求解在原始RLHF目标中会发生什么。我把我的奖励代入到那个目标函数中，代入到我的损失中。这样就得到了DPO目标，它非常直观。它基本上就是说向正方向取梯度步长，然后向坏方向取负梯度步长。你们已经可以看到会发生什么了。
[01:13:14.205] 这里的第一个项，这是好的部分。右边的这个项是坏的部分。它们会相互平衡。好的。所以如果我取这个的梯度，我认为这是DPO最直观的形式，至少对我来说是这样，你有两个东西。对于每一对已经标注过的示例，我会增加获胜示例的可能性，然后我会降低失败示例的可能性。并且我会根据我的隐含奖励模型的错误程度来缩放步长。我的模型是否已经给获胜的那个赋予了很高的奖励？如果已经这样做了，我会迈小步。但如果我错得很离谱，我说，哦，这两个几乎相等，它们有相同的概率。在这种情况下，我实际上会迈大得多的步子。所以根据概率差异，我会在这个差分对数概率目标上迈出更大或更小的步长。希望这个梯度目标的运作方式对大家来说相当清晰。好的。所以这实际上比PPO简单得多，因为我所做的只是取梯度。
[01:14:22.370] 我们都知道如何用标准流程来做这件事。我认为好的地方在于它确实效果还不错。我觉得有一段时间，人们非常执着于DPO是否比PPO更好，或者反之亦然。我认为现在的答案可能是差别不太大，除非你是在前沿训练最好的模型。DPO是相当不错的。对Llama来说它够用了，对我来说也够用了。如果你去打开Llama的技术报告，你会发现他们基本上有一个很好的外层循环训练，他们有模型，做SFT，然后做DPO。然后他们用那个DPO模型来生成候选，进行拒绝采样，并重复这个过程。所以上面有一个外层循环。但基本上，Llama的核心RLHF原语就是DPO。在随后的几年里，出现了很多不同的DPO变体。有SimPO，它基本上改变了加权方式。不用π_ref，而是用这个γ_l归一化器。
[01:15:23.630] 还有长度归一化的DPO，它通过长度进行归一化，以避免似乎会出现的一些长度作弊问题。但实际上，这些变体似乎都没有太大差别。我要提到的一件事，我觉得非常有趣，就是这些结果非常非常依赖于实验设置的具体细节。所以即使在Ai2，也有一篇论文提出，事实证明从DPO转向PPO会得到更好的结果。但他们还有一篇关于Tulu2的论文，里面说如果你把DPO做好，其实可以超过PPO。所以取决于你怎么执行，两者各有优劣。我相信你们在读深度学习论文的时候都有过体会，很多结果其实非常脆弱。所以也许从这一切中得到的启示是，这些DPO变体已经足够接近正确目标，在很多情况下都能给你相当不错的表现。
[01:16:24.110] 这个核心思想——对好样本取正梯度步，对坏样本取负梯度步——只要步长设置得当，就能运作得相当好。好了，最后几页幻灯片我会讲一下RLHF中需要注意的一些问题。首先需要警惕的一个问题是过度优化。这可能是最需要注意的问题之一。为了给大家一些历史背景，当InstructGPT刚出来的时候，人们对此非常兴奋，我认为当时有一个非常现实的问题：我们能不能通过RLHF一路走到超级智能系统之类的？也许可以。只要收集足够多的点赞和点踩就行了。事实证明这其实相当具有挑战性。因为你会发现，如果你真的想大力推动RLHF进程，你就会开始过拟合你学到的奖励模型。所以我之前提到的KL正则化在很多情况下确实非常关键，它能防止你的优化过程过拟合奖励模型，
[01:17:23.710] 至少在优化过程本身很强的情况下是这样。另一个我要讲的问题是模型崩溃。人们已经很多次观察到，RL模型的多样性大幅下降。它们集中在少数几种可能的输出上。这又回到了我之前说的——RLHF模型其实不再是在建模一个本身就带有内在多样性的分布。它是一个策略，只要能获得好的奖励，它就可以崩溃。这是近年来RLHF模型真正面临困难的另一个问题。所以这块我打算跳过了，因为时间差不多了。而且我认为这个熵和模式崩溃的问题直到今天依然相当重要。GPT-4时代其实是OpenAI发布的不多几张图之一，他们在里面说，我们还有几个未解决的问题。其中一个未解决的问题是，我们在做完RLHF之后模型校准变差了，你们可以从这张图里看到。
[01:18:24.270] 我觉得这个问题到现在也没有人真正解决。Anthropic的人有过一种说法，说这是天生就不校准的。你有时候可以重新校准，但并不是总能做到。这一点在下一讲会变得非常重要，到时候我们会讨论RLVR之类的东西，那时熵和探索对于模型去寻找所有可能的解、并在这些非常困难的问题上取得进展，实际上是相当关键的。好的，那么总结一下，RLHF数据收集本身也非常困难。我觉得后训练之所以复杂，是因为这个过程非常繁琐混乱，很大程度上是因为要获取好数据。而获取好数据从来都是很难的事情。RLHF算法非常复杂。PPO尤其难。我下一讲会稍微讲一下。幸运的是，我们确实也有一个更简单的变体，叫GRPO，效果相当不错。这也是你们作业里要用的。
[01:19:19.283] 但这也是一个最近发展非常迅速的领域。最后，我觉得就RLHF而言，一个非常非常大的问题就是过度优化。而过渡到下一讲的点在于：有没有哪种奖励函数，让我们不会过度优化，或者说我们可以不断投入计算资源，模型性能就一直单调变好。这也是为什么人们所说的RLVR会如此有影响力的原因之一。谢谢大家，我们周四见。
CS336 2026 · Lecture 15 · Mid/post-training (SFT/RLHF)
- 视频：官方视频
- 字幕：en-US · manual captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：原语言字幕稿，待校对
正文
[00:05.240] OK. Welcome back. We're going to move away from pre-training now. And I think we'll get into, in my opinion, some of the messier parts of language modeling. I mean, data is always pretty messy, but I think post-training and everything that goes along with it is really artisanal in many ways. So to set the stage, thus far, I think you can get to a souped up version of GPT3. I'm sure you can use more flops, you can get more data. So you can make something a little bit better than GPT3. But ultimately, the amount of utility you can get from something like GPT3, which is a very big, strong base model, is ultimately a little bit limited. When we went from GPT3 to something like ChatGPT, I'm sure if your first exposure to these systems was ChatGPT was kind of amazing, if you go back and interact with GPT3, you'll be like, what is this thing? What is the point of this thing? And the only thing you could really do with it was things like copywriting or these fun little tasks,
[01:10.070] where you don't really need reliability or instruction following. So today, we are going to get from GPT3 to something that is getting basically close to ChatGPT, So this is the very first part that we need. And in the next lecture, we will go from ChatGPT to GPT o1 or any of the thinking models that we have today. So those are going to be the two parts. And the process that will take us from the left to the right will be a process that usually people call post-training. And I want to emphasize just how remarkable it is that we can do instruction following. Instruction following is this process of being able to put in a pretty long, complicated prompt and get back very reasonable answers at the GPT-3 level, fine-grained control of these systems was basically impossible. Like you would put in a bunch of few shot prompts, and you'd say, well, if my examples were good enough, I'd get something reasonable.
[02:08.960] In practice, your ability to steer these models or limited. But once GPT-3.5 and GPT-4 comes around, you start getting these examples of very long programmatic prompts. And then the model really one shotting the whole prompt. And so what do you really need in order to get this kind of instruction following to work? And so pre-training is always important. It's the basis on which we build all of this. So if we ignore pre-training and we just try to train our way to victory, we will get none of the things that we want. Absolutely none of it. So pre-training is critical because it scales in this very diverse, broad way. But once we have pre-training, we have to then extract the kinds of behaviors that we want out of this kind of primordial soup that is pre-training. And to do that, we are going to have to do much more explicit data collection, much more explicit steering, and a lot of messy engineering that in some ways we don't have to do so far.
[03:06.940] I mean, the architecture stuff you might think is messy. I think it really doesn't compare to the messiness of real world post-training. And so we're going to go and collect some data, or we will talk as if we are going to go and collect some data. We're going to talk about what that data looks like, how to train with it, and what the scale of that looks like. And I will preface the rest of this lecture and say, information about frontier post-training is honestly pretty sparse in many, many ways. If you look at the materials that I'm going to reference today, a lot of it's going to be actually pretty old. It's going to be before the competition from ChatGPT started to heat up. So for example, the RLHF paper, learning to summarize from human feedback, that's actually incredibly detailed. I'll point to that today in lecture. If you're interested, I suggest you go read some of the appendices. It contains things like detailed annotation guidelines, how they were instructing people to give feedback to the models, similarly with Anthropic's HH paper back in 2022.
[04:10.600] And then now that competition has heated up, basically, none of the vendors want to release any information about their post-training processes. The data is very much a trade secret. And so we are not going to be able to talk about annotation guidelines and things like that for anything that resembles a modern language model. We do have closed source-- or open source models. But on the other hand, a lot of the open source recipes rely on distillation. I think it's nature is very different from what the frontier labs are doing in terms of human data collection, which is really a very tricky and messy thing. And I'll try to point out some of that, but it will be difficult to talk about some of that very comprehensively. OK. And then, just as an example of this, I was doing some pre-research for this lecture earlier, and I found this very fun example back in 2023 about the trade secretness of post-training data, where I guess, scale AI's internal documents got leaked at some point.
[05:12.430] And they had a whole bunch of very frank conversations about what was happening in post-training data land. And one of them was like, well, they were trying to make Google Bard better, but they couldn't figure out why GPT-4 was so much better. And so they said, well, we need to reverse engineer what's happening, and we need to get our annotators to produce responses that are more detailed and better than GPT-4. So these kinds of competitive dynamics are happening all the time. And that's also why some of this detail is going to be missing from today's lecture. That said, algorithm wise, we do have a pretty good handle on what is happening. But algorithms are not really the secret sauce. It's not where a lot of the leverage leverages. It's going to be the data. So if we look at the RLHF papers, we see a very straightforward recipe, two-part recipe. The first part is you go and collect some demonstration data.
[06:04.480] So for every prompt you want some annotator to provide supervised fine tuning data, so reference, responses for that prompt. And you're going to do some fine tuning. Then after you have that, you're going to do some reinforcement learning of some kind to try to shape the behavior of the model to be more aligned or more in line with what humans think are good responses. So I'm going to start by talking about this very first phase, which is the SFT part of post-training. And this is the part that's really basically all about data. We all how to SFT a model. That's basically exactly the same as pre-training. So the only real difference with some minor variation is going to be the training data. So I will focus on the training data almost entirely for this part of the lecture. OK. With a little bit of discussion about the method. So we will go through the data. We are going to look at some examples. I'll talk about some historical developments and what the trends are, at least in open source post-training data land.
[07:10.560] And then I will talk about some pitfalls, some ways in which collecting this kind of data is much harder than you think. And if you've taken classes like 224N, you're going to find some things that are very reminiscent of issues that you see when you start building classic supervised deep learning models from scratch. So let me take a moment to pause, and then just talk about the progression of SFT data over the years. I think I'm going to go top left to bottom right. That's roughly my chronological ordering. And there's been a lot of different tricks and a lot of things that people have done in the open source language modeling land to try to both improve system performance and to try to understand what factors of this kind of data are useful to train models. I think, really, the oldest and in some ways the most visionary of this is things like FLAN. This was used to train the T5 model by Google. There a rough idea for this was to say, well, NLP people have already collected a bunch of supervised data sets of inputs and outputs.
[08:18.780] We should just train the model on all of these. Reasonable thing to try. Turned out, for various reasons, this was not the right thing to do. I'll talk about that later. Self-instruct, and other follow ups to that, this was also very forward-looking in terms of its thinking was saying, well, why can't we use the model itself to generate data? Well, models are getting better all the time. And in fact, they might even be better than some of our annotators. Why don't we get models to write very good, high quality responses to our inputs? And that's the self-construct idea. And then there came a bunch of distillation style approaches. Alpaca is one of them. The Berkeley folks had one called Vicuna in which they were using online user shared prompts as the way to-- as the inputs to distill on. And then there were other efforts that were more entirely human-driven, like Open Assistant. You can think of this kind of like Wikipedia.
[09:15.972] It was a bunch of people getting together to write prompts and responses. And then you have a new generation WizardLM and Tulu3 and others of saying, well, that language models are very good synthetic data generators. Why don't we come up with increasingly complicated ways of generating instruction following data using language models? And then finally, I'll touch briefly on this. SFT and post-training has changed yet again in recent years because I think the focus is less on chat, and much more on agents and tool use. And so there is now a new generation of SFT pipelines that are designed to generate tool use and agentic data. So this is roughly a bunch of different developments in post-training data land, at least in the open source world. So in the closed source world, where-- well, some of the companies may be distilling each other, but where you're also collecting human data.
[10:09.260] How you're collecting the human data is a big important ingredient. That's not reflected in any of this discussion. OK. I can pause here in case anyone has questions before I move on to what we're going to do next is look at some of the details of these data sets. Good. Oh, yeah. So for post-training, we need the input-output pairs, right? That's right. So input-output pairs. Does the correctness of those input-output matters? How far does it-- Yeah. So the question was how much does the correctness of the input-output pairs matter? That's a nuanced question. I mean, I think the top level answer is you should go collect the highest quality responses you can because the bad stuff will teach the model to do bad stuff. On the other hand, it turns out that you can SFT models on all sorts of strange things, and they will learn to follow instructions.
[10:58.720] I think Percy's former student had a paper in which you're like training models without even the responses or something. And you can train models to be instruction following. And so there's a lot of pre-training generalization behaviors that let you get away with worse quality data. So it's actually a pretty nuanced question, what you need in instruction following data. OK. Good. So I want to maybe start with FLAN. I think FLAN is very much like a forward-looking data set for its time. It's established this idea of doing this multitask post-training. If you're going to do downstream tasks, go collect all the downstream tasks and train on it. Very reasonable, but smart idea at the time. But if we look at what's in FLAN, we actually find that it's a little bit of a strange data set. You have this thing that's like, well, here's an email.
[11:56.592] And then there's this thing that gets appended, which is write a subject line for this email, and then the subject line goes on the right. This is a very strange thing. I don't think you've ever probably prompted a model to do a task like this, let alone the instructions at the very end. And the reason why you get something like this is because FLAN is generated from existing data sets. So there's a data set called Enron emails. This is basically a way to turn Enron emails into a supervised task because it has a body, and a subject, you can make this subject prediction task. Similarly, you have a summarization task. This is-- I don't know if this was taken from CNN, Daily Mail or Exxon, but it was taken from one of the big, major summarization data sets. Highlights for this article, take the article part of a summarization data set, put the summary on the right.
[12:45.820] But you immediately notice, compared to something like ChatGPT, the summaries are very short. In fact, if you look really closely, the summaries are often hallucinated. There's a bunch of details that are not in the inputs. And in some ways, related to this question, the original NLP data sets, from which font was built, was not the highest of quality. And so when you train on this thing, you also inherit a whole bunch of deficiencies from those data sets. And the unnaturalness of the structure of these data sets also carry down all the way. And so this is a very smart idea. FLAN is a very smart idea to create this big data set out of these data set-- existing benchmarks. But really, the unnaturalness really gets you. And I'll mention one more thing, going back one slide. I think when font was originally constructed, part of the theory of how post-training should work was that you needed scale. In pre-training, we know we need a lot of scale.
[13:44.270] So you might also think that in post-training, we need a ton of scale to unlock capabilities. And so because of that, they took tons of data sets and lots of data, and had this gigantic data set. We'll see later on that a lot of these data sets just get really much smaller, and it still works. And so really, part of the change from font to later data sets is this observation that if you have a sufficiently strong, big, pre-trained model, you can actually get away with very few high quality examples because of pre-training generalization is going to get you quite a bit of the way. And so it's the wrong point in some ways on the quality, quantity trade off. Although, there was no way to until we explored the full space of these data sets. And then I think, after that, maybe next point in how instruction tuning data sets are understanding of it changed was after ChatGPT came out, some of my students did this thing called Alpaca, where
[14:41.570] they distilled ChatGPT traces to get input-output pairs. So this is an example from that data set. You get more natural-looking inputs, longer chatty outputs. Because of course, it's taken from ChatGPT. And what we found was that these kinds of examples, now, reliably induced ChatGPT-like behavior on models. Importantly, only when we did it to the original Llama models, so pre-training, post-training, both of them need to work out. But we did find that this was the point at which these things started to work. And so I think people realized at this point that if you got this kind of chat-style data, it was actually not that hard to build ChatGPT-style systems. Getting the details right is still hard, but to get most of the way. So then I think there was this big push to create, essentially, the highest quality human-driven version of these kinds of data sets. To give you a little bit more context, at the time, I think after ChatGPT came out and Alpaca came out, I think there was this enormous optimism
[15:50.250] that if only we could collect a sufficiently high quality and large, instruction-tuning data set, we could then catch up and match a lot of the performance of these big closed source labs. So there's a lot of energy and excitement to building these data sets. And so in that context, Open Assistant is this big crowdsourced effort to build such a data set. So a lot of volunteers got together and they said, well, we're going to come up with really hard and interesting prompts. We're going to come up with good, high quality responses, and we're going to do it at scale. In the same way that Wikipedia managed to produce something very high quality at scale. A very admirable, very impressive effort that generated actually a decent amount of data. I forget if it's like 10,000 or more examples before stalling out as a project.
[16:36.190] I can maybe talk about that if someone's interested. But if you look now, we have chat-style inputs, very long, detailed, high quality expert responses on the right. Good stuff. I'll talk about some of the pitfalls later as well. OK. Finally, I've taken this from some of the newest of the instruction-tuning data sets. We've really started to move as a field, or in terms of the products that people want, from just a chat interface to something that's like a full agent system. So we don't want just textual responses. We want tool calls. We want to-do lists. If you've ever used Cloud Code or Codex, you know that it produces a to-do list to check off as it goes.
[17:16.010] We want all of those things. And so SFT data now, if you look at, for example Nemotron, so that's NVIDIA's open source efforts, their SFT data, a big chunk of it is these kinds of agentic SFT examples, where not only do you have the responses, which are provided in the assistant field here, but also tool calls that can happen in parallel alongside the text. And this is SFT data. So this is explicitly supervised into the model. And so now, we're moving much more towards these structured formats. And I've been talking about this as I've gone through. But we've seen big, higher level transitions in how we think about and build these kinds of data sets. It's useful to roughly internalize these perspectives. One of them is this chattiness thing. It's hard for me to fully articulate what this is, but there is a sense in which the classic NLP data sets, even the high quality ones, were very input to programmatic output almost.
[18:18.690] But people don't really want to talk to NLP benchmark, they want to talk to people. And so there was this big shift that has happened towards more detailed human-like responses. There's a big shift towards higher quality annotators, and much more detailed. Open Assistant is a good example of this, of having experts write your responses. And then finally, tool use and figuring out the right interface and API for all of this has been a big final shift that's been happening. I'm sure things will keep changing, but these are some high level factors to think about in terms of how SFT data has changed over the last few years. OK. I'll talk about all sorts of pitfalls or challenges in a moment. Any questions on the specifics of data sets and things like this? Good. OK. So now, I think, let's say, you are put in charge of collecting human annotation data for SFT. What kinds of things should you be mindful of?
[19:19.910] There's lots of formatting things that you might want to think about. There's also things like knowledge, how much knowledge should be put into the responses? How much data points do you want? And how do you deal with safety of your model? What kinds of safety data should you be collecting? And I think one thing I want to highlight is length is-- and style variation, in general, is a very, very big part of this post training process. People say, Claude has a different tone than ChatGPT. ChatGPT is too chatty, so on and so forth. All of these are conscious decisions that are made by the data collection folks. And also, we see very wide variation if you look at the data sets in terms of the length of the responses. And one thing I do want to highlight is when you go and evaluate preferences, these stylistic factors matter a ton. So people will get very easily tricked.
[20:19.490] I don't know if tricked is the right word. They will very often select responses that have bullet pointed lists in their outputs, or they will select responses that have more and longer detail. I think part of this is very natural, especially if you're in an evaluation setting, where you put in front of two examples and someone asks you which one is better, I think more detailed and list like responses are generally good. But this does induce very explicit distortions in the kinds of tone that a chatbot has relative to people. These factors-- oh, yeah. So [INAUDIBLE] complex matter in the data set? I'll talk about that later. And you can ask me questions if I don't fully answer your question then. But I think the thing about style, and the thing to be careful about, is if you're looking at engagement signals, as most of these companies are, it's very, very easy to fool yourself into thinking you're getting better data, when in reality, your models capabilities are not changing.
[21:23.990] So if you look at all of these different kinds of post-training data sets that you can train on, you'll see big variations in preferences when you train on some of these things. So that's the right column over there, AlpacaEval, for example. But really, it doesn't necessarily change standard benchmarking evaluations. So your models aren't necessarily smarter because you've trained on certain kinds of post-training data, at least in these ones. But you can very much shift the engagement signals. And so this is telling you, you want to think about style control separately from capabilities control. OK. The other thing, and this, I think, is a quite, in many ways, important thing to think about in terms of how we collect and think about post-training data. I think Open Assistant is very instructive because it was an effort that was intended to be the highest quality in some ways, post-training data.
[22:17.890] And so you often see things like this. Like for example, this reference-- sorry, this response contains a reference to an article. This J Michel, m the pay of corporate executives, and blah, blah, blah. And so this is actually teaching the model two different things at once. Let's say, you're SFT-ing on this data. So first of all, you're doing next token prediction. So it's teaching the model about this Bivens J & Michelle L citation. It's also telling the model another thing, which is when you're giving a response, a good thing to do is to give a reference. So these are two different mechanisms that are going on at once. You're teaching the model about different pieces of knowledge, which is good because you want the model to be knowledgeable, but you're also teaching the model to emit detailed knowledge. And this is sometimes problematic because what this is going to force the model to do is sometimes it will misgeneralize,
[23:14.670] and it will hallucinate out a reference instead of actually giving a proper reference. Models don't really know necessarily whether a reference is true or false. And there's been a lot of interesting arguments that people have made around this. And I think the Folklore, which I think is not wrong. There's some good empirical evidence to support this Folklore. Is that if you train a model, especially at the SFT phase, to emit facts that it does not, this will make it hallucinate. Because the model, as I said, is trying to generalize two behaviors at once. It's trying to generalize the knowledge but also the format. And so teaching it something where you have the format in a piece of unknown knowledge is kind of teaching the model to forcibly emit unknown knowledge. And people have shown that if you train on, for example, information that you don't really know, then this leads to hallucination and overfitting to pieces of knowledge.
[24:14.232] Whereas if you're training on only known facts, you don't necessarily see this. John Schulman makes a separate argument, which is that this is the reason why you need reinforcement learning and training. I mean, this is unsurprising because if you know John Schulman, he's like a reinforcement learning guy. He made PPO. Of course, he is very pro reinforcement learning. But the argument is sound. That, in some ways, in order to teach the model what it knows and doesn't do have to be policy-dependent. You can't have an external person shoving knowledge down your throat if you want the model to be calibrated about what it knows and doesn't. OK. And so to put that piece together, you might not want to train on the highest quality data if the model doesn't already that data. Tail knowledge can actually sometimes be actively harmful in that it might induce hallucinations, especially if it's associated with markers like reference colon.
[25:07.950] They'll force the model to emit a reference afterwards. I'll talk about RLHF later, but this is one of the reasons why reinforcement learning is important and useful. For things that are model-dependent, knowing what you know, RL is a very useful way to teach models to not say things that it does not know. And so knowledge storage is actually very messy and a very nuanced concept. You don't really want to just say yes, just pack knowledge in there. You want to be very careful about exactly what supervision you're throwing in at the post-training stage. This is one of the reasons why I think post-training is quite messy. You have to deal with all of these realities of serving a system eventually to a user. Yes. What's the definition of tail knowledge? Yeah. I don't think there's a formal definition of knowledge. For example, in the past when I've worked on things like this, it's been things like the number of times that-- sorry, the length of a Wikipedia article, for example,
[26:03.070] might be a proxy for its well-knownness. And then you might say, yes, if we train on stuff that's not well known, we do see that the model hallucinates more. But there's no formal like, yes, this is tail and this is not knowledge, because knowledge itself can't really be pinned down, precisely. OK. Yeah. [INAUDIBLE] The question was why would RL help for hallucinations? And the reason why RL might help, let me give a folk story for why this might help. Imagine, the model has internal to it. I know something direction inside it's activations. SFT time, maybe what's happening is you forced it to generate references from your SFT data. And so it's just going to generate references no matter what. But when you RL, you might notice that you get good rewards when you generate references in the I know stuff direction. And you get bad rewards when you generate references in the I don't know direction.
[27:03.550] And so then you might then learn to generalize that I know things or don't things activations into whether you emit a reference. So it's possible that if you know what somewhere inside the model, RL can help you extract that into your output policy. If you don't know what you know at all, at any level, RL cannot help you there. It can't help you with the problem of having some level of calibration. Yeah. So just to follow up on that. But if it emitted a bad reference, where do you get penalized in the loss function? In SFT, well, in some ways, implicitly, you're getting penalized for anything that is not a reference, like assigning any probability mass to something that is not this exact sequence is penalized. So in some ways, SFT isn't doing the wrong thing. This is a correct sequence. The problem is SFT is the generalization properties of this. That I can't perfectly generalize off this one example. There's two things, two disentangled things that are being taught here, one is the reference template,
[28:03.270] and then the other thing is the actual content of the knowledge. And if I generalize based on the template, I'm in big trouble. Great. Great questions. Yeah. OK. The other thing that I want to talk about is safety. Once we get the post-training, we have to really start engaging with all of the messy realities of how are people going to use this. I think pre-training people live in this ivory tower, like let's just compress the world. But post-training people have to say, yes. But what if people are using our system for political manipulation and disinformation? And post-training people might say, yes, we see individualized spearfishing. We need to prevent this from happening. You are the last line of defense, in some ways, between you and misuse of the systems. And so what do you do? Well, you have to apply safety controls to the system. That will refuse to respond to malicious inputs. This is the de facto way by which these companies enforce the safety controls.
[29:03.830] And so it falls upon the post-training team. Unfortunately, safety SFT information seems even more sparse than capabilities SFT information in publicly available documents. I pulled out the Llama 2 description of how they do safety SFT. If anything, this is one of the more detailed descriptions. But you'll see that they don't even talk about how many examples. They used for safety tuning, which is kind of crazy to me. But regardless, really, the framework by which all the safety tuning approaches operate is to balance two things. You want to balance how often am I allowing bad queries to get through, which is, I think the violation rate here. And then you also want to not over refuse because someone might ask a question like, how do I kill a Python process. And then if you're a bot, it's like, no, I can't let you kill anything, that's very frustrating to the user. And so the false refusal rate also needs to be low.
[29:58.470] So want to control this and have a good Pareto trade between the two. And so what you do is you basically create various kinds of tailored data to navigate this trade off. In terms of the scale of the amount of data you collect, you usually have something like a few thousand or tens of thousands of examples. For example, in Llama 2, I think it's about a few thousand examples that are used for this process. This state of affairs is a little bit unsatisfying if I left you with only that slide. So I will talk about a little bit more detail. I think if you want to look at how post-training is done, maybe the only good reference right now would be two or three. This would be the post-training pipeline for the OLMo models that Allen AI put out. And given the lack of details about post-training in many places, this is one of the few places where you can see a reasonably performant post-training pipeline in action.
[30:57.370] And they do have, for example, a safety component, which consists of something like 50,000-ish examples total. And if you look, they all follow the same fairly simple strategy for generating SFT data, which is the following. In a previous project, they had this thing called WildChat, where basically they gave free API or free chat access to a bunch of people so that they can go out and use various kinds of chat interactions. In exchange, they collected what they were doing on those chats. And from that, they were able to basically filter out a whole bunch unsafe behaviors that were being attempted to be done on these chat interfaces, as well as jailbreaks that people attempted to do. And so they mined these at scale, and then created essentially the preferred response, which is to resist the jailbreak in one case, and to say no in another case. Fairly, very straightforward approach. And I think from what we see, at least in the technical reports and the model cards of the various closed source companies, similar things are happening, and that you
[32:03.010] look at your usage information, you find unsafe behaviors. You get your annotators to play whack-a-mole with these bad behaviors. OK. The final thing I'll mention, and this is related to what I said about the number of examples that you need for SFT. I think one thing, if you haven't played with SFT before that you will be surprised by, is if you have a sufficiently capable model, it does not take very many examples to steer these systems. So if you want to do something safety tuning, if you take 500 examples, let's say, in the vein of you find a bunch of unsafe stuff, or you synthesize a bunch of unsafe stuff, and then you write refusals and you put that into the system, you'll find that even with as little as 500 examples, the rate at which you follow malicious instructions or other kinds of hate speech instructions, and so on, drops dramatically. And with 500 examples, you're really applying this fairly surgical-- not surgical, very small push to the model. But that results in a fairly nice across the board reduction in these safety issues over here.
[33:07.790] And so one way to think about it is models already have am I going to be a safe model or an unsafe model access inside of it after pre-training so it does not take very many examples to pull this out. Now, one thing to also remember at the end of this is just because you can steer a model with very few examples, doesn't mean that there's no benefit to more examples. If you're OpenAI or Anthropic, and you want to enforce really fine grained distinctions about what is safe and not safe, you do need very large scale data collection does. This doesn't really change that story. OK. So SFT at its best is this thing where if you're extracting pre-training behaviors, like it's already in there somewhere in pre-training, and all you want to do is pull out the right modes out of the model, then instruction fine tuning works very, very well. With very little data, as long as you have the right high quality ones, you can get the model to do what you want. And adding data or factually correct ones can sometimes hurt you, at least in terms of hallucination.
[34:11.429] And then finally, it doesn't take very much data. And so you can often focus on quality instead of quantity. Good. OK. This is going to be the end of the data part for SFT. So if anyone has questions, ask me before I move on. Yes. How do we know what is already a behavior [INAUDIBLE]? Yeah. The question was how do we whether something is in pre-training? I think this is the sloppiness of what I was saying, which is that we don't really know per se. We can know what things aren't in pre-training. We can look for the kinds of generalization to, for example, programming languages that are very rare that's very hard using SFT. So we can show those things. We can't show necessarily that, for example, safety is something that's in pre-training per se. Yes. Can you talk more about this when you talk about RL. But there's this whole question of SFT destroying features with RL invading them.
[35:06.570] That's like [INAUDIBLE] trying to talk about that. I'm not going to talk about that later. The question was whether SFT is destroying features in RLs like pushing up or down. I think the lines are very blurry between the two, to be honest, especially, when we get into something like expert iteration and all these other things, that look like SFT with extra bells and whistles. So I don't necessarily think it's a core distinction between RL and SFT, rather that it's more of a distinction in the kinds of feedback that we get. SFT is this dense feature supervision, whereas RL is the self-taught policy supervision. And the other thing about RL is that you're getting your own output from your own policy. So there's a sense in which you don't maybe deviate quite as far in what you're being reinforced on. And that's probably an important part of what's happening. OK. So now, let's talk about methods. For most of this class, we've been excited to talk about methods.
[35:59.350] In this part, it is about the most boring thing because you just do gradient descent. I just put the picture here as almost a joke. You just take loss.backwards and that's basically it. But there is one nuance I do want to talk about that is important, which is this increasing trend of turning instruction tuning into part of pre-training. It used to be that when these things first appeared, the pre-training and post-training were just these separate ideas, that you would pre-train on web data, and then you would post-train afterwards. But people realized why separate two things when you can mix them together. And so lots of high quality data, and even instruction tuning data, is now getting mixed in at the tail end of training when the decay phase usually of training happens. And this really allows you to scale up instruction tuning. It also allows you to essentially emphasize higher quality data.
[36:54.450] The impact of this has been quite dramatic. And so everyone, as far as I know, is doing it. You can see it in most of the model release reports, that there's a mint training or a second phase pre-training with a different data mix. And importantly, one thing that I'll-- this is my pet peeve. I'll say is that now, when someone tells you something is a base model, that's kind of a lie. Because a base model is usually about, yes, predicting the next word on internet data or something. Well, really, base models today are pre-trained on ultra chat, and who knows what else. Those are like chat data sets that are synthetically designed to make you good at chat. And so it's very hard to say that this is like a base model in the traditional sense of the word. I've taken this example of a two-phase training. I think this one's from MiniCPM.
[37:45.610] Because I think they show very nicely the change in the mix. So this first bar is the very standard pre-training phase, where you've got all sorts of internet data on the left. And on the right, you're kind of switching to this higher quality, and also very chatty data set. You've got Stack Exchange QA. You've got Ultra Chat. You've got all sorts of other SFT data. And then you've decreased the fraction of standard general purpose pre-training data. So almost everyone does this today. And you can think of the point of this as saying, well, the boundaries between pre-training and instruction tuning, and also high quality data is just being blurred very much. Yes. Are the queries masked in this kind of data? No, this is pure pre-training. And so you're also predicting the prompt as well. But actually, that's not a huge difference because some SFT recipes do also predict the prompt. Yes. [INAUDIBLE] OK.
[38:53.590] So the question was whether the data quality is lower in the decay. Usually, the intuition is you want the reverse, that the decay is the most important part of your training. It's the part that's closest to deployment during pre-training. Also, it's the part where the lowest learning rate. And so for both of those reasons, maybe you want to put the highest quality stuff into the decay. And so I guess, Book Chinese, and all these things are supposed to be higher quality. Certainly, Wiki and Stack Exchange is usually considered high quality. Yes. How is the mixture-- I guess, for pre-training, you discussed other mixtures beside these [INAUDIBLE] cheaper models. And then from there, you can use statistical analysis to decide. But then, I guess, for midtraining and post-training, you have to essentially run it on the larger model, or do people still have some smaller model they use as a proxy?
[39:38.990] I assume there's some reduction in the amount of information you can build. I guess, you mentioned what's good there about these data mixtures and where do they get them. Yeah. So data mixtures, I mean, both for pre-training and post-training are very trial and error. We do have algorithms like many people have written papers about things in this space. But I think they're fairly unreliable. There's a lot of trial and error and intuition that happens for this space. But the nice thing about this two-phase training and this midtraining stuff is that midtraining is much shorter than full pre-training. So if you're doing data mix optimization for midtraining, you can run something like 10 of these for each one of these on the left right. And so this lets you not quite brute force the problem, but it does let you try a lot of ablations. And so actually, I think the usual way that I've heard a lot of this done is actually in the decay phase, you do a lot of data ablations, which
[40:33.330] are cheap to run and so on, and then you get estimates of data quality and then you reflect that even back to the first stage-- Into the pre-training. Into the pre-training stage. Yeah. So to decide the pre-training mix, often, you actually run a bunch of ablations on decay. And then you will figure out what to put back or how to set the pre-training mix. The reason why you don't just make pre-training high quality is you just don't have enough tokens. You'll run out of tokens if you try to make Wikipedia your whole pretrained. Is there any intuition based on the information you get from, I guess, how good the [INAUDIBLE] performance was with your prior mixture. Is there any intuition with how those are-- how it's adjusted, I guess? That's very case by case of how you go from downstream deltas on ablations to mixes.
[41:20.125] There are systematic ways you can fit models and do these things, but they're often brittle. What I've heard from people is that actually, it's much more trial and error than you think. You ablate a domain, you make rankings over which ablations had the most impact, and then you decide what to put in. Actually, a good example of this, a fun example of this, that's been leaked is when Meta got sued for using Books. In the court documents, there's actually a bunch of documents showing some researchers doing ablations and trying to estimate, how useful are each of these Book subsets. And so that's exactly the kinds of things that happen. And then ultimately, they'll decide based on that what the mixes are. Cool. OK. So now, we're done with SFT. The algorithm part was very short. And now we get to talk about RLHF. So this is the second phase.
[42:08.600] We're done with doing supervised fine tuning. Now, we're going to do something very different from pre-training, where we're just going to upweight or downweight different model outputs based on how a rater, or in some cases, a reward model decides how good these responses are. And before we get into algorithms and stuff, I do want to pause for one moment on this slide and talk about a conceptual difference that I think is actually quite important. Maybe it's less important for people that weren't originally coming from StatML background and so on. But I think there is actually a very different way of thinking about SFT or even pre-training and RLHF. So in pre-training, what are we doing? We're doing generative modeling of some sequence. So you have a collection of sequences and you're trying to predict the next word. SFT is also the same. You've changed the distribution, but you're predicting the next word. So you can think about this using your generative modeling brain. This is purely a generative modeling problem. And then you're just going to fit some distribution.
[43:06.680] Now once we get to the second part, the RLHF, we are no longer playing a fit a distribution game. We are fitting a maximize a reward game. So what we want is we want to find some distribution. We still have a distribution. This is a policy now. Such that this policy is going to maximize some downstream reward. And maybe that reward is something how long your users are engaged on your platform. Maybe that reward is like what math problems you can solve, but there's some of reward. And I don't care if I've mimicked some distribution. And this is conceptually very different because now, in the second world, I can totally collapse out my distribution onto a single point for every input. So for every prompt, my model could have a single answer, not a distribution. And that would be OK, as long as it got a good reward. So that's a big important conceptual distinction to keep in your mind. And I wanted to highlight it before we get into some of the later bits.
[44:00.880] OK. So now, why do we want to do this thing of doing reinforcement learning? We could just say, why don't you forever go and collect SFT data? Well, I think one thing that is important for thinking about RL is when we try to optimize what humans want, there is sometimes a big difference in what they say they want and what they generate. And that's one thing that we found in the past study. I put this here because I find this so funny, even though it's a little bit old now. We asked a bunch of freelance writers to summarize news documents. And we found that a few annotators actually preferred Instruct Davinci, which was the predecessor to ChatGPT, over their own writing.
[44:48.700] And that was very strange. And we were like, are these guys actually doing annotation? So we interviewed them, and we asked them like, why did you like the AI outputs over your own writing? And they said, well, I looked at it and I thought, actually, this is pretty good stuff. And so it turned out that, they're writers, they're good, they're good at writing. Their summaries are high quality. We check it for quality. But actually, when they looked at it, they were like, I didn't realize that's actually a better way of doing it. So people aren't really these optimal systems. And so when they judge things, it's actually different. And so because of this gap, sometimes you might want to rate outputs rather than just to generate demonstrations. Another reason is that in some domains, verification is a lot easier than generation.
[45:31.460] Math, which I'll talk about more next lecture, is a prime example of this, where verifying a proof is probably much easier than generating the proof. And folks like DeepSeek have gone down that path for self-verification using models. That's another place, where this idea of having reinforcement learning and judging yourself is a good idea. OK. I'm not going to cover what people call RLVR today. That's its own universe. It's going to be an entire lecture next time. I'll talk about the exciting promises of reasoning models. But today, I'm going to talk about RLHF. In RLHF, we'll have different components. And I'm going to talk about data, how do we collect it. And then I'm going to talk about algorithms, how do we actually do it. And I'll talk about two major families of algorithms before talking about maybe some pitfalls at the very end. OK. Good. So in RLHF, the high level way that we do things is now we have a good model after SFT.
[46:29.280] It follows some instructions. We now put prompts into the model. The model is going to generate a couple of different outputs. And we usually do this by just sampling with temperature one. At the end of SFT, the model is pretty diverse, so this is a reasonable thing to do. A rater comes in and rates all of the examples and provides a ranking over the examples. Sometimes, even a binary ranking. And then a model is going to be trained on those rankings. And we're going to use standard reinforcement learning techniques to maximize the score that we get on that reward model. And we go through the reward model because it might be easier to train a verifier than to train a model that does well directly. OK. So how do we get data for this thing? As I said before, the general way in which we do this is to get pairwise ratings. And so you're going to have an interface like this, where you have a couple AI responses and you're
[47:19.500] going to give a response saying response one or two is better. This is just a standard annotation interface that I've gone and found, but most places have similar things. We can now look at what an actual company would do in order to do this. It's not that different, although they have some nuances. If you want to go and really read up on this, I would suggest you look at the InstructGPT appendix. It's the last point at which we have a glimpse into data collection processes from industry. And so at least in the InstructGPT case, they say, OK, we want you to rate these outputs for being helpful, truthful, and harmless. So they have to balance three things. Helpfulness is how clear the writing is, or being sensitive to internationality, or not giving overly long answers. You want to be truthful, so don't hallucinate. And then harmless, if it's unsafe, we want you to say that-- or we want you to upweight things that are refusing to respond to questionable prompts.
[48:18.200] So this makes sense as an omnibus objective for training chatbots. Another very similar example that I found is, this is, I think, the only other example of a closed source lab annotation being out in the public. This one is for Google Bard. So that was quite a while ago, but their annotations got leaked by some annotators. And you can see they have a very similar structure of saying, we want helpfulness, we want the presentation of the response to be good. And actually, their rating in a Likert scale rather than pairwise feedback, but they have a very similar structure. It's not a very super long set of instructions, but you get the sense what kinds of things are included, like don't contain inaccurate information or be coherent and be easy to consume, so short for the user to read. Things have changed quite a bit since then. There are a couple of different surveys of annotators. I think the high level thing that I'll say before going through the next two slides is, in general, I think the worker distribution for annotation has shifted
[49:26.940] upwards, more towards experts, more towards higher cost. And we see that in compensation and things like this. So if you look at the education level of data annotators, this is for one of ScaleAI's platforms. So it's not representative of the full space of data annotators for language models. But this is probably a reasonable subset of those. The majority, or a large fraction of them, are bachelor's degrees or master's holders. So that's like 70% some of these annotators. And the age is something like 35 is the modal age. And the task that they're doing are often like creative writing and technical writing and so on, are often the tasks that these annotators end up performing. This is one step towards experts, but I think even more, there's been a growth in the last year or two in bespoke annotators. And I think this is because a lot of the labs are now pushing into deploying these systems in actual white collar jobs.
[50:28.120] And so you want doctors, you want lawyers, you want these people to annotate the responses. You want them to give SFT data. And so there are these projects that are internal to these companies like OpenAI, where annotators are being paid a pretty decent wage per hour to do these annotations. And so if you look at the median wages, they're above $50 across a range of different topics. But if you look at the experts, some of these experts are being paid more than $100 per hour to do annotation work. And so I think this is to say, if your mental model of an annotator was low cost pairwise feedback from somewhere overseas, that's not really the full picture. There's a lot of annotation happening right now that's very bespoke, very expensive, but it's kind of also a pyramid. It's not like that lower cost scalable annotation has gone away. OK. And to emphasize this point to you, getting this kind of data is actually very challenging.
[51:30.520] That's why the pay is high. That's why there's so many data collection startups. I think the hard challenges now is getting verifiable annotators, especially ones where you're sure that they're not using AI. If you all have done survey research, or crowdsourcing recently, I'm sure you found that people are using ChatGPT as part of their survey responses or as part of their annotation workloads. Basically, preventing people from doing this is extremely, extremely difficult. It's also very difficult to get truly correct responses when people are under time pressure. For example, that Google Bard example that was part of, I think, some labor dispute where the annotators for Google Bard were saying, we have to check the correctness of these long chat responses in under a minute. It's impossible for us to do that. There's no way that we can actually follow these instructions. And to add, there's a growing amount of expert data collection, but there's also a very large amount, still, of low paid annotation groups.
[52:36.380] ScaleAI, I think, did quite a bit of this initially. I don't know if they still do, where they outsourced a lot of this low cost annotation and got into quite a bit of trouble. And so in the same way that I guess the modern economy has this bifurcation in terms of high and low wage jobs, we see the same thing with annotations as well. The final thing I want to talk about in terms of annotators is demographics. One thing that's quite important is that annotators have a surprising amount of influence over what the model does. So post-training comes at the end. And it's, in some sense, the final shaping step of the model before it gets shipped out. So if you're not really precise with how the annotators steer the model, and your annotation groups have biases, you can actually end up getting some pretty surprising behavior. So one of the studies that we did in the early days of instruction tuning with one of my postdocs, between Percy and I, was that we looked at essentially which kinds of ideological opinions
[53:41.700] language models were more aligned to. And we did this by asking standard opinion, poll questions to language models, and seeing which human demographics they were closest to. And so what you find-- or what we found is that base models of various kinds were kind of similar to Protestants or Roman Catholic opinions, and far from Buddhist or Hindu ones. And then what we found is that if you post-train these models, so the three right columns, you find that they're more different from Protestants and Roman Catholics, and becomes more similar to Buddhist, Hindus, and atheists in terms of the kinds of opinions that they give. And we're like, that's really strange. Why these three groups? It turns out, if you look at the appendix of the InstructGPT paper, you find the annotators, who were annotating for these models, and they line up, like a lot of Southeast Asians, and also people in the West Coast of America.
[54:37.560] And that's roughly the demographic groups corresponding to these. You also find, I guess, another added note in the recent years, there's been a lot of findings that very subtle biases can transmit through data. So one example of this is something called emergent misalignment. If you have a piece of data trained from a model or generated from a model, that's been trained to say like, I like owls or something, and you train on that innocuous looking data, the model trained on it will actually inherit a preference for owls. And so there's all these weird kinds of subliminal transfer effects that can happen with data that are quite hard to catch. And so that's one thing to be really, really quite careful about. And this is true, more generally. I think this whole study of LLM political opinions and so on can be quite fragile and sketchy at times. But the annotator distribution and who your annotators are also matters quite a bit for the different kinds more material errors that your model might make.
[55:37.560] So this is a study that's quite nice by Hosking, et al, where they studied like if I get really expert annotators, like people who really care about doing the job right, versus random collection of crowdworkers, how do their annotations differ? And they show very nice, striking results that people who are just non-expert annotators, that are just kind of paid on a platform really like to emphasize formatting effects, which is this row over here. You see, it's red. It's over enriched in these non-expert annotators. On the other hand, things like factuality or inconsistency are things that are much more emphasized than the expert annotators. Of course, it's much, much harder to test for factuality, and so it's no surprise that if you don't have experts, you don't actually end up checking for these. This is another way in which your distribution of annotators ends up mattering, not demographic, but here, how much they care and what their expertise is.
[56:33.580] It's been shown for a while now that because of all these-- oh, was there a question? Yeah. I was going to ask how do you measure quality and [INAUDIBLE]? What are the [INAUDIBLE] to be higher quality than another? Yeah. So the question was, what does it mean for one annotator to be higher quality than another? I think there's no gold standard rule where you can mechanically turn the crank and be are a good annotator. I think, I'll maybe give two different kinds of answers. One of them is a sufficiently-detailed annotation guideline should hopefully be semi objective. You will say, it should be factual. And then you'll define what factuality is. And then you might even have objective criteria. Like in the first three pages of the Google search, you don't find anything that contradicts it. And if you can find something that doesn't follow the guideline, you can be like, yes, you're not following the rule.
[57:21.960] That's one notion of a good annotator. The other notion of a good annotator is inter-annotator agreement. And that's a statement about a population of annotators, whether they have high variance or not. I think the only issue with that is it doesn't tell you what the bias is. It tells you the variance. And some tasks inherently have variance. If you're asking do you like this, that's a task with lots of variance. So inter-annotator agreement is not useful. If you're asking, is it factual, maybe it should be lower. But even if it's zero, if you've asked the right question. Or if they're all using ChatGPT, the variance will also be zero. So it's very hard to assess quality even though there's ways of maybe getting at that question. So is the reason you're seeing annotators or companies starting to recruit more experts, or how we educated people to be annotators because they give somewhat higher quality or more lower grade on answers?
[58:16.820] Or is it because the data that you're collecting is just something that only these people can answer? I think it's more of the latter. Maybe you need a lawyer to check if a Blue Book annotation is correct. You need to be relatively well trained to be able to do that. And there's also tacit knowledge of various kinds that maybe you want to extract. So that's one of the reasons. It is true that I think, in general, the platforms are moving more towards higher quality annotators, even for general annotation tasks. I think a big part of this is because LLMs have gotten good, and if you don't like in-person supervised people, they will use the cheapest LLM to generate plausible-looking answers. And so I think there's a lot of annotator shops whose value proposition is basically like we have real people that do real things, and we will verify that this is true for you. Cool. OK. Yeah.
[59:09.100] What's the largest problem-- and I hear you might be getting to this. What's the largest problem using a domain-specific model as an annotator? So the question was, what's the problem with using domain-specific models as an annotator? I guess, there's two separate questions there. One is like are domain-specific models good? And then the other question is like should we use models to annotate data? Model-based data annotation is quite good. If you're not at the frontier, and what you want to do is basically catch up to the frontier, model-based annotation is very, very good. Modern models are probably much, much better than random crowd worker you can get. And so model-based annotations, good idea. Now, domain-specific models, well, can you get domain-specific models that are better than the strongest open models? Sometimes, quite difficult. And so for that reason, I don't think there's a unique advantage, for example,
[01:00:01.080] to annotating with domain-specific models. In general, model-based annotations are a good idea. OK. So I didn't plant that question, but we are moving towards model-based annotations now. If you look at LLM-generated annotations of various kinds, you will find that these models are surprisingly good. When GPT-4 came out, some of my students and I did comparisons between GPT-4's annotations, and carefully curated human annotations. And if you compare the rankings of systems, they're pretty good. If you compare the agreement between humans to models, the mode of humans to models, they're pretty good. And their cost is an order of magnitude less. So that's all nice and good. And at the time, when we initially did this, back in the GPT-4 era, it was still not clear which way things were going to go, like in the open world, for example, where are we going to move towards expensive human annotation, or will everything just become, in some ways, distillation from stronger models.
[01:01:03.980] I think at this point, it's been enough years that we know the answer, which is there's basically no space for human collected data if all you want to do is to catch up to the frontier and capabilities. I think one of the things that I found very interesting at the time and instructive is Huggingface was trying to build an open source model called Zephyr. And one of the things that they really wanted to do at the time was to not do any model distillation. And so they said, well, we're not going to use any model-based annotations or model-based data, and we're going to spend a lot of time and effort collecting human data. So they went to the same vendors that OpenAI and others do to basically go and collect a bunch of human data. But basically, they found that it was extremely time-consuming, costly, and the results they were getting were not actually better than using model-based annotations.
[01:01:55.720] And so in the end, they basically just used model-based feedback for this, which I think is instructive. They gave it a very real good shot using the resources that they had. But in the end, they ended up using AI feedback. And this has basically been the case. So for SFT, and also for RLHF, we find that UltraChat and UltraFeedback, which are model-generated are now very standard. Tulu3, which I mentioned as one of the big, almost flagship, open source post-training processes, uses model-based annotations for all of its pipeline. And so this is basically where we've ended up in terms of should we use model-based annotations or not. Models are very good at following instructions, very scalable. So you can collect the right kinds of data that you want. And this is roughly where things are.
[01:02:49.000] Of course, if you want to push the frontier out, you don't get to necessarily play these games, except in limited circumstances. And so you're still very much reliant on human-driven data collection. OK. Yeah. [INAUDIBLE] example. What is the [INAUDIBLE] model [INAUDIBLE] because 70 is [INAUDIBLE] very small. Well, 70 is small now. I guess, at the time, I guess, 70 was a very respectable open source model. It's like Llama is a 7B. Everyone loves Llama 7B. We too would like to train a 7B. And so in that sense, it was respectable. I don't know if they went and tried it for their bigger runs, but at least, at that scale, they weren't seeing any differences.
[01:03:34.038] I would be very surprised if they switched to the bigger one, and suddenly there was a big difference. Cool. OK. There are other places where you might do model generation that is not purely distillation. Anthropic had an early work on self-training. This was Constitutional AI, where they kind of prompted a model to generate safety data, and then they trained the model itself on that prompted data. And that allowed them to get a safer model out of the model. And you can think of this as a very early self post-training data generation loop. And the other paper-- sorry, the other project that I mentioned to you of a self instruct, that's a different kind of capability centric version of this kind of idea.
[01:04:14.520] So you can use models to bootstrap your own post-training data. But if you want something like lawyers or scientists, you can't get world knowledge without getting people to annotate. So you're kind of stuck, at least, for that, with collecting human annotators. The other thing I'll mention is that models are also very susceptible to the same kinds of biases as humans, sometimes in ways that are quite problematic as well, even more problematic. There's been a couple studies, after this kind of wave of model generated data, stuff started coming out, showing that actually you could just push length of your responses way out. And you would continue to get improvements in the win rates of model-judged performance. And if you look, the one outlier here is ChatGPT 3.5, basically saying, well, they're not just length hacking, they're actually a model that is actually better. Similarly, others showed-- this was a very nice paper showing that you can just RLHF on length alone,
[01:05:19.040] and you could do quite well on many of these benchmarks. So that's roughly the data story. Who's collecting the data? What kinds of things to be conscious of? And then now in the remaining time that I have, I'm going to talk about the algorithm. I'm only going to briefly talk about PPO because both in the interest of time and because I'm just going to do a more extended treatment next lecture. And then I'll talk a little bit about DPO, which is the cool fun bit. OK. So now, remember, we're in the RLHF regime. The goal is the following. I'm going to maximize, under my policy, the reward that I get by sampling from that policy. And this is like baby reinforcement learning. Because people might say we're playing bandits. This is not true multi-turn interesting RL. And that's roughly right. And so our algorithms will also be quite simple. So the objective that I just showed you, maximize reward, it appears almost exactly in the InstructGPT paper.
[01:06:17.600] If you look at-- open the paper up, and you go to equation 2, you'll find exactly the kind of thing I wrote down, where you sample from your RL policy, I want to maximize my reward, subject to this second term, which is really just a KL divergence. All that it's saying is I want to stay close to my pre-trained model because I don't want to go too far and become degenerate. Same thing is true from Stiennon, et al. This is another paper. You see, the reward in this case is a pairwise feedback model that you've learned. So you train a binary classifier on which of a pair of examples is better. And then you hill climb on that reward. So very simple structure that we have. Now, how do we actually do that hill climbing? We use an algorithm called PPO. If you've taken RL class, you, of course, know what PPO is. If you haven't, I will give you a baby description of what PPO is. Also, you'll have to understand it for the assignment.
[01:07:14.150] So the very basic starting point is the policy gradient identity. What I want to do is I want to maximize my reward sampling from my policy. How do I optimize that? Well, everything in machine learning is gradient descent. So I take gradients. So I want the gradient with respect to my parameters. And you can push the gradient in and do the policy gradient trick. And you can show that that's equivalent to the equation on the right. So I'm just going to take the gradient of my log probabilities and weigh it by my rewards. And this really just looks like SFT, but with weighted examples, effectively. Now, you might say, OK, policy gradients are great. I love policy gradients, but the problem with policy gradients is I need to sample every time I want to take an optimization step. And sampling is very expensive. We've already talked about in the systems parts, inference is often very complicated and hard, whereas training is very arithmetically intense and good.
[01:08:05.990] And so I want to roll out once, and I want to reuse that rollout multiple times. So this is something called off-policy. To do off-policy, I want to take multiple steps but not go too far. Because if I go too far, my estimates of my local rewards kind of blow up. So this is something called TRPO. The basic idea is I want to take policy gradients, but I want to stay a little bit close to where I am while you do something called an importance weighting correction. I'm not going to go into that in detail. Finally, PPO says, well, TRPO is a good idea, but this distance constraint is actually kind of hard to deal with. And so I'm going to come up with a heuristic clipping thing that's just going to discourage the RL algorithm from going to places where I'm too far from the original policy. I'll go through the details of this in the next lecture. So you don't need to worry about it.
[01:08:55.010] But the main progression of policy gradient off-policy, you end up at PPO. Hopefully, you roughly understand. PPO, already, the equations look a little bit gnarly. And so a lot of people for many years have asked the following question. Can we get rid of PPO? And this was a big thing for many people. And many reasonable people thought about lots of good ways of getting rid of PPO. And I will tell them to you so that you do not necessarily repeat them in your own research. For example, you can say, I have a pair of examples, and I can do SFT, but with a special kind of SFT. For the pair that's good, I will prepend a good token. And for the example that's bad, I'm going to prepend a bad token. So you're going to append some tokens. And then when I generate, I'm just going to prefix my response with good so my model only generates good stuff.
[01:09:44.170] So this is reducing PPO to SFT or RL to SFT. It does not work. People have tried training the model only on the good stuff. Also does not work very well. You can train the reward model, get LLM outputs, select only the stuff that the reward model selects and train on them. That does not work as well, although it does somewhat work. So many people have tried, many different iterations. And in the end, we do now have a thing that is much less complex than PPO, works pretty well. And is, in many ways, looks just like SFT. And this is DPO. And so the algorithm is very, very simple. What we're going to do is we're going to try to get rid of the complicated parts of PPO. What are we going to get rid of? We're going to get rid of the reward model. We're going to get rid of anything that has to do with on-policy stuff.
[01:10:32.025] And by getting rid of these two things, we're going to try to make a much more simple RLHF algorithm. And it relies on a very simple intuition. Everything in deep learning is just taking gradient steps in the direction of good things. And so we're going to take steps in the direction of the log loss of the good stuff. And then we will take negative gradient steps on the direction of the bad stuff. In other words, we're going to reverse for the log probability of anything that is bad. This is basically the most naive thing you can do. Because you're doing SFT on one hand, and you're doing kind of negative SFT on the bad thing. It turns out that if you wait these two appropriately, you will get an algorithm that is actually pretty good. And I'll give you the derivation because it's all very simple. So remember, our goal is to optimize this equation over here. So this should look familiar to you.
[01:11:23.070] The first term is my expected reward under my policy pi theta. The second term is saying that's my KL distance. I want my policy to remain close to my reference. That's what I pretrain. Now, I'm going to make a assumption. This is the one strong assumption that I'm going to make. And I'm going to assume that pi, my policy, is actually not a neural network at all. It is the set of all possible policies. It is a nonparametric thing. It can approximate anything. So if pi can be anything, I can actually solve that top problem closed form. I actually know what the minimizer is. And the minimizer is the following. The minimizer is actually to take my original reference policy, and to exponentially tilt it by my reward. In other words, every response gets upweighted or downweighted with a weight, x of 1 over beta r based on how good the reward is.
[01:12:17.770] This makes a lot of sense. If the reward is really bad, I'm going to exponentially downweight. If the reward is really good, I'm going to exponentially upweight it. So this is the closed form solution if pi can be anything. Now, we can then-- now that this pi of r, this perfect solution, I can solve for the implied reward. In other words, I can solve for what is the r that would induce this. And so I can solve for that. I can put the pi on the left side, and I can solve for the reward that we would have. And then we can take that implied reward that I wrote down. And then I can solve for what would happen in the original RLHF objective. I plug-in my reward into that objective here, into my loss. And this gives the DPO objective, which is very intuitive. It basically says take gradient steps in the positive direction, and then take negative gradient steps in the bad direction. 0 you can already see what's going to happen.
[01:13:14.205] This first term over here, this is the good stuff. This is the bad stuff on the right term here. And they're going to be balancing each other. OK. And so if I take the gradient of this, I think this is the most intuitive form of DPO, at least to me, you have two things. For every pair that you've had annotated, I'm going to increase the likelihood of the example, that one, and I'm going to decrease the likelihood of the example that lost. And I'm going to scale the step size in some sense by how much my implied reward model is wrong. Did my model already assign a very high reward to the winning one? If it did so, I will take a small step. But if I was very wrong, and I said, oh, these two are almost equal, I had same probability. But in that case, I will actually take a much bigger step size. So based on the probability differences, I will take bigger or smaller step sizes on this differential log prob objective. So hopefully, the action of this gradient objective is quite clear to people. OK. So this is actually much, much simpler than PPO because all I'm doing is taking gradients.
[01:14:22.370] And we all how to do that with our standard processes. And I think the nice thing is that it does work reasonably well. I think for a while, people were very obsessed with is DPO better than PPO or vice versa? I think the answer now is maybe it doesn't matter very much, unless you're like at the frontier training the very best model. DPO is reasonably good. And it's good enough for LlaMA. It's good enough for me. If you go and open up the LlaMA tech report, you'll find that they basically had this nice outer loop training, where they had their model, they SFTd it, they did DPO. And then they used that DPO model to then generate candidates that the rejection sampled and they repeated this process. So there's the outer loop on top of it. But basically, the core RLHF primitive for Llama was DPO. In the years since, there's been a lot of different variants of DPO. There was SimPO, which basically changes the weighting. So instead of using pi ref, you have this y of l normalizer.
[01:15:23.630] There was length normalized DPO, which normalizes by the length in order to avoid certain length hacking issues that seem to arise. But really, none of these variants maybe seem to matter very much. One of the things that I'll mention, that I thought was very interesting, was these results are very, very contingent on the specifics of the experiment setup. So even at Ai2, there was a paper that was like, well, it turns out that if you go from DPO to PPO, you'll get a better result. But then they also had a paper for Tulu2, where they said, actually, if you do DPO right, you can do better than PPO. And so depending on how you execute this, one can be better than the other. I'm sure you have experiences reading deep learning papers and so on, where the results are very fragile. So maybe the takeaway from all of this is that these DPO variants are actually close enough to the right thing to give you pretty good performance in many cases.
[01:16:24.110] This core idea of take gradient step in the right direction and take negative gradient step from the bad stuff works reasonably well as long as you set the step sizes right. OK. I will end with the last few slides on things to watch out for in RLHF. So one of the things to watch out for is over optimization. And this is maybe one of the biggest things. To give you also historical context, when InstructGPT came out, and people were very excited about this, I think there was a very real question can we RLHF our way to superintelligent systems or whatever? Maybe we can. Just collect enough thumbs up, thumbs down. Turns out that is actually quite challenging. Because you find that if you try to really push the RLHF process forward, you'll start overfitting to your learned reward model. So the KL regularizer that I talked about is really critical in a lot of cases in order to prevent your optimization process from overfitting your reward model,
[01:17:23.710] at least if your optimization process is very good. The other thing that I'll talk about is model collapse. So people have seen lots of times, where RL models have much less diversity. They're concentrated on a few different possible outputs. And this connects back to what I said before right RLHF models are really no longer modeling a distribution which comes with its own inherent diversity. It's a policy that can collapse as long as it gets a good reward. So this is another thing that RLHF models have really struggled with in recent years. So actually, I'm going to skip this because we're at time. And I think this entropy and mode collapse thing actually ends up being quite important even today. And the GPT-4 era actually was one of the few plots that OpenAI put out, where they said, actually, we have a few open problems left. And one of the open problems they had was, actually, our models are uncalibrated after we do RLHF, which you can see in this plot over here.
[01:18:24.270] I don't think anyone has really solved that yet. The Anthropic folks also had an argument saying, well, it's naturally uncalibrated. You could recalibrate sometimes, but not always. This becomes very important in the next lecture when we talk about things like RLVR, where the entropy and exploration is actually quite critical for the model to explore all the possible solutions, and to make progress on some of these very hard problems. Great. So to put it all together, RLHF data collection is also very hard. I think part of the thing about post-training is it's very complicated, messy process because a lot of it is getting good data. And getting good data is always very difficult. RLHF algorithms are quite complex. PPO is especially difficult. I'll talk a little bit about it next lecture. Thankfully, we do also have a simpler variant. They're called GRPO. That works pretty well. That's what you'll do in your assignments.
[01:19:19.283] But this is one place where there's been a lot of recent developments. Finally, I think, in terms of RLHF, one of the big, big problems is over optimization. And the transition to the next lecture is going to be, is there rewards where we won't over optimize or we can just dumping compute and the model performance just keeps monotonically getting better. And that's one of the reasons why what people call RLVR is just so been so impactful. Thanks. I'll see you all Thursday.