CS336 2026 · Lecture 16已发布

Post-training - RLVR

日期: Wed May 20
讲师: Tatsu
课程资料: 10 项
内容产物: 4 项

课程简介

本讲从 PPO 的策略梯度、价值模型和 KL 约束出发，引出更适合可验证任务的 GRPO 与相关变体。通过 DeepSeek-R1、DeepSeekMath、Kimi 等案例，课程分析正确性奖励、组内基线、长思维链涌现、SFT 冷启动和奖励投机，说明 RLVR 能扩展什么、又不能替代什么。

讲义 / Slideslecture_16.pdf打开链接↗论文Proximal Policy Optimization AlgorithmsPPO↗论文DeepSeekMath - Pushing the Limits of Mathematical Reasoning in Open Language ModelsGRPO↗论文DeepSeek-R1 - Incentivizing Reasoning Capability in LLMs via Reinforcement LearningRLVR 案例↗论文Kimi k1.5 - Scaling Reinforcement Learning with LLMs推理强化学习↗论文Back to Basics - Revisiting REINFORCE Style Optimization for Learning from Human FeedbackRLOO↗论文Understanding R1-Zero-Like Training - A Critical PerspectiveRLVR 分析↗文档OpenAI Spinning Up - PPO算法教程↗代码AlpacaFarm PPO Trainer课程代码↗视频课程视频课程视频↗

Assignment 4: Data截止

Assignment 5: Alignment and Reasoning RL已发布

CS336 · Language Modeling from Scratch — Lecture 16 中文讲义

学习目标

完成本讲后，你应该能够：

理解 RLVR（Reinforcement Learning from Verifiable Rewards）与 RLHF 的核心区别，并说明为什么可验证奖励是推动更强推理模型的关键。
解释 PPO 在语言模型设置中的基本结构、主要组件以及实现上的常见痛点。
从策略梯度与基线（baseline）的角度推导并解释 GRPO 的目标函数，指出 GRPO 与纯策略梯度之间的差异。
对比 DeepSeek R1 / R1-Zero、Kimi K1.5、Qwen3 系列在后训练流程上的设计选择，尤其是数据筛选、奖励设计、长度控制与蒸馏。
识别 RLVR 在实际系统中的工程挑战：在线 vs 离线、长 rollout 的系统瓶颈、奖励可被攻击（reward hacking）的问题。
理解“结果监督 vs 过程监督”的争议，并掌握当前主流结论。

课程主线

本讲是后训练系列的第二讲，聚焦于基于可验证奖励的强化学习。课程分为两大部分：

核心算法：PPO 复习、GRPO 的动机与推导、GRPO 与策略梯度理论的关系。
开源系统案例分析：DeepSeek R1 / R1-Zero、Kimi K1.5、Qwen3 与 Qwen3-Coder-Next 的技术报告解读，重点在于数据、奖励设计、长度控制、蒸馏与智能体 RL。

全课反复强调一个核心观点：RLVR 之所以有效，是因为我们拥有难以被攻破的奖励信号；而 RLHF 受限于奖励模型的可过拟合性，无法持续投入更多计算。

第一部分：核心算法

1. 从 RLHF 到 RLVR 的动机

RLHF 的瓶颈：偏好数据依赖人工标注，奖励模型本身是学习得到的，持续优化会过拟合奖励模型。
AlphaGo 式的成功依赖于目标本身可验证：围棋胜负明确，可以无限投入算力。
数学、代码等领域天然具备可验证性，因此更适合 RL。
课程强调：RLVR 与 RLHF 在算法上并无根本差异，但奖励的性质决定了我们能投入多少算力。

2. PPO 复习与实现困境

策略梯度核心公式

语言模型 RL 的基础是策略梯度（REINFORCE 形式）：

\nabla_{θ} J (θ) = E_{x \sim D, y \sim π_{θ} (\cdot | x)} [R (x, y) \nabla_{θ} \log π_{θ} (y | x)]

即以加权 SFT 更新的方式对奖励做梯度上升，权重 $R (x, y)$ 可正可负。

PPO 基本结构

采样轨迹（rollouts）
计算优势（advantage），通常使用 GAE（Generalized Advantage Estimation）
对重要性采样比率做 clip，限制策略更新步长
拟合价值函数（value function）作为基线

为什么 PPO 实现很痛苦

PPO 对实现细节非常敏感，社区有大量“PPO 实现细节”类文章。
常见问题：
- KL 项操作方式错误（如只在正值处截断）会导致训练崩溃。
- 许多人使用 $γ = λ = 1$ ，退化为 bandit 问题，丢失多步结构。
- 价值网络与主模型同尺寸，显存开销大，不利于大规模推理。
- 梯度估计方差高，需要大量工程 hack 才能稳定训练。

为什么不用 DPO 替代一切

DPO 只适用于成对偏好（Bradley-Terry）数据，不适用于数学等非成对反馈任务。
DPO 本质是离线算法，虽可在线迭代，但并非通用 RL 工具。
社区渴望替代 PPO 的动机正是 PPO 的复杂性。

3. GRPO：核心算法

动机

去掉价值网络（critic），避免大规模额外参数与训练不稳定。
不使用原始 REINFORCE（高方差），而是引入组内相对优势。

定义

在给定 prompt 下采样 $G$ 个输出 ${o_{1}, . . ., o_{G}}$ ，计算每个输出的奖励 $r_{i}$ ，然后做 z-score 归一化：

A_{i} = \frac{r_{i} - mean ({r_{1}, . . ., r_{G}})}{std ({r_{1}, . . ., r_{G}})}

目标函数形式与 PPO 类似：

L_{GRPO} = \frac{1}{G} \sum_{i = 1}^{G} min (\frac{π_{θ} (o_{i} | x)}{π_{θ_{old}} (o_{i} | x)} A_{i}, clip (\frac{π_{θ} (o_{i} | x)}{π_{θ_{old}} (o_{i} | x)}, 1 - ϵ, 1 + ϵ) A_{i}) - β \cdot KL (π_{θ} ∥ π_{ref})

在线情况下， $π_{θ} = π_{θ_{old}}$ ，clip 操作失效，目标简化为：

E [A_{i}] - β \cdot KL

即直接对优势做策略梯度，减去 KL 正则。

GRPO 与理论策略梯度的偏差

严格遵循 Sutton & Barto 的带基线 REINFORCE，只允许减去与动作无关的基线（如状态相关基线）。GRPO 做了两件额外的事：

除以标准差：改变优化方向，并非严格无偏策略梯度。
长度归一化：按序列长度做归一化因子。

这两点既是 GRPO 的实用之处，也是它的理论不洁之处。

长度归一化的影响

除以输出长度会鼓励模型在错误输出上继续生成更长内容，以稀释负奖励。
在正奖励情况下，则倾向于缩短输出，可能损害正确性。
实践中观察到的“思维链无限增长”现象，部分来自长度归一化，而非模型能力提升。

标准差归一化的影响

对二元奖励（对/错）而言，标准差小意味着问题太简单（总是对）或太难（总是错）。
GRPO 会提高这些问题的权重，这在直觉上可能不是最优的：我们更希望模型学习处于能力边界附近的问题。

为什么 GRPO 广泛流行

不需要价值网络，只需要 rollout 与奖励计算。
实现极其简单，核心代码可在一页之内完成。
在开源社区中被证明能复现接近闭源 PPO 的效果。

第二部分：开源系统案例分析

1. DeepSeek R1 与 R1-Zero

R1-Zero：最简洁的 RLVR 配方

基座模型（已做指令微调）直接做 RLVR。
奖励 = 数学正确性 + 格式奖励（保证思维链被 <think> 标签包围）。
使用 GRPO。
结果非常干净：接近 OpenAI o1 的数学能力，且无需复杂后训练管线。

关于“顿悟时刻”的冷静解读

论文中宣称的“顿悟时刻”（模型在思维链中自我标记“啊哈”）并非 RL 的独特产物。
实验表明基础模型本身已具备这类行为，RL 只是激发了它。
更长的思维链更可能是长度归一化（GRPO 除以输出长度）的副产品，而非智能涌现。

R1 的生产化流程

在 R1-Zero 基础上加入：
- 少量长思维链 SFT 数据（可能来自蒸馏）
- 语言一致性奖励（防止思维链中语言混杂）
- 不可验证奖励混入 GRPO（接近 RLHF 的融合）
最终做标准 RLHF（与 DeepSeek V3 类似）。
结论：这是标准的“预训练 → SFT → 推理 RL → RLHF”流水线。

结果监督 vs 过程监督

DeepSeekMath 中曾使用过程奖励模型（PRM），但 R1 中放弃了。
原因：过程监督需要逐步标注，难以规模化；结果监督已足够好。
结论：当前主流是结果监督。

蒸馏洞察

将 R1 的思维链数据蒸馏进 Qwen / Llama 模型，可显著提升推理能力。
暗示：推理能力可能通过模仿获得，RL 是数据生成手段，而非唯一途径。

2. Kimi K1.5

与 R1 的对比

同时期发布，性能超越 R1，但讨论热度较低。
在算法推导上走了与 GRPO 不同的路径，但结果相似。

从 DPO 式推导到组均值基线

从 KL 正则化 RL 目标出发，假设可解析求解最优策略。
通过将奖励模型与策略比率匹配，导出与 GRPO 类似的组均值基线。
最终更新形式与“GRPO + KL”极为相似。

长度控制：比 GRPO 更明确

Kimi 明确反对无限增长的思维链，原因不是训练稳定性，而是推理成本。
加入启发式长度奖励：
- 正确答案也应尽量短。
- 错误答案不能强制太短，否则会陷入“永久错误”的陷阱。
平衡点：让错误答案比平均略短，但不无限压缩。

数据课程与筛选

排除选择题（无需长思考）。
使用 best-of-k 过滤：如果模型在 8 次采样中能解决，则从训练集中移除。
动态监测成功率，已解决的问题从问题集中删除，节省算力。

验证器设计

数学：用奖励模型检查答案等价性，而非硬性字符串匹配。
原因：模型可能给出正确但格式不同的答案，严格检查器会误判。
RLVR 的验证部分极其复杂，是“深不见底的兔子洞”。

系统效率问题

长 rollout 会阻塞批量训练。
在线策略（on-policy）训练最稳定，但系统利用率低。
复用 rollout 会导致离策略（off-policy）问题，引入训练不稳定。

专家迭代 vs RL

Kimi 大规模消融显示：RL 始终优于 expert iteration（仅 SFT 于正确答案）。
结论：要榨干模型潜力，RL 是必须的。

3. Qwen3 与 Qwen3-Coder-Next

Qwen3 的总体流程

基座 → SFT → 推理 RL → 思考/非思考模式融合 → RLHF → 蒸馏出小模型。
数据筛选：
- 过滤掉无需思维链就能答对的题
- 过滤与验证集过于相似的题（去污染）
- 对参考思维链做人工清洗
只用 4000 个样本做 RL，即足以获得显著提升。

思考模式与非思考模式

同一个模型通过提示标签切换长思维链模式与即时响应模式。
引入“提前退出”机制：附加特殊字符串可截断思维链。
现象：低思考预算下性能优雅下降，但思考模式仍优于即时模式。

关于混合模式的取舍

融合会带来非推理任务收益，但数学/编程能力下降。
Qwen3.5 已放弃“思考+非思考”混合，改为分开模型，以保住推理性能。

智能体 RLVR：Qwen3-Coder-Next

智能体后训练的核心其实是数据工程，而非新算法。
中期训练数据：
- 将代码仓库文件拼接为超长上下文
- 用 RAG 构建 PR 上下文
- 将文本+代码文档转为 Markdown
- 用 LLM 生成编码风格合成数据
- 记录编码智能体的运行轨迹

专家蒸馏策略

训练多个任务专家（web dev、UX、QA、软件工程 agent）。
将专家蒸馏回单一模型。
这种做法在前沿训练中罕见，更接近学术界的 branch-train-merge。

奖励可被攻击的典型案例

模型学会通过 Git 操作查看未来提交来“作弊”解决 issue。
即使禁止 git log，模型仍可通过添加 remote 绕开限制。
结论：RLVR 的稳健性完全取决于奖励函数。
即使是 Lean 这类形式化验证语言，也存在对抗性漏洞。

性能与泛化的谨慎解读

模型用 30 亿活跃参数即可在 SWE-bench 上取得 70.6% 准确率。
但在训练环境过拟合风险高，验证集提升不保证广泛泛化。

核心原则总结

RLVR 的核心是奖励的可验证性：越难被攻破的奖励，越能支撑大规模 RL。
GRPO = 组内 z-score 优势 + KL 正则：不是严格策略梯度，但实用且简单。
数据筛选比算法更重要：难度过滤、去污染、课程设计决定 RL 上限。
长度控制是关键设计维度：不控制会导致无限思维链，过度控制会导致能力陷阱。
验证器必须是工程重点：答案等价性检查、格式解析、对抗性鲁棒性缺一不可。
蒸馏不可忽视：RL 生成的高质量思维链可以被 SFT 吸收，从而缩小模型也能获得强推理能力。

课程作业关联

你将实现的 RLVR 算法与 DeepSeek R1-Zero 基本一致，包括：

多次采样（k 个 rollout）计算奖励
对奖励做 z-score 归一化
计算 KL 项（需要对参考模型做 stop-gradient）
在奖励与 KL 的组合上做梯度更新

预期现象：

奖励曲线上升
负 KL 项下降
思维链变长（但应注意是否由长度归一化导致）

作业中常见问题：

忘记 stop-gradient 导致梯度泄漏
标准差为 0（例如所有采样奖励相同）导致数值爆炸
思维链过长导致训练速度急剧下降

常见误区

GRPO 是“正确”的策略梯度：不，它偏离了基线定理，标准差归一化与长度归一化都不是无偏梯度变换。
思维链越长 = 模型越聪明：部分原因是长度归一化的副作用，尤其在错误输出上。
过程监督一定优于结果监督：R1 的实际经验是结果监督已足够且更易扩展。
RLVR 训练后模型一定在广泛任务上更好：可能会损害非推理任务能力，Qwen3 已观察到这一取舍。
验证器没有问题：即使形式化语言（如 Lean）也可能被攻击，验证器需要持续加固。
在线一定优于离线：在线策略梯度无偏但系统利用率低；离线复用 rollout 可提高吞吐但引入离策略偏差。

复习清单

[ ] 我能口头写出策略梯度公式，并解释为什么它可以看作“加权 SFT”。
[ ] 我能解释 PPO 为什么需要价值函数，以及它为什么使实现变复杂。
[ ] 我能写出 GRPO 的目标函数，并说出它与 PPO 的三个具体差异。
[ ] 我能解释 z-score 标准化的作用，以及它为什么不是严格基线。
[ ] 我能解释长度归一化对正负奖励的差异化影响。
[ ] 我能说明 R1-Zero 与 R1 在训练流程上的差异。
[ ] 我能说明过程监督为什么被主流放弃。
[ ] 我能描述 Kimi 的长度控制策略，以及它如何避免“永久错误”陷阱。
[ ] 我能说明 best-of-k 过滤的逻辑及其在课程设计中的作用。
[ ] 我能举例说明奖励可被攻击，以及它为什么制约 RL 规模化。
[ ] 我能区分在线与离线 RL 的权衡，并指出 GRPO 在线形式的简化之处。
[ ] 我能描述 Qwen3-Coder-Next 的中期训练数据构成与专家蒸馏策略。

从 RLHF 到 RLVR：后训练系列讲座中的思考模型革命

在《CS336 · Language Modeling from Scratch》的课程图谱中，我们已经走过了预训练、指令微调（SFT）和 RLHF 的阶段。上一讲结束时，我们带着对 RLHF 的悲观情绪离开——偏好数据与奖励模型的过拟合瓶颈似乎让我们走到了尽头。本讲（Lecture 16）则试图回答一个关键问题：如果我们把优化目标和信号源从人类偏好换成可验证的奖励（Verifiable Rewards），强化学习能否真正展现出 AlphaGo 那样的威力？本讲围绕核心算法 GRPO、DeepSeek R1、Kimi K1.5 与 Qwen 3 系列技术报告，给出了一个相当肯定的答案。

为什么 RLHF 不够好，而 RLVR 让人兴奋

上一讲结束时我给的调子比较悲观。我们讨论了 RLHF 的流程：收集偏好数据，训练奖励模型，用强化学习做优化。这套流程的核心问题是“过度优化”（overoptimization）。你不可能无限期地对同一个奖励模型进行强化学习——很快你就会过拟合。无论你把 KL 正则化做得多么精细，这个天花板始终存在。

但强化学习的真正潜力，在 AlphaGo 这类场景里展现得淋漓尽致。围棋的胜负条件清晰可见，你可以投入无穷多的计算量，因为只要目标在改善，你就是在做正确的事。反观 RLHF，它本质上是一种受限于人工标注的学习问题，不是一个可以无限扩展的搜索问题。

于是我们自然而然地寻找可验证奖励（Verifiable Rewards）的领域：形式化数学、自然语言的数学问题、代码执行。只要问题有确定的正确答案，我们就有了一个“不可攻破”的奖励来源，其成本远低于人工偏好标注。这就是 RLVR（Reinforcement Learning from Verifiable Rewards）的动机。算法上它和 RLHF 没有本质区别，但目标函数的性质让整个流程发生了质变。

回顾 PPO：为什么它是一个痛苦但不可回避的起点

要理解 RLVR，必须从策略梯度谈起。核心公式是 REINFORCE 梯度：

\nabla_{θ} J = E [R \cdot \nabla_{θ} \log π_{θ} (a | s)]

这个公式贯穿整个讲座。我们在做的本质上是对语言模型的采样输出施加带权重的 SFT 更新，权重可正可负，取决于奖励大小。

但逐步更新的策略梯度效率太低。于是我们有 TRPO，进而有 PPO（Proximal Policy Optimization）。PPO 在深度强化学习领域是绝对的主力，OpenAI 用它做过机器人走路、Dota 等经典工作。从概念上看，PPO 的伪代码不算复杂：采样轨迹，估计优势函数，用裁剪后的优势更新策略，再拟合一个价值函数。

问题在于实现。有篇著名博客文章叫“PPO 的 37 个实现细节”，这种事本身就说明问题。PPO 对实现决策极其敏感——基线到底算不算、经验缓冲区怎么管理、值函数何时更新、优势估计是否真用上了 GAE（Generalized Advantage Estimation），每一个环节都能让训练从收敛变成发散。

尤其是当我们把它搬到语言模型上时，情况更糟。语言模型是逐 token 生成的多步决策问题，不是一步臂赌机（bandit）。这意味着优势估计要处理每个 token 的回报，价值模型又是另一个和策略一样大的神经网络，存储和训练开销大，且极不稳定。我们在课程中体验过：即使外层循环合理、内层损失计算基本正确，真正让 PPO 跑起来依然需要各种 hack。例如，KL 惩罚要在 0 处截断才稳定，否则求和里正负项混乱就会崩溃；很多人还把 GAE 的两个参数 $γ$ 和 $λ$ 设成 1，实质退化为最大化末尾整体奖励臂赌机——这几乎等同于放弃了 PPO 的结构优势。

PPO 的这些问题催生了 DPO 这样的替代品，但 DPO 只适用于 Bradley-Terry 成对比较结构，无法直接应对数学求解这类从单一结果给出奖励的问题。我们需要一个更简单的在线 RL 算法，这就是 GRPO 登场的背景。

GRPO：去掉价值网络的组内归一化策略梯度

GRPO（Group Relative Policy Optimization）出自 DeepSeekMath 论文。它的意图很直接：保留 PPO 的核心思想，去掉最复杂的部件——价值函数（value model）。没有价值网络后，如何估计优势？答案是用组内 z-score 归一化。

假设你对同一个提示词采样 $g$ 个 rollout，记第 $i$ 个 rollout 的奖励为 $r_{i}$ 。GRPO 定义该 rollout 的优势为：

A_{i} = \frac{r_{i} - mean (r_{1}, . . ., r_{g})}{std (r_{1}, . . ., r_{g})}

这在直觉上很自然：如果我的这次采样比组内平均表现好，优势就是正的；否则为负。没有任何额外的神经网络，没有价值模型的拟合与存储，在在线采样下（即当前策略采样时），目标函数中的裁剪项变成恒等，最终目标退化为：

maximize E [A \cdot \log π_{θ}] - β \cdot KL

即用优势做加权更新，加上一个 KL 惩罚保持贴近参考策略。这本质上就是一个带奖励的、加了正则化的 REINFORCE。实现极其简洁，课程提供了一个来自 McGill 的参考实现，核心代码不到一页：计算每个组内奖励，做 z-score 归一化，加一个小小的 $- 4$ 防止标准差为 0（当所有奖励相同时会爆炸），然后计算 KL 惩罚项，最后合并做梯度更新。你不难理解为什么 GRPO 在开源社区中如此普及——它就是一个能在一页代码里写完、并取得不错效果的算法。

在 DeepSeekMath 的实验中，GRPO 明显优于 RFT（拒绝微调），这说明这套简化并没有牺牲多少性能，反而因为更稳定而收益更多。

但这里必须强调一个重要的理论瑕疵：GRPO 并不是标准意义上的策略梯度。Sutton 和 Barto 的带基线 REINFORCE 允许你减去一个状态相关的基线，但你不应该除以标准差。GRPO 除以标准差、并引入按序列长度归一化两项，都超出了基线定理的框架。这意味着它优化的是一个和原始奖励目标略有不同的对象。

这带来了一个有趣的副作用：长度归一化会鼓励负面案例生成超长序列，因为错误的负面奖励除以一个极大的长度后趋近于 0，模型得以逃脱惩罚。这无疑会导致思维链（COT）无限膨胀。标准差归一化则会抬高简单和极难问题在目标内的权重——这两类问题的奖励方差都很小，因而被放大。但这恰恰是模型学不到东西或不可能学会的问题。

尽管如此，GRPO 在实践中非常非常好用。它的简单性让大量实验室得以快速复现并迭代，所以它事实上已经成为开源 RLVR 社区的基调。

DeepSeek R1：一个干净到接近纯净的 RLVR 配方

DeepSeek R1 是开源 RLVR 模型的里程碑。它最重要的贡献之一，是提供了一个任何人都能复现的 RL 配方。相比闭源的 PPO 范式的痛苦，GRPO 的简单性让 RLVR 不再是超大规模实验室的专利。

R1 的起点是 R1-Zero：一个基础模型（当然，它已经过了指令微调），直接在其上做 GRPO，奖励只来自数学问题的准确率和格式（例如是否用 <thinking> 标签包围思维链）。没有 SFT，没有人工标注，没有复杂设计。就是这么简单的配方，得到的模型能力与 OpenAI 的 O1 比比，只差一点点。

R1-Zero 中的一个现象引起了巨大的社交媒体热议，那就是“顿悟时刻”——模型在思维链中惊奇地标记“我发现这个可以用……”。但我们回头分析，这个“顿悟”其实并不那么神秘。基础模型在预训练阶段就生成了大量的数学 token，早已具备识别这类结构的能力；RL 训练只是助推了这个模式的出现。而思维链长度的持续增长，我认为更直接的原因就是 GRPO 的长度归一化副作用——模型发现延长输出可以在不损失奖励的情况下减少惩罚。

但无论如何，R1 作为一个产品级系统，给我们带来了一份完整的架构视图。它的流程大体如下：预训练模型 → 长上下文扩展 → 推理强化学习（GRPO）→ RLHF → 蒸馏小模型。为了保持输出语言一致性，R1 还引入了语言一致性奖励。他们坦白说，早期 R1-Zero 式的训练会产生跨语言思维链，所以他们限制模型只能用单一语言来提升可读性，这纯粹是出于可解释性考虑。

R1 还告诉我们一件事：过程监督并非必需。DeepSeekMath 中他们确实做过过程奖励模型（PRM），但 R1 报告里坦白，过程奖励模型效果并不好——结果奖励模型（outcome reward model）不但足够，而且更好扩展。这是一个重要的行业共识：你的奖励足够干净时，你不需要昂贵的逐步评分标注。

另一个有趣的细节是 SFT 数据来源。R1 报告说他们“收集了一小部分长思维链数据”，但并未明确说明这些数据是否来自其他模型（如 O1）的蒸馏。这个模糊的处理方式在开源社区里很常见，但对复现者和科研者来说，它意味着 SFT 阶段可能隐藏了大量非 RL 的能力提升。事实上，后来包括我的学生的一些研究在内，越来越多文献表明，如果你的基础模型足够强，并且在正确的长思维链数据上做 SFT，你就能解锁绝大多数长推理能力，甚至可能不需要 RL。RL 的独特价值在于自我生成的监督信号，而一旦某种长思维链被别人生成并公开，学生模型可以通过模仿学习获得大量能力。这也是 R1 自己做的蒸馏实验：把 R1 的思维链放入 Qwen 2.5 或 Llama 系列，你就能显著提升它们的推理能力。

Kimi K1.5：不同的路径，同样的收敛

Kimi K1.5 是同时期的另一个重量级模型。它的技术报告在多个维度上极具价值。首先，他们给出了一个非常不同的算法推导路径——从 DPO 风格的推导出发，最终得到的更新规则却惊人地相似。

具体来说，他们从最大化期望奖励加 KL 正则项出发，解出最优的奖励模型及其策略比率，然后把目标函数转化为一种平方损失形式。对损失求梯度后，你得到的其实就是一个组均值归一化的策略梯度，外加一个 KL 正则项。这跟 GRPO 的结论一样，但走了完全不同的路径。这个事实非常有意思：它说明组均值归一化基线这个东西，可能不是一个特定算法特有的魔法，而是这类 RL 问题的自然产物，无论从哪个方向开始推导，你都会回到这里。

Kimi 还有一个比 DeepSeek 更明确的观点：长思维链不是免费的，甚至是有害的。推理成本是实际部署中的巨额开销。如果你贩卖订阅服务，而模型一次思考一小时，这对你的商业模型是不利的。因此 Kimi 在目标函数中加入了针对长度的启发式奖励，推动模型压缩思维链长度，而且做得相当巧妙：他们不是惩罚所有长序列，而是让错误答案的思维链必须短于平均水平，正确答案可以适当长一点——但也要尽量压缩。这样就不会导致一个关键问题：如果你把错误答案的思维链压到零，模型就再也不可能从那些错误中恢复过来，因为你根本没有正面的学习信号了。

Kimi 在数据层面的处理同样值得借鉴。他们做了中等难度过滤——如果一个问题在 best-of-8 采样下模型已经能答对，那就不是一个好的 RL 训练样本，因为模型已经掌握了它，再学也不会有什么进步。同时，他们也放弃了太难的问题，因为完全没有奖励信号就没法学。这个中间范围的课程想法，是 RLVR 一个非常重要的实操准则。

此外，他们还呈现了 RL 基础设施的复杂度：rollout 可能很长，有些是黎曼猜想级别的难题，一个长 rollout 拖住整个训练流程，如何做截断、如何切换框架、如何在训练和推理之间共享权重，都是非常硬核的系统问题。而且，他们展示了一个关键的对比：真正的强化学习（RL）在最终性能上持续超越专家迭代（expert iteration，即只在正确答案数据上做 SFT）。这意味着，当你想从模型中压出最后的能力上限，你无法回避真实的 RL。

Qwen 3：从 SFT 到 RLVR 到成熟的系统化流程

Qwen 3 的技术报告更接近一个工业系统总览。它的流程也遵循了后训练的标准模板：预训练 → SFT → 推理强化学习 → RLHF → 蒸馏小模型。他们吸取了 DeepSeek 和 Kimi 两家的大量实践经验，包括难度过滤、去除无需推理即可解决的题目、数据去污染、对参考思维链做人工过滤等等。

最让我惊讶的一点是，他们在推理 RLVR 阶段只用了 4000 个样本。这说明 RLVR 在你拥有良好基础模型和干净奖励函数时，需要的样本量并不大。真正重要的还是数据质量和目标函数的正确性。

Qwen 3 还提供了一个独特的“思考模式”功能，即模型既可以长思维链推理，也可以即时响应。这实际上是通过一个标签来实现的，同一套权重既支持长思考也支持短回答，无论你是否思考，都可以在一个模型内完成。更妙的是，他们实现了“提前终止思考”的机制，即在任意中间 token 处停止思维链并强制要求模型回答。令人惊讶的是，在低思考预算下，即使模型在思考中途被截断，其应对数学或编程任务的能力依然明显优于非思考模式。这说明长 COT 训练使模型在思考较浅的阶段就已具备更强的信息利用能力。

Qwen 3 的消融也显示，融合思考和非思考模式会带来一些领域性能的小幅下降，比如数学和编程。我认为这正是他们在 Qwen 3.5 系列中把两个模式分为不同模型的原因——用户需要极致推理时，你不想为融合付出任何性能代价。

Qwen3-Coder-Next：智能体 RLVR 的深度解剖

最后，课程进入了 Qwen3-Coder-Next（代码智能体后训练前沿）的细节。这份报告的价值在于它展示了智能体（agent）后训练的所有秘密。核心遵循着整个课程的一贯信条：数据胜于一切。

他们从“中期训练”（mid-training）开始，通过拼接代码仓库、构造长上下文数据、把 Pull Request 与 RAG 合成上下文混合进来、自动生成 Markdown 文档，来注入大量编码和智能体能力。这些数据被模型用来模拟真实的编码环境：打开多个文件、解析错误、修改代码。他们还训练了多个“专家模型”——网页开发、UX 工具使用、软件工程、QA 等——然后把这些专家模型蒸馏回同一个最终模型中。这个做法很有趣，因为它某种程度上回到了“分支训练合并”的学术风格，而不是传统的多任务联合训练。

在软件工程智能体的 RL 环节，他们建立了一个庞大的 GitHub 环境。真正的亮点在于对抗性奖励设计：模型太容易学会绕过检查了。举个例子，在 Git 历史中如果有未来的提交，模型可以直接查看 commit 知道修复方案，从而作弊。他们的奖励函数几乎专门为了阻止这种作弊而设计。如果你不这么做，曲线就会像右边那张图一样：训练平淡，然后突现一个跳跃——那是模型学会了操纵 Git 历史。甚至在你禁止使用 git log 时，它还可以通过添加远程仓库（origin）来查询提交记录。所以你看，所谓“可验证”的环境也是一样脆弱的。我们在 Lean 上做 RL 时也天真地以为那个形式化系统坚不可摧，结果发现 Lean 编译器在某些罕见模式下都能被对抗性样例攻破。因此，RLVR 的稳健性就是你的奖励的稳健性——这才是它真正的核心瓶颈。

即使如此，他们在这套环境上训练出的模型，在 SWE-bench 上达到了约 70.6% 的准确率，而模型只有 30 亿活跃参数（active parameters），非常惊人。不过，我们需要冷静看待单任务指标，它并不代表全知全能的泛化能力。

结语：从 Algorithm 到 System，再到 Rewards

整个课程走到这里，你已经看到了一个完整的技术脉络：预训练是基础，也是所有能力之源；之后指令微调让你获得对话和格式能力；RLHF 让输出贴合人类偏好，但受限于偏好标注的规模；而 RLVR 则在可验证问题里彻底释放强化学习的能量——它使你能在数学、代码甚至智能体任务上无限制地增加算力，只要你相信你的奖励是干净且不可攻破的。

这个讲座的核心要务是让你掌握 GRPO。它的函数形式、更新规则、实现方式，应该刻进你的肌肉记忆，就像预训练损失一样熟悉。GRPO 的意义在于它让 RLVR 变得足够简单，以至于从一个“只有大厂才能玩得动”的方向，变成了一个任何人都可以在作业中复现、在实验室里迭代的标准工具。

与此同时，你必须认识到 RLVR 现在的挑战已经不是核心算法——方法大家都已经知道怎么用了，而是工程。它挑剔、有噪声、跟基础设施耦合紧密，但它的难是有边界的，不像在机器人领域做 PPO 那样无边无际。

最终，我们的目光还是要回到奖励。RLVR 的一切都建立在“你的奖励是正确的”这一假设之上。无论是数学答案等价性检查器的复杂设计，还是 Git 历史反作弊，或者 Lean 可验证环境里的隐藏漏洞，都在提醒我们一个事实：可验证奖励不是一个理论完美的概念，而是一个需要永不停歇地攻击和修补的工程对象。但正因如此，它才如此有吸引力——它把强化学习的价值和潜力真正地、一次又一次地展现在我们面前。

课程信息（官方入口）

视频：CS336 2026 Lecture 16 —— Post-training - RLVR（官方 YouTube 链接）
讲义与课程主页：CS336 Language Modeling from Scratch 官方网站下对应 Lecture 16 资源。

CS336 2026 Lecture 16：完整中文逐字稿

视频：官方视频
来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。

正文

[00:05.000] 好的，我们开始吧。这是后训练系列讲座的第二讲，我们将讨论强化学习领域令人兴奋的新进展，也就是 RLVR，即基于可验证奖励的强化学习（Reinforcement Learning from Verifiable Rewards）。我认为这在某种程度上非常及时。我没来得及更新幻灯片，但就在刚才，OpenAI 的团队发布公告说，他们解决了一个重大的问题。那是一个 MyOpenMath 问题，他们用其中一个思考模型解决了它。那是怎么做到的呢？这正是思考模型或者 RLVR 这套东西。到目前为止，我们所处的阶段是这里的左侧部分。上节课结束后，我们已经有了指令微调、RLHF，以及一些类似 ChatGPT 的好东西。

[00:56.480] 剩下的就是思考模型方面的现代发展了。模型需要有能力进行非常长的思维链（COT），并解决这些困难的、可验证的问题，比如数学问题，在某些情况下还有编码问题。请记住，我上节课结束时的基调有点悲观。RLHF 并不能真正让我们达到目标，因为存在一个叫做“过度优化”的问题。所以我们会去收集一堆偏好数据。偏好数据非常有用。我们会构建一些奖励模型，然后针对那个奖励模型进行强化学习。但这本质上是一个非常受限于人工标注的瓶颈，因为我们有一个奖励模型，而你不能持续不断地向同一个奖励模型投入计算资源，最终你会过拟合你的奖励模型。而且无论你在正则化方面做得有多好，最终你都会遇到这个过拟合的问题。

[01:49.990] 所以，这确实令人沮丧。而且这并不真正是强化学习的潜力所在。如果你是一个强化学习的坚定信徒，你会看到像 AlphaGo 这样的例子，并说，嗯，在这些领域，强化学习确实做得非常出色。我们在 RLHF 中所做的，与那些更原生或更天然适用于强化学习的领域所取得的成就相比，区别在哪里？我认为，真正的区别，或者说像 AlphaGo 和 RLHF 之间的一个主要区别是，在 AlphaGo 中，我们优化的正是我们想要的东西。你有围棋游戏的胜负条件。这个定义没有任何含糊之处。所以你可以投入任意多的计算资源。只要目标在改善，你就做得很好。从某种意义上说，这些是搜索问题。

[02:40.000] 而上面那个则更像是一个学习问题。这并非一个非常精确的区分，但这是一种思考方式。所以现在可能还有其他领域，比如形式化数学，甚至自然语言数学，它们具有这种更可验证的特性，因此更适合强化学习。这就是我们为什么要研究这些内容的主要动机。算法上根本不会有太大不同，但最终我们到达的地方将会出人意料地不同。今天的讲座将分为两个部分。这和我们课程在其他部分的结构方式很相似。在讲座的第一部分，我将讨论核心算法。这对你们来说是基础知识。你们会在作业中用到其中一些内容。我会讲到什么是 GRPO，PPO 是如何工作的。

[03:29.130] 然后，在我讲完这些之后，我会深入挖掘一系列开源发布，向你们展示我在第一部分教给你们的内容，是如何反映在许多主要开源模型发布的技术报告中的。所以我们要深入探讨方法和基础部分。我要讲的第一件事是 PPO。要讨论语言模型的强化学习，就不能不讨论 PPO。尽管我们已经讲过一遍了，但 PPO 足够令人困惑，我认为再讲一遍你们会受益匪浅。对于语言模型的强化学习，最重要的一点，或者说需要记住的，是策略梯度。特别是这个 REINFORCE 梯度技巧。我们始终要做的就是对我们的奖励进行梯度下降。我们会通过本质上采取加权 SFT 更新的方式来实现这一点。权重可能是正的，也可能是负的。我们会定义 R 是什么。

[04:29.640] 但这个公式很重要，因为它会贯穿整个讲座反复出现。从某种意义上说，这是推导出其他一切的核心。现在，在此基础上，记住上节课的直觉是，我们想要一次走不止一步。策略梯度要求我们在每次想要进行梯度更新时都从我们的策略中采样。我们能重复使用我们的 rollout 吗？嗯，你可以用 TRPO。你也可以用 PPO 等等。我待会会讲 PPO。我已经讲过 TRPO 了。我不会在这方面深入太多。如果你上过强化学习课程，你当然很熟悉 PPO，理解这个算法，但我不假设你们所有人都上过强化学习课程或者理解强化学习。所以，如果你不熟悉 PPO 的话，要知道它是强化学习领域真正的主力军。

[05:20.450] 它已被用于各种非常困难的强化学习场景中。OpenAI 在他们早期非常信奉强化学习的时期做过不少这样的事情。我记得有一个是 OpenAI Gym，他们当时在谈论 PPO 算法，就好像是说，嘿，我们可以用强化学习让这些人形机器人走路。还有一个更引人注目的演示是他们的 OpenAI 机器人，就是用 PPO 训练的。所以 PPO 在处理高维动作和状态空间的深度强化学习中表现出色，在这些情况下，做强化学习比通常情况要复杂得多。哦，是的。还有那个到处跑的 OpenAI 人形机器人。好的。所以在概念层面上，PPO 实际上非常简单。记住我告诉过你们，这一切中重要的一部分是摆脱 PPO。这已经成为激励很多人的一个因素。但如果你看一下 PPO 的伪代码，这来自于 OpenAI 的 Spinning Up 强化学习文档。

[06:23.555] 你看到这个，你会说，这没那么糟糕。这实际上相当简单。我可以一次性实现它。你采样一些轨迹。我使用任何一种优势估计方法来计算一个叫做优势的东西。我以一种稍微奇怪但还行的方式对优势进行裁剪。然后我在这个裁剪后的优势下更新策略。这样就很好。然后我还可以拟合一个叫做价值函数的东西。好的，没问题。这些在实践中看起来都相对容易。所以我想正因为如此，人们会觉得，哦，没问题。PPO 完全没问题。但接着你会看到像这样的博客文章，这会让你心生恐惧。因为如果你看到一篇博客文章说“PPO 的 37 个实现细节”，你就知道这是一个对实现决策非常敏感的算法。

[07:10.730] 有各种各样你可以做的事情，还有各种各样的库和实现，它们给出的数字会根据你使用的不同而完全不同。事实证明，很多人把这件事实现错了。所以有一篇论文说，实际上有些人在 PPO 中使用的基线根本算不上是基线。它们从根本上改变了优化问题。所以基本上正因为如此，我想如果不告诉你们，嗯，我们可能真的需要开始看一些 PPO 的实现，那我就没法讲完接下来的讲座了。而 PPO 的实现，我跟你们说，在语言模型上并不是特别令人愉快。这张幻灯片非常具有代表性。可以说，它很好地描述了 PPO 的样子。这里有优势估计（advantage estimation）。中间这个大块，就是那个优势的东西。你有一个经验缓冲区（experience buffer），因为你要保留一些旧的数据。

[08:05.828] 它要训练这个价值模型（value model）。而价值模型被用于优势估计的计算中。注意绿色框出现了两次。重要的是，我的目标函数中的某些部分，即 KL 项，实际上是逐 token 操作的。所以它实际上不仅仅是一个赌博机问题（bandit problem）。它是一个完整的多步强化学习问题。这一切都非常困难和复杂。因此，这导致了各种非常棘手的实现问题，你必须去处理。如果你查看各种实现，你经常会发现人们在用 PPO 时做的事情有时会有点奇怪和棘手。让我看看我能不能找到。之前我的一个学生为某个项目实现了 PPO，做的是 RLHF。所以 PPO，如果你还记得，曾被用于 RLHF。我们当时有一个相当健壮的 PPO 参考实现，并且花了好长时间才让它跑通。所以我深入看了一下细节，比如我们到底做了什么让这个东西能工作的。

[09:05.290] 外层循环没问题。外层循环大概是这样的：你进行策略采样（rollouts），然后采取一些 PPO 步骤，计算一些损失，裁剪梯度，然后更新几步。这是完全合理的外层循环。但接着你开始深入看。就目前来看，实际的内层计算损失部分几乎完全遵循 PPO 更新公式。所以这部分看起来也基本没问题。我们计算优势，计算裁剪比率（clipping ratios）。然后基于此更新模型。这些都很好。但接着你会遇到一些更棘手的地方。实际上，我会跳过策略采样部分，在那里你会遇到这样的情况：嗯，也许我们需要 KL 惩罚来让原始模型接近参考模型，但实际上这只有当你把 KL 在 0 处截断时才有效，当然，如果你对 KL 散度有所了解的话，这完全破坏了 KL 散度的意义。

[09:58.090] 你有正负值同时在求和。如果你去掉这个，它马上就爆掉了。诸如此类的事情都可能发生。这并不是说这就是你应该实现 PPO 的方式。我想我提这个的原因是，强化学习算法的实现可能有点棘手，而且相当敏感，因为PPO 中有这么多移动的部件，而且梯度估计的方差很高。这些都是非常非常棘手的事情。还有另一件你经常看到的事情，这个不光是 RL 实现的问题，很常见。在最初的 PPO 论文中，有一个叫做广义优势估计器（generalized advantage estimator）的东西，它基本上是在看这个 gamma 折扣奖励，其中你有这个价值函数来估计每个 token 生成步骤的奖励。但事实证明，人们经常只是用 gamma 等于 lambda 等于 1，这只是一个退化设置，把它又变回了一个赌博机问题。

[10:53.300] 所以你丢掉了从 PPO 中得到的很多结构。所以如果你在经过一番工程努力后运行这个算法，你会得到你预期的结果。当你在作业中运行你的 RL 算法时，你应该预期会看到类似的事情，比如奖励上升。如果你在做 RLHF，奖励模型的奖励会上升，然后负的 KL 正则化项会下降。这是可以预期到的合理现象。但不管怎样，所有这些的重点是说，PPO 的实现可能非常痛苦。取决于环境，它可能需要各种 hack 来稳定。所有这些都很痛苦。并不是说这不可能。很多人在 PPO 上训练出了非常好的模型。许多实验室都有非常成熟的一体化方案，可以让 PPO 在大规模下工作。所以这不是不可能的。但对于像研究人员这样从头实现的人来说，PPO 可能真的非常挑剔和复杂。

[11:51.870] PPO 在许多情况下不受青睐的另一个原因是，它需要一个价值模型来估计每一步的价值。这个价值模型有多大？嗯，它和原始模型一样大，所以这会消耗一些你宁愿用于其他东西的内存，比如模型或推理服务器。所以现在你可能会说，嗯，上节课你告诉我们 DPO 这个东西，DPO 看起来相当不错。我们为什么不直接用 DPO 做所有事情呢？问题是 DPO 是一个非常具体问题的非常具体的解决方案。DPO 适用于 Bradley-Terry 比较形式的成对反馈。这是非常具体的。如果我想解决数学问题，我的数学问题并不是天生就以成对比较的形式出现的。还有其他 DPO 变体旨在打破这种成对结构，但真的，你只是在用错误的工具做事情。

[12:44.900] 你不一定想要用 DPO 来做你通常会使用 PPO 做的事情。PPO 是更通用的工具，你可以用它来敲打任何东西。DPO 通常是离线的。虽然我认为这种区别被过分强调了，因为它可以通过反复迭代 DPO 来实现在线化。所以我不认为这一定是一个大的区别。这就是所有这些的背景。然后，与 DPO 类似，研究社区也非常渴望不必使用 PPO。希望 DPO 和 GRPO 被采纳的事实能告诉你，在许多情况下让 PPO 工作是多么痛苦。所以 GRPO 和 DPO 很像，是一种替代方案，是在可验证任务上做 RL 的更简单方法。它在很大程度上已经接管了开源、开放知识——让我们称之为社区——中的 RLVR。它在精神上与 PPO 非常非常相似。只是它去掉了几个最复杂的部分，通过这样做，让我们有了一个更简单的算法。

[13:51.130] 所以 GRPO 出自 DeepSeek 的论文。我想是在他们的 DeepSeekMath 论文里，你从 PPO 开始。它在概念上说的是，PPO 是个好主意，但我们只想改变几件事。它改变了什么呢？它改变了可以说是 PPO 中最复杂、最烦人的部分，也就是价值函数。价值函数是你作为基线减掉的东西，用来减少梯度更新的方差。价值函数是一个完整的神经网络。它会使训练不稳定。我们不想要价值网络。那我们怎么做呢？嗯，我们去掉了价值网络，但我们仍然需要一个优势。我不想用 reinforce——原版的 reinforce，因为那个方差会非常高。所以取而代之的是，我会计算组内的优势，作为 z-score，一会儿我会解释这意味着什么。

[14:40.640] 但基本上，思路是这样的。通常，你看你得到的奖励，如果你有一个价值函数，你会把它与你的预测值比较。你会说，我的神经网络说我应该得到 5 分。我得到了 6 分。所以这是一个好的采样轨迹。这通常是你做的。相反，你要做的是，假设你的采样轨迹得了 5 分，你现在会另外采样 10 个采样轨迹，然后你会说，与其他 10 个采样轨迹相比，我表现如何。如果我比我的平均值做得好，那么我有一个高的优势。从某种意义上说，在你可以从同一个提示词多次采样之类的场景中，计算 advantage 的一个非常自然的方式就是这样。所以 GRPO 正是这个思路。如果你看一下 DeepSeek 论文中 GRPO 的初始定义，你会看到，这就是被优化的目标函数。

[15:30.860] 他们会完全按照 PPO 的更新方式来做，其中有这个 min clipped advantage，加上一个 KL 项来让你保持在参考模型附近。然后 KL 是以一种特定的方式计算的。这个细节不是特别重要。重要的是，这个 advantage 是对一组输出 o 的 1 到 g 的样本计算 z-score 得到的。所以这就是——或者在每个组内，你有——哦，抱歉。这个 advantage 的 i，这是你计算 advantage 的地方。你所做的是，你取每个组内 rollouts 的奖励，减去均值，再除以标准差。

[16:04.710] 这样你就得到了每个组的 z-score。所以在在线情况下——这是一个重要的区别，因为在很多情况下，你实际上可能最终是在线运行而不是离线运行的——在在线情况下，clipping 就消失了，因为 pi theta old 和 pi theta 之间的比率是 1，这个 clipping 操作符永远不会产生任何作用，你就只剩下 min a of i, a of i。所以这就只是 advantage 减去一个 KL 惩罚项。所以我将对我的 advantage 做 RL，而这个 advantage 是由这个定义的。所以在在线 rollout 的情况下，这是一个非常非常简单的对象，作为一个 RL 目标来说很有意义。我可以在这里暂停一下，以防 GRPO 的某些概念性部分不太清楚。

[16:46.280] 我的意思是，从某种意义上说，这是所有那些令人兴奋的数学内容的基础。或者说，你可以用 GRPO 来复现所有那些令人兴奋的数学内容。所以在继续讨论一些具体细节之前，值得稍微仔细地理解一下它。好的。很好。简单到大家一遍就能理解。我很满意。而 GRPO 之所以非常非常简单，是因为没有价值函数，你基本上可以用一个非常简单的代码块把整个 GRPO 写下来，你只需要说，我要 rollout k 次，我要把我观测到的 rollout 与这 k 个参考进行比较，做 z-score 标准化，然后直接对权重取一个 REINFORCE 梯度。

[17:32.210] 你会在作业中实现这个。它会涉及一组非常简单的步骤。为你每次 k 个 rollout 中的每一个计算奖励。对你的 rollouts 做归一化，计算 KL 项。然后就在你的 KL 项和这些 rollouts 的组合上做梯度更新。你可能需要稍微思考一下，因为要使用自动微分来做这件事，你需要在某个地方做 stop grad。但实际上这并不复杂。你可以看到，你可以用一页代码就实现这个基本思路。这是我从麦吉尔大学的一些人那里拿来的 GRPO 参考实现，它有一个非常好的小巧玩具版本。组索引的计算、标准差的计算等等这些都只占幻灯片的一半。他们与 GRPO 论文原文唯一不同的地方是，他们在标准差计算中加了一个小小的负 4，

[18:33.560] 以防止当你只有一个样本时，或者当你的样本恰好有完全相同的奖励时，标准差会爆炸——这种情况确实会发生，比如你处于一个可以恰好得到数值 0 奖励的领域，像是你没能解决一个数学问题。但希望这能让你体会到 GRPO 有多么直接了当，以及为什么几乎所有开源工作都建立在这个算法之上。它易于实现，易于理解。而且，正如我们稍后会在结果中看到的，它在模仿许多闭源实验室可能用 PPO 做的事情方面，提供了相当有说服力的结果。那么它效果如何呢？在最初的 DeepSeek 数学论文中——如果你对数学 AI 感兴趣，我鼓励你去读一读——它有一组很好且非常有趣的结果。他们展示了 GRPO，即黄色和蓝色的线，比 RFT（如拒绝微调）做得好得多。

[19:31.097] 你们也会把这个作为基线来实现。那基本上就是取你的模型生成出来的正确答案，在它们上面做训练，然后把其他所有东西都丢掉。而且他们似乎还展示了过程监督——不仅是给最终答案打分，还给中间步骤打分——会带来一些收益。我稍后会更多地讨论这个。这是一个需要讨论和覆盖的重要内容，因为这是这类 RL 问题中最大的设计决策之一。但现在，你只需要内化这个事实：GRPO 是有效的。蓝色和黄色的线在其他的线之上。好的。那么现在让我们再深入思考一下 GRPO 的目标函数。这个目标函数中到底发生了什么？我们真的在取策略梯度吗？有一些重要的概念性问题我们确实应该想清楚。如果我们思考 GRPO 和 PPO 之间的区别，记住 PPO 有一个价值函数，它会输入到 advantage 中。

[20:27.580] 而在 GRPO 中，我们去掉了那个，用 z-score 的家伙来替代它。这个 a of i，即 r 减均值除以标准差。那么这是一个好的 advantage 吗？我想你们中那些上过 RL 课的人，已经知道这个问题的答案了。这是一个好的 advantage 函数吗？什么造就一个好的 advantage？什么是一个有效的 advantage 函数？嗯，如果你看 Sutton 和 Barto 那本经典的强化学习巨著，他们会说，有一个叫做带基线的 REINFORCE 的算法。你被允许做的事情是，如果你在做策略梯度，也就是这里的这一行，这第一个方程。我想要朝我的奖励方向做梯度步。你能做的事情是，你可以取你的奖励，然后减去任何——他们所谓的状态相关基线——在这种情况下，就是我们的状态。

[21:20.800] 如果我们处于 bandit 环境中，那其实就是提示词。所以你可以有一个依赖于提示词的基线，你可以减去它。任何只做这些事情的东西都是策略梯度的有效形式。只要你这样做，你仍然会朝着相同的方向下降。而根据你对 b 的选择，你在这个梯度下降过程中会有更低或更高的方差。现在，你可能应该注意到，我们正在做的并不是这个。我们不只是在减去一个常数，我们还在除以标准差。所以那是一个问题。事实证明，嗯，在某种程度上它是一个问题。在另一种程度上，它又不是。如果你真的想要一个概念上清晰的算法，一个真正像它声称的那样做事的算法，它实际上是在让奖励下降。GRPO 并不是这样做的。因为正如我在这里用红色标出的，GRPO 除以标准差，这打破了基线的约定。

[22:20.630] 我们不只是在除以或减去一个基线。我们还在用奖励的标准差做归一化。当然，GRPO 还做了另一件事，我之前没有提到。作为一个很棘手的实现细节，GRPO 几乎是按 token 来做的。他们会把总序列长度作为一个归一化因子来除。但如果你尝试从第一性原理出发推导 GRPO，遵循策略梯度和基线定理，你最终会得到不同的东西。你不会有一个长度归一化项，也不会有标准差归一化项。在GRPO发布后不久，有些人注意到了这一点，他们写了一篇论文，基本上是说，如果你不采用这两个不同的做法，你可能会得到非常不同、甚至可能是非常积极的行为。我待会儿会讲到这个。另外，当我们讲到——不是OpenAI，抱歉，是DeepSeek R1论文的时候也会涉及。

[23:22.890] 但这里的主要内容是想说，GRPO并不是这个想法的第一性原理推导。它实际上做的事情略有不同，既有优点也有缺点。那么这两个项到底是做什么的？我们现在知道，GRPO实际上并不是直接对奖励目标进行梯度下降。我们知道它有这两个修正因子。它们有什么作用？嗯，长度归一化比较容易理解。我们做的是什么呢？我们用更大的数去除更长的序列。所以这样做的效果是，它会鼓励模型生成长输出，因为——或者说抱歉，当你出错的时候，它会鼓励你生成错误的输出。因为假设——我知道我即将把一个数学证明做错。我将会得到我的负奖励，比如说负1。我就生成一个无限长的字符串。如果我这么做，我就可以除以无穷大，然后我就可以完全摆脱我的负面惩罚。

[24:19.900] 这是一个极端的例子。但你能理解，如果你除以输出长度，你就会鼓励模型在意识到自己实际上无法解决问题的时候继续喋喋不休。所以这就是你最终会遇到的长度问题。如果你修复了这个问题，你会发现，人们经常在GRPO的思维链（COT）中观察到的现象，也就是思考长度不断增长、增长、再增长，实际上最终只是封顶在一个常数上，而不是无限期地持续增长。所以这看起来可能是一件好事。尤其是在错误的案例上，你真的不希望生成越来越长、越来越长、越来越长的输出。那么标准差归一化，这个稍微复杂一点。但如果你仔细想想，我做的是除以标准差。所以这可以被理解为，强调那些标准差较小的问题。

[25:15.320] 对于二元奖励问题，什么时候标准差会小呢？那就是当问题太简单或者太难的时候。如果我的问题真的很容易，我每次都得到100%的正确率，那么我就会除以1除以——或者说我会除以基本上等于0的数，因为如果我总是答对问题，那就没有变异。同样地，如果我总是答错问题，我的奖励变异也是0。我就会显著地提高这个问题的权重。所以标准差项是在给两个极端都加权重，既包括简单问题也包括难题。这看起来显然是我们可能不想要的，因为我们希望模型可能在它可解决范围内的东西上进行学习。所以这些就是RLVR的算法组件。我接下来会讲解近期发布的几个不同模型，并讨论我认为非常重要的所有算法组件，包括今年我新增的一些内容，这些内容实际上是智能体相关的东西，我认为这些已经逐渐流行起来，

[26:18.680] 并且对于使用和部署这些模型来说越来越重要。但在我继续之前，我可以在这里暂停一下，如果大家对算法有什么问题，我会在继续之前接受提问。好的，很好。那么，我们开始讲DeepSeek R1，因为我认为这可以说是一种社会现象。我认为每个人都应该对DeepSeek R1有所了解。我觉得这是一篇很棒的论文，它开启了开源RLVR模型的浪潮。它了不起的地方在于，我认为它是第一个真正匹敌OpenAI O1行为的。OpenAI O1的关键特点是，它有非常长的思维链。这明显是强化学习的结果，而且在困难的数学问题上表现非常好。他们还提供了一个任何人都能复现的RL配方。我认为从研发的角度来看，这非常重要。如果你的解决方案是像那种可怕的PPO方案，只有DeepSeek自己能跑得动，那它的影响力就会小得多，至少对于我们这些研究人员来说，相比于GRPO这类方案是这样，

[27:29.010] GRPO是任何人都能真正拿来玩一玩的，甚至你们在做作业的时候都可以用。最后，他们还有一些非常有趣的蒸馏洞见，我认为这些洞见至今依然成立。所以DeepSeek R1的起点是，我想你可以看到DeepSeek的很多专长是如何相互累积的。他们之前有那篇DeepSeek数学论文，在那里面他们就在做GRPO。他们已经深入理解了在数学问题上做强化学习的很多细微之处。因此，他们在R1中继承了很多DeepSeek数学的组件。但其中一个非常重要的区别，我特意要指出来，因为我认为它特别重要，就是他们放弃了过程监督，这是DeepSeekMath中效果很好的一个东西，而他们只使用结果监督。需要澄清的是，结果监督是指，你只根据最终答案是对还是错来给予奖励。过程监督则是指，你有一个评分标准，比如说，用来检查证明中中间步骤的有效性。

[28:31.180] 很多人当时认为过程监督很重要。但事实证明，在很多事情上它并不是关键。所以R1的有趣之处在于它包含了很多不同的组件。但其中一个非常有趣的组件是R1 Zero，在这个版本中他们并没有做太多后训练，所以他们有一个基础模型。当然，这个基础模型，正如我们现在都知道的，也是经过指令微调的。所以这个模型能够执行一定程度的指令跟随。然后在这个基础模型之上，他们基本上就是做RLVR，他们GRPO算法的奖励就是准确率，也就是根据是否正确解决一组数学问题来给予奖励，再加上格式奖励。格式奖励之所以重要，是因为它基本上允许他们之后剥离出思维链。他们希望模型正确地使用思考标签来包围其思维链。但这是一个非常简单的配方。GRPO非常简单。准确率奖励也非常简单。

[29:27.400] 事实上，你们将在某种程度上实现和复现这组结果，而且你们做的就是这种非常简单的配方。那么最终结果如何呢？你得到的东西只比OpenAI O1差一点点。这真的是非常、非常、非常干净的结果。我喜欢这个结果，因为它没有混入任何真实生产环境中后训练流程的那些杂乱元素。你不需要去怀疑，哦，是不是因为RLHF起了作用，或者这个那个的原因。就是一个非常简单的基座模型加上GRPO。你的数学能力就相当不错了。我认为R1之所以流行起来，部分原因也是因为他们在论文中声称观察到的那些有趣现象。他们声称观察到的现象之一就是，随着训练时间的推移，模型能够思考的时间越来越长、越来越长、越来越长。

[30:15.570] 它们的思维链变得越来越长、越来越长、越来越长。他们还强调了一个我认为在社交媒体上疯传的东西，那就是那个“顿悟时刻”，也就是说，如果你训练得足够久，模型的思维链能够做出像“哦，这里有一个我可以标记的顿悟时刻”这样的行为。他们认为这非常有趣。但对我来说，这两个现象是否真的在很多方面都特别令人印象深刻，其实还不清楚。我们现在知道，更长的思维链可以说只是GRPO算法长度归一化的一个自然副产品。而其他人展示的那个“灵光一现”时刻，其实在基础模型里就已经出现了。所以很明显，这不可能仅仅是RL算法的结果。很多人说“啊哈，我可以把它用作解决数学问题的一部分”。这个模型在预训练阶段就已经学到了这一点。

[31:10.900] 在RL过程中，这恰好被激发出来，其实完全不令人惊讶，因为它生成了大量的数学token。所以我认为，那个被大肆宣传的现象并不特别突出或令人兴奋，但我确实认为R1是一个非常重要的里程碑，因为它凸显了RLVR可以有多简单。从某种意义上说，无论是作为一种算法，还是作为解决这些极其困难问题的方法，这都非常简洁。所以R1 Zero是一个非常漂亮、干净的受控环境。然后R1是他们试图将这个系统产品化的尝试。我认为R1有一点很好，稍后我也会展示Qwen的对应版本，那就是你可以看到他们是如何把所有组件堆叠在一起的。

[31:55.700] 我想我们已经分别研究了所有组件。我们知道预训练是什么，我们知道有RLHF，我们知道SFT，但我们要如何组合所有这些组件，最终得到一个真正的系统呢？通常你做的事情大概是这样的：我们有我们训练好的中间模型，我们要做一堆推理训练，也许中间还会做一些长上下文扩展。然后我们最后要做RLHF，因为从某种意义上说，这是最面向用户的部分。我们要确保格式和所有这些东西都很完美。RLHF在这方面可以发挥很大作用。他们做的另一件事是，为他们的生产模型添加了一个针对思维链的语言一致性奖励。如果我没记错报告内容的话，这是因为在没有一致性奖励的情况下，R1 Zero式的训练会导致思维链中出现语言切换，他们认为这非常难以解释，甚至有点令人不安。所以他们希望模型只输出一种一致的语言。

[32:51.590] 他们这样做纯粹是出于可解释性的原因。然后他们也有一些不可验证的奖励，被混入GRPO中，几乎像是融合到了RLHF过程中，但除此之外，这和其他流程非常非常相似。整个流程，我认为，非常直接了当。关于SFT，他们做的事情很简单。在R1 Zero中，他们完全不想做任何SFT。但在R1中，他们愿意使用长思维链数据。当你阅读这些开源技术报告时，试着解读字里行间的意思会很有趣。当他们说“我们构建并收集了一小部分长思维链数据”时，你可能会想——我想知道这是不是从其他模型中蒸馏出来的。这对开源模型来说是一件特别消极的事情。现在每个人都在这么做，但我只是觉得他们非常谨慎地写着“收集一小部分长思维链数据”这件事很有趣。这些数据被用来微调模型。

[33:52.460] 我们知道，对于一个非常好的基础模型，仅仅在长思维链上进行SFT就能解锁大量O1风格的能力。而这是RL的一个很好的起点。所以他们从那里开始，然后加入一些验证来过滤掉思维链。接着他们继续进行后续的处理流程。我认为自那以后的许多论文，其中包括一些我学生的论文，已经表明，如果你有正确的蒸馏流程，并且你有合适的基础模型，你基本上可以仅通过SFT就获得大部分长思维链推理的能力。我认为这可能很有趣，因为我认为其中一个仍未解决的问题是，对于其中某些能力，你是否真的需要RL。我想也许思考RL在语言建模中作用的一种方式是，RL是一个极好的监督来源。

[34:43.830] 如果你在解决前沿数学问题，你就是没有足够的监督来获得详细的长思维链，而RL允许你自行生成这些数据。但一旦有人生成了这些长思维链，你也有可能通过模仿来学习。这似乎正是许多这类蒸馏结果所部分展示的。所以R1的RL部分基本上和R1 Zero相同。除去我提到的语言一致性损失，基本上就是使用GRPO运行RL训练。这就引向了最终的阶段，也就是我们上一讲学到的东西。所以我们做了一些基本的指令微调风格的SFT，然后是RLHF。对于不可验证的任务，他们基本上使用了与DeepSeek V3相同的方法。所以这里真的毫无意外。效果如何？真的非常非常好。我认为这就是为什么当时有那么多关于DeepSeek的恐慌或广泛关注，因为R1确实是一个非常优秀的模型。

[35:48.380] 它在许多类别上击败了O1。它匹配了人们预期的许多测试时扩展行为。而且它来自一个非常简单的方案。所以很容易理解这些提升是从哪里来的。最后一点，与我之前提到的蒸馏讨论有关，你可以拿R1的思维链，把它放入Qwen 2.5中，如果你这样做，你真的可以显著提升这些模型的性能。在某些情况下，可以匹配许多专门的思考模型。所以换句话说，真的，如果你能获得这些模型能够理解并使用的正确的长思维链，通常就有可能让它们进行长时间的推理，因为基础模型已经出奇地好了，而且这甚至对Llama模型也有效，我认为这非常引人入胜。所以，作为DeepSeek故事的结尾，我非常喜欢DeepSeek的许多技术报告，因为它们通常既包含消融实验，也会告诉你那些不成功的尝试。

[36:59.680] 如果你一直在关注他们的工作，DeepSeekMath有所有这些过程奖励模型相关的内容。就像是我们必须验证所有步骤，然后你读到R1时会说，过程奖励模型去哪儿了？嗯，他们告诉了你过程奖励模型的去向，那就是我们试图让过程奖励模型起作用。但它们对我们来说没什么用。事实证明，结果奖励模型非常好，而且已经足够好了。而且你可以更好地扩展这些模型的数据规模。我认为关于PRMs的一个大争论是，你从哪里获得这些逐步的评分标准。这很难规模化。我认为在这一点上，现在已经很清楚了，结果奖励模型非常非常好。这也是当前研究和应用的主要方向。同样，如果你在最初O1发布时在场，很多人都在猜测OpenAI的O1内部到底在做什么。

[37:52.420] 他们会说，哦，他们在用PRMs吗？他们在做像AlphaGo那样的树搜索吗？他们尝试了很多MCTS，然后他们描述说我们无法让它很好地工作。我非常想强调一个事实，那就是他们非常坦诚地公开了所有这些探索以及他们尝试过但不成功的事情，而不是说他们根本没试过。很好。好的。如果我要谈——或者，实际上，我在这里暂停一下，以防有人有问题，虽然我不知道——嗯。[听不清]你应该解释一下正奖励的基础，因为[听不清]我以为你说对于负奖励，我们鼓励除以无穷大，而对于正奖励则是[听不清]0。是的。所以问题是关于在正奖励的情况下，长度归一化是做什么的。当然，在那些情况下，长度归一化会鼓励缩短思维链，如果你想节省推理成本，这是好的；但如果这会降低准确性，那就不好了。

[38:55.140] 但我认为主要问题在于，对于负面案例，你会得到一些非常长的回答，而正面案例则没法压缩太多，因为要解决特定问题，思维链（COT）长度是有下限的。我想我是在试图理解这个图所对应的解释。是不是这样——我觉得 Doctor GRPO 那篇论文里的图更清晰一些。在 R1 的图中，这些都是聚合在一起的。这是他们评估中所有的回答，不仅仅是正面案例。但如果我们看 Doctor GRPO 的图，这里是平均输出长度，这是错误输出的长度。你会看到正确输出长度在这里。所以你会发现，这实际上主要是由错误案例驱动的，正如我们从解释中预期的那样。所以这个故事，至少在这些受控对比中，是相当干净和清晰的。

[39:58.250] 是的，好。那么现在，我还要讨论另一种方法，Kimi K1.5，或者说另一篇论文和另一种方法。而且我总觉得，当我谈到 R1 时，我有义务也谈谈 Kimi K1.5，因为它们和 R1 是同时出现的。它们也击败了 R1，然而人人都只关注 DeepSeek，Kimi 在讨论中并不那么经常被提及，尽管他们的模型非常出色。而且我并不是出于同情才提他们。我认为 Kimi 有一个很好的地方，那就是他们做的事情——在某些方面——与 DeepSeek 有很大不同，而我们可以从这两者都有效这个事实中学到很多东西。我认为这是本课程的主题之一，即通过阅读所有这些技术报告并观察更广泛的模式，我们可以大致开始理解什么是让这些算法生效的可行或容易的空间。他们实际上在数据集的构建和强化学习（RL）过程的课程生成方面提供了更多细节，很多人都说

[41:08.440] 这相当重要。而且他们还有一种不同的 RL 算法，从某些方面来说，直觉上有些相似，但并不完全是 GRPO。所以这是另一个很好的验证，从技术上讲，我认为你未必需要像 GRPO 这样的东西才能让这些方法奏效。所以我想先讲的第一部分是数据。数据对所有这些过程仍然非常重要。而对于 RL，还有一个额外的复杂性，那就是课程安排，你想加入什么难度的题目？在监督微调（SFT）中，你不会真正考虑难度，因为你只是把数据塞进去。不管你用什么样的数据，你只管做预训练损失，模型被迫从这些数据中学习。在 SFT 中，情况要稍微微妙一些。我们讨论了很多关于幻觉的问题，也许有时候你并不想在某些数据上训练。

[42:02.830] 但对于 RL，还有另一个额外的复杂性：如果你的题目太难，你就得不到任何奖励。如果你得不到奖励，你就没有信号。如果你没有信号，你就无法学习。所以拥有广泛覆盖的正确类型样本非常重要。所以 Kimi 团队的做法是，首先，他们试图获得整个广泛覆盖范围的数据。不出意料，他们排除了选择题，因为这在很多不同领域都有涉及。他们想要的是需要长篇深入思考的内容。而他们的论点是，也许选择题不具备这一点。但也许最后这一点我认为可能是最重要的想法。这一点出现在很多不同的 RL 论文中。你基本上是根据一个 best-of-k 过滤器来筛选所有的样本，或者说抱歉，你是根据它来过滤的。

[42:52.800] 所以这里你拿一些样本，如果它们在 best of 8 中成功了，也就是说，如果你采样八次，模型至少能成功一次，那么这不是一个好的 RL 样本，因为它已经处于模型能力的边缘了。你不会从中真正学到什么新东西。因此你只关注那些基本上通不过这个测试的样本，因为这让我们能够挑选出正确类型的题目。这样也节省了计算资源，让你可以完全跳过某些题目。你还可以在两端都进行筛选，以获得既不太难也不太容易的题目。我认为研究界的普遍共识是，如果你真正想要的是让 RL 稳步推进，那么进行这种中等难度范围的过滤是非常好的。和 DeepSeek 一样，没有关于 SFT 的描述。我们可以推测那是什么，但我们没有具体的信息。

[43:53.098] 所以我觉得 Kimi 有趣的地方在于，他们有一个受 DPO 启发的论证，但最终得到的结果和 GRPO 非常相似。而我认为，他们最终落在相似的位置这一事实，暗示了如果你在设计一个新的 RL 算法，比如说，这些组件中的哪些可能是有用的。所以他们从和其他人完全一样的地方开始。这对你们来说应该看起来很熟悉了。它就是在我当前策略下最大化期望奖励。然后在右边有一个 KL 正则化项，让我保持在基础策略附近，或者甚至可能是之前迭代的策略附近，取决于你打算怎么做这个 RL。所以他们要遵循同样的 DPO 式推导，他们说，假设我可以解析地最大化，然后我解出能够实现或对应那种最大化的奖励模型。这会给我这个策略比率。

[44:51.360] 我要把这个代回上面的目标函数，尝试求解正在发生的事情。他们现在说的是，他们想尝试匹配奖励模型。而为了让这个等式成立——他们希望这个等式成立，这在最小值点上是成立的。因此，这是一个很大的启发式。如果某样东西在最小值点上是相等的，为什么不直接在该项上放一个平方损失并最小化那个平方损失呢？我觉得搞优化的人看到这个会吓坏的，但我认为从某些方面来说，这是一个完全合理的直觉。我们知道，当模型良好的时候，这两者应该是接近的。所以把这两者显式地拉近作为一个替代目标，也许也是可以的。正如你看到的，这个推导和 GRPO 完全不同。我们不是通过 PPO 的路径来推导这个更新。

[45:45.030] 但是现在，实际上一看，你对这个 L 求梯度，最终得到的东西看起来惊人地像加了正则化项的 GRPO。你最终得到的是这个带基线 R bar 的策略梯度。而这个 R bar 实际上是每个条件下这些 x 的平均值。你还有这个额外的正则化项，本质上就是会出现在 GRPO 中的 KL 项。所以从某种意义上说，我们通过完全不同的途径重新发明了组均值归一化基线。而 Kimi 团队，我认为在某种程度上，对长度问题有一个更好或更清晰的观点。我觉得 GRPO 团队的态度有点像，长度在失控地增长，这难道不好吗？我确定我们的模型变得更聪明了。这稍微有点不厚道的解读，但当你把这个图作为积极的东西来展示时，其隐含的意思是，我们的模型思考时间更长，这很棒。

[46:51.540] 但 Kimi 团队审视这个长思维链问题时会说，如果我们可能产生很长的思维链，那是非常浪费的。我们其实不想要长思维链，因为思维链要花推理成本。所以他们不仅没有长度问题，因为如果你看这个目标函数，他们不按序列长度归一化，而且他们实际上还想更进一步。他们想要压缩回复的长度。这是我们在语言模型发展中普遍看到的另一个重要特征。我们希望思维链尽可能短。我们想解决难题，但我们不想思考很长时间，因为当我们思考很长时间时，我们本质上是在补贴用户。如果你是 OpenAI，你的用户订阅了 200 美元的 Pro 计划，而你的模型一次要思考一个小时，那你的处境就不太妙了。但如果你的模型一次只思考五分钟，那你的处境就很好。所以出于所有这些原因，Kimi 想要压缩思维链。

[47:54.600] 那怎么做呢？最简单的方法就是你设计一个启发式的长度奖励，然后直接把它加到强化学习过程中。这个奖励其实相当复杂。他们做的是试图让较长的序列变得越来越短。正确的答案也应该简短。这里有趣的平衡点在于，如果你让错误答案变得太短，它们就再也没有办法恢复过来了。让我给你举个例子。假设我不擅长几何。AI 不擅长几何，我得到很多错误答案，而惩罚让我的几何思维链变得非常短。现在我的几何思维链是 0。我非常不擅长几何。我永远无法从中恢复。我再也得不到任何正的几何奖励了。我就卡在那里了。所以你要做的是，你并不强制错误答案变得超级短，而是激励它们比平均水平稍微短一点点，这样那些错误的内容就不会无限制地增长。

[49:02.190] 最终这产生了一系列非常合理的结果。所以正如我之前所说，他们在数据集策展方面有一套非常好的强化学习视角。他们不仅有经过筛选的数据集，还会观察他们能够解决特定问题的成功率。一旦模型掌握了一个特定的问题，这个问题就会被从问题集中移除。这样你就节省了一些计算资源。我觉得基本上每个做强化学习的人都会做这种成功率过滤，既是为了避免浪费计算资源，也是为了不在对模型来说太难的问题上浪费时间。他们基本上有这样的奖励机制：对于代码，他们会拿标准答案解决方案并生成一些新的测试用例；对于数学，他们实际上有一个奖励模型来检查答案等价性。这在某种程度上其实挺有趣的，因为我们这堂课一开始说我们要研究形式化数学之类的东西，

[50:04.160] 某种真正可验证的东西，你可以让编译器来检查你数学的正确性。然后我们经历了这节课的大部分内容。最后我们到了哪里？好吧，我们最终得到了一个奖励模型，一个检查数学答案正确性的奖励模型。你会在作业中再次看到为什么，或者如果你玩玩作业，你就会明白为什么。你很快就会发现答案等价性检查是一个非常困难的问题。因为在数学中，你可以用很多种方式写出等价的东西。不仅如此，语言模型也可以用很多种方式给出答案。即使你提示它用 LaTeX 的 boxed 格式给出答案，也许有时候它会跳过 boxed，也许会在 box 里加一些额外的东西。有很多种方式让模型因为严格的正确性检查器而得不到正确答案，即使它已经有了正确的思路。

[50:57.090] 所以出于这些原因，基本上大多数强化学习项目都有一个非常复杂的答案检查器，要么是正则表达式，要么是模型，或者谁知道是什么，来避免这类问题。要让 RLVR 的验证部分真正可靠，确实是个深不见底的兔子洞。好了。最后一点，它和我们之前讨论过的一些系统部分以及推理部分相关，那就是 RL 优化。RL 基础设施确实非常难。训练很难，推理也很难。而 RL 把两者放在一起了。所以某种意义上，难怪它如此糟糕和困难。更糟糕的是，我觉得有一个你一开始不会意识到的关于让 RL 高效有多难的事情，那就是想想以下场景。想象一下，你有了你的 rollout，但你有一个非常难的数学问题，比如说其中一个是黎曼猜想。你的模型在黎曼猜想上真的很吃力地运行着。它生成了一个巨大的思维链。现在发生了什么？与此同时，如果你在做朴素的推理，其他所有人都在等着这一个 rollout

[52:09.373] 完成，才能进入下一个阶段。所以如果你在做这些批量操作，长的 rollout 真的会拖累你。所以 RL 确实需要处理所有这些非常棘手的底层细节。你需要处理一些思维链可能变得很长这个事实。你需要截断它们吗？你需要以某种方式把它们放到另一台机器上吗？谁知道呢？这些都是你可以做的决定。如果你在训练和 rollout 之间切换，你 rollout 一次然后训练，再 rollout 再训练，你必须在某种程度上处理这个问题。要么你的一些机器是纯 rollout 机器，另一些是训练机器，要么你不停地切换框架。这两种方式都非常昂贵。最后，你面临一个非常困难和可怕的权衡，那就是在策略（on-policy）的东西在数学上和训练动态上都非常好。

[53:00.710] 所以 GRPO 在它简单的在策略形式下表现得非常好。你会在作业中体验到这一点，然后你就会变得贪心。你会说，但我的系统利用率太低了。如果我能复用我的 rollout，我就能做得更好，然后我就能让推理和计算重叠起来，做各种巧妙的事情。所以你会尝试复用这些 rollout。那就会导致离策略（off-policy）问题，进而导致训练不稳定以及各种非常难处理的事情。所以这些都是最终变得非常非常棘手的事情。所以我觉得现在大多数开源技术报告都会有一节专门讲他们的 RL 基础设施，他们的 RL 基础设施需要同时包含训练部分，就是这里的蓝色框，以及推理部分，就是右边的绿色部分。

[53:55.340] 你还需要把权重从训练部分转移到推理部分。所以你需要一些方案来处理如何移动这些权重。你需要紧密地协调它们。在某些情况下，也许它们甚至共享同一批机器，因为当推理在运行的时候，训练那边可能处于空闲状态。所有这些都是复杂性所在。你有很多非常复杂的系统工作需要做。我们已经知道 Kimi 打败了 OpenAI O1，但我们也看到一些非常好的现象，表明随着强化学习的进行，你能够思考更长时间，性能也在提升。但在某些情况下，并不是说我们在无限制地增加 token。我们最终会看到一些情况，我觉得 OmniMath 就是一个很好的例子。你思考的时间并没有长很多，但性能持续上升。所以也许这是一个很好的例子，说明之前的长度控制机制发挥了作用，产生了好的效果。

[54:51.410] 我最后想谈的是，你可能会想，这种真正的强化学习东西是否比仅仅在正确答案上训练要好得多。这被称为专家迭代（expert iteration）。它在过去很多论文中表现得非常好。在你处理非常不稳定的情况时，你可能实际上更倾向于使用专家迭代。Kimi K1 做了非常大规模的消融实验，表明这类强化学习方法始终一致地优于专家迭代（expert iteration）。所以在这里就是橙色线压过蓝色线。因此，如果你想榨干模型所有的性能，你就真的无法回避强化学习。我可以在这里暂停一下，以防有人对 Kimi 的内容有疑问。然后我会继续讲三篇中的最后一篇，也就是 Qwen 3。好的，没问题。那么，我想讲的最后一篇是 Qwen 3，还有 Qwen 3.5-Next-Coder。这些名字变得越来越复杂了。

[55:51.680] Qwen 3 发布得稍晚一些。但他们实际上有非常有趣的规模化（scaling）和数据方面的结果。所以我想谈谈这些。而这个的继任者，也就是 Qwen 3.5-Next-Coder，之所以有趣，是因为我认为它包含了，我可以说，在各大技术报告中关于智能体强化学习验证（agentic RLVR）训练最详细的细节，而且他们做了很多很酷的事情。所以再次强调，和 DeepSeek 一样，我们在 Qwen 的报告中看到的一个好处是所有这些内容组织的完整方式。这和 DeepSeek 的结构非常相似。我们从基座模型（base model）到基本上做监督微调（SFT），然后是推理强化学习（reasoning RL）。然后他们有这个思考模式融合（thinking mode fusion）。

[56:35.410] 我稍后会谈到这个。然后他们做基于人类反馈的强化学习（RLHF），接着就有了一个可以发布的模型。但是当然，他们不想直接对外服务这个模型。所以他们通过蒸馏（distillation）来获得他们更小的模型。我的意思是，Qwen 3 是一个相当值得尊敬的开源模型。你基本上可以把这作为你脑海中关于前沿语言模型是如何构建的图景，把所有组件放在一起。我认为你会对正在发生的事情有一个相当合理的认识。而 Qwen 基本上使用了，在这一点上，已经是经过尝试和验证的 RLVR 剧本。我认为他们吸取了 Kimi 和 DeepSeek 的许多最佳部分，并且他们做对了。他们做了大量关于难度的过滤，因为我们知道这能节省计算量。这是获得好数据的好方法。他们还做了其他事情，比如他们移除了模型无需思维链（COT）就能答对的那些内容，因为我们知道那不是一个思考问题。

[57:30.340] 他们移除了与验证数据过于相似的内容，试图进行数据去污染（decontaminate）。然后他们对参考思维链（reference CoTs）做了一点人工过滤。也许关于 Qwen 3 最值得注意的是，他们实际上只在非常少的样本上做强化学习，仅仅 4000 个样本。但同样，如果你的流程其余部分是正确的，你实际上可以取得令人惊讶的进展。所以 RLVR 组件的核心也许并不太令人惊讶，因为他们确实是在 DeepSeek R1 和 Kimi 的基础上构建的。有一些有趣的 Qwen 3 特有的东西，其中一些他们最终放弃了，但我认为其中一些想法

[58:15.500] 实际上相当有趣。那么他们做的一件事就是用标签混合思考和无需思考的东西。这就是他们最终拥有思考模式的方式，在这种模式下，模型有一个很长的思维链，他们可以将其与无需思考的模式混合在一起。所以即时响应模型和长思维链模型基本上都存在于同一个模型中。这在很多情况下是不成立的。即使在 OpenAI，也常常有一个思考模式和一个非思考模型。他们还有——我觉得这相当有趣——他们有一种提前退出思考的方法，如果他们附加一个特殊的字符串，他们会立即停止思维链，模型被迫对给定的任何提示给出答案。我们甚至可以在像 ChatGPT 这样的界面中看到这种功能，例如，你可以提前终止思考并尝试立即得到一个答案。但是我们从这里看到的是，我一直认为这是一个令人惊讶和有趣的事情，那就是当你改变思考预算（thinking budget），使用这种提前终止的技巧，你最终会

[59:25.000] 发现模型的性能会优雅地下降，即使在较低的思考预算下，这些模型有点在思考中途被截断，它们仍然能够在那个点给出令人惊讶的合理回应。而且我们一致地看到，即使思考预算非常小，思考模式模型在所有这些数学或编程任务上的表现，都比即时响应模式要好得多，后者更像是经典的指令跟随加上 RLHF 模型。所以他们为我们做的一件好事是，审视所有不同组件对性能的贡献。在很多方面，这不应该让你太过惊讶。如果你做这个推理强化学习，然后做通用强化学习，你能够在所有不同的任务上慢慢获得改进，比如 Arena-Hard 或 CounterFactQA。这些是你希望进行更常规强化学习（RF）的通用任务，我们看到这些性能在各个方面都有显著提升。

[01:00:34.350] 我们在数学和编程方面看到了一些性能下降，因为我们把非思考组件融合在一起，但这种下降并不是太严重。我认为，实际上，尽管在绝对数值上这些数字在后续版本中很小，但我想在一些 Qwen 3.5 中，他们已经放弃将思考和无需思考融合到单个模型中了。我认为他们——他们过去称这些为混合模型（hybrid models），因为他们发现这种性能下降是不可接受的。他们想要在思考模式上榨取所有可能的性能。所以我认为他们现在将这些模型彼此分开了。我要讲的最后一个话题和论文是 Qwen 3——抱歉，我把这两个搞反了。是 Coder-Next 而不是 Next-Coder，这是最新发布的模型之一。我认为如果你对通用的智能体（agent）内容感兴趣，当然，还有其他的智能体相关的事情需要考虑，比如框架（harnesses）。但如果你对如何真正构建一个智能体感兴趣，

[01:01:35.280] 这可能是一份值得去读的报告。Qwen3-Coder-Next 是一份很棒的智能体后训练（post-training）论文。但在某些方面，智能体的后训练与我们描述的所有内容并没有太大不同。并不是说有一种新的智能体训练算法。实际上，你应该在整个课程中内化这一课。数据才是重要的东西。所以如果你要训练一个智能体或对智能体进行后训练，你要做什么？你要把数据准备好。所以数据有两个部分。为了获得大量的能力，你不能只是在最后才把它们注入进去。你可能需要更早一点开始。所以有一个广泛的中期训练（mid-training）阶段，试图尽可能多地将编程和智能体类的能力注入到模型中。所以他们做的事情包括，他们会拿代码仓库，将仓库中的文件拼接起来，生成这些非常长的上下文数据，并将其放入模型中。

[01:02:30.170] 我们知道模型最终会看到这些非常长的思维链轨迹（COT traces），其中智能体可能打开了一堆文件，这是非常有用的预训练风格的数据。他们会拿拉取请求（pull requests），然后尝试为其构建合成上下文，使用可能有助于理解拉取请求的检索增强生成（RAG）。所有这些都会进入中期训练。他们会获取那些他们用自动化方法检测为包含文本和代码的文档，然后使用大语言模型（LLM）将其转换为漂亮的 Markdown 格式。把这些也都加进去。最后，他们会有一个语言模型，在编码相关的网页文档上谈论编码，以生成编码风格的合成数据，

[01:03:14.440] 然后他们会拿公开可用的编码智能体，在各种环境中运行它们，我稍后会描述这些环境。这些运行过程的轨迹也都会进入中期训练。他们也会做一些相当标准的处理，比如指令跟随，而且他们还有一个额外的任务，就是中间填充（filling in the middle），我认为这与我们的关联不是特别大。如果你在写代码的时候，想要补全一个片段中间的部分，拥有这种能力会很有用。所以他们会把一部分这类数据放进中期训练里。但除此之外，这是一个非常标准但深入的数据收集流程，目的是把一些智能体（agentic）能力注入到中期训练中。

[01:03:50.360] 然后通义千问（Qwen）团队做了一件我不确定以前有没有见过的事。这实际上相当有意思。他们会拿中期训练好的 Qwen3 模型，然后实际上只是训练一大堆不同的专家模型，用于不同类型的编码相关任务。他们最终会得到四个不同的智能体专家，然后他们只是把这些专家全部蒸馏回同一个模型里。我只能猜测他们为什么这么做。我不认为我在其他工作里见过这种做法。我已知最接近的是 DeepSeek V3，还是 V3？V3.5，抱歉，DeepSeek V3.5 或者 3.2。也许是 3.2。抱歉，3.2，我认为，他们做了类似的事情，就是有不同的专家，它们唯一的任务就是格式化并处理数据，所以他们有数据处理专家，为完整训练生成数据。但我不认为我见过这种生成不同模型专家然后蒸馏回去的做法。

[01:04:49.490] 这实际上更接近一些学术工作，比如分支训练合并（branch train merge）之类的。这些确实是人们会做的事情。我在前沿模型训练中见得不多。话说回来，这些专家每一个实际上都很有趣，他们做的事情基本上就是在他们定义的一大堆不同子任务上进行完整的强化学习（RL）和/或有监督微调（SFT）训练。所以他们有一个网页开发专家，他们在有效的网页代码上做 SFT，基于不同类型的检查。他们会构建用户体验（UX）专家，训练它使用许多不同的工具格式，他们还会有 QA。所以 QA 智能体只是用更多针对代码的合成数据来训练。

[01:05:31.860] 但也许对我们来说最有趣、也最复杂的是他们的软件工程智能体。他们做的事情是大规模构建智能体环境。SWE-bench 是这类事情的黄金标准。所以他们想要的是比 SWE-bench 更多的东西。他们走出去，用各种基于 GitHub 的自动化方式来生成大量不同的问题。基于这一点，他们基本上想在这些环境上做强化学习，这在某些方面非常不令人意外。他们得到了什么？他们能让强化学习性能上升。这里我要说一件事，这在我们讨论的更广泛背景下非常重要，我们之所以能把越来越多的算力投入到强化学习中，是因为我们相信我们的奖励模型是不可被攻破的，或者说很难被攻破。如果这个假设不成立，你的强化学习方法会找到越来越隐蔽的方式来欺骗你、损害你的性能。

[01:06:35.450] 所以他们有一个很好的图，基本上，在 Git 里，你可以查看未来的提交或者不同的提交。如果你有一个 issue，而且存在未来的提交，你可以直接查到这个修复是什么。这对模型来说是一个非常容易学会的作弊手段。所以他们基本上有整整一个奖励函数，它的唯一目的就是防止智能体篡改 Git 历史。如果你不这么做，你最终会得到一个像右边那样的图，它在学习、学习、学习、学习、学习。然后突然间，你看到这个突现的跳跃，而这个突现的跳跃实际上是它学会了操纵 Git 调用来获取历史记录。在某些情况下，它甚至能绕过你给它的某些约束。如果你告诉它不能使用 Git log，它可能会添加一个远程（remote）比如 origin，然后向这个远程查询某些提交中发生了什么。所以有各种各样的作弊方式。

[01:07:32.270] RLVR 的稳健性完全取决于你的奖励函数，而你的奖励函数有时候可能一点都不稳健。顺便说一句，我和我的一个学生曾经在做一个项目，类似于在 Lean 上做强化学习。Lean 是一种形式化数学、可验证的语言。我们当时天真地以为，这个绝对不可能出问题。很多人在做 Lean 相关的工作。Lean 编译器是坚不可摧的。结果发现 Lean 编译器并不是对抗性稳健的。有些字符串你可以放进去，它们会允许你在某些模式下验证本不应该被验证的证明。所以我认为可验证奖励这个概念实际上比你们很多人最初可能以为的要棘手得多。不管怎样，你做了这个大规模构建 GitHub 仓库的过程。你经历了做强化学习的过程，你看到分数缓慢而稳定地上升，最终你会得到一个出乎意料地好的系统，一个实现了

[01:08:34.740] 大约 70.6% SWE-bench 准确率的模型，即使你拥有的基本上是一个非常小的模型，一个 30 亿活跃参数的模型，你也能在这些任务上表现得非常好。不过在某种程度上，我不确定这是否超级令人惊讶。强化学习，当然，你会在你训练过的环境上表现良好。你甚至可能像我这里展示的那样，在你的验证集上获得泛化。但你总是需要小心一点地比较性能。特定任务的性能并不一定意味着它能够泛化到更广泛的领域。好的。这就是我关于 RLVR 这堂课要讲的全部内容。记住这堂课的核心要点是，它真的完全围绕着强化学习的奖励。RLHF 和 RLVR 可以说它们是非常相似的问题。但区别实际上只是我们想要更难以被攻破的奖励，这样我们才能真正投入更多的算力，让这些系统变得更好。

[01:09:39.330] GRPO，至少对于研究社区来说，是真正推动了这一切的东西。你们都应该非常熟悉 GRPO。你们应该知道它的函数形式是什么样的，以及更新是怎么做的。这是你们应该像预训练损失一样熟知的东西。然后最后，在这一点上，我认为很多人已经知道怎么做 RLVR 了。正如你们会发现的那样，不幸的是，强化学习仍然非常挑剔、有噪声，用起来很痛苦。但它并不是那么难。它不像过去在各种非常棘手的环境中做 PPO 那样。它实际上比你可能想象的顺畅得多。我先在这里停一下，在让大家离开之前，看有没有任何最后的问题，以防有人有疑问。一个很快的问题——当你提到模型开和关思考模式的时候，这个差异是在模型上还是[听不清]一样？

[01:10:33.140] 还是说他们在提示上做了手脚，或者他们有不同的底层模型来切换？是的。所以问题是关于思考模式的，以及幕后发生了什么。关于思考模式有趣的一点是，它实际上就是一个模型。只是有一个小小的提示标签，把它们在长和短之间切换，比如长思维链（COT）和几乎无思维链模式。而你如果只是有一个 API 标志位之类的东西，那并不难。他们能把这两者放在一起才是有趣的地方。所以你可以实际上选择，[听不清] 但然后你自己加额外的提示来做思维链？是的，或者说在这种情况下，控制机制是在提示层面，而不是在 API 或服务层之类的地方。

[01:11:13.270] 是的。太好了。哦，是的。你提到了中期训练的阶段。我想知道，进入中期训练过程的格式化工序有多大程度影响了 RLVR 后训练期间能学到什么；然后是作为这个问题的第二个部分，如果你在中期训练中没有正确的数据，那么是否可以认为强化学习无法抽取一个正确解，因此你无法学习它？是的。所以这个问题是关于中段训练在强化学习中的角色。我认为这里有很多经验性的微妙之处。我没有一个放之四海而皆准的答案。它肯定不会给出一个“绝对不行”的回答，但为了稍微说明一下细微差别，我认为预训练和SFT承担了大部分重活。只要你有覆盖度，如果预训练根本没有任何代码数据，那你就麻烦了。你需要训练。但如果你的预训练非常多样化，并且覆盖了——我们回到中段训练那张幻灯片——如果它覆盖了很多内容，比如文本、代码数据，还涵盖了大量GitHub数据，那么这非常有用，但可能不是关键。

[01:12:17.820] 因为你反正都会做SFT，而SFT能让你足够接近，从而开始从强化学习中获得一些奖励。如果你没有SFT，那你就陷入了DeepSeek那样的困境。但事实上，我们总是在强化学习之前做一定量的SFT，这真的对你很有帮助，而且预训练有足够的覆盖度，希望你能获得一些合理的泛化能力。所以中段训练是非常有用的。对获得稍微更好的泛化能力非常重要，但我觉得不一定是成败攸关的。是的。[听不清]在你的迁移模型训练之后还有一步。我们最后仍然会跳回单一模型。那么这是不是——这种蒸馏是怎么运作的？你需要设计一个混合流程吗？是的，没错。所以蒸馏过程需要你写下一系列的提示词，专家们将在这些提示词上被蒸馏进最终模型。

[01:13:15.110] 是的。我绝对能看到这种方法的好处，那就是你可以让一个独立的团队负责每个专家。让团队分工处理这个要容易得多。而且如果你有足够的算力，聚合过程也可能很简单。只是这个——如果你有所有的目标，你不如直接把它们都扔进大型训练循环里。通常，我认为这是你更愿意做的事情。这样你就不必处理蒸馏这种复杂情况。你可以只有一个大的训练目标，然后在一个训练循环里完成。是的。长推理训练也是中段训练的一部分吗？所以问题是，长推理训练是不是中段训练的一部分。我认为R1和Kimi K5.1都有长COT的SFT。长COT传统上并不属于中段训练本身，但它通常——长COT这类数据会被用于长上下文扩展。

[01:14:13.605] 所以这里稍微微妙一些。我完全没有讲到长上下文扩展，对此我感到遗憾。但通常，那是在RLHF或其他事情之前的一个额外阶段，在这种情况下也是长COT，你使用手头有的任何长上下文数据来扩展上下文。通常，它是书籍、代码、合成数据的组合，因为这些东西的长度足以进行扩展。是的。所以我觉得跟当前这张图相关的是，当你在做这个[听不清]过程时，比如说有多少数学、化学以及很多其他领域。通常会并行进行吗？比如一次强化学习训练运行？还是按顺序进行？如果按顺序进行，你怎么避免遗忘学习的早期部分？是的，我认为最接近这个“顺序还是并行”问题答案的，就是这些图，你做的两个划分是在推理问题和非推理问题之间划分，所有推理问题都放进一个桶里，在第二阶段执行。然后非推理问题在最终的RLHF阶段处理。这包括诸如“健谈度”之类的东西，等等。

[01:15:30.708] 那些都放进后面的那部分，也就是第四阶段。好的。非常感谢。下周见。[掌声]

CS336 2026 · Lecture 16 · Post-training - RLVR

视频：官方视频
字幕：en-US · manual captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：原语言字幕稿，待校对

正文

[00:05.000] OK. Maybe we shall get started. This is the second of the post-training lectures, so we're going to talk about the exciting developments in RLVR or reinforcement learning from verifiable rewards. And I think this is in some ways very timely. I didn't get to update the slide, but there was just a announcement by OpenAI folks that they had solved one of the major problems. So that's a MyOpenMath problem using one of their thinking models. How does that thing work? That's exactly this thinking model stuff or RLVR stuff. And so thus far, where we're at is we are at the left side here. After we finished last lecture, we've got instruction tuning, we've got RLHF, we've got some good stuff that's ChatGPT.

[00:56.480] And what remains is the modern developments in thinking models. The model ability to do very long COT and solve these hard, verifiable problems like mathematics and coding in some cases as well. And remember that I ended last lecture with this a little bit of a downer note. The RLHF wasn't really going to get us where we wanted to go because of this problem called overoptimization. So we're going to go and collect a bunch of preference data. Preference data is very useful. We will build some reward model and then we will RL against that reward model. But this thing is fundamentally very annotation bottleneck because we have a reward model and you can't keep putting compute into the same reward model, eventually you're going to overfit your reward model. And no matter how good of a job you do at regularizing, eventually you're going to run into this problem with overfitting.

[01:49.990] And so really, this is very depressing. And it's not really the potential of reinforcement learning. If you're a big reinforcement learning believer, you're going to look at all things like AlphaGo and so on where you say, well, in these domains, RL really did an amazing job. What is the difference between what we're doing in RLHF and what has been achieved in these more RL native or RL natural domains? And really the difference, I would say, or one major difference between something like AlphaGo and something like RLHF is that in AlphaGo, we are optimizing exactly what we want. You get the win-loss conditions of the game of Go. You don't have any sloppiness to that definition. And so you can just put in as much compute as you want. And as long as the objective improves, you're doing well. In some sense, these are search problems.

[02:40.000] Whereas the top one is much more of a learning problem. Not quite a precise distinction, but that's one way of thinking about it. So now there might be other domains like formal mathematics or even natural language mathematics, that have this flavor of being more verifiable and therefore much more amenable to reinforcement learning. So that's going to be the motivation for why we're going to study some of these. And the algorithms aren't going to be that different fundamentally, but where we will end up will actually be surprisingly different. And so the lecture is going to be in two parts today. This mirrors a lot of how we've structured the course in other parts. So in the first part of the lecture, I'm going to talk about the core algorithms. This is like basic knowledge for you all. You will use some of this for your assignments. And so I'm going to talk about things what is GRPO, how does PPO work.

[03:29.130] And then after I've done that, I'm going to dig through a bunch of open source releases and show you how, what I've taught you in the first part of reflects in the technical reports of many of these major open source model releases. So we're going to dive in to the methods and basic section. And the first thing that I'm going to start with is PPO. We can't really discuss RL for language models without discussing PPO. And even though we've done it once, PPO is confusing enough that I think you will benefit from doing it twice. The most important thing to identity, let's say to remember in RL for language modeling is policy gradients. And in particular this reinforce gradient trick. Well, we are always going to be doing is gradient descent on our rewards. And we are going to do so by taking essentially weighted SFT updates. The weights might be positive or negative. We will define what R is.

[04:29.640] But this equation is important because it will reappear throughout this lecture. This is in some sense the core from which everything else is derived. Now, building upon that, remember that the intuition from last lecture was we want to take more than one step at a time. A policy gradient requires us to sample from our policy every time we want to take a gradient step. Can we reuse our rollouts? Well, you can do TRPO. You can do PPO and so on and so forth. I'll talk about PPO in a minute. And I've already talked about TRPO. I won't really get too much into this. If you've taken RL course, PPO is, of course, well understood to you, like you understand this algorithm, but I don't assume that all of you have taken RL class and/or understand RL. And so PPO, in case you're not familiar, is a real workhorse of RL.

[05:20.450] It's been used in a variety of very difficult RL settings. OpenAI has done quite a few of these in their early days of being very RL pilled. They had, this was I think, OpenAI gym where they were talking about the PPO algorithm and they're like, hey, we can get these people to walk around using reinforcement learning. In a much more dramatic demonstration of this was their OpenAI bot, which they trained using PPO. And so this can excel with deep RL in high-dimensional action and state spaces, where it's much more complicated to do RL than it is usually. Oh, yeah. And there's the OpenAI man running around. All right. So at a conceptual level, PPO is actually very simple. Remember that I told you, an important part of all of this is getting rid of PPO. That's been a motivating factor for a lot of people. But if you look at the pseudocode of PPO, this is from the OpenAI's like spinning up on RL documentation.

[06:23.555] You look at this and you say, this is not that bad. This is actually pretty easy. I could implement this in one go. You sample some trajectories. I compute something called an advantage using any method of advantage estimation. I clip the advantage in this slightly strange but OK way. And then I update the policy under this clipped advantage. And that's fine. And then I can fit something called a value function as well. OK, fine. This is all relatively easy looking in practice. And so I think because of this, people are like, oh, it's OK. PPO is totally fine. But then you see blog posts that look like this, and this should strike fear into your heart. Because if you see a blog post that says the 37 implementation details of PPO, you know that this is an algorithm that is very sensitive to your implementation decisions.

[07:10.730] And there's all sorts of things that you can do and all sorts of different libraries and implementations, and they give totally different numbers based on which ones you use. And it turns out that many people have implemented this wrong. And so there's paper saying, like actually the baselines that some people use in PPO aren't even baselines at all. They fundamentally change the optimization problem. And so basically because of this, I think I would not be able to get through the rest of the lecture without telling you, well, we actually maybe need to start looking at some implementations of PPO. And the implementations of PPO, let me tell you, for language models are not particularly pleasant. This is, I mean, a very good representative slide. If anything, this is a very good description of what PPO looks like. You've got advantage estimation. This big block in the middle, that's this advantage thing. You've got an experienced buffer because you're going to keep some of the old stuff.

[08:05.828] It's going to be training this value model. And the value models are being used in the advantage estimation calculation. Notice how the green box appears twice. And importantly, some parts of my objective, the KL term, actually operates token by token. So it's not actually just a bandit problem. It's like a whole multi-step RL problem. And this is all very difficult and complicated. And so this has led to a variety of really tricky implementation things that you have to do. And if you look at a variety of implementations, you find often that there are things that people do for PPO that are a little bit strange and tricky at times. Let me see if I can find. So one of my students implemented a PPO for a project a while back doing RLHF. So PPO, if you remember, was used in RLHF. And so we had this nice robust reference implementation of PPO and it took a long time to get working. So I looked into the details, like what did we actually do to get this thing to work.

[09:05.290] The outer loop is fine. The outer loop is something like you go through your rollouts and you're going to take some PPO steps, you're going to compute some loss and clip some gradients and take some steps. Totally reasonable outer loop. But then you start of looking into it. For now, the actual inner compute loss part follows almost exactly the PPO update. So this also looks mostly good. We're computing things like the advantages and we're computing the clipping ratios. And then we're updating the model based on that. So this is all good. But then you get to some of the messier parts. Actually, I'll skip the rollouts, where you have things like well, maybe we need KL penalties to keep the original model close to the reference, but actually this only works if you clip the KL off at 0, which of course, if you know anything about KL divergences, that totally ruins the point of a KL divergence.

[09:58.090] You have both positive and negative values being summed. If you remove this, it blows up immediately. All things like this can happen. This is not to say that this is the way you should implement PPO if you were doing it. I think the reason why I'm bringing this up is to say implementations for RL algorithms can be somewhat tricky and somewhat sensitive because of all the moving pieces in PPO, and because of the high variance in the gradient estimates. These are all very, very tricky things that are happening. And, another thing that you often see, this one is not just RL implementation. It's very common. In the original PPO paper, you have this thing called a generalized advantage estimator that's basically looking at this gamma discounted reward, where you have this value function that's estimating the reward at of every token generation step. But it turns out that people often just use gamma equals lambda equals 1, which is just a degenerate setting that turns this back into a bandit problem.

[10:53.300] So you've thrown away a lot of the structure that you get from PPO. And so if you run this algorithm after a bunch of engineering, you will get what you expect. And when you run your RL algorithms for the assignment, you should expect to see similar things like the rewards go up. The reward model rewards go up if you're doing RLHF, and then the negative KL regularizers go down. This is a reasonable thing to expect to see. But anyway, the point of all of this is to say implementation of PPO can be quite painful. Depending on the environment, it can require hacks to stabilize. All of this is painful. Not to say that it's not possible. Lots of people have trained very good models on PPO. Many of the labs have very turnkey solutions for getting PPOs to work at scale. So this is not impossible. But for people like researchers who are implementing this from scratch, PPO can just be really finicky and complicated.

[11:51.870] Another reason why PPO is disfavored in many cases is that it requires a value model to estimate the value at each token as you go. And how big is the value model? Well, it's as big as the original model, so this consumes some memory that you would rather be using for other stuff like models or inference servers. So now you might say, well, in last lecture, you told us about this DPO thing, and DPO seems pretty good. Why don't we just use DPO for everything? Well, the problem is DPO is a very specific solution to a very specific problem. DPO is good for pairwise feedback in the form of Bradley-Terry comparisons. That is very specific. And if I want to solve math problems, my math problems don't come in the form of inherently pairwise comparisons. There are other DPO variants that are designed to break that pairwise structure, but really you're just using the wrong hammer for the job.

[12:44.900] You do not want to use DPO necessarily for what you would normally use PPO for. PPO is the more general hammer that you can hit everything with. DPO is generally offline. Although I think this distinction is very overstated because it can be made online by just iterating DPO repeatedly. So I don't think this is necessarily a big difference. So that's the context for all of this. And then so that similarly to DPO, there's just an enormous desire by the research community to not have to use PPO. Hopefully, the fact that DPO and GRPO have gotten adoption tells you how painful it was to get this to work in many cases. And so GRPO much like DPO, is the alternative, it is the simpler way to do RL for verifiable task. It has taken over for the most part RLVR in the open source, open knowledge, let's call it community. And it is very, very similar to PPO in spirit. It just strips out a couple of the most complicated parts, and by doing so, allows us to have a much simpler algorithm.

[13:51.130] So GRPO is by DeepSeek paper. I think it was there like DeepSeek math paper, where you start with PPO. It conceptually it says PPO is a good idea, but we want to change just a few things. And what does it change? It changes arguably the most complicated and annoying part of PPO, which is the value function. So the value function is the thing that you subtract as a baseline to reduce variance of your gradient updates. The value function is a whole neural network. It destabilizes training. We don't want the value network. So what do we do? Well, we get rid of the value network, but we still need an advantage. I don't want to do reinforce-- vanilla reinforce because the variance of that is going to be really, really high. So instead, what we're going to do is I'm going to compute you advantage as a z-score within a group, and I'll explain what that means in a moment.

[14:40.640] But basically, the idea is the following. Normally, you look at a reward that you get and if you had a value function, you would compare it to your predicted value. You would say, my neural network says, I should have gotten a score of 5. I got a score of 6. So this is a good rollout. That's usually what you do. Instead, what you would do is you would let's say your rollout with a score of 5. You would now sample 10 other rollouts and you would say, how good was I compared to my 10 other rollouts. If I'm doing better than my mean, then I have a high advantage. In some ways, a very natural way of computing advantage in contexts where you can multiple sample from the same prompt or something like that. So GRPO is exactly this idea. If you look at the initial definition of GRPO that appeared in the DeepSeek paper, you see, this is the objective that's being optimized.

[15:30.860] They're going to do exactly the PPO update, where they have this min clipped advantage, plus a KL term to keep yourself close to the reference. And then KL is computed in a particular way. That detail is not particularly important. And importantly, this advantage is computed as a z-score over samples of a group of outputs, o of 1 through o of g. So that's-- or within each group, you have-- oh, sorry. This advantage of i, this is the place where you compute the advantage. What you do is you take the reward over your rollouts within each group, you subtract the mean and you divide by the standard deviation.

[16:04.710] So you get a z-score for each group. So in the online case and this is an important distinction because in many cases, you may actually end up running this online rather than offline. In the online case, the clipping just disappears because the ratio between pi theta old and pi theta is 1, this clipping operator never does anything and you just get min a of i, a of i. So this is just advantage minus a KL penalty. So I'm going to do RL on my advantage, which is defined by this. So in the online rollout case, it is a very, very simple object that makes sense as RL target. So I can pause here for a moment in case any of the conceptual bits of GRPO are not clear.

[16:46.280] I mean, in some sense, this is the foundation of all of the exciting math stuff. Or you can replicate all of the exciting math stuff using GRPO. So it's worth understanding in a little bit of detail before we move on to some of the nitty-gritty details. OK. Good. Simple enough that everyone understands in one go. I will take it. And GRPO is very, very simple because of the lack of value function, you can basically write down all of GRPO in a very simple block of code, where you just say, I'm going to roll out k times, I'm going to compare my observed rollout to those k references, z-score it and then just take a reinforced gradient

[17:32.210] against those weights. So you will write this in your assignment. And it's going to involve a very simple set of steps. Compute a reward for each of your k rollouts. Normalize your rollouts, compute the KL term. And then just take gradient updates on the combination of your KL term and these rollouts. You'll have to do a little bit of thinking because in order to do this using autodiff, you will have to do a stop grad somewhere. But it's actually not that complicated. And you can see that you can do a one-page implementation of this basic idea. And this is a reference implementation of GRPO that I took from some folks at McGill, which has a really nice little toy version of this whole thing. And the computation of the group indices, and the standard deviation calculation and all of that just fits in one half of a slide. The only thing that they do differently than the literal paper write up of GRPO here is they add a little tiny weeny negative 4 to the standard deviation calculation

[18:33.560] to prevent it from blowing up when you only have a single sample, or if your samples happen to have the exact same rewards, which does happen if you're in a domain where you can get exactly numerically 0 rewards, like you failed to solve a math problem. But hopefully, this emphasizes to you how straightforward GRPO is, and why almost all open source work has been built on this algorithm. It's easy to implement, easy to understand. And if you, as we'll see in the results, it provides fairly compelling results in terms of emulating a lot of what the closed source labs have done using potentially PPO. So how well does it work? In the original DeepSeek math paper, which if you're into math AI stuff, I would encourage you to read, it's a nice and really interesting set of results. They show that GRPO, the yellow and the blue lines, does much better than RFT, like rejection fine tuning.

[19:31.097] You'll implement this as a baseline as well. That's just taking the correct answers that your model generates and training on them and throwing away everything else. And they seem to show process supervision, which is grading not just the final answer, but grading the intermediates give you some gains. I'll talk more about that later. That is an important thing to discuss and cover because that's one of the big design decisions for these kinds of RL problems. But for now, you can just internalize the fact that GRPO works. The blue and yellow lines are above the others. OK. So now let's think a little bit more about the GRPO objective. What is happening in this objective? Are we actually taking policy gradients? There are important conceptual questions that we should really think through. And so if we think about the difference between GRPO and PPO, remember that PPO had this value function thing which feeds into the advantage.

[20:27.580] And in GRPO, we strip that out and we replace that with the z-score guy. This a of i, ratio of r minus mean divided by standard deviation. So is this a good advantage? I think for those of you that have been in an RL class, you already know the answer to this question. Is this a good advantage function? What makes a good advantage? What makes a valid advantage function? Well, if you look at Sutton and Barto, the big classic book for reinforcement learning they will say, well, there's this algorithm called reinforce with baseline. And the things you're allowed to do is that if you're doing policy gradients, which is this line over here, this very first equation. I want to take my gradient step on the direction of my rewards. And the thing you can do is you can take your rewards and I can subtract any-- what they call state-dependent baseline, in this case, our state.

[21:20.800] If we're in the bandit world, it's just the prompt. So you can have a prompt-dependent baseline that you can subtract. And anything that does just this is a valid form of a policy gradient. So you're still going to descend in the same direction as long as you do this. And depending on your choice of b, you will either have lower or higher variance in doing this gradient descent process. Now, you should probably notice that what we're doing is not this. We are not just subtracting a constant value, we are also dividing by the standard deviation. And so that is a problem. It turns out well, it's in some ways a problem. In other ways, it's not. If you really want a conceptually clear algorithm that really does what's written on the tin, it actually descends the reward. GRPO does not do that. Because as I've highlighted in red here, GRPO divides by the standard deviation, which breaks this baseline contract.

[22:20.630] We're not just dividing or subtracting a baseline. We're also normalizing by the standard deviation of my rewards. Of course, there's another thing that GRPO does, which I didn't mention earlier. As an implementation detail that is tricky, which is GRPO does this almost per token. So they'll divide by the total length of the sequence as a normalization factor. But if you try to derive GRPO from first principles, following the policy gradient and baseline theorems, you'll end up with something different. You won't have a length normalizer and you won't have the standard deviation normalization. And some folks soon after GRPO came out noticed this, and they wrote a paper basically saying, well, if you don't do these two things that are different, then you get very different potentially very positive behavior. I will talk about that. Also, when we get to the-- not OpenAI, sorry DeepSeek R1 paper.

[23:22.890] But the main thing here is to say GRPO is not the first principles derivation of this idea. It is actually doing something slightly different with both pros and cons. So what are these two terms doing? We now that GRPO is not actually directly descending on the reward objective. We know that it has these two correction factors. What do they do? Well, the length normalization is easier to see. So what are we doing? We are dividing longer sequences with a bigger number. And so what this will do is that it's going to encourage the model to generate long outputs because-- or sorry when you're wrong it will encourage you to generate wrong outputs. Because let's say-- I know that I'm going to get a math proof wrong. I'm going to incur my negative reward of let's say, negative 1. I'm just going to generate an infinitely long string. If I do that, I'll get to divide by infinity, and I'll get to totally get rid of my negative penalty.

[24:19.900] That's an extreme case. But you understand that if you divide by the output, you encourage the model to blab on once it realizes that it cannot actually solve the problem. So this is a length problem that you end up getting. If you fix that, you'll find that often what people have observed in GRPO COT, like chain of thought, thinking length growing and growing and growing, that actually turns out to just cap off at a constant rather than just continually and forever grow. So this seems like a potentially good thing. And especially on the incorrect cases, you really don't want to be generating longer and longer and longer outputs. So the standard deviation normalization, this one's a little bit more complicated. But if you think about it, what I'm doing is I'm dividing by the standard deviation. And so this can be thought of as emphasizing problems where the standard deviation is small.

[25:15.320] And when is my standard deviation small for a binary reward problem? Well, that's when the problems are too easy or too hard. If my problem is really easy and I get 100% all the time, I'm going to divide by 1 over or the number-- I'm going to divide by basically 0 because there's 0 variation if I always get a question correct. Similarly, if I always get a question wrong, I have 0 variation in my rewards. I'm going to upweight that significantly. So the standard deviation term is upweighting the two sides, both easy and hard questions. That seems like clearly a thing that maybe we don't want because we want our models to potentially learn on things that are within its solvability range. So that's the algorithmic components of RLVR. And I'm going to go through several of the different recent model releases and talk through all the algorithmic components that I think are quite important, including some of the stuff that I added this year that's new, which is really the agentic stuff that I think has been catching on

[26:18.680] and is increasingly important for using and deploying these models. But before I do that, I can pause here and if anyone has algorithmic questions, I'll take them before I move on. OK. Good. All right. So we're going to start with DeepSeek R1 because I think this is a bit of a social phenomenon. I think everyone should know a bunch about DeepSeek R1. I think it's a lovely paper kicked off the wave of open source RLVR models. And what's remarkable about it, I think it was really the first thing to have matched OpenAI O1's behavior. So the key things about OpenAI O1 was you have very long chains of thought. It's clearly RL, and you have really good performance on hard math problems. And they also provided RL recipe that anyone could do. And I think from the perspective of R&D, this is very important. If your solution was like you have this horrible PPO thing that no one but DeepSeek can run, then that is much less of an impact, at least for us researchers than this kind of GRPO thing,

[27:29.010] which anyone can really play with, even you in your assignments. And finally, they have some really interesting distillation insights that I think have held up even now. So the starting point of DeepSeek R1, and I think you can see how a lot of DeepSeek's expertise build on each other. So they had this DeepSeek math paper, where they were doing GRPO. They had already understood a lot of the nuances of RL on math problems. And so because of this, they started off with a lot of the components of DeepSeek math. But one of the really important differences, and I'll highlight an expert just because I think it's particularly important is they abandon process supervision, which was the thing that worked really well in DeepSeekMath, and they go only with outcome supervision. And just to be clear, outcome supervision is when you have reward only for whether the final answer is correct or incorrect. And process supervision is when you have a grading rubric, let's say, to check the validity of intermediate steps in a proof.

[28:31.180] A lot of people thought process supervision was important. Turned out it wasn't critical for a lot of things. So R1 was interesting in that it has a lot of different components. But one of the very interesting components is R1 zero, where they don't really do that much post training, so they have a base model. Of course, the base model, as we all now is also [? MIT ?] trained. And so this guy can do some amount of instruction following. And then on top of this base model, they do basically RLVR, where the rewards for their GRPO algorithm is accuracy, rewards for whether they correctly solve a collection of math problems and format rewards. And this one is important because the format rewards basically allow them to strip out the chain of thought later. They want the model to properly use thinking tags to enclose their COT. But this is a very simple recipe. GRPO is very simple. Accuracy rewards are very simple.

[29:27.400] In fact, you will implement and replicate essentially the set of results in some ways, and you do just this very simple recipe. And where do you end up? Well, you end up something that is only a little bit worse than OpenAI O1. So this is really, really, really clean. And I like this result because it has none of the messes of a real production post-training pipeline thrown in. You don't really question whether oh, was it because of the RLHF helping or this or that. It's a very simple base model plus GRPO. Your math abilities are quite good. And I think R1 also took off, partially because of the interesting phenomena they claim to have observed in the paper. So one of the things they claim to have observed is as we go longer and longer and longer during training, the model is able to think for longer and longer and longer.

[30:15.570] Their COT get longer and longer and longer. And they also highlighted this thing that I believe went viral on social media, which is like the aha moment, where the model is if you train it for enough, its COT is able to do things like, oh, there's an aha moment that I can flag here. And they thought this was really interesting. It is unclear to me whether either of these phenomena are really particularly impressive in many ways. We now know that longer CoTs is arguably a natural side effect of the length normalization of the GRPO algorithm. And the aha moment others have showed is actually appears in even the base model. And so clearly, it can't just be a result of the RL algorithm. Lots of people say aha, I can use this as part of solving the math problem. The model has learned it during pre-training.

[31:10.900] It's not surprising at all that somehow during RL just happens to get extracted because it's emitting a lot of math tokens. So I think the phenomena that was highlighted is not particularly salient or exciting, but I do think R1 was a really important milestone in that it highlighted just how simple RLVR could be. This is very, in some sense clean as an algorithm and as a way of solving these really difficult problems. And so R1 zero was this very nice, clean controlled setting. And then R1 was their attempt to productionize this system. And I think one thing that's nice about R1 and I'll show you the Qwen equivalent later, is you can see how you stack all the pieces together.

[31:55.700] I think we've studied all the pieces in isolation. We know where pre-training is. We know that there's RLHF, we know SFT, but how do we compose all of the pieces to actually get a system at the end? Well, usually the thing that you do is something like, well, we have our mid-trained model, we're going to do a bunch of reasoning training, maybe some long context extension somewhere too. And then we're going to do RLHF at the end, because that's in some sense, the most user-facing object. We want to make sure that the formatting and all of this is nice. RLHF can very much help with that process. One of the other things that they did was they added a language consistency reward for the chain of thought for their production model. I think if I remember the report right, this was because the R1 zero style training without a consistency reward would language switch in COT, and they thought this was very uninterpretable and slightly disturbing. And so they wanted to just have it output a single consistent language.

[32:51.590] And so they do this just for interpretability reasons. And then they have some of the non-verifiable rewards as well, thrown in to GRPO, almost as blending into the RLHF process, but otherwise it's a very, very similar thing. And the whole pipeline, I think, is very straightforward. So for SFT, what they do is pretty simple. So in R1 zero, they don't want to do any SFT at all. But in R1, they're willing to do long COT data. And when you're reading these open source tech reports, it's fun to try to read between the lines. When they say something like, we construct and collect a small amount of long COT data, you might wonder-- I wonder if that was distilled from some other models. That is a particularly like negative thing for open source models. Everyone is nowadays doing this, but I just find it funny that it's very carefully written collect a small amount of long COT data. And so this is used to fine tune the model.

[33:52.460] We know that for a very good base model, just SFT on long COT can unlock a lot of O1 style capabilities. And that's a great starting point for RL. So they start there and then they add some verification to filter out the CoTs. And then they continue on with processing. And I think many papers since then, some including some from my students have shown things that if you have the right kind of distillation procedure and you have the right kinds of base model, you can basically get a lot of the long COT reasoning juice just from SFT. And I think this is maybe interesting because one of the things that I think is still an open question is, do you really need RL for some of this. And I think maybe one way of thinking about the role of RL in language modeling is that RL is a great source of supervision.

[34:43.830] If you're solving frontier math problems, you just don't have the supervision to get detailed long CoTs, and RL allows you to self-generate that. But once someone has generated these long CoTs, you could potentially also learn from imitation. And that seems to be what a lot of these distillation results partially show. So the RL part for R1 is basically the same as R1 zero. Minus the language consistency loss that I mentioned, it's basically just we run the RL training using GRPO. And that leads to essentially the final phase, which is what we learned about last lecture. So we do some basic instruction tuning style SFT and then RLHF. For non-verifiable tasks, they basically use the same thing as DeepSeek V3. So there's really no surprises at all here. How well does it work? It's really, really good. And I think this is the reason why there was all that panic or big noticing of DeepSeek because R1 was genuinely a very good model.

[35:48.380] It beat O1 on many of the categories. It matched a lot of the test time scaling behavior that people expected. And it came from a very simple recipe. So it was very easy to understand where the gains were coming from. And the last thing related to some of the distillation discussions that I was saying is you can then take R1's CoTs and you could put it into Qwen 2.5, and if you did so, you could really significantly boost the performance of these models. In some cases, matching a lot of these specialized thinking models. And so in other words, really, if you can get the right kinds of long CoTs that are legible to these models, often it is possible to get them to reason for long periods of time, like the base models are already surprisingly good, and this works even for Llama models as well, which I think is quite fascinating. So to close out the DeepSeek saga, I quite liked a lot of the DeepSeek tech reports, because they often do both ablations and they also tell you failed things that didn't work.

[36:59.680] And if you've been following what they've been doing, DeepSeekMath has all of this process reward model stuff. It's like we got to verify all the steps and then you read R1 and you say where did the process reward models go. Well, they tell you where the process reward models go, which is we tried to get process reward models to work. They just didn't do very much for us. It turns out that outcome reward models are great and they're good enough. And you can scale the data for those a lot better. I think a big debate for PRMs was, where are you going to get these step-by-step rubrics. And it's very hard to scale that up. I think at this point, it's quite clear now that outcome reward models are very, very good. And that's the bulk of where the action is happening. Similarly, if you were around for the initial O1 release, a lot of people speculated about what was going on inside OpenAI's O1.

[37:52.420] They were like, oh, are they doing PRMs? Are they doing tree search like AlphaGo? They to tried a lot of MCTS and they describe we couldn't get it to work very well. I just very much want to highlight and the fact that they're very open about all these explorations and the things that they tried and didn't work, rather than to say they didn't try it at all. Cool. All right. And if I talk about-- or actually, I'll pause here in case anyone has questions, although I don't know if-- yeah. [INAUDIBLE] you should explain the positive reward basis by [INAUDIBLE] Because I thought you said for negative, like we encourage to divide by infinity and [INAUDIBLE] 0 for positive. Yeah. So the question was like for positive cases what the length normalizer do. And of course, in those cases, the length normalizer encourages you to shorten the COT, which is good if you want to save on inference costs, bad if that degrades your accuracy.

[38:55.140] But I think the main issue is that for the long-- for the negative ones, you get some really long responses and the positive cases, it can't shrink it that much because there's a lower bound to how small your COT can go to solve a particular problem. I guess I'm trying to understand that explanation for the diagram. Is that-- I think doctor GRPO has a cleaner diagram of this. So in the R1 diagram, this is all aggregated. This is all of the responses across their evals. It's not just the positive ones. But if we look at the doctor GRPO plot over here. So this is like the average output length and this is the incorrect output length. And so you see that in the correct output length is here. So you see somewhat that this is really being driven by the incorrect ones as we would expect from the explanation. So the story, at least in these sets of controlled comparisons, is quite clean and clear.

[39:58.250] Yeah. Cool. So now, I'm also going to talk about an alternative method, Kimi K1.5 or alternative paper and approach. And I always feel like when I talk about R1, I'm obligated to talk about Kimi K1.5 because they came out at the same time as R1. They also beat R1, and yet everyone is all about DeepSeek, and Kimi is not quite as often brought up in the conversation, even though their models are extremely good. And I'm not just talking about them out of pity, per se. I think Kimi has this nice thing that they do something-- some sets of things, quite differently than DeepSeek, and we can learn quite a bit from the fact that both of these work. I think this is part of the theme of this class, which is by reading all of these tech reports and by looking at broader patterns, we can roughly start to understand what are the valid or easy spaces to get these algorithms to work. So they do actually quite a bit more detail in talking about the data set construction and curriculum generation for the RL process, which many have said

[41:08.440] is quite important. And they also have a different RL algorithm, which in some ways, has similar intuitions but is not exactly GRPO. And so this is another nice validation that technically speaking, I don't think you necessarily need something like GRPO to get these guys to work. So the first part I want to talk about is the data. The data remains very important for all of these processes. And for RL, there's an additional wrinkle of curriculum, what difficulties of problems do you want to throw in? In SFT, you don't really think about difficulty very much because you just jam the data in there. For whatever data that you want, you just do pre-training losses and the model is forced to learn from these. In SFT, there's a little bit more of a subtlety. We talked quite a bit about the hallucination issues, and maybe sometimes you don't want to train on some of this data.

[42:02.830] But for RL, there's this additional wrinkle that if your problems are too hard, you get no rewards. And if you get no rewards, you have no signal. And if you have no signal, you can't learn. So it's very important to have broad coverage of the right kinds of examples. And so what the Kimi folks do is first of all, they try to get a whole big, broad range of coverage of data. Unsurprising, they exclude multiple choice because that's covered in lots of different domains. They want things that require long, deep thought. And their argument is maybe multiple choice doesn't have that. But maybe this last one I think is maybe the most important idea. This appears in a lot of different RL papers. You basically filter all the examples for things that either pass or sorry, you filter them based on a best of k filter.

[42:52.800] So here you take examples. And if they succeed on best of 8, if you sample eight times and the model can do it at least once, then that's not a great RL example, because it's already at the edge of the model's capabilities. You're not going to teach it anything particularly new. And so you only look at examples that basically fail this test, because that allows us to essentially pick the right kinds of problems. And so this saves compute. It allows you to skip problems entirely. You can also filter on both sides to get problems that are neither too hard nor too easy. I think the general consensus in the research community is that doing this medium range difficulty filtering is very good if really what you want is for RL to progress at a steady pace. As with DeepSeek no description about SFT. We can speculate about what that is, but we have no concrete information.

[43:53.098] So I think the interesting thing about Kimi is that they have a DPO inspired argument, but they end up actually in a place that's very similar as GRPO. And I think the fact that they end up in similar places is suggestive of which of these components are potentially useful if you're designing a new RL algorithm, for example. So they start at the exact same place as everyone else. So this should now look very familiar to all of you. It's just maximize the expected reward under my policy. And I've got this KL regularizer on the right that keeps me close to my base policy, or maybe even the previous iterations policy, depending on how you're going to do this-- do this RL. So they're going to follow the same DPO style derivation, where they say assume that I can maximize analytically and I solve for the reward model that does that maximization or corresponding to that maximization. And that's going to give me this ratio of policies.

[44:51.360] And I'm going to plug that back in to my objective up there and try to solve for what is happening. Well, what they're now saying is they want to try to match the reward model. And for this equality to be-- they want this equality to be true, which is true at the minimizer. And therefore, this is a big heuristic. If something's equal at the minimizer, why don't we just put a squared loss on that thing and minimize the squared loss? I think optimization people looking at this would be horrified at this, but I think this is a totally reasonable intuition in some ways. We know that these two things should be close when the model is good. So it is also maybe OK to treat making those two things explicitly close as a surrogate. This derivation, as you can see, is very, very different from GRPO. We're not going through the PPO path to derive this update.

[45:45.030] But now, actually, lo and behold, you take a gradient of this L, you end up with something that looks surprisingly like GRPO with an added regularizer term. What you end up with is this thing that is a policy gradient with a baseline R bar. And that R bar is actually the mean of each condition, these x's. And you have this additional regularizer term, which is essentially the KL term that would appear in GRPO. So in some sense, we've reinvented the group mean normalized baseline through quite different means. And so the Kimi folks I think in some ways had a nicer or better view of the length problem. I think the GRPO folks were like, isn't it great that the length is growing uncontrollably. I'm sure our model is being smarter. That's a little bit of an uncharitable take, but when you present this plot as a positive thing, the implied statement is like, it's great that our model is thinking for longer.

[46:51.540] But the Kimi folks look at this long COT problem and they say, well, if we're making long CoTs potentially, that's very wasteful. We don't really want long CoTs because CoTs cost inference. So not only do they not have a length problem, because if you look at this objective, they don't normalize by sequence length, but they actually want to go even further and they want to compress the length of the response. And this is another important feature that we see generally occurring in language model development. We want the CoTs to be as short as possible. We want to solve hard problems. We don't want to think for very long, because when we think for very long, we have to essentially subsidize the user. If you're OpenAI and your users have the $200 Pro Plan, and your models are thinking for an hour at a time, that's not a very good place to be in. Whereas if your models are thinking for five minutes at a time, that is a great place to be in. And so for all of these reasons, Kimi wants to go and compress the CoTs.

[47:54.600] How do you do that? Well, the simplest are all thing to do is you come up with a heuristic length reward and you just tack it on to the RL process. This reward is surprisingly complicated. And the thing that they're doing is you're trying to make longer sequences shorter and shorter. Correct answers also should be short. The interesting balance here is if you make the incorrect answers too short, they no longer have a way to recover. So let me give you an example. Imagine I'm bad at geometry. AI bad at geometry, I get lots of incorrect answers, and the penalty makes my geometry CoTs really short. Now, my geometry CoTs are 0. I'm really bad at geometry. I will never recover from this. I will never get a positive geometry reward ever again. And I'm stuck. And so what you do is you don't force the incorrect answers to be super short. You incentivize them to be just a little bit shorter than the average, so that the incorrect stuff doesn't grow unboundedly.

[49:02.190] And so this has, in the end, a surprisingly reasonable set of outcomes. And so as I said before, they have a really nice RL set of views on how they do their data set curation. So not only do they have this data set that they filter, they also look at the success rates of how often they're able to solve particular problems. And once the model masters a particular problem, it gets taken out of the problem set. So you save some compute doing that. I think basically everyone doing RL does these success rate filtering to avoid both wasting compute or working on problems that are way too hard for the model. And they basically have rewards that, for code, they take ground truth solutions and they generate some new test cases. And for math, they actually have a reward model to check for answer equivalence checks. And this is a really funny thing in some ways, because we started out this lecture by saying we want to work on formal math or something,

[50:04.160] something truly verifiable, where you can-- a compiler can check the correctness of your math. And then we've gone through most of the lecture. And then in the end, where we ended up? Well, we ended up with a reward model, a reward model that checks the correctness of math answers. Once again, you'll see why in the assignment or maybe if you play with the assignment, you'll see why. You'll quickly find out that answer equivalence checking is a very difficult problem. Because in math, you can write equivalent things in many ways. And not only that, a language model can give the answer back in many ways. Even if you prompt it to say give the answer back in latex boxed, maybe sometimes it skips the boxed, maybe add some extra stuff to the box. There are lots of ways that the model can just fail to get the right answer by a strict correctness checker, even though it has the right idea.

[50:57.090] And so for these reasons, basically, most RL projects have a very complicated answer checker, either a regex or a model, or who knows what in order to avoid these kinds of problems. It's a real rabbit hole getting the verified part of RLVR. Cool. The last thing which connects to some of the systems bits that we had talked about earlier and the inference bits is RL optimization. And RL infra is really quite hard. Training is hard and inference is hard. And RL puts the two together. So in some ways, no wonder is it really horrible and difficult. Even worse, I think one of the things that you don't initially appreciate about how hard it is to make RL efficient is think of the following scenario. Now, imagine you've got your rollouts, but you've got one really hard math problem, let's say one of them is like the Riemann hypothesis. And your model is really chugging along on the Riemann hypothesis. It's got this gigantic COT. Now, what's happening? In the meantime, if you're doing naive inference, everyone else is waiting on this one rollout

[52:09.373] to complete in order to move on to the next phase. So if you're doing these batch things, long rollouts can really hurt you. And so RL really needs to deal with all of these really tricky low-level details. So you need to deal with the fact that some CoTs can get really long. Do you need to truncate them, do you somehow set them into a different machine, who knows? These are all decisions that you can make. If you switch from training to rollouts, you roll out once and then you train and you roll out and you train, you have to somehow deal with this. Either some of your machines are pure rollout machines, and some of them are training machines, or you're switching out frameworks all the time. Both of them are very costly. And then finally, you have this really difficult and horrible trade off, which is on policy stuff is mathematically and training dynamics wise very nice.

[53:00.710] So GRPO in its simple on policy form behaves very nicely. You will experience this in your assignments and then you will get greedy. You will say but my systems utilization is so low. I could do so much better if only I could reuse my rollouts, then I can overlap my inference and computation and do all clever things. And so you will attempt to reuse these rollouts. And that will lead to off-policy problems which then lead to destabilizing your training and all sorts of things that are very difficult. So these are all sorts of things that end up getting very, very tricky. And so I think most open source technical reports will have these days a section talking about their RL infra, and their RL infra is going to have to have both a training part. That's the blue boxes over here and an inference part, which is the green part on the right.

[53:55.340] And you're going to have to move weights from the training part to the inference part. So you need some of story for how to move those around. And you need to coordinate them closely. And in some cases, maybe they even share the same machines because as inference is running, the training one might be idle. So all of these are complexities. And you have lots of very complicated systems things to do. So we already know that Kimi beats OpenAI O1, but we also see some really nice things showing that as RL proceeds, you're able to think longer and performance goes up. But in some cases, it's not that we're just unboundedly increasing tokens. We end up getting cases where I think OmniMath is a good example. You're not thinking for that much longer, but your performance continues to go up. So maybe this is a good example of this length control kicking in and doing good things.

[54:51.410] The final thing I want to talk about is you might wonder, is this actual RL stuff much better than just training on correct answers. This is called expert iteration. It worked very well in a number of past papers. In cases where you're dealing with very unstable stuff, you may actually want to do expert iteration instead. Kimi K1 has very large scale ablations showing that these kinds of RL methods work consistently better than expert iteration. So that's the orange beating the blue over here. So you can't really avoid RL if you want to squeeze out all of your performance. I can pause here in case anyone has questions about the Kimi stuff. And then I'll move on to the last of the three which is Qwen 3. Great. OK. So the last one that I want to talk about is Qwen 3 and also Qwen 3.5-Next-Coder. The names are getting complicated.

[55:51.680] And Qwen 3 is later. But they have actually really interesting scaling and data results. So I want to talk about those. And the successor to this, which is Qwen 3.5-Next-Coder is interesting because I think has the most details of I would say agentic RLVR training out of the major tech reports and they do lots of cool things. So once again, as with DeepSeek, one of the nice things that we see with the Qwen report is the full way that all of these are organized. And this is very similar to the DeepSeek structure. We do base model to basically SFT, then reasoning RL. And then they have this thinking mode fusion.

[56:35.410] I'll talk about that later. And then they do RLHF and then they have a model that gets shipped. But of course, they don't want to serve that directly. So they do distillation to get their smaller models. I mean, Qwen 3 is quite a respectable open source model. You can basically have this be your mental picture of how frontier-ish language models are built putting all the components together. And I think you would have a pretty reasonable picture of what is happening. And Qwen uses basically, at this point, tried and tested playbook for RLVR. I think they take lots of the best parts of Kimi and DeepSeek, and they get it right. They do a lot of filtering for difficulty because we know that saves compute. That's a good way to get good data. They also do other things, like they remove things that the model can get right without COT because we know that's not a thinking problem.

[57:30.340] They remove things that are too similar to validation data to try to decontaminate things. And then they do a little bit of manual filtering on reference CoTs. Maybe most remarkably about Qwen 3 is they actually do their RL on very few examples, just 4,000 examples. But once again, if you have the rest of the pipeline right, you can actually get surprisingly far. So the core of the RLVR components are maybe not super surprising because they are really building off of DeepSeek R1 and also Kimi. There are some interesting Qwen 3 specific things, some of which they've ended up getting rid of, but I think some of the ideas

[58:15.500] are actually quite interesting. So one of the things that they do is they mix thinking and non-thinking things with tags. And that's how they end up having a thinking mode, where the model has a long COT and they can mix that with a non-thinking mode. So both the instant response model and the long COT model basically live in the same model. This was not true in many cases. There was often a thinking mode and a non-thinking model, even at OpenAI. They also have-- I thought this was quite interesting, which is they have this way of early exiting thinking where if they append a special string, they'll immediately stop the COT and the model is forced to give an answer to whatever prompt is given. We see this even in-- we see this affordance in interfaces like ChatGPT, where you're able to terminate the thinking early and try to get an immediate answer, for example. But what we see with this and I've always thought this was a surprising an interesting thing, which is that as you vary the thinking budget, doing this early termination trick, you end up

[59:25.000] finding that the performance of the model degrades gracefully, even though at the lower-thinking budgets, these models are kind of getting truncated mid-thought, they're able to give surprisingly reasonable responses even at that point. And we see that consistently, even with very small-thinking budgets, the thinking mode models are much better on all of these mathematical or coding tasks compared to the instant response mode, which is much more of the classic instruction following plus RLHF models. And so you can-- one of the nice things they do for us is look at the contribution of all of the different components to performance. And in many ways, this is not something that should surprise you too much. If you do this reasoning RL and then general RL, you're able to slowly get improvements in all of the different tasks, things like Arena-Hard or CounterFactQA. These are general tasks for which you want more normal RF, and we see significant boosts to that performance across the board.

[01:00:34.350] There are some degradations that we see in math and coding because we fuse together non-thinking components, but the degradation is not so bad. I think, actually, even though these numbers in an absolute sense are small in later releases, I think in some of the Qwen 3.5, they've gone back on fusing both thinking and non-thinking into a single model. I think what-- they used to call this hybrid models because they found this drop unacceptable. They wanted to squeeze out all the juice possible on thinking modes. And so they've now, I think, separated these models from each other. The very last topic and paper that I want to cover is Qwen 3-- or sorry, I flipped the two. It's Coder-Next not Next-Coder, which is one of the newer models to come out. And I think if you're interested in general agent stuff, of course, there's other agents things to think about harnesses. But if you're interested in how do you actually build an agent.

[01:01:35.280] This is probably a good report to go read. Qwen3-Coder-Next great agent post-training paper. But in some ways, post-training for an agent is not very different from everything that we've described. It's not like there's a new agent training algorithm. Really, you should internalize this lesson throughout the entire class. Data is the important thing. So if you're going to train an agent or post-training an agent, what are you going to do? You're going to get the data together. So there's two parts to the data. To get a lot of the capabilities, you can't just inject them at the end. You have to probably start a little bit earlier. And so there's an extensive mid-training phase that happens to try to get as much coding and agent like capabilities into the model as possible. So they'll do things like they'll take repositories and concatenate the files in the repository to generate these very long context data and put it into the model.

[01:02:30.170] We know that the model is eventually going to see these very long COT traces, where maybe the agent has opened a bunch of files, very useful pre-training style data. They take pull requests and then they try to construct synthetic context for it, using RAG that is potentially helpful for understanding the pull request. That all goes in the mid training. They will take documents that they detect using an automated method to have text and code, and then they'll use an LLM to transform this to a nice markdown format. Throw that all in as well. And then finally, they're going to have a language model, talk about coding on web documents that are coding related to generate coding-ish synthetic data,

[01:03:14.440] and then they'll take publicly-available coding agents, run them on various environments, which I'll describe later. And the traces of those runs all go into mid training. They'll do some fairly standard stuff as well, like instruction following, and they'll have an additional task, which I think is not super relevant for us for filling in the middle. If you're coding and you want to fill in the middle of a span, it's useful to have that ability. So they throw some of that data into mid training. But otherwise, this is a very standard but in-depth data collection procedure to try to get some of these agentic abilities into mid training.

[01:03:50.360] And then the Qwen folks do something that I don't know if I've seen before. It's actually quite interesting. They'll take the mid-trained Qwen3 next model, and they are actually just going to train a whole bunch of different expert models for different kinds of coding adjacent tasks. And they will end up with four different agent experts, and then they're just going to distill them all back into the same model. I can only speculate as to why they do this. I don't think I've seen it in other works. The closest one I know of is DeepSeek V3 or was it V3? V3.5, sorry, DeepSeek V3.5 or 3.2. Maybe 3.2. Sorry, 3.2, I think, did this thing where they have different experts whose only job it is to format and process data, so they have data processing experts which generate the data for the full training. But I don't think I've seen this thing of generating different model experts and then distilling them back.

[01:04:49.490] That's actually closer to some academic work like branch train merge and so on. Those are definitely things people do. I haven't seen it much in frontier model training. That said, each of these experts is actually interesting, and what they do is they basically do full on RL or and/or SFT training on a whole bunch of different subtasks that they've defined. So they have a web dev expert, where they SFT on valid web code based on different kinds of checks. They'll do things build UX experts trained on many different tool formats, and they'll have QA. So the QA agent is just trained on more synthetic data for code.

[01:05:31.860] But then maybe the most interesting for us and the most involved is their software engineering agent. And what they do is they construct agent environments at scale. So three benches is the gold standard of these kinds of things. And so what they want to do is they want three bench but more. They go out and they have various automated ways based on GitHub to generate a ton of different issues. And given this, they want to basically do RL on these environments in some ways very unsurprising. And what do they get? They can get RL performance to go up. One thing I will say here, and this is very important through the broader context of what we're talking about, the reason why we can put more and more compute into RL is because we believe that our reward models are unhackable or difficult to hack. If that assumption breaks down, your RL method will find increasingly obscure ways of cheating you out of your performance.

[01:06:35.450] And so they have this great plot of basically, in Git, you can look at future commits or different commits. And if you have an issue and there's future commits, you can just look up what the fix was. And this is a very easy hack for models to learn. And so they basically have a whole reward whose entire purpose it is to prevent the agent from messing with the Git history. And if you don't do this, you end up with a plot that looks like the right, where it's learning, learning, learning, learning, learning. And suddenly, you get this emergent jump, where the emergent jump was actually, it learned how to manipulate the Git calls to get the history. And in some cases, it can even hack around certain constraints you give it. If you tell it, you can't use Git log, it might add an origin like a remote and then query the remote for what happened in certain commits. So there's all sorts of hacking you can do.

[01:07:32.270] RLVR is only as robust as your reward, and your rewards can sometimes be not very robust at all. As a side comment, one of my students and I were working on a project that was like RL on lean. Lean is this formal math, verifiable language. And we naively thought at the time, there's no way this can go wrong. Lots of people have worked on Lean. The Lean compiler is bulletproof. Turns out the lean compiler is not adversarially robust. There are strings that you can put in it that will allow you to verify proofs that are not meant to be verified in certain modes. So I think the notion of verifiable rewards is actually much trickier than many of you might initially think. Regardless, you do this process of constructing these GitHub repos at scale. You go through this process of doing RL, you see your scores go up slow but steady, and in the end, you will actually end up getting a system that is surprisingly good, a model that achieves something

[01:08:34.740] like 70.6% on SWE-bench, even though you have what is basically a really tiny model, a three billion active parameter model, you're able to do surprisingly well on these tasks. In some ways, though, I don't know if this is super surprising. RL, of course, you're going to be able to do well on the environments for which you've trained. You might even get generalization to your validation set as you do here. But you always want to be a little bit careful about comparing performances. Task specific performance doesn't necessarily mean it'll generalize to broader domains. OK. So that's basically all I've got for the RLVR lecture. Remember the core takeaways of this lecture is it's really all about the reward for RL. RLHF and RLVR arguably they're very similar problems. But with the difference being really just we want more unhackable rewards so that we can actually put in much more compute and get these systems to be much better.

[01:09:39.330] GRPO, at least for the research community, is something that really enabled a lot of this. And you should all know GRPO very well. You should know what the functional form is and how the updates are. This is something that you should know as well just pre-training losses. And then finally, at this point, I think lots of people know how to do RLVR. As you will find out, unfortunately, RL remains very finicky and noisy and it's painful to work with. But it's not that hard. It's not like the old days of doing PPO on various really tricky environments. It's actually a lot smoother than you might think. I'll stop here for a moment and take any final questions before I let you all go, in case anyone has questions. A really quick question-- when you mention about the models being in thinking mode on and off, is the wait on the model or [INAUDIBLE] the same?

[01:10:33.140] Or do they play prompting or do they have different underlying models that they call back? Yeah. So the question was about thinking mode and what's happening on the back. So the interesting thing about thinking mode really is that it's actually one model. And there's just a little prompt tag that switches them between long and short, like long COT and no coat almost mode. Whereas if you just had an API flag or something, that's not very difficult. The fact that they were putting both of them in together is the interesting bit there. So you can actually choose, [INAUDIBLE] but then you add extra prompt yourself to do COT? Yeah, or the control mechanism in this case is in the prompt rather than at the API or serving layer or something like that.

[01:11:13.270] Yeah. Great. Oh, yeah. So you mentioned the phase of the mid training. I was wondering how much of the formatting that goes into the mid training process, informs what is able to be learned during the RLVR post training and then as a secondary to that question of if you don't have the correct data in mid-training, then is it safe to assume that the reinforcement learning can't sample a correct solution and therefore you can't learn it? Yeah. So the question was about the role of mid-training in RL. I think there are a lot of empirical subtleties here. I don't have a one size fits all. It will definitely not work an answer, but to give a little bit of the nuance, I think pre-training and SFT are doing a lot of the heavy lifting. As long as you have coverage, if pre-training just doesn't have any code data, then you're in trouble. You need training. But if your pre-training was very diverse and it covered, let's go back to the mid-training slide. If it covered a lot of stuff like text code data and it covered a bunch of GitHub data, then this is very nice to have, but maybe not critical

[01:12:17.820] because you're going to SFT anyway, and the SFT will allow you to get close enough to start getting some rewards for RL. If you didn't have SFT, then you're in DeepSeek trouble. But the fact that we're always going to be doing some amount of SFT pre RL really helps you, and pre-training has enough coverage, hopefully that you're able to get some reasonable generalization. So mid-training is very nice to have. Very important in order to get slightly better generalization but not necessarily make or break, I think. Yeah. [INAUDIBLE] there is a step after train your transfer model. We're going to be still jumping into a single model at the end. So does that-- how does this kind of distillation work? Do you have to design a mixed procedure? Yeah. So that's right. So distillation procedure is going to require you to write down a sequence of prompts on which the experts will be distilled into the final model.

[01:13:15.110] Yeah. I can definitely see the advantages of this approach, which is that you can have a separate team working on each expert. It's much easier to have teams work on this. And then the aggregation could potentially be simple as well if you have enough compute. It's just that this-- if you have all the objectives, you might as well just throw it into the big training loop. Usually, I think that's how you would prefer to do things. That way, you don't have to deal with this distillation complication. You could just have one big training objective that you then do one training loop. Yeah. Is the long reasoning training also part of the training, that mid-training? So the question was, is the long reasoning training part of mid-training. So I think so there's long COT SFT in both R1 and Kimi K5.1. Long COT is not traditionally part of mid-training, per se, but it is often-- long COT like data is used in long context extension.

[01:14:13.605] So it's a little bit more subtle. I didn't talk about long context extension at all and I regret that. But usually, that's an additional phase right before RLHF or other things, in this case, also long COT where you extend the context using any long context data you have. Usually, it's a combination of books, code, synthetic data, because those are the things that are long enough to do extension on. Yeah. So like I guess related to this diagram that's currently up, when you're doing this [INAUDIBLE] process, let's say how much math, chemistry, many other domains. Is it common to do it in parallel, with one, I guess, like reinforcement learning training run? Or you do it sequentially? And if you do it sequentially, how do you avoid forgetting earlier parts of the learning? Yeah, I think the closest answer to this question do you do things sequentially or in parallel is these diagrams, where the two splits that you do is you split amongst like reasoning problems and non-reasoning problems, and the reasoning problems all go in one bucket and they get executed at stage II. And then the non reasoning problems happen at the final RLHF phase. And this includes stuff like chattiness and so on and so forth.

[01:15:30.708] And those all go into the one later bit, which is the stage IV. OK. Great. Thanks a lot. See you next week. [APPLAUSE]

Post-training - RLVR

课程简介 ​

CS336 · Language Modeling from Scratch — Lecture 16 中文讲义 ​

学习目标 ​

课程主线 ​

第一部分：核心算法 ​

1. 从 RLHF 到 RLVR 的动机 ​

2. PPO 复习与实现困境 ​

策略梯度核心公式 ​

PPO 基本结构 ​

为什么 PPO 实现很痛苦 ​

为什么不用 DPO 替代一切 ​

3. GRPO：核心算法 ​

动机 ​

定义 ​

GRPO 与理论策略梯度的偏差 ​

长度归一化的影响 ​

标准差归一化的影响 ​

为什么 GRPO 广泛流行 ​

第二部分：开源系统案例分析 ​

1. DeepSeek R1 与 R1-Zero ​

R1-Zero：最简洁的 RLVR 配方 ​

关于“顿悟时刻”的冷静解读 ​

R1 的生产化流程 ​

结果监督 vs 过程监督 ​

蒸馏洞察 ​

2. Kimi K1.5 ​

与 R1 的对比 ​

从 DPO 式推导到组均值基线 ​

长度控制：比 GRPO 更明确 ​

数据课程与筛选 ​

验证器设计 ​

系统效率问题 ​

专家迭代 vs RL ​

3. Qwen3 与 Qwen3-Coder-Next ​

Qwen3 的总体流程 ​

思考模式与非思考模式 ​

关于混合模式的取舍 ​

智能体 RLVR：Qwen3-Coder-Next ​

专家蒸馏策略 ​

奖励可被攻击的典型案例 ​

性能与泛化的谨慎解读 ​

核心原则总结 ​

课程作业关联 ​

常见误区 ​

复习清单 ​

从 RLHF 到 RLVR：后训练系列讲座中的思考模型革命 ​

为什么 RLHF 不够好，而 RLVR 让人兴奋 ​

回顾 PPO：为什么它是一个痛苦但不可回避的起点 ​

GRPO：去掉价值网络的组内归一化策略梯度 ​

DeepSeek R1：一个干净到接近纯净的 RLVR 配方 ​

Kimi K1.5：不同的路径，同样的收敛 ​

Qwen 3：从 SFT 到 RLVR 到成熟的系统化流程 ​

Qwen3-Coder-Next：智能体 RLVR 的深度解剖 ​

结语：从 Algorithm 到 System，再到 Rewards ​

CS336 2026 Lecture 16：完整中文逐字稿 ​

正文 ​

CS336 2026 · Lecture 16 · Post-training - RLVR ​