Training 2: Reinforcement Learning Basics
- 日期
- Tue, Sep 22, 2026
- 课程资料
- 8 项
- 内容产物
- 4 项
课程简介
CMU 11-768 Fall 2026 · Lecture 9 · Training 2: Reinforcement Learning Basics 课程学习笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
依据：Lecture 9 官方 Slides（
lecture-09-rl-basics.pdf，页码引用均指该 PDF 页码）与课程视频自动字幕清洗稿的中文候选译稿。两者若有不一致，以 Slides 为准；图示与公式的视觉细节需对照官方 PDF 核验（待核）。本笔记已由用户审核确认。
一、学习目标
学完本讲，你应当能够：
- 用数字搜索（Number Search）这一贯穿例子，说明智能体与环境交互如何被形式化为轨迹、观测、动作与奖励（Slides 第 2、9、10、11 页）。
- 说清 SFT 用于智能体/序列预测时的三类问题：任务不匹配、数据不匹配、暴露偏差（Slides 第 4、5、6 页）。
- 写出期望奖励目标
，并独立推导 REINFORCE 的得分比率（score-ratio / log-derivative）技巧（Slides 第 13、18、19 页）。 - 把策略梯度落到实现层面：掩码（masking）与按奖励重缩放（rescaling）SFT 损失（Slides 第 21、22 页）。
- 说明基线（baseline）与优势（advantage）为何有用、为何不引入偏差，以及它降低方差的直觉（Slides 第 27–34 页）。
- 描述组相对优势估计与 GRPO 更新，并说出 DrGRPO 的两处改动及其动机（Slides 第 37–40 页）。
- 识别本讲方法的共同失效模式（组内奖励全同 → 无梯度）与常见补救方向（Slides 第 41 页）。
二、概念主线
本讲的叙事是一条“问题 → 目标 → 估计 → 改进 → 变体”的主线：
上一讲回顾（SFT）→ SFT 的三个问题 → 让智能体自己生成轨迹并按奖励加权 → 期望奖励目标
主线的一句话版本：本讲所有方法都只是在给“智能体自己采取的动作”配一个权重，区别只在这个权重是什么（Slides 第 44 页）：
| 方法 | 权重 |
|---|---|
| ReST-EM | 成功轨迹上为 1，其余为 0（等价于对保留数据做 SFT） |
| REINFORCE | |
| GRPO / DrGRPO |
统一形式（Slides 第 44 页）：
三、贯穿例子：数字搜索（Number Search）
设定（Slides 第 2 页）：隐藏整数在 1 到 16 之间，智能体最多猜 4 次；每次猜错后环境返回“higher”或“lower”；只有猜中时奖励为 1，否则为 0。本讲的任务实例隐藏数字为 11。
四条样例轨迹（Slides 第 2 页）：
: 8 → higher → 12 → lower → 10 → higher → 11 → correct， : 4 → higher → 8 → higher → 12 → lower → 11 → correct， : 16 → lower → 8 → higher → 9 → higher → 10 → timeout， : 1 → higher → 2 → higher → 3 → higher → 4 → timeout，
关键观察：
奖励设置的一般化（Slides 第 9、11 页）：奖励是标量，可在轨迹末尾给出，也可逐步给出。字幕中提到 RLVR（带可验证奖励的强化学习）常使用 0/1 二元奖励，例如代码是否通过全部单元测试；本讲方法更通用，也适用于连续标量奖励。
四、符号与假设
先集中给出本讲用到的符号与它们背后的假设，后续公式都以此为基准。
- 任务实例
（如隐藏数字 11）。 - 轨迹
；奖励 （Slides 第 11 页）。 - 历史
，即策略真正条件化的东西（Slides 第 10 页）。 - 环境状态
：决定环境的一切，可能含智能体看不到的隐藏状态（例如后端数据库内容、隐藏数字）。 - 策略
： 是驱动智能体的语言模型参数。 - 轨迹概率（Slides 第 12 页）：
- 目标（Slides 第 13 页）：
。
假设说明：
- 部分可观测（POMDP）：字幕中明确回应提问——本讲处理的是 POMDP，智能体看不到底层状态，只能看到观测，因此
的概率依赖于此前全部动作与观测历史，而非仅上一状态与动作。表述方式是为了便于处理，在此类情形下与其他等价写法一致。 - 环境不可微：动作是离散且采样的；环境（例如单元测试）不可微（Slides 第 18 页）。因此不能直接反向传播，需要策略梯度。
- 环境概率未知但可采样：我们不知道
，但可以通过与环境交互来采样（Slides 第 12 页）。字幕中称之为“有点神奇”的地方在于：只靠交互采样就足以估计出依赖于该分布的参数梯度。 - 奖励位置：一般设置下可在每个动作/观测后给奖励；数字搜索只在轨迹末尾给 0/1 奖励（Slides 第 11 页）。两种奖励位置在公式上可互相等价处理。
五、SFT 的三个问题
Slides 第 3 页给出 SFT 损失：
目标动作可来自人类、更强的模型（蒸馏），或（本讲语境下）智能体自己的一条成功轨迹。
三个问题（Slides 第 4、5、6 页）：
- 任务不匹配（task mismatch）：SFT 最大化“被示范动作”的概率，我们真正想要的是最大化“完成任务”的概率。二者相似但不同：完成任务有多种方式（
与 ）；某些动作明显更差（例如在“12 → lower”之后不应猜 13），但 SFT 对所有未被示范的动作一视同仁，不会区分好坏。 - 数据不匹配（data mismatch）：我们希望也能从次优数据中学习。失败的轨迹（
、 ）不应被 SFT 用来提高其动作概率，但我们仍希望有方法让模型“不要走这条路”。 - 暴露偏差（exposure bias）：训练时模型在固定历史下预测下一个动作，而生成/评估时历史由模型自己产生（自回归）。若模型生成了训练分布之外的动作（例如收到“higher”后却猜了更低的数），由于训练中从未见过这类情况，它的响应可能完全失准（Slides 第 6 页示意：训练示范是 8 → higher → 12，而生成为 8 → higher → ?）。
字幕中补充：RL 可视为一种让模型生成自己的训练数据、并学会从自己错误中恢复的方法；这一“让模型产生自己的训练数据”的性质称为同策略（on-policy）。它通常有利，因为监督信号与模型当前能力匹配，也有效率上的权衡（下一讲/后续讲座会讨论放宽这些假设、更偏 off-policy 的方法）。
六、专家迭代与 ReST
在完整策略梯度之前，本讲先给一个朴素但有效的基线：专家迭代中的一类——强化自训练（Reinforced Self-Training, ReST）（Slides 第 14–16 页）。
流程（Slides 第 15 页）：
- Grow：从当前策略采样轨迹并计算奖励。
- Improve：用奖励过滤数据（二元奖励时保留
），扩充已有数据集，训练策略。 - Repeat：用更新后的策略继续。
损失仍是标准 SFT 损失，只作用在高奖励轨迹上（Slides 第 16 页）：
课堂澄清点（来自字幕）：它与“自蒸馏”的区别在于用奖励做过滤；若模型完全无法产生正奖励轨迹，RL/ReST 可能并不适用，可考虑先用好示范做冷启动（字幕提到推理模型常有冷启动阶段）、课程学习（先选有足够多可解样本的问题），或给予中间奖励。
七、策略梯度与 REINFORCE
7.1 目标与推导（Slides 第 18、19 页）
不能直接反向传播的两个原因：动作是离散且采样的（改变
起点是把期望写成求和：
取梯度：奖励
得分比率技巧：乘除
这是可采样估计的期望。REINFORCE（Williams, 1992，Slides 第 19 页标注）。
7.2 三个组成部分（Slides 第 20 页）
：通过与环境交互采样得到； ：来自环境的任务级评估； ：机制与 SFT 中相同，但动作是策略采样出来的，而不是示范动作。
7.3 轨迹对数概率的分解（Slides 第 21 页）
环境项不含
一个动作可能对应多个 token。若
7.4 实现：掩码 + 奖励缩放（Slides 第 22 页）
实际计算“基本上就是对标准 SFT 损失做掩码与重缩放”：
其中
7.5 训练循环（Slides 第 23 页）
- Collect：采样任务实例，在其上从智能体采样；
- Evaluate：为每条轨迹计算奖励；
- Form loss：用奖励给动作对数概率加权；
- Update：反向传播，改变
，再采集。
损失：
7.6 与 ReST 的关系（Slides 第 24 页）
二元奖励下，REINFORCE 与 ReST 形式相近：
差异在于 ReST 做批量更新并聚合数据集，而 REINFORCE 完全在线：采样后直接更新。
八、基线与优势
8.1 动机：奖励应当是相对的（Slides 第 26 页）
- 对成功轨迹
，所有动作都被 上调； - 对失败轨迹
，基础 REINFORCE（以及 ReST）完全不产生梯度，因为被奖励 0 缩放。
但我们其实希望“学会不去做”
8.2 优势的定义（Slides 第 27 页）
即“采取该动作后的期望奖励”减去“从该历史出发的期望奖励”。课堂具体例子：历史为 8 → higher、12 → lower，数字只能是 9、10 或 11，还剩两次猜测。若策略足够好，
- 猜 11：若错则剩两个候选、只剩一次猜测，故
； - 猜 10：即使错也还剩一次猜测，可以往高或往低猜，故
。
由于
8.3 带优势估计的策略梯度（Slides 第 28 页）
baseline
| baseline 选择 | 对应方法 |
|---|---|
| 常数 | 本讲的 bandit 例子 |
| 已见奖励的运行平均 | 随策略改进而自适应 |
| 训练出的价值模型 | actor–critic、PPO |
| 组内均值 | GRPO（无需额外模型） |
Slides 第 28 页强调：这些行是估计同一个
8.4 基线不引入偏差（Slides 第 29 页）
对只依赖历史的基线：
利用
8.5 两臂老虎机例子（Slides 第 30–33 页）
单参数策略：
无基线：
- 采样 A（奖励 1）：梯度
； - 采样 B（奖励 0）：梯度
。
加基线
- 采样 A：
，梯度 （无基线时 +0.75）； - 采样 B：
，梯度 （无基线时 0）。
两者都让
方差对比（Slides 第 32 页）：精确期望
长期行为（Slides 第 33 页）：反复采样的滑动平均都收敛到 0.1875，但无基线时 SD 0.325，基线
8.6 基线的局限（Slides 第 34 页）
有好处：标记轨迹比预期更好还是更差；好的基线降低梯度方差。
未解决：同一轨迹内所有动作共享同一个优势
九、中间奖励与 reward-to-go
若环境在最终步之前也给奖励，可用 reward-to-go 降低方差（Slides 第 35 页）：
一个动作只被它之后的奖励加权；更早的奖励不依赖
十、组相对优势与 GRPO / DrGRPO
10.1 组相对优势（Slides 第 37 页）
思路：对同一实例多次采样策略，用组内奖励的均值作为 baseline，无需额外模型。
Slides 第 37 页的表格（沿用
字幕中所属动机：GRPO 出自 DeepSeekMath 论文，目的是效率——过去把 RL 用于语言模型常需要单独模型来预测 baseline，这很耗算力；组相对方法去掉了这个模型，很优雅。组大小常见取 8（字幕提到后续讲座会有关于组大小选择的指导）。
10.2 GRPO 更新（Slides 第 38 页）
批中包含多个任务实例，每个实例有自己的组：
| 任务实例 | 组内奖励 | 优势估计 | |
|---|---|---|---|
| 1 1 0 0 | 0.50 | +1 +1 −1 −1 | |
| 1 0 0 0 | 0.25 | +1.73 −0.58 −0.58 −0.58 | |
| 1 1 1 0 | 0.75 | +0.58 +0.58 +0.58 −1.73 |
注意：每个任务实例单独标准化（各自的
Slides 第 38 页明确注明：这是针对当前策略轨迹写出的形式，省略了原论文的 ratio、clipping 与 KL 项。字幕也确认今天只讨论 GRPO 内部的优势函数，ratio/clipping/参考模型正则在后续讲座讨论。字幕中有提问确认该优势是无偏的（因为它只依赖提示与模型采样），回答为是。
10.3 DrGRPO 的两处改动（Slides 第 39 页）
- 不除以组标准差，只减去组内均值。理由：除以
会让优势失去“问题难度”的意识——实例的难度尺度被标准化掉了，从而“几乎总是通过”或“几乎总是失败”的实例权重被放大（Slides 第 39 页“Advantage estimate”行）。字幕补充：很多自称 GRPO 的实现实际上已经去掉除以标准差的步骤。 - 不按生成长度归一化，改为除以全局常数
（论文使用生成预算）。理由：按长度归一化时，长而错误的回答每个 token 被惩罚得更少，于是策略会漂移向更长的错误答案（Slides 第 39 页“Response aggregation”行）。
Slides 第 39 页底注：奖励本身不变，改变的是任务实例与回答之间的相对权重。
10.4 DrGRPO 的 token 系数（Slides 第 40 页）
对单个任务实例，轨迹
Slides 第 40 页给出
10.5 退化情形（Slides 第 41 页）
组内奖励完全相同时，优势全为 0，该组不贡献任何梯度：
- 全部失败
：可能对当前策略太难； - 全部成功
：可能对当前策略太容易。
若采样的轨迹全部失败，可考虑（Slides 第 41 页）：换任务实例、扩大探索、加入示范、提供更有信息量的奖励。字幕中进一步列出：冷启动/示范微调、课程学习、中间奖励与奖励塑形（例如给“找到正确文件”的中间奖励）。
十一、常见误区
- “策略梯度需要知道环境模型”。不需要。未知
可通过交互采样绕过（Slides 第 12 页）。代价是算法偏慢、算力需求大。 - “奖励为 0 的轨迹没有信息”。在基础 REINFORCE 里确实没有梯度（Slides 第 26 页），但加 baseline 后，失败也会提供学习信号（Slides 第 31 页）。
- “减 baseline 会改变优化目标”。不会；对只依赖历史的 baseline，期望梯度不变（Slides 第 29 页），变化的是方差（Slides 第 32、33 页）。
- “baseline 能解决信用分配”。不能。同一轨迹所有动作共享同一个
（Slides 第 34 页）。reward-to-go 是另一条独立思路（Slides 第 35 页）。 - “GRPO 就是论文完整损失”。本讲只讲优势函数，省略了 ratio、clipping、KL 项（Slides 第 38、40、42 页）。
- “GRPO 的除以标准差是必要的”。字幕指出很多实现已去掉它；DrGRPO 正是把它作为两处改动之一，并给出“失去难度意识”的动机（Slides 第 39 页）。
- “ReST 与 REINFORCE 是两种完全不同的东西”。二元奖励下二者损失形式高度相似，差别在于批量更新/数据集聚合 vs 完全在线（Slides 第 24 页）。
- “组内奖励全对/全错只是数据问题”。它是算法层面的退化：优势恒为 0，该组无梯度（Slides 第 41 页）。
十二、复习问题
- 写出
的定义，并完整推导 REINFORCE 的表达式，指出每一步用到了什么假设（Slides 第 13、18、19 页）。 - 为什么
中环境概率项会消失？这一性质对算法有什么实际意义（Slides 第 21 页）？ - 用 Slides 第 2 页的四条轨迹解释“任务不匹配”：为什么
与 的存在说明 SFT 目标与“完成任务”的目标不同？ - 在数字搜索给出历史“8 → higher、12 → lower”时，为什么
而 ？这对策略意味着什么（Slides 第 27 页）？ - 在两臂老虎机例子中，基线
如何把“采样 B 得到 0 梯度”变成“+0.125 的梯度”？为什么这不改变期望梯度（Slides 第 30、31 页）？ - 已知无基线时 SD 0.325、基线 0.5 时 SD 0.108，而两者滑动平均都收敛到 0.1875。这说明基线的收益体现在哪里（Slides 第 32、33 页）？
- 为什么 DrGRPO 去掉除以
？为什么去掉按长度归一化？分别对应 Slides 第 39 页表格中的哪一行？ - 如果某个任务实例的四条 rollout 奖励全是 0，GRPO 会怎样？可以采取哪些补救措施（Slides 第 41 页）？
- 用 Slides 第 44 页的权重视角统一比较 ReST-EM、REINFORCE、GRPO/DrGRPO。
- 本讲的三类 SFT 问题，分别被本讲的哪一部分机制所解决（Slides 第 7、45 页）？
十三、待核事项
以下内容仅靠文本提取无法确认，需对照官方 PDF 与视频核验，本笔记不做猜测：
- 所有公式的精确排版、上下标位置与希腊字母（Slides 第 12、13、19、20、21、22、28、29、37、38、40、44 页）。
- 图示与曲线：Slides 第 7 页 RL 流程图、第 15 页 ReST 三阶段图、第 18 页不可微示意、第 30–33 页的 sigmoid 曲线与梯度采样可视化、第 34 页信用分配示意、第 35 页 reward-to-go 时间轴、第 42 页 PPO/GRPO 对比图。
- Slides 第 1 页的 SWE-bench 标注（演示中含有小字“SWE-bench verified”一类字样）——文本提取不可靠，待核。
- 逐字稿中的若干专名与术语被自动字幕误写（例如 “Swedbench”、“gpo”、“DRGP”、“Dr. gpo”、“Aorva”、“Graham”、“PO”、“deal or no deal” 论文名），语义应以官方 Slides 术语（PPO、GRPO、DrGRPO、ReST-EM 等）为准。
- Slides 第 32 页标注基线
时 SD 0.108 与无基线 0.325 的数值来源与计算细节。 - 逐字稿中提到的具体论文（Williams 1992、ReST/ReST-EM、DeepSeekMath、DrGRPO）在 Slides 中以脚注链接形式给出，具体链接与引用格式待核。
十四、官方来源链接（给定资料）
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频（Lecture 9）：https://www.youtube.com/watch?v=paAcPaaYZGM&list=PLSN0qpDfUvTM&index=9
- 官方课程日程：https://www.cmu-agents.com/#schedule
- Lecture 9 slides：https://www.cmu-agents.com/slides/lecture-09-rl-basics.pdf
从 SFT 到策略梯度：CMU 11-768 Lecture 9 强化学习入门笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
这篇讲稿想解决什么问题？
上周我们讲了如何用监督微调（SFT）训练智能体。SFT 的思路很直观：拿一条人类或更强模型给出的"标准答案轨迹"，让模型学会在每一步都输出那个被示范的动作。但讲者（Daniel Fried）用一整个 Lecture 9 来说明：SFT 的目标和我们真正想要的东西并不一样。
这节课的定位是"三次强化学习讲座中的第一次"（逐字稿开头）。第一次讲最基本的策略梯度方法，第二次讲 off-policy 和稳定性技巧，第三次讲 RL 系统和超参数。贯穿全课的沙盒例子是 Number Search：一个隐藏整数在 1 到 16 之间，智能体最多猜四次，每次猜错环境回复"higher"或"lower"，猜对得奖励 1，超时得 0（Slides 第 2 页）。
这节课的主线可以概括为一句话：把"模仿一条示范"换成"让高奖励轨迹更可能发生"，然后一步步推导出怎么在没有梯度直通、环境不可微的条件下，仍然能对期望奖励求梯度。
SFT 的三个问题
讲者在进入 RL 之前，先回顾了 SFT 的目标：最大化示范动作的对数概率之和（Slides 第 3 页）：
然后他列出 SFT 在智能体场景下的三个不匹配（Slides 第 4–6 页）：
任务不匹配（Task mismatch）。 SFT 最大化的是"每一步示范动作的概率"，但我们想要的是"智能体完成任务的成功概率"。这两者相似但并不相同。原因有两层：第一，完成同一任务有多种方式——Slides 第 2 页给了两条都成功的轨迹 τ₁（8→12→10→11）和 τ₂（4→8→12→11），但一条示范只展示了一种；第二，有些动作明显比其他动作差，比如在"12→lower"之后绝不该猜 13，但 SFT 并不区分那些没有被示范的动作，只要它们不是目标 action，损失是一样的。
数据不匹配（Data mismatch）。 我们希望也能从次优数据中学习。Slides 第 5 页给出 τ₃ 和 τ₄ 两条失败轨迹（一个从 16 开始、一个从 1 开始然后线性递增），这两条轨迹本身不该被 SFT 模仿，但我们确实希望模型能从"这样做会失败"里获得信号。
暴露偏差（Exposure bias）。 SFT 训练时，历史是固定的示范序列；但生成时，历史是模型自己产生的。Slides 第 6 页把这个画得很清楚：训练时是 8 → higher → 12 → lower → 10 → higher → 11 correct，生成时模型可能走到 8 → higher → ?，而它从未见过"自己猜错后"的分布。讲者的原话是：如果只在最优策略上训练，模型一旦采样出一个更低的数字，"可能就完全被打乱了，因为它在训练时没有见过那样的数据"。
RL 的基本框架：轨迹、策略、奖励
RL 的思路（Slides 第 7 页）：让智能体自己生成轨迹，评估奖励，然后调整参数，使高奖励轨迹概率上升、低奖励轨迹概率下降。讲者指出这恰好同时回应了上面三个问题——目标直接是期望奖励（解决任务不匹配）、能从所有轨迹学习（解决数据不匹配）、轨迹来自当前策略即 on-policy（缓解暴露偏差）。
符号（Slides 第 8–12 页）
- 任务实例 x：Number Search 里就是隐藏数字（本课固定为 11）。
- 观测 o、动作 a、奖励 r：环境每步返回观测和奖励，智能体根据历史产生动作。
- 历史 h_t：到第 t 步为止所有动作和观测的序列，
h_t = (o_0, a_0, o_1, …, o_t)。策略 π_θ(a_t | h_t) 只依赖历史，不依赖隐藏的环境状态。 - 轨迹 τ = (o_0, a_0, r_1, o_1, …, a_{T-1}, r_T, o_T)，奖励 R(τ) = Σ r_t。
Slides 第 9 页给了一张跨任务对照表：文本生成里任务实例是 prompt、动作是下一个 token、奖励是评判模型打分；Web agent 里任务实例是任务和网站、动作是点击/输入/滚动、奖励可以是已完成的子任务数；Number Search 里就是猜测。讲者特意强调，虽然大多数设置只在任务结束时给奖励，但课上讲的东西"也适用于具有中间奖励的设置"。
轨迹概率（Slides 第 12 页）是策略概率和环境概率的乘积：
讲者的关键观察是：我们不知道环境概率 P，但可以通过与环境交互来从中采样。这被称为 RL 里"有点神奇"的地方——不需要知道环境模型，采样本本身就提供了求梯度所需的信息。有学生在这里问了 Markov 假设的问题，讲者回答说我们在处理的是 POMDP，下一个观测的概率必须依赖所有之前的观测和动作，因为你看不到底层状态。
学习者解释：这里之所以"神奇"，是因为策略梯度推导只需要 log π_θ 对 θ 的导数，而环境那部分 P 与 θ 无关，求导时直接消失。采样提供的是 R(τ) 的估计，而 R(τ) 不需要可微。两者的分工让"不可微环境"不再是障碍。
目标函数（Slides 第 13 页）
Slides 第 13 页用四条轨迹算了具体的数字：π 给 τ₁ 0.30、τ₂ 0.20、τ₃ 0.25、τ₄ 0.25，前两条奖励 1、后两条奖励 0，期望奖励 = 0.30+0.20 = 0.50。训练的图景就是把概率质量从红色的失败轨迹搬到蓝色的成功轨迹上。
一个简单基线：ReST
在真正推导策略梯度之前，讲者先给了一个"非常简单的 SFT 修改"，叫 Reinforced Self-Training，ReST（Slides 第 14–16 页）。两步循环：从当前策略采样轨迹并计算奖励；用奖励过滤，把成功轨迹加入数据集，在上面做标准的 SFT。Slides 第 16 页的损失就是只在 R(τ)=1 的轨迹上求和的标准 SFT 损失。
讲者明确说这在技术上不算强化学习，但和高层思想很接近，而且"有二元化奖励就很适合"作为起点。有学生追问如何保证一定有正奖励的轨迹，讲者承认这是 RL 算法的核心难题，给出的方向包括：先用示范做冷启动（提到 R1 论文这类推理模型的做法）、课程学习（先训练能产生奖励的简单问题）、以及中间奖励/奖励塑形。
学习者解释：ReST 和后面 REINFORCE 的关系，Slides 第 24 页写得很直白——在二元奖励下，REINFORCE 的损失（R(τ) 只取 0 或 1）恰好退化成"对成功轨迹做 SFT"。区别在于 ReST 会攒一个大数据集反复训练多个 epoch，而 REINFORCE 是采样一批、更新一次。
为什么不能直接反向传播
进入策略梯度前，讲者先解释了障碍（Slides 第 18 页）：
- 动作是离散且采样得到的。改变 θ 会改变动作概率，但只要没跨过某个"阶跃点"，采样得到的动作本身不变，所以样本对参数的导数几乎处处为零。
- 环境不可微。讲者的原话很生动："你怎么对单元测试关于代码求导？"
虽然 p_θ(τ|x) 随 θ 变化，R(τ) 是在 rollout 之后才观测到的标量，两者无法端到端求导。
策略梯度（REINFORCE）推导
Slides 第 19 页把推导一步步写出来。核心是对期望奖励求梯度：
因为 R(τ) 不依赖 θ，梯度只作用在概率上。接着做得分比率技巧（score ratio trick）：分子分母同乘 p_θ，再用恒等式 ∇p/p = ∇log p：
这就是 REINFORCE（Williams, 1992，Slides 第 19 页的引用）。
Slides 第 20 页把期望拆成三项可操作的东西：用采样得到 τ；从环境得到 R(τ) 这个标量； ∇_θ log π_θ(a_t|h_t) 用和 SFT 相同的机制计算。
轨迹对数概率（Slides 第 21 页）
第一项与环境概率相关，不依赖 θ，求导时是常数，直接消失。于是：
还有一处细节：一个动作 a_t 可能由多个 token 组成（比如一次复杂的工具调用），动作的对数概率是各 token 对数概率之和：
实现（Slides 第 22 页）
Slides 第 22 页给了一张带 masking 的示意图：彩色的 token 是策略采样出来的，灰色的是 prompt、观测、脚手架添加的内容。损失是：
讲者感叹说，他第一次看到在复杂任务上用 REINFORCE 的代码（提到一篇叫 “Deal or No Deal” 的论文）时很惊讶，因为实现"基本上就是做一个 SFT 梯度计算，然后把那个梯度乘以一个奖励"。
训练循环（Slides 第 23 页）
采样任务实例 → 在它们上面采样轨迹并算奖励 → 用奖励加权动作对数概率 → 反向传播、更新 θ、重新采集。
奖励应该是相对的：基线（Baseline）
讲者说"人们很少会直接开箱即用地运行标准的 REINFORCE"，最大的动机来自 Slides 第 26 页：奖励应该是相对的。看两条轨迹——一条全部正确（R=1），每条动作都被奖励 1 加权；另一条 τ₃（16→8→9→10 timeout，R=0）。在基本 REINFORCE 里，第二条根本不产生任何梯度，因为奖励是零、乘出来就是零。
但我们明明希望"学会不做这个"。讲者的另一种说法是：更新应该取决于策略在这个例子上通常表现如何。如果模型 95% 的情况下都能拿 1，那 5% 的愚蠢错误应该得到一个很重的反向更新。
优势函数（Slides 第 27–28 页）
Slides 第 27 页把优势定义成：
前者是"采取这个动作之后能拿到的期望奖励"，后者是"从 h 出发的期望奖励"。讲者用 Number Search 举了一个很好的例子：
假设已经猜了 8（得到 higher）、猜了 12（得到 lower），那么数字在 9、10、11 之间，还剩两次机会。此时：
- 猜 11 很糟：它落在范围的一端。如果不对，还剩两个候选但只剩一次猜测，A^π(h, 11) < 0。
- 猜 10 很好：无论对错，剩下一个候选还剩一次机会，可以往高或往低猜，A^π(h, 10) > 0。
Q^π 和 V^π 我们都不知道，所以用采样的奖励来估计优势：
Slides 第 28 页列了几种基线选择：常数、奖励的滑动平均、训练出的价值模型 V_φ(h_t)（即 actor-critic / PPO）、以及组内均值（GRPO，下一节）。
基线不引入偏差（Slides 第 29 页）
Slides 第 29 页给了一个简短的证明。所有只依赖历史 h_t 的基线都不会让梯度估计有偏。证明的关键一步是：
也就是"概率之和恒为 1，梯度为零"。
学习者解释：这个证明的含义是：减去一个和历史相关的常数，梯度期望不变，但样本的方差可以小很多。这正是为什么几乎所有实用的策略梯度算法都会减去某种基线。
双臂老虎机示例（Slides 第 30–33 页）
讲者用一个只有两个动作、只有一个参数 θ 的简化例子把基线的作用讲得非常具体（他说"基线对我来说一直有点神秘，做这些可视化也是为了说服自己"）。
设 p = σ(θ) 是选 A 的概率，θ = −1.1 给出 p = 0.25。A 奖励 1、B 奖励 0。这个策略是坏的，因为它把更高概率放在了没奖励的动作上，我们希望 θ 变得更正。
不使用基线时（Slides 第 30 页）：
- 采样 A：R=1 × ∂/∂θ log π = 1 × (1−p) = +0.75
- 采样 B：R=0 × ∂/∂θ log π = 0
如果采样 B，梯度是 0——这就是讲者一直强调的"失败轨迹不给信号"的现象在最小例子里的体现。
加入基线 b = 0.5 后（Slides 第 31 页）：
- 采样 A：Â = +0.5 × 0.75 = +0.375（无基线是 +0.75）
- 采样 B：Â = −0.5 × (−0.25) = +0.125（无基线是 0）
两个样本都变成正的、都朝正确方向更新 θ，而且都更接近真实梯度 0.1875。
Slides 第 32–33 页给出方差对比：无基线时梯度样本的标准差是 0.325，基线 0.5 时是 0.108，而两者的运行平均都收敛到 0.188（即真实梯度 0.1875 的采样近似）。讲者的总结是：期望不变，方差变窄。
基线的局限与 reward-to-go（Slides 第 34–35 页）
Slides 第 34 页诚实地列出：基线能标记一条轨迹比预期好还是差、能降低方差；但不能解决 credit assignment——同一条轨迹里所有动作共享同一个 Â；而且估计 V^π 可能需要单独的价值模型。
Slides 第 35 页给了一个处理中间奖励的技巧：reward-to-go。如果一个动作 a_t 之后的奖励才与它有关，那就只用后续奖励：
讲者解释说，早于 a_t 的奖励不依赖 a_t，丢掉它们可以降方差而不引入偏差。Number Search 只在最后给奖励，所以对每个动作都有 R_t = R(τ)。
组相对优势：GRPO 与 DrGRPO
（Slides 第 36–42 页）
讲者用最后一段时间介绍了 GRPO（Group Relative Policy Optimization，来自 DeepSeekMath）的简化版本。动机是：好的基线需要估计策略下的期望奖励，而训练一个单独的价值模型很贵（Slides 第 42 页把它对照 PPO）。GRPO 的思路是对同一个任务实例采样多次，用组内奖励的均值作为基线。
Slides 第 37 页给了四条 Number Search 轨迹（两条 R=1、两条 R=0）的完整计算：R̄ = 0.5、σ_R = 0.5，中心化后 ±0.5，再除以 σ_R 得到优势 ±1。
Slides 第 38 页把 GRPO 损失写成：
并强调每个任务实例有自己的 R̄_i 和 σ_i，各自归一化。页脚注明这是"针对当前策略轨迹的写法，不含论文中的 ratio、clipping 或 KL 项"。讲者也明确提醒，原论文的损失有额外项（ratio、clipping、对策略偏离起点的正则），Graham 的下一次讲座会讲，今天只讨论 GRPO 内部的优势函数。
DrGRPO 的两处改动（Slides 第 39–40 页）
讲者说这两处改动"你也会在作业中做"（指 Assignment 3，但本讲稿不涉及具体题目）：
- 去掉除以标准差。理由是除以 σ_R 会让优势不再意识到问题难度——相当于把所有难度范围重新缩放为标准化的。Slides 第 39 页的理由栏写着：σ_R 是组内测量的，所以实例间不同；几乎总是通过或几乎总是失败的实例 σ_R 很小，反而会占更大权重。
- 去掉除以生成长度。原来的 GRPO 会除以 |y_j|，这会让一个又长又错的回答按 token 惩罚得更轻，于是"策略会漂向更长的错误答案"。DrGRPO 改为除以一个全局常数 C（论文里用生成预算）。
Slides 第 40 页给出 DrGRPO 的每 token 系数示意：同一条轨迹的每个生成 token 共享该轨迹的优势：
退化组（Slides 第 41 页）
Slides 第 41 页指出一个重要边界：如果组内所有奖励都相同，优势全为零，贡献的梯度也是零。全失败 [0,0,0,0] 说明对当前策略可能太难；全成功 [1,1,1,1] 说明可能太简单。讲者列出几个可选做法：换任务实例、扩大探索、加入示范，或者提供更有信息量的奖励（奖励塑形，比如给"找到正确文件"发奖励）。
Slides 第 15 和 37 页引用了 ReST、ReST-EM、DeepSeekMath、DrGRPO，均以幻灯片上的论文标签为准。
把这节课的方法放在一起看
Slides 第 44 页给了一张非常清楚的对照表。所有方法都在用某种权重 w_t 加权"智能体实际采取的动作"，区别只在权重：
| 方法 | w_t |
|---|---|
| ReST-EM | 成功轨迹为 1，否则为 0 |
| REINFORCE | R(τ)，轨迹的奖励 |
| GRPO / DrGRPO | Â，相对于同任务实例其他 rollout 的奖励 |
统一的形式是：
Slides 第 45 页把三个问题收尾：任务不匹配（目标直接是期望奖励）、数据不匹配（能用次优轨迹里学到的信息，只要基线合适）、暴露偏差（同策略轨迹让模型学会从自己的错误中恢复）。
结尾的预告
讲者在最后提到下一次讲座会讲 async RL / off-policy data 和 importance weighting（Slides 第 46 页），也就是"放宽 on-policy 假设"的方向——这正是他在讲暴露偏差时埋下的伏笔：on-policy 数据"与模型当前能预测的内容匹配得很好"，但可能有采样效率上的代价。
写在最后
本篇笔记依据 Lecture 9 的逐字稿整理稿和 Slides 文本提取整理。逐字稿注明状态为"待审核；自动字幕错词与图示、公式需对照原视频"，Slides 文本提取也注明"图示、公式和布局需对照官方 PDF"。本文中所有公式和符号均按 Slides 提取文本转写，关键的公式排版、图示细节（如 Slides 第 22 页的 token 着色示意、第 33 页的方差曲线）建议对照官方 PDF 核验，本文不对未在提取文本中出现的数值和结论做任何补充或推测。
给定资料中的官方来源：
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=paAcPaaYZGM&list=PLSN0qpDfUvTM&index=9
- Lecture 9 slides：https://www.cmu-agents.com/slides/lecture-09-rl-basics.pdf
CMU 11-768 Fall 2026 · Lecture 9 · Training 2: Reinforcement Learning Basics｜完整中文逐字稿
- 视频：官方视频
- 来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
- 说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。
正文
[00:00.480] 好的，那我们开始吧。所以今天是关于强化学习入门的介绍，它将建立在我们上周讲过的智能体SFT的基础上。这将是我们关于强化学习的三次讲座中的第一次。在这第一次讲座中，我们将讨论基本的策略梯度方法，这些方法是智能体 与环境交互并获取反馈，然后更新它们的权重，使那些成功并获得高奖励的事情更可能发生，使那些获得低奖励的事情更不可能发生。然后我们会有第二次讲座，实际上是在下周，关于更高级的强化学习算法，讨论离策略方法，以及一些你需要做的来稳定 强化学习并让它工作的事情。然后我们的第三次讲座将关于强化学习系统，深入一些实践框架，以及你在训练这些智能体时需要考虑和关注的一些超参数方面的事情。所有这些应该能为你们很好地准备好第三次作业，它将聚焦于在两种设置下用强化 学习训练模型。所以第一种设置将是一个简单的沙盒式设置，我们实际上会在这节课中把它作为贯穿的例子来使用。它将被称为数字搜索。第二种则是一个简单的有点像Minecraft的合成环境。所以对于这些，它们肯定需要比你们之前的作业更多的计算 资源。所以你们会更大量地使用计算额度，但你们应该在前两次讲座之后就有了开始做那些作业的基础。嗯是的，我们期待看到你们为那些作业所做的成果。我们周四还会有一场客座讲座，来自Akari Asai，她是CMU这里的一位教师，是一位
[01:58.159] 深度研究方面的专家，她会把那个作为智能体的另一个领域来讨论，补充我们到目前为止谈过的那些领域。好的。在我们开始之前，有没有任何快速的流程性问题？好的。是的。所以今天我们会有从监督微调到策略梯度方法的桥梁，后者算是
[02:25.280] 最基础但也最优雅的强化学习形式之一。所以我们将使用的贯穿例子是这个我们称为数字搜索的任务，它基本上就是训练一个智能体基于关于你是太高还是太低的反馈来猜一个数字。所以这是一个相当简单的任务，但你知道，容易理解，也容易想出一个合理的策略 来处理它。所以设置是这样的：有一个数字，它是这个任务环境的一部分，但智能体并不知道，它是一个1到16之间的整数。智能体将有四次机会来猜这个数字。在它做出猜测之后，如果它猜对了，环境会说正确。但否则，环境会说，嗯，你应该往高猜，或者 你应该往低猜。然后智能体可以再做一次猜测，总共最多四次猜测。如果在任何时候智能体猜对了，任务就结束，它获得奖励1。但如果你用完了四次猜测还没有猜对，那么奖励就是0。
[03:30.480] 好的。所以，我们将要走过的很多轨迹都来自一个例子，其中隐藏的数字是11。所以智能体会尝试猜它。这里有四次可能的尝试，在第一次中智能体说8，得到反馈，你应该往高猜。它说12，得到反馈，你应该往低猜。嗯10，往高，然后 11，所以它在最后一次猜测中猜对了。在另一次尝试中，我们可能从更低的数字开始。所以这实际上在这里什么是最优策略？最好的起始数字是什么？中间，对吧？所以这有点像20个问题。你想要切分空间，对吧？所以最好的第一次猜测是8。但嗯，实际上你们中有多少人玩过Wordle？ 你知道，就是《纽约时报》的那个猜词游戏。这是个有趣的游戏。它有点在类似的空间里，你可以做一个最优的第一次猜测，它以信息论的方式缩小空间。这里概念类似但更简单。所以你可以从一个你知道次优的第一次猜测4开始，但你仍然可以成功完成这个任务，通过 考虑这个反馈，往高走，然后往高走，然后往你知道低走，最终猜对它，但你也可能出错，对吧，所以我们可以从一个非常糟糕的猜测16开始，嗯除非我们猜对了，反馈会是往低，在这种情况下我们可能会超时，所以我们在四次猜测内没猜对，类似地
[05:05.199] 类似地，我们可能有另一个相当糟糕的策略，从1开始，然后2、3、4，嗯然后超时，任务失败。大家对设置都清楚了吗？好的，很好。所以，我们也会为轨迹设置奖励。奖励只是一个标量值，它告诉你那条轨迹有多好或多坏。所以在智能体 设置中，这可能对应于比如你写的代码是否通过了例如Swedbench中所有的单元测试，如果通过了所有测试你就得到奖励1，如果没有通过所有测试就得到奖励0，或者你也可以有你知道不是0或1的值，不是二元奖励，比如你通过的 测试的比例是多少，但在如今人们训练的很多领域中，在所谓的RLVR，即带可验证奖励的强化学习中，奖励要么是0要么是1。嗯但今天我们将讨论的方法更通用。嗯它们也适用于标量值的奖励。
[06:11.520] 所以快速提醒一下SFT是如何工作的。所以在监督微调中，你有一个任务的示范，你训练智能体通过最大化被示范动作的概率来复制那个示范。对吧？所以如果我们把一条轨迹分解成这些步骤，我们有一个给智能体的提示。嗯我们有一个猜测，嗯来自 环境的反馈：往高。我们有另一个猜测，来自环境的更多反馈，等等。我们的任务是，给定这些被示范的动作，我们会用星号标记，我们想要最大化智能体在给定到那个点为止所有过去历史的条件下分配给那些动作的概率。对吧？所以这将只是你知道的监督 微调损失是最大化该序列概率的对数，也就是各个动作概率对数之和，对吧，而我们有很多种方式可以获得这些我们试图模仿的目标动作。所以它们可以来自给出任务示范的人。它们可以来自我们试图复制的更强的模型
[07:21.759] 以进行蒸馏。它们也可以来自智能体自身的轨迹，我们试图在模型中强化这些轨迹。但这希望是你们对之前LLM训练课程以及SFT讲座的复习。好的。但SFT存在一些问题，而强化学习将要解决这些问题。所以大致有三个方面 我们将重点关注。第一个方面可以称为任务不匹配。所以监督微调最大化的是每个时间点上示范动作的概率。但我们真正感兴趣的是最大化智能体成功完成任务的概率。 这两者是相似的，但并不完全相同。这有几个原因。第一个原因是，成功完成任务有多种方式，对吧？所以，呃，从8开始算是最好的做法，呃，作为第一个猜测，但是，呃，从，你知道，4开始也是合理的
[08:29.840] 只要你能，呃，把反馈考虑进去。而对此，你知道，在这个第一步反馈之后，我们也有各种合理的动作可以采取。而一个示范，无论它来自，你知道，一个人还是另一个模型，只展示了解决问题的一种方式，但我们希望智能体能够呃也潜在地使用其他方式。 第二个问题是，你可以采取的某些动作比其他动作更差，但对SFT来说，它们看起来都一样。所以如果我们在反馈说应该更高时，输出一个低于8的数字，那将是非常糟糕的，对吧，但SFT不区分那些未被示范的动作 所以如果呃，你知道，智能体在一个更低的备选动作上概率最高，而不是在一个更高的、但并非被示范的那个备选动作上概率最高，它会得到相同的损失。所以这就是SFT损失与我们真正感兴趣的目标之间的不匹配问题之一，即最大化你知道的期望奖励或
[09:46.000] 任务成功的概率。第二个问题是数据不匹配。所以我们希望能够从呃次优数据中学习。即使我们假设我们的示范是完成任务的最佳方式，你知道，如果我们能获得其他不那么最优的、甚至可能不成功的示范，那些也能为我们的模型提供 有用的信号。所以比如呃，假设我们知道这是一个糟糕的轨迹，比如它超时了。我们不想仅仅对它做SFT，对吧？我们不想最大化那些动作的概率，但我们可能想要呃有一种学习方法能够呃让模型不采取这样的轨迹。而强化学习将是 实现这一点的一种方式。第三个问题就是所谓的暴露偏差。所以在SFT中，你训练模型在给定固定的过去动作序列的情况下预测下一个动作，对吧？那个固定的动作序列来自某个地方。但在评估时，之前的动作将由模型自身生成，对吧？ 因为你必须一次产生一个动作。你有你过去产生的动作。这就是与环境交互的自回归过程。所以如果智能体呃产生了一个与它所训练的历史分布非常不同的动作，它可能就没有——它产生的响应可能就没有 经过校准，因为它从未见过那样的数据。所以假设我们只在示范上训练，其中呃下一个整数是，你知道，确实与我们得到的反馈匹配的。所以如果反馈是更高，它就总是更高。现在如果在生成时模型犯了一个错误，采样了一个更低的数字，它可能就完全被打乱了，因为它在训练时没有见过那样的数据
[11:54.240] 如果我们只在那些最优策略上训练的话。这合理吗？所以看待强化学习的一种方式是，它是一种让模型生成自己的训练数据、并学会如何从自己犯的错误中恢复的方法，这样当你在生成时，它就准备好处理自己的错误。而呃这种让模型生成 自己的训练数据的特性被称为同策略（on-policy）。这通常是一件好事，呃，因为它让模型准备好，你知道，处理自己的错误，而且你获得的监督信号与模型当前能够预测的内容很好地匹配。呃但它可能有一些效率上的权衡，在Graham的讲座中我们会讨论一些——实际上会讨论很多关于呃
[12:39.600] 放宽这些假设、更加偏离策略（off-policy）的方法。好的。所以这些就是呃监督微调用于智能体和序列预测的三个问题。关于这些有什么问题吗？好的。那么，强化学习的一般思想是，不是像呃一条固定的轨迹那样，你通过你知道的 最大化动作的概率来学习模仿它。我们将让智能体或者你知道的模型呃生成自己的动作，然后我们会评估这些动作在轨迹中的奖励，呃然后我们会调整智能体的参数，以便呃它使高奖励的轨迹更可能发生，低奖励的轨迹更不可能发生。所以形式上，我们稍后会看到 你知道这个的公式，我们将训练智能体最大化其期望奖励，你知道，在智能体下和环境下的概率期望。所以呃，仅仅示意性地，如果我们有你知道一个像这样的任务，在这个数字搜索游戏中，环境中的底层目标是B猜数字11，我们会让智能体生成多条 轨迹，就是通过呃在每一步从智能体产生可能的动作。你知道，就像我们通常会在环境中评估智能体的方式一样，对于每一条轨迹。呃我们会通过呃在环境中执行这些动作来获得一个奖励，然后我们会用那个奖励作为强化学习方法的反馈，这将改变
[14:26.079] 智能体的参数，嗯，使得这些高奖励轨迹更有可能发生。所以这将解决我们嗯列出的监督微调（SFT）的三个问题。任务不匹配问题得到了解决，因为我们现在直接训练智能体来最大化期望奖励。我还没有向你们展示这是如何做到的。讲座第一部分的大部分内容将围绕我们如何做到这一点展开。我们将 构建期望奖励的公式。我们将对它求导。我们将直接优化它。这也会解决数据不匹配的问题，因为正如我们将在讲座中间和结尾部分看到的那样，智能体现在能够从所有轨迹中学习。所以它会从这些糟糕的轨迹中获得一些信号，并在未来避免采取导致此类轨迹的行动。 这也有助于处理暴露偏差，因为智能体现在将生成用于训练的轨迹。它将是同策略的。所以如果它犯了错误并看到这些错误导致糟糕的奖励，它将因此受到惩罚，然后学会在未来避免这些错误。所以它在某种程度上已经准备好应对自己的错误。
[15:38.160] 是的。嗯是的，就像人们在大约三四年前对语言模型遇到的很多问题，比如嗯它们有嗯那种重复偏见。它们会陷入循环。一旦人们开始在强化学习中进行最终的后训练步骤，很多这些问题就得到了解决，此时模型现在能够
[16:00.240] 从自己的错误中恢复。好的。那么我们将讨论我们如何，你知道，将与环境交互形式化为轨迹，这将让我们能够建立我们优化的目标。好的。嗯我们不会在这上面花太多时间，因为这应该只是课程第一部分的回顾。 但我们有一个智能体，它将嗯被表示为嗯一个策略，我们称之为嗯π theta。Theta是驱动智能体的语言模型的参数，你知道的，智能体将嗯通过从语言模型采样来产生行动，嗯然后你知道的，将这些嗯嗯采样结果应用于它所运行的框架中，那将产生 行动嗯，然后你在环境中执行这些行动，环境给你返回观测和奖励。我们将研究的很多设置只在任务结束时给你奖励，比如当你实际执行测试用例时。嗯，但我们讨论的几乎所有内容也适用于具有嗯中间奖励的设置，即对于 你采取的个别行动，你也可以获得一些关于嗯该行动价值的奖励。但只是一些例子来说明如何将这个通用框架应用于文本生成。嗯，任务实例可能是你试图产生补全的提示。观测可能是你目前已有的token。行动可能是下一个token。奖励可能是
[17:40.240] 你从评判模型得到的分数。例如，如果你在做嗯基于人类反馈的强化学习，用于一个网页智能体，就像我们一两周前讨论的设置，任务实例可能是任务和网站。这里环境的状态——即决定环境的一切——和智能体所能 观测到的之间存在差异。可能存在一些隐藏状态，比如后端数据库的内容是什么，但智能体只能观测到它当前所在的页面。它可以采取诸如点击、输入或滚动之类的行动。它可能获得奖励，比如中间奖励，比如任务描述中它到目前为止已完成的子任务数量。而对于 数字搜索，我们已经嗯看到任务实例是智能体试图猜测的数字，这个数字将对智能体隐藏。所以在数字搜索中，每个任务在开始时对智能体来说看起来都是一样的。嗯观测将是偏高、偏低或正确。
[18:41.280] 行动将是下一个猜测。如果智能体猜对了就给奖励，那是一分，嗯如果超时则得零分。所以给定这些嗯这些观测和行动以及任务，我们可以开始建立一些符号，这将让我们推导出我们试图最大化的嗯奖励项——期望奖励项。所以一条轨迹是一个 交替的行动和观测的序列，我们将嗯在给定时间步之前的观测和行动序列表示为历史h。所以历史ht将包含到那个时间点为止的所有行动和观测。而我们的智能体策略π theta——记住theta是智能体内部语言模型的参数——那将给我们一个 在特定历史下我将采取的嗯下一个行动上的分布。对吧？所以你把历史输入你的智能体框架，你得到一个下一个token上的分布，这些token对应环境中的行动。关于这些有什么问题吗？
[19:54.480] 好的。所以我们还有奖励，在最一般的设置中，你可以在沿途每嗯个行动或观测之后获得奖励。这些公式，无论是行动之后还是观测之后，彼此可以是等价的。嗯在数字搜索设置中，我们只在你嗯要么超时要么得到正确反馈之后才获得奖励。所以
[20:18.080] 在数字搜索设置中，你的奖励将是二值的，并且它们只会在轨迹的末尾出现。我们将轨迹的奖励写为所有中间时间步上奖励的总和。所以tau——哦——将表示轨迹，也就是你知道的这些观测、行动和奖励的序列。 好的。这就是符号，给定这些我们也可以你知道的，有轨迹上的概率，一条轨迹的概率将取决于两件事。所以第一件事是智能体分配的概率。所以我们可以将其分解为各时间步上概率的乘积，其中嗯h——抱歉我在这里漏了一个p，但嗯这一项前面应该有一个p 那是环境在给定这个历史、智能体在那个时间步采取的行动以及任务的条件下，得到这个观测和这个奖励的概率，来自环境的那个概率我们将乘以来自智能体策略的、产生这个下一个特定行动的概率。所以这 just 是链式法则嗯分解这个 将生成这整条轨迹的概率分解为其组成部分概率的乘积。RL 的一个关键点是，我们实际上并不知道，我们不必了解关于环境的任何信息。我们不必知道，比如说，得到正确结果作为下一个观测的概率是多少，或者在我与 Amazon.com 之类交互时得到某个下一个观测的概率是多少。RL 的关键在于，我们将与环境交互，通过采取动作，这将成为从观测和奖励的分布中采样的一种方式。而这实际上将提供足够的信息，使我们能够学会如何对依赖于这个分布的、关于我们参数的
[22:27.039] 目标函数求导，并更新参数。从某种意义上说这有点神奇，但这也使得强化学习算法有点慢，因为你基本上只能去尝试各种东西，看看会发生什么。嗯，这就是为什么强化学习需要大量算力来运行，因为你必须做这些实验，嗯，与环境做实验，看看 会发生什么，并把它作为一种间接的反馈来源，来了解你的奖励如何依赖于你的参数。关于这一点有什么问题吗？>> 是的。>> 是的。所以我嗯把这里改写了一下，所以问题是，在经典 RL 中有一个马尔可夫假设，嗯，它说的是到下一个状态的转移只取决于前一个状态
[23:21.679] 以及动作。所以我们实际上在这里处理的是一个 POMDP，一个部分可观测马尔可夫决策过程，嗯，其中存在一个区别：底层状态你看不到，而观测你可以看到。所以在 POMDP 中，嗯，你的下一个观测的概率必须依赖于所有之前的观测和动作，因为你无法访问
[23:45.520] 状态。嗯，但对于 POMDP 来说，这些表述应该是等价的，而这个只是对我们来说稍微更容易处理一些。好问题。好的。是的，抱歉那里少了个 P。好的，所以我们的目标将是最大化期望奖励。嗯，那么这意味着什么呢？所以我们想做的是，我们想嗯从 我们的智能体那里采样，就像我们想让智能体在环境中生成轨迹。嗯，而且嗯我们通常会通过从语言模型中采样嗯动作，然后把这些动作送入智能体框架来实现这一点，对吧？嗯，所以我们可以把它写成从这个分布中采样一条轨迹，这个分布是由 语言模型本身参数化的。对于我们从那个分布中采样得到的每一条轨迹——这是一个非常复杂的分布，它依赖于智能体、依赖于语言模型、也依赖于环境——我们会得到一条轨迹，我们会评估那条轨迹的奖励，这给我们一个数值。我们想要最大化我们
[24:59.840] 在这里获得高奖励的机会。所以那就是最大化这个期望。我们就直接把这个目标写成 J of theta，因为它是我们语言模型参数的函数。所以在我们这个非常简单的设定中，嗯，让我们嗯，你知道，我们有了这个 number search，我们假设只有四条轨迹实际上是语言模型可以生成的。所以 它学到的是一种非常尖锐的策略。嗯，假设它把概率 0.3 放在第一条轨迹上，0.2 放在第二条上，0.25、0.25 放在第三和第四条上。嗯，一般来说蓝色将表示成功，奖励为一，红色将表示失败，奖励为零。 所以，如果这是智能体唯一能生成的这四条轨迹，那么期望奖励将是 0.5。对吧？所以就是，你知道，对这些轨迹奖励按智能体策略下的概率做加权求和，而训练的目标将是对其进行优化。所以我们想要调整参数 theta 来最大化 这个期望奖励。所以你可以把这看作，这是一个当前比较糟糕的智能体，因为它有相当高的概率 0.5 生成一条奖励为零的轨迹。我们想要转移其中一部分概率。我们想要理想情况下把所有这些概率都转移到预测那些奖励为一的轨迹上。所以就是把概率质量从红色转移到蓝色。
[26:32.960] 这显然是简化过的，因为你知道，实际上智能体可以生成的数量不是无限的，但会是组合意义上指数级大的可能轨迹数量，不过我们在这里只是稍微简化一下可视化。所以你可以做到这一点的一种方式，是对 SFT 做一个非常简单的修改，而且它有很好的动机。它叫做专家迭代。 嗯，专家迭代的一种特定类型叫做强化自训练，简称 rest。所以在 rest 中的想法是，我们将从我们当前的嗯策略、从语言模型的当前参数中生成数据，然后我们会选择那些成功的轨迹，并把它们用作微调数据，并在 这些数据上更新模型。这就是 reinforced 这个词的来源。它会以某种方式强化智能体已经在产生的正确行为。嗯，所以它有两个步骤，这两个步骤将被迭代进行。嗯，rest em 实际上被命名为 rest，我想是 expectation maximization，因为它看起来和 EM 算法非常相似，嗯，你们中的一些人可能在 机器学习课上见过。但其想法是，我们将在这两个步骤之间交替进行。第一步是你从当前策略中采样轨迹。你得到它们的奖励。第二步是你将嗯让模型通过仅仅对收集到的这些轨迹做一堆 SFT，使那些嗯轨迹变得非常可能。嗯，然后你重复，你会从 当前策略中获得更多轨迹，并把这些轨迹加入到你已有的好的轨迹集合中，然后你会嗯在所有这些轨迹上训练你的模型。所以你有了这个不断增长的数据集，而你正在让模型嗯迭代地变得非常擅长生成它们。
[28:32.799] 嗯，是的，所以这就是一个高层次的描述，嗯，这在机制上起作用的方式是，你会生成一堆轨迹。你会嗯评估每一个的奖励。你会嗯使用像阈值这样的方式来筛选，如果你的奖励是二值的，这就很容易。嗯，如果它是二值的，你就只保留奖励为一的轨迹，然后扔掉 那些奖励为零的轨迹。然后你把所有实例中保留下来的这些轨迹组合起来。这些就是你下一轮训练集的。所以目标就只是 SFT 目标。我们想要最大化这些轨迹中这些动作的概率。那个奖励为二元奖励一的情况。
[29:23.600] 这说得通吗？对此有什么问题吗？所以，从示意图上看，这实际上会看起来和我们接下来要讨论的强化学习方法非常相似。嗯，这在技术上并不被称为强化学习。嗯，但它确实与之有很多相似之处，因为你正在从模型生成数据。嗯，你正在以某种方式基于那些数据更新 这种方式由轨迹的奖励决定。嗯，我们稍后会看到强化学习如何以更精细的方式来做到这一点。但这是一个非常简单且相当有效的嗯基线，你可以从它开始，它复用了监督微调的所有相同机制，并且如果你有二元化的奖励，它就很适合。
[30:12.720] >> 是的。那为什么这是自？ >> 所以与自蒸馏的区别就在于你是在用奖励来做过滤。自蒸馏也可能指嗯在模型产生的所有轨迹上进行训练。 >> 是的。所以它被称为自训练，因为它看起来很像嗯你知道的类似那样，但它是被强化了的，因为你在使用奖励，所以你在 强化好的轨迹。是的。我的意思是这是一个更基础的问题，但你怎么能保证确实有具有正奖励的轨迹呢？ >> 如果比如说在最坏的情况下你只有一堆糟糕的轨迹呢？ >> 是的。所以这是一个很好的问题，比如你如何保证你的一些轨迹有好奖励？嗯，如果
[31:01.200] 你的模型无法产生具有好奖励的轨迹，那么强化学习和这个方法嗯可能就不太适合。所以你可以做几件事。一是你可以嗯有一些好轨迹的示范，你在这些示范上训练模型。所以你知道比如嗯 R1 论文嗯以及很多推理模型嗯会有一个初始的他们称之为冷启动阶段 在这个阶段他们会在开始做强化学习之前先在推理轨迹上训练模型。所以这会让强化学习或类似方法更容易产生具有高奖励的轨迹。你也可以像是看课程学习。所以你可以嗯选择那些确实有足够多能产生奖励的轨迹的问题。先在这些上面训练，然后随着 模型变得更好，你就能解决更难的问题。你也可以有像是中间奖励，比如如果你对问题有一些了解，并且知道先探索代码仓库是有用的。你可以因为模型这样做而给它奖励，然后希望等它学会这样做之后，它会在某种程度上解锁解决整个任务。但这是一个非常大的
[32:08.480] 嗯问题，强化学习算法必须应对，而这些是一些解决方案。是的。酷。好的。所以现在嗯让我们嗯真正进入策略梯度方法。所以我们感兴趣的是最大化这个量，嗯，J of theta，也就是奖励的期望。我们不能直接使用反向传播来做这件事嗯，原因有两个。第一个是我们正在 从我们的策略中采样动作。第二个原因是我们随后在这个像是黑箱环境中执行这些动作，这个环境潜在是、几乎总是不可微的。嗯，所以这破坏了我们的反向传播能力，去你知道直接你知道像你在机器学习课上学到的那样使用链式法则来得到嗯你的奖励 相对于参数的导数。所以嗯动作是离散的，而且它们是采样得到的。所以嗯一个样本相对于参数的导数是几乎处处为零的，因为你知道参数会改变概率，但在你到达某种阶跃点之前，不会嗯改变采样得到的动作，而且环境通常是不可微的。所以一般的解决
[33:32.000] 方案实际上是一个相当经典的方法，叫做 reinforce，嗯，它来自 Williams 在 1992 年的工作，嗯，它将使用这个嗯期望奖励的定义，然后基本上只用一些代数技巧来嗯计算一个估计值，这个估计值使用嗯来自模型的样本，以及嗯我们之前看过的相同的反向传播过程，来得到偏，你知道
[34:03.600] 来得到相对于这些参数 theta 的偏导数，我们会在下一张幻灯片上展示如何做到这一点。所以这就是我们想要优化的目标。它是这个在我们的环境和我们的策略下对轨迹的期望，以及这些轨迹奖励的期望。所以如果你回想一下期望是什么 它是对这些东西、这些轨迹嗯的一个求和，嗯，求和的是这里的这一项，也就是奖励，但按该轨迹的概率加权。所以这是在一个非常大的空间上的求和，比如语言模型所有可能的轨迹。我们几乎永远无法精确计算它，但如果我们要计算它，这就是它会得到的结果。对它取 梯度。所以我们只是，你知道，对参数 theta 取偏导数。嗯，我们可以把那个嗯移到求和里面，而这个嗯奖励项只依赖于轨迹。所以一旦我们知道轨迹，我们就知道奖励是什么。
[35:16.480] 它不再依赖于参数了。参数产生轨迹。嗯，但随后奖励只是轨迹的一个函数。所以嗯这个奖励项会乘以这个概率的偏导数。但这仍然很困难，因为这是对一个潜在非常大量轨迹的求和。所以我们 要在这里做一个代数技巧。这叫做得分比率技巧。嗯，我们要只是嗯乘以并除以这一项 p of theta。所以我们这里有那个，我们在分母里也有那个。它们抵消了。所以这跟那个是一样的。然后我们可以使用这个技巧，也就是嗯 log p 相对于 theta 的偏导数。我们可以直接用链式法则 它就是嗯 p 的偏导数除以 p。所以我们可以在这里使用这个恒等式，然后我们得到这一项。这将会有点神奇，因为我们现在有了这个，它将是一个对轨迹的期望嗯乘以一个奖励项再乘以这里的这一项。
[36:27.839] 而且，嗯，这个项将会是——这与我们在SFT中所做的计算相同。对吧？在SFT中，我们也是对一堆单元token的对数关于语言模型的参数求偏导数。而且，嗯，我们将能够通过采样轨迹然后计算这里的这个项来估计这个量。而
[36:57.440] 这将给我们一个关于这个期望奖励的梯度的估计。对此有什么问题吗？>> 所以这非常通用，不对语言模型或环境做任何假设。>> 是的。>> 你是要说一系列的状态和动作，但这些动作是由大模型生成的，它们也是序列？>> 是的，协调这一点的常规方式 >> 是的，我们会在几张幻灯片中分解这一点，但没错，这是个好问题。所以，嗯，我们就先把那个项复制到这里，然后我们会把它拆解成各个组成部分。所以，嗯，这些轨迹将是通过让语言模型以及在其之上的智能体与环境交互来采样的，嗯，这就是你知道的，当你在 只是运行这个智能体时会使用的相同过程。那会给你一条轨迹。嗯，而我们，呃，这应该是——这应该是对轨迹的一个求和，应该是对轨迹的平均，因为我们可以通过从该分布中抽取一定数量的样本，然后对期望内部的量取平均来估计一个期望。所以如果我们对从该分布中抽取的一堆轨迹取平均，这个 这里的量，嗯，那么我们就会得到这个梯度的估计。那么，嗯，这里面这些项是什么？嗯，由于这是一条轨迹，我们把它分解成步骤。所以它将是按时间步求和，嗯，由那个特定动作所导致的梯度。我们将在幻灯片中看到如何计算它。
[38:54.160] 所以是的，这里三个组成部分是轨迹。我们通过采样得到它们。嗯，任务级别的评估，我们从环境中得到。然后是这里的这个梯度，它将使用与我们在监督微调中获得梯度相同的方式产生。但关键之处在于，这里的这些动作将是从智能体采样的动作，而不是
[39:18.800] 由，你知道，某种真值演示的动作。好的。所以是的，我们如何得到轨迹的对数概率？呃，我会稍微快速地过一下这个。要记住的关键是，轨迹的概率既取决于来自语言模型加智能体的概率，也取决于从环境中得到 特定观测和奖励的概率。但关键在于，这个概率不以任何方式依赖于语言模型的参数。所以如果我们对这个轨迹的对数概率关于参数求导，这一整项就直接消失了，因为它是关于
[40:10.160] 参数的一个常数。所以我们可以直接忽略它，而轨迹的这个梯度就是轨迹内各动作梯度的求和。所以这主要只是一种为了严谨性的记账。嗯，但有一件重要的事情需要考虑，就是由智能体产生的这些动作，嗯，可能对应于你的语言模型的多个token， 对吧？所以比如也许你需要说，呃，你需要说你正在做一个复杂的工具调用，而它会拆分成多个token。嗯，当然，这些token的对数概率只是被加起来，以产生整个动作的对数概率。
[41:02.319] 而当你有了轨迹的这个梯度，一条轨迹的梯度时，你可以直接乘以奖励，即来自环境的一个标量值，来得到你采样的这条轨迹对期望的贡献。那么这在实际例子上运行起来，嗯，实际看起来是什么样的呢？所以你会，嗯，有你的 轨迹，它是从智能体采样的。嗯，这些蓝色的token是实际由模型生成的。所有灰色的token要么是提示的一部分，要么是环境的一部分，要么是由脚手架添加的。但这些是实际通过从语言模型生成而产生的。而那些将 是对策略梯度损失有贡献的。所以我们将有所有这些生成的token。我们将得到，嗯，在那一刻从语言模型在特定token上的分布中选出的那个生成动作的对数概率。然后我们只需将所有这些token对数概率按对应于这条轨迹的奖励进行缩放。
[42:28.800] 这讲得通吗？然后如果你对它求梯度，梯度会传进去并作用到这里的这个对数概率上。所以你的梯度将是奖励乘以梯度。所以当我第一次看到在一个相当复杂的任务上实现reinforce的代码时，它像是使用RL来优化对话模型，来自一篇叫deal or no deal的论文。它 真的有点令人惊讶，实现竟然如此简单，你基本上就是做一个SFT梯度计算，然后把那个梯度乘以一个奖励，或者把损失乘以一个奖励并对其求梯度。但它确实有效，数学上也成立。
[43:13.680] 是的。所以我们这里会有一个掩码，你知道，你只用掩码来选择你生成的token，然后你按轨迹的奖励对所有这些进行缩放。这就是你实现reinforce策略梯度算法的方式。所以训练循环就是你采样任务实例，在它们上面评估奖励。嗯，使用 我们刚刚展示给你的那张幻灯片得到损失，然后，嗯，呃，对你这批示例中的所有实例做一次更新。稍后，当我们讲到gpo时，我们会讨论如何为一个给定的任务示例I生成多条可能的轨迹，然后在更新中使用所有这些轨迹。但如果你只是做标准的reinforce，你会
[44:06.000] 只对每个示例使用一条轨迹，然后，嗯，从中计算更新。所以你，嗯，批量大小将是你训练所用的任务实例数量。嗯，但当我们讲到gpo时，嗯，你还会每个任务实例有多个样本。对此有什么问题吗？
[44:29.760] 好的。是的。所以，嗯，很快地说，对于二元奖励，这看起来很像我们讨论过的rest，嗯，过程。这两者都只是，嗯，用奖励来缩放动作的对数概率。区别在于，这个reinforce方法是完全在线的。你是在采样这些轨迹，然后直接对它们进行更新。而 REST，你构建一个非常大的数据集，然后在所有这些数据上训练，也许做一整个 epoch 或多个 epoch 在所有这些实例上。但它们的直觉基本上是一样的。好的，但这也有很多问题。所以人们很少会直接开箱即用地运行标准的 reinforce 策略梯度。
[45:17.920] 所以我认为最大的——接下来我们要讨论的方法最强的动机是思考奖励应该如何——你从奖励中做的更新应该真正是相对的。让我们看看这两个不同的轨迹。这个成功的，所以如果我们使用我们讨论过的更新规则，所有动作都会被奖励 1 上加权。 这个是不成功的。所以我们第一个猜测很差，最终超时了。如果我们只是使用基本的 reinforce 或 rest，我们根本不会基于这个例子做任何更新。对吧？因为它被奖励缩放了。奖励是零。但我们实际上希望能够学会不做这个，对吧？所以
[46:06.800] 下一部分的直觉将是关于平移我们的奖励，这样我们仍然能得到梯度的正确估计，但我们能够从所有不同尺度或平移的奖励中学习。另一种思考方式是，你做的更新应该取决于策略在这个例子上预期表现有多好。
[46:31.200] 所以假设这只是一个简单的例子，模型大约 95% 的时间在这个例子上得到奖励 1。如果 5% 的时间你犯了一个非常愚蠢的错误得到奖励 0，你应该对那些代价高昂但罕见的错误进行非常重的反向更新。我们接下来要讨论的基线和优势将是一种做到这一点的方法，它将 把奖励与你期望从策略在这个例子上运行得到的奖励进行比较。其数学看起来像这样。我们将定义一个优势，它将是到特定时间步的历史和动作的函数。那就是说，你采取这个动作——你采取这个特定动作，然后你从 语言模型采样剩余的轨迹，这会告诉你——你把它与如果你只是从语言模型采样包括那个动作然后继续下去进行对比。这会告诉你这个特定动作有多好，相对于——考虑到语言
[47:49.440] 模型将在未来采取所有剩余动作，这个特定动作的价值是什么。所以也许一个具体的例子会有所帮助。假设我们在这个猜数字游戏中进行到一半，我们猜了 8 得到反馈更高，猜了 12 得到反馈更低。所以我们知道数字必须在 8 和 12 之间。所以它可以是 9、10 或 11。所以我们 不知道这个价值函数确切是什么。它取决于我们的策略是什么。但我们可以想，它可能是 2/3 或 1 左右，对吧？所以如果策略做最好的可能的事情——总是——你应该能够总是从这个情况中获胜。但如果它是在有效范围内更随机的策略，它可能更像 2/3。 但我们可以思考特定猜测相对于这个基线的优势函数。所以猜 11 会很差，因为它在这个范围的一端，如果 11 不正确，那么我们有两个候选而只剩一次猜测，所以这个优势应该很低，而 10 会很好，因为无论——即使这是错的，我们还剩一次猜测 我们可以往高或往低猜。所以很多强化学习算法都源自于估计这些优势和价值的不同方式——这个优势作为动作的个体 Q 值和价值的函数，我们将在本讲座剩余部分讨论其中一些。但一个关键想法是，如果我们不知道
[49:39.040] qpi 或 vpi，我们可以通过行动来估计这个特定动作的期望奖励，我们可以使用我们将称之为基线函数的东西来估计这个我们预期表现有多好的价值。我们将在接下来几页幻灯片中看到一些基线函数的例子。
[50:07.359] 所以基线只是一种重新缩放奖励的方式。所以这是期望回报的 reinforce 估计器。它用奖励来缩放——对数的对数概率，使用奖励来重新缩放对数概率的梯度。这些基于优势的方法通常会有这里这种形式，从这个奖励中减去 一个基线。这些是不同的——它们在数值上彼此不等价，但它们都是估计策略梯度的不同有效方式。我们接下来会有一页幻灯片展示这也是同一个量的有效无偏估计器。 但对于我们可以定义的不同基线，我们会得到不同的强化学习算法。所以我们接下来会有一个例子，它在某种程度上说明了为什么减去一个基线是一件好事。在那个例子中，基线只是一个常数。基线也可以是你目前为止从所有见过的例子中得到的所有奖励的运行平均值。 它也可以是一个训练好的神经网络模型，这就是 actor critic 方法或一种叫做 PO 的方法的情况，我们可能会在下一讲中涉及。但你需要一个与你的策略分开的模型，这可能会很昂贵。在本讲座的最后部分，我们实际上不会有单独的模型，但我们只会通过从策略本身采样来估计
[51:49.280] 策略下的这个期望奖励。那就是 gpo 算法。好的。所以，关于这个有什么问题吗？我们也会在接下来几页幻灯片中看到关于基线的更多动机。[喷鼻声] 所以，只是关于基线和偏差的快速说明。
[52:24.319] 所以这两个项都是对底层期望奖励的估计。一个估计可能有偏差。估计有偏差和方差，对吧？偏差就像这个估计的均值可能偏离真实值多少。方差就像我为一个估计抽取的值有多大的分散。所以你可能会担心 关于估计器中的偏差。有一个引理表明，减去一个仅依赖于到该点为止的历史的基线——我们在这里定义的所有这些基线都满足这个条件——不会使梯度产生偏差。理解这一点的方法是，这是我们的优势，是我们基于基线的期望回报估计，我们可以把这个期望分解成这里的这些项。 所以，利用结合律，把这个放在这里，那是偏导数的定义，那是J的梯度减去这里B乘以这个的期望，如果这里这一项的期望为零，那么这个就等于这个。我们可以通过简单地取 这一项，然后把期望写成这样来看出这一点。它是关于从该策略中采样的动作的期望。把不依赖于到该点为止历史之外的任何东西的基线项提出来，我们就得到动作的概率之和。
[54:26.480] 然后我们可以使用同样的对数导数技巧，把它重写成这样。然后这就变成一，它会是基线项乘以一的偏导数，也就是零。所以这告诉你，你可以减去一个仅依赖于到该点为止的动作历史——到该点为止的动作和观测——的基线，而不会使你对策略梯度的估计产生偏差。
[54:56.559] 但让我们看一个例子，看看这些基线实际上如何能减少你梯度估计中的噪声。这实际上非常重要，因为强化学习中大量的工作都是在减少你梯度估计中的方差，
[55:24.720] 以使学习稳定，并且对你从策略和环境采样中得到的所有变化具有鲁棒性。但为了引出这一点，我们来看一个非常简单的例子，你只有两个动作。没有多个时间步。你只做一个单一决策，有两个动作。你要么猜A要么猜B。我们有一个非常简单的 策略，它只有一个参数theta。我们通过对theta应用sigmoid函数来得到一个概率策略。sigmoid是这个压缩函数，对吧？它把从负无穷到正无穷的实值映射到0到1之间的值。我们把这个值作为选择A的概率。 假设对于我们当前的策略，theta是负1.1，这给我们这些非常方便的概率：猜A的概率为0.25，奖励为一；猜B的概率为0.75，奖励为零。这是一个好策略还是坏策略？ 想想概率和奖励是如何对应的。这是坏的，对吧？因为它把更高的概率放在了没有奖励的选项上。所以我们希望我们的策略梯度更新这个参数theta。我们希望theta朝哪个方向更新？对，我们希望它变得更正，对吧？因为如果theta更大，我们就在这条
[57:12.640] 曲线上更靠上，我们会得到猜A更高的概率，而A是奖励更高的选项。所以让我们在不使用基线的情况下计算梯度，看看策略梯度是否告诉我们正确的事情。我们可以计算期望，我们可以计算我们从每个可能采取的动作中得到的策略梯度估计值 在我们可能采样的策略下。所以如果我们采样猜A，我们对梯度的估计是什么？我们通过将奖励乘以偏导数得到它，利用sigmoid的性质，这里的偏导数是75。所以梯度就是它们的乘积，是75。 我们可以做同样的事情。如果我们不采样A而是采样B，我们对梯度的估计是什么？因为奖励是零，实际上给我们返回了零梯度。所以这看起来像我们之前看到的轨迹，那些奖励为零的没有给我们任何梯度。这里如果我们采样
[58:23.280] 这个奖励为零的动作，我们也得不到任何梯度。但如果你取策略梯度的精确定义，即对所有动作的期望，在这个只有两个动作的设定中，我们可以精确地计算它。我们可以把每个梯度乘以采样该动作的概率，那将像我们几页后会看到的精确数字， 但我想大概是0.18，所以实际的策略梯度，对所有动作的期望是0.18，但我们将只有一个非常嘈杂的估计，25%的时间我们会得到一个75的样本，远高于0.18，然后75%的时间我们会得到零作为估计。我们真正想要的是两个可采样的 估计值都更接近真实值0.18。基线实际上做到了这一点。与之前的修改是我们只是从奖励中减去一个基线，其他计算相同，结果是我们采样猜A时现在有一个梯度的估计，采样B时我们有一个 .125的梯度估计。所以期望乘以这些动作的概率仍然是.18。所以它和之前一样。但这些单独的样本更接近那个真实值，因为基线实际上减少了我们得到的样本中的方差。
[01:00:11.359] 我们会在另一页幻灯片中有一个可视化。但这里的关键是期望不变。它只是让估计器中的方差变窄，这是好事，因为你将在真实数据上抽取这些估计，进行更新，你不希望你的梯度让你来回跳动得比需要的多。
[01:00:35.839] 好的。是的。所以，这只是一个可视化。如果我们有一个为零的基线，这是我们的梯度估计。它们有这个期望值，就是0.18。当我们拖动基线时，我们可以看到我们得到的这些梯度估计收敛到这个真实的期望梯度值。 在所有情况下，我们都会进行一个让模型更正确的更新。我们会进行一个将theta向更正值方向移动的更新，这会增加获得更多奖励的概率，或者将更多概率放在奖励为一的事物上。但是，如果我们在这个估计中方差更低，我们就会更快地做到这一点。
[01:01:30.000] 这里有一张图展示了这一点。这里紫色的线是对梯度的估计进行抽样，不使用基线。所以没有B项，抱歉，只有优势，只是尺度为一或零。然后绿色的线是减去基线的估计，只是一个非常简单的基线0.5，你知道，那个 在这个图上看起来是这样的。所以它让它们稍微更接近这个真实期望。所以我们只是抽取一些样本，这模拟了在完全相同的例子上反复进行策略梯度步骤。所以这里在这个情况下我们采样了猜测A。所以我们得到了正梯度，而且如果我们不使用基线，我们会得到一个大得多的梯度。 所以我们会迈出一个非常大的步子，然后我们又采样了A。这里的这条线将会是我们目前为止所有梯度样本的滑动平均值。所以当我们采样B时，现在我们有一个低于真实值的梯度样本。所以我们会更接近，我们的滑动平均值会更接近那个真实值。但你可以看到这里，当我们有基线时，我们的 方差更小。所以我们的滑动平均值已经更接近那个真实值了。然后就这样继续下去。我们碰巧连续采样了很多猜测B。我们采样猜测A，它又回升了。它最终确实会收敛，但当我们用这个基线降低了方差时，我们通常会离它更近。
[01:03:15.680] 关于这个有什么问题吗？好的。希望这没有太过于琐碎，但是，嗯，基线对我来说一直有点神秘，我想做这些可视化也是为了说服自己它们是如何工作的。希望它们对你们也有帮助。这个非常简单的双臂老虎机设置中的直觉确实大体上可以迁移到这些更复杂的
[01:03:40.480] 你知道，序列决策设置中。是的。所以基线是有帮助的，因为它们可以标记一条轨迹比预期更好或更差。而且如果你选择了一个好的基线，很多强化学习就是关于选择好的方式来表示这个优势函数。它可以帮助减少梯度方差并稳定你的 学习。但是你也有一些缺点，而且关于你用什么作为基线有很多选择。所以，我会跳过这个，只是说如果你在一个在中间时间步给你奖励而不是只在轨迹末尾给奖励的设置中工作，你会把你的奖励缩放替换为奖励到
[01:04:28.799] 也就是未来时间步上奖励的总和，这也会帮助减少方差。我们会保留这张幻灯片，如果这跟你相关的话，我们也可以线下讨论。但是，这就是你如何处理不是所有奖励都在任务末尾出现的情况。
[01:04:51.399] [哼鼻声] 好的。所以，用我们剩下的最后一点时间，我们会覆盖一个简化版本的GRPO，我相信是组相对策略优化。它是一种获得这个优势函数、估计基线的方式，使用来自模型自身的展开。而且这实际上是最近才被提出的。所以它是在 DeepSeek的一篇论文中，那篇论文训练推理模型，非常注重效率。所以很多过去关于将RL应用于语言模型的工作都需要一个单独的模型来预测这个基线。预测基线来减少方差非常重要，但很有挑战性，人们为此使用单独的模型。但DeepSeek 对效率非常感兴趣，训练模型不使用太多计算量，而拥有一个单独的模型需要大量计算。所以他们提出了这种方法，去掉了单独的模型，只使用来自模型自身的多个样本，来估计策略下的期望值、期望奖励，这真的很优雅。所以想法是
[01:06:11.200] 我们可以对给定实例从模型中多次采样，然后得到每一个的奖励。然后我们可以通过对这个组中所有这些实例的奖励取均值来中心化那个奖励。我们得到一个均值。我们基本上用那个均值作为我们的基线。这很合理，对吧？因为你知道基线是你的期望，一个好的基线是你的 策略下的期望奖励。而一个非常好的估计就是多次运行策略并得到奖励，取这些奖励的均值。所以我们在这里得到了中心化的奖励。在原始的GRPO论文中，他们还对此进行了标准化。所以它们不只是平移它，它们还重新缩放它。它们计算这个组 轨迹中奖励的标准差，然后除以那个标准差来得到优势函数。现在实际上，似乎除以奖励的标准差并不是必要的，而且很多自称GRPO的实现实际上去掉了这个除以标准差的步骤。 所以它更接近优势项。有一篇我们会在几张幻灯片后讨论的论文叫DRGP，Dr. gpo或者done right gpo，它更多地讨论了这样做的动机，但实际上在很多gpo的实现中它们不会除以标准差，它们只会减去均值，但是是的，gpo的更新非常简单
[01:07:57.119] 看起来很像我们之前定义的策略梯度，但所以你有一个任务实例，就是你知道的一个要解决的RL问题，你会从中生成多个展开。所以你会多次解决同一个问题，重置环境，通过从你的智能体采样得到一条完整的轨迹，这通常会，Aorva在他的讲座中会给一些 关于如何选择这个组大小的指导，我相信，但这通常是像八这样的值。然后你会得到这些展开上奖励的均值，你会从中得到你的优势估计。我们会在另一张幻灯片中展示两种不同的获得这些优势估计的方法，但那会给你一条轨迹的一组展开以及优势 对他们来说。如果你的GPU还有空间容纳更多样本，你也可以为额外的实例获取相同大小的组。而且你只会在每个示例内部计算组均值。所以这很重要，对吧，因为价值函数应该只是对……它应该只是对这个特定示例上奖励的估计。所以我们的优势 应该只看这个示例内部的奖励。但你会通过使用那个公式来缩放该轨迹的奖励，从而为每条轨迹得到一个优势，该公式会减去均值，可能还会除以该实例内奖励的标准差。
[01:09:39.359] 所以对于策略梯度损失，我们之前只是为每个实例取一个样本，然后重新缩放。但在GPO中，我们会在批次中有多个实例，然后每个实例的组内有多个轨迹，我们会为其中每一个得到一个优势。我们会用它在求导之前重新缩放我们的对数概率。
[01:10:06.800] 关于这个有什么问题吗？是的。>> 这是无偏的吗？是的，因为你的优势函数在这里只依赖于提示和模型。所以你在这里从奖励中减去的基线函数只是来自语言模型的采样。所以这是无偏的。
[01:10:40.159] 好的，很好。是的。所以，Dr. GRPO对GRPO做了两处改变。我还应该指出，GRPO也——所以我在这里写的并不是论文中定义的GRPO损失。论文中的损失有一些额外的项，我想Graham会在他的讲座中更多地讨论，那些项也会裁剪梯度，这样你就不会在某个 方向上迈出太大的步子，并且还会对策略进行正则化，这样你就不会与你开始的策略偏离太多。这两者都是有助于稳定训练的额外内容。我们今天不讨论它们。我们只讨论GRPO内部的优势函数。但是的，这会帮助你开始做第三次作业，当它
[01:11:32.400] 发布的时候。好的，很好。所以，我们时间还够。那么，DRGRPO对GRPO做了两处改变。你们也会在作业中做这些。它们似乎通常是有帮助的，因为它们有助于避免GRPO中的一些偏差。这两处改变就是——第一处我们已经稍微谈过了。它不除以标准差。它只是减去 组内奖励的均值。嗯，这样做的动机是，如果你除以标准差，你的优势就不再意识到问题的难度了。因为你有点像是在把所有难度范围重新缩放为标准化的。所以那实际上可能是有害的。GRPO还除以了 生成的token数量，这可能会使模型偏向于……偏向于冗长的错误答案，所以DRGRPO只是除以一个常数，即可能生成的最大token数。好的。所以，和我们之前看到的同一张图。当你实际实现它时，这看起来是什么样的？你有蓝色的生成token。你计算一个优势函数，它会 取你这条轨迹的奖励，然后减去你所有其他轨迹的均值奖励。然后你得到一个每token奖励，它只是用一个常数C来归一化它，C通常是可能生成的token数量。 然后你对组内所有轨迹的所有奖励都这样做，这些奖励可能是正的也可能是负的。损失看起来是这样的。关于这个有什么问题吗？是的。嗯，是的，所以再次注意，原始论文还有一个比率，它有助于对策略进行正则化，还有裁剪，但我们现在不讨论那些。
[01:13:46.400] 好的，所以我们之前有一个问题，比如如果你所有的奖励都是零怎么办？那很糟糕，因为如果你所有的轨迹都失败了，那意味着对当前策略来说这很可能太难了，即使你有基线，你也不会从中得到任何信号。如果你所有的轨迹都成功了，你也有问题，因为 它们对当前策略来说太容易了。所以就像你知道的，如果这是二元奖励，一是我们能做的最好的，那就很糟糕，一般来说，如果我们的示例内没有任何分布差异，那就很糟糕，因为那样我们就不会从GRPO估计的优势函数中得到任何信号。
[01:14:33.120] 那么如果每一个你知道的样本轨迹都失败了，我们能做什么呢？我们可以选择不同的任务实例。我们可以向提示中注入一些知识，鼓励模型产生更有可能正确的、更有引导性的解决方案。我们可以添加演示。 有一些做离策略强化学习的方法，会把正确的演示放入组中。或者更简单地说，你可以先在那些正确演示上微调模型，然后再运行强化学习。或者你可以有一个更有信息量的奖励，它鼓励模型在产生整条轨迹之前做出正确的中间决策。那叫做
[01:15:14.159] 奖励塑形。所以，比如给模型找到正确文件的奖励，就是一个例子。我们会跳过这个，因为我们之后可能会更多地讨论PO。但PO基本上就是，不使用轨迹来计算优势函数，而是有一个单独的模型，通常与策略共享一个主干，来预测基线。
[01:15:44.800] 好的。那么非常快地比较一下我们到目前为止看到的方法。每个方法都让智能体在环境中采取行动，然后我们只是用智能体产生的那些行动来计算损失。所有这些在某种程度上看起来都像自训练，但它们的不同之处在于它们重新加权那些生成行动的方式。所以restm，我们假设我们有零或一的二元奖励 然后我们只过滤出正确的轨迹，然后对它们做SFT。Reinforce使用轨迹的奖励来重新加权损失，这对应于重新加权梯度，我们花了很多时间展示这是正确的。然后GRPO使用这些优势函数，其中奖励被偏移，可能还被模型为同一示例采样的 其他轨迹的奖励重新缩放。所有这些方法都有助于解决我们在讲座最开始提到的三个问题。所以任务不匹配现在应该解决了，因为我们现在直接最大化奖励。直接最大化期望奖励。数据不匹配。我们现在可以从奖励为零的轨迹中学习，
[01:17:09.199] 只要我们以适当的方式做基线。而且，这些方法的同策略性质有助于缓解暴露偏差，因为模型在训练时基本上是在自己驾驶。所以它应该学会从自己的错误中恢复。嗯，代价就是，你知道，你必须真的在环境中尝试各种东西，这可能代价高昂，可能 会给你的梯度引入一些噪声，我们已经介绍了一些试图让这个过程更稳健的方法，但在接下来的两讲中，从下周开始，我们还会讨论一些其他方法，好了，这就是我们
CMU 11-768 Fall 2026 · Lecture 9 · Training 2: Reinforcement Learning Basics
- 视频：官方视频
- 字幕：en-orig · auto captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
正文
[00:00.480] Okay, so let's go ahead and get started. So today is going to be about an intro to reinforcement learning which will build on the SFT for agents uh that we had last week. This is going to be the first of three lectures that we'll have on reinforcement learning. And so in this first one, we're going to be talking about basic poly gradient methods, which are ways where agents interact with the environment and get feedback and then update their weights to make things that were successful that got high reward more likely and things that got low reward less likely. We'll then have a second lecture which will actually be next week on uh more advanced RL algorithms talking about off policy methods about um some of the things that you need to do to stabilize RL and get it to work. And then our third lecture will be on RL systems and going into some of the practical frameworks and you know things that you need to think about and pay attention to in hyperparameters when you're training these agents. And all of these should set you up uh pretty well for assignment three, which is going to be focused on training models with reinforcement learning in two settings. So the first setting is going to be a simple kind of sandboxed setting, which we'll actually use as a running example in this lecture. It's going to be called number search. And the second one is going to be like a simple kind of uh crafting environment that's a little bit like Minecraft. So for these, they're definitely going to require more compute than your previous assignments have. Um so you'll be using the compute credits more heavily, but uh you should have the foundations to get started on those assignments after these first two lectures. And um yeah, and we're looking forward to uh to seeing what you all do for those. We'll also have a guest lecture on Thursday from um Akari Asai who's a faculty here at CMU uh who's an
[01:58.159] expert on deep research and she'll be talking about that as another domain for agents uh that complements the the ones that we've talked about so far. Cool. Any uh any quick logistic questions before we get started? Okay. Yeah. So uh today we'll have a bridge from uh supervised fine-tuning to policy gradient methods which are kind of the most uh the most basic but also
[02:25.280] one of the most elegant forms of reinforcement learning. So the running example that we'll use is this task that we'll call number search which is basically like training an agent to guess a number based on feedback about whether you're too high or too low. So this is a pretty simple task but you know easy to understand and easy to come up with a reasonable policy for it. So the setup is there's this number which is part of the environment for this task but isn't known to the agent and it's an integer between 1 and 16. The agent is going to get four attempts to guess the number. And after it makes a guess, the if it got it correct, the environment will say correct. But otherwise, the environment will say, uh, you should go higher or you should go lower. And then the agent can make another guess up to four guesses in total. If at any point the agent gets it correct, the task ends and it gets a reward of one. But if you reach the end of your four guesses and you haven't gotten it correct, then it's a reward of zero.
[03:30.480] Okay. So, a lot of the trajectories that we'll walk through are from an example of this where the hidden number is 11. So, the agent is going to try to guess that. And here's four possible attempts at this where in the first one the agent says 8, gets feedback, you should go higher. It says 12, gets feedback, you should go lower. uh 10 higher and then 11 so it gets it correct on the last guess. In another attempt we might start with a lower number. So this is actually what's the optimal policy here? What's the best starting number? The middle, right? So it's kind of like 20 questions. You want to like split the space, right? So the best first guess is eight. But uh and actually how many of you folks have played Wordle? you know the guessing game from the New York New York Times. It's a fun game. It's kind of in a similar space and there's like an optimal first guess that you can make which reduces the the the space in an information theoretic way. Similar concept here but simpler. So you could start off with a you know suboptimal first guess of four but you could still succeed in the task right by taking this feedback into account going higher then going higher then going you know lower and eventually getting it correct but you could also go wrong right so we could start with a very bad guess of 16 and um unless we got that right the feedback's going to be lower and in this case we might time out so we don't get it within four guesses and similar
[05:05.199] Similarly, we could have another pretty bad policy of starting with one and then going two and three and four and um and timing out and failing at the task. Everybody clear on the the setup? Okay, great. So, we'll also have rewards for trajectories. And a reward is just a scalar value which tells you how good or bad that trajectory is. So in agentive settings, this might correspond to like did the code that you write pass all of the unit tests in like Swedbench for example and you get a reward of one if it passes all the tests a reward of zero if it doesn't pass all the tests or you could have you know like a value that's not zero or one that's not a binary reward too like what's the fraction of tests that you passed but in a lot of the domains that people are training on these days um in what's called RLVR like reinforcement learning with verified rewards. The rewards will be either zero or one. Um but the methods that we'll be talking about today are more general. Um they also apply to scaler valued rewards.
[06:11.520] So just a quick reminder of how SFT works. So in supervised fine-tuning, you have a demonstration of a task and you're training the agent to copy that demonstration by maximizing the probability of the demonstrated actions. Right? So if we break down one trajectory into these steps, we have a prompt that's given to the agent. Um we have a guess um feedback from the environment higher. We have another guess more feedback from the uh environment and so on. And our task is going to be given these demonstrated actions which we'll write a star. We want to maximize the probability that the agent assigns to those actions conditioned on all the past history up to that point. Right? And so this will be just you know the supervised fine-tuning loss is maximizing the log of the probability of that sequence which is the sum of the log of the probabilities of the individual actions right and there's a lot of ways we could get these target actions that we're trying to imitate. So they could be from a person who gives a demonstration from the task. They could be from like a stronger model that we're trying to copy
[07:21.759] to distill from today. They could also be from uh trajectories from the agent itself that we're trying to reinforce in the model. But this should hopefully be a review from you know your previous classes on LLM training and and from the SFT lecture. Okay. But SFT has some problems which reinforcement learning is going to address. So there's roughly like three of these that we'll focus on. The first one of these is what we might call task mismatch. So supervised fine-tuning maximizes the probability of demonstrated actions at each point in time. But what we're really interested in is maximizing the probability that the agent carries out the task successfully. And these are similar to each other, but they're not quite the same. And there's a couple reasons for this. So the first reason is that there are multiple ways to carry out the task successfully, right? So, uh, starting with eight is kind of the best thing to do, um, as a first guess, but, um, it would also be reasonable to start with, you know, four
[08:29.840] as long as you're able to, uh, take the feedback into account. And for this, you know, here after this first step of feedback, we have various reasonable actions that we could take as well. and a demonstration, whether that's, you know, from a person or another model, only shows one way of solving the task, but we want the agent to be able to um potentially use other ways as well. And a second problem is that some actions that you can take are worse than others, but to SFT, they all look the same. So it would be very bad for us to in response to the feedback you should go higher it would be very bad for us to like output a number below eight right but SFT doesn't make a distinction between the actions that weren't demonstrated and so it would get the same loss if um you know the agent had highest probability on um a lower alternative action rather than a higher alternative action that wasn't the one that was demonstrated. So that's one problem of the mismatch between this SFT loss and the objective that we're actually interested in of maximizing you know expected reward or
[09:46.000] the probability of success on the task. The second problem is data mismatch. So we'd like to be able to learn from um suboptimal data too. Even if we assume that our demonstrations are the best way to achieve a task, there's, you know, if we had access to other demonstrations that were less optimal or maybe even weren't successful, those could provide useful signal for our models too. So like uh say that we know that this is a bad trajectory like it times out. We wouldn't want to just do SFT on this, right? We wouldn't want to maximize the probability of those actions, but we might want to uh have some learning method that could um have the model not take trajectories like this. And reinforcement learning is going to be one way of doing that. The third problem is what's known as exposure bias. So in SFT, you're training a model to make predictions of the next action given a fixed sequence of past actions, right? And that fixed sequence of actions comes from somewhere. But then at evaluation time, the previous actions are going to be generated by the model itself, right? Because you have to produce actions one at a time. You have the actions you produced in the past. It's this auto reggressive process of interacting with the environment. So if the agent um produces an action that is very out of distribution from the histories that it was trained on, it might just not have the the response that it produces might just not be calibrated because it never saw data like that. So say that we only trained on demonstrations where uh the next integer was, you know, did match the feedback that we got. So it was always higher if the feedback was higher. Now if at generation time the model makes a mistake and samples a lower number, it might just totally throw it off because it didn't see data like that in training
[11:54.240] time if we only trained on these optimal policies. Does that make sense? So one way of viewing reinforcement learning is it's a way of letting the model generate its own training data and also learn how to recover from the mistakes that it makes so that it's prepared to handle its own mistakes when you generate from it. And um this property of having the model generate its own training data is called being on policy. It's generally a good thing to have um because it prepares the model to you know deal with its own mistakes and the supervision that you get is a good fit for what the model's able to currently predict. Um but it can have some efficiency trade-offs and in Graham's lecture we'll talk a little bit we'll talk a lot actually about um uh
[12:39.600] methods that relax these assumptions that go more off policy. Cool. So these are three problems with um supervised fine-tuning for agents and for sequential prediction generally. Any questions about them? All right. So, reinforcement learning the general idea is instead of having like a uh trajectory a fixed trajectory which you learn to imitate by you know maximizing the probability of the actions. We're going to have the agent or you know the model um generate its own actions and then we'll evaluate the reward of those actions in the trajectory and um then we'll adjust the parameters of the agent so that um it makes high reward trajectories more likely and lowreward trajectories less likely. So formally and we'll see you know the formulation of this in a little bit we're going to be training the agent to maximize its expected reward you know expectation in probability under the agent and under the environment. So for um just schematically if we have you know a task like this number search game with the underlying goal in the environment B to guess number 11 we would have the agent generate multiple trajectories just by um producing likely actions at each step from the agent. You know just in the way that we would typically evaluate the agent in the environment for each of those trajectories. uh we'll get a reward um by uh by executing these actions in the environment and then we'll use that reward as feedback for the reinforcement learning method which is going to change
[14:26.079] the parameters of the agent uh to make these highreward trajectories more likely. So this is going to address the three problems that we uh laid out with SFT. So task mismatch is addressed because we're now training the agent directly to maximize expected reward. I haven't shown you how that's done yet. A lot of the first part of the lecture is going to be about how we do that. We'll formulate expected reward. We'll take derivatives of it. We'll optimize it directly. It will also fix data mismatch because as we'll see especially towards the middle and end of the lecture, the agent will now be able to learn from all trajectories. So it will get some signal from these bad trajectories and will avoid taking actions that lead to trajectories like that in the future. And it will also help deal with exposure bias because the agent is now going to be generating these trajectories that are used in training. It's going to be on policy. So if it makes mistakes and sees that those mistakes lead to a bad reward, it will get penalized for that and then learn to avoid those in the future. So it's sort of being prepared to deal with its own errors.
[15:38.160] Yeah. And um yeah, like a lot of the issues that people had with like language models up to maybe like three or four years ago where they were um being like uh they had kind of repetition biases. They would get stuck in loops. A lot of those issues got fixed once people started doing a final post-training step with reinforcement learning where the models were now able
[16:00.240] to recover from their own mistakes. Okay. So we'll talk about how we you know formulate interactions with the environment as trajectories which will then let us set up the objectives that we optimize. Okay. So um we won't spend too much time on this because it's should just be a recap from the first part of the course. But we have an agent which is going to be uh represented um as a policy which we'll term um pi theta. Theta are the parameters of the language model that you know that drives the agent and the agent's going to be producing actions um by sampling from the language model and you know then applying those um uh those samples in the harness that it's operating in that will produce actions um which you then take in the environment and the environment gives you back observations and also rewards. A lot of the settings that we'll look at only give you a reward at the end of the task, like when you actually execute the test cases. Um, but pretty much everything that we talk about will also apply to settings that have um intermediate rewards where for individual actions you take, you can also get some reward about um the value of that action. But just some examples of how you can apply this general framework for text generation. Um, the task instance might be the prompt that you're trying to produce a completion to. The observation might be the tokens you have so far. The action might be the next token. And the reward might be the
[17:40.240] score that you get from a judge model. For example, if you're doing um reinforcement learning from human feedback for a web agent, like the setting we talked about a week or two ago, the task instance might be the task and the website. And here there's a difference between the state of the environment, sort of everything that determines the environment and what the agent gets to observe. There might be some hidden state like what's the content of the backing databases, but the agent only gets to observe the page that it's currently on. And it can take actions like clicking, typing or scrolling. And it might get reward like intermediate reward like what are the number of subtasks from the the task description that it has completed so far. And for number search we've uh seen the task instance is the number that the agent is trying to guess which is going to be hidden from the agent. So in number search every task looks the same to the agent at the start. Um the observation would be high or lower or correct.
[18:41.280] Action would be the next guess. And the reward is given if the agent gets it correct that's a one or um if it times out then it gets a zero. So given these um these observations and actions and tasks we can start to formulate some notation which will let us derive the um reward terms that we're expected reward terms that we're trying to maximize. So a trajectory is a sequence of interled actions and observations and we'll represent um the sequence of observations and actions up to a given time step as the history h. So the history ht is going to have all of the actions and observations up to that point in time. And our agent policy pi theta remember theta are the parameters of the language model inside the agent. that's going to give us a distribution over the next actions that I would take um g at a particular history. Right? So you take the history, you plug it into your agent framework, you get a distribution over next tokens which correspond to actions in the environment. Any questions about that?
[19:54.480] Okay. So we also have rewards and in the most general setting you could be getting rewards along the way after each um action or observation that you take. Those formulations whether it's after actions or observations can be equivalent to each other. Um in the number search setting, we only get a reward after you um either time out or get the feedback that it was correct. So
[20:18.080] in the number search setting, your rewards will be binary and they'll just come at the end of the trajectory. And we'll write the reward of the trajectory as the sum over the rewards at all intermediate time steps. So toao is going to represent trajectory which is you know the sequence of these observations actions and rewards. Okay. So that's the notation and given this we can also you know have probabilities over trajectories and the probability of a trajectory is going to depend on two things. So the first thing is the probabilities assigned by the agent. So we can decompose this into a product of probabilities over time steps where um h sorry I dropped a p here but uh there should be a p before this term which is the probability from the environment of getting this observation and this reward conditioned on this history the action taken by the agent at that time step and the task and that probability from the environment we're going to multiply by the probability ility from the agent policy of producing this next particular action. So this is just the chain rule um decomposing this probability of producing this entire trajectory into the product of its component parts. And a key thing in RL is we don't actually know we we don't have to know anything about the environment. We don't have to know, you know, what's the probability of getting correct as our next observation or what's the probability of getting a next observation as I'm interacting with Amazon.com or something. The key thing about RL is we're going to interact with the environment which will by taking actions which will be a way of sampling from this distribution over observations and rewards. And that's actually going to be enough information for us to be able to learn how to differentiate um an objective that depends on this with
[22:27.039] respect to our parameters and update the parameters. And that's kind of magic in some sense, but it also makes reinforcement learning algorithms a bit slow because you kind of just have to like try stuff and see what happens. Um, and that's why reinforcement learning takes a lot of compute to run because you have to do these experiments um, experiments with the environment, see what happens, and use that as a somewhat indirect source of feedback about how your reward depends on your parameters. Any questions about this? >> Yeah. >> Yeah. So I've uh rewritten this a bit from so the question was in classical RL there's a marov assumption which um says that the transition to the next state depends just on the previous state that
[23:21.679] you're in and the actions. So we're actually working with um a palm dp here a partially observed marov decision process um where there's a difference between the underlying state which you can't see and the observations that you can. So in a palm dp um your probability of the next observation has to depend on all of the previous observations and actions because you don't have access to
[23:45.520] the state. Um but for a pom dp these formulations should be equivalent and this one's just a little bit easier to work with for our purposes. Good question. Okay. Yeah, sorry about that P missing. Okay, so our objective is going to be to maximize the expected reward. Um, and so what does this mean? So what we want to do is we want to uh take samples from our agent like we want to have the agent generate trajectories through the environment. Um and like uh we're typically going to be doing that by sampling um actions from the language model and then feeding those actions through the agent framework, right? Um so we can write that um as sampling a trajectory from this distribution which is parameterized by the language model itself. And for each of those trajectories that we get by sampling from that distribution, it's a very complex distribution that depends on the agent, on the language model, and on the environment, we're going to get a trajectory and we're going to evaluate the reward of that trajectory that gives us a number. And we want to maximize our
[24:59.840] chance of getting a high reward here. So that's going to be maximizing this expectation. And we'll just write that objective as J of theta because it's a function of the parameters of our language model. So in our very simple setting um let's uh you know we have this number search we have say there's only four trajectories which are actually producible by the language model. So it's learned kind of a very sharp policy. Um say that it puts probability of 0.3 on this first trajectory 0.2 on the second and 0.25 25 on the third and fourth. Uh generally blue is going to mean success, a reward of one and red is going to mean failure, a reward of zero. So the expected reward if these are the only four trajectories which the agent can produce is going to be 0.5. Right? So just you know taking a weighted sum of these trajectory rewards weighted by the probability under the agent policy and the objective of training is going to be to optimize this. So we want to adjust the parameters theta to maximize this expected reward. So you can think of this as like this is kind of a bad current agent because it has a pretty high probability 0.5 of producing a trajectory with zero reward. We want to shift some of that probability. We want to shift ideally all of that probability to predicting trajectories that have reward of one. So shifting the probability mass from red to blue.
[26:32.960] This is obviously simplified because you know really the agent could produce an inf not infinite but like combinatorally exponentially large number of possible trajectories but we're just kind of simplifying the visualization a bit here. So one way that you can do this is a pretty simple modification to SFT that's really well motivated. It's called expert iteration. And um a particular uh type of expert iteration is called reinforced self-training or rest for short. So the idea in rest is going to be that we will just generate data from our current um policy from our current parameters of the language model and then we will choose the trajectories that are successful and use those as fine-tuning data and update the model on those. And so that's where the reinforced term comes from. It's going to sort of reinforce the correct behaviors that the agent is already producing. Um so it has two steps which are going to be iterated. Um rest em is actually named like rest I think expectation maximization because it looks really similar to the em algorithm uh which some of you might have seen in an ML class. But the idea is we will alternate between these two steps. The first step is you sample trajectories from the current policy. You get the rewards of them. The second step is you will um have the model make those uh trajectories very likely by just doing a bunch of SFT on those collected trajectories. Um and then you repeat you will get more trajectories from the current policy and you add those trajectories in to the existing set that you have of good trajectories and then you'll um train your model on all of those. So you have this growing data set which you are having the model um iteratively get very good at producing.
[28:32.799] Um yeah so that's kind of a high level description and uh the way this works mechanically is you will generate a bunch of trajectories. You will um evaluate the reward of each one. You'll select um using like a threshold and this is easy if your reward is binary. Um, if it's binary, you just keep the trajectories that have a reward of one and throw out the ones that have a reward of zero. And then you combine all of these kept trajectories across your instances. And those are the um those are the next iteration of your training set. And so then the objective is just the SFT objective. We want to maximize the probability of these actions in these cap trajectories. the one where the reward is for binary rewards one.
[29:23.600] Does that make sense? Any questions about that? So, schematically, this is actually going to look quite a bit like the reinforcement learning methods that we will talk about. Um, this isn't like technically referred to as reinforcement learning. Um, but it really has a lot of similarities to it because you're generating data from the model. um you're updating on that data in a way that is determined by the reward of the trajectories. And um we'll see how reinforcement learning does this in a little bit more of a fine grained way later. But this is a very simple and pretty effective uh baseline that you can start with which reuses all the same machinery of um supervised fine-tuning and is a good fit if you have binarized rewards.
[30:12.720] >> Yeah. Why is that self? >> So the difference from self distillation is just that you're using the reward to filter. Self-distillation could also refer to um training on all of the trajectories produced by the model. >> Yeah. So self it's called selfraining because it looks a lot like um you know like that but it's reinforced because you're using the reward and so you're reinforcing the good trajectories. Yeah. I mean this is a more basic question but how can you guarantee trajectories that do have a positive reward? >> What if like for example in the worst case you just have a bunch of bad trajectories? >> Yeah. So that's a great question like how do you guarantee that some of your trajectories have good reward? Um if
[31:01.200] your model can't produce trajectories that have good reward then reinforcement learning and this method um are probably not a great fit. So there's a few things you can do. one is you can um have some demonstrations of good trajectories which you train the model on. So you know like um the R1 paper um and a lot of reasoning models um will have an initial what they call cold start stage where they will train the model on reasoning traces um before they start doing reinforcement learning. So that just makes it easier for RL or something like this to be able to produce trajectories that have high reward. You can also like look at curriculum. So you can um select problems that do have enough trajectories which produce reward. Train on those first and then as the model gets better you're able to tackle harder problems. You can also have like intermediate rewards like if you have some knowledge of the problem and like know it's useful to explore the code repo first. You could give the model reward for doing that and then hopefully like after it learns to do that it'll kind of unlock solving the whole task. But this is a really big
[32:08.480] kind of issue that RL algorithms have to grapple with and those are a few solutions to it. Yeah. Cool. Okay. So now um let's uh actually get into the policy gradient methods. So we're interested to maximize this quantity uh J of theta the expectation of reward. And we can't just use back propagation to do this uh for two reasons. The first is that we are sampling actions from our policy. And the second reason is that we are then executing these actions in this like blackbox environment which is potentially which is almost always like non-ifferiable. Um and so this breaks our ability to do back propagation to you know directly you know like use the chain rule like you saw in a machine learning class to get the derivative of um your reward with respect to the parameters. So um actions are discrete and they're sampled. So the derivative with respect to um the parameters of a sample is zero almost everywhere because you know the parameters change the probabilities but don't um until you get to kind of a step point don't change the action sampled and the environment is usually not differentiable. So the general solution
[33:32.000] to this is actually a pretty classic method called reinforce um which was from Williams in 1992 and um it's going to use this uh definition of the expected reward and then basically just some algebra tricks in order to um compute an estimate of this that uses um samples from the model and um the same back propagation procedures that we've looked at before um to get par you know
[34:03.600] to get partial derivatives with respect to these parameters theta and we'll show you how to do that on the next slide. So this is the objective that we want to optimize. It's this expectation over trajectories from our environment and our policy and the expectation of the reward of those trajectories. So this is a if you recall what an expectation is it's a sum um over these things these trajectories uh of the term in here the reward but weighted by the probability of that trajectory. So this is a sum over a really large space like all possible trajectories from the language model. We're almost never going to be able to compute that exactly, but this is what it would come out to be if we wanted to compute it. Taking the gradient of this. So we just, you know, take the partial derivative with respect to the parameters theta. Um, we can move that uh inside the sum and this uh reward term depends only on the trajectory. So once we know the trajectory, we know what the reward is.
[35:16.480] it doesn't depend on the parameters anymore. The parameters produce the trajectory. Um but then the reward is just a function of the trajectory. So um this reward term is going to be multiplied by this partial derivative of this probability. But this is still difficult because this is a sum over like a potentially very large number of trajectories. So we're going to do an algebra trick here. And this is called the score ratio trick. um we're going to just uh multiply and divide by this term p of theta. So we have that over here and we also have that in the denominator. Those cancel. So this is the same as that. And then we can use this trick which is um the partial derivative of log p with respect to theta. We can just use the chain rule and it's get it's the partial derivative of um p / p. So we can use that identity here and we get this term. And this is going to be kind of magical because we now have this which is going to be an expectation over trajectories um times a reward term times this term here.
[36:27.839] And um this term is going to be this is the same computation that we do in SFT. Right? In SFT, we're also taking a partial derivative of the log of a bunch of unit tokens um with respect to the parameters of the language model. And um we're going to be able to uh estimate this quantity by sampling trajectories and then computing this term here. And
[36:57.440] this will give us an estimate of the um of the gradient of this uh expected reward. Any questions about that? >> So this is really general doesn't make any assumptions about um the language model or about the environment. >> Yeah. >> Are you going the series of states and actions but the actions are generated by large are also sequences of >> yeah normal way to reconcile that >> yeah we'll we'll break that down in a couple slides but yeah it's a good question yeah so um we'll just copy that term here and then we'll um break this into the component pieces so um these trajectories are going to be sampled by having the language model and the agent on top of it interact with the environment um and that's you know the same procedure that you would use when you're just running this agent. That gives you a trajectory. Um and we uh uh this should be um a sum over this should be an average over trajectories because we can estimate an expectation by drawing some number of samples from this distribution and then averaging the quantity inside the expectation. So if we average over a bunch of trajectories drawn from this distribution this quantity here um then we will get an estimate of this gradient. So um what are these uh terms inside here? Well it's um since this is a trajectory we break this up into steps. So it'll be a sum over time steps of um the gradient that's due to that particular action. And we'll see how to compute that in a slide.
[38:54.160] So yeah, so the three components here are the trajectories. We get those by sampling. Um the task level evaluation which we get from the environment. And then this gradient here which is going to be produced using the same way that we get gradients in in supervised fine-tuning. But the key thing is these actions here are going to be the ones sampled from the agent rather than ones
[39:18.800] that are demonstrated by uh you know some ground truth. Okay. So yeah, how do we get log probabilities of trajectories? Uh I'll go a little bit quickly through this. The the key thing to keep in mind is that the probability of a trajectory depends both on the probability from the language model plus agent but it also depends on the probability of getting the particular observations and rewards that you did from the environment. But the key thing is is that this probability doesn't depend on the parameters of the language model in any way. So if we differentiate this log probability of you know the uh trajectory with respect to the um parameters this entire term just goes away because it's a constant with
[40:10.160] respect to the parameters. So we can just ignore it and this um gradient uh for the trajectory is the sum of the gradients for the actions within the trajectory. So this is mostly just kind of like uh bookkeeping for rigor. Um but one thing that is important to think about is these actions that are being produced by the agent um might correspond to multiple tokens from your language model, right? So like maybe you need to say uh you need to say you're you're making a complex tool call and that splits up into multiple tokens. Um, of course, the log probabilities of those tokens are just going to be um summed up to produce the log probability of the entire action.
[41:02.319] And when you have this uh gradient of a trajectory, the gradient for a trajectory, you can just multiply by the reward, which is a scalar value from the environment to get the contribution of this trajectory that you sampled to the expectation. And how does this uh like actually look if you are running this on a real example. So you would um have your trajectory which is sampled from the agent. And um these blue tokens are the ones that are actually generated by the model. All the gray tokens are ones that are either like part of the prompt or part of the environment or added by the scaffold. But these are the ones that are actually produced by generating from the language model. and those are going to be the ones which um contribute to the loss for policy gradient. So we'll have all of these generated tokens. We'll get um log probabilities for those um of that generated action selected from the distribution of the language model over particular tokens at that point. And then we will just scale all of those um token log props by the reward that corresponds to this trajectory.
[42:28.800] Does that make sense? And then if you take the gradient of this, the gradient passes inside and gets applied to this log probability here. So that your gradient is going to be the reward times the gradient. So when I first looked at the code that implemented reinforce on like a pretty complex task, it was like using RL to optimize dialogue models from a paper called deal or no deal. It was kind of like really surprising that the implementation is so simple like you just do basically an SFT gradient computation and then multiply that gradient by a reward or multiply the loss by a reward and take the gradient of that. But it it works and the math works out.
[43:13.680] Yeah. So we'll have a mask here which is going to be you know you mask only to select the tokens that you generate and then you scale all of those by the reward for the trajectory. And that's how you implement the reinforce policy gradient algorithm. And so the training loop is just you sample task instances you evaluate reward on them. um get the loss using that slide that we just showed you and then um uh do an update on all of the instances in your batch of examples here. Later on we'll uh when we cover gpo we'll talk about how you could generate multiple possible trajectories for a given task example I and then use all of those in the update. But if you're doing just like standard reinforce, you would
[44:06.000] just be using one trajectory for each example and then um computing the update from that. So your um batch size would be the number of task instances that you're training on. Um but when we get to gpo um you will also have multiple samples per task instance. Any questions about this?
[44:29.760] Okay. Yeah. So um just very quickly with binary rewards this looks a lot like the rest um procedure that we talked about. Both of these are just um scaling log probabilities of actions by a reward. The difference is is that this reinforce method is like fully online. You're like sampling these trajectories and then updating on them directly. Whereas with REST, you build up a very large data set and then train on all of those and maybe do like, you know, a full epoch or multiple epochs on all of those instances. But um the intuition behind them is basically the same. Okay, but this has a lot of issues too. And so it's very rare for people to just run standard reinforce policy gradient out of the box.
[45:17.920] So the biggest I think the the the way of kind of the strongest motivation for the next methods that we'll talk about is thinking about how rewards should really be the update that you do from rewards should really be relative. So let's look at these two different trajectories. Um this one is successful and so all the actions are going to be upweighted by a reward of one if we use the update rules we've talked about. This one is unsuccessful. So we had a bad first guess, we eventually time out. And if we're just using basic reinforce or arrest, we are not going to do any update based on this example at all. Right? Because it's scaled by the reward. The reward is zero. So but we actually want to be able to learn not to do this, right? Um so
[46:06.800] the intuition for the next part is going to be about shifting our rewards so that um we still get a correct estimate of the gradient but um we're able to learn from um uh rewards at all different um scales or shifts. Um and another way of thinking about this is that the update that you do should depend on how well the policy uh would be expected to do on
[46:31.200] this example. So say that this is just like an easy example and um it and the model like 95% of the time gets a reward of one on it. If 5% of the time you make a really dumb mistake and get a reward of zero, you should update very heavily against those um costly but rare mistakes. And um baselines and advantages that we'll talk about next will be a way of doing that which will compare rewards to the reward that you would expect to get from the policy running on this example. And the math for that looks like this. We'll define an advantage which is going to be a function of the history up to a particular time step and an action. And that's going to be um say that you take this action from you take this particular action and then you sample from the language model for the rest of the trajectory that will tell you and you contrast that against if you just sample from the language model including that action and then continue on. This will tell you how good is this particular action um relative to uh how good is the what's the value of this particular action taking into account that the language
[47:49.440] model is going to take all the rest of the actions in the future. So maybe a concrete example will will help with this. Let's say that we are partway through this uh guessing game and we've guessed eight and gotten feedback higher, guessed 12, gotten feedback lower. So um we know that the numbers have to be between 8 and 12. So they can either be 9, 10 or 11. So we don't know exactly what this uh value function is. It depends on what our policy is. But we can think about um it's probably something like 2/3 or one, right? So like if the policy is is uh does the best thing possible uh always you should be able to just always win from this situation. But if it's a more random policy within the set of things that are valid it might be more like 2/3. But we can think about the advantage functions of particular guesses relative to this baseline. So like a guess of 11 would be bad because um it's at one end of this range and if 11 is not correct then we have two candidates and one guess left so this advantage should be low whereas 10 would be good because whether or you know even if this is wrong then we have one guess left and we could go either higher or lower. So a lot of reinforcement learning algorithms are derived from different ways of estimating these advantage and value th this advantage as a function of uh this individual Q value for actions and values and we'll talk about a few of those for the rest of this lecture. But a key idea is that if we don't know um
[49:39.040] qpi or vpi, we can estimate them uh by we can estimate the expected reward of this particular action just by acting and we can estimate um this value of how well we would expect to do using what we'll call a baseline function. We'll see some examples of baseline functions uh for the next few slides.
[50:07.359] So a baseline is just a way of um rescaling your rewards. So this is the reinforce estimator of the expected um return here. Um which you know scales uh the reward by um the log probabilities of which uses the reward to rescale this gradient of the log probabilities. And um these advantage-based methods will generally have this form here of um subtracting off a baseline from this reward. And these are different these are not numerically equivalent to each other but they are both different valid ways to estimate the policy gradient. We'll uh have a slide next that shows that this is also a valid unbiased um estimator of this same quantity. But for different like baselines that we can define um we'll get different reinforcement learning algorithms. So we'll have an example coming up which kind of motivates why subtracting a baseline uh can be a good thing to do. Um so and in that example the baseline's just going to be a constant. The baseline could also be a running average of all the rewards that you've gotten so far from all the examples that you've seen. It could also be a trained like neural model and this is the case in actor critic methods or a method called PO which we may cover in the next lecture. Um but you need a separate model from your policy for that that can be expensive. And in the final part of this lecture we'll actually not have a separate model but we will just estimate
[51:49.280] this expected reward under the policy by sampling from the policy itself. Um, and that'll be the gpo algorithm. Okay. So, any questions about this? We'll see a little bit more motivation for baselines in the next few slides, too. [snorts] So, just a quick uh note about um about uh baselines and bias.
[52:24.319] So both of these terms are estimates of the underlying expected reward. And um an estimate uh could have a bias. Estimates have bias and variance, right? A bias is like um how much is the mean of this estimate potentially shifted from the true value. And the variance is like how much spread is there in the values that I'm uh drawing for that estimate. So uh you might be worried about bias um in your estimator. And there's this uh this lema that shows that subtracting a baseline which depends only on the history up to this point which all of these baselines that we've defined here do um does not bias the gradient. So the way to see this is this is our advantage our baseline based estimate of the expected um return and we can uh decompose this the expectation of this into these terms here. So um using just the associative property putting this here um that's the definition of uh the uh partial derivative that's the that's the um the gradient for J minus this expectation here of B * this and this is going to be um equal to this if this um expectation of this term here is zero And we can see that by just um taking this term here and uh writing out the expectation like this. Um it's the expectation over um actions uh sampled from this policy. Um and bringing out this baseline term which does not depend on anything other than the history up to this point. We just get the sum over actions of the probability of those actions.
[54:26.480] And um that's going to uh we can use this same uh log derivative trick and rewrite that as this. And then this will just become one um and it'll be um a bias term times the partial derivative of one which is going to be zero. So this tells you that you can uh subtract um a baseline which depends only on the history of um actions up to that point, actions and observations up
[54:56.559] to that point without biasing your estimate of the policy gradient. But let's look at an example of how these like baselines can actually reduce the noise in your estimate of the gradient. And this is going to actually be really important because uh a huge amount of the work in reinforcement learning is uh reducing the variance in your estimates of the gradient um so
[55:24.720] that learning is stable um and robust to all the variation that you have from sampling from the policy and from the environment. But to motivate this we'll look at just a very simple example um where you just have two actions. Uh there's no multiple time steps. There's just a single decision that you make and two actions. You're either guessing a or b. And we'll have just a very simple policy which only has one parameter theta. And we'll get a probabilistic policy from this by applying the sigmoid function to theta. So sigmoid is this squashing function, right? Which um maps a real value from negative infinity to infinity um into a value between 0 and one. and we'll take that value as the probability of choosing a. So say that for our current policy we have this theta is negative 1.1 which will give us these uh very convenient probabilities of 0.25 for guessing a which has reward of one and probability 0.75 of guessing B which has probability zero. Is this a good policy or a bad policy? Think about how the probabilities correspond to the rewards. This is bad, right? Because it puts higher probability on the option that has no reward. So we want our policy gradient to update this parameter theta. Which way do we want theta to get updated? Yeah, we want it to become more positive, right? Because um if theta is larger, we'll be farther up on this
[57:12.640] curve, we'll get a higher probability for guessing A, which is the option that has higher reward. So let's compute our gradients uh without using a baseline and see if policy gradient tells us the right thing. So um we can compute the um expectation we can compute um the uh values that we we would get um as estimates of the policy gradient from each of the actions that we could take under the policy that we could sample. So if we sample guess A, what's our estimate of the gradient? Well, we get this by multiplying the reward by the partial derivative, which um using properties of sigmoid, the partial derivative here is 75. So the gradient is just the product of those, it's 75. We can do the same thing. Uh what's our estimate of the gradient if instead of sampling A, we sample B? That because the reward is zero actually gives us a zero gradient back. So this is kind of looks like the trajectories that we saw before where we didn't get any gradient from ones with reward zero. Here we're not getting any gradient if we sample
[58:23.280] this action which has reward zero. But um if you take kind of the exact definition of the policy gradient which is the expectation over all the actions in this setting where there's only two actions we can compute that exactly. we can just uh multiply each of these gradients by the probability of sampling that action and that'll be like we'll see the exact number in a couple slides but I think it's like8 so the actual policy gradient the expectation over all the actions is8 and um but we will only we'll have kind of a very noisy estimate of that where 0.25% 25% of the time we will get a sample of 75 which is much higher than 0.18 and then 75% of the time we'll get zero as an estimate for it. And what we really want is to have both of those estimates that we could sample be much closer to the true value.18. So a baseline actually does this if um so the modification from the past thing is we're just subtracting a baseline from our reward here and otherwise the computation is the same and it works out to be that if we sample um guess A we now have an estimate for the gradient of and if we sample B we have an estimate for the gradient of.125. So the expectation uh multiplied by the probability of these actions is still going to be like.18. So it's going to be the same as what we had before. But these individual samples are much closer to that true value because the baseline actually reduced the variance in the samples that we get.
[01:00:11.359] We'll have a visualization of this in another slide. But the key thing here is that that expectation does not change. It just kind of like makes the um the variance in the estimator more narrow, which is a good thing because you're going to be drawing these estimates on real data, doing updates, and you don't want your gradient to bounce you around a bunch more than you need to.
[01:00:35.839] Okay. Yeah. So, here's just a visualization of this. Um if we have a baseline of zero, here's our gradient estimates. They have this expected value, which is8. And as we drag the baseline, we can see that um we're more we get these estimates for the gradient which um converge on this uh on this true value for the expectation of the gradient. And in all cases we are going to be doing an update which makes the model more correct. We're going to be doing an update which shifts theta to be more positive which is going to increase the probability of a put more reward or put more probability on the thing with reward one. But um we're just going to be doing that more quickly if we have lower variance here in this estimate.
[01:01:30.000] And here's a graph that shows that. So um the purple line here is going to be doing draws of um the estimate for the gradient without using the baseline. So no B term just advantages sorry just scales of one or zero. And then the green line is going to be estimates that subtract a baseline just a very simple baseline of 0.5 which was you know which looked like this on this plot. So it's moved them a little bit closer to this true expectation. So uh we'll just like draw some samples and this simulates like doing steps of policy gradient on the exact same example over and over again. So here are sampling in this case we sampled guess A. So we got positive gradient um and we got a much larger gradient if we weren't using a baseline. So we'll be taking a very large step and we sample A again. And this line here is going to be our running average over all of the samples of the gradient so far. So when we sample B, now we have um a sample of the gradient that is below the true value. So we're going to be closer, our running average is going to be closer to that true value. But you can see here we're um we have less variance when we have the baseline. So our running average is already closer to that true value. And this continues like this. We happen to sample a lot of guess B's in a row. We sample guess A, it goes back up again. And it does converge eventually, but we're generally closer to it when we've reduced the variance with this baseline.
[01:03:15.680] Any questions about this? Okay. Hopefully this wasn't too in the weeds, but like um baselines have always been a little bit mysterious for me and I wanted to make these visualizations also to convince myself how they work. Um hopefully they're helpful for you too. The intuitions for this very simple uh two armed bandit setting do um transfer generally to these more complex
[01:03:40.480] um you know sequential decision-m settings too. Yeah. So um baselines are helpful because they can mark a trajectory as better or worse than expected. And uh if you've chosen a good baseline, a lot of reinforcement learning is about choosing good ways of representing this advantage function. It can help to reduce gradient variance and and stabilize your learning. Um but you have some downsides too and a lot of choices about what you use for the baseline. So with um and I'll skip over this um other than just to say if you are working in a setting that gives you rewards at intermediate time steps rather than just at the end of the trajectory um you'll you'll replace your reward scaling with um the reward to go
[01:04:28.799] like the sum over rewards um over future time steps and this will also help to reduce the variance. Um, we'll have this slide here and I we can also talk about this offline if it's something that's relevant to you. Um, but uh it's um how you would handle not having all of your rewards come at the end of the task.
[01:04:51.399] [snorts] Okay. So, with the last um bit of time that we have, we'll cover a simplified version of um GRPO um which is uh group relative uh policy optimization I believe. Um, and it's a way of um e of getting this advantage function of estimating a baseline that uses rollouts from the model itself. Um, and this was um this was actually introduced pretty recently. So it was in one of the papers from deepseek that was training reasoning models um with a big focus on efficiency. So a lot of past work on um applying RL to language models required a separate model to predict this baseline. Predicting baselines to you know reduce the variance is very important but was challenging and people were using separate models for it. But DeepSeek was very interested in efficiency um and training models without using much compute and having a separate model takes a lot of compute. So they proposed this approach which gets rid of the separate model and just uses multiple samples from the model itself to estimate the expected value the expected reward under the policy which is really kind of elegant. Um so the idea is that
[01:06:11.200] we can sample multiple times from the model for a given instance and then get the reward on each of those. Um and then we can um center that reward by taking the mean of the reward across all of these instances in the group. We get a mean. We um basically use that mean as our baseline. And this makes sense, right? Because you know the baseline is your expected a good baseline is your expected reward under the policy. And a really good estimate of that is just running the policy multiple times and getting the reward taking the mean of those. So we get our centered reward here. And in the original GRPO paper, they're also standardizing this. Um so they're not just shifting it, they're rescaling it. they compute the standard deviation over the rewards in this group of trajectories and then they divide by that standard deviation um to get the advantage functions. Now there's um like actually it seems like dividing by the reward is not necessary and a lot of implementations of uh that that call themselves GRPO actually get rid of this um division by the standard deviation. So it more closely matches the advantage term. Um there's a paper we'll talk about in a few slides called DRGP uh Dr. gpo or done right gpo um which talks more about the motivations for that um but actually in a lot of implementations of gpo they won't divide by the standard deviation they will just subtract the mean um but yeah the gpo update is very simple
[01:07:57.119] looks a lot like policy gradient that we defined before but so you have like a task instance which is you know like one RL problem to solve you will generate multiple rollouts from it. So you will solve the same problem multiple times, resetting the environment, um getting a full trajectory by sampling from your agent and um this will typically uh Aorva in his lecture will give some guidance on like choosing values for this group size I believe, but this is typically something like eight. Um and uh you'll get the mean of rewards over those rollouts and you'll get your advantage estimates from that. Um, we'll show two different ways of getting those advantage estimates in another slide, but that gives you a group of rollouts for one trajectory along with advantages for them. If you have room in your GPU uh to fit in more examples, you can also get groups of the same size for additional instances. And you'll be computing group means um only within each example. So this is important right because the value function should just be an estimate of how um it should be an estimate of the reward on this particular example. Um so our advantages should only look at the reward within this example. But you will get an advantage for each trajectory by um scaling the reward for that trajectory using the um the formula that shifts by the mean and maybe divides by the standard deviation of the rewards within that instance.
[01:09:39.359] So for policy gradient loss we were just taking a single sample for each um for each instance and then rescaling. But in gpo we are going to be um having multiple instances in our batch and then multiple trajectories within the group for each instance and we'll get an advantage for each of those. We'll use that to rescale our log probabilities before differentiating.
[01:10:06.800] Any questions about this? Yeah. >> Um is this unbiased? Yes, because um you your advantage function here um depends on nothing except the prompt and the model. So your baseline function here um which you're subtracting from the reward is just um rollouts from the language model. So this is unbiased.
[01:10:40.159] Okay, cool. Yeah. So, um, Dr. GRPO makes two changes to to GRPO. And I should note too, GRPO also, so what what I wrote here is not the loss for GRPO as defined in the paper. Um the loss in the paper has some additional terms which I think Graham will talk more about in his lecture which also like clip the gradient so that you don't take steps too far in one direction or another and also regularize the policy so that you don't uh deviate too much from the policy that you started with. Both of those are additional things that help stabilize training. We're not going to talk about them today. We're just talking about the advantage functions within GRPO. Um but uh yeah, this will help you uh get started on um assignment three when that
[01:11:32.400] comes out. Okay, great. So, we're good on time. Um so, uh DRGRPO um makes two changes to GRPO. Um and you will also do these in your assignment. Um and they seem generally helpful to do um because they help avoid some biases within GRPO. And the two changes are just the first one we've talked about a little bit already. It doesn't divide by a standard deviation. It just subtracts the mean of rewards within the group. And um yeah, the motivation for that is that uh uh if you're dividing by the um standard deviation, your advantages are no longer aware of the difficulty of the problem. um because you're sort of like rescaling all ranges of difficulties to be standardized. Um so that can actually be bad. GRPO also um divided by the number of tokens generated which can bias the model towards um uh towards uh long wrong answers and so DRGRPO just divides by a constant which is the max number of tokens that could be generated. Okay. So, same figure that we saw before. How does this actually look when you're implementing it? You have your generated tokens in blue. Uh, you compute an advantage function which is going to um take your reward for this trajectory and just subtract off the mean uh reward from all of your other trajectories. And then you get a per token reward which just normalizes that by a constant C, which is typically the number of tokens that could have been generated. And um you do this for all of the rewards in your all the trajectories in your group which could have varied rewards positive or negative. And the loss looks like this. Any questions about this? Yeah. And um yeah, so just to note again the original paper also has like a a ratio which helps regularize the policy and clipping, but we won't talk about those for now.
[01:13:46.400] Okay, so we had a question before about like what if all of your rewards are zero? Um, that's bad because if all of your trajectories fail, that means it's just probably too hard for the current policy and you're not going to get any signal from that even if you have a baseline. Um, you also have a problem if all of your trajectories succeed because they're too easy for the current policy. So like you know if this is binary rewards and um one is the best that we could possibly do um it's bad um and generally it's bad if we don't have any spread um within our examples because then we won't have any signal from the advantage function as estimated by gRPO.
[01:14:33.120] So what could we do if every you know sample trajectory fails? We could choose different task instances. We could um inject some some knowledge into the prompt that encourages the model to produce more guided solutions that might be correct. We could add demonstrations. Um there's ways of doing off policy RL that put like uh correct demonstrations into the group. Um or more simply, you could just fine-tune the model first on those correct demonstrations before running RL. Or you could have a more informative reward which um uh encourages the model to make correct intermediate decisions before producing an entire trajectory. That's called
[01:15:14.159] reward shaping. So the like giving a model reward for finding the correct file could be an example of that. Um we'll skip this because we'll uh potentially talk more about PO later on. But PO basically just um instead of using the trajectories to compute the advantage function has a separate model which usually shares a backbone with the policy um to predict the the baseline.
[01:15:44.800] Okay. So very quickly just comparing the methods that we've seen so far. every method has the agent take actions in the environment and then we'll just um compute a loss uh using those actions produced by the agent. All of these look like selftraining in some way but they just differ in the way that they rewe those generated actions. So restm um we assume that we have binary rewards of zero or one and we just filter to the trajectories which are correct and then do sftt on those. Um reinforce uses the reward of the trajectory to rewe the loss which corresponds to reweing the gradient which we spent a lot of time showing is correct. And then GRPO uh uses these advantage functions where the reward is um is uh shifted and potentially also rescaled by the reward of other trajectories sampled by the model for the same example. And all of these methods help to address the three problems that we had at the very beginning of the lecture. So um task mismatch uh should now be solved because we're now maximizing reward directly. um expected reward directly data mismatch. We can now learn from trajectories that have reward zero as
[01:17:09.199] long as we're doing baselines in the appropriate way. And also um the on policy nature of these methods helps with exposure bias because the model's like essentially driving at training time. So it should be learning to recover from its own errors. um it just comes at the expense of you know you have to actually just try stuff in the environment which can be costly can introduce some noise into your gradients and we've covered some ways of trying to make that process more robust but in the next two lectures starting next week we'll we'll talk about some others too okay that's all we