CMU 11-768 Fall 2026 · Lecture 5已发布

Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination

日期: Tue, Sep 8, 2026
课程资料: 38 项
内容产物: 4 项

课程简介

规划、任务分解与多智能体协调，组织长程任务。

CMU 11-768 Fall 2026 · Lecture 5 学习笔记

来源：平台字幕与官方 Slides。内容已由用户审核确认。

Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination

依据：官方 Lecture 5 slides（https://www.cmu-agents.com/slides/lecture-05-planning.pdf）与本次讲座中文逐字稿整理稿。逐字稿为机器翻译、状态：用户审核通过，图示与公式需对照官方 PDF/视频核验。本笔记已由用户审核确认。

一、学习目标

学完本讲后，你应当能够：

用一句话给出 LLM 智能体语境下“计划”的工作定义，并说明它与“技能（skill）”的区别（Slides 第 5 页）。
解释为什么长时程任务会出现“结构”，并用 GPU 采购任务说明并行、依赖、迟到信息与不可逆步骤（Slides 第 2–3 页）。
对比推理（CoT）与在环境中行动（acting）的区别，说明为什么行动给规划带来可行性、后果、信息与恢复四个新问题（Slides 第 12 页）。
复述经典规划的基本要素：状态即谓词集合、动作有前置条件与效果、规划即搜索（Slides 第 13 页）。
说明 LLM 时代“难的东西变了”：写一个看似合理的计划更容易，判断世界模型是否正确更难（Slides 第 14 页）。
列举计划的三种表示谱系——形式化计划、自然语言计划、程序——并说明各自的可检查性与通用性（Slides 第 15 页）。
说明“执行前承诺”与“执行中重规划”的权衡（Slides 第 16 页）。
说明把规划器与执行器分离的动机、方法与风险（Slides 第 19–21 页）。
解释可供性（affordance）建模如何给计划打分（Slides 第 22 页）。
解释“思考 vs 行动”的计算分配，以及“过度思考”的三种坏行为模式（Slides 第 24–25 页）。
说明长时程任务中错误自条件化（self-conditioning）如何放大失败（Slides 第 28–30 页）。
描述 MACU 的基于图的多智能体框架如何分解、执行与重规划（Slides 第 38–44 页）。
使用讲末设计清单在“是否加入规划结构”上做判断（Slides 第 46 页）。

二、概念主线

本讲的主线可以概括为：计划是对未来行为的显式表示；它在长时程任务中带来模块化、反馈、长时程与控制四种压力；每种压力对应一类方法；但每一种方法都有代价。

开篇的动机例子是 GPU 工作站采购（Slides 第 2 页）：任务是跨多家供应商网站比较配置与价格，对照内部 wiki 检查兼容性，最后提交采购申请。一个反应式智能体用 ReAct 循环可能要走数百步：去 vendor-a.com、Workstations、spec sheet、GPU comparison……第 31 步观察到 wiki 说“机架接受 4U 机箱；每槽 1600W”；上下文被压缩 128k → 14k tokens；搜索了四家供应商，只有 D 符合规格，但交货期 20 周；第 201 步它要在“选 D？找第五家？放宽需求？”之间做决定（Slides 第 2 页）。这就是“一拳打在嘴上”的现实版本——计划必须可被环境改写（Slides 第 5 页引用 Mike Tyson）。

Slides 第 3 页把这个任务画成依赖图：Vendor A–E 的搜索相互独立，兼容性检查依赖各家返回的信息，采购申请是后到的、不可逆的一步；“当四家都不合适时，图本身也要改”——加进第五个搜索节点。这一页给出本讲的四个问题：价值（何时有用/何时有害）、表示（如何表示）、承诺（何时制定、什么可以改）、监督（人何时介入）。

随后讲座把规划放进课程地图（Slides 第 6 页）：本讲讲提示式计划与 harness 结构；编码智能体一讲讲 Agentless 的固定工作流；Interaction 1 讲多智能体角色与通信；Interaction 2 讲澄清与人工批准；Search 1–2 讲对计划与预测未来的搜索，并重访 MACU 与 RAO。

历史脉络分两支。第一支来自 LLM 的思维链：Kojima et al. 2022 与 Wang et al. 2023 表明“Let's think step by step”等提示能让模型更准；CoT 提供额外计算与草稿纸，计划与解答在同一次生成中产生，没有任何显式检查（Slides 第 8 页）。第二支是分解：Least-to-Most（Zhou et al., ICLR 2023）先写子问题再按序回答并向前传递答案，理由包括子问题更简单、所需上下文更少、随问题变长收益增大（Slides 第 9 页）；Decomposed Prompting（Khot et al., ICLR 2023）用分解器生成子任务并路由到专用处理器，例子是“1910 年出生的人制作的电影拿过什么奖？”先变成“谁出生于 1910 年？”（简单 QA）再变成“#1 是哪些电影的制片人？”（位置 QA）（Slides 第 10 页）。

但真实世界中行动会失败：Slides 第 12 页逐项对比——CoT 只改文本、可被后续文字撤销、中间步骤全部可见；世界中的动作改变智能体下一步面对的状态、可能揭示信息、可能失败、可能不可逆（submit、send）。因此计划必须处理可行性、后果、信息与恢复。

接着回望经典规划（Slides 第 13 页）：状态是谓词集合（on-table(x)、on(x,y)、clear(x)、holding(x)、hand-empty），动作有前置条件与效果（pick-up(x): on-table(x) ∧ clear(x) ∧ hand-empty → holding(x) ∧ ¬on-table(x) ∧ ¬hand-empty），规划就是在搜索空间里找一条满足每个动作前置条件的有效动作序列（示例：unstack(blue,orange)、put-down(blue)、pick-up(orange)、stack(orange,blue)）。Slides 第 14 页点出转变：对 LLM 来说一切都是隐式的，写一个看似合理的计划更容易，判断世界模型是否正确更难；因此当代智能体设计“少做深搜、多查世界”——观察、验证、从错误恢复，但搜索与动作适用性仍是有用概念。

然后是表示谱系（Slides 第 15 页）：形式化计划可检查，语言计划通用，程序居中；2023 年前后一批工作（Code as Policies、ProgPrompt、Binder）用代码语言模型写调用感知与控制 API 的程序；程序可运行、可检查，但受 API 与程序结构限制；不过 API 本身可以调用 LLM 或另一个神经网络。

承诺与重规划的权衡（Slides 第 16 页）用一张轴表示：开发者定义工作流（固定供应商列表的 for 循环）→ 先计划后执行（模型决定查哪些供应商，执行前提交）→ 自适应智能体（四家失败后依据执行反馈去找新的）。事先承诺更快、更便宜、更可预测；重规划让模型在揭示新信息时适应；当反馈值得额外成本与复杂度时才重规划。Slides 第 17 页给出加入规划结构的四个理由：modularity、feedback、long horizons、control，并引用 Fred Brooks：“Plan to throw one away; you will, anyhow.” Slides 第 18 页把四者映射回 GPU 任务：搜索分开做（模块化）、都不合适就改计划（反馈）、需求与进度贯穿全程（长时程）、提交前由人批准（控制）。

四类压力各有对应方法。模块化：UGround/Agent S 用 GPT-4o 做视觉接地规划器、用合成数据训练的 7B 执行器把自然语言指令映射为屏幕坐标（Slides 第 19 页）；Plan-and-Act 用教师模型对成功轨迹分段打标签，再微调规划器与执行器（Slides 第 20 页）。但固定角色不安全：Slides 第 21 页引 Neubig 2024 指出编码角色分解成功有限——角色在任务到来前就固定，验证者无法把故障定位到自己的答案；交接是摘要报告，可能丢掉下一个智能体需要的上下文。反馈：SayCan（Ahn et al., CoRL 2022）为每个机器人技能学一个价值函数预测完成概率，与 LLM 给出计划的概率相乘（Slides 第 22 页，公式形式待核）；LLM-Planner、Inner Monologue、SwiftSage 是环境反馈重规划的代表（Slides 第 23 页）。思考 vs 行动：Test-Time Interaction（Shen et al., NeurIPS 2025）在相同计算预算下比较“更长 CoT”“采样多候选”“更多与环境交互”三种方法，交互往往更好（Slides 第 24 页，具体数值需核验原图）。过度思考：Cuadron et al. 2025 识别三种模式——分析瘫痪、流氓动作链、过早退出，过度思考与问题解决率负相关（Slides 第 25 页）。思考 vs 观察：Calibrate-then-Act（Ding, Tomlin, Durrett, 2026）以分隔符、引号字符、跳过的表头行共十二种组合的 CSV 解析为例，把格式似然写进提示，模型读出“分隔符最可能是 ';'，约 0.85……我不确定，也许该跑单元测试确认”（Slides 第 26–27 页，数值待核）。

长时程：Vending-Bench 让智能体长时间经营自动售货机，最长约 2000 条消息、约 2500 万 token、最多 222 个模拟日；一个例子是“到期应付 ≠ 已配送”，环境正确报告“物品不可用”，智能体却推断“生意失败了”，从未等待履约邮件或复查，错误持续到把每日费用当成“欺诈”并发出“紧急：业务终止后的未授权费用”邮件（Slides 第 28 页）。Sinha et al. 2025 说明更好的单步准确率延长可执行视界，而自条件化会让后续步骤更不准；更大的模型能执行更长任务但自条件化更强，RL 是否能修复这一点尚属开放（Slides 第 29–30 页）。RAO（Gandhi et al., 2026）把委派本身当作可训练的动作：launch_subagent(goal) 启动自身副本处理子任务，副本还能再委派；每个节点按自身任务得分加子节点成功频率计分；只在中等难度任务上训练，却在难任务上委派得更深（Slides 第 31 页）。安全视角：ReAct 让每个页面都能影响下一个动作，形成注入路径；程序可以接受来自页面的值而不接受新动作（Piet et al., 2026，Slides 第 32 页）。控制：Cursor 的 Plan Mode 演示只读调查→澄清问题→写出计划→人工审阅→执行，并允许直接编辑计划（Slides 第 33 页）。全局约束：TravelPlanner（Xie et al., ICML 2024）提醒航班、酒店、餐饮、景点可以干净分解，但预算、交通、饮食是全局的，跨子任务常被违反（Slides 第 34 页）。

最后把一切拼起来：Odysseys 基准（Jang, Koh, Fried, Salakhutdinov, 2026）中“为腿部 ACL 找 ACL 外科医生”这类任务需要数百步、半小时到一小时，并有评分标准（Slides 第 36–37 页）。MACU（Koh, Salakhutdinov, Fried, 2026）用 manager 把用户任务分解成 DAG，按依赖顺序用并行子智能体执行，manager 检查结果、要求重做或改图，循环直到图处理完（Slides 第 38–39 页）。消融显示：更强 manager 改善协调（Qwen3.5-4B worker，36 个任务；Slides 第 41 页），更弱 worker 被提升更多（Opus 4.6 manager；Slides 第 42 页），Odysseys easy 子集 45 个任务上有 worker 并行度结果（Slides 第 43 页），修改图的次数影响成功率（Slides 第 44 页）——具体数值均需核验原表。

三、关键机制

1. 计划模式（Plan Mode）。 Cursor、Claude Code、Codex 都引入：只读调查→澄清问题→书面计划→人工审阅→执行，可能带子智能体并行（Slides 第 4 页）。逐字稿补充的动机：先做非破坏性信息收集，再让人验证或直接编辑计划，确认后点 build 执行。

2. 计划的工作定义。 计划是对预期未来行为的显式表示：智能体将尝试的动作或子目标，以及它们之间的排序或依赖；计划是“这个任务的”，技能是跨任务可复用的；计划是提议而非保证，可被用户、另一个模型或环境交互修改（Slides 第 5 页）。

3. 经典规划 vs LLM 规划。 经典规划：状态=谓词集合，动作=前置条件+效果，规划=搜索（BFS/DFS/A* 加启发式）；LLM 规划：一切隐式，写计划容易、验证难，所以用观察、验证、恢复替代深搜（Slides 第 13–14 页）。逐字稿补充的类比：经典规划里你确切知道动作何时适用、做了什么；LLM 里这些大多在智能体外部，只能到环境里试。

4. 计划表示谱系。 形式化（可检查）—程序（居中，可运行、可检查，受 API 限制）—语言（通用）（Slides 第 15 页）。

5. 规划器/执行器分离。 不同模块可用不同模型与训练数据；UGround 用 7B 执行器补足 GPT-4o 的像素级操作短板（Slides 第 19 页）；Plan-and-Act 用教师标注轨迹微调两者（Slides 第 20 页）。

6. 可供性重打分。 为每个技能学价值函数，预测在特定上下文中完成该自然语言子任务的概率，再与 LLM 的生成概率相乘（Slides 第 22 页；乘积形式与符号定义待核）。逐字稿例子：视野里没有苹果时“拿起苹果”概率低，“找到它们”概率更高。

7. 交互式重规划。 执行失败后把自然语言反馈（“找不到土豆，但看到冰箱”）回灌模型，重预测计划（Slides 第 23 页）。

8. 计算分配。 相同预算下比较更长 CoT、多候选采样、更多环境交互；交互通常更值（Slides 第 24 页）。过度思考三模式：分析瘫痪、流氓动作链、过早退出（Slides 第 25 页）。

9. 置信度校准。 把隐藏格式的似然写进提示，让模型显式读出不确定性并选择验证动作（Slides 第 26–27 页）。

10. 递归委派。 launch_subagent(goal) 作为动作被训练；节点得分=自身任务分+子节点成功率；中等任务训练、难任务泛化（Slides 第 31 页）。

11. 计划作为安全边界。 程序只接受页面传来的值，不接受页面传来的新动作（Slides 第 32 页）。

12. MACU 图执行。 manager 生成 DAG→按依赖并行执行→检查结果→重做或改图→直到完成（Slides 第 38–44 页）。

四、公式的假设与符号

本讲幻灯片中的“公式”主要是定性示意，文本提取无法确认完整数学形式，以下均标注为待核：

SayCan 的重打分（Slides 第 22 页）：文字描述为“把这个价值与 LLM 产出计划的概率相乘”。设 LLM 给子任务 ℓ 的分数为 p_LLM(ℓ|context)，技能价值函数为 p_afford(ℓ|state)，组合分大致形如 p_LLM · p_afford。假设：两个量被当作可乘的独立或近似独立打分；p_afford 由机器人学技术学得，预测“在特定上下文中执行该自然语言子任务”的概率；LLM 分数被描述为“类似 logits 的和”（逐字稿），是否先做 softmax 后再相乘、是否有温度或归一化，待核原图与论文。
长时程成功率相乘（Slides 第 28–30 页）：逐字稿给出直觉“0.9 × 0.9 × …”，即轨迹成功率≈单步准确率的步数次幂。假设：各步近似独立；实际并非独立，因为后续动作以早期上下文为条件（自条件化），而这正是 Slides 第 28–30 页强调的失效来源。
递归节点得分（Slides 第 31 页）：“每个节点按自身任务得分加子节点成功频率计分”。符号未在提取文本中出现，加权方式待核。
RAO 的奖励（逐字稿）：“优化模型执行任务的能力，也优化模型生成代码的能力，以及子任务成功的频率”。具体奖励项、权重与训练目标待核原论文。
执行视界图（Slides 第 29–30 页）：横轴单步准确率、纵轴可正确执行的任务长度，模型大小与自条件化的关系，均为图，待核原图。

五、误区

“计划就是保证。” 不是。计划是提议，不是保证，可被用户、另一个模型或环境修改（Slides 第 5 页）。
“CoT 就是规划。” CoT 是计划的一种，但计划与解答在同一次生成中产生，没有任何显式检查（Slides 第 8 页）；而行动会改变状态、可能失败、可能不可逆（Slides 第 12 页）。
“LLM 规划不需要搜索，所以经典规划没用了。” Slides 第 14 页明确说搜索与动作适用性仍是有用概念，只是重点从深搜转向观察、验证与恢复。
“多智能体角色分解总是更好。” Slides 第 21 页提醒：角色在任务到来前固定，验证者无法把故障定位到自己答案；交接是摘要，可能丢掉下一个智能体需要的上下文。
“想得越多越好。” Slides 第 25 页：分析瘫痪、流氓动作链、过早退出都与更差的问题解决相关。
“只要模型够大，长时程就会好。” Slides 第 30 页：更大模型执行更长任务但自条件化更强，RL 能否修复仍是开放问题。
“计划一旦确认就不能改。” Slides 第 44 页的消融反而显示允许修改图的次数越多，成功率上升；限制修改次数约为 25%（逐字稿给出，待核原表）。
“分解一定安全。” Slides 第 32 页：ReAct 让每个页面都能影响下一个动作，形成注入路径；程序化计划只接受值、不接受动作，才构成边界。
“分解能覆盖所有约束。” Slides 第 34 页：预算、交通、饮食等全局约束跨子任务，常被违反。
“有了 plan mode 就不需要人工。” Slides 第 4、33 页：plan mode 的核心之一正是人工审阅与直接编辑计划。

六、复习问题

用工作定义说明计划与技能的区别（Slides 第 5 页）。
在 GPU 采购任务里，哪些步骤可并行、哪些有依赖、哪一步不可逆、哪一步是“图本身要改”的例子（Slides 第 2–3 页）。
为什么说“对 LLM 来说难的东西变了”（Slides 第 14 页）？
写出 pick-up(x) 的前置条件与效果，并说明什么是一条有效动作序列（Slides 第 13 页）。
形式化计划、程序、语言计划各自的可检查性与通用性如何权衡（Slides 第 15 页）？
开发者定义工作流、先计划后执行、自适应智能体三者的成本与适应性差异是什么（Slides 第 16 页）？
为什么把规划器与执行器分开？UGround 与 Plan-and-Act 分别怎么训练（Slides 第 19–20 页）？
固定角色多智能体的两个具体弱点是什么（Slides 第 21 页）？
可供性价值函数如何与 LLM 分数结合？它解决什么问题（Slides 第 22 页）？
Test-Time Interaction 比较了哪三种推理方法？结论是什么（Slides 第 24 页）？
过度思考的三种模式是什么？它与问题解决率的关系如何（Slides 第 25 页）？
用 CSV 解析例子说明“思考 vs 观察”的差别（Slides 第 26–27 页）。
Vending-Bench 的例子中，错误是如何从“暂时缺货”一步步演化成“欺诈”的（Slides 第 28 页）？
自条件化为什么会让长时程任务更脆弱（Slides 第 28–30 页）？
RAO 把什么当作动作来训练？为什么只在中等任务上训练却能泛化到难任务（Slides 第 31 页）？
为什么说“计划可以是安全边界”（Slides 第 32 页）？
TravelPlanner 提醒的全局约束问题是什么（Slides 第 34 页）？
复述 MACU 的 manager–worker–DAG–重规划循环（Slides 第 38–44 页）。
用讲末设计清单的四个问题评估一个你熟悉的任务（Slides 第 46 页）。
Slides 第 47 页列出的四个开放问题分别是什么？

七、官方来源链接

课程主页：https://www.cmu-agents.com/
课程日程与阅读：https://www.cmu-agents.com/#schedule
作业与项目要求：https://www.cmu-agents.com/#assignments
Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
课程视频：https://www.youtube.com/watch?v=S8v-dR4s29M&list=PLSN0qpDfUvTM&index=5
官方课程日程：https://www.cmu-agents.com/#schedule
Lecture 5 slides：https://www.cmu-agents.com/slides/lecture-05-planning.pdf

注：本笔记基于逐字稿整理稿（机器翻译、待审核）与 slides 文本提取（待核）编写；所有图示、公式、数值与未在文本中出现的结论均标注为待核，未作猜测。

从 ReAct 到计划图：CMU 11-768 第五讲拆解智能体的规划、任务分解与多智能体协调

来源：平台字幕与官方 Slides。内容已由用户审核确认。

一、一个让人犹豫的任务

先看一个具体的任务。你要为一个研究实验室采购新的 GPU 工作站：比较不同供应商网站上的配置和价格，对照内网 wiki 上的集群兼容性要求做检查，然后提交采购申请。这个任务并不“难”，但它很长。课程幻灯片把它拆开来看：一个使用标准 ReAct 循环的智能体需要大约数百步——访问 vendor-a.com，找到 Workstations 页面，读取 spec sheet，做 GPU 比较，第 15 步才配置出“2× RTX 6000、128 GB RAM、850 W PSU”，第 31 步才从 wiki 上读到“机架只接受 4U 机箱，每个插槽 1600 W”。上下文中途被压缩，从 128k 降到 14k token。等它搜完四个供应商，发现只有 D 符合要求，但交货期长达 20 周。第 201 步它面临一个抉择：选 D？找第五家？放宽某个要求？第 202 步它还是去采购门户提交了申请（Slides 第 2 页）。

这就是本讲的核心张力：一个只会“反应”的智能体，在长任务上会积累上下文、丢失结构、在关键时刻无法回头。Daniel Fried 在开场就抛出这个问题：规划——让智能体对未来行为做出显式承诺——到底有没有用？什么时候它不值得额外的成本和僵化？怎么表示计划？什么时候该改计划？人应该在什么时候介入（Slides 第 3 页）？

学习者解释：这里的“ReAct 循环”指智能体反复执行“思考—行动—观察”的单步循环。它的特点是每一步只看当前上下文，不为未来行为保存显式结构。逐字稿中讲者把它称为“标准 React 循环”，并说这个任务“可能需要大约数百步”。

二、本讲对“计划”的工作定义

课程给了一个可以直接引用的工作定义（Slides 第 5 页）：

计划是对预期未来行为的一种显式表示：智能体将尝试的动作或子目标，以及它们之间的排序或依赖关系。

三个限定词很关键。第一，计划是针对这个任务的，而技能（skill）是跨任务可复用的指导；计划可以比技能更具体，甚至绑定到某个特定仓库。第二，计划组织未来的工作，它可以规定动作或子目标，以及排序和依赖。第三，计划是提议，不是保证，它可以被用户、另一个模型或环境交互检查与修改。幻灯片引了泰森那句话：“每个人都有一个计划，直到嘴上挨了一拳。”

讲者在逐字稿中用了同一段话来解释这个定义的边界：计划是模型可以参照的产物，它既总结过去做过什么（类似上下文压缩），也指定未来应该做什么。它建立在思维链的思路之上，但在智能体场景里更麻烦——因为智能体会从环境中揭示新信息，计划必须始终可修改。这就引出了本讲的第一个权衡：智能体应该在多大程度上承诺某个行动方案（便于人类理解和监督），又应该有多灵活（能应对沿途出现的事情）。

三、为什么长任务需要结构

把那个 GPU 任务画成图，结构就出来了（Slides 第 3 页）。Vendor A/B/C/D 的搜索彼此独立，可以并行；兼容性检查依赖各供应商返回的信息，但供应商之间互不依赖；采购申请在兼容性检查之后，而且它是不可逆的。更麻烦的是，图本身会变：如果四家都不合适，就得加入第五个搜索节点。这正是“重新规划”（replanning）——你有一个结构化的表示，并根据环境观察修改它。

幻灯片把本讲的问题拆成四个维度（Slides 第 3 页）：

价值：规划什么时候有用？什么时候有害？
表示：计划怎么表示——文本？结构化图？代码？
承诺：计划何时做出，什么可以改变它？
监督：人应该在何时、以何种方式介入？

学习者解释：依赖图（DAG）在这里的作用是让“哪些子任务可以并行、哪些必须等待”变得显式。幻灯片第 3 页明确标出“Vendor E”是后来加入的节点，用来说明图结构本身也要能被修改，而不只是图上的执行结果会变。

四、思维链：规划的史前史

讲者先快速回顾了 LLM 之前的“规划”传统（Slides 第 8 页）。Kojima 等人 2022 年的工作发现，只要提示模型“让我们一步一步地思考”，就能提高答案准确率；Wang 等人 2023 年则提示模型先“理解问题并制定计划”，再执行计划。这些工作发生在模型被训练来显式产生思维链之前。思维链的作用是给模型提供额外计算和一块草稿板：它接收这些 token，在预测答案前做额外操作，也能回头参考中间结果。这与智能体规划的动机相同。

但幻灯片也点出了关键差别：计划和解答在同一次模型生成中到达，没有任何东西显式检查这个计划。后来的工作（STaR、DeepSeek-R1）把思维链训练成提高正确答案概率的手段，但“检查”这件事仍然缺席。

接下来是分解（Slides 第 9 页）。Least-to-Most（Zhou 等人，ICLR 2023）把“规划”和“求解”分开：第一阶段写出子问题，第二阶段按顺序回答并向前传递答案。动机是模型不擅长一次回答涉及多个推理步骤的问题，而回答子问题所需上下文更少，改进幅度随问题长度增长。Recursive Language Models（Zhang 等人，2025）被归为同一类。

学习者解释：逐字稿里讲者用了一个具体例子来说明“向前传递”：先问“每趟旅行需要多长时间”，得到“五分钟”，再把完整问题和这个答案一起喂给下一个子问题。对当时的模型来说，这很关键，因为它们不太擅长一次完成多个推理步骤。

分解还带来模块化（Slides 第 10 页）。Decomposed Prompting（Khot 等人，ICLR 2023）用一个分解器生成子任务，并把每个子任务路由到专门的处理器。幻灯片给的例子是：“由 1910 年出生的人制作的电影获得过什么奖？”被拆成“谁出生在 1910 年”（simple QA）和“哪些电影的制作人是 #1”（position QA）。模块化允许独立改进处理器模型（通过上下文示例或训练），也提供了程序化/结构化控制。

五、当计划开始作用于世界

思维链只改变文本：它可以被更多文字撤销，每个中间步骤都可见。而世界中的动作会改变智能体接下来遇到的状态、可能揭示信息、可能失败、可能不可逆——提交、发送都是例子。幻灯片第 12 页总结道：行动意味着计划必须处理可行性、后果、信息和恢复。

这就把本讲引向经典规划（Slides 第 13 页）。经典规划里，状态是一组谓词，动作有前置条件和效果。幻灯片用积木世界举例：pick-up(x) 的前置条件是 on-table(x) ∧ clear(x) ∧ hand-empty，效果是 holding(x) ∧ ¬on-table(x) ∧ ¬hand-empty。规划是一个搜索问题：找到一串有效动作，从初始状态到达目标。幻灯片给的示例计划是 unstack(blue, orange) → put-down(blue) → pick-up(orange) → stack(orange, blue)。它引用了 STRIPS（Fikes & Nilsson, 1971）和 PlanBench（Valmeekam 等人, NeurIPS 2023）。

但 LLM 时代“难的东西变了”（Slides 第 14 页）：对 LLM 来说一切都是隐式的。写一个看似合理的计划更容易，判断它对世界的模型是否正确更难。所以智能体设计用更少的深度搜索，更多地去检查世界：观察、验证、从错误中恢复。但搜索和动作适用性仍然是有用的概念。

学习者解释：这里不是“经典规划被淘汰了”，而是重心转移。逐字稿中讲者说，LLM 见过大量关于各种情境的信息，所以能直接生成一个大概率合理的初步计划；问题不在于“生成”，而在于“检查它是否真的可行和可执行”。

六、计划可以采用什么形式

幻灯片第 15 页给出一个谱系：形式化计划可检查，语言计划通用，程序在两者之间。2023 年前后有一系列工作用代码 LLM 写出调用感知和控制 API 的代码（Code as Policies、ProgPrompt、Binder）。程序可运行、可检查，但受限于 API 和程序结构允许什么。而 API 本身可以调用 LLM 或另一个神经网络模型。

讲者在这里补了一句重要的现实判断：这类方法在机器人领域没有像端到端多模态 VLA 系统那样流行起来，但对编码智能体仍然非常有用——下一讲的 CodeAct 会涉及。

关于“什么时候重新规划”，幻灯片第 16 页给了三档（Anthropic《Building effective agents》2024 被引为来源之一）：

开发者定义的工作流：开发者指定过程，比如检查一个固定的供应商列表。幻灯片给了一段伪代码：for v in VENDORS: specs[v] = read_specs(v)，然后 ok = [v for v in specs if fits(v)]，最后 requisition(cheapest(ok))。
先计划后执行：模型规划要检查哪些供应商，然后在执行前承诺。
自适应智能体：四家供应商失败后，执行反馈让模型去寻找另一家。

权衡是：提前承诺一个计划更快、更便宜、更可预测；重新规划让模型在執行揭示新信息时能适应。当反馈值得额外成本和复杂性时，才用重新规划。

七、规划压力的四个来源

幻灯片第 17 页把“为什么要加规划结构”归纳成四个词：模块化、反馈、长时程、控制。第 18 页用 GPU 采购任务把这四个词具体化：

模块化：搜索分别执行。
反馈：都不合适，所以修改计划。
长时程：需求和进度贯穿整个任务。
控制：人在智能体提交前批准。

接下来的方法论文献，就是围绕这四个压力展开的（幻灯片第 19–32 页）。

7.1 模块化：规划者与执行者分离

UGround（Gou 等人，ICLR 2025）和 Agent S（Agashe 等人，ICLR 2025）是幻灯片给的例子。规划者模型（当年是 GPT-4o）有视觉 grounding，但不擅长产生像素级动作，于是让它用语言描述下一步动作；一个单独训练的 7B 执行者模型把这些描述映射到屏幕坐标。讲者解释了为什么这在当时有效：最强模型不擅长处理网站的细粒度视觉细节，而分解并模块化让你能独立优化每个组件，在弄清瓶颈后对每个组件做“爬山”。

幻灯片第 20 页给了怎么训练规划者：用教师模型对成功轨迹做分段和标注，产生计划；然后在这些标注轨迹上微调规划者模型和执行者模型（Plan-and-Act，Erdogan 等人，ICML 2025）。讲者指出这有点类似上一讲的工作流归纳：都从一条轨迹开始，做分段和标注，区别在于这里是用它来生成训练数据。

7.2 模块化的代价：警惕固定角色

幻灯片第 21 页是一个重要的警告，标题是“Beware of fixed roles and multi-agent systems”，并标注了 Graham Neubig。要点有两个：为编码做角色分解的成功有限；角色在任务到达之前就固定了，验证者无法定位故障来检查自己的答案；交接是摘要报告，可能丢掉下一个智能体需要的上下文。讲者在逐字稿中展开说：一些工作让模型模拟软件公司或工程师团队，有专门的测试角色、编辑角色，甚至产品经理；这些方法及其变体（如非常针对 GitHub 问题的 Agentless）在任务总是匹配那种结构时取得了成功，但缺乏适应性，而且模块之间传递信息很难——如果你在一个单一智能体里做，并且妥善处理了所有过去动作的可用上下文，那些信息本来都在。

7.3 反馈：可供性、反思与交互

可供性（Slides 第 22 页）。SayCan（Ahn 等人，CoRL 2022）把 LLM 规划者与一个技能可供性模型结合：为每个机器人技能学一个价值函数，预测完成它的概率，然后把这个概率与 LLM 给出该计划的概率相乘。讲者举的例子是：模型可能提议“拿起苹果”，但如果视野里没有苹果，那就是一个糟糕的子任务；价值函数会给“找到它们”更高概率，给“放下它”更低概率，因为你还没拿到。

在环境反馈上重新规划（Slides 第 23 页）。LLM-Planner（Song 等人，ICCV 2023）、Inner Monologue（Huang 等人，CoRL 2022）、SwiftSage（Lin 等人，NeurIPS 2023）被列为例子。讲者用 ALFRED 具身环境解释：任务是“煮一个土豆并把它放进回收箱”；模型产生初始计划并调用原语，但计划在上下文中可能无法执行，于是它得到反馈“我找不到土豆，但我看到了一个冰箱”，把反馈输入语言模型重新预测计划，计划更新为“打开冰箱，然后希望里面有土豆”。

思考 vs 行动（Slides 第 24 页）。Test-Time Interaction（Shen 等人，NeurIPS 2025）在三种推理方法之间匹配计算量：产生更长的思维链、采样多个候选、或与环境更多交互。讲者说，在给定 token 花费下，用那些 token 与环境更多交互通常表现更好——只要你不是在做会让你进入糟糕状态的事情。

过度思考（Slides 第 25 页）。Cuadron 等人 2025 识别了三类坏行为：分析瘫痪、流氓动作链、过早脱离。更多过度思考在各模型类型上都预测更少的问题解决。讲者补充说，这些行为只需要知道花了多少 token 在推理上，可以从很多 API 模型获取；更大或更强的模型可以缓解其中一些问题。

思考 vs 观察（Slides 第 26–27 页）。Calibrate-then-Act（Ding, Tomlin, Durrett, 2026）的例子很精巧：每个文件有一个隐藏格式，三个部分——分隔符（, ; \t）、引号字符（" '）、跳过的表头行数（0 1），十二种组合，只有三个都对文件才能解析，而文件名是唯一线索：sales_fr.tsv 指向 tab。幻灯片给出的提示词增加了“格式似然”，模型读出“分隔符最可能是 ;，概率约 0.85……但我不确定，所以也许我应该跑些单元测试确认”。逐字稿中讲者用 CSV 的 price 列做类比：大多数价格 CSV 有 price 列，但不是所有都有。

7.4 长时程：错误会相乘，也会自我条件化

幻灯片第 28–30 页集中处理长时程失败。Vending-Bench（Backlund & Petersson, 2025）的任务是长时间经营一台自动售货机：定价、订货、补货；时间跨度最多 2,000 条消息、约 25M token、最多 222 个模拟日。幻灯片给了一个失败链：过早补货——“今天到期”不等于“已经送达”；环境正确报告“商品不可用”，智能体却推断“生意失败了”，从不等待履约邮件或再检查；错误持续，每日费用变成“欺诈”，邮件标题成了“紧急：业务终止后的未授权费用”。讲者的总结是：后来的动作会条件化在智能体早先的错误上。

幻灯片第 29–30 页引用 Sinha 等人 2025：更好的步骤准确率能延长时程，而自我条件化会让后续步骤更不准确；更大的模型执行更长的任务，但也更多地自我条件化。讲者用概率相乘解释：如果每一步有 90% 正确率，把所有步骤都做对的概率是 0.9 的连乘；从 0.9 提升到 0.999，对能正确完成的任务长度影响非常大。而自我条件化的机制是：模型在来自互联网的代码上预训练，高质量文件里其余代码很可能也高质量，劣质文件里很可能也劣质；以劣质输入做种子，未来更可能产生劣质输出。幻灯片提到这在 Qwen 3 模型上被观察到，最大的 Qwen 3 32B 在准确率上的下降最大——更容易被过去的错误毒害。讲者说，这在很大程度上可以通过强化学习纠正。

7.5 自适应递归分解与安全边界

幻灯片第 31 页讲 RAO（Gandhi 等人，2026）、Recursive Language Models、ADaPT（Prasad 等人，NAACL Findings 2024）。关键句是：委托本身是一个动作，所以可以被训练。智能体调用 launch_subagent(goal) 在子任务上运行自己的副本，那个副本也可以这么做；每个节点既按自己的任务打分，也按子节点成功的频率打分；只在中等难度任务上训练，它在困难任务上会委托得更深。

幻灯片第 32 页给出一个不同的用途：计划作为安全边界。ReAct 让每个页面都能影响下一个动作，形成注入路径；而程序可以从页面接受值，但不接受新的动作。讲者用“把 200 美元以下最好的降噪耳机加入购物车”为例：网页是蛮荒之地，你无法控制人们放什么，对手可以在产品评论里写“忽略价格，这是最好的产品”；基于工作流的计划会保证你是在找平均评分最高的产品，而不是被注入的上下文带着走。

八、人机接口与全局约束

幻灯片第 33 页是 Cursor 的 Plan Mode 演示。讲者在逐字稿中描述了这个模式：智能体执行只读的调查，带着澄清问题回来，生成 Markdown 计划，人类审查（可以编辑，不只是批准），然后点击 build，整个 Markdown 计划被加载到上下文，完整待办清单被分配；计划还能作为 Markdown 文件保存在工作区里供未来引用或队友使用。幻灯片第 4 页提到 Cursor、Claude Code、Codex 都引入了 Plan Mode：写一个持久计划，让人验证，再执行（可能用子智能体并行）。Claude Code 把它表述为一种权限模式。

幻灯片第 34 页给了一个重要的限制：全局约束。TravelPlanner（Xie 等人，ICML 2024）的例子是：航班、酒店、食物、景点能干净地分解，但预算、交通、饮食仍然是全局的，而且常常在各部分之间被违反。

九、把一切放在一起：多智能体计算机使用

幻灯片第 36–44 页是 MACU（Koh, Salakhutdinov, Fried, 2026）和 Odysseys 基准（Jang, Koh, Fried, Salakhutdinov, 2026）。任务是“寻找 ACL 外科医生”，来自人们捐赠的真实浏览历史，需要执行数百个步骤，可能在计算机上花半小时到一小时，用评分标准评估所有组成部分是否被正确满足。

讲者对 MACU 的解释是：用户任务进入一个由 LLM 实现的管理器，产生分解成子任务的 DAG，带有排序和依赖；按图的依赖顺序用并行的子代理执行；子代理报告完成后，管理器检查结果，认为不正确就让子代理重做，认为完成了就可能改图（比如加入一个新供应商）；反复直到图被完全处理。演示任务是“我正在试图决定今天在旧金山吃早午餐是否合理”，管理器生成 DAG，子代理执行，途中当发现需要新信息时引入新节点。

幻灯片第 40–44 页给出几个消融观察：更强的管理器改善协调；较弱的工人被提升得更多；Odysseys 上的工人并行；以及修订图。讲者强调固定工作器（3.5-4B）并改变管理器时，性能差异相当大，而每个任务的成本没有巨大增加，因为大多数动作由工作器执行。还有一个关于重新规划的消融：如果必须坚持最初提出的图，成功率约 25%；允许一次更改会上升一点，允许更多更改、更多重新规划步骤则上升更多。但幻灯片没有给出这些图的具体数值，图示需要对照官方 PDF 核验。

十、一条设计清单和一组开放问题

幻灯片第 46 页给出一条设计清单，在添加规划结构之前问：

子任务可分吗？规划者和执行者是否会受益于不同的模型或训练？
哪些失败或观察应该改变计划？
系统如何阻止错误和上下文累积？
什么信息和权限保持分离——谁批准不可逆动作？

幻灯片第 47 页给出四个开放问题：检查——语言计划和子目标图没有通用验证器；校准——智能体误判何时该思考、观察、询问或修改；全局状态——跨子任务约束抵抗分解和更长上下文；脚手架——训练出的推理是否会吸收这些结构，仍然开放。

十一、我的理解：本讲真正在教什么

以下为学习者解释，不是讲者原话。

如果把这一讲读成“规划方法的目录”，会漏掉它真正的论点。它在教一个设计判断：规划不是默认更好的，它是用承诺换可理解性、用灵活性换可靠性的一笔交易。

四类压力（模块化、反馈、长时程、控制）不是四个可以同时最大化的目标，而是四个会互相拉扯的旋钮。模块化让你能独立优化每个部件，但固定角色会在任务到达前就锁死，让验证者无法定位故障；重新规划让智能体能适应，但增加了成本、复杂性和不可预测性；限制子代理的上下文能提高效率，但全局约束会在各部分之间被违反。

最有说服力的证据来自长时程那一节：错误会相乘（步骤准确率连乘），而且会自我条件化（过去的糟糕上下文让未来更可能糟糕）。这两点合起来解释了为什么“平均每步都还行”的智能体在长任务上会崩溃，也解释了为什么规划结构——无论是显式的 DAG、单独训练的执行者，还是一个人的批准点——最终是在为“在正确的位置检查、在正确的位置恢复”付费。

待核：逐字稿为自动字幕清洗稿的机器翻译，其中若干模型名称、论文标题和数字（如 Vending-Bench 的 2,000 条消息 / 约 25M token / 222 天，MACU 的 25% 成功率，Qwen 3 32B 的表述）需要对照官方 Slides 与视频核验；本文已尽量按幻灯片文本转述，但图示和表格数值不应被当作已确认的精确数字。

官方来源

课程主页：https://www.cmu-agents.com/
课程日程与阅读：https://www.cmu-agents.com/#schedule
作业与项目要求：https://www.cmu-agents.com/#assignments
Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
课程视频（Lecture 5）：https://www.youtube.com/watch?v=S8v-dR4s29M&list=PLSN0qpDfUvTM&index=5
Lecture 5 slides：https://www.cmu-agents.com/slides/lecture-05-planning.pdf

CMU 11-768 Fall 2026 · Lecture 5 · Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination｜完整中文逐字稿

视频：官方视频
来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。

正文

[00:03.110] 好的。是的，我们开始吧。那么，今天的内容将是关于规划和任务分解的。呃，规划是一个挺有意思的话题，因为它有点像是一个不断变化的目标。嗯，它是关于——好吧，我们到时候会看到。它是关于智能体如何产生一种它们计划在未来做什么的表示。嗯，它建立在一些来自思维链的想法之上，比如说。嗯，但在智能体场景中，它更加有趣和具有挑战性，因为智能体会从环境中揭示一些信息，而这些信息会改变它应该做什么。所以计划应该始终是可修改的，而且有一个问题，比如说你知道，智能体应该在多大程度上承诺于某一个行动方案，这可能有助于让事情对一个人来说是可理解和可监督的，相对于智能体应该有多灵活、能够应对沿途出现的事情，这两者之间如何权衡。我们会看到设计空间中在这些之间权衡的不同点。这会稍微涉及一点多智能体，因为你知道，如果你有一个计划，并且你试图并行地执行其中的某些部分，那自然就是一个多智能体系统。但我们会在后面的一讲中更多地深入多智能体以及智能体之间相互通信的方式。不过作为一个激励性的例子，让我们想一想一个你现在就可以让智能体去执行的任务。你可以想一想你是否会信任这个智能体去完成这个任务或完成所有这些任务。但也许你想要——嗯，也许你想要智能体为你正在运营的一个研究实验室做一些关于GPU的研究，你需要比较来自不同供应商的不同配置，但你也有一些约束。你需要检查它们是否与你现有的集群兼容，对此有一些文档，然后如果

[01:52.240] 一切顺利的话，智能体实际上应该购买这些GPU。所以，一个使用标准React循环来执行这个任务的智能体，可能需要大约数百步，对吧？嗯，它需要去第一个供应商那里，嗯，浏览他们的网站，呃，然后，呃，你知道，找到某个特定的配置，嗯，获取一些细节，然后回到一个有约束条件的wiki，将可用的选项与之比较。然后你需要对若干个其他供应商重复这个过程。而且你知道，你怎么确定要查看多少个供应商呢？嗯，你会不会临时改变这个数量？在某个时候，你可能需要压缩上下文，对吧？我们之前讨论过压缩，但你可以把它看作是一种产生一个产物的方式，这个产物总结了你过去做过的事情。规划会有点像那个。它会是模型可以参照的一个产物。但计划还需要指定未来应该做什么，嗯，这可以帮助你知道，为智能体的行动提供信息。所以在大概数百步之后的某个时候，也许智能体已经搜索了四个不同的供应商，但还没有找到非常合适的。只有一个供应商满足规格要求，但GPU需要20周才能到货。这是我们几周前遇到的一个真实例子。嗯，那么现在你怎么办？智能体是

[03:24.800] 直接买那个吗？它会问人吗？它会寻找更多的供应商吗？嗯，所以计划，智能体需要采取的行动可能需要根据它在环境中遇到的情况而改变。所以这里有几点需要注意。呃，你不必线性地执行所有这些，对吧？有四个不同的供应商，嗯，你可以对每一个进行独立的调查，而它选择的这四个供应商网站中没有一个特别合适。所以你需要改变，嗯，并对你从环境中得到的反馈做出反应。而且在某个时候，可能也是一个向人征求反馈的好机会。嗯，我是否应该继续做这个不太理想的事情？我应该做别的吗？如果是的话，正确的做法是什么？所以，计划和任务分解会给我们一种通用的方式来解决其中一些挑战，嗯，但会有一些权衡，我们到时候会看到。所以，思考这个特定例子的一种方式是，嗯，智能体需要执行的任务有一些底层结构，你可以把它看作是一种依赖图。我们会在讲座末尾讨论的一篇论文中看到这一点。但供应商A、B、C和D，嗯，对每个网站的搜索可以彼此独立地进行。如果你只有任务的一些规格说明，你可以并行地执行每一个，然后你可以把从每一个获得的信息拿来与你所拥有的规格列表进行核对。所以这个兼容性检查依赖于来自供应商的信息，但每个供应商之间嗯并不相互依赖。

[05:15.039] 然后在你完成这个兼容性检查之后，理想情况下你会想要做一个采购申请，对吧？也许你开始这个任务时想，我就看那四个供应商，做兼容性检查，然后做采购申请。但如果它们实际上都不匹配，那么也许我们需要添加一些新的子任务，我们可以把它们表示为比如说一个图中的新节点，嗯，基于我们沿途从环境中得到的一些反馈。所以这叫做重新规划，就像嗯，你有这个计划，这个你想要做什么的结构化表示，然后你根据来自环境的观察对它进行修改。

[05:57.029] 所以是的，我们今天要问的一些问题是，规划，就像让智能体承诺它打算在未来做什么，这有用吗？嗯，什么时候它可能有用？什么时候它可能不值得额外的成本或它对智能体施加的额外的僵化性？表示计划有哪些方式？只是文本吗？是像这样的结构化图吗？嗯，代码也是一种你可以用来表示计划的方式。而且如果你做了一个计划，你应该什么时候改变它？比如在这个例子中你应该添加另一个供应商来搜索吗？还有，一个人应该如何以及何时介入？所以嗯，你们中有多少人会愿意让智能体实际上为你购买GPU？呃，是的，你可能会要求输入从这个人那里。嗯，你可能会在这件事接近尾声时、在你收集到更多信息之后才询问。但对于风险较低的任务，也许你可以接受由这个人、你可以接受智能体嗯只是执行一个计划并在你只批准了初始计划的情况下进行调整，或者也许你甚至根本不需要批准这个计划。

[07:04.720] 那么，有多少人在编码智能体中使用过嗯规划模式？这些变得——好的。是的。所以，有不少人——这些在大约一年前变得相当流行，我想。嗯，它们存在于嗯它们存在于 Cursor 中。我想 Cursor 也许是最早的之一。而且它也相当有趣并且功能齐全。我们稍后会在讲座中看到一个演示。Claude Code 和 Codex 也有。但它们之间共同的想法是，你用嗯一个复杂任务来提示智能体。它会用自然语言描述它将要执行的步骤。嗯有时是在执行了一些初步的、非破坏性的嗯操作之后，比如探索你的代码库或采取其他信息收集操作，这些操作实际上不会改变任何东西，只是获取那些有助于让你判断这个计划是否好的信息。嗯所以由人类来验证它，或者也许对它做一些编辑，嗯通过直接编辑计划，或者通过给出一些你知道的反馈消息来让智能体更新计划。然后在计划被确认之后，你会做一些类似点击构建的操作，然后智能体就会执行它。这也是一个将计划执行并行化的机会，嗯比如使用子智能体来执行其中的各个单独项目。我们稍后会看到一些研究论文，展示实现这一点的方法。

[08:32.560] 那么，为什么这可能会有帮助呢？有人能提出一些你可能会想让智能体进行规划的理由吗？有人举手吗？>> 是的。当然。是的，对于很难信任智能体的任务，计划可以帮助嗯建立这种信任，嗯帮助你确信它很可能会做正确的事。 >> 是的。是的。所以，这就像是在 Claude Code 中，这实际上被表述为一种权限模式，对吧？就像嗯，以同样的方式，智能体要么执行所有操作，要么对每个操作都询问。有一种叫做计划模式，你执行信息收集操作，然后之后你可以更宽松地行动。 >> 是的，>> 也许计划出来嗯比嗯有更低的全局成本？>> 是的，先规划再执行可能比贪心地执行有更低的总成本。是的，那是真的。而这是规划的一个很大的动机。嗯有几种方式可以实现这一点。比如一种可能是，这些任务更简单，这些，你知道，计划中的项目更简单，所以你可以嗯让一个更弱的智能体来执行它们。嗯但你需要一个更强的智能体，运行成本更高，来实际生成一个好的计划。嗯如果能让智能体并行化，你知道，子智能体调用，也可能更快。所以它可能嗯即使你对所有事情都使用嗯相同规模的智能体，嗯也许你并没有花费更少的计算量，但也许它确实嗯在时间上更高效，而且计划也可能嗯避免让智能体犯错，使智能体更容易不犯那些会让它嗯陷入你知道的循环并导致更多操作的错误。嗯是的 >> 有时候就像当你有课程时你

[10:37.279] 没有非常结构化的——怎么解释呢——所以你可能只是学习所有嗯你有的那些想法，你可能会把它结构化并规划出来，既规划你的智能体也规划你自己 >> 对，是的，所以计划可以有点像一个嗯草稿纸，用来嗯思考如何解决问题，由人和智能体共同分享。是的，没错。酷。是的，也许再来一个。请讲。>> 一个非常长的任务，需要被分成多个子任务。所以就像把它明显地拆分成多个任务，让它自己结构化地呈现出来。>> 是的，没错。所以拥有一个结构化的产物，记录了什么已经完成、什么未来还需要完成，可以帮助智能体跟踪嗯它在任务中的进展。是的。所以这也就像，你知道，那个动机有点类似于我们讨论过的关于上下文管理的那些。嗯是的，但它是以一种人类可解释、人类可编辑的方式来做这件事。是的。很好。这些都是关于规划的很好的嗯论据，而且嗯是的，我想我们——我想我们已经涵盖了所有那些。

[11:55.200] 酷。所以是的，我们已经谈到了这一点，但为了本次讲座的目的，我们对计划的工作定义是：对智能体未来可能做什么的某种表示。所以这些动作是它将尝试的子目标，以及它们之间的任何排序或依赖关系，就像你知道我们在那个图中看到的依赖边，或者只是某种像线性的步骤排序。所以嗯技能，我们上次讨论过的，是关于适用于跨任务的可复用指导，嗯是不同任务中常见的东西，而计划只是针对这个特定任务。嗯并且不那么涉及通用性。嗯所以计划可以比技能更具体，并且可以嗯呃与特定的仓库相关联。嗯例如，所以是的，就像我们说的，计划是嗯对未来工作的一种组织，而且它只是一个提议，对吧？嗯智能体可能需要根据它在环境中发现的一些情况偏离那个计划。所以它可能需要重新规划，但嗯重新规划的能力也会让智能体的行为变得不那么可预测，或者说更难以被人明确指定。所以这将是一个我们稍后会嗯稍微处理的权衡。是的。而且我喜欢这句话，每个人都有一个计划，直到他们嘴上挨了一拳，这出自拳击手迈克·泰森。嗯所以有一个计划是好的，但也能即兴调整你的计划也是好的。

[13:34.639] 所以我们会在本课程中嗯讨论规划几次。所以主要是今天，但在编码智能体讲座中，我想 Graham 会讲关于 agent lists 的内容，那是一种嗯结构化的嗯呃计划，智能体在嗯解决嗯代码时会坚持遵循它，嗯比如在实现 pull request 时、在解决编码问题时。我们也会稍微谈谈多智能体。好吧，我们会在其中一次交互讲座中大量讨论多智能体，它与规划有联系。嗯我们会在第二次交互讲座中更多地讨论人类如何嗯通过计划向智能体提供监督，计划是其中一种机制，而且在搜索讲座中我们也会嗯讨论嗯呃搜索应用于多智能体系统

[14:23.199] 嗯，包括MACU和我们今天要讨论的row系统。是的。在我们开始之前有什么问题吗？好的。那么，规划有一段历史，这段历史的第一部分是最近的部分。它来自大语言模型和思维链。所以，我认为大家应该都见过思维链，对吧？就像嗯，人们熟悉推理模型。很快地。是的，我们会非常非常快速地回顾一下这些。嗯，但思维链的代表性工作是这两篇论文，它们提出了提示词，嗯，甚至在模型被训练来显式产生思维链之前。它们是在，你知道，嗯，在模型被训练来产生思维链之前，模型可以以此为条件来产生答案，这就像R1或01这样的推理模型，嗯，人们只是通过提示让模型产生思维链。嗯，他们发现提示模型产生思维链，嗯，使它们更擅长产生答案。这里有一些人们探索过的提示词。所以这篇Kojima等人的论文中，他们有一个挺有趣的表格，他们列出了使用的提示词，然后列出了模型在以那种方式提示时的准确率，他们发现最成功的是“让我们一步一步地思考”。嗯，所以这在一段时间内变得很有名，就像“让我们一步一步地思考”就是你所需要的全部。嗯，然后你知道，后来嗯模型被训练来产生思维链以一种对提示词更鲁棒的方式。嗯，但这些工作的基本思想是，思维链将为模型提供一种使用额外计算的方式。所以它接收这些token，嗯，在它产生的思维链中，嗯，这允许它在预测答案之前做额外的操作。它也提供了一个草稿纸，让

[16:34.320] 模型可以回过头来参考。所以对于复杂问题，写下你得到的一些中间结果，把解决复杂问题的过程外化出来是很有用的。这与我们对智能体规划的动机是相同的。嗯，所以在这些工作中，模型在一次生成过程中完成思维链的生成和解答模型一次生成一个思维链的token，然后产生答案。嗯，我们也会看到一些智能体规划方法看起来像这样。但你也可以想象，比如显式地，你知道，修改思维链或检查思维链，然后使用那个受控的思维链作为答案的条件上下文。对于智能体规划也有类似的类比。嗯，所以我认为目前对于智能体来说，我们大概处于这里，就像在这些之后，你知道，我们大概处于这两种方法之后，开始做的工作是训练一个智能体系统来产生一个对答案也有用的计划。嗯，但那在推理模型中已经被更彻底地探索过了。嗯，像star和deepseek是最早做这件事的论文之一。我们将在本讲座接近结尾时看到一些论文，比如Ralph，它们训练模型产生一个对产生可验证答案也有用的计划。

[18:08.950] 所以，在思维链的基础上，人们开发了将复杂推理问题分解为子任务的方法，并表明这可以提高性能。所以做这件事的几篇论文，一篇叫最少到最多提示，来自iClar，几年前。还有一篇更近期的，我也认为属于这类，是递归语言模型论文。嗯，但这里的想法是，你将，你将调用你的推理模型一次，你将调用你的LLM一次，来把一个复杂问题分解成部分。所以像这篇论文，最少到最多论文是在数学问题上验证的，嗯，他们让语言模型拿一个更复杂的问题，然后从中预测一些子问题，想法是这些子问题对语言模型来说比初始问题更容易解决。所以模型可以，你可以然后对更简单的问题进行连续的LLM调用，嗯，依次解决每一个，其中一些调用也可以依赖于更早问题的答案。所以比如我们可能首先开始解决每趟旅行需要多长时间，我们会得到那个的解。每趟旅行需要五分钟，然后你会继续到语言模型生成的下一个问题，作为那个问题的输入，你还会喂入，嗯，你知道，完整的复杂问题和之前问题的答案。对于当时的模型来说，这真的很有帮助，因为模型不太擅长一次回答涉及多个推理步骤的问题。嗯，而且与完整问题相比，回答子问题也不需要那么多上下文。对吧？所以也许要回答这个，你只需要初始问题的一些部分，嗯，以及你所依赖的一些任务的答案。

[20:16.480] 所以这些动机对于我们现在的智能体来说是一样的。就像一个智能体解决一个子任务可能更容易，因为它不需要完整任务所需要的所有上下文。比如嗯，如果你在搜索某一个特定供应商的GPU，你不需要对其他供应商的信息进行推理。你

[20:39.919] 可以把它带回来，然后再聚合。嗯，将复杂任务分解为子部分的另一个好处是，你可以使用不同的模型来解决一些子部分。有人能想到这样做的动机吗？为什么使用不同的模型来解决不同类型的问题可能是有用的？没错。是的，你可以找到或开发一个对那个特定类型有点专门化的模型。嗯，我们会看到一些例子。嗯，另一个动机是我们之前提到的，嗯，它可能更高效。嗯，就像嗯，你可能不需要一个运行成本那么高的模型来解决一个更简单的问题。所以做这件事的一篇论文叫分解提示。想法与我们刚刚看到的类似，你拿一个复杂问题，你让一个分解模型从中生成子问题，每个问题也预测了一个类型。所以他们专注于问答。嗯，所以他们有不同类型的问答任务，比如一个是通用QA，嗯比如那些出生于 910 年、1910 年的人，嗯，以及其他人，涉及一些关于人物及其工作的关系信息，嗯，比如这部电影是哪部——这里会回溯到之前问题的答案。嗯，对于哪些电影，论点一是制片人，那会被预测为这种特定类型的问答问题。所以在这项工作中，我认为他们使用了不同的提示词来让语言模型实现这些不同类型的问答模型，其中模型有一些少样本示例，这使它更擅长通用问答，或者使它更擅长这种位置性问答。但你也可以想象其中一些模型能够访问专门的知识，或者

[22:56.400] 在语言模型的语境下，能够访问其他模型没有的专门工具。这篇论文还有一种简单形式的程序化控制，你基本上指定了未来要执行的子任务的顺序，并且允许将某个子任务的答案中的一些信息传递到下一个子任务。比如，嗯，这会指——这会允许

[23:25.120] 将前一个子任务的答案作为一个参数在这里使用。而且他们为此有一种相当结构化的表示。但就像在技能那一讲中一样，我们从某种模板走向了代码。你也可以想象从一种结构化的领域特定语言，走向直接使用代码来控制这种子任务的编排，我们稍后会看到这一点，

[23:48.080] 当我们讲到 route 的时候。好的。所以这是来自 OLM 思维链方面的一些简要动机，但我也想谈谈当你转向智能体设置时，什么会失效，那些设置实际上需要在世界中行动。所以思维链、推理步骤，它只是在生成文本，对吧？嗯，它不像智能体那样在环境中做出改变。而且它通常也不会揭示更多信息，对吧？思维链中生成的一切基本上都来自语言模型。嗯，我们没有使用工具调用。嗯，模型可以通过说类似“算了”之类的话来撤销推理步骤，然后说出你知道的那个更改，理想情况下它之后就能够以此作为条件。所以如果我们真的在世界中行动，我们会有很多挑战。嗯，所以智能体将采取会揭示信息的行动，比如查明供应商有哪些 GPU 可用。行动可能会失败。所以你可能有一个非常好的计划，但后来发现，你知道，你需要用来执行它的东西根本不可用。嗯，

[25:03.440] 而且你还可能做出不可逆地改变环境状态的行动，比如删除你所有的文件。嗯，这些是推理模型不存在的挑战。所以 AI 中有一个关于规划和计划的悠久研究传统，经典 AI 中的计划与我们用来描述 LLM 智能体的计划方式有一些重要区别。但经典规划的核心是，你有一个结构化的、基于逻辑的世界表示。你有状态，状态是一组谓词。比如，你知道，那些可以对特定参数为真的函数。所以这来自比如机器人领域，其中有对象，这些对象实例化了这些 X 和 Y 参数，而你有谓词这些谓词关于这些参数为真或为假，比如一个对象可以在桌子上。一个对象可以在另一个对象 Y 上，等等。而且你还有你的 AI 系统可以在世界中采取的行动，这些行动会更新状态。所以这些基本上也是应用于两个参数的函数，为了能够表示状态的变化，你会定义每个行动函数的前置条件和效果。所以前置条件基本上就是，什么时候——你什么时候被允许应用那个行动。比如，如果对象在桌子上，且对象上面没有任何东西，且你的机器人手是空的，那么你就可以拿起一个对象，然后在你应用那个行动之后，你就有了一个新状态，它是谓词的合取。所以你正握着那个对象，对象不在桌子上，且机器人手不是空的。这是一个叫做 strips 的语言，它曾被非常广泛地用于规划。嗯，PDDDL 是另一个，人们实际上把它用作 LLM 的评估目标。嗯，如果你感兴趣可以去看看那篇论文。嗯，人们正在思考

[27:19.919] 如何将两者的优势结合起来。但一个规划问题是，你有一个初始状态，你有一个目标，目标也是一个状态。你的问题是如何找到一组有效的行动序列，把你从初始状态带到目标？而一个行动序列是有效的，如果每个行动的前置条件都被你当前所处的状态满足。所以这里有一个示例计划。比如如果你有蓝色和橙色方块，蓝色方块在橙色上面，且红色、蓝色和黄色上面都没有任何东西，你可以有这样一组行动序列，希望我检查得足够仔细。你可以看一看，这是一个有效的行动序列，而且它确实能把你带到目标。所以规划中的很多工作过去是、现在仍然是关于搜索的，比如你开发搜索算法，像 BFS、DFS、A*，配上不同的启发式方法，在给定这种非常结构化的表示的情况下，找出一个从起始状态到结束状态的计划。

[28:29.520] 所以这就是经典规划，它与用 LLM 做规划有很多不同，但也有一些重要的相似之处。对于 LLM 来说，这里的一切都是显式的，对吧，你确切知道一个行动何时适用，你确切知道这个行动做什么，很多时候你也对世界了解很多。嗯，也有一些方法可以在这些方法中表示不确定性，但对于 LLM 来说，一切都是隐式的，对吧，嗯，对于一个 LLM，你不知道一个行动究竟意味着什么，比如你什么时候可以应用这个行动，其中大部分存在于智能体外部，你只能像在环境中尝试各种东西，看看它是否有效。但对于一个 LLM 智能体来说，你不必做大量搜索就能写出一个看似合理的计划，因为，你知道，LLM 已经见过大量关于我们正在操作的所有这些不同情境的信息。所以，它们可以只是生成一些文本，那大概率是关于如何把一个方块移到另一个方块，或者像如何调试代码库中的问题这类更复杂任务的合理初步猜测。所以对 LLM 来说，规划不太在于做搜索，尽管我们稍后会在课程末尾想办法把搜索引入来改进 LM 智能体，以及更多关于就是检查你的计划是否真的可行和可执行的内容。嗯，但重要的概念在某种程度上仍然是搜索，还有动作适用性。所以我们没有像经典规划中那样做前置条件检查的方法，但我们会看到一些方法

[30:03.360] 嗯，试图把关于你在环境中能做什么的信息融入到选择计划的方式中。所以现在可能是一个很好的时机来开始思考，如果我们不做经典规划，嗯，那我们的计划实际上看起来是什么样的？而像我们在 STRIPS 中看到的那种形式化计划，它们是可检查的。语言计划，就像你知道的，我们在讲座开头在计划模式中看到的那些，是相当通用的。嗯，程序也是一种表示计划的方式，它介于两者之间。而且有一系列论文，我想 2023 年左右是这些工作的开端，它们使用代码语言模型来基本上写出一个代码结构的计划，嗯，它控制系统，嗯，它被应用于嗯一些领域，包括机器人学，还有其他领域，包括那种结构化问答，这篇论文 Binder，我们也在这里做了一些工作，嗯，用于那种半结构化问答。但这里的基本思想是，你有一个任务进入你的 LLM，比如把积木堆到空碗上，然后 LLM 会被提示生成一个程序，嗯，它调用嗯领域特定的 API 或函数，而那些函数将使用嗯某种类似于嗯机器人系统里的机器人原语来实现，比如嗯对图像调用一个物体检测器来看看有没有积木，或者有没有碗，嗯，然后实际上也嗯调用物体操作器。所以为了让这个工作，你需要有嗯好的 API 和好的原语。嗯，但你也需要让你想让系统执行的任务能够被代码很好地表示。而代码有一些不错的特性，比如你可以说，嗯，我还没把积木放到另一个积木上，重试做这件事，这在某种程度上是一种嗯实现类似搜索和后置条件的方式。

[32:18.480] 嗯，但这也像我们在技能讲座中谈到的，有点更僵硬。是的。而让它更灵活的一种方式是，你这里的 API 函数也可以是调用 LLM，例如，或者另一种神经网络模型，比如嗯你知道一个基于 ResNet 的物体检测器。所以你得到一些好处，比如嗯拥有结构化的、程序化的代码控制，但同时也嗯你可以使用嗯定制的模型，它们可以是机器学习的，更灵活。关于这个有什么问题吗？是的，这些方法已经开始，至少在机器人领域。我的理解是，这些方法并没有像嗯你知道端到端的、嗯基于多模态 LLM 的 VLA 嗯类型系统那样流行起来，那些系统可以

[33:11.039] 直接训练。嗯，但这些类型的方法仍然非常有用，例如对于编码智能体。嗯，我们会在下一讲中通过 CodeAct 嗯看到一些相关内容。所以另一个要思考的维度是，根据你从环境中得到的反馈来改变计划，嗯，这样做是否有用。所以 Anthropic 几年前发了一篇博客文章叫《构建有效的智能体》，嗯，他们在其中基本上是在建议你不要从智能体开始，而是从更简单、更容易推理、你可以提前指定的东西开始，他们称之为嗯工作流。所以在一个工作流中，你有一个嗯固定的程序化方式来调用语言模型，嗯，它指定了很多你关于任务所拥有的信息。所以嗯比如说这个 GPU 的例子，假设我们要非常频繁地买 GPU，嗯，因为我们很有钱，而且我们总是想遵循同样的流程来做这件事。所以我们有一个固定的供应商列表，我们要去他们的网站做研究。我们会读那些规格。我们会检查它们是否合适然后我们会选最便宜的那个并买下来。嗯，这实际上有点类似于我认为我们会在下一讲中讨论的 agentless 框架。嗯，所以这很好，因为你基本上嗯知道智能体将要执行的任务的高层结构。它有一些灵活性来适应，比如如果供应商网站变了，它希望能从中找到正确的信息，嗯，而且你知道它会嗯选最便宜的那个，所以这很好，嗯，但它也相当僵硬，对吧，比如你知道如果嗯这些供应商不再是做研究的最佳选择呢，所以放松其中一些约束也可能是有用的，并且

[35:08.240] 例如让智能体预测嗯应该被研究、搜索的供应商列表，嗯，也许你做一个搜索来产生那个列表。嗯，或者你完全自适应，根本没有这种结构，只是给智能体某种嗯你想要它做的事情的描述。

[35:27.119] 你放开它，没有多少规划发生，嗯，以一种由人提前指定的方式。所以，这里有一个权衡。比如提前承诺一个计划。嗯，它可能更快、更便宜、更可预测，嗯，因为你可以也许用更小、更简单的模型来实现每一个部分。嗯，但如果你完全不改变这个，你就失去了适应环境变化的能力。而且嗯我们稍后会看到重新规划嗯有时可能值得额外的成本、复杂性和它给你的系统带来的不可预测性。关于这个或其他事情有什么问题吗？是的。所以如果你看看这篇嗯博客文章，他们会说我们已经弃用了这个，嗯，然后指向另一篇。但我认为他们这样做的部分原因是，我们现在的智能体更强了，嗯，用最新模型的最新智能体在很多时候更擅长做出合理的动作，嗯，不受工作流的约束。我想我们会在编码智能体上也看到这一点，嗯，这个 agentless 框架不再嗯是解决比如问题的最成功的框架。嗯，所以当你的智能体在某个特定领域变得更强时，其中一些脚手架可以被替换，但你也可能嗯失去可靠性。而且如果你在做一个新问题，你还没有嗯，你知道，智能体还没有

[37:06.960] 训练成能够对环境变化做出反应，从工作流开始可能是更稳妥的方式。有问题、想法吗？好的。那么，为规划添加一些结构的原因。我们会讨论其中一些，它们是由我们最初的例子所激发的。再次回到那个例子，我们有你知道的这种模块化，比如独立地执行每一个这些子任务，彼此独立。反馈，比如根据你在环境中观察到的东西潜在地改变计划。贯穿所有这些的一个非常普遍的特点是它们是长时程的，比如你需要跟踪信息，并且对你要在未来做什么有一些预测可能会很有帮助，然后还可能控制一个人可能在计划上介入的地方，或者沿途对计划的修改。这将构成讲座下一部分的结构，主要讨论研究文献中为处理这些中的每一个而开发的各种方法。

[38:25.430] 所以首先是模块化，我们之前稍微触及过这一点，我们说过拥有以不同方式专门化以执行任务不同部分的不同模型可能是有用的。这类方法在一段时间内确实被广泛用于计算机使用智能体，因为用于生成比如计划的那些最强模型并不擅长处理关于网站的某种低层次细粒度视觉细节。所以2025年有几篇论文通过分离，比如你知道在网络上解决任务，比如找最便宜的4K显示器，相当成功，它们会把这分解为一个规划步骤，在那里你实际上是一个规划步骤的序列，你有一个强的规划器模型，比如当时的GPT40，会接收网页并生成一个自然语言描述，说明应该是什么样的，比如你知道在页面顶部的搜索栏中输入，但这个模型会非常不擅长比如实际点击搜索栏，尤其是如果你需要比如点击特定的像素。所以他们做的是训练了一个小的，这是U Groundwork。他们在合成数据上训练了一个小的70亿参数模型，在那里他们你知道有网页，他们能够比如为页面的特定元素合成像这样的文本描述，一旦他们让那个模型变得非常准确，那么你就可以用它来执行由规划器模型生成的这些自然语言指令。对吧？所以分解并使系统模块化可以让你独立地优化每个组件，并在弄清楚瓶颈是什么之后对每一个进行爬山优化。这些论文是这方面很好的例子。在这项工作完成之后，训练这些大型语言模型的公司，比如GPT40的后继者，开始把大量比如网络数据，带有同样那种细粒度的你知道的东西，直接放入大模型的训练中。所以这类方法对于仅仅提升性能来说不再那么必要，但仍然可能

[40:45.599] 对提升效率有用。我们稍后会看到一些关于这方面的内容。是的。所以你可以训练你知道你可以训练一个执行计划部分的模型，就像他们在这里训练了grounding模型一样，但你也可以生成数据来训练一个规划器。你可以做到这一点的一种方式被这篇plan and act论文探索了，在那里你想要，所以他们在做的任务是，是的，他们在这里做的任务也涉及比如你知道在环境中的动作，与网页交互，他们想要有一个规划器模型能够生成一个看起来像这样的计划，比如第一步将是搜索这个特定的地方，第二步将是分析结果，然后把计划的执行交给一个模型，它看起来有点像我们在上一张幻灯片上看到的执行模型。所以他们需要，你并没有这些计划的某种自然产生的数据源。所以他们做的是他们会获取整个轨迹，执行任务，把轨迹过滤为他们预测为正确的那些，然后让一个教师LLM用基本上是对动作的一个分段以及针对每个动作序列的自然语言标签来标注那个轨迹。一旦你有了那个，你就可以比如解压它并为规划器创建你的训练数据，它从你知道任务的自然语言描述到自然语言步骤的序列。然后执行器也是，它从一个步骤的自然语言描述映射到你在上下文中采取的实际动作。这说得通吗？在某些方面，这有点类似于我们上一讲讨论的工作流归纳的东西，对吧？比如工作流归纳也是从一条轨迹开始，并对它做了某种分段和标注。

[42:52.800] 嗯，你在这里做类似的事情，但你在用它来为生成那些描述的模型生成训练数据。所以当你转向像这样的模块化系统时也有一些权衡。你在把一个任务分解为你知道一组子任务以及它们的排序。这是一个抽象，可能会遗漏一些细节并且不灵活。比如用于编码的某种分解已经有了一些有限的成功。有一系列论文比如说过我们会让模型实现某种软件公司或者像一个软件工程师团队，可能会有比如专门的测试角色、编辑角色。有些甚至有比如一个产品经理，那些方法已经，并且它们的一些变体比如像agentless，它有点非常针对性地用于解决GitHub问题，在任务总是匹配那种结构的情况下取得了成功，但它们也有一些弱点。嗯，正如我们之前讨论过的，它们不具备适应性，而且它们还需要比如从一个这些模块向另一个传递信息，可能很难比如获得另一个智能体解决其任务所需的正确信息。而如果你只是在一个单一智能体中做这件事，并且你很好地处理了过去动作的所有可用上下文，那么所有那些信息都会在那里供智能体使用。所以这些是一些权衡，在

[44:49.280] 这些方法要么只是让单个智能体尝试做所有事情，要么使用基于规划的分解。所以如果你有一系列想要在环境中执行的子任务，经典规划中缺少的一点是判断你是否真的能够完成该子任务。所以当LLM开始在NLP任务上变得非常成功时，人们开始看看它们是否可以被用来控制例如机器人系统，并取代过去使用的一些规划系统。所以你会给系统类似这样的指令，比如你如何把一个苹果放在桌子上，然后让LLM产生一个类似于计划的文本替代品，你可以从语言模型中采样很多不同的可能的下一步计划，在每一个下面你都会从语言模型得到一些分数。嗯，这些会是类似logits的和。嗯，其中一些在语言模型下可能相当合理，但语言模型实际上并不知道机器人系统在特定上下文中能做什么。所以它可能会提议比如拿起苹果，但如果智能体的视野范围内没有苹果，那实际上就是一个糟糕的子任务。所以这篇叫SEAN的论文，嗯，将用作规划器的LLM与一个可供性模型整合在一起，即可供性模型判断规划器产生的子任务是否真的能在环境中执行。他们使用了机器人学中的一些经典技术来开发那个模型，那个价值函数。嗯，但那个价值函数基本上能够预测在特定上下文中执行任何这些自然语言描述的子任务的概率。当你有了这个之后，比如你知道，如果你不在苹果或可乐罐附近，那么那些的概率就很低，而找到它们的概率就更高，对吧？然后，放置它的概率更低，因为你还没有拿到它。所以将这种关于你是否真的能执行子任务的价值函数与规划器结合起来的一个简单方法，就是直接将价值函数给你的概率与语言模型给你的概率相乘。

[46:59.440] 关于这个有什么问题吗？嗯。所以接下来我们希望智能体能够做的是整合来自环境的反馈，对吧？我们在讨论反思时看到了这方面的一个前身，对吧？就像你让一个智能体尝试一条轨迹，你看到它是成功还是失败，你得到一些自然语言反馈，你以此作为条件再试一次。而

[47:22.720] 这也可以，这在那些你无法从初始状态就看到所有东西的环境中特别适用。所以这个例子来自一篇论文，它在这个叫Alfred的具身环境中运作，你有一个机器人在模拟房屋内导航，你有像这样的任务煮一个土豆并把它放进回收箱。这些任务是合成生成的，所以有些挺搞笑的。嗯，但模型会产生一个初始计划，调用这些原语，这些原语可以执行诸如导航到土豆、拿起土豆、把它放进回收箱之类的任务，它会尝试执行那个计划，但嗯，那个计划在那个上下文中可能并不真的能被机器人执行。所以你可能会得到这样的反馈：我找不到土豆，但我确实看到了一個冰箱。你可以直接把那个反馈输入语言模型，然后尝试重新预测计划，此时计划会更新为比如打开冰箱，然后希望里面有土豆，我应该继续下去。所以这是一种简单的基于反思的重新规划方式，可以有所帮助。所以通常存在一种权衡，嗯，你知道，在不与环境交互的情况下进行推理步骤和环境中采取行动之间的权衡，这之间可以存在权衡，不同的方式在不同程度上可能有价值，嗯，但我觉得这篇论文挺有意思的，它展示了嗯，如果你有一个在环境中行动的智能体，嗯，你有关于如何花费计算资源的选择。所以你可以嗯，如果这是一个推理模型，你可以在采取某个特定行动之前产生更长的推理链，嗯，或者你可以直接采取更多行动，与环境更多地交互，

[49:40.800] 嗯，然后你可能会获得更多信息，但你也需要用你的语言模型处理那些信息，这也会是一种花费计算资源的方式。嗯，他们嗯，他们在他们研究的这些任务上比较了这两种方法，嗯，他们还比较了一种基线方法，嗯，我们会在搜索讲座中更多地介绍这个，但嗯，那个方法只是从环境中采样多条不同的轨迹，然后使用一个评判者来在它们之间进行选择。那就是这里的橙线。所以一种只是让模型思考更长时间、在采取每个行动之前产生更长思维链的方法，是这条蓝线。而一种与环境更多交互的方法是绿线。对于给定的 token数量嗯花费来说，通常这条绿线，即使用那些token与环境更多交互，表现更好。这说得通吗？描述对大家来说清楚吗？所以，我觉得这是一种很好的思考方式，嗯，就是关于交互的价值。嗯，从环境中获取更多信息，只要你不是在做那些

[50:58.240] 会让你进入糟糕状态的事情，嗯，通常是非常有帮助的。另一篇从另一个角度研究这个的论文，嗯，Graham实际上也参与了，它研究了嗯，编码模型过度思考的危险。所以他们嗯，识别了推理模型嗯所使用的或者产生思维轨迹的模型作为智能体基础所使用的不同种类的行为，那些模型可能产生的不同种类的坏行为，嗯，这些会影响它们在环境中采取的行动。所以，其中一种类型是模型花太多时间规划，而没有嗯采取足够多的行动。另一种是采取无用的行动，但对环境造成有害的变化。嗯，还有第三种，比如过早结束任务。所以这些嗯他们识别出的行为，嗯，实际上不需要访问推理轨迹，因为如果你知道，它们只需要访问花了多少token在推理上，你可以从很多API模型获取这个。嗯，但他们研究的是某种平衡你可以在环境中实际看到的动作，以及所花费的推理 token 数量。他们为这些不同类型的行为开发了一些分类器，并用它来生成一个分类器，为在代码库中行动的、由语言模型支持的给定 agent 给出某种过度思考分数。他们把这些画出来，比较在这个

[53:07.839] 解决 SWEBench 问题的基准上的表现与过度思考分数，并发现相当强的负相关。所以那些过度思考更多的模型，像是那种没有恰当行动、做了太多思考的模型，在这些基准上也表现更差。所以他们总体上发现，其中一些问题可以通过使用更大或更强的模型来解决。嗯，

[53:32.319] 所以你可以看到，比如你知道，推理努力较低的 o1 更差。推理努力较高的 o1 更好，过度思考更少，也做得更好。嗯，甚至非推理模型——你知道，它们在采取行动之前仍然会产生一些文本——也有这种类似的行为，尽管它不像推理模型那样糟糕。

[53:54.960] 所以嗯是的，从环境中获取反馈是有用的。嗯，而且你不想花太多时间做规划。嗯，你还需要按需重新规划。是的。所以嗯，这篇论文我们没时间深入细节，嗯，为了节省时间，但我认为这是一篇非常有趣的论文，如果这个想法让你觉得有意思，我鼓励你去看一看。但想法是，语言模型可以采取的某些动作，嗯，只是看起来像是合理的，嗯，你也许可以在不实际检查环境细节的情况下做一些合理的事情。所以比如说，你你知道，让一个编码 agent 去调查一个 CSV 文件并找到价格最高的条目。所以大多数与价格相关的 CSV 文件可能都有一个叫 price 的列，但并不是所有都这样。所以语言模型可以采取这样的动作，大多数时候它会是对的，但嗯有时候它会出错。而这篇论文大致是关于，嗯，我们如何明确地表示某个关于环境的事情为真的概率，比如你知道它是否具有 CSV 文件的这些特定属性，而如果我们明确地表示这些概率，它是否能让 agent 更有可能做正确的事情，他们发现它可以。是的。所以如果这听起来让你感兴趣，我会鼓励你去看看那篇论文。

[55:43.990] 是的。所以部分动机也是为了，让模型能够更好地执行嗯长时程任务。嗯，我认为长时程任务的一个典型例子是这个叫做 vending bench 的任务，它之所以有挑战性，是因为它的时间跨度很长，而不是因为它需要像复杂的推理，但想法是，比如长时间经营一个简单的生意，这个生意只是运营一台自动售货机。嗯，所以你有一些物品，它们有价格。如果库存变低，你需要订购新物品，并把它们放进自动售货机。但挑战在于，你必须这样做长达 222 天，这可能需要模型消耗大约 2500 万个 token。所以你需要能够比如随时间跟踪状态，并一遍又一遍地做出好的决策，嗯，而不被你过去做错的事情所毒害。嗯，随着模型被训练，嗯，用长上下文训练，并用 RL 训练以能够从错误中恢复，模型在这方面已经变得更好了。

[56:55.680] 但有一些分析，嗯，研究了嗯，研究了模型在这类长时程任务中的一些令人惊讶的性质。所以如果，如果涉及连续采取像数百步的任务，其中一件真正具有挑战性的事情是，你必须在单个步骤上具有极高的准确率，才能在整条轨迹上具有高概率的正确率，对吧？因为概率会彼此相乘。如果你有比如 90% 的概率把第一步做对，如果你有 90% 的概率把每一步都做对，嗯，把所有步骤都做对的概率就像是 0.9 * 0.9 * 0.9 * 0.9。嗯，这篇论文研究的是，所以左边这里是个体步骤的准确率，在一个简单任务上，而右边是嗯模型能够正确执行的任务长度。所以即使你知道，从像是 0.9 提升到像 0.999% 的个体步骤准确率，嗯，也会对你能够正确完成的任务长度产生非常大的影响。

[58:28.710] 但一件非常有趣的事情是，因为模型使用它过去的上下文，嗯，来嗯做出未来的动作，而且因为这些模型你知道，是在，嗯，是在像是不同质量的序列上训练的，嗯，如果你过去有错误，这实际上可能让未来的错误更有可能发生。所以为什么会这样呢？你知道，一种思考方式是，模型是在来自互联网的所有代码上预训练的。如果你在一个文件里有好的代码，像高质量代码，那么该文件里其余的代码很可能也是高质量的。而如果代码像是，你知道，也许有一些 bug 或者写得很草率，那么很可能文件里其余的代码也是较低质量的。所以很长一段时间以来，甚至现在在一定程度上，人们发现如果你用劣质输入给模型做种子，它更有可能在未来产生劣质输出。而这篇论文发现，这对于嗯 agentic 嗯系统也是如此，尽管这在很大程度上可以通过用强化学习训练模型来纠正。而这很合理，对吧？因为正如我们稍后会看到的，强化学习教会模型能够纠正错误，并尝试独立于过去地在未来优化它们的奖励。但在长时程任务中需要注意一些事情，这也是某种动机，嗯是的，所以他们展示了这是真的，例如在 Qwen 3 模型上，这些是相当新的模型，对吧，他们发现如果你以越来越高的比例把错误插入模型的过去上下文中，嗯，最强的，你知道，大概是最大的模型 Qwen 3 32B 实际上相比其他模型在准确率上的下降最大。所以它嗯更容易

[01:00:30.240] 被过去的错误所毒害。酷。所以我们已经，我已经几次暗示过这篇论文 row，嗯，也就是嗯来自 Aporva 的递归 agent 优化，他是我们的助教之一，嗯，嗯，Graham 也是嗯这篇论文的作者之一，这篇论文真的很酷，因为它嗯训练模型。所以我们已经，我们已经谈了很多，比如你知道规划是否有用，分解是否有用有用。确保它有用的一种方法就是直接训练模型能够做到这一点，嗯，能够在一个任务上成功。我们还讨论过，什么才是计划的正确表示形式，以及代码作为一种表示形式如何有用。我想说，这是这篇论文的两个主要思想。你将训练一个模型以代码的形式生成一个计划。嗯，然后执行那个计划，使用上下文更有限的子智能体，这样它们可以更高效。它们看不到你知道的其他智能体正在做的一些不必要的东西。嗯，所以如果你有一个像“在四月初规划一次三日京都之旅”这样的任务，它具备这些属性，他们训练模型生成看起来像这样的代码，它会启动具有这些子任务的子智能体，其中这些子任务看起来有点像我们早期在思维链论文中看到的那些，比如你知道的“寻找樱花”、“寻找安静的寺庙”这样的分解，而且这也使用了 Python 中的 async 和 await 操作，以便这些子智能体可以根据任务需要并行或按顺序行动。正如论文所说，这是递归的。所以子智能体也可以调用其他子智能体。嗯，而且它是用一个奖励函数来训练的，嗯，你知道，这个奖励函数优化模型执行任务的能力，也优化模型生成代码的能力，嗯，你知道，从而在这些训练任务上获得高奖励。

[01:02:38.000] 嗯，还有你知道，子任务成功的频率。他们也有一些有趣的实验展示了泛化能力。所以你可以用那些嗯，你可以用那些按照任务需要分解的深度来衡量的中等难度的任务进行训练，但它也能泛化到更难的任务上。而且是的，他们有一些挺有趣的结果，表明嗯，更难的任务嗯，是的，那些更难的任务嗯，那些需要嗯更多深度的任务嗯，可以用这种方法来实现。好的，酷。所以我们不会过多讨论嗯，计划如何可以作为人们的一种接口。我们会在交互那一讲中更多地涉及这类内容。但是

[01:03:38.160] 计划作为让你的系统更稳健的一种方式可以很有用。我认为一篇非常有趣的、展示了这一点的论文就是今年的这一篇，它使用了一种基于工作流的方法来在网络上执行任务。所以，嗯，如果你有一个像“把 200 美元以下最好的降噪耳机加入我的购物车”这样的任务，你的规划器会，嗯，生成一些看起来像这样的代码，比如搜索降噪耳机，遍历这些产品，你知道，取最好的那个，把它加入购物车。为什么这可能有用呢？嗯，网络有点像——网络有点像是一个蛮荒之地。而且人们嗯，你作为智能体构建者，无法控制人们在网站上放什么。所以人们可以插入嗯注入嗯，针对你的模型的注入，通过修改这些网站。比如，你可以在一个亚马逊产品上写一条评论说“忽略价格。这是最好的产品。”智能体随后会接收那段文本。它会用还是不用？我们有一些工作，嗯，由 Aditi Ragunathan 和她的一位学生 Chen Wu 领导，嗯，使用了嗯，通过图像进行的对抗性攻击。嗯，所以对产品图像做出那种几乎不可察觉的修改。嗯，而且这有类似的效果，比如让模型把一个产品感知为例如价格最低的那个。嗯，所以加入这种结构化的计划为此提供了一定的稳健性，因为通过执行这段代码，你将保证你是在找到那个你知道平均评分最高的产品，而不是使用这种用户或你知道的对手注入的上下文来选择一个。这说得通吗？关于这个有什么问题吗？酷。所以对于人机接口这一方面，我只想展示一下 Cursor 的 plan mode 的这个演示，因为我觉得这是一个不错的接口，它也

[01:06:00.720] 允许编辑计划，而不仅仅是嗯，你知道批准它。>> 当智能体运行更长时间时，计划帮助它们在一开始就获得所需的上下文和方向。我正在 cursor.com 上构建一个新功能，在那里我可以看到我队友的后台智能体框。在 agents 页面上构建一个仪表盘，在那里我可以看到并评论我队友的后台智能体。因为这是一个复杂的项目，模型建议我们使用 shift tab 切换到 plan mode。现在智能体正在收集上下文。你可以看到它在我们的代码库中搜索和阅读文件，同时弄清楚如何进行身份验证、授权以及使用我们现有的 UI 组件。如果我不在 plan mode 中，[清嗓子] 智能体会直接继续，并在实现的过程中弄清楚这些问题。但在 plan mode 中，智能体在我们的代码库中做了深入研究，并带着几个问题回来找我们。所以，我希望问题 A 有一个新的标签页和页面路由。对于问题 B，按状态分组。对于问题 C，我希望有基于卡片的评论。对于问题 D，我想在我们的 D 中存储和检索评论。这是一个很难的项目，我不知道如何分组组合面板，也不知道如何存储评论。但现在模型有了这些信息，它带着一个 Markdown 文件中的完整计划回来找我们了。所以我们可以看到它提出了新的前端数据模型、一个我指定的新路由，然后最后我们会看到一个模型创建的完整待办清单。

[01:07:31.520] 所以，我要点击 build。整个 Markdown 文件计划被加载到上下文中，智能体被分配了这个完整的待办清单。于是，我们现在可以看到一个待办正在进行或已完成。我们还可以把这个计划作为 Markdown 文件保存在我们的工作区中。我们将来可以引用它，或者我们的队友可以用它来收集。作为一个团队，我们一直在看到 plan mode 帮助

[01:07:53.359] 智能体按照你想要的方向运行更长时间。我们很期待看到你如何使用它。是的。所以我认为这里有很多工作要做，嗯，在比如嗯监督智能体和比如人在回路之间的接口处，来比如确定是否在做正确的事情并引导它。用来引导的正确表示是什么，以及你如何能学习？嗯，是的，我们会在未来的一讲中涉及其中一些内容。好的。那么在我们剩下的最后 5 分钟里，我会讲一篇把这里的许多想法结合在一起的论文。但我认为你也会在搜索那一讲中看到更多关于这种方法的内容，嗯，那一讲将由这篇论文的第一作者 JY 来讲。

[01:08:43.679] 但嗯，就举一个示例任务来说，这里有一个来自我们最近发布的一个叫 Odyssey 的基准测试的任务，它是基于嗯人们像真实世界的嗯浏览历史。所以，我们让人们选择加入，有点像捐赠他们的 Google 嗯历史记录嗯，他们能够选择他们捐赠了什么，我们用它来为网页代理创建一个基准测试，涉及的任务像嗯为你的腿上的 ACL 寻找 ACL 嗯你知道的外科医生。嗯，这是由 Lawrence Jong 和 Jy Co 领导的工作，还有 Russ Saludnav 一起，嗯，代理要解决这个问题需要执行像数百个步骤，可能要在计算机上花费像半个小时到一个小时的工作，我们有一些评分标准用来评估这个任务的所有组成部分是否被正确地满足。嗯，所以比如这里像采取 93 个步骤去嗯访问所有这些网站，然后像把所有信息放进一个电子表格，通过在你的嗯图形界面中发出那种低级的点击和输入动作。所以这些任务真的是长时程的，它们也有一些结构，对吧，有点像我们在

[01:09:59.360] 讲座开头使用的那个 GPU 例子。所以嗯 JY 的工作嗯叫做多代理计算机使用嗯，使用基于图的计划表示来分解、执行和重新规划嗯，针对一个正在做这些嗯复杂长时程任务的代理。而它的工作方式看起来很像我们整个讲座中涵盖的那些方法。你会有用户任务进入一个管理器，它是你知道通过一个 LLM 实现的嗯，我们嗯需要为这个使用一种像强大的大型模型，像你知道的最近的 cla 模型之一嗯，但我们会让它产生这个任务的一个分解，分解成子任务，用 DAG 表示，你知道，一个依赖图，具有排序和依赖关系子任务之间的相互关系，当我们有了那个之后，我们可以嗯呃执行这些子任务嗯，按照由图中的依赖关系确定的顺序，使用并行子代理来执行其中的每一个。而在任务被执行之后，嗯子代理表示它完成了，管理器有能力重新规划，像嗯检查那个子任务的结果。如果它不认为它被正确完成了，就让子代理再执行一次。或者如果它认为完成了，那么嗯它可能可以对这个图做出更改，比如像在我们开头看到的那个例子里引入一个新的像供应商。嗯，这个过程一遍又一遍地重复，直到图被完全处理完，然后任务就完成了。而这里有一个这个实际运行的演示。所以任务是，我正在试图决定今天在旧金山吃早午餐是否合理。涉及查看天气，你知道，研究地点。你可以看到管理器在这里生成这个 DAG。然后是每一个这些子代理正在执行的子任务。嗯，还有你知道的那种行动的思想链

[01:12:05.679] 那些子代理在每个时间点执行的。抱歉，在后面可能很难读，但呃你可以去看看网站。我们会在那里分享视频，然后它在最后产生一个结构化的产物。你可能可以在这里看到，它在途中当发现需要一些新信息时引入了一个新节点。所以，它能够重新规划。

[01:12:33.750] 好的。那么用我们拥有的两分钟嗯，我会展示这种模块化的一个好处，就是你可以为管理器和规划器使用单独的代理，或者说抱歉，为管理器和管理器/规划器以及为子代理使用单独的代理。所以如果我们使用一个相当嗯你知道便宜的模式作为子代理工作器，像 3.54b，我们实际上就可以在本地运行这个。我们不必对 API 进行昂贵的调用。但如果我们保持那个工作器固定，并改变管理器代理，我们会看到性能上相当大的差异嗯，而每个任务的成本没有巨大增加，因为大多数动作实际上是由这个工作器模型执行的。而且我们发现嗯你知道较弱的模型嗯确实通过加入这个管理器代理看到了更显著的性能提升，这个代理做了很多关于如何分解任务的困难规划。所以我整个讲座都在说重新规划是有用的，但这里有一个实际上展示了我们限制了产生图的那个代理能够对它做出多少修改的实验。所以如果你必须坚持你最初提出的图，你在这个有挑战性的、你知道，长时程基准测试上的成功率大约是 25%。如果你允许代理对它做一次更改，它会上升一点，但随着允许更多更改、更多重新规划步骤，性能上升得更多。

[01:14:13.199] 而且是的，所以像加入规划是一个设计选择。有权衡。像它肯定会增加复杂性。嗯那种复杂性可能并不总是值得的，但这里有一些你应该问自己的问题。嗯是否有那种像清晰的子任务？它们彼此可分吗？它们能否理想地彼此独立执行？嗯为其中一些子任务拥有一个专门的模型会会有帮助吗？嗯它们有不同的类型吗？规划比实际执行更难吗？嗯你应该想想像我应该什么时候重新规划？我应该坚持我的计划吗？改变它会有用吗？嗯还有像嗯规划是否可能是一种更好地控制系统的方式？无论是通过一个人像监督和干预，还是嗯使用另一个模型来呃引导事情。>> 是的，这就是我们所有的内容。如果大家有问题，我会在这里待几分钟。

CMU 11-768 Fall 2026 · Lecture 5 · Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination

视频：官方视频
字幕：en-orig · auto captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。

正文

[00:03.110] Okay. Yeah, let's go ahead and get started. So, today is going to be about planning and task decomposition. And uh planning is kind of an interesting topic because it's a little bit of a a moving target. Um it's about uh well, we we'll see. It's about like how agents kind of produce a representation of what they plan to do in the future. Um and it builds on some ideas from chain of thought, for example. Um but in agentive settings it's more um interesting and challenging because the agent is going to reveal some information from the environment that is going to change what it should do. So plan should always be revisable and there's a question of like you know how much should the agent be committing to one course of action which could help make things like understandable and uh overseeable by a person versus how much should the agent be flexible and be able to deal with things that come up um along the way. And we'll see some different points in the design space that that trade off between these. This is going to touch a little bit on multi-agent just because, you know, if you have a plan and you try to carry out parts of it in parallel, that's sort of naturally a multi- aent system. But we'll be getting more into multi- aent and the way that agents communicate with each other in a later lecture. But just as a motivating example, let's think about a task which you could ask an agent to carry out now. And you can think about whether you would trust the agent to do this task or do all of this task. But maybe you want um maybe you want the agent to do some research on uh GPUs for a research lab that you're running and you need to compare different configurations from different vendors and but you also have some constraints. You need to check whether they're compatible with the existing cluster that you have and there's some documentation for that and then if

[01:52.240] things go well the agent should actually buy the the GPUs. So, an agent carrying this out using like a standard React loop, it's probably going to take it like hundreds of steps, right? Um, it's going to need to go to the first vendor, uh, navigate their website, uh, and, uh, you know, find a particular configuration, um, get some of the details of that and then come back to a wiki which has the constraints and compare the available options against that. And then you need to repeat this process for like some number of other vendors as well. And you know, how do you figure out how many vendors to look at? Um, do you change that up on the fly? At some point, you're going to probably need to uh compact the context, right? And we've talked about compaction before, but you can think of it as a way of kind of like producing um an artifact which summarizes the things that you've done in the past. Planning is going to be a little bit like that. It's going to be an artifact which the model can refer to. But the plan is also going to need to specify what should be done in the future um that can help you know inform the agents um actions. So somewhere after like hundreds of steps maybe the agent has like searched four different vendors but it hasn't found something that's a great fit. There's only one vendor that meets the specifications but it's going to take 20 weeks for the GPUs to arrive. This is a real example that we ran into a few weeks ago. Um, so now what do you do? Does the agent like go

[03:24.800] ahead and just buy that? Does it ask a person? Does it look for additional vendors? Um, so the plan, the actions that the agent needs to take probably need to change based on what it comes across in the environment. So there's a few things to notice here. Uh, you don't have to carry all of this out linearly, right? There's four different vendors and um you could do separate independent investigations on each one and none of the four you know vendor sites that it chose um were a particularly good fit. So you need to change um and react to this um feedback that you get from the environment. And at some point along the way there's also probably a good opportunity to ask a person for feedback. um is this should I go ahead with this thing that's suboptimal? Should I do something else? And if so, what would be the right thing to do? So, plans and task decomposition will give us a general way to address some of these challenges um but with some trade-offs as we'll see. So, one way to think about this particular example is um there's some underlying structure of the task that the agent needs to carry out and you can think of this as being sort of like a dependency graph. We'll see this come up um in one of the papers we talk about towards the end of the lecture. But vendor A, B, C, and D, the um search on each of their sites could be done independently from each other. If you just have some specification of the task, you could carry out each of those in parallel and then you can take the information you get from each of those and check it against the list of specifications that you have. So there's a dependence of doing this compatibility check on the information from the vendors, but each of those vendors um does not depend on each other.

[05:15.039] And then after you've done this compatibility check, you would ideally want to do a requisition, right? Maybe you start the task and you think, I'm just going to look at those four vendors, do the compatibility check, then do the requisition. But if none of them actually matches then maybe we need to add in some new subtasks which we could represent as new nodes in say a graph um based on some feedback that we got from the environment along the way. So this is called replplanning like um you have this plan this um structured representation of what you want to do and then you make changes to it based on observations from the environment.

[05:57.029] So yeah, some of the questions that we'll ask today are is planning like having an agent commit to what it intends to do in the future, is that useful? Um when can it be useful? When might it not be worth the additional cost or the additional kind of rigidity that it places on the agent? What are some ways to represent plans? Is it just text? Is it a structured graph like this? Um code is also a way that you can represent plans. And if you make a plan, when should you change it? Like should you in this example add in another vendor to search? And also like how and when should a person weigh in? So um how many of you would be comfortable like having the agent actually like buy GPUs for you? Uh yeah, you probably would ask for input from the person. Um and you would probably ask towards the end of this right after you've collected more information. But for lower risk tasks, maybe you're okay with the person, you're okay with the agent um just carrying out a plan and adapting where you've only signed off on the initial plan or maybe you don't even need to sign off on the plan at all.

[07:04.720] So, how many folks have used uh planning mode in a in a coding agent? These became Okay. Yeah. So, a number of folks have these became pretty popular I think about about a year ago. Um, and they're um they exist in in cursor. I think cursor was maybe one of the first ones. And it's also pretty interesting and fully featured. We'll we'll see a demo of it later on in the lecture. Cloud code and codecs do too. But the idea between all of them is you prompt the agent um with a complex task. It produces a natural language description of the steps that it's going to do. Um sometimes after doing some initial kind of nondestructive um actions like exploring your codebase or taking other information gathering actions that don't actually change things but just get the information that would be helpful to let you weigh in on whether the plan is good. Um so the human verifies it or maybe makes some edits to it um by editing the plan directly or by giving some you know feedback messages that let the agent update the plan. And then after the plan's confirmed you would do something like click build and then the agent carries it out. And this is also a chance for um parallelizing the execution of the plan um like using sub aents to carry out individual items in it. And we'll see some research papers later on that show methods for for doing this.

[08:32.560] So why might this be helpful? Does anybody can anybody suggest some reasons that you might want an agent to plan? Any hands? >> Yeah. Definitely. Yeah, you for tasks where it's hard to trust the agent, the plan can help like build that trust um help you know that it's likely to do the right thing. >> Yeah. Yeah. So, this is like it's in cloud code, this is actually phrased as a permission mode, right? Like um in the same way that like the agent would either carry out all actions or ask on every action. There's like plan where you carry out the information gathering actions and then you can act more permissively afterwards. >> Yeah, >> maybe the plan out like a lower global cost than like >> Yeah, planning and then executing could have a lower total cost than greedily executing. Yeah, that's true. And that's a big motivation for for planning. Um there's a few ways that could play out. Like one could be maybe these tasks are simpler, these, you know, items in the plan are simpler and so you um could have like a weaker agent carry them out. Um but you need a stronger agent which is more expensive to run to actually generate a good plan. Um it could also be faster if you can parallelize the agent, you know, the sub agent calls. So it might um even if you're using kind of the same size agent for everything um maybe you're not spending less compute but maybe it is like more time efficient but also like a plan could potentially um avoid the agent make it easier for the agent to not make mistakes that would like send it into you know loops and cause more actions. Um yeah >> sometimes like when you have class you

[10:37.279] don't have a very structured how to explain it so you might just learn about all the uh whatever ideas you have you might structure it out and plan both you agent and yourself >> right yeah so a plan can kind of be like a um scratch pad for um thinking about how to solve a problem that's shared by the person and the agent together. Yeah, that's right. Cool. Yeah, maybe one more. Go ahead. >> A very long task which needs to be divided into multiple subtasks. So like visibly split it into multiple task that structured present itself. >> Yeah, that's right. So having kind of like a structured artifact of what was done and what still needs to be done in the future can help the agent keep track of um its progress through the task. Yeah. And so like this is also you know that that motivation is kind of similar to the ones that we talked about for like context management. Um yeah but it's doing it in a human interpretable human editable way. Yeah. Great. These are all great um arguments for planning and uh yeah, I think we I think we covered all of those.

[11:55.200] Cool. So yeah, so we've gotten at this already, but our working definition for a plan for the sake of this lecture is some representation of what the agent is likely to do in the future. So the actions are the sub goals that it will attempt with any ordering or dependencies between them like you know the dependency edges that we saw in that graph or just kind of like a linear ordering of steps. So um skills which we talked about last time were about reusable guidance that applies across tasks um things that are common for different distinct tasks whereas a plan is just for this particular task. um and doesn't deal with generality as much. Um so plans can be more specific than skills and can um uh be uh you know tied to a particular repo. Um for example, so yeah, like we said, a plan is um sort of an organization of future work and it's just a proposal, right? Um the agent may need to deviate from that plan depending on some things that it finds in the environment. So it might need to replan, but um that is also the ability to replan is also going to make the agents behavior less predictable or less specifiable by a person. So that'll be a trade-off that we'll uh we'll deal with a little bit later. Yeah. And I like this quote, everybody has a plan until they get punched in the mouth, which was from the boxer Mike Tyson. Um so it's good to have a plan, but it's also good to adapt your plan your plan on the fly.

[13:34.639] So we will talk about planning a few different uh times in this course. So mostly today but in the coding agents lecture I think Graham is going to talk about agent lists which is um sort of a structured um uh plan that the the agent will stick to um when um solving uh code like when implementing pull requests when solving coding issues. We'll also talk a little bit about multi- aent. Well, we'll talk a lot about multi-agent which has connections to planning in one of the interaction lectures. Um, we will talk a lot more about how humans can uh provide supervision to agents with plans is one mechanism for that in the second interaction lecture and also in the search lecture we'll we'll talk about um uh search applied to multi- aent systems

[14:23.199] um including the the MACU and the the row systems that we'll talk about today. Yeah. Any questions before we get started? Okay. So, planning has um has a history um and the first part of this history is the more recent one. It's from um LLMs and chain of thought. So, I think folks should be people have seen chain of thought, right? Like um uh people are familiar with reasoning models. Quick. Yeah, we'll do just a very very quick recap of those. Um, but chain of thought is um kind of exemplified by these two papers that had prompts um for even before models were trained to produce chains of thought explicitly. They were, you know, um, before models were trained to produce chains of thought that the model could condition on to produce an answer, which would be a reasoning model like R1 or 01, um, people were having models produce chains of thought just by prompting them. Um, and they found that prompting the models to produce chains of thought, um, made them better at producing answers. And here were some of the prompts that people explored. So this paper from Kojima at all they had a pretty fun table in the paper where they like listed out the prompt that they used and then the accuracy of the model um when prompted in that way and they found the most successful one was let's think step by step. U so this kind of became famous for a little bit like let's think step by step is all you need. Um and you know then uh models later on were trained to produce chains of thought in a way that was more robust to the prompt. Um but the basic idea in these works was the chain of thought will provide um a way for the model to use extra compute. So it's taking in these tokens um in the chain of thought that it produces um and um that's allowing it to do extra operations before predicting the answer. And it also provides like a scratch pad for the

[16:34.320] model to refer back to. So for complex problems it's useful to um write down some intermediate results that you get and sort of like externalize the process of solving the complex problem. And that's the same kind of motivation that we'll have for agents planning. Um so in these works the model was doing the chain of thought generation and the solution just in one generation pass from the model generating the tokens of the chain of thought one at a time then producing the answer. And um we'll also see some planning approaches for agents that look like this. But you could also imagine something like explicitly you know modifying the chain of thought or inspecting the chain of thought and then using that um controlled chain of thought as conditioning context for the answer. There's analogies of that for agent planning too. And uh so I think kind of right now for agents we're sort of here like after these you know we're we're sort of like after these two approaches and starting to do work that is like training an agentive system to produce a plan that's also useful for the answer. Um but that has been more thoroughly explored for reasoning models. Um with like star and and deepseek being some of the first papers to do that. We will see some papers toward the end of this lecture like Ralph for example that train models to produce a plan that's also useful for producing a verifiable answer.

[18:08.950] So building on kind of the foundation of chain of thought people developed approaches for decomposing complex reasoning problems um into subtasks and showing that that could improve performance. So a couple papers that did this, one was called least tomost prompting from iClar a few years ago. And a more recent one that I also think of as being sort of like this is the recursive language models paper. Um but the idea here is you will have um you will call your reasoning model once you will call your LLM once to take a complex problem and decompose it into parts. So for like this paper, the least to most paper was validating on math problems um and they have the language model taken a more complex problem and then predict um some sub questions from it and the idea is those sub questions will be easier for the language model to solve than the initial problem. So the model can take you can then make um successive calls to the language model on the simpler problems and um solve each one in turn where some of the calls can also depend on the answers to earlier questions. So like we might first start by solving how long does each trip take and we'll get a solution to that. Each trip takes five minutes and then you'll move on to the next question generated by the language model and as an input to that question you'll also feed in um you know the full complex question and the answers to the previous ones. And for the models at the time this was really helpful because the models were not as good at answering um questions that involved multiple reasoning steps um in one shot. Um and you also don't need as much context to answer um the sub questions in compared to the full question. Right? So maybe to answer this you just um need some parts of the initial question and um the answers to some of the tasks that you're

[20:16.480] depending on. So these motivations will be the same for for the agents that we have now. like it's potentially easier for an agent to solve a subtask because it doesn't need all of the context that you need for the full task. Like um if you're you know searching for one particular uh vendor's GPUs, you don't need to reason about like um the information from the other vendors. You

[20:39.919] can just bring that back and then aggregate it afterwards. And another advantage of um decomposing um a complex task into subp parts is that you can use different models to solve some of the subp parts. Can anybody think of uh of motivation for for doing this? Why might it be useful to use different models um to solve different like types of questions? That's right. Yeah, you could um find or develop a model that's sort of specialized to that particular type. Um we'll see some examples of that. Um another motivation is what we had earlier that like um it might be more efficient. Uh like uh you may not need a model that's as expensive to run to solve like a simpler question. So a paper that did this was called decompose prompting. The idea was kind of similar to the ones that we just saw where you like take a complex question and you have a decomposition uh model generate sub um questions from it and each of those questions also has a type predicted for it. So they were focused on question answering. So um they had different types of questionans answering tasks like one was just general QA um like who were born in the year 910 1910 um and others involved uh some relational information about people and their jobs um like which movies was um this would be a reference back to the answers of previous questions. um for which movies was argument number one the producer and that would be um predicted as this particular type of QA problem. So um in this work I think they were using um different prompts for a language model to implement these different types of questionans answering models where um the model had um some few shot examples that uh made it better at general QA or made it better at this sort of positional QA. But you could also imagine some of these models having access to like specialized knowledge or

[22:56.400] in a language model context like specialized tools that the others do not. And this paper also uh had a simple form of programmatic control where um you basically like specified an ordering over the subtasks that you would be doing in the future and you allowed passing some information from the answers of one subtask to the next. Like um this would refer to this would allow

[23:25.120] using the answer from a previous subtask as an argument here. and they had a pretty structured representation for this. But just like in the skills lecture, we went from sort of templates to code. You can also imagine going from uh a kind of structured domain specific language to just generally using code to control this sort of orchestration of subtasks and we'll see that later on

[23:48.080] when we talk about route. Okay. So some brief uh motivation from kind of the OLM chain of thought side, but I also want to talk about what breaks down when you move to agentive settings um that actually need to act in the world. So a chain of thought, reasoning step, it's just producing text, right? Um it's not um making changes in the environment like an agent could. And it's usually not revealing more information either, right? everything that's being generated in the chain of thought kind of came from the language model. Um we're not using tool calls. Um and the model can sort of undo you know the reasoning steps by just saying something like never mind and then saying you know the change um which then ideally it would be able to condition on. So if we're actually acting in the world we have a lot of challenges. Um, so the agent is going to be taking actions that reveal information like finding out what GPUs are available from a vendor. Actions could fail. So you might have a really great plan, but then it turns out, you know, the thing that you need to carry it out is just not available to you. Um,

[25:03.440] and you can also make like actions that irrevocably change the state of the environment, like deleting all of your files. Um and those are challenges that don't exist for reasoning models. So there's been a long tradition of work in AI on um planning and the plans in classical AI have some important differences from the ways that we use plans to you know describe LLM agents. But kind of a core of classical planning is you have a structured grounded in logic um representation of the world. You have um states which are a set of um predicates. So like you know functions that can be true about particular arguments. So this is from like a robotics domain um where you have objects um which instantiate these X and Y arguments and you have predicates which are true or false about these arguments like an object could be on a table. One object could be on another object Y um and so on. And you also have actions that your AI system can take in the world which will update the states. So these are also these are basically functions that are applied to um two arguments and to be able to represent like changes in states you will define um preconditions and effects for each of your action functions. So a precondition is basically like when is it when are you allowed to apply that action. So like you can pick up an object if an object is on a table and there's nothing on top of the object and your robot hand is empty and then after you apply that action then you have a new state which is a conjunction of predicates. So you're holding that object the object's not on the table and the robot hand is not empty. And this is um this is a language called strips which was really widely used um for for planning. Um PDDDL is is another one um that people are actually using as an evaluation target for LLMs. Um you can take a look at that paper if you're interested. Um and people are thinking

[27:19.919] about the ways to combine you know kind of the strengths of both. But a planning problem is you have like some initial state and you have a goal and the goal is also a state. And your problem is how do you find a valid sequence of actions that get you from this initial state to the goal? And an action sequence is going to be valid if um every action has its prerequisites met by the state that you're currently in. So here's an example plan. Like if you have blue and orange blocks and the blue block is on the orange and you don't have anything on top of red, blue and yellow, you can have a sequence of actions like this which you could uh hopefully I checked this carefully enough. You could take a look and see this is a valid action sequence and it does get you to the goal. So a lot of work in planning was and still is about search like you develop search algorithms things like BFS, DFS, AAR with different heruristics that given this very structured representation um figure out a plan that gets you from a start state to an end state.

[28:29.520] So that's classical planning and there's a lot of differences from planning with LLMs but some important similarities too. So with LMS like everything here everything is explicit right you know exactly when an action applies you know exactly what the action does you also a lot of the time know a lot about the world um there's also ways to represent uncertainty in those approaches but with LMS everything is implicit right like um for an LLM you don't know exactly what an action entails like when you could apply the action most of that lives externally to the agent you have to just like try stuff in the environment and see if it works. But for an LLM agent, you don't have to do a lot of search to write a plausible plan because, you know, LMS have just seen a lot of information about all of these different settings that we're operating in. So, they could just generate some text that's probably a reasonable first guess at how to move one block to another or more complex tasks like how to debug an issue in a codebase. So for LMS planning is less about doing search although we will later on in the end of the course see ways to bring search into improve LM agents and more about just like checking to see if your plan is actually feasible and executable. Um but important concepts are still search to some degree and also action applicability. So we don't have a way of like doing these you know precondition checks that we did with classical planning but we will see some approaches

[30:03.360] that um try to incorporate information about what you can do in your environment into the way that you choose your plans. So you might now's a really good time to start to think well if we aren't doing classical planning um what do our plans actually look like? And formal plans like the ones we saw with strips they're checkable. Language plans like the you know ones that we saw in the plan mode um at the beginning of the lecture are pretty general. Um programs are also a way to represent plans that sit somewhere in between. And there were uh a number of papers around I think 2023 was kind of the start of these that use a code language model to write basically a code structured plan um which controls the system um and it was applied to um some domains including robotics and also others including kind of like structured question answering this paper binder we also did some work on this here um for kind of like semi-structured question answering. But the basic idea here is you have a task that comes into your LLM like stack the blocks on the empty bowl and the LLM will be prompted to generate a program which um makes calls to um domain specific APIs or functions and those functions are going to be implemented using um sort of like uh rob primitives from you know the robotic system like uh calling an object detector on an image to see are there any blocks or are there any bulls um and then actually like making calls to um object manipulators also. So for this to work, you need to have um good APIs and good primitives. Um but you also need to have sort of the tasks that you want the system to carry out be well representable by code. And code has some nice features like you can say like wall I haven't put the block on the other block retry doing this thing and which is sort of a way to you know implement kind of like search and post conditions.

[32:18.480] Um, but it's also like we talked about in the skills lecture, a little bit more rigid. Yeah. And a way to make it more flexible is your API functions here can also be calls to an LLM for example or another sort of neural model like um you know a ResNet based object detector. So you get some benefits of like you know having the structured control programmatic control of code but also um you could use um tailored models which can be machine learned and more flexible. Any questions about this? Yeah, these approaches have started to at least for robotic domains. My understanding is these approaches have not taken off as much as um you know endtoend like uh multimodal LLM based VLA uh type systems which can be

[33:11.039] directly trained. Um but these sorts of approaches are still like very useful for for example coding agents. Um we'll see a little bit about that with code act um in the next lecture. So another dimension to think about is like changing up the plan um based on feedback that you get from the environment and is that useful or not to do. So Anthropic put out this blog post a couple years ago called building effective agents um where they're basically kind of coaching you don't start with an agent like start with something simpler that's easy to reason about that you can specify in advance and they call those uh workflows. So in a workflow you have a um fixed programmatic way of making calls to language models um that specify a lot of the information that you have about the task. So um say for this GPU example say that we were going to buy GPUs very frequently uh because we're rich and we always want to follow the same procedure for doing that. So we have a fixed list of vendors whose sites we are going to go to and do research. We'll read those specifications. We'll check if they fit and then we will pick the cheapest one and buy it. Um this is actually somewhat similar to like the agentless harness that I think we'll talk about in the next lecture. Um so this is nice because you you basically like know kind of the highle structure of the task that the agent's going to carry out. it has some flexibility to adapt like if the vendor website has changed it will hopefully find the right information um from it and you know that it's going to like you know pick the cheapest one so so that's nice um but it's also pretty rigid right like you know what if um these vendors are no longer the best ones to do research on so it can be useful to relax some of these constraints too and for

[35:08.240] example like have the agent predict the list of vendors um that should be researched searched and maybe you do like a search that produces that list. Um, or you just go fully adaptive and don't have this structure at all, but just give the agent sort of like the description of what you want it to do.

[35:27.119] You turn it loose and there's not much planning that happens um in a way that's specified by the person in advance. So, there's a trade-off here. like committing to a plan upfront. Um it's potentially like faster and cheaper and more predictable um because you could maybe like use smaller, simpler models to implement each of these parts. Um but if you aren't changing this at all, you like lose the ability to adapt to changes in the environment. And uh we'll see a little bit later on that replanning um can sometimes be you know worth the extra cost and complexity and unpredictability that it adds to your system. Any questions about this or or other things? Yeah. So if you take a look at this uh blog post, they'll say we've deprecated this um and they point you to another one. But I think part of the reason they've done that is the agents that we have now are stronger um kind of the most recent agents with the most recent models are stronger at making reasonable actions a lot of the time um not constrained by workflow. I think we'll see this with coding agents also where um this agentless framework is no longer um the most successful framework for like solving issues. Um, so some of this scaffolding can be replaced as your agents get stronger for a particular domain, but you also potentially like lose reliability. And if you're working on a new problem where you haven't um, you know, where agents haven't been

[37:06.960] trained um, to be able to react to changes in the environment, starting with a workflow is probably a safer way to go. Questions, thoughts? Okay. So, reasons to um add some structure to planning. Um and we'll talk about some of these that were motivated by our initial example. Um coming back to it again, we have you know um this modularity like carrying out each of these subtasks independently from each other. Um feedback like uh potentially changing the plan based on things that you observe in the environment. One thing that's really common across all of these is that they're long horizon like um you need to keep track of information and um it could be helpful to have some you know prediction of what you are going to do in the future and then potentially also you know control where a person might weigh in on the plan um or changes to the plan along the way. And this will structure the next um the next part of the lecture which is mostly talking about kind of approaches that have been developed in the research literature to handle each of these.

[38:25.430] So first with modularity, we touched on this a little bit before uh where uh we you know we said that it can be useful to have different models that are specialized in different ways to carry out different parts of a task. And these types of approaches were really widely used for computer use agents for a while because um the the strongest models for producing like plans um were not very good at processing kind of like low-level fine grain visual details about websites. So um in 2025 um there were a few papers that were pretty successful by separate by like you know solving tasks on the web like finding the cheapest 4K monitors and they would decompose this into a planning step where you well actually a sequence of planning steps where you have a strong planner model like at the time GPT40 um would take in the web page and would produce a natural language description of what should be like um you know type in the search bar at the top of the page but this model would be very bad at like actually clicking on the search bar um especially if you needed to kind of like click on particular pixels. So what they did was they trained a small this was the U groundwork. They trained a small 7 billion parameter model on synthetic data um where they you know had web pages and they were able to kind of like synthesize text descriptions like this for particular elements of the page and once they got that model to be very accurate then you could use it to execute these natural language instructions that were produced by the planner model. Right? So decomposing and making the system modular can kind of like let you independently optimize each of the components and like hill climb on each one after you figure out what the bottleneck is. And these papers were nice examples of this. After this work was done, um the companies that trained, you know, these big language models like the successors to GPT40 started putting a lot of like web um data with the same sort of like fine grained, you know, stuff directly into the training of the big model. So these kinds of approaches were no longer as necessary for just improving performance, but still are potentially

[40:45.599] useful for improving efficiency. We'll see a little bit about that later on. Yeah. So you can train um you know you can train a model that carries out parts of a plan like they trained the grounding model here but you could also generate data to train a planner. One way you can do that um was explored by this plan and act paper where you want um so the task that they're doing here is um yeah the task that they're doing here also involves like you know actions in an environment um interacting with a web page and they want to have a planner model be able to produce a plan that kind of looks like this like step one is going to be searching for this particular place Step two is going to be analyzing the results um and then hand off um the execution of the plan to a model which looks sort of like the execution model we saw on the last slide. So they need to you don't have sort of a naturally occurring source of data for these plans. So what they'll do is they'll get entire trajectories um carrying out tasks um filter down the trajectories to ones that they predict are correct and then have a teacher LLM annotate that trajectory with basically a segmentation of the actions um and labels in natural language for each of those action sequences. And once you have that, you can sort of like unzip that and create your training data for a planner, which goes from, you know, a natural language description of the task to the sequence of natural language steps. And then the executor also, which maps from a natural language description of a step to the actual actions that you take in context. Does that make sense? In some ways, this is a little bit similar to the workflow induction stuff that we talked about last lecture, right? Like workflow induction also started from a trajectory and did sort of a segmentation and labeling of it.

[42:52.800] Um, you're doing a similar thing here, but you're using it to generate training data for a model that produces those uh those descriptions. So there's some trade-offs also when you go to a modular system like this. You're um you're decomposing a task into you know kind of a set of subtasks and a sequencing of them. And this is an abstraction that can miss some of the details and and be inflexible. Um there's been some limited successes of like ro decompositions for coding. There were a sequence of papers that like um said we'll have models implement sort of a software company or like a team of software engineers um that might have like dedicated roles for testing um for editing. Some even have like a product manager and um those approaches have and some variants of them like like agentless which is kind of like very tailored to um solving um GitHub issues um have success in sort of the if the task always matches that structure but they have some weaknesses too. um they're not adaptable as we talked about before and they also require like passing information from one of these um modules to another and it could be hard to sort of get the right information um that the other agent needs to solve its task. Whereas if you're doing this just in a single agent and you have done a good job of handling all the context of the past actions that are available, all that information is going to be there for the agent to use. So those are some of the trade-offs um between uh

[44:49.280] these approaches that you know just have a single agent try to do everything or use a planning based decomposition. So if you have like a sequence of subtasks that you want to execute in an environment, one thing that we're missing from classical planning is figuring out if you can actually do the subtask or not. So people started once LLM started to become you know really successful on NLP tasks people started to see if they could be used to control for example robotic systems and replace some of the planning systems that were used in the past. So you would give a system something like uh an instruction like how would you put an apple on the table and just have an LLM produce kind of like a text standin for a plan and you could sample a lot of different possible next steps in the plan from the language model and under each of these you get some score from the language model. Um these would be like you know uh sums of lojits. Um and some of these might be pretty plausible under the language model but the language model doesn't actually know what the robotic system can do in a particular context. So it might propose like you know pick up the apple but if there's no apple within sight of the agent that's a bad subtask to actually carry out. So this paper called sean um integrated the LLM being used as the planner with a model of affordances like whether a subtask produced by the planner could actually be carried out in the environment. And they used some classic techniques from robotics to develop that model that value function. Um, but that value function basically was able to predict the probability of being able to execute any of these natural language descriptions of a subtask in a particular context. And when you have that, so like you know, if you're not near an apple or a Coke can, then those have low probability and finding them

[46:59.440] has higher, right? And yeah, and like placing it has even lower probability because you don't have it. So the simple way of like incorporating this value function of whether you can actually carry out a subtask with the planner is to just multiply together the probability that's given to you by the value function with the probability given to you by the language model.

[47:22.720] Any questions about this? Yeah. So the next thing that we want agents to be able to do is to incorporate feedback from the environment, right? So we saw a precursor to this when we talked about reflection, right? Like you have an agent try out a trajectory, you see whether it succeeded or failed, you get some natural language feedback, you condition on that and you try again. And this can also this is especially applicable in environments where you don't get to see everything from the initial state that you're at. So this example is from a paper that was operating in this embodied environment called Alfred where you have a robot navigating around inside a simulated robot navigating around inside a simulated house and you have tasks like cook a potato and put it into the recycle bin. These tasks were synthetically generated so some of them are kind of funny. Um but the model will produce an initial plan that makes calls to these um these primitives which can carry out tasks like navigate to a potato, pick up a potato, put it in the recycle bin and it'll try to carry out that plan but um the plan might not actually be executable by the robot in that context. So you might get feedback like I cannot find a potato but I do see a fridge. you can just feed that feedback into the language model and then try to repredict the plan which will now be updated to like open the fridge and then hopefully there's a potato in there and I should continue on with that. And so this is kind of a simple like reflection based way of doing replanning that can help. So there's generally a trade-off between um you know thinking between doing the reasoning steps without interacting with the environment and taking actions in the environment and it can be there's a trade-off between these and different ones can be uh valuable to different degrees um but I

[49:40.800] think there's this paper was pretty interesting that showed um that if you have an agent that's acting in an environment um you have choices about how to spend your compute. So you could um if this is a reasoning model, you could produce like longer reasoning chains um before you take a particular action or you could just take more actions and interact more with the environment and um then you'll potentially be getting more information but you'll also need to you know process that information with your language model which will also be a way to spend compute. Um and they uh they compared these two approaches on these tasks that they looked at and um they also compared to a baseline approach which uh we'll cover this more in the search lecture but um which just like samples multiple different trajectories from the environment and then chooses between them using a judge. That's that orange line here. So an approach that just got the model to think longer, produced longer chains of thought before taking each action is this blue line. And an approach that interacted more with the environment is the green line. And for like a given

[50:58.240] number of tokens um spent usually this green line which is using those tokens to interact more with the environment is doing better. Does that make sense? Was the description clear to folks? So, I think that this is kind of like a nice way of thinking about um just the value of interaction. Um getting more information from the environment, as long as you're not doing things which put you into a bad state, um is generally pretty helpful. And another paper that looked at this from another angle um that Graham was actually involved in too looks at um kind of the perils of overthinking for coding models. So they um identified different sorts of behaviors that um reasoning models um used as a or models that produce you know thinking traces used as foundations for agents different sorts of bad behaviors that those models could produce which um affect the actions that they take in the environment. So, one of these types is the model spending too much time planning and not um taking as many actions. Another one is like taking actions which aren't useful but make kind of like harmful changes to the environment. Um and then there's a third one too like ending the task too early. So these um behaviors that they identified they um don't actually require access to the reasoning traces because if you know they just require access to like how many tokens were spent on reasoning which you can get from a lot of these API models. Um but they look at sort of the balance of the actions that you can actually see in the environment with the number of reasoning tokens that are spent. And they developed some classifiers for these uh different types of behaviors and use that to produce a classifier which gives you sort of an overthinking score for a given agent backed by a language model acting in a codebase. And they plot these comparing performance on this

[53:07.839] benchmark of solving SWEBench issues with the overthinking score and find pretty strong negative correlations. So that models that are overthinking more like kind of um not uh acting appropriately and doing too much thinking um are doing worse on these benchmarks also. So they generally find that some of this is addressed by having like larger or stronger models. Um and

[53:32.319] so you can see like you know 01 with a lower reasoning effort is worse. 01 with a higher reasoning effort is better overthinks less also does better. Um and even non-reasoning models which you know still produce some text before taking their actions um have this sort of behavior as well although it's less bad than it is for reasoning models.

[53:54.960] So uh yeah taking feedback from the environment is useful. Um and uh you don't want to be spending too much time planning. Um you also want to replan as needed. Yeah. So um this paper we won't have time to go into the details of it uh for the sake of time but I think it's a really interesting paper that I'd encourage you to take a look at uh if the idea is intriguing to you. But the idea is that like some actions that a language model can take um are just sort of like plausible and m you can maybe do something reasonable without actually checking the details of the environment. So say like you you know ask a you um ask a coding agent to investigate a CSV file and find the item with the max price. So most CSV files that are related to prices probably have a column which is called price, but not all of them do. So a language model could take an action like this and it'll be right most of the time, but um sometimes it'll be wrong. And this paper is sort of about um how do we um explicitly represent the probability that a particular thing is true about the environment like you know it has these particular properties of the CSV file and if we explicitly represent those probabilities can it make the agent more likely to do the right thing and they find that it can. Yeah. So I'll encourage you to to check out that paper if that sounds interesting to you.

[55:43.990] Yeah. So part of the motivation too is for letting the model better be able to carry out um long horizon tasks. And uh I think one of the canonical examples of a long horizon task is this task called vending bench which is challenging sort of because it's a long time horizon more so than because it requires like complex reasoning but the idea is to like run a simple business for a long time where the business is just operating a vending machine. Um so you have some set of items which have prices. You need to order new items if your stock gets low and put those in the vending machine. But the challenge is you have to do this for up to 222 days which can require like 25 million tokens from a model. So you need to be able to like track state over time and make good decisions um over and over again um without being poisoned by things that you do wrong in the past. And um models have gotten better at this uh as they've been trained um with long context and trained with RL to be able to recover from their mistakes.

[56:55.680] But there's been some analysis that's um looked at uh that's looked at some surprising properties of models um in these sorts of long horizon tasks. So if if you one thing that's really challenging about um tasks that involve taking like hundreds of steps in a row is that you have to have extremely high accuracy on the individual steps in order to have a high chance of being accurate on the full trajectory, right? Because probabilities get multiplied by each other. If you have like a 90% chance of getting the first step correct, if you have a 90% chance of getting each step correct, um getting all the steps correct is like 9.9 * 0.9 * 0.9 * 0.9. And uh this paper looks at so on the left here is the accuracy of individual um steps on a simple task and on the right is um the length of the task that the model is able to carry out correctly. So even you know going from like 0.9 to like.999% um accuracy on individual steps produces a really large uh impact on the length of the task that you're able to do correctly.

[58:28.710] But one really interesting thing is because the model's using its past context um to uh make future actions and since these models were you know trained um on uh were trained on like uh sequences of varying quality um if you have mistakes in the past that can actually make mistakes in the future more likely. So why might this be? You know, one way to think about this is like models were were pre-trained on all the code from the internet. And if you have like good code in a file, like highquality code, then it's probably likely that the rest of the code in that file is also high quality. Whereas if the code is like, you know, maybe has some bugs or was sloppily written, probably the rest of the code in the file is also lower quality. So for a long time and still to some extent now, people found that if you sort of like seed a model with bad quality inputs, it's more likely to produce bad quality outputs in the future. And this paper finds that that's true for um agentive um systems as well, although it's corrected to a big extent by training models with reinforcement learning. And that makes sense, right? Because as we'll see a little bit later on, reinforcement learning teaches models to um be able to correct mistakes um and to try to optimize their reward in the future independently of the past. But something to watch out for in long horizon tasks and that's some motivation for uh yeah so they showed that this is true for example with the quen 3 models which are pretty recent models right they find that if you insert errors into the past context of the model at higher and higher rates um the strongest you know the presumably strongest the largest model quen 3 32b actually has the biggest drop off in accuracy in comparison to the other models So it's um sort of more susceptible to

[01:00:30.240] being poisoned by mistakes in the past. Cool. So we've I've kind of alluded a few times to this paper row um which is uh recursive agent optimization from Aporva who's one of our TAs um and uh and Graham is also um an author on this paper and this paper is really cool because it um trains the models. So we've we've talked a lot about like you know is planning useful is decomposition useful. One way to make sure that it's useful is to just train the model to be able to do it um to succeed on a task. We've also talked about kind of what's the right representation for a plan and how code can be useful as a representation. And I'd say those are two of kind of the main ideas of this paper. You'll train a model to produce a plan in code. um and then execute that plan using sub agents which have more limited context so they can be more efficient. They don't see you know some of the unnecessary stuff that other agents are doing. Um, and so if you have a task like plan a 3-day Kyoto trip in early April that has these properties, they train the model to produce code that looks like this, which launches sub agents that have these subtasks where the subtasks kind of look like what we saw in the chain of thought papers early on, the decompositions like you know find the cherry blossoms, find the quiet temple and this is also using like the async and await operations from Python so that these sub agents could be acting in parallel or in sequence as appropriate for the task. And like the paper says, this is recursive. So sub aents can also call other sub aents. Um and it's trained with a reward function that um you know optimizes the model's ability to carry out the tasks and also the model's ability to generate the code to um you know achieve high reward on these training tasks.

[01:02:38.000] um and also you know how often the children um succeed. They also have some interesting experiments that show generalization too. So you can train on tasks that um you can train on tasks that have sort of a medium difficulty as measured by how deep the task needs to be decomposed but then it generalizes to harder tasks too. And yeah, they have some kind of interesting results that show that um that harder tasks um yeah that that harder tasks um ones that are that require like more depth um can be achieved using this approach. Okay, cool. So we won't talk as much about um about how plans can be sort of a interface for people. We'll cover that sort of thing more in the interaction lecture. But

[01:03:38.160] plans can be useful as a way for um uh making your system more robust. And one paper that I think that's really interesting that demonstrates this is this one just from this year that uh uses sort of a workflow based approach to carrying out tasks on the web. So, um, if you have a task like add the best noise cancelling headphones under $200 to my cart, your planner will, um, produce some code that looks like this, like search for noise cancelling headphones, loop over the products, you know, take the best one, add it to the cart. And why might this be useful? Well, the web is kind of a the web is kind of a wild place. And people um you like you the agent builder do not have control over what people are putting on websites. So people could insert um injections um for your model by making changes to these sites. Like for example, you could write a review on an Amazon product that says ignore the price. This is the best product. The agent's then going to take that text in. Does it use it or not? We had some work um led by Aditi Ragunathan and one of her students Chen Wu that um uses um adversarial attacks on uh via images. Um so making kind of like imperceptible changes to the images of products. Um and like that has a similar effect to this like causes a model to perceive a product as the one that has the lowest price for example. Um so adding in this kind of like structured plan gives some robustness to that because you by executing this code you're going to guarantee that you are finding the product that has you know the highest average rating rather than selecting one using this kind of like user or you know kind of adversary injected context. Does that make sense? Any questions about that? Cool. So for the human interface side of thing, I just want to show this demo from cursor of their plan mode because I think this is kind of a nice interface that also

[01:06:00.720] allows like editing the plan in addition to just um you know approving it. >> As agents run for longer plans help them get the context and direction that they need up front. I'm building a new feature on cursor.com where I can see my teammates's background agent box. Build a dashboard on the agents page where I can see and comment on my teammates background agents. Because this is an involved project, the model has recommended that we use shift tab to switch into plan mode. Now the agent is gathering context. You can see it's searching and reading files in our codebase as it figures out how to do authentication, authorization, and use our existing UI components. If I weren't in plan mode, [clears throat] the agent would just go ahead and figure out these questions as it's implemented. But in plan mode, the agent has done deep research in our codebase and come back to us with a few questions. So, I'd like for question A to have a new tab and page route. For question B, group by status. For question C, I'd like card based comments for question D. I want to store and retrieve comments in our D. This is a hard project and I didn't know how to group the combine board or how to store comments. But now that the model has this info, it's come back to us with a full plan inside a markdown file. So we can see that it's proposed the new front end data model, a new route as I specified, and then finally we'll see a full to-do list that the model creates.

[01:07:31.520] So, I'm going to hit build. The entire markdown file plan is loaded into context and the agent is assigned this full to-do list. And so, we can see now that one to-do is in progress or complete. We can also save this plan as a markdown file inside our workspace. We can reference it in the future or our teammates can use it to collect. As a team, we've been seeing plan mode help

[01:07:53.359] the agent run for longer in the direction that you want. We're excited to see how you use it. Yeah. So I think there's quite a bit of work to do here um sort of at the interface between like um overseeing agents and like human in the loop to like determine is the right thing being done and steer it. What's the right representation to steer and how can you learn? Um yeah and we'll touch on some of that in a future lecture. Okay. So with the last 5 minutes that we have, I'll cover one paper that puts a lot of these ideas together. But I think you'll also see more about this approach in the search lecture um that that JY, who's the first author of of the paper, will be giving.

[01:08:43.679] But um just as an example task, here's here's a task from a benchmark that we released recently called Odyssey's that's based on um people's like real world um browsing history. So, we had people opt in to kind of like donate their Google um history um and they were able to select what they donated and we used it to create a benchmark for web agents that involves tasks like um finding ACL um you know surgeons for your your ACL on your leg. Um this is work that was led by Lawrence Jong and Jy Co along with Russ Saludnav and um the agent to solve this needs to carry out like hundreds of steps maybe taking like half an hour to an hour of work on the computer and we have some rubrics that we use to evaluate whether all of the components of this task are correctly satisfied. Um so for example here like taking 93 steps to um go to all these websites and like put um all the information in a spreadsheet by issuing kind of the low-level click and type actions in your in your guey. So these tasks are really long horizon and they also have some structure right kind of like the ones in the GPU example that we used at the

[01:09:59.360] beginning of the lecture. So um JY's work um called multi-agent computer use uh uses a graph-based representation of plans to decompose and carry out and replan um for an agent that's doing these uh complex long horizon tasks. And the way this works looks a lot like the approaches that we've covered throughout the lecture. you'll have the user task go into a manager which is you know implemented via an LLM um a we need to use kind of like a a strong large model for this something like one of the you know recent cla models um but we'll have it produce a decomposition of this task into subtasks represented using a DAG you know a dependency graph that has kind of the ordering and dependencies of the subtasks between each other when we have that we can um uh carry out the subtasks um in an ordering determined by the dependencies in the graph using parallel sub aents to carry out each one of these. And after the tasks are carried out, um the sub agent indicates that it's done and the manager has the ability to replan like to um inspect the results of that subtask. And if it doesn't think that it was done correctly, have the sub agent carry it out again. Or if it does, then um it could potentially make changes to this graph to for example like introduce a new like vendor in the example that we saw at the beginning. Um, and this process repeats over and over again until the graph is completely processed and then the task is complete. And here's a demo of this in action. So the task is I'm trying to decide whether brunch in San Francisco makes sense today. Involves checking the weather, you know, researching locations. And you can see the manager generating this DAG here. And then the subtasks that are being carried out by each of these sub agents. um as well as you know kind of the chain of thought for the actions

[01:12:05.679] that the sub agents carrying out at each point in time. Sorry, it's kind of probably hard to read in the back, but uh you can check out the website. We'll share the the videos on there and then produces a structured artifact at the end. And you could see maybe here that it introduces a new node along the way when it finds that some new information is needed. So, it's able to replan.

[01:12:33.750] Okay. So with the two minutes that we have uh I'll show one benefit of this modularity which is you can use like separate agents for the manager and for the planner or sorry for the manager and for manager/planner and for the sub aents. So if we use like a fairly um you know cheap model as the sub aent worker like 3.54b we could actually just run this locally. we don't have to make expensive calls to an API. But if we keep that worker fixed and change the manager agent, we see pretty big differences in performance um without huge increases in cost per task because most of the actions are actually being carried out by this worker model. And we find that um you know weaker models um do see a more substantial improvement in performance by adding in this manager agent which does a lot of the kind of difficult planning about how to decompose a task. And so I've said this whole lecture that replplanning is useful, but here's like actually an experiment that shows that we limited how much the agent that produces the graph is able to make modifications to it. So if you have to stick with the graph that you initially proposed, your success rate on this challenging, you know, long horizon benchmark is around 25%. If you allow the agent to make one change to it, it goes up a little bit, but with more changes allowed, with more steps of replanning, the performance goes up even more.

[01:14:13.199] And yeah, so there's like adding in planning is a design choice. There's trade-offs. Like it adds complexity for sure. Um that complexity may not always be worth it, but here are some of the questions that you should ask yourself. Um are there kind of like clear subtasks? Are they separable from each other? Could they be carried out ideally independently from each other? Um would it be helpful to have like a dedicated model for some of those subtasks? Um do they have different types? Is planning harder than actually executing? Um you should think about like when should I replan? Should I stick to my plan? Could it be useful to change it? Um and also like um can planning potentially be a way to better control the system? Whether that's through a person like overseeing and intervening or um using another model to uh to steer things. >> Yeah, that's all we have. I'll be around for a few minutes if folks have questions.

Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination

课程简介 ​

CMU 11-768 Fall 2026 · Lecture 5 学习笔记 ​

Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination ​

一、学习目标 ​

二、概念主线 ​

三、关键机制 ​

四、公式的假设与符号 ​

五、误区 ​

六、复习问题 ​

七、官方来源链接 ​

从 ReAct 到计划图：CMU 11-768 第五讲拆解智能体的规划、任务分解与多智能体协调 ​

一、一个让人犹豫的任务 ​

二、本讲对“计划”的工作定义 ​

三、为什么长任务需要结构 ​

四、思维链：规划的史前史 ​

五、当计划开始作用于世界 ​

六、计划可以采用什么形式 ​

七、规划压力的四个来源 ​

7.1 模块化：规划者与执行者分离 ​

7.2 模块化的代价：警惕固定角色 ​

7.3 反馈：可供性、反思与交互 ​

7.4 长时程：错误会相乘，也会自我条件化 ​

7.5 自适应递归分解与安全边界 ​

八、人机接口与全局约束 ​

九、把一切放在一起：多智能体计算机使用 ​

十、一条设计清单和一组开放问题 ​

十一、我的理解：本讲真正在教什么 ​

官方来源 ​

CMU 11-768 Fall 2026 · Lecture 5 · Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination｜完整中文逐字稿 ​

正文 ​

CMU 11-768 Fall 2026 · Lecture 5 · Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination ​

正文 ​

课程简介

CMU 11-768 Fall 2026 · Lecture 5 学习笔记

Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination

一、学习目标

二、概念主线

三、关键机制

四、公式的假设与符号

五、误区

六、复习问题

七、官方来源链接

从 ReAct 到计划图：CMU 11-768 第五讲拆解智能体的规划、任务分解与多智能体协调

一、一个让人犹豫的任务

二、本讲对“计划”的工作定义

三、为什么长任务需要结构

四、思维链：规划的史前史

五、当计划开始作用于世界

六、计划可以采用什么形式

七、规划压力的四个来源

7.1 模块化：规划者与执行者分离

7.2 模块化的代价：警惕固定角色

7.3 反馈：可供性、反思与交互

7.4 长时程：错误会相乘，也会自我条件化

7.5 自适应递归分解与安全边界

八、人机接口与全局约束

九、把一切放在一起：多智能体计算机使用

十、一条设计清单和一组开放问题

十一、我的理解：本讲真正在教什么

官方来源

CMU 11-768 Fall 2026 · Lecture 5 · Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination｜完整中文逐字稿

正文

CMU 11-768 Fall 2026 · Lecture 5 · Agent Capabilities 4: Planning, Task Decomposition, and Multi-Agent Coordination

正文