Agent Capabilities 3: Skills and Memory
- 日期
- Thu, Sep 3, 2026
- 课程资料
- 27 项
- 内容产物
- 4 项
课程简介
CMU 11-768 Fall 2026 · Lecture 4 · Agent Capabilities 3: Skills and Memory 学习笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
依据：Lecture 4 slides 文本提取（Slides 第 1–46 页）与课程逐字稿整理稿。图示与公式细节以官方 PDF 为准，本文对无法从文本确认的部分标注“待核”。本文为学习整理，内容已由用户审核确认。
一、学习目标
学完本讲，应能：
- 说明智能体在没有重新训练的前提下改进自身的三条路径：上下文窗口、外部产物、模型权重更新，并比较三者的代价与收益（Slides 第 3 页）。
- 区分四类“值得保留的经验”：episode（完整轨迹）、fact（事实）、skill（技能），以及 memory 与 skill 的交集（Slides 第 4–5 页）。
- 说明人类编写技能与从经验中归纳技能这两条路径的分工与负担（Slides 第 6 页）。
- 描述 Agent Skills 的包结构与“渐进式披露”三层加载机制（Slides 第 9–11 页）。
- 说明外部记忆的读写方式，以及记忆冲突的四种处理：ADD / UPDATE / DELETE / NOOP（Slides 第 17–18 页）。
- 说明 Reflexion 式反馈与相似轨迹条件化如何改善后续尝试（Slides 第 19–20 页）。
- 比较文本技能与代码技能作为技能表示的优势与缺陷，并解释代码技能为何更可测、更高效也更脆弱（Slides 第 29、33、36 页）。
- 说明技能生命周期：学习（judge → induce → admit）、使用（retrieve → act）、维护（add → repair → retire）（Slides 第 25、37 页）。
二、概念主线
本讲的中心问题是：智能体如何在不从头开始的前提下变得更好（Slides 第 1 页）。
主线可以概括为三步推进：
- 先定位经验存放的位置。 上一讲讨论上下文窗口内保留什么；本讲讨论把东西放到窗口之外，即数据库、文件系统等外部产物；第三种是对模型权重做 SFT 或强化学习，留待后续课程（Slides 第 3 页）。外部产物可检查、可编辑、可检索、可跨模型移植，但必须被“归纳、选择、维护”。
- 再区分经验的粒度。 Episode 保真度最高但最贵；fact 是可在未来复用的知识片段；skill 是可复用的“如何行动”的知识，可用代码、文本或两者表示（Slides 第 4 页）。
- 最后讨论技能从何而来、如何被使用和淘汰。 技能可以由人编写（编码偏好与组织政策），也可以由智能体从自身情节中归纳。两者互补：前者来源明确、质量可控但耗时；后者可自动化，但可能带噪声并产生意外后果（Slides 第 6 页）。
一个反复出现的张力贯穿全讲：多少行为留给模型解释，多少行为固定在产物里（Slides 第 36 页）。
三、关键机制
3.1 共享任务结构与三类经验
课程用购物网站任务说明“共享结构”：任务 1“把 Sony 耳机加入愿望清单”与任务 2“查询无线键盘价格区间”，都包含“进入商店 → 找到搜索框 → 填入查询 → 点击搜索”这一共享过程，区别只在后续的 TASK-SPECIFIC FINISH（Slides 第 2 页）。由此提出三问：表示该用文本还是代码？如何从经验中归纳？技能何时被检索、检查、修订或遗忘？
三类经验的典型例子（Slides 第 4 页）：
- Episode：完整轨迹，每一个观察和动作；“如果确切的商品、价格或序列会再次起作用”时有用。
- Fact：“这家商店只在商品页显示价格”；当事实会在未来任务中复现或需要被更新时有用。
- Skill：
填搜索框 → 点击搜索 → 打开一个结果；当后续任务会重复同一模式时有用。
3.2 人类编写的技能与 Agent Skills 标准
课程用例：同一段请求被反复输入每个 PR 对话——用 Black（88 字符行宽）、Ruff（自定义规则）、公共 API 类型提示、Google 风格 docstring、所有新函数用 Pytest。当你想按固定规范反复做某事时，就把这段指导存成技能（Slides 第 8 页）。
技能的包结构（Slides 第 9 页）：
python-review/
├── SKILL.md # 必需：指令 + 元数据
├── scripts/ # 可选：可执行代码
├── references/ # 可选：文档
└── assets/ # 可选：模板、资源SKILL.md 开头的 YAML 元数据含 name、description，示例中还出现 triggers 列表。description 的写法是给出“何时该应用该技能”的指导，例如用户说 “review Python code”“check Python style”“lint Python” 或提到 code quality 时使用。
渐进式披露分三层（Slides 第 9 页表）：
| 层级 | 文件 | 进入上下文窗口的时机 |
|---|---|---|
| 1 | SKILL.md metadata（YAML） | 总是加载 |
| 2 | SKILL.md 正文（Markdown） | 智能体触发该技能时加载 |
| 3 | 附带文件（文本、脚本、数据） | 智能体读取这些文件时加载 |
系统提示中始终只有技能索引（名称与描述）。Slides 第 10 页给出 HermesAgent 的提示片段，要求模型“回复前扫描技能列表，若匹配或部分相关就必须用 skill_view(name) 加载并遵循其指令”，只有确实无关时才可不加载。Slides 第 11 页展示四轮中进入上下文的内容：第 1 轮只有索引（Hermes 中约 3k token），第 2 轮 skill_view('python-review') 把正文作为工具结果带入，第 3 轮终端运行脚本只把输出带入——代码本身从不进入上下文，第 4 轮按需读取 references 中的文档风格规则。
3.3 技能的两个来源与维护案例
技能包的来源有三类（Slides 第 14 页）：随 harness 捆绑；由人安装或编写（放入 .agents/skills/ 或从仓库、注册表安装）；由智能体自己创建（Hermes 的 skill_manage 可创建、修补、删除；OpenHands 有 /skill-creator 从刚完成的工作流起草）。
OpenHands 的建议是：不要从零开始写技能；在智能体完成一个你想重复的工作流之后立刻创建，然后测试几次。
代码审查技能的维护案例（Slides 第 15 页）：维护 review 技能 → 记录每次运行 → 给每次运行打分 → 找出反复出现的失败 → 修补 SKILL.md。OpenHands 在自己的仓库上对每个 PR 运行该技能，PR 合并后由模型评判者统计有多少建议被人保留；模型分析失败记录后指出两类反复问题——忽视仓库约定（约 15%）和批准带严重缺陷的 PR（约 10%）；再由同一模型起草 SKILL.md 更新：发现严重问题时要求改动，发布建议前先对照仓库检查。
3.4 外部记忆与冲突处理
MemGPT 式设计把记忆分层：不更新的系统指令、从外部记忆拉取内容的工作上下文，读写都由模型发出的函数调用控制；另设单独的存储保存实际消息（Slides 第 17 页）。逐字稿中的例子是用户提到 Six Flags，系统发出工具调用在过往交互中检索相关提及，把结果放进工作上下文，模型便能据此作答。
Mem0 式流程则解决“记忆会变”的问题：消息到达后先由语言模型抽取候选记忆，用基于嵌入的检索器在数据库中找到相关的前 K 条，再由语言模型判断每条新记忆属于哪种情况（Slides 第 18 页）：
- ADD：尚无相似内容被存储；
- UPDATE：消息改变了已有记忆；
- DELETE：消息否定了已有记忆；
- NOOP：消息已被存储，或不相关。
课程在讨论中也把它与上一讲的 Delta 类比：线性注意力只能追加值，Delta 能检索并改写为特定查询存储的向量，这里的文本事实更新是同一目标的不同实现。
3.5 轨迹反馈与相似轨迹条件化
Reflexion（Slides 第 19 页）的循环是：尝试 → 评估者预测奖励 → 生成自然语言反思 → 重试同一任务。课程给的具体例子：问题是“1776 年 10 月 28 日在纽约白原附近进行的一系列战役是什么？”第一次回答 Finish[Battle of White Plains] 错误；生成的反馈指出“问题问的是一系列战役，我只给了一场战役的名字……我会确保提供更多上下文，例如战役的名称”；第二次回答 Finish[The New York and New Jersey campaign] 正确。
相似轨迹条件化（Slides 第 20 页，ExpeL 等）的三个要点：存储的是完整轨迹，成功与失败都存，有时在入库前被改写和标注；轨迹池可以离线收集、在线增长或预先手写；使用时用文本嵌入检索与新任务最相似的运行，作为 few-shot 示例放入上下文。
3.6 归纳技能：从文本工作流到代码技能
Slides 第 22 页把技能定义为任务中可复用的片段：搜索商品被表示为“fill('element145', {query}) → click('element147')”，并配一段自然语言说明；价格区间报告则复用同一搜索过程再加上读取价格和汇报的步骤。这里也点出问题所在：元素 ID 一变技能就失效。
在线学习评估设置（Slides 第 24 页）：评估任务顺序到达，成功任务在下个任务到来前更新记忆，指标是所有已见任务的累计成功率；技能记忆随评估流增长，例如“搜索商品”“把商品加入愿望清单”“报告价格区间”。
技能生命周期（Slides 第 25 页）：
- LEARN：episode → judge → induce → admit → memory
- USE：memory → select/retrieve → act → outcomes
- MAINTAIN：outcomes → add → repair → retire → memory
Agent Workflow Memory（Slides 第 26–28 页）的做法：给出一条查询“告诉我我们商店迄今收到的、提到 satisfied 这个词的评论数量”及对应动作序列，评判模型先判断最终状态是否可能正确，若通过则用少样本提示让语言模型归纳出工作流，把非固定元素写成描述性变量名：
Task: Search for reviews matching "{term}"
# To find reviews for {term}, I will search for "{term}" in the reviews page
click("All reviews")
fill(757, "{term}")
click("Search")Slides 第 27 页的归纳提示词要求：找出多个任务中重复的动作子集，每个工作流是常用的子例程，不得生成相似或重叠的工作流，每个工作流至少两步，用描述性变量名表示非固定元素。
技能作为文本与技能作为代码（Slides 第 29 页）对照：
- 文本 + 示例（AWM）：灵活，是智能体解释并适配当前页面的提示；但手工，智能体仍要自己发出每个底层动作。
- 代码（ASI）：
def search_product(search_box_id: str, query: str):
"""Search for a product using the search box. ..."""
click(search_box_id)
fill(search_box_id, query)
keyboard_press('Enter')优点是可作为工具使用、可测试、可分层（技能调用技能）、高效（一次调用替代多步模型生成）；缺点是死板——即使页面已变，它也严格照做。
代码技能的归纳与准入（Slides 第 30–31 页）：从 episode 归纳出 search_reviews(search_box_id, search_button_id, search_term) 和 open_marketing_reviews() 等函数，评判模型检查最终状态与技能是否被使用，并且可以执行生成的抽象代码再评判结果。SkillWeaver 用伪代码描述该循环：candidates = propose_skill(task)、episode = practice(candidates, task)、若奖励模型通过则 memory.add(candidate)，否则 revise(candidate, task)。Slides 第 31 页给出一个真实 lint 例子：identify_pill(page, imprint="M366", color="White") 中 color 参数被定义却从未使用，于是补上使用 color 的分支。
压缩效应（Slides 第 32–33 页）：无记忆的基线是 50 步上限；有代码技能时 4 步完成。在五个 WebArena 站点上平均，checkpoint 到达率从 41.3（无记忆）到 59.5（文本技能）再到 80.2（代码技能），每任务步数从 24.5 降到 20.6 再降到 15.0。
泛化失败案例（Slides 第 34 页）：在一个网站归纳的技能拿到 Target 上评估——技能假设排序是点击一个下拉菜单，而 Target 的排序打开的是侧边栏，因此失败。PolySkill（Slides 第 35 页）的应对是用代码抽象：组合逻辑针对抽象方法写一次，每个网站只实现原始的浏览器动作。
表示方式的后果总结（Slides 第 36 页）：检索到的 episode 保留具体行为但长且难迁移；文本技能提供灵活指导但不提升效率；代码技能可执行、可组合、高效但可能脆弱。
3.7 从失败中学习、评判者质量与过度检索
ReasoningBank 的失败案例（Slides 第 38 页）：任务要求给出 Sony 蓝牙耳机的完整名称及价格区间，智能体搜索 “Bluetooth headphones Sony” 得到 5,578 条跨部门结果、每页 12 条，一路翻页直到步数上限，什么都没发给用户。归纳出的策略是：收紧查询、调整每页显示数量、使用侧边栏的类别过滤器。
不同记忆类型对失败的受益不同（Slides 第 39 页）：在归纳策略时失败帮助很大（ReasoningBank）；对轨迹（Synapse）或工作流（AWM）帮助较小——课程提出疑问“展示不该做什么的示范？”是否有效。
评判者准确率的影响（Slides 第 40 页）：用模拟评判者（从 oracle 到抛硬币，以固定概率翻转真标签）替代真实评判者后，性能随评判准确率变化而下降，但在相当宽的准确率区间内记忆仍带来改进。
过度检索的代价（Slides 第 41 页）：检索更多经验反而可能让性能下降（ReasoningBank 的例子；SkillsBench 对人类策划的技能也发现同样现象）。两种解释：部分经验确实无关（记忆本身的局限）；智能体条件化能力有限（可能因上下文变长而分心）。
记忆整合（Slides 第 42 页）：TroVE 会丢弃使用次数少于 ½·log₁₀(n) 的函数，n 是迄今见过的示例数。
训练归纳器（Slides 第 43 页）：技能的价值是延迟兑现的——任务 1 的奖励只说明任务是否成功，不说明保存解法是否有价值，只有后续任务才能体现。因此把后续任务放进同一个训练样本：两个相关任务在一次 rollout 中（SAGE），或一条长轨迹（AgeMem）。奖励形式可为 r₁ + 1[both succeed] · 1[task 2 reused its skill]。幻灯片给出的数字是（AppWorld test-normal，Qwen2.5-32B）：仅看结果 55.4，两个任务都成功 56.6，成功复用技能 60.7。
3.8 技能所有权与人的介入
Slides 第 45 页给出决策分工表：当政策已知、正确性是规范性的或高风险、需要上下文判断的罕见例外、需要问责归属时，人的努力更有价值；当技能必须通过交互发现、结果是可执行且廉价可测的、路由重复并产生反馈、漂移在结果中可见且可修复时，智能体的努力更有价值。
四、公式与数值的假设和符号
本讲的定量内容以对比数字为主，形式化程度不高，需注意：
- 累计成功率（Slides 第 24 页）：在任务顺序到达的设置下，评估的是“迄今见过的所有任务”的成功率，而非单个任务的成功率。这意味着后出现的任务与先出现的任务被同等计入，且记忆中已积累的技能会影响后续任务。
- TroVE 的丢弃阈值：
使用次数 < ½·log₁₀(n)，n 为迄今见过的示例数（Slides 第 42 页）。符号含义在幻灯片上明确，但阈值的推导依据未在提取文本中给出，待核。 - SAGE 的奖励表达式：
r₁ + 1[both succeed] · 1[task 2 reused its skill]（Slides 第 43 页）。指示函数表示条件成立取 1。各项权重未在提取文本中说明，待核。 - Slides 第 33 页的三组数字（41.3 / 59.5 / 80.2 与 24.5 / 20.6 / 15.0）是五个 WebArena 站点上的平均，比较条件是“具有重复结构的 web 任务”，不应当外推到所有任务分布。
- Slides 第 13 页的 33.9% → 50.5% 是 SkillsBench 主实验的平均通过率变化；同时明确“87 个任务中有 13 个技能仍有损害”。这些数字只在该基准、该策划技能组合下成立。
- 逐字稿提到 Hermes 系统提示中技能描述总长约 3,000 token，与 Slides 第 11 页的“≈3k tokens in Hermes”一致。
五、常见误区
- 以为有长上下文就不需要外部记忆。 课程明确说，即便有百万级 token 上下文，跨多个任务仍会超出，且你希望从过去任务中保留某些东西（Slides 第 3 页；逐字稿同段论述）。
- 把技能等同于工具或函数。 技能可以是文本指导，也可以是代码；文本技能灵活但效率不变，代码技能高效但死板（Slides 第 36 页）。
- 以为技能总是有益。 Slides 第 13 页指出技能在 87 个任务中的 13 个上仍造成损害，“技能可能取代更好的默认策略”；Slides 第 41 页指出检索更多经验可能导致性能下降。
- 把失败经验一律当成有用的记忆。 失败对归纳策略帮助大，对轨迹或工作流帮助较小（Slides 第 39 页）。
- 假定元素 ID 稳定的技能可以跨页面或跨站点复用。 元素编号变化或界面结构不同都会使技能失效（Slides 第 22、34 页）。
- 忽视触发器由模型决定。 逐字稿中的问答指出，触发与否完全取决于模型的工具调用，取决于模型能力和描述写法，因此需要对技能做评估。
- 以为技能越多越好。 逐字稿与 Slides 第 41、42 页都指向：无关技能会分散注意力，且可以通过使用频率剔除。
- 把记忆当作只增不减的日志。 记忆需要 UPDATE 与 DELETE 来处理冲突（Slides 第 18 页），也需要 repair 与 retire（Slides 第 25 页）。
六、复习问题
- 上下文窗口、外部产物、模型权重更新三条路径各自的优势与代价是什么？为什么外部产物更容易在人与系统之间充当接口？（Slides 第 3 页）
- 给定一条智能体轨迹，你会把其中哪些部分存为 episode、fact、skill？判断依据是什么？（Slides 第 4 页）
SKILL.md的 YAML 元数据中name与description各自承担什么作用？为什么 description 需要简短？（Slides 第 9–10 页）- 渐进式披露的三层分别是什么？在第 3 层中，为什么“代码本身从不进入上下文”是重要的？（Slides 第 9、11 页）
- ADD / UPDATE / DELETE / NOOP 四种记忆冲突处理各自的触发条件是什么？举例说明 UPDATE 与 DELETE 的区别。（Slides 第 18 页）
- Reflexion 的例子中，第二次尝试为什么能答对？这种反馈能推广到不同但相似的任务上吗？（Slides 第 19 页）
- 文本技能与代码技能在“测试”和“效率”两个维度上的差别是什么？Slides 第 33 页的数字说明了什么？（Slides 第 29、33 页）
- Target 网站上的失败案例说明了技能泛化的什么限制？PolySkill 用什么软件工程手段缓解它？（Slides 第 34–35 页）
- 为什么说技能的价值是延迟兑现的？Slide 43 的奖励设计如何把后续任务纳入训练样本？（Slides 第 43 页）
- 在什么情况下人的努力更有价值，什么情况下智能体的努力更有价值？（Slides 第 45 页）
七、给定资料中的官方来源链接
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读 / 官方课程日程：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=6zigF2a-2Pw&list=PLSN0qpDfUvTM&index=4
- Lecture 4 slides：https://www.cmu-agents.com/slides/lecture-04-memory-and-skills.pdf
（本文所引论文、博客与基准名称均来自 Slides 与逐字稿中给出的引用标签；除上述官方链接外，未附加其他链接。）
从上下文窗口到技能库：CMU 11-768 Lecture 4 中的智能体技能与记忆
来源：平台字幕与官方 Slides。内容已由用户审核确认。
一、一个反复出现的问题：智能体为什么总要"重新学"同一件事
想象一个 GUI 智能体被要求"把 Sony 耳机加入愿望清单"。它需要打开购物网站、找到搜索框、输入关键词、点击搜索、打开某个结果、点击"加入愿望清单"。下一次任务变成"查询无线键盘的价格区间"，它又要打开同一个网站、找到同一个搜索框、输入不同关键词、点击同一个搜索按钮——只是最后几步不同。
Slides 第 2 页把这种结构总结为"共享搜索流程 + 任务专属收尾"：TASK 1 与 TASK 2 共享 goto store → find search box → fill(...) → click Search，随后才分叉。这带来 Lecture 4 的核心问题：智能体如何在不重训模型的前提下，把这类可复用的结构从一次任务迁移到后续任务？
讲者在开场明确说，上一讲讨论的是"上下文窗口内保留什么"，而本讲的重点是"如何超越上下文窗口，在外部存储中跨任务保存信息"（逐字稿 00:01–00:56）。这就是 Lecture 4 的主线：技能归纳（skill induction）与记忆（memory）。
二、三种更新智能体的位置：一张表撑起整讲的框架
Slides 第 3 页给出一个三行表格，几乎是最重要的"路线图"：
| 位置 | 代表方法 | 优点 | 缺点 |
|---|---|---|---|
| 上下文窗口 | 上一讲 | 对发生过的事高保真 | 昂贵、噪声大、不会判断什么重要 |
| 外部产物 | 本讲 | 可检查、可编辑、可检索、可移植 | 必须被归纳、筛选和维护 |
| 模型权重 | 后续 SFT 讲 | 推理更快、行为改变更广 | 更新慢、不透明、模型专属 |
其中"外部产物可移植"这一点讲者特别强调（逐字稿 06:21–06:49）：描述"如何在某网站搜索商品"的文本可以被多个模型共用；而权重更新只对那个特定模型有效，且难以被人检视和修改。这也解释了为什么本讲大量篇幅花在文本技能与代码技能上——它们是人与系统之间的接口。
【学习者解释】这张表之所以关键，是因为它把"agent 如何变强"从单一维度（改权重）变成了三维坐标。Lecture 4 关心的不是"哪种最好"，而是"给定当前任务分布和数据量，应该把经验放在哪一层"。
三、经验的三种粒度：Episode、Fact、Skill
Slides 第 4 页将"什么经验值得留下"分为三类：
- Episode（情节）：完整轨迹，包含每一步观察和动作。当确切商品、确切价格、确切顺序在未来仍会重现时有用。
- Fact（事实）：例如"这家店只在商品页显示价格"。当该事实会在未来反复出现、而且可能被更新时有用。
- Skill（技能）：例如"填入搜索框 → 点击搜索 → 打开结果"。当后续任务会重复同一模式时有用。
讲者在逐字稿 09:10–11:35 的对应讨论里补充了两点：Episode 保真度最高但 token 最贵；Fact 类记忆是早期记忆系统（例如 ChatGPT、Anthropic 的记忆设置界面）的主要形态，本质是"从交互中由 LLM 生成的一段文本，被存在外部"。
Slides 第 5 页进一步区分了 Memory 与 Skill：Memory 是从智能体自身交互中保存下来的内容（比如上次买过的商品），Skill 是"如何行动"的可复用知识，可以是代码、文本或两者。这两者会交叉——"agent 从自己成功的搜索里归纳出的技能"就是交叉点。
【学习者解释】把经验拆成这三类，实际是在回答一个工程问题：你希望未来的智能体在多大程度上"记起具体细节"，又在多大程度上"学会方法"？ 前者容易过拟合到某个站点某次版本，后者容易抽象过头而失效。这组张力贯穿整讲。
四、人来写技能：Agent Skills 的标准结构与渐进式披露
4.1 什么时候应该写技能
Slides 第 8 页给出一个很具体的判定："当你想要反复做同一件遵循固定规范的事"——例如加测试、审 PR、升级依赖。页面上展示的反例是一段每次都粘贴到 PR 对话里的指令：Black 88 字符、Ruff 自定义规则、公共 API 类型注解、Google 风格 docstring、新函数用 Pytest。
【学习者解释】这个例子的说服力在于：它先给出"笨办法"——每次都手打一遍。技能的价值不是让模型变聪明，而是把"固定规范"从重复劳动变成一份可维护的产物。
4.2 三层的目录结构与 YAML 头部
Slides 第 9 页展示了一个 Python review 技能的目录：
python-review/
├── SKILL.md # 必需：指令 + 元数据
├── scripts/ # 可选：可执行代码
├── references/ # 可选：文档
└── assets/ # 可选：模板、资源SKILL.md 是必需文件，开头是 YAML 元数据，包括 name、description、triggers 等。讲者在 15:31–17:28 补充：description 的作用是"告诉语言模型什么时候应用这个技能"，因此必须简洁；references/ 是补充文档，agent 按需加载；assets/ 存放模板等资源。
4.3 渐进式披露：三层按需加载
Slides 第 9 页表格把加载分为三层：
- 层 1：
SKILL.md的 YAML 元数据——始终加载。 - 层 2：
SKILL.md正文（Markdown）——当 agent 触发该技能时加载。 - 层 3：打包文件（文本、脚本、数据）——当 agent 主动去读时加载。
Slides 第 10 页展示 Hermes Agent 的系统提示词片段，里面有一份 <available_skills> 索引，只列出技能名与描述。Slides 第 11 页把一次典型交互拆成四步：
- 系统提示词含技能索引（Hermes 中约 3k token）；
skill_view('python-review')返回SKILL.md正文；python scripts/run_checks.py src/只把脚本输出放进上下文，被检查的代码本身不进入；- 需要时再
skill_view('python-review', 'references/docstring-style.md')。
讲者在 18:41–19:36 明确说明："你们会在作业一中实现这个"——这是课程与 Slides 第 11 页的直接对应。
【学习者解释】渐进式披露是技能系统里最容易被低估的设计。它的真正含义是：技能不是"一段长的系统提示"，而是一份可被模型主动检索的文档索引。因此"技能写得好不好"不仅是文本质量问题，也是"模型能不能判断何时该打开它"的检索问题。这也直接引出讲者在 19:36 之后的提醒：技能越多，元数据总量越大，性能会受影响。
五、SkillsBench：人类写技能到底有多大用？
Slides 第 12 页介绍了 SkillsBench：8 个领域、87 个真实工作任务，人工为每个任务挑选技能包，主实验在 18 个模型-框架组合 × 87 个任务上对比"无技能 vs 精选技能"。
Slides 第 13 页给出几个关键结果：
- 平均通过率从 33.9% 提升到 50.5%；
- 任务仅配 1–3 个技能时增益更大，≥4 个技能时增益更小；
SKILL.md更短的一组增益也更大； - 87 个任务中有 13 个，技能反而使性能下降——技能可能"挤掉更好的默认策略"。
【学习者解释】最后一行的意义常被忽视。它说明技能不是纯收益，而是一种注意力与行为的重分配：一旦模型开始遵循某份技能，"更好的默认策略"反而被压制。这与讲者 07:59 之后现场提问得到的学生反馈一致：模型常会"过度推断"，把技能写得比实际需要更宽。
六、技能的来源与维护
Slides 第 14 页列出三种来源：随框架捆绑、人工安装或写入（常放在 .agents/skills/）、由 agent 自己创建（Hermes 有 skill_manage，OpenHands 有 /skill-creator）。页面上的 OpenHands 建议是："不要从零写技能，在 agent 刚完成一个你想重复的工作流之后立刻创建，然后测几次"。
Slides 第 15 页给出一个具体案例：OpenHands 在自己的仓库里对每个 PR 跑 python-review 类技能，然后：
- 用模型判断"人类保留了多少建议"来评估；
- 让模型分析失败模式，识别出"忽略仓库约定（约 15%）"和"批准了含严重缺陷的 PR（约 10%）"；
- 据此更新
SKILL.md：发现严重问题时一律要求改动，发建议前先对照仓库。
这一部分的"数值"来自 Slides 第 15 页原文，属于课程材料中的例子，而非本讲新做的实验。
七、从"人工写"转向"自动归纳"：记忆的四条基础线索
进入归纳技能之前，讲者先交代了几类外部记忆的基础工作：
- MemGPT（Slides 第 17 页）：把读写记忆变成 agent 自己发起的函数调用，把"工作上下文"和"外部事实存储"分层。
- Mem0（Slides 第 18 页）：处理记忆冲突，操作分四类——ADD / UPDATE / DELETE / NOOP（无相关就丢弃）。
- Reflexion（Slides 第 19 页）：把"对轨迹的自然语言反馈"作为下一次尝试的条件。页面上的例子是：第一次回答"Battle of White Plains"被判错，反馈说"问题问的是一系列战役，我只给出了一场"，第二次改为"The New York and New Jersey campaign"即正确。
- 相似轨迹条件化（Slides 第 20 页）：把整条轨迹（成功和失败都算）存进经验池，用文本 embedding 检索出与新任务最相似的几条，作为少样本示例。
讲者在 37:18–38:47 现场引导学生回忆上一讲的"Delta"方法，把"文本级事实更新"与"线性注意力中的键值更新"类比对——【学习者解释】这个类比是"记忆在符号层和向量层是同构问题"的一个直观入口。
八、把技能当作任务中的可复用块
Slides 第 22 页把"搜索商品"这个子任务抽象成：
# To find {query}, I will type it into the store's search box and submit.
fill('element145', {query})
click('element147')同一段动作在"报告价格区间"任务里又被复用，然后追加报告步骤。Slides 第 23 页把这条线追到更早的工作：Voyager（Minecraft 里逐层制造函数）、DreamCoder（压缩程序语料归纳函数）、Stitch、LAPS、LILO——它们共同构成"技能归纳/程序归纳"的前史。
Slides 第 24 页给出评估设定：任务按顺序到达，某个成功任务更新记忆，然后影响之后的任务，最终衡量"累计成功率"。Slides 第 25 页将这一生命周期抽象为三段：LEARN（判断→归纳→准入→入记忆）→ USE（检索→行动→产出）→ MAINTAIN（新增→修复→淘汰→回记忆）。
九、文本技能与代码技能：本讲的核心对比
9.1 文本+动作示例（AWM）
Slides 第 26 页展示 Agent Workflow Memory（AWM）的流程：NL 查询 + 动作轨迹 → 判定器判断最终状态是否正确 → 少量样本提示 LM 归纳 → 得到"任务描述 + 模板化动作序列"。Slides 第 27 页给出归纳提示词原文，其中值得注意的要求包括：每个工作流至少两步、不要生成相似或重叠的工作流、用描述性变量名表示非固定元素。
Slides 第 28 页展示跨任务记忆复合带来的收益：随着任务流推进，带子任务记忆的 agent 解决问题数量显著高于无记忆基线，并能执行更复杂的任务。
9.2 代码技能（ASI）
Slides 第 29 页做了一个清晰对比：
- 文本 + 示例：灵活（模型可解释、可适配当前页面），但每个底层动作仍需模型自己发出；
- 代码：可用作工具、可测试、可分层（技能调用技能）、高效（一次调用取代许多模型步骤），但僵化——"即使页面已经变了，它仍会照字面执行"。
Slides 第 30 页展示 ASI 的代码技能归纳：从同一条轨迹里生成 search_reviews 与 open_marketing_reviews 两个函数，并给出用函数重写的轨迹。Slides 第 31 页给出"代码技能可被单独测试"的例子：SkillWeaver 中归纳出的 identify_pill 函数定义了 color 参数却从未使用它，被 linter 抓到后补上实现。
9.3 效率与成功率的量化
Slides 第 32 页给出一个直观对比：同一任务在无记忆情况下 50 步触顶，代码技能下 4 步完成。Slides 第 33 页在五个 WebArena 站点上取平均：
- checkpoints reached：无记忆 41.3、文本技能 59.5、代码技能 80.2；
- steps per task：24.5 → 20.6 → 15.0。
（这些数值来自 Slides 第 33 页原文，属于课程对该论文的引用，本讲没有新增实验。）
9.4 泛化问题与 PolySkill
Slides 第 34 页给出失败案例：在一个站点归纳的 sort_listings（点击下拉菜单）到 Target 上失效，因为 Target 的排序展开的是一个侧边栏。Slides 第 35 页给出 PolySkill 的应对方案：抽象类 + 每个网站的具体实现——组合逻辑写一次，网站只实现底层浏览器动作。这同时也是"同一时间只有一小部分技能是活跃的"的机制。
Slides 第 36 页总结三种表示的强弱点：检索到的 episode 保真但难迁移；文本技能提供灵活指导但不提升效率；代码技能可执行、可组合、高效但可能脆弱。
【学习者解释】第 36 页最后一句话"多少行为留给模型，多少固定在产物里"，是整讲的元问题。文本技能把决定权留给模型但牺牲效率；代码技能把决定权固定下来但牺牲鲁棒性。PolySkill 之类的方案本质上是在这两端之间寻找一个可维护的中间层。
十、从失败中学习、过度检索与记忆合并
Slides 第 38 页给出 ReasoningBank 的一个案例：查询"列出 Sony 蓝牙耳机的完整名称和价格区间"返回 5,578 条结果、每页 12 条，agent 一路翻页直到触顶失败。归纳出的策略是"收紧查询"、"提高每页显示数"、"使用筛选器"。Slides 第 39 页给出一个反直觉的结论：失败经验对"策略归纳"很有帮助，但对"轨迹型/工作流型记忆"帮助较小——后者本质上是把"不该做什么"作为示例，效果有限。
Slides 第 40 页讨论"判定器准确率有多重要"：他们用真实标签加固定概率翻转来模拟不同质量的判定器，结论是性能依赖判定器质量，但在相当宽的准确率范围内，记忆仍然带来改进。
Slides 第 41 页给出过度检索的代价：检索更多经验反而会使性能下降，SkillsBench 对人类精选技能也观察到同样现象。原因可能是经验本身不相关，也可能是模型无法区分哪些上下文真正相关。
Slides 第 42 页给出记忆合并的机制：TroVE 会丢弃使用频率低于 ½·log₁₀(n) 次（n 是迄今见过的样本数）的函数；并提示"并非所有技能都帮忙——对某类任务有用的技能可能对另一类有害"。
十一、训练"技能归纳器"本身
Slides 第 43 页是本讲的前沿部分：把技能归纳过程直接放进 RL 训练。关键洞察是：任务 1 的奖励只说明任务 1 成功，并不能说明"保存这个技能"是否值得——只有后续任务才能证明。 所以把两个相关任务放进同一次 rollout（SAGE）或一条长轨迹（AgeMem）里训练。页面上给出的对照是（AppWorld test-normal，Qwen2.5-32B）：
- 仅结果奖励：55.4
- 两个任务都成功：56.6
- 成功复用技能：60.7
Slides 第 45 页给出一张"谁来做决策"的对照表：技能/策略已知时人工更划算，需要从交互中发现时 agent 更划算；正确性规范或高风险时人工更划算，结果可执行且测试便宜时 agent 更划算。
十二、课程与作业的对应
本讲明确与作业一对接：Slides 第 11 页的脚注写着"Assignment 1 · §1.4 Load skills"，讲者在 18:41 说"你们会在作业一中实现这个"。课程主页的 Assignment 1 · Harness 是相关材料入口；Assignment 2 · Eval 与本讲反复提到的"判定器质量""评估技能是否真的帮忙"直接相关，因为本讲多次强调需要评估来判断技能是否"真的让输出更好"。
如果想按官方顺序追这一讲，直接看 Lecture 4 slides 与 课程视频。
十三、几点需要核实的地方
以下几处，是逐字稿或幻灯片文本提取无法单独确认的，写作时按"待核"处理，不作猜测：
- 本讲中"约 3k token""33.9% → 50.5%""41.3 / 59.5 / 80.2""24.5 / 20.6 / 15.0""55.4 / 56.6 / 60.7"等数值均直接来自 Slides 第 11、13、33、43 页原文；如有疑问应对照官方 PDF 逐页核验。
- Slides 第 17、18、26、27、28、30、31、35、38、40、41、42、43 页的图、表、公式在纯文本提取中无法完整重建，本讲对应的结构描述请以官方 PDF 为准。
- SkillsBench 的任务数（87）与领域数（8）以 Slides 第 12 页为准；如有更详细的统计口径，请核对原论文。
- 逐字稿中出现的模型/框架名（如 Hermes、OpenHands、
agent skills标准）应结合其官方文档确认具体拼写与版本。
官方来源（给定资料中的真实链接）
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频（Lecture 4）：https://www.youtube.com/watch?v=6zigF2a-2Pw&list=PLSN0qpDfUvTM&index=4
- Lecture 4 slides：https://www.cmu-agents.com/slides/lecture-04-memory-and-skills.pdf
CMU 11-768 Fall 2026 · Lecture 4 · Agent Capabilities 3: Skills and Memory｜完整中文逐字稿
- 视频：官方视频
- 来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
- 说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。
正文
[00:01.520] 好，欢迎大家回来。>> 那么，嗯，第一个作业已经发布了，嗯，正如 Graham 所说，我们为选修这门课的同学提供了学分，让你们可以去做这个作业。大家有什么流程上的问题吗？很好。好的。那么我们在上一讲中谈到了智能体如何，嗯，选择在其上下文窗口中保留什么内容，作为一种在任务中 存储信息的方式，并利用这些信息，嗯，来判断什么对于在任务中做得更好是重要的。所以今天很大一部分内容将是关于智能体如何超越上下文窗口，在外部存储中跨任务存储信息，嗯，这些信息将对它们在未来提升任务表现有所帮助。这将是技能归纳。嗯， 然后，嗯，我们会基于过去在智能体记忆系统方面所做的一些工作进行展开。嗯，并展示如何将这些应用于智能体。但关于技能还有另一种视角。技能是人们为智能体编写指令的一种方式，可以编码类似于最佳实践或你关于任务应该如何完成的知识。而这些 实际上将是互补的。嗯，所以我们在这节课中只会稍微涉及一点，但嗯，你可以想象一个系统，嗯，它有点像有人类在环路中，嗯，依赖智能体根据它所拥有的经验提出一些潜在的最佳实践，然后由人来编辑这些内容，来回迭代，或者反过来，比如你知道，让一个技能和 由人编写的指令，由智能体根据经验来改进。所以是的，有很多不同的视角，但希望到讲座结束时，你能够看到它们如何能够融合在一起。那么让我们从一些动机开始，关于为什么你可能想要从一个任务到另一个任务记住一些东西。假设我们有一个智能体
[02:09.039] 它在一个网站比如购物网站上执行任务，你可能给它一个任务，比如把索尼耳机加到我的愿望清单。所以当我们讲到 GUI 智能体时，我们会深入细节，你知道，关于表示空间、动作空间，这样一个系统是如何工作的。但在高层面上，它会产出像嗯 导航到商店网站、找到它的搜索框、在那个搜索框中输入一些文本，然后点击搜索这样的动作。这就是搜索耳机的部分，对吧？然后我们会用它们做一些事情。我们会找到匹配用户请求的那些，并把它们加入我们的愿望清单。
[02:51.840] 我们可能在未来执行另一个任务，比如查找无线键盘的价格范围。如果我们在同一个网站上做这件事，很多动作将会被复用，对吧？我们会去商店，我们会找到搜索框，我们会以类似的方式搜索不同的产品，然后我们会用它做一些不同的事情，比如你知道，获取价格 并将这些报告给用户。但如果智能体很难弄清楚如何做这个搜索，一旦它知道怎么做了，它就应该能够在未来复用这些知识，以便在未来任务上更有效率。所以很多结构是重复出现的，今天的内容将是关于我们如何表示这种共享结构。例如，最好只是嗯保留 智能体在执行这个共享子任务时产生的原始动作，还是最好尝试将其抽象为例如一个代码函数，可以作为黑盒来运行，或者也许可以被测试，甚至像分层地调用其他函数。
[03:57.120] 我们还会感兴趣的是，智能体如何确定一个任务的哪些部分应该在未来被复用。嗯，正如我们所知，任何时候我们有一个从经验中构建起来的记忆，就会有一些问题，关于嗯，我们如何增长这个记忆？我们是一直在往里面添加吗？如果我们得到一些冲突的信息，或者 我们找到了做某件事的更好方式，会发生什么？我们可能想对它进行编辑。如果某些东西不再适用，我们可能想删除它们。所以，我们如何嗯管理这些技能以及这些学到的子任务表示的生命周期呢？[哼声] 那么，我想说有三种通用的方式可以让智能体进行更新。所以，第一种我们在上一
[04:47.759] 讲中已经讲过了。它就是嗯你在模型上下文窗口中维护的内容。所以这是嗯，你知道，标准的 React 风格的方式。它是你采取过的过去的观察和动作、从用户那里得到的消息、来自环境的反馈的历史，你需要，你需要选择在上下文窗口的限制 以及模型使用它的能力之下，保留什么内容。所以嗯我们谈了很多关于压缩以及管理上下文窗口中内容的方法。今天我们将重点关注将内容存储在上下文窗口之外。所以即使有了我们现在拥有的像一百万个 token 这样非常长的上下文窗口，嗯，用于嗯多个，用于像用智能体执行多个
[05:34.560] 任务，嗯，你也会超出那个上下文窗口，而你希望从过去的任务中记住一些东西来帮助你在未来的任务中表现更好。所以如果我们将内容存储在窗口之外，嗯，那看起来会像是某种数据库，嗯，需要在某种程度上进行管理，嗯，或者嗯像是智能体可以访问的文件系统。嗯，这节课将是关于 将内容放入这些类型的存储中，嗯，使用它们并更新它们的方法。然后最后一种方式是实际上对模型的权重进行更新。对吧？所以这看起来更像是标准的监督微调或强化学习。我们将在后面几节课中讲到这些。但每一种都有
[06:21.600] 各自的优缺点。所以嗯，我认为使用外部产物的一个主要优点是它们可以跨模型移植，对吧？所以如果你嗯有一个关于搜索产品方式的描述，嗯，那可以被智能体框架中多个不同模型中的任何一个使用。而如果你实际上是在更新模型的权重，那只是 针对那个特定模型的更新，它不会迁移到其他模型。所以外部产物也更容易被人检查。嗯，所以你可以写下最佳实践或在网站上如何搜索。智能体可以使用这些。智能体也可以提出，你知道，关于这些的某种文本，你可以查看它并对它进行修改。所以这更容易
[07:11.199] 有点像在人和系统之间有一个接口，如果你的更新方法、如果你记忆的方法是可解释的。关于这个有什么问题吗？有多少人在与编码智能体的交互中使用过技能？比如你们有多少人——好的，所以大多数人都有。你们中有多少人是自己编写技能的？ 好的，所以大概是使用技能的人中有一半。你们中有多少人让智能体为你编写过技能？好的，所以实际上很多人有。是的。有哪些事情——你们让智能体编写技能的经验是什么？你有多经常能够直接开箱即用？智能体构建出来的东西 有人有特别失败的案例或特别成功的案例吗？所以为了录制再重复一下，有时候模型会在技能里写太多东西，它们会过度推断应该做什么，然后当这个被交给另一个模型时，它会逐字遵循，做比你实际想要的更多的事情。 >> 其他人还有别的例子吗？>> 不错。哦是的，请讲。我的意思是，当你在比如尝试处理某个特定系统的生产问题时，如果你给它三四个支持案例，你告诉它看看过去人们是如何遇到这个问题的，为自己创建一个技能，其他智能体也可以用同样的方式使用 >> 这非常 >> 哦不错 >> 好的不错，是的，所以指示它为另一个智能体编写指令，这就足够好了，这有点像模拟智能体会做什么。是的，这很有趣。我有时发现模型在模拟心理理论方面相当差，就是你知道的另一个人或智能体在
[09:10.160] 查看指令时会有的那种心理理论，但当你提示它们去做这件事时，它们会好很多。不错。好的。是的。所以我们可能想为智能体保留几种不同类型的经验。第一种是情节，这就是你从环境中、从用户那里得到的观察序列以及所采取的行动，这 有点像最高保真度，对吧，你只是保留所有东西，你有那个特定环境的所有细节以及用户给出的特定指令，这也将是最昂贵的，对吧，它需要最多的token来表示，而且你可能不需要所有这些，对吧？就像当我们做监督 学习时，我们不想保留每一个细节，我们想抽象出重要的事情以便在未来泛化。所以我们也可以保留一些事实，比如关于环境的一些知识或关于用户的知识，在记忆系统的早期工作中，很多都集中在保留事实上。比如用户与助手交互，可能给出一些个人 信息，比如他们多大年纪或住在哪里。系统可以将这些保留在记忆中，然后使用这个事实来改善未来的交互。如果你用过比如Anthropic或ChatGPT的记忆系统，它们基本上就是在做这个，对吧？你可以进入设置，你可以查看它们关于你的记忆，那里有一个文本表示
[10:50.399] 来自一个模型，它从你过去的交互中产生了那些事实。看起来有点像压缩。它由LLM生成。它被存储在外部的东西里。然后最后一个更特定于智能体。所以它是关于你如何以某种从任务细节中抽象出来的方式来完成任务的某部分。 比如在我们之前看的例子中，你如何在特定网站上搜索产品，而不必保留搜索的细节，你知道，你搜索的特定产品？所以关于如何表示技能有很多选择。它只是文本吗？它是你实际采取的行动吗？或者它甚至可以是代码吗？
[11:35.680] 当然，技能只有在你将不止一次地做同样的子任务时才有用。所以我们可以思考记忆和技能以及它们之间的交集。记忆只是我们从智能体交互中保存的东西。例如，你之前在某个网站上购买的产品是什么？技能是关于如何在环境中行动的可重用 知识。它可以用多种方式之一来表示。比如，如果你要发布一个应用程序，把它推到生产环境，这是你在做之前需要过的一个清单。所以，技能可以由人来编写，就像这个例子中一样。但我们也可以有记忆和 技能的组合。所以我们可以从智能体过去的经验中归纳出技能并存储那些，然后用那些来希望在未来做得更好。所以如果智能体过去成功搜索了一个产品，如果它成功了，如果我们认为它成功了，那么我们就能提取出看起来像技能的东西来
[12:45.680] 在未来指导智能体。我们会在本讲座的剩余部分更具体地了解这是什么样子的。所以我们可以把技能看作指令，技能作为指令可能由人编写，用于编码他们自己的偏好，或者可能是组织政策，比如这是我们在某个特定公司编写和测试代码的 方式。人类编写的技能我们知道它们来自哪里。我们可能对质量有更多控制，但编写和维护它们需要很多时间。如果我们在学习技能，那么智能体在产生它们。这很有帮助，因为我们可以以自动化的方式来做这件事。也许我们可以有一个循环，我们归纳出技能。 我们尝试使用技能来重做同样的任务。我们看看在某个指标上事情是否变好了。但它也可能是潜在有噪声的，有——特别是如果你在为其他智能体使用技能，就像我们在大家给出的那些例子中谈到的，它也可能有意外后果。
[13:54.800] 好的，所以让我们先谈谈人类编写的技能，以及这些的标准和它们在当前智能体中的使用方式。可能很多听起来像你们中的很多人已经有这方面的经验了。所以也许你一直在类似这样的例子的背景下使用这些。 呃，Open Hands 有一篇很好的博客文章，是关于使用技能的指导。嗯，对于你们中还没有使用过技能的人，或者如果你想知道一些关于如何创建和使用技能的最佳实践的建议。但他们有一个很有启发性的例子，比如说你发现自己总是给编码 代理类似的指令。嗯，所以你想要你想要检查已经写好的代码，使用几种不同 linter 中的一种。你想给出类型提示。你想使用特定类型的文档字符串。你想对所有函数应用 pyest。所以如果你总是这样做，你可以总是把这些指导输入到系统中。嗯，但是你知道，一种自然且更快的 做法就是直接把指导存储为一个文本文件，然后让代理把那个指导当作指令来使用。所以当你想要按照某种固定规范反复做某事时，就应该使用技能。代理会根据上下文来决定如何解释这个规范，但嗯总体指令是一样的。
[15:31.680] 而目前在技能标准中，呃，实现这个的方式叫做 agent skills，就是把它作为代理能够读取的一组文件。这里有一个例子。所以如果我们为 Python 审查构建一个技能，嗯，我们会有一个像这样结构的文件夹。技能有这种标准结构，它们会 有一个 skill.md 文件，而且你必须要有它。Skill.md 嗯是一个 markdown 文件，开头是一些 YAML 元数据。所以你需要给技能起一个名字，还需要一个简短描述，呃，用来给语言模型提供关于何时应该应用该技能的指导。我们会在再过一两张幻灯片后看到模型如何访问这些 YAML 头部中的信息。然后嗯在头部之后，你会有一个更详细的描述，说明嗯该技能应如何应用，比如说会列出嗯这里是你应该应用的所有 linter，这里是如何进行测试等等。所以你必须要有 skill.md。你还有一些其他嗯文件夹，它们 是可选的，但可以包含一些额外资源，供代理在执行技能时使用。比如 scripts，里面有可执行代码。例如，如果你有一个会运行 linter 的 Python 脚本，你可以把它放在那里，然后在这个 skill.md 文件中描述它。还有 references，它像是补充文档，也许 你不需要每次使用技能时都用到它。嗯，但你可能会在技能文件中引用那个文档。代理可以在需要时从 references 文件夹加载它。然后 assets 基本上就是，你知道，额外的模板或资源，比如如果你在做某件涉及 Ginja 的事情，并且你需要
[17:28.319] 代理有条件地加载它。所以这可能会是很多信息，对吧？我们嗯不想用模型可能使用的所有技能来填满它的上下文窗口。嗯所以有一种机制叫渐进式披露，它会按需只向代理展示技能中相关的部分。所以第一层是嗯这个 YAML 文件中的元数据。呃，也就是名称和描述，如果技能被加载、如果模型可以使用它，那么它总是会在系统提示中展示给模型。所以你需要在这个嗯描述中保持简洁。嗯这样如果你有很多技能，就不会把内容填得太满。 还有第二层，就是 markdown 文件中的其余文本，嗯如果模型判断该技能相关，并决定通过工具调用读取文件其余部分，那么它会被有条件地加载。我们马上会看到一个例子。然后下一层大概就是这里的所有资源，它们也会 通过工具调用被加载。所以嗯你需要在 skill.md 文件中有足够的信息，让模型知道在执行技能时应该去这些额外文件夹中查找更多资源。关于这个有什么问题吗？你们会在作业一中实现这个，或者你们中有些人可能已经在开始实现了。
[19:05.840] >> 是的。那么高的话，如果那占用了太多，会不会影响性能？ >> 是的。所以如果你有很多技能，即使技能元数据相当短，它也可能影响性能吗？是的，它肯定会影响。嗯所以你要让技能数量保持相对较少，也许只是有条件地加载它们。 嗯，在一个最先进的代理中，Hermes agent，我想我不确定技能的数量，但系统提示中技能描述的总长度大约是 3,000 个 token。嗯有一些论文我们过一会儿会看到，它们表明如果你有太多技能，嗯它会让性能变差。比如加入无关技能 可能会降低性能，嗯因为你知道，上下文变长了。模型可能会被实际上无关的东西分散注意力。所以是的，你肯定需要小心技能的数量，但嗯我认为拥有像 10,000 或 1,000 个技能会相当罕见。一般来说，它是，你知道，更小的，而且你可能会按需加载技能，比如针对你正在执行的 软件任务。 >> 是的。 >> 那些触发器的敏感度如何？ >> 是的，这是个很好的问题。触发器的敏感度如何？所以触发器会是由模型生成的工具调用。因此它们是否被调用，完全取决于模型。嗯，你知道，模型基于过去上下文以及它对技能的了解来预测 它是否应该被触发，这取决于模型的能力。所以你知道，这取决于模型，取决于你写的描述，嗯这就是为什么我们需要为此做评估。嗯我们将会看的一篇论文朝这个方向迈出了一步。
[20:59.919] >> 是的。像一个标准化名称。 >> 是的。呃 skill.md 是一个标准化名称。所以这个标准叫做嗯，我想是 agent skills 标准，嗯模型、代理被训练来使用它，代理框架被训练来加载它。嗯所以是的，你嗯如果你想使用另一种格式，好吧，首先把它放在你知道尽可能接近这个 如你所愿，但你可以做，你知道，如果你有特定的，如果你有特定的，嗯，你想要的东西，比如在 references 文件夹里，你可以在 skill.md 中描述它们，就像所有这些都只是给语言模型的指令，所以它也可以对其他配置具有鲁棒性，特别是如果你描述了它们的话。
[21:50.799] >> 是的。为什么是 >> 是的，为什么是 markdown？我认为这是机器可读性和人类可读性之间的平衡。>> 是的，这是个很好的问题。有没有比 skill.md 更好的格式？是的。你有什么想法吗？我觉得，嗯，所以呃，我的意思是它就像线性的，对吧，嗯，你可以有超链接，这很好，你也可以嵌入图片，嗯 >> 是的，嗯，但是呃，就像它很难 你知道，表示动态的东西，所以也许你想向模型说明，嗯，数据可以被可视化的不同方式，嗯，你知道，也许用像 JavaScript 交互式 JavaScript 应用程序之类的东西会更好。嗯，是的，你可以，你知道，把它作为某种代码文件放在这里被执行。嗯，但是也许像 在 markdown 中被引用。嗯，所以我觉得 markdown 是某种最好的尝试，拥有某种通用的、可读的东西，但然后你会依赖语言模型来解释其他格式。>> 我认为在呃 Daniel 将要谈论的科学文献中，基本上有两种方式。有文本技能和程序化技能
[23:18.960] 文本技能出现在 markdown 中，程序化技能出现在脚本中。所以如果你考虑这些的话，它的范围是相当广的。>> 是的。酷。>> 是的。>> 有没有人做，为了不必把所有的东西都放在上下文中？例如，如果你在做一个修改云基础设施的 agent，你在某个大银行工作，AWS 里有十亿 个东西，能够只是让模型修改某些东西会很好，但然后因为它是十亿个东西，你不能只是为每个小 component 准备十亿个工具可用。是的。>> 所以看起来某些东西会这样做。是的，肯定在一些研究论文中，所以问题是像人们会从大型技能集合中检索吗。嗯，是的 肯定在我们将要看的一些研究论文中，嗯，他们会这样做，嗯，要么使用你知道的只是像基于文本 embedding 的检索，要么做一些更像是领域条件化的东西，比如我们知道这些技能适用于这个站点，所以当我们与那个站点交互时，我们会加载那些技能。是的，我想象生产系统也这样做，但是嗯，是的，我
[24:30.799] 手头没有任何参考资料。酷。好的，所以我们将呃只是呃快速地向你们展示渐进式披露是如何工作的。嗯，我们首先需要给模型访问这个 level one，对吧？让它知道哪些技能是可用的。你会有一个看起来像这样的 system prompt。这只是取自 Hermes agent，它是一个有点像标准的开源嗯 agent，能够使用和诱导技能，你会嗯描述什么是技能。呃，你会告诉模型关于这个 skill view 工具，它可以用它来读取 markdown 文件的其余部分，然后你会列出你拥有的所有技能。
[25:14.080] 所以这里只有两个技能。嗯，有一个代码质量技能和一个工作流技能。那只是那些技能文件的 YAML metadata。所以，嗯，模型能够使用 tool calls 来，你知道，展开技能的其余部分。这是它可能看起来的样子。所以，它在 system prompt 中总是有技能的索引，但它可以选择进行 tool call 来打开 Python review 技能。嗯，如果那看起来与你知道的用户要求对某些 Python 代码进行 review 相关的话，然后它会像嗯你知道的获取那个 markdown 文件，将它作为一条消息插入到系统中，作为那个 skill view 工具的结果的 observation。嗯，系统可能选择从 scripts 目录运行某些东西，嗯，然后它会得到运行那个 脚本的结果作为 observation，把它放进窗口中，嗯，并且可能选择你知道的像查看那个技能包中的额外文件。好的。所以是的，我们这些我们有的问题，像你知道的额外的技能会不会有害，嗯，技能是否在正确的地方被触发。呃，我们通过构建 evaluations 来回答像这样的问题，嗯，这方面的
[26:33.600] 工作有点嗯新。它是某种正在进行的，但是嗯这个领域一个非常棒的首次尝试叫做 skills bench，他们做的是他们嗯从嗯网上的各种来源收集一堆技能。他们实际上得到了像 200 万个技能。嗯，他们也会让人写看起来可能从技能中受益的任务。呃，所以他们会像得到 400 个任务，嗯，那些任务会独立于技能被写出来，但他们会让任务标注者从这个技能池中选择看起来可能适用于这些任务的技能。所以你会得到某种像一小部分精心策划的技能。嗯，然后你会把这些技能提供给一个 agent，当它试图解决任务时，他们会在一像嗯你 知道的四种不同的 agent frameworks 嗯上用，我认为像 18 个模型之类的来评估这个，他们会只是看像拥有技能是否嗯在给定 agent framework 和模型组合的所有任务上，相比没有技能提高了性能，对吧，所以这会评估你知道的，它会隐式地评估技能质量，但会被设计来评估一个模型和 framework 是否能够你知道的适当地使用技能，而且他们在试图选择技能。所以他们有像一个过滤器，嗯，它检查对于一小部分模型，当你给模型访问技能时，任务上的性能确实提高了，然后你在看这对于更广泛的 frameworks 和模型集合是否也成立。
[28:15.039] 是的。这个总体设置说得通吗？酷。所以平均而言，你知道，有了这个精心策划的嗯技能集合，呃，这些任务上的平均 pass rates，使用像人工编写的验证标准，确实增加了。所以你可以看到像实心条到阴影条，跨越一堆不同的模型嗯和 frameworks。嗯，但实际上呃技能像 需要更少技能来解决的任务，从拥有可用技能中受益更多，比那些需要更多技能的任务。嗯，所以这部分是难度的问题，但也部分是模型可能因为拥有更多可用技能而分心。你可以想象，在像大得多的技能数量下它会变得糟糕得多。嗯，我们会看到更多证据 这部分内容我们稍后会在讲座中讲到。而实际上，对于特定类型的模型，技能在很多任务上仍然会带来负面影响。嗯，所以模型可能会不恰当地使用技能，这实际上可能导致它以不同于使用者所指定的方式去执行任务。所以这在一定程度上说明了检查你编写的技能是否真正
[29:31.440] 让你的输出变得更好是很重要的。所以技能可以来自，你知道的，很多不同的来源。它们可以与harness捆绑在一起。嗯，它们可以由人来编写。嗯，你们中很多人已经这样做过了，听起来是这样的，然后你，你知道的，以你所使用的框架所指定的方式安装它，但它们也可以由 agent来创建。而且，嗯，Hermes有一个技能管理工具。嗯，OpenHands有一个技能创建器（skill creator）命令，嗯，可以帮助你走完创建技能的流程。当你在创建技能时，我认为非常重要的是将这个技能建立在你先前的某些交互之上，而不是试图仅仅先验地去做。嗯，而实际上
[30:19.520] 在skills bench论文中有一些结果表明，当模型被要求仅根据任务描述来自行创建技能时，它们在这方面非常差。而且这实际上会让性能变得更糟。所以，作为一个人，你可能会做得更好，因为你大致知道你想要它被完成的方式，但一般来说最好将它建立在，你知道的，过去的 你所做过的事情之上，因为当你在做一个任务时，它会揭示一些关于你希望它如何被完成的事情，而这些是你事先可能不知道的。所以有一个很好的案例研究，嗯，来自OpenHands博客上的一篇文章，关于你如何，比如你有一个你已经创建的代码审查技能。你如何检查并看看它是否有效，然后在必要时改进它。
[31:01.360] 所以这聚焦于比如审查pull request，你可以做的是，嗯，在agent生成审查之后，然后一个人会去浏览这个pull request，查看每一条审查评论，并决定是否处理它们。之后你可以应用一个基于模型的评判器（judge），并统计其中有多少实际上 被这个人采用了，这会让你对pull request的质量有所了解。你可以，嗯，在大量由agent使用这个技能创建并审查的pull request上做这件事。而且，嗯，如果你在，你知道的，数百或数千个例子上做这件事，你会得到大量关于什么有效、什么无效的证据，你可以用一个推理模型来总结所有 这些，嗯，并用它来，比如，识别出生成忽视仓库约定的代码是没有帮助的。人们确实相当多地采纳了那些修改。然后你可以让模型提出对技能文件的一些更新，使系统生成的审查更
[32:14.159] 有可能被人采纳。好。在我们继续之前有什么问题吗？好的。那么接下来我们将讨论我们如何让模型自动记住它们过去经验中的事情，然后我们会以此为基础，嗯，来讨论从经验中归纳出技能。
[32:40.480] 所以这是一篇，这是一篇较早的论文。这是2023年的，嗯，但它相当有影响力。嗯，它是，呃，最早的几篇论文之一，它叫做MEMGPT。它是最早的几篇论文之一，呃[哼鼻]引入了一种你可以在语言模型的上下文窗口之外管理记忆的方式。所以这真的是一篇受系统设计启发的论文。我想我们几节课前讲过 关于，你知道的，拥有像记忆层级这样的东西，你知道的，某种访问速度快的东西，对语言模型来说就是上下文窗口，然后某种更大但更慢的东西，在这种情境下就是外部的事实存储，而在这篇论文写就的几年前，上下文窗口还要更短，比如 你知道的，8000个token，嗯，所以这种方法即使仅仅是为了与助手系统交互也是非常有根据的。嗯，如今你知道的，我们的上下文窗口更大了，但你想要存储在其中的东西也大得多。所以这种方法对于agent来说也是相当有根据的。
[33:48.399] 但这里的基本思想是，你会有，嗯，你会有，你知道的，系统指令（system instructions），它们不会被更新。你会有一个工作上下文（working context），嗯，它会，呃，从外部记忆中拉取东西，嗯，同时你也会把东西存储到这个外部记忆中，而那些读写将由来自模型的工具调用来控制， 嗯，你还会，你知道的，为系统接收到的实际消息设置一个单独的存储，嗯，但那将以与这个，呃，这个事实记忆非常相似的方式来控制。所以，这实际上看起来是这样的，抱歉这里因为亮度有点难以看清，但比如你，嗯，你可能会与 模型有一次交互，用户，比如提到了six flags，嗯，系统可以生成一个工具调用，在这个存储中搜索它过去交互中关于six flags的提及。它会提取那些内容，并把它们放进这个，比如，工作上下文中。嗯，然后模型，嗯，这个工作上下文被存储在语言模型的上下文窗口内。所以模型将 能够在输出响应时使用那些内容。但重要的是能够做的不仅仅是，比如，添加到记忆中，因为例如，嗯，也许你在和系统谈论你的工作。你说我在X工作，但如果你后来换了工作，你希望系统能够，比如，使用最新的信息。
[35:26.000] 嗯，这篇叫做Me Zero的论文，呃，引入了一种让记忆也能被更新的方式。嗯，所以它的工作方式是，你会有，比如，消息进来，嗯，每次你收到一条消息，你会应用一个语言模型从中提取一些记忆。嗯，一些，比如，你可能想要存储或更新的潜在事实，你会有一个外部的，呃，数据库存储，嗯， 你会尝试检查并看看我在那个数据库存储中是否有任何与这些，嗯，我从这些消息中提取出的事实相关的东西。所以你会检索前K个，为此你会使用，比如，一个基于嵌入的检索器模型，就像，你知道的，我们刚才稍微谈到过的那些。嗯，你会提取出类似的事实，比如，嗯，six flags，或者比如，嗯，我在X工作，以及 你会应用一个语言模型来检查从这些最新消息中提取出的所有记忆，嗯，它们是否存在于这些相似的检索到的记忆中？如果它们不存在，那么我们可能就应该把它们添加进去。嗯，如果它们是对已有内容的更新，我们就应该用新内容替换掉为该项存储的内容。如果它 否定了已经存在的内容，比如，你说“我在X工作”，但后来你说“我被裁员了”，你可能就想直接删除这个事实，而不是嗯去更新它。嗯，或者如果它完全不相关，你就会直接忽略它。
[37:01.280] 嗯，我会让你们去看论文里的细节，了解这方面的详情，嗯，因为细节也会随着智能体系统而变化，但我认为，如果你有冲突，能够更新知识也是有用的，这个想法可能相当有用。
[37:18.880] 那么，嗯，有人还记得上下文管理那节课里某个与此有相似目标、能够更新或删除记忆的东西吗？是的。呃，哈希。>> 是的。是的。基本上，有人记得那篇具体的论文吗？是的。是的，就是哈希意义上的，因为在那里我们有键值和查询，对吧？嗯，我们当时存储的是值，也就是向量。嗯 而这里我们存储的是文本，嗯，作为事实。是的。所以Delta那篇论文实际上就做了这个，对吧？嗯，线性注意力只能把值追加到你的，嗯，实际上就是你的记忆中，但Delta嗯呃实际上会检索为特定查询存储的东西，然后改变它们，改变为它们存储的向量 而这有助于提升性能。所以这算是它的一种基于文本的类比，有着相似的目标。是的，你也可以想想，嗯，对于第三种学习方式，也就是更新模型权重，能够遗忘东西在那里可能也是有用的。嗯，尤其是如果你有某种冲突的知识，而且人们开始开发
[38:47.440] 一些方法来做这件事，比如遗忘学习，嗯，就是一种方式。好了，那么这些就是一些关于记忆的基础性论文，嗯，它们是为智能体助手开发的。嗯，对轨迹的反馈可能非常有用。所以这篇叫反思的论文嗯非常有影响力，它真的是一种嗯，真的是一种相当简单但有效的嗯，让智能体利用反馈并
[39:20.480] 可能做得更好的方法。所以它绝对值得了解。如果你在做一个可能希望智能体能够从其过去经验中受益的项目，这是一个很好的实现选择。嗯，它的工作方式是，我们试图在某个特定任务上经过多次尝试后提升智能体的表现。那么，假设你让智能体去嗯， 你知道，在一个网站上搜索一个产品。它做了一次尝试。嗯，它说它完成了，而你能够获得一些关于它是否可能正确的反馈。所以这可能来自比如一个人查看它。也可能来自应用一个模型作为评判者。嗯，但有了那个反馈，你会希望模型再试一次，并试着做得更好。 而这篇论文所做的是表明，嗯，对这次尝试的自然语言反馈可以是非常有帮助的东西，让模型在第二次尝试中以它为条件。所以比如如果它试图回答这个关于某一场特定系列战役是什么的问题，它可能只提到某一场特定的战役，然后给出的反馈嗯，由一个语言模型基于 知道这是不正确的，并查看问题和答案而生成，可能是嗯，你只给出了单场战役，而不是一个系列。所以你可以想象，这对模型来说会是非常有帮助的反馈，让它在嗯对这项任务做另一次尝试时可以以此为条件，并且它能够答对。嗯，这篇论文表明，嗯，以这种对 哪里出错了的自然语言描述为条件可以非常有用。所以我们也可以把这个推广到嗯智能体在做不同任务时不断改进。对吧？这是在同一个任务上改进。但你可以想象，嗯，反馈可能普遍有助于提升在相似任务上的表现，只要反馈嗯足够通用。
[41:15.280] 而我们可以嗯让智能体改进的最后一种方式，或者你知道，帮助智能体改进的最后一个基础性东西，嗯，就是记住轨迹，嗯，也就是它们过去做过的整个轨迹、整个情节，并利用它来塑造它们在未来任务上的行为。
[41:34.240] 有一堆论文做了这个。我们在这张幻灯片底部有一些指向它们的链接。嗯，但所有论文中的关键点是，你会有某个训练任务的集合，那些任务可能是嗯一组示范，也可能是智能体在逐个执行任务时产生的。嗯，而你会把所有这些任务 汇集在一起。也许你还会额外地嗯生成一些文本来描述来自它们的反馈，有点像上一张幻灯片里的反思所做的那样。然后你会有那个经验池。然后当你接到一个新任务嗯进来时，你会从这个嗯轨迹集合以及嗯可能生成的关于它们的文本中检索，来找到与你正在执行的任务 相关的东西。你会把它放进模型的上下文窗口里，它一般会帮助你做得更好。嗯，所以这有点像是面向智能体的rag，对吧？嗯，呃，它相当有效。关于这个有什么问题吗？是的。>> 在这种情况下，你说的轨迹是什么意思，是像调用序列吗。
[42:42.800] >> 是的。是的。所以对于智能体来说，轨迹一般就是它从环境或用户那里得到的嗯，像消息序列。所以是观察结果嗯，它采取的行动，比如工具调用，嗯，呃，可能还有你知道它产生的思维链。所以一种理解方式基本上就是，嗯，就是所有的嗯，呃，消息序列嗯
[43:07.680] 在模型的上下文窗口里 >> 而你通过图像以不同方式检索它。你可以做这种常规的嵌入检索，或者比较过程序列并做一些集合比较。>> 是的。你可以做检索的不同方式有哪些？所以通常你想让模型，你给模型一个它应该执行的新任务的描述。所以你只有 那个文本描述，你会用嵌入模型把它嵌入，然后你会嗯基于其他任务的描述来检索，每个任务都有一个与之关联的轨迹，也许你还会把它约束成比如我们只检索我们正在处理的这个特定领域的任务。嗯，是的，好问题。关于这个还有别的问题吗
[43:57.359] 方法？一般来说，这有点像是一整类方法。嗯，有不同的方式来实现它。嗯，细节方面，比如你如何构建经验？你如何进行检索？嗯，但我只是想让你有一种感觉：如果我们有过去成功的轨迹，并且把这些作为示例展示给智能体，它就能帮助智能体做得更好。
[44:22.240] 好的。那么有了这些，我们就有了讨论实际归纳技能的基础。嗯，所以我们可以把技能看作是任务中可以在不同任务之间复用的部分。所以在一开始我们谈到了比如搜索一个产品，那可能看起来像是我们告诉智能体展示这个特定查询的结果，然后 智能体可能会产生这样一条思维链，然后它可能会产生这些实际与网页交互的特定动作，比如嗯呃点击由这个特定ID标识的这个元素，也就是搜索框，在里面输入一个查询，然后点击另一个元素，也就是实际用于搜索的按钮。 而这一块任务可能会在未来的任务中被复用。比如如果我说嗯这个特定商品的价格范围是多少，它可能也需要再次执行这些完全相同的步骤，以及之后的一些额外步骤。所以我们可以把技能看作是嗯任务中一个特定的可复用部分。
[45:38.640] 这是表示技能的好方法吗？有人能想到这有什么问题吗？>> 是的。只是 >> 是的，没错。是的。所以，比如如果页面上的元素ID发生变化，技能就会失效。而且你知道，这种情况可能发生的方式有很多。 比如，你知道，也许元素……也许是同一个页面，但东西只是编号不同了。你可以对动作空间做一些修改来处理这个问题。我想JY会在他的讲座中谈到这一点。但比如如果我们去一个不同的网站，它有不一样的搜索界面，嗯它就会失效，我们稍后会看到一些这样的例子。
[46:31.440] 是的。所以我们将像Graham说的那样，一般会比较文本和代码作为表示技能的方式。我想简要强调一些嗯非常酷的论文，嗯我认为它们为技能归纳、嗯从经验中学习技能奠定了基础。呃，一篇叫做嗯Voyager，嗯他们在Minecraft中控制一个智能体。嗯，他们使用呃代码作为一种 控制智能体的方式，其中代码函数嗯实现了诸如嗯合成某个特定物品或与僵尸战斗之类的事情。把这些子任务表示为函数，使得函数可以相互调用。这样当智能体嗯学习了简单事情的函数后，它就可以比如在其他函数中调用它们来组合这些函数，你就能执行越来越复杂的任务。 还有一条来自嗯程序归纳社区的研究路线嗯也在做这个。我认为一个非常好的代表是嗯呃叫做dreamcoder的工作以及建立在它之上的一些论文。在那里你嗯有点像是……你基本上有关于你正在构建的东西的语法，嗯比如对象，嗯然后你嗯比如有一些完整对象的示例，这些对象 是使用非常低级的函数构建的，你在其中搜索反复出现的模式，你找到可以抽象为函数的模式，你反复这样做，你有点像在压缩构建这些对象的程序。
[48:07.680] 然后还有一些后续工作嗯集成了比如大型语言模型、Python代码以及嗯自然语言描述。而那些论文中的一些相同想法对于使用语言模型与世界交互的智能体也很有用。我们评估归纳和使用技能的系统的一个典型方式是在线学习设置。所以 当你与嗯一个智能体交互时，你可能是让它连续执行一系列任务，其中一些任务彼此相似。你希望系统能够从与它的交互中学习，然后在未来能够做得更好，对吧？所以嗯这是一个在线学习设置，任务会一个一个地到来。所以 比如，你可能会说，把一副Sony蓝牙耳机加到我的愿望清单里。我们会在解决这些任务的过程中逐步构建技能记忆。所以嗯会有一个归纳步骤，我们将在幻灯片上或过一两张幻灯片谈到。那个归纳步骤会创建一个技能，比如搜索一个产品并把产品加到愿望清单，这些 对解决这个特定任务很有用。嗯你知道也许我们有另一个像这里这个任务一样的任务，我们可以在未来的任务中应用这些技能，比如这个任务也可能涉及搜索一个产品，希望我们在这个任务上会做得更好，因为我们有这个从之前任务中归纳出来的技能，以此类推。所以我们感兴趣的是， 如果我们能衡量每个任务上的成功，我们的成功率是不是在变得越来越好？随着我们从经验中学习并逐步构建嗯呃完成这些更简单任务的方式的表示，我们是否能够解决越来越复杂的任务？关于这个设置有什么问题吗？
[50:11.760] 好的，很好。那么我们将在讲座剩余部分看的论文考察的是这个嗯呃从经验中学习技能、拥有这个技能库的空间的不同部分，并提出问题：在我们嗯归纳出技能之后，我们应该怎么处理这些技能。所以一般会有一个学习步骤，我们会让这些嗯任务到来。 我们会尝试从中提取一个技能。会决定是否把这个技能嗯存储到我们随着时间逐步构建的记忆中。还有嗯实际使用这个技能记忆来潜在地提升我们正在执行的任务的性能的步骤。然后最后，我们可能想要，你知道，如果我们得到了太多技能，我们可能 想要尝试简化那些，比如删除那些没被使用的，或者如果它们太具体了，我们可能想要让它们更通用。嗯或者比如把多个东西整合并合并在一起。[清嗓子] 我们将更深入地看一篇论文，嗯我认为它相当能代表归纳技能的方式。呃它也是在CMU完成的，
[51:24.880] 是由Zora Wong领导的工作，她将在课程稍后某个时候就一些其他主题做一次客座讲座。嗯但这篇论文叫做agent workflow memory，嗯它的工作方式是我们把它应用于网页任务。呃这也是与Graham以及嗯我一起做的工作，嗯我们把它应用于网页任务，你有一个像这样的查询，你知道告诉我评论的数量 我们的商店到目前为止收到的，当智能体执行这个时，这些就是它将产生的实际工具调用。所以我们有这一系列工具调用，我们想尝试识别其中哪些部分将来可能可以复用，你们大概能猜到我们是怎么做的。我们让一个语言模型来看看这一系列工具调用，并且 识别出那些看起来在未来任务中可能有用的。然后我们会让模型输出我们所谓的工作流，它们有点像是基于文本表示的技能——文本和动作。一个工作流会有对子任务的描述，比如搜索某个特定物品。 然后它会有你用来完成这件事的动作序列。我们让模型把这个做得更通用一点，这样它就能适用于比如搜索其他词条。所以我们让它使用——创建一个模板，可以抽象出初始动作序列的某些部分。你们可能在想，这开始看起来 像代码了，对吧？确实如此。稍后我们也会把代码和这个做比较，展示代码在某些方面如何更好，但也会让事情变得稍微更脆弱一些。这里一个相当重要的部分是，我们会把一个评判模型应用到这条轨迹上，来决定——让评判者预测这条轨迹看起来是否正确。我们会用这个来
[53:28.400] 决定我们是否真的应该把归纳出的工作流加入记忆中。因为我们不想记住那些不正确的东西，因为那些将来可能会把模型引向错误的方向。>> 关于这个有什么问题吗？好的。那么我们让模型访问这些工作流集合的方式，就是把它们塞进 模型的上下文窗口里。所以这看起来很像你们知道的这个智能体技能标准。你有选择，是展示全部内容，还是只展示那种文本描述。你在这里怎么做可能取决于你有多少技能，以及你知道的模型的上下文长度。
[54:20.000] 那么为了节省时间，我会跳过这个归纳提示词，但我们确实发现，拥有这种关于过去子任务的记忆，可以在智能体在线学习时帮助提升它的表现。这里x轴上是来自这个基准测试的、按固定顺序排列的不同任务，涉及与地图工具的交互，这里的蓝线是 没有记忆的基线智能体，它大概收敛到这些任务上大约10%的准确率，这里的标签是智能体正确解决的任务。然后这条黑线加入了这种关于子任务的记忆，提升相当显著，而且它能够——你知道，两者之间的差距出现了，但它也能够 执行这些越来越复杂的任务，最终做到比如在这个位置附近找一家酒店、给我展示步行路线，而这些是基线智能体根本做不到的。所以这是在用这种文本和示例动作的混合来表示技能。
[55:33.599] 但我们也可以考虑使用代码，对吧？这两个都是某个搜索产品函数的实现。如果我们只用这个文本和示例，这会被放在模型的上下文窗口里。然后它在新的上下文中就需要产生这些工具调用，对吧？当它试图执行同一个子任务时。所以它是灵活的，比如 模型可以选择插入某个特定的搜索词来搜索不同的产品。如果网站变了，我们不再有这个特定的搜索框元素，那么智能体可以选择使用正确的那个。但这也意味着智能体需要自己实际产生所有这些工具调用。如果我们使用代码，智能体
[56:25.760] 实际上可以直接把这个函数作为一个工具来调用，对吧？如果它传入了正确的参数，并且如果函数是正确的，那么它就不需要做任何事。它不需要自己产生所有这些底层动作。工具会在环境中执行时直接完成这些。有人能想到把这样的函数用作技能表示的一些优缺点吗？ 有什么缺点吗？>> 它非常死板。>> 是的，它非常死板。所以比如模型将需要——调用这个工具的模型会精确地执行这三步，没有任何偏离的余地。是的，这是它主要的——这是它的一个主要缺点，但我们也得到了使用代码的很多优点，对吧，比如我们可以嵌套函数，它 可以是分层的，我们实际上也可以为那个特定函数生成测试用例，独立于它被执行所在的整个轨迹。我们还可以做比如重构之类的事情。所以Zora有另一篇论文，叫做归纳程序化技能或智能体技能归纳，它做的就是这些，而且它看起来和
[57:47.839] 我们用来归纳这些文本工作流的方法很相似。所以你有一条像这样的轨迹，带着这些工具调用，我们会让语言模型——我们就利用语言模型也擅长生成代码这一点，我们会提示模型生成函数，抽象出这条轨迹中的很多行为，而这些是真实的 例子。所以语言模型会写出这个搜索评论函数，以及它的文档字符串，然后你知道它在函数里把这些底层工具调用作为行。另一个是打开营销评论，但我们也会让它生成一个使用那些工具的、对原始轨迹的某种重写。关于代码的一个非常好的地方是，我们可以通过 执行它来测试它。所以不像之前那样把评判者应用到原始轨迹上，我们可以让语言模型。嗯，我们可以执行这个生成的抽象代码，然后把评判者应用到由此得到的最终结果上。如果那看起来是正确的，那就更像是技能——你归纳出的这些代码技能也是 正确的——的一个更强的指标。如果它们是正确的，那么我们就把它放进记忆里，它们就作为工具对语言模型可用。所以这是一种建立工具库的方式，模型能够在未来的例子上使用这个工具库，而且我们展示了它也你知道提升了表现。
[59:19.680] 关于这个有什么问题吗？是的。所以代码技能的一个好处是，它们允许单独对技能本身进行测试。有一篇不错的论文叫做 skillw weaver，就是做这个的。所以这大概是我刚刚展示给你的那个循环的一些伪代码，你有一个任务，你为它提出一组代码技能，然后你会尝试使用
[59:50.079] 那些技能来执行任务。你会得到一个 episode，你会对它应用一个奖励模型。如果成功了，那么你可以把所有这些归纳出来的技能放进你的记忆里。但如果没有成功，那么你可以对它做一些修订。技能可能不成功的方式有很多种，对吧？所以，我们能做的一件事就是，比如对一个归纳出来的技能应用一个代码 linter。那么， 这里是论文中的一个真实例子。有一个被归纳出来的技能叫做 identify pill。比如，用于和药品网站交互。而这个函数接受很多参数，比如丸药的品牌、颜色，但在 agent 写出的这个技能的最初版本中，它实际上并没有使用所有参数。所以它忽略了这个 颜色参数。所以你可以从一个 linter 那里得到关于这个技能的反馈，然后以一种反思式的方式，以它为条件，然后比如创建这个函数的一个更新版本并存储它，或者尝试执行它并存储它。你也可以使用来自这个奖励模型的反馈，并在你对技能做这些修订时以它为条件
[01:01:00.400] 。所以这些是代码的一些不错的好处，即使实际的代码函数本身更僵硬一些，这些好处也可能成为优势。另一个不错的好处是，因为这些被归纳出来的函数现在是工具调用，它真的可以减少你的 agent 需要与环境交互的次数。所以这里 有一个来自 Zora 的 ASI 论文的例子。我们有这样一个任务，比如在一个页面上更新几个地址，对于基础 agent，你必须做所有非常低层次的动作，比如浏览所有这些表单，输入地址中的特定内容。但如果我们有这个已经归纳出这些代码技能的模型，你可以，而这是来自这项工作的一个真实例子， 我们在其中归纳出这个 navigate to address settings 和 update address details 函数。我们实际上可以在三步内解决这个任务，我们假设这对这个网站很合适。我们可以直接调用这两个函数，然后完成任务。
[01:02:12.319] 而且这个 agent 实际上快得多，因为它不需要重新观察网站，然后生成工具调用，再重新观察新网站，生成新的工具调用。正如你可能会在 JY 讲的 guey 讲座中看到的那样，对这些模型来说，一个非常大的瓶颈实际上是它们必须与网站交互的次数，因为 这些图像会占用大量 token，而且在每个动作之前实际生成一段思维链也会占用大量 token。所有这些都必须被生成。所以效率在这里真的很重要。而在这项工作中，我们确实发现，我们做了一次受控比较，比较使用文本技能和代码技能。这是在一组确实具有 更多重复结构的任务上进行的。但我们发现，与没有记忆相比，文本技能有帮助，但代码技能帮助更大，不仅体现在系统的准确率方面，而且我们在效率上也获得了相当大的提升，也就是 agent 需要与页面交互的次数。
[01:03:18.960] 很好。所以，正如我们之前已经谈过的那样，技能的一个很大的潜在问题是它们可能不具备泛化性。这里是这种方法的一个失败案例，实际上也是这些代码技能的一个失败案例。我们试着看看技能能在多大程度上跨网站泛化。我们在一个网站上归纳出这些代码技能，然后我们试着把它们应用到一个真实网站上， 比如 Target 购物网站。所以这个在不同网站上被归纳出来的技能在那个网站上有效，但它假设你正在交互的这个特定元素有一个像这样的下拉菜单。而在 Target 网站上，它是这样一组单选按钮，所以它在那里失败了。而且你必须让模型能够实际上
[01:04:09.359] 根据它在与环境交互时从环境得到的反馈来编辑这个技能。但代码对此有一个解决方案，对吧？我们有抽象类，而它们可以以不同方式被实例化。有一篇论文叫做 poly skill，就是做这个的。所以他们使用代码作为表示，某种程度上建立在我们所做的这项 ASI 工作上。 但是，当模型第一次与一个新网站交互时，它会写一个抽象类，这个抽象类没有这些方法的实现，但可以有像搜索产品、加入购物车、结账这样的存根。然后它还会为它写一个网站特定的实现，比如这个给 Amazon 的。然后当它与另一个网站交互时，你 可以有一些元数据，或者你甚至可以直接让语言模型触发，表明这个特定的具体实现不再适用。然后你会写这个抽象类的另一个实例化。而且他们表明这也有帮助。这还很好，因为它给了你一种方式，让你可以拥有大量特定技能，但在给定时间只有其中一个子集 是活跃的、实际相关的。所以，这有点像说得通，对吧？这只是软件工程的最佳实践，也被应用到了 agent 上。好的。是的，我们已经谈过所有这些了。你们对每种表示类型的优缺点已经有很好的理解，但知道这一点是有帮助的：
[01:05:45.760] 技能标准允许你把两者结合起来，而且关于把这些表示结合起来的最佳方式，还有大量研究要做。就像说的，好的，所以我们会快速过几篇论文，关于这个技能生命周期中不同部分的论文。这是一个非常开放的研究课题，比如什么是最好的方式来归纳和
[01:06:08.240] 管理这些技能。你会拿到论文以了解更多细节。但如果你感兴趣，你也可以在讲座末尾或课程末尾的项目中探索这个。到目前为止我们谈论的一切都是从成功中学习，但从失败中学习也可能相当有帮助。比如说，模型 比如尝试执行这个任务，比如嗯你知道搜索蓝牙耳机。嗯但是对于这个特定的网站，它们的搜索词是以 or 而不是 and 连接的，所以你得到的是一个包含所有蓝牙或耳机或索尼的结果列表。这是来自 Google 的 reasoning bench 论文中的一个真实例子。所以你得到比如 5,000 个结果，然后智能体就崩溃了 在这个上面，它必须翻遍所有这 5,000 个结果。它确实这么做了，你知道，它失去了对任务是什么的追踪，然后失败了。但如果我们观察那个嗯失败的轨迹，以一种反思风格的方式，我们可以呃生成一些文本，给出如何在未来避免它的指导。比如说嗯我们应该给出一个更具体的搜索词，而不是这个更 通用的词，我们应该你知道让显示的项目数量更多，这样模型一次能在上下文窗口中获取更多内容。这篇论文嗯基本上使用了和呃智能体工作流记忆、智能体技能归纳相同的框架，但不是仅仅拥有这些正确轨迹的抽象，它生成了这种文本反馈嗯，这种反馈嗯也适用于
[01:07:42.880] 失败的轨迹，而且也是你知道的那种你自己写技能时会写的东西嗯如果你在写一个技能的话，他们发现这相当有帮助。所以这些是不同的嗯记忆类型。嗯所以 synapse 使用轨迹，比如完整轨迹，包含所有动作和观察、所有工具调用和消息。嗯这是智能体 工作流记忆，它使用这些稍微抽象化的轨迹部分。而这是他们的方法，使用这些策略的文本描述，嗯添加正例和负例，你知道添加正例和负例作为记忆的输入。嗯呃当你使用轨迹或嗯工作流时，并不会提升性能，但当你拥有那种 文本描述时，确实会大幅提升，这从我们之前看到的例子来看是合理的，因为你可以用一种通用的方式描述如何不失败，而不是记住那些失败并以它们为条件。另一个重要的考虑是，我们在决定是否存储轨迹嗯你知道是否将东西存入记忆，或者从例子中推断什么样的策略
[01:09:02.000] 这基于我们判断该例子是否成功。在大部分这类工作中，使用的是模型作为评判者，它查看轨迹的最终状态，查看你试图解决的任务，然后预测看起来我是否做对了。所以这里一个非常好的问题是，你知道那个评判者的质量 对算法性能有多大影响，因为我们知道评判者并不完美。嗯这篇论文 reasoning bank 对此做了控制。所以他们嗯用完美评判者运行了他们的方法。嗯他们还喜欢，所以他们使用了关于是否成功的某种真实标签知识。嗯但然后他们往里面加入了一些噪声，这样你就能控制嗯准确率， 模拟的评判该轨迹的准确率，性能确实会下降，但他们发现存在一个嗯甜点区域，在那里你确实能从记忆中获得改进嗯，而且嗯存在一个相当宽的准确率范围，在这个范围内使用评判者来归纳记忆是有帮助的，但这里肯定还有未来工作的空间。
[01:10:19.600] 好的。嗯然后是的，在最后几分钟里，嗯我想回到之前我们有一个问题，就是拥有超过必要数量的技能嗯是否会潜在损害性能？我们还有一个问题，比如嗯如果我们从一个集合中检索技能嗯那个检索器重要吗？比如那个检索器的质量重要吗？所以嗯在这篇 reasoning bank 论文中有一些很好的消融 实验，嗯控制了嗯你从你拥有的记忆中检索多少个经验，他们表明随着你拥有更多可用经验，性能确实会下降下降。嗯所以这可能是由于几个不同因素。所以嗯你知道也许你只是还没有在那么多任务上运行过你的智能体，也许 你的记忆中没有相关的东西。所以放入更多东西就注定是不相关的。那将是在那种设置下应用这种方法的一种基本限制。对吧？只有当你学习的东西和你应用的东西之间存在某些相似性时，你才能学习。但这也可能是模型本身的限制。 也许模型因为上下文窗口中有更多可用的东西而受到干扰。这有可能通过训练模型来修复，对吧？让模型更好地不被不相关的上下文分心，更好地推理什么是真正相关的。
[01:11:56.080] 然后嗯是的，明确地减少你的记忆大小嗯也可能是有用的，这样语言模型从其中选择东西的负担就更小。嗯我们在这方面做了一些工作，使用了类似缓存类型的方法嗯，你只需查看你记忆中的某个特定项目过去被使用过多少次嗯，如果 它没有被非常频繁地使用，你就会丢弃它。这可以你知道相当程度地提升这些方法的效率，因为你不必在上下文窗口中放入那么多东西，但如果模型被不相关的东西分心，这也可以提升它们的成功率。
[01:12:41.520] 是的。然后我们将覆盖的最后一件事是，所以所有这些在某种程度上都有点临时拼凑，对吧？就像我们只是让语言模型查看这些过去的经验，然后生成一个技能，这个技能会被放入记忆并在未来使用。但我们只是像提示语言模型去预测那些看起来可能 在未来有用的东西。一个非常自然的问题是，你能不能直接训练语言模型来做这件事？嗯你可以。嗯这是一个最近才开始被探索的方向。所以这些论文嗯刚刚出现在你知道刚刚出现在今年的 ACL 上。嗯但这里的基本思想是，我们要在多个 按顺序看到的任务上进行强化学习。所以我们一直在用于评估的这种在线学习设置，我们也要在它上面进行训练。我们会按顺序获得多个相似的任务，我们会像之前一样通过从经验中归纳技能并在未来任务上使用记忆来建立记忆。所以这是一个非常复杂的， 你知道，过程，对吧？就像我们在这里使用了几个不同的模型。呃，我们把东西存储在记忆中。我们检索那些东西。我们使用它们。但好消息是强化学习不在乎。你可以直接把强化学习应用到像这样复杂的不可微过程中。
[01:14:04.960] 我们会有几节课向你们展示如何做到。嗯，你可以从呃这个第二个任务是否成功以及它是否使用了过去归纳出的任务中获得奖励。所以你可以直接对这个奖励做强化学习，在相似任务序列上的技能归纳上进行训练。嗯，有几篇论文探索过这个。它们都发现这是有帮助的。而且呃 其中一篇有一个有趣的结果，就是呃如果你还加入这个奖励项来激励技能的复用，它有助于提高成功率。所以你还可以想象引入其他种类的奖励项，比如最小化你已经归纳出的技能库的大小，试图你知道吧，让事情保持正则化，不要 过度拟合太多。嗯，我认为这里有很多有趣的像特征工作可以探索。而且嗯是的，这就是呃这就是我们有的全部时间了。呃但是呃希望你们对技能被归纳以及被书写的方式有了一点感觉。嗯你们会在你们的项目中应用那些，你们也可以在研究项目中探索它们。我会再待几分钟 如果大家有任何问题。
CMU 11-768 Fall 2026 · Lecture 4 · Agent Capabilities 3: Skills and Memory
- 视频：官方视频
- 字幕：en-orig · auto captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
正文
[00:01.520] Okay, welcome back everyone. >> So, um the first assignment is out and uh as Graham said, we we have credits available for you to to work on it for those of you who are enrolled in the class. Do you folks have any logistic questions? Cool. Okay. So we talked in the last lecture about how agents can uh choose what to retain within their context window as a way of you know storing information within a task and using that to um to determine what's important to do better within the task. So a big part of today is going to be about how can agents go beyond the context window and store information across tasks in external stores um that will be useful for them to improve performance on tasks in the future. And this is going to be skill induction. Um, and uh, we'll build on some some work that was done in the past on memory systems for agents. Um, and show how to apply those to to agents. But there's also another view on skills. And skills are a way for people to write instructions for an agent that can encode kind of like best practices or the knowledge that you have about how a task should be done. And these are actually going to be complimentary to each other. Um so we'll cover it just a small amount in this lecture but um you can imagine a system that um is sort of having a human in the loop um that's relying on the agent to uh propose some potential best practices from the experiences that it has a person edit those and goes back and forth or vice versa like you know having a skill and instruction that's written by a person be refined by the agent based on experience. So yeah, it's a lot of different viewpoints, but hopefully by the end of the lecture, you'll see how they can all sort of fit together. So let's just start off with some motivation about why you might want to remember things from one task to another. So say that we have an agent
[02:09.039] that's carrying out tasks in a website like a shopping site and you might give it a task like add Sony headphones to my wish list. So when we cover guey agents, you'll we'll get into the details of, you know, the representation space, the action space for how a system like this works. But at a high level, it's going to be producing actions like um uh navigating to the store website, finding the search box for it, typing some text in that search box, and then clicking search. And that's going to be the part that like searches for the headphones, right? And then we'll do something with them. We'll find the ones that match the request the user had and put them into our wish list.
[02:51.840] We might have another task that we carry out in the future like find the wireless keyboard price range. And if we're doing that on the same site, a lot of the actions are going to be reused, right? We'll go to the store, we'll find the search box, we'll search for the different product but in a similar way and then we'll do something different with it like you know getting the prices and reporting those back to the user. But if it's difficult for the agent to figure out how to do this search, once it knows how to do it, it should be able to reuse that knowledge in the future to be more effective on future tasks. So a lot of structure recurs and today is going to be about how can we represent this shared structure. So for example, is it best to just uh keep kind of the raw actions that the agent produced when it was carrying out this shared subtask or is it better to try to abstract it into for example a code function which could be run as a blackbox or maybe tested or even like hierarchically call other functions.
[03:57.120] We'll also be interested in how can the agent like determine which parts of a task should be reused in the future. And um as we're you know any time we have a memory that we're building up from experience, there's questions about um how do we grow that memory? Do we are we always adding to it? What happens if we get some conflicting information or we find a better way to do something? We might want to make edits to it. we might want to delete things if they no longer seem to apply. So, how can we um kind of manage the life cycle of these skills um and these subtask representations that are learned? [snorts] So, I'd say there's three general ways that we can get agents to update. So, the first one we covered in the last
[04:47.759] lecture. It's um just what you maintain in the context window of the model. So this is um you know in the standard React style way. It's the history of the past observations and actions that you've taken, messages you've gotten from the user, feedback from the environment, and you need to you need to choose like what to retain given kind of the limitations of the context window and the model's ability to use that. So uh we talked a lot about like compaction and methods for managing what's in that context window. Today we're going to be focusing on storing things outside of the context window. So even with like really long context windows that we have now of like a million tokens uh for um multiple for like carrying out multiple
[05:34.560] tasks with an agent um you're going to exceed that context window and you want to remember things from past tasks to help you on future ones. So if we are storing things outside of the window, um that's going to look something like a database um that needs to be managed um in some some way or um like a file system that the agent can access. And uh this lecture is going to be about methods for um putting things into those types of storage and um using them and updating them. And then the final way is going to be like actually making uh updates to the weights of the model. Right? So this looks more like standard supervised fine-tuning or reinforcement learning. We'll cover that in a few lectures later on. But each of these has
[06:21.600] like pros and cons. So um I think like a main pro of using external artifacts is that they're like they can be ported across models, right? So if you um have a description of the way to search for a product um that could be used by any one of a number of different models um within an agent framework. Whereas if you're actually making updates to the weights of the model that's just an update for that specific model and it doesn't transfer to others. So external artifacts are also more inspectable by people. Um, so you could write down best practices or how to search on a site. The agent can use that. The agent could also propose, you know, kind of text for that and you could look at it and make changes to it. So there's it's easier to
[07:11.199] kind of like have an interface between the person and the system if your um method of updating, if your method of remembering is interpretable. Any questions about this? How many folks have uh have used skills in their interactions with the coding agent? Like how many of you Okay, so most folks have. How many of you are are writing the skill yourself? Okay, so maybe about half the folks who are using skills. How many of you have uh gotten an agent to write a skill for you? Okay, so a lot of folks actually have. Yeah. What are some things that what how what's your experience been for the agent writing the skills? How often have you been able to use it just right out of the box? The thing that the agent constructs anyone have any spectacular failure cases or spectacular successes? So just repeating that for the recording like sometimes the models will write too much in the skill like they'll kind of overinfer the what should be done and then when that's handed off to another model it will kind of follow it to the letter and do a lot more than you actually wanted it to. >> Any other examples from folks? >> Cool. Oh yeah, go ahead. I mean in cases when you're like trying to like a production issue for like a particular system if you give it like three or four support and you tell it that look at how people have this issue in the past create a skill for yourself other agents could use to it in the same way >> and that's very >> oh nice >> okay cool yeah so instructing it to like get write the instructions for another agent it's good enough it's sort of like simulating what the agent would do. Yeah, that's interesting. I've sometimes found that the models are like pretty bad at kind of like simulating the theory of mind that you know another person or an agent would have when they
[09:10.160] like look at instructions, but they get a lot better when you prompt them to do that. Cool. Okay. Yeah. So, a few different like types of experience that we might want to retain for agents. So the first is like the episode and this is you know just the sequence of observations that you get from the environment from the user and the actions that are taken and this is sort of like the maximum fidelity right you're just you're retaining everything you have all the specifics of that particular environment and the particular instructions that the user gave also going to be like the most costly right it'll need the most tokens to represent and you probably don't need all of that right? Like uh just like when we're doing you know supervised learning we don't want to retain every detail we want to abstract to the things that matter to generalize um in the future. So we could also retain some facts like um some knowledge about the environment um or about the user and in the early work on memory systems uh a lot of it was focused on retaining facts. So like a user is interacting with an assistant and maybe gives some personal information like how old they are or where they live. The system can retain that in its memory and then use that fact to improve future interactions. If you've um used uh like Anthropic or Chatbt's uh memory systems, uh they're doing this basically, right? Like you can go into the settings, you can view the memory that they have about you and there's, you know, a text representation
[10:50.399] there from a model that's produced those facts um from your past interactions. Looks a little bit like compaction in some way. It's generated by an LLM. It gets stored in this external thing. And then the final one is more specific to agents. So it's about like how do you do part of a task um in a way that's abstracted from the details of the task. So like in the example we looked at before, how do you search for products on a particular website without maybe retaining the details of searching, you know, the particular product that you searched for? And so there's a lot of choices about how you would represent a skill. Is it just text? Is it kind of the actual actions that you took? Um or could it even be code?
[11:35.680] And of course, skills are only going to be useful if you're going to be doing the same subtask more than once. So we could think about like um memory and skills and the intersection between them. So memory is just something that we're saving from the agents interaction. So for example, what's a previous product that you purchased on a site? A skill would be reusable knowledge about how to act in an environment. um which could be represented in one of a variety of ways. Like it could be, you know, if you're releasing a um application um by pushing it to production, here's a checklist that you need to go through before you do that. So, a skill could be written by a person like in this example. But we could also have um a combination of memory and skills. So we could induce skills from the agents past experiences and store those um and then use those to hopefully do better in the future. So if the agent searched successfully for a product in the past um and uh yeah and if it was successful, if we think that it was successful, then we could pull out something that looks like a skill to
[12:45.680] guide the agent in the future. And we'll get a lot more concrete about what this looks like through the rest of the lecture. So we can think about skills as instructions where uh skills as instructions might be written by a person um for encoding their own you know preferences or might be organization policies like this is the way that we write and test code in this particular company. Um and human authored skills um we know where they came from. we have more maybe control over the quality but it takes you know a lot of time to write them and to maintain them. If we're learning skills then the agent is producing them. Um and this is helpful because we can do this in an automated way. Maybe we can have like a loop where we induce the skill. We try to use the skill to redo the same tasks. We see if things get better on a metric. Um, but it can be, you know, potentially noisy too and have, you know, especially if you're using the skills for some other agent like we talked about in those examples everybody gave, um, it could maybe have unintended consequences too.
[13:54.800] Okay, so let's talk first about uh, humanridden skills and um, the standards for these and the ways that they're used in incurrent agents. So probably a lot of it sounds like a lot of you have experience with this already. Uh so maybe you've been using these in the context of an example kind of like this. Uh there's this nice blog post from open hands that has guidance on using skills. Um for those of you who um who haven't used skills yet or if you you know like some suggestions on how to uh how to have best practices for creating and using skills. But they have this motivating motivating example of say that you find yourself like always giving similar instructions to a coding agent. Um so you um want the you want to check the code that has been written using one of a couple different llinters. You want to give type hints. You want to use a particular type of dock strings. And you want to apply piest to all the functions. So if you're always doing this, you could always type this guidance into the system. Um, but you know, a kind of natural and faster way to do this would be to just have the like store the guidance as a text file and then have the agent use that guidance um as an instruction. So you should reach for a skill when you want to do something repeatedly following kind of like a fixed specification. And the agent's going to choose how to interpret this specification in context, but the um general instruction is the same.
[15:31.680] And the way that this is implemented in the current standard for skills um which is called agent skills is just as a collection of files which the agent is able to read. And here's an example of of that. So if we are constructing a skill for Python review um we'll have a folder which has a structure like this. So skills have this standard structure where they'll have a skill.md file and you have to have that. Skill.md um is a markdown file that starts with um some YAML metadata. So you need to have a name for the skill and you need to have a short description uh which is going to give guidance to the language model about when the skill should be applied. And we'll see in a slide or two about how the model gets access to this information in the YAML header. And then um after the header you'll have a much more detailed description of um how the skill should be applied which would for example you know list um here are all the llinters that you should apply here's how testing should be done and so on. So you have to have skill.mmd. You also have a few other um folders which are optional but can contain some extra resources for the agent to use as it carries out the skill. So there's scripts which has executable code. So for example, if you had like a Python script that would run a llinter, you could put it in there and then describe it in this skill.md file. There's also references, which would be sort of like supplementary documentation that maybe you don't you you don't need to use it every time you use the skill. Um, but you'll have maybe a reference to that documentation in the skill file. The agent could load it as needed from the references folder. And then assets basically just has like, you know, additional templates or resources like if you're, you know, doing something that involves like Ginga and you need
[17:28.319] the agent to load that conditionally. So this can be a lot of information, right? And we um don't want to fill up the context window of the model um with all of the possible skills that it could use. So there's a mechanism called progressive disclosure which is going to just show relevant parts of skills to the agent as they're needed. So the first level is um the metadata in this YAML file. Uh so the name and the description and that's always going to be um shown to the model in the system prompt if the skill is loaded if it's available to the model to use. So you need to be kind of concise in this uh description. Um so that if you have a lot of skills you don't fill things up too much. There's the second level which is the rest of the text in the markdown file and um that'll be loaded conditionally if the model decides that the skill is relevant um and uh decides to read the rest of the file via a tool call. We'll see an example of that in a minute. And then the next level is sort of like all the resources in here which will also be loaded um using tool calls. So you kind of need to have enough information in the skill.md file for the model to be aware that it should look into these additional folders to get more resources um when it's carrying out the skill. Any questions about this and you'll implement this in assignment one or maybe some of you are already starting on implementing that.
[19:05.840] >> Yeah. are so high like what if that takes too much of that effect performance? >> Yeah. So if you have a lot of skills even if the skill metadata is pretty short could it affect performance? Yeah, it definitely can. Um so you want to keep the number of skills somewhat small and maybe just load them conditionally. um in one of the state-of-the-art agents, Hermes agent, I think the I'm not sure the number of skills, but like the total length of the skill descriptions in the system prompt is like 3,000 tokens. Um there's some papers that we'll see in a little bit that show that if you have too many skills, um it can make performance worse. Like adding in irrelevant skills can drop performance um because, you know, the context gets longer. The model might be distracted by things that are actually irrelevant. So yeah, you definitely need to be careful about the number of skills, but um I think it would be pretty rare to have like 10,000 or a thousand skills. Generally, it's, you know, smaller and you would maybe load skills as needed for like the software tasks that you're carrying out. >> Yeah. >> How sensitive are those triggers? >> Yeah, that's a great question. How sensitive are the triggers? So the the triggers will be tool calls that will be generated by the model. And so whether they get called or not is a function is totally dependent on the model. Um and you know the model's ability to predict that it should be triggered based on the past context and what it knows about the skill. So that you know depends on the model depends on the description you write and um that's why we need evaluations for this. Um and one of the papers that we'll look at takes a step in that direction.
[20:59.919] >> Yeah. like a standardized name. >> Yeah. Uh skill.md is a standardized name. So the the standard is called the um I think the agent skills standard and um you like models agents are trained to to use that agent frameworks are trained to load that. Um so yeah you um if you want to use an alternate format well first put it you know as close to this as you can but you could do you know if you have particular if you have particular um things that you want like in the references folder you could describe them in skill.md and like all this is just going to be instructions to the language model so it could be robust to other configurations also especially if you describe them.
[21:50.799] >> Yeah. Why are >> Yeah, why markdown? It it is a balance of machine and human readability, I think. >> Yeah, that's a great question. Could there be better formats than than skill.md? Yeah. Do you have things in mind? I think like um so uh I mean it's just like linear right um it's you can have hyperlinks which is good you can embed images too um >> yeah um but uh like it's difficult to you know represent things that are dynamic so like maybe you want to illustrate to the model like um different ways the data could be visualized um you know maybe that would be better with like a JavaScript interactive JavaScript application or something like that. Um, yeah, you could, you know, have that be sort of like a code file here that gets executed. Um, but and maybe like referenced in the markdown. Um, so I think like markdown is sort of the best attempt at having something that's general and like readable, but then you would rely on the language model to interpret other formats. >> I think in the scientific literature that uh Daniel's going to talk about, there's basically two ways. There's textual skills and programmatic skills
[23:18.960] and the textual skills appear in the markdown and the programmatic skills appear in the script. So it kind of it's pretty broad if you consider those. >> Yeah. Cool. >> Yeah. >> Does anyone do to not have to put it all in context? For example, if you're making an agent that modifies cloud infrastructure, you work at some big bank and have a billion things in AWS and it would be nice to be able to just ask the model to modify something, but then because it's a billion things, you can just have a billion tools for every small component available. Yes. >> So looks like something would do. Yeah, definitely in some of the research papers that so the question was like do people retrieve from large collections of skills. Um and yeah definitely in some of the research papers that we'll look at um they do that um either using you know just like a text embedding based retrieval or doing something that's more kind of like domain conditional like we know these skills apply to this site so when we're interacting with that site we'll load those skills. Yeah, I imagine production systems do this too, but um yeah, I
[24:30.799] don't have any references off the top of my head for that. Cool. Okay, so we'll uh just uh kind of quickly to show you how progressive disclosure works. Um we first need to give the model access to this level one, right? Let it know what skills are available. And you'll have a system prompt that looks something like this. This is just taken from Hermes agent which is sort of like a standard open- source um agent that is able to use and induce skills and you'll um kind of describe what skills are. Uh you'll tell the model about this skill view tool which it could use to read the rest of the markdown file and then you'll list out all the skills that you have.
[25:14.080] So there's just two skills here. Um, there's a code quality skill and a workflow skill. And that's just the YAML metadata from those skill files. So, um, the model is able to use tool calls to, you know, expand the rest of the skill. And here's what that might look like. So, it always has the index of skills in the system prompt, but it could choose to make a tool call to open the Python review skill. um if that seems relevant to you know the user asked for a review of some Python code and then it's going to like um you know get that markdown file insert it as a message to the system as an observation the result of that skill view tool. Um the system might choose to run something from the scripts directory um and then it'll get the result of running that script as an observation put it in the window um and might choose to you know like view additional files within that skill package. Okay. So yeah, we these questions that we had about like you know could additional skills hurt um are skills being triggered in the right place. Uh we answer questions like this by building evaluations and um the the work
[26:33.600] on this is a little bit um new. It's sort of ongoing but um a really great first effort in the space is called skills bench and what they do is they um collect a bunch of skills uh from uh various sources online. they actually get like 2 million skills. Um, and they'll also have people write tasks which seem like they could benefit from skills. Uh, and so they'll like get 400 tasks and um those are going to be written independently from the skills, but they'll have the task annotators choose from this pool of skills ones that seem like they could apply to the tasks. And so you'll have kind of like a small set of curated skills. um and you'll supply those skills to an agent as it's trying to solve the task and they'll evaluate this on like um you know four different agent frameworks uh with I think like 18 models something like that and they'll just see like does having the skills um improve performance over not having the skills across all the tasks for a given agent framework and model combination right so this will evaluate you know it'll implicitly evaluate the skill quality but will be designed to evaluate is a model and framework able to you know use the skills appropriately and they're trying to choose the skills. So they have like a filter um which checks to see that for a small subset of models the performance on the tasks does improve when you give models access to the skills and then you're seeing if that's also true for kind of a wider set of frameworks and models.
[28:15.039] Yeah. Does the general setup make sense? Cool. So on average, you know, with this cur curated set of um skills, uh the average pass rates on these tasks using like humanridden verification criteria does increase. And so you can see like the solid bars to the shaded bars across a bunch of different models um and frameworks. Um but actually uh ski like tasks that require fewer skills to solve benefit more from having the skills available than ones that require more skills. Um and so this is partly a difficulty thing but also partly that the models might be getting distracted by having more skills available. And you could imagine that it's getting much worse with like a much larger number of skills. Um we'll see some more evidence for that later on in the lecture. And actually skills still hurt on a number of the tasks for particular model types. Um so the model might be using the skill inappropriately which could actually make it do the task in a way different than what the person specified. So this kind of motivates the importance of like checking to see if the skills that you wrote are actually
[29:31.440] making your outputs better. So skills can come from, you know, a bunch of different sources. They could be bundled with the harness. Um, they could be written by a person. Um, and a lot of you have done this, it sounds like, and you then, you know, kind of install it in a in a way that's specified by the framework that you're using, but they can also be created by the agent. And, um, Hermes has a skill manage tool. um open hands as a skill creator uh command that help work you through the process of creating a skill. And when you're creating a skill, I think it's really important to base that skill on some prior interactions that you've had rather than trying to just do it sort of a priori. Um and actually
[30:19.520] there's some results in the skills bench paper that show that models when they're tasked to create a skill by themselves just based on a task description, they're really bad at it. and it actually makes performance worse. So, as a person, you're probably better because you kind of like know the way that you want it to be done, but it's generally best to base it on, you know, past things that you've done because when you're doing a task, it'll reveal some things about the way you want it to be done that you might not know in advance. So there's a nice case study of this um from the open hands blog post on how do you like so you have this code review skill that you've created. How do you check and see if it's working and then improve it if necessary.
[31:01.360] So this is focused on like reviewing poll requests and what what you could do is like um after the review is produced by the agent then a person is going to go through the poll request and look at each of those review comments and either address them or not. And you can apply a modelbased judge after that and count up how many of these were like actually used by the person which will give you a sense of how good the pull request was. And you can like um do this over a large number of pull requests that were created and then reviewed by the agent using this skill. And um if you do this over um you know like hundreds or thousands of examples, you'll get a lot of evidence about what's working and what's not, you can summarize all of that with a reasoning model um and use that to like for example identify that it's not helpful to um you know produce code that ignores the repo conventions. people did like incorporate those changes quite a bit. And then you could like have the model propose some updates to the skill file that will make the you know reviews produced by the system more
[32:14.159] likely to be incorporated by the person. Cool. Any questions before we move on? Okay. So next we'll talk about how do we get models to automatically remember things from their past experience and then we'll use that to like build on um to talk about inducing skills from experience.
[32:40.480] So this is this is an old paper. This is from 2023 um but it was pretty influential. Um and it's uh it's one of the first papers it's called MEMGPT. It's one of the first papers to uh [snorts] introduce a way that you can manage memory outside of the context window of a language model. So this is like a really a systems inspired paper. I think we talked about a couple lectures ago about you know kind of having like memory hierarchies like you know something that is fast to access which for the language model is the context window and then something that's like larger but but slower and in this setting that would be like an external store of of facts and back when this paper was written uh a few years ago context windows were even shorter like you know 8,000 tokens um so this sort of approach was really well motivated even for just interacting with like an assistant system. Um nowadays you know our context windows are larger but what you want to store in them is also much larger. So this sort of approach is pretty well motivated for agents too.
[33:48.399] But the basic idea here is that you're going to have like a um you'll have you know the system instructions which don't get updated. you'll have a working context um which is going to uh pull things from an external memory um and also you'll be storing things to this external memory and those reads and writes are going to be controlled by tool calls from the model and um you'll also you know have like a separate storage for the actual messages that the system received um but that's going to be controlled in a pretty similar way to this uh this memory over fax So, what this actually looks like, sorry it's a little bit hard to read with the brightness in here, but like if you um you might uh have an interaction with the model where the user like mentions six flags and um the system could produce a tool call that searches within this storage for past mentions of six flags from its past interactions. It'll take those and it'll put them into this like working context. Um and then the model um this working context is stored within the context window of the language model. So the model will be able to use those as it outputs its response. But it's important to be able to do more than just like add to memory because for example like um maybe you're talking to the system about your job. you say I work at X, but then if you change jobs later on, you want the system to be able to like use the latest information.
[35:26.000] And um this paper called me zero uh introduces a way for memories to be updated as well. Um so the way that this works is you'll have like messages coming in and um each time you get a message, you'll apply a language model to extract some memories from it. um some like potential facts that you might want to store or update and you'll have an external uh database store um and you'll try to check and see do I have anything in that database store that's relevant to these um facts that I pulled out of these messages. So you'll retrieve the top K and to do this you'll use like an embedding based retriever model like you know the ones we talked about a little bit ago. um you'll pull out similar facts like um six flags for example or like um I work at X and you'll apply a language model to check for all of these extracted memories from these latest messages um do they um are they present in these similar retrieved memories? If they're not present then we should potentially add them in. um if they are an update to something that's there, we should replace what's stored for that with the new thing. If it negates something that's already there, like for example, you say, "I work at X," but then later on you say, "I got laid off," you might want to just delete that fact rather than um than updating it. Um or if it's not relevant at all, you would just ignore it.
[37:01.280] And um I'll refer you to details or for the paper for for details about this um because the details also change with with agentive systems but I think the idea that um it's useful to be able to like update knowledge as well if you have conflicts is potentially pretty useful.
[37:18.880] So, um, does anybody remember something from the lecture on context management that had a similar goal to this being able to update or delete memory? Yeah. Uh, hashing. >> Yeah. Yeah. Basically, does anybody remember the specific paper? Yeah. Yeah, it's hashing in the sense that there we had like key values and queries, right? Um and we were like storing values which were vectors. Um here we're storing like text um as facts. Yeah. So the Delta paper actually did this, right? like um linear attention is only able to append um values to your um what's effectively your memory but Delta um uh like effectively retrieves um things that are stored for particular queries and then changes them changes the vectors that are stored for for them and that helps improve performance. So this is sort of like a textbased analog of that that has a similar goal. Yeah, you could also think about like um for the third type of learning for updating the weights of a model, it's potentially helpful there to be able to forget stuff also. Um especially if you have kind of like conflicting knowledge and and people are starting to develop
[38:47.440] some approaches for that like unlearning um is one way to do it. Okay, so those are some foundational papers about memory um that were developed for assistance for agents. Um feedback on trajectories can be really useful. So this paper called reflection was um a really impactful it's a really uh it's a really pretty simple but effective um the way to get agents to use feedback and
[39:20.480] potentially do better. So, it's definitely worth knowing about. It's a good thing to implement if you're working on a project that might want an agent to be able to benefit from its past experiences. Um, and the way it works is we're trying to improve the performance of an agent given multiple attempts on a particular task. So, say that you're asking the agent to like um, you know, search for a product on a website. It makes one attempt at it. um it says that it's finished and you're able to get some feedback about whether it was likely correct or not. So that could come from like a person looking at it. It could come from applying a model as a judge. Um but given that feedback, you'll want the model to try again and try to do better. And what this paper does is show that natural language feedback um on the attempt can be a really helpful thing for the model to condition on in its second attempt. So like if it's trying to answer this question about what was a particular series of battles, it might just mention one particular battle and then the feedback that's given um generated by a language model based on knowing this is incorrect and looking at the question and the answer might be um you gave just a single battle rather than a series. So as you can imagine this would be really helpful feedback for the model to condition on as it you know does another attempt at this task and it can get it correct. Um and the paper shows that you know conditioning on this natural language description of what went wrong can be really helpful. So we can generalize this also to um the agent improving as it does different tasks. Right? This is improving on the same task. But you could imagine um feedback might be generally helpful to improve performance on similar tasks as long as the feedback's like sufficiently general.
[41:15.280] And the final way that we can like um have agents uh improve or you know kind of the final foundational thing for helping agents improve um is remembering trajectories uh like entire trajectories, entire episodes that they've done in the past and using that to shape what they do on future tasks.
[41:34.240] There is a bunch of papers that have done this. We have some links to them at the bottom of this slide. Um, but the key thing in all of them is you'll have some collection of like training tasks and those tasks could be like a collection of demonstrations or they could be produced by the agent as it's carrying out tasks one at a time. Um, and you'll pull all of those tasks together. Maybe you'll additionally like um generate some text that describes uh feedback from them sort of like reflection did on the last slide. And you'll have that pool of experiences. And then when you get a new task um that comes in, you'll retrieve from this uh set of trajectories and um maybe generated text about them to find something that's relevant for the task that you're carrying out. You'll put it in the context window of your model and it'll generally help you to do better. Um so this is sort of like rag for agents, right? Um and uh it's pretty effective. Any questions about this? Yeah. >> What do you mean by trajectory in this case is like sequence of calls.
[42:42.800] >> Yeah. Yeah. So a trajectory generally for an agent is the sequence of um like messages that it gets from the environment or the user. So the observations um the actions that it takes like the tool calls and um and uh potentially also you know the chains of thought that it produces. So one way to think about it is basically just like all the sequence of um of uh messages um
[43:07.680] in the context window of the model >> and you retrieve it by image differently. You can do this regular embedding retrieval or compare to process sequences and do some set comparison. >> Yeah. What are the different ways that you could do retrieval? So typically you want the model to you're giving the model the description of a new task that it should carry out. So you have just that text description and you'll embed that um with an embedding model and you'll retrieve um based on the descriptions of other tasks that each have like a trajectory associated with them and maybe you'll also constrain that to be like we'll just retrieve the tasks for this particular domain that we're working in. Um yeah, good question. Any other questions about this
[43:57.359] approach? Generally, this is sort of like a whole class of approaches. Um there's different ways to instantiate it. Um the details like how do you build up the experiences? How do you do the retrieval? Um but just want to kind of give you a sense that if we have past successful trajectories and we show those as examples to the agent, it can help it to do better.
[44:22.240] Okay. So with that we kind of have the foundation for talking about um actually inducing skills. Um so we can think about skills as being like parts of a task that are reusable across different tasks. So at the very beginning we talked about like searching for a product and that might look something like we tell the agent show me results for this particular query and the agent might produce a chain of thought like this and then it might produce these particular actions that actually interact with the web page like um uh clicking on this element identified by this particular ID which is the search box typing a query in there and and clicking on another element which is like the actual button to search. And this uh chunk of a task might be reused on future tasks. Like if I say what is the price range for um this particular item, it might need to do these same exact steps again as well as some additional ones afterwards. And so we can think of a skill as being um a specific like reused part of the task.
[45:38.640] Is this a good way to represent a skill? Can anybody think of any issues with this? >> Yeah. just >> Yeah, that's right. Yeah. So, like if the element ids on the page change, the skill will break. And you know, there's a number of ways that that could happen. Like, you know, maybe the elements are maybe it's the same page, but things are just numbered differently. You can make some changes to the action space to deal with that. I think JY will talk about that in his lecture. But like if we go to a different website that has like a different search interface um it's going to break and we'll see some examples of that later on.
[46:31.440] Yeah. So we'll generally compare as Graham said text versus code as ways to represent skills. I want to briefly highlight some uh really cool papers um from the past that I think kind of like lay a foundation for skill induction um for learning skills from experience. Uh, one is called um Voyager and um they're controlling an agent in Minecraft. Um, and they're using uh code as a way to control the agent where the code functions um implement things like um crafting a particular item or combating a zombie. And representing these like subtasks as functions allows functions to call each other. So that as the agent like learns functions for simple things, it can then like compose them calling them in other functions and you can carry out increasingly complex tasks. There's also a line of work um from the program induction community um that that does this. I think a really nice representative of this is um uh work called dreamcoder and a few papers that built on it. And there you're um sort of uh you have basically like grammarss over things that you're constructing um like objects um and you uh like have uh some examples of complete objects which are constructed using like very low-level functions and you search for recurrent patterns in that you find patterns that can be abstracted into functions and you do that repeatedly and you're sort of like compressing the programs that construct these objects.
[48:07.680] And then there's also some follow-ups to this that uh integrate like large language models, Python code, and uh natural language descriptions too. And some of the same ideas in those papers are also useful for agents that are um using language models to interact with the world. A typical way that we'll evaluate systems for inducing and using skills is an online learning setting. So when you're interacting with um an agent, you are maybe having it carry out a bunch of tasks in succession and some of those tasks are going to be similar to each other. You want the system to be able to learn from your interactions with it and then be able to do better in the future, right? So um this is an online learning setting where we'll have tasks come in one at a time. So for example, like you might say, add a Sony Bluetooth headphone to my wish list. And we'll be building up a memory of skills um incrementally as we solve those tasks. So um there will be an induction step that we'll talk about in a slide or two. And that induction step will create a skill like search for a product and add a product to the wish list that are useful for solving this particular task. And um you know maybe we have another one like this task here and we can apply those skills on future tasks like this task might also involve searching for a product and hopefully we'll do better on that task given that we had this skill induced from the previous task and so on. So what we're interested in is if we can measure success on each of these tasks, is our success rate like getting better and better? Are we able to solve increasingly complex tasks um as we're learning from experience and building up um uh representations of the ways to do these simpler tasks? Any questions about this setting?
[50:11.760] Okay, cool. So the papers that we'll look at for the rest of the lecture look at sort of different parts of this space of um uh learning skills from experience having this library and ask questions about what should we do with the skills like after we've um induced them. So there's generally like a learning step where we'll have these uh tasks coming in. We'll try to get a skill out of them. will decide whether to store that skill um in the memory that we're building up over time. There's um the step of like actually using this memory of skills to potentially improve performance on the tasks that we're carrying out. And then finally, we might want to, you know, if we're getting too many skills, we might want to try to simplify those, like delete the ones that aren't being used, or if they're too specific, we might want to make them more general. um or like consolidate and merge multiple things together. [clears throat] We'll go a little bit deeper into one paper um that I think is pretty representative of ways for inducing skills. Uh it was also done here at CMU
[51:24.880] as work led by Zora Wong who will be giving a guest lecture on some other topics a little bit later on in the course. Um but this paper's called agent workflow memory and um the way it works is it we applied it to web tasks. Uh this is work with with Graham also um and me and uh we applied this to web tasks and you have a query like this and you know tell me the number of reviews that our store received so far and when the agent's carrying this out these are the actual tool calls that it'll produce. So we have this sequence of tool calls and we want to try to identify which parts of this might be potentially reusable in the future and you can probably guess how we do this. We ask a language model to take a look at the sequence of tool calls and identify ones that seem like they might be useful on future tasks. And we'll have the model output what we call workflows um which are sort of like a textbased representation of skills text and actions. And a workflow will have like a description of the subtask like searching for a particular item. And then um it'll have the sequence of actions that you use to do that. And we're having the model make this a little bit more general so that it could apply to like searching for other terms. So we have it use like create a template that can abstract out some parts of the um initial action sequence. You might be thinking like this is starting to look like code, right? That's true. Um and a little bit later on we'll compare code to this also and show how that can be better in some ways but also uh make things a little bit more uh fragile. A pretty important part of this is we'll um apply a judge model to this trajectory to decide um have the judge predict whether the trajectory seem to be correct or not. And we'll use that to
[53:28.400] determine whether we should actually add the workflows that were induced to the memory. Um because we don't want to be um remembering things that were incorrect because those might steer the model wrong in the future. >> Any questions about this? Okay. And so the way that we'll um give the model access to these uh collection of workflows um is just by sticking them in the context window of the model. So this looks a lot like you know this agent skill standard. Um you have choices about whether you show everything or whether you show just you know kind of the text description. um and what you do there could depend on how many skills you have and you know the context length of the model.
[54:20.000] So um for the sake of time I I'll skip over this induction prompt but we do find that um having this memory of like past uh subtasks um can help to improve the performance of the agent as it's learning online. So on the x-axis here we have like um different tasks in a fixed ordering from this benchmark involving interacting with like a map tool and the blue line here is the baseline agent that doesn't have a memory that sort of converges to around you know like 10% accuracy on these tasks and the labels here are the tasks that are being solved by the agent correctly. Um and then this black line which adds in this memory of um subtasks improves pretty substantially and it's able to um you know this gap emerges between the two but it's also able to carry out these like increasingly complex tasks eventually doing things like you know finding a hotel near this location showing me the walking path um that the baseline agent just isn't able to do. So this is using, you know, sort of this mix of text and example actions to represent skills.
[55:33.599] But we could also consider using code, right? And both of these are implementations of a search product um function. And if we're using just this text and examples, this is going to be in the context window of the model. and then it's going to be needing to produce these tool calls um in a new context, right? As it tries to carry out the same subtask. So, it's flexible like the model could choose to insert a particular search term to search for a different product. If the website has changed and we no longer have like this particular um element for the search box, then the agent could choose to use the correct one instead. Um, but it also means that the agent needs to like actually produce all of those tool calls itself. If we're using code, the agent
[56:25.760] could actually just call this function as a tool, right? If it passes in the right arguments and if the functions correct, then it doesn't have to do anything. It doesn't have to produce all of those low-level actions itself. Um, the tool will just do that as it executes in the environment. Can anybody think of uh some pros and cons of using a function like this as a representation for a skill? Any cons? >> It's very rigid. >> Yeah, it is very rigid. So like um the model will need to the model calling this tool will do exactly these three steps and there's no room to to deviate from it. Um yes that's a main call that's a main like con to it but we also get a lot of the pros of using code right so like we can nest functions it can be hierarchical we can actually like generate test cases for that specific function too independent from the entire trajectory that it's executed in. And we can do things like you know refactoring also. So uh Zora has another paper um called inducing programmatic skills or agent skill induction which um which do this and it looks pretty similar to the
[57:47.839] approach we were using to induce these textual workflows. So you have a trajectory like this with these tool calls and we'll have the language model um we'll we'll just use the fact that language models are also good at generating code and we'll um prompt the model to generate functions which abstract a lot of the um uh behaviors in this trajectory and these are real examples. So the language model will write this search reviews function and um a doc string for it and then you know it has these low-level tool calls as um lines in the function. This other one for open marketing reviews but we'll also have it generate sort of a rewrite of this original trajectory that uses those tools. So a really nice thing about um code is that we can test it by executing it. So instead of applying the judge to the original trajectory like we did before, we can have the language model. Well, we can execute this generated abstracted code and then apply the judge to the end result that you get from that. And if that appears to be correct, that's sort of like a stronger indicator that the skills, these code skills that you induced are also correct. And if they are, then we put them in the memory and they become usable to the language model just as tools. So this is a way of like building up a tool library that the model's able to use on future examples and um we show that it also you know improves performance.
[59:19.680] Any questions about that? Yeah. So a nice thing about um code skills is that they allow testing just of the skill by itself. Um there's this nice paper called skillw weaver which does this. Um, so this is sort of some pseudo code for the uh the loop that I just showed you where you have a task, you um propose some set of like code skills for it and then you'll try to use
[59:50.079] those skills to execute the task. You'll get an episode, you'll apply a reward model to it. Um, and if it succeeds, then you can put all those induced skills into your memory. But if not, then you could make some revisions to it. And there's a number of ways that skills might not succeed, right? So, um, one thing we could do is like apply a code llinter to an induced skill. So, here's a real example from the paper. There's this induced skill called identify pill. Um, you know, for like interacting with a drug website. And this uh function like takes a lot of arguments like you know the um the make of the pill the color but in the initial version of this skill that the agent wrote it actually didn't use all the parameters. So like it ignored this color parameter. So you could get feedback from a llinter on this skill and then sort of in the reflection type way um condition on that and then like you know uh create an updated version of the function and store that or like try executing that and store that. You could also use feedback from this reward model and condition on that when you're um
[01:01:00.400] making these revisions to the skill 2. So these are some nice benefits of code um that can be advantages even if you know the actual code function itself is a bit more rigid. Another nice benefit is that because these functions that are induced are now tool calls, it can really reduce the number of times that your agent needs to interact with the environment. So here's an example from um Zora's uh ASI paper. We have this uh task like uh updating a couple addresses on a page and for the base agent you have to you know like do all the very low-level actions of navigating through all these forms typing the particular um things from the address. But if we have this model that has induced these code skills, you can and this is a real example from uh from the work where we induce this navigate to address settings and update address details functions. We can actually solve this task in three steps where we um you know assuming that this is a good fit for the site. Um we can just call these two functions and um complete the task.
[01:02:12.319] Um, and the agent is actually much faster because it doesn't need to reobserve the website and then generate the tool call, then reobserve the new website, generate the new tool call. Um, as you'll probably see in the guey um, lecture that JY is giving, um, a really big bottleneck for these models is actually the number of times that they have to interact with the site because these images take up a lot of tokens and actually producing a chain of thought um, before each action takes up a lot of tokens too. All of those have to be generated. So efficiency is really important here. And um in this work we did find so we did a controlled comparison between using text skills and code skills. This was on a set of tasks which did have sort of more repeated structure. Um but we found that uh in comparison to no memory text skills um helped but code skills helped even more in terms of you know the accuracy of the system but also we got pretty big improvements in efficiency the number of times that the agent needed to interact with the page.
[01:03:18.960] Cool. So, as we've kind of talked about before, a big potential issue with skills is that they might not be general. And here's a failure case of this approach and actually a failure case of these code skills. So, we tried to see how well could skills generalize across websites. And we induced um these code skills on one site uh and then we tried to apply them on a real website uh like the Target shopping site. So this skill um that was induced on a different site worked on that site but like um uses assumes that this uh this um particular element that you're interacting with has a drop down like this. Whereas on the target site it's you know this set of radio buttons and so it it fails there. And um you have to have the model be able to like actually
[01:04:09.359] edit the skill based on um feedback from the environment as it's interacting with it. But code has a solution to this, right? Uh so we have um abstract classes and those can be instantiated in different ways. There's this uh paper called poly skill which does this. So um they're using code as the representation sort of building on this ASI work that we did. Um, but the first time that the model interacts with a new website, it writes an abstract class which doesn't have implementations for the methods, but could have stubs like searching for a product, adding to a cart, checking out. And um, then it'll also write a website specific implementation of it like this one for Amazon. Then when it's interacting with a different site, you could have some metadata or you could even just have the language model trigger that this particular concrete implementation no longer applies. and then you'll write a different instantiation of the abstract class. And they show that this helps too. Um, this is also nice because it gives you a way to have like a large number of specific skills, but only have a subset of them be active at a given time that are actually relevant. So, this kind of makes sense, right? It's, you know, just a software engineering best practice that's also applied to agents. Okay. Uh, yeah, we talked about all this already. um you guys had a good sense of the the pros and cons of each of these types of representation, but it's helpful to know that, you know, the
[01:05:45.760] skills standard allows you to combine both and um there's a lot of research to be done on the best way to combine these representations together. Um like uh said, okay, so we'll go quickly through a few papers um on different kind of like parts of this skill life cycle. This is a really open research topic of like what are the best ways to induce and
[01:06:08.240] manage these skills. Um you'll have the papers for for more details. Um but you can also explore this in your projects uh towards the end of the lecture or the end of the class if you're interested. Everything we've talked about so far has been learning from success, but it's also potentially pretty helpful to learn from failures. So say that the model like tries to carry out this task like um you know searching for Bluetooth headphones. Um but for this particular website they sort of have like their search terms are or rather than and so you get a list of everything that's Bluetooth or headphones or a Sony. This is a real example from this reasoning bench paper from Google. So you get like 5,000 results and the agent just breaks on this like it has to pagionate through all of these 5,000 results. it does, you know, it loses track of what the task is and it fails. But if we observe that um failed trajectory sort of in the reflection style way, we could uh produce some text that gives guidance on how to avoid it in the future. Like saying um we should give a more specific search term rather than this more general one and we should you know have like a larger number of items displayed so the model gets more in its context window at once. And this paper um generally uses the same framework as like uh agent workflow memory, agent skill induction, but instead of just having these kind of abstractions of the correct trajectories, it generates this text feedback um which um is applicable
[01:07:42.880] to failed trajectories too and is also you know the kind of thing that you would write in a skill yourself um if you were writing one and they find that this is pretty helpful. So these are different like types of memory. Um so synapse uses trajectories like full trajectories with all of the actions and observations all the tool calls and messages. Um this is agent workflow memory that uses these slightly abstracted parts of trajectories. And this is their approach that uses these text descriptions of strategies and um adding and negatives going you know adding and negative examples as inputs to the memory. um uh doesn't improve performance when you're using either trajectories or um workflows, but does improve substantially when you have sort of these text descriptions, which kind of makes sense given the example we saw before because you can describe in a general way how not to fail rather than remembering and conditioning on those failures. Another important thing to think about is we're deciding whether to store trajectories um you know store things in the memory or what type of strategies to
[01:09:02.000] infer from the examples based on whether we judged the example to be successful or not. That's in most of this work using a model as a judge that looks at the final state of the trajectory, looks at the task that you were trying to solve and predicts does it seem like I got it correct or not. So a really good question to ask here is what you know how much does the quality of that judge affect the performance of the algorithm because we know the judges aren't perfect. Um this paper reasoning bank controlled for this. So they um run their methods with a perfect judge. Um and they also like to so they use kind of the ground truth knowledge of whether it succeeded or not. Um but then they add some noise into that and so you're able to control um the accuracy the simulated accuracy of judging this trajectory and the performance does fall off but they find there's kind of a sweet spot where um you do get improvements from the memory um and uh there's sort of a widish range of accuracies where using the judge to induce the memory is helpful but there's definitely room to do future work here.
[01:10:19.600] Okay. And uh yeah, with the last few minutes, um I want to come back to we had a question before about does having more skills than are necessary um potentially hurt performance? And we also had a question about like um we're if we're retrieving skills from a set um does that retriever matter? Like does the quality of that retriever matter? So um there were some nice ablation experiments in this reasoning bank paper that uh control um how many experiences you are retrieving from the memory that you have and they show that performance does fall out fall off as you have more available. Um so this could be because of a couple different factors. So um you know maybe you just haven't run your agent on that many tasks and maybe there's nothing relevant in your memory. So putting more stuff in is just bound to be irrelevant. That would be sort of like just a fundamental limitation of this approach applied in that setting. Right? You can only learn if there's some similarities between the things you're learning from and the things you're applying to. But it could also be like a limitation of the model itself. Maybe the model gets thrown off by having more things available in the context window. That could potentially be fixed by training the model, right? To be better at not being distracted by irrelevant context, to be better at reasoning about what's actually relevant.
[01:11:56.080] And um yeah, it can also potentially be useful to uh explicitly reduce the size of your memory um so that there's less of a burden on the language model to select from the things that are in it. And um we did some work on this that sort of uses like a caching type approach um that you just look and see how often has a particular item in your memory been used in the past um and if it hasn't been used very frequently, you'll drop it. And this can you know improve this efficiency of the approaches quite a bit because you don't have to put as much in your context window but can also improve their success if the models are getting distracted by things that are irrelevant.
[01:12:41.520] Yeah. And the very last thing that we'll cover is so all of this was in some ways a little bit ad hoc, right? Like we were just having the language model look at these past experiences and then produce a skill which would be put in the memory and used in the future. But we're just like prompting the language model to predict things that seem like they might be useful in the future. And a really natural question is could you just train the language model to do that directly? Um you can. Um this is a direction that's starting to be explored just really recently. So these papers um are just appearing at you know just appeared at ACL um this year. But the basic idea here is that we're going to do reinforcement learning over multiple tasks seen in sequence. So this online learning setting that we've been covering for evaluation, we're also going to run training on it. We'll get multiple similar tasks in sequence and we'll be building up the memory. um in the same way that we've been doing before by inducing skills from the experience and using the memory on the future tasks. So this is a very complex, you know, process, right? Like we're using a couple different models here. Uh we're storing things in a memory. We're retrieving those. We're using them. But the good thing is reinforcement learning doesn't care. You can just apply reinforcement learning to complex non-ifferiable processes like this.
[01:14:04.960] We'll have a couple lectures that show you how. Um and you can get reward from uh did this second task succeed and did it use tasks that were induced in the past. So you can just do reinforcement learning on this reward being trained on um skill induction on sequences of similar tasks. And um there's a couple papers that have explored this. They both find that this is helpful. And uh one of them had this interesting result that um it helps to improve success if you also include this reward term that incentivizes reuse of the skills. So you could also imagine like introducing other sorts of reward terms that like minimize the size of the skill libraries that you've induced to try to you know kind of keep things regularized and not overfit too much. Um I think there's a lot of interesting like feature work that could be explored here. And um yeah that's uh that's all the time that we have. Uh but uh hopefully you got a sense for the way that skills are induced and also written. Um you'll apply those in your projects and you can explore them in the research projects too. I'll stick around for a few minutes if folks have any questions.