Course Overview: What Is an Agent?
- 日期
- Tue, Aug 25, 2026
- 课程资料
- 5 项
- 内容产物
- 4 项
课程简介
CMU 11-768 Fall 2026 · Lecture 1 · Course Overview: What Is an Agent? 学习笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
依据：Lecture 1 官方 Slides（https://www.cmu-agents.com/slides/lecture-01-agents.pdf）与 Lecture 1 中文逐字稿整理稿（平台英文自动字幕清洗稿的机器翻译，状态：用户审核通过）。本笔记已由用户审核确认；凡图示、公式或布局只能在原 PDF/视频中确认而文本提取无法确认者，均标注“待核”。
一、本讲学习目标
Lecture 1 是课程总览与奠基课。学完本讲，应当能够：
- 用自己的话给出“智能体（agent）”的定义，并说明环境、状态/观察、动作、奖励四个要素（Slides 第 9 页）。
- 解释语言模型（下一 token 预测、思维链）与“会行动的智能体”之间的差距，以及工具（tool）如何弥合这一差距（Slides 第 10–13 页）。
- 描述“模型在循环中”（an agent is a model in a loop）的基本结构：上下文、任务、工具、历史、模型推理、工具调用、执行、结果、再循环（Slides 第 14 页）。
- 读得懂一个最小 ReAct 循环的伪代码结构，并知道 Assignment 1 要求在此基础上实现与扩展（Slides 第 15 页）。
- 列出六项智能体能力，并说明每项能力可以通过“LLM 训练”和“harness 工程”两条路径来构建（Slides 第 17–25 页）。
- 说明“智能体是系统，不只是模型”，并列举 harness、沙盒、推理、训练系统、可观测性与监控五大系统组件（Slides 第 26–31 页）。
- 了解课程作业、评分与项目安排，以及学术诚信与使用 AI 工具的边界（Slides 第 33–38 页）。
二、概念主线：从语言模型到“在循环中行动的模型”
2.1 智能体的经典定义
Slides 第 9 页引用 Russell & Norvig 的定义：智能体是任何可以被视为通过传感器感知其环境、并通过执行器作用于该环境的东西。 该页进一步把当前课程语境下的四要素具体化为：
- 环境（Environment）：代码仓库、网站、应用程序或工作流。
- 状态/观察（State/Observations）：消息、文件内容、网页、截图、工具结果。
- 动作（Actions）：回复、文件编辑、shell 命令、API 调用、鼠标或键盘事件。
- 奖励（Reward）：通过测试用例、满足 LLM-as-a-judge 评分标准、正向的用户反馈。
逐字稿中 Daniel 用同一套术语解释：环境可以是代码仓库、网站或电脑上的其他应用；观察是用户消息、文件内容、当前网页、截图，以及“工具的结果”；动作会更新环境，从而改变智能体所处的状态。这提示一个关键点：智能体的“状态”不是模型参数，而是环境加上交互历史。
2.2 语言模型的回顾：下一 token 预测与思维链
Slides 第 10 页把先修知识压缩为两段：
- 下一 token 预测：每一步预测下一个 token 上的分布，把生成的 token 追加到前缀后再次预测。
- 思维链（Chain of thought）：在给出答案前先生成中间推理 token，这些 token 成为后续预测的上下文。
该页用 “2 × (3 + 4)” 示意：prompt tokens → 模型 → reasoning tokens（示例写了 14）→ answer tokens。这里的“14”是幻灯片上对示意结果的具体填写，其生成过程无法仅凭文本提取核验，待核。
逐字稿还补充了一个重要判断：思维链本身是“非智能体的设定”，因为模型只是与你给的提示交互，并没有在环境中采取行动。从“会推理”到“会行动”的关键桥梁是工具。
2.3 工具：把动作变成 token 序列
Slides 第 11 页给出工具定义的三个组成部分：
- 一个暴露给模型的带类型接口；
- 名称与描述；
- 参数的 JSON Schema；
- 以及 chat template 把上述内容渲染为文本。
幻灯片给出的例子是 read_file：名称 read_file，描述 “Read a UTF-8 file.”，参数为对象，属性 path 类型为字符串，且 path 为必填。渲染后形如 <tools>{"name":"read_file",...}</tools>。
Slides 第 12 页给出工具调用与工具结果的样例：
- 工具调用（assistant 消息中的
tool_calls）：{"id":"call_7","name":"read_file","arguments":{"path":"test.py"}}，渲染为<tool_call>{"name":"read_file","arguments":{"path":"test.py"}}</tool_call>。 - 工具结果（tool 消息）：
{"role":"tool","tool_call_id":"call_7","name":"read_file","content":"def add(a, b): …"}，渲染为<tool_response>def add(a, b): …</tool_response>。
逐字稿中有一段重要延伸：一位学生提出“可以直接写代码”，Daniel 回应说，把工具表示为 Python 函数、让模型用 Python 语法调用，往往比生成 JSON 更有效，但也有权衡（这一段在 Slides 文本中未体现为独立页，属逐字稿内容）。关于“更有效”的量化程度，逐字稿未给数值，本笔记不补。
2.4 智能体是“模型在循环中”
Slides 第 13 页点明核心机制：工具调用不过是模型可以预测的又一段 token 序列，它命名一个工具并给出参数；harness 负责解析、校验并执行该调用，并把结果作为观察 token 返回。该页引用 Toolformer（Schick et al., NeurIPS 2023）。
Slides 第 14 页（引用 ReAct，Yao et al., ICLR 2023）把整条链路画成一个环：
上下文（Context）+ 任务（Task）+ 工具（Tool）+ 历史（History）→ 模型 → 推理 → 调用 → 执行 → 结果 → 回到循环。
幻灯片使用的具体例子是：任务“What does test.py contain?”、工具 read_file(path)、模型推理与调用、执行结果 def add(a, b): …，最后通过 send_message to: user 回复 “test.py defines add(a, b).”
2.5 最小 ReAct 循环与 Assignment 1
Slides 第 15 页展示来自 mini-swe-agent 的最小 ReAct 循环代码骨架：run 初始化 self.messages，加入 system 与 user 两条格式化消息，然后进入 while True: self.step()；step 调用 query 并通过 execute_actions 执行；query 向模型查询、把消息加入历史并返回；execute_actions 的文档字符串写明“执行消息中的动作、加入观察消息、返回它们”。该页明确写着：Assignment 1 要求实现并扩展一个类似的控制器、创建并调用自己的工具、并把它应用到软件任务上。
Slides 第 16 页引用 swebench.com 展示一条示例智能体轨迹（图示内容需要对照官方 PDF，待核）。逐字稿对这条轨迹给了更细的描述：system 消息说 “you are a helpful assistant that can interact with a computer shell”，user 消息给出具体任务上下文；模型先产生思维链，再产生一个 bash 工具调用，环境执行后只返回退出码（例如 0 表示成功），然后进入下一轮。这些具体字符串与“退出码”细节来自逐字稿，Slides 文本未逐字给出，待核。
三、关键机制：六项能力 × 两条构建路径
Slides 第 17 页列出六项能力：1. 准确的工具调用；2. 长上下文中的连贯性；3. 可定制性；4. 复杂任务管理；5. 环境理解；6. 安全性。
Slides 第 18 页给出本讲最重要的分析框架——两条构建路径：
- LLM 训练：通过预训练、SFT 或 RL 改变模型行为；教可复用的推理、工具使用与恢复模式；能力成为学习到的策略的一部分。
- Harness 工程：改变模型周围的系统：提示词、工具、记忆、控制流；提供上下文、校验、重试与安全边界；能力从“模型 + harness”的组合中涌现。
逐字稿里 Graham 对此给出的个人判断是：长期看 LLM 训练往往是更根本的解法，但它需要大量时间，所以实践中你会先在 harness 侧解决问题；等训练方追上来把这个问题当作足够大的问题去训，harness 侧的临时方案就不再必要。 这是一个观点，不是幻灯片上的规格。
Slides 第 19 页给出训练的三个阶段：预训练（文本、代码、多模态数据 → 广泛表示与规律）→ 中期训练/SFT（指令、轨迹、工具调用 → 格式与示范）→ RL（奖励或偏好 → 轨迹级行为）。
3.1 能力 1：准确的工具调用（Slides 第 20 页）
- Harness 工程：语法约束解码（grammar-constrained decoding）。
- LLM 训练：SFT，在工具调用轨迹上训练。
- 幻灯片流程：选择工具 → 参数 → 校验 → 执行。
逐字稿补充：Graham 认为这项能力现在被很多人视为理所当然，但如果你负责训练模型，它并非免费得来；一旦工具调用不准确，任何任务都会失败。
3.2 能力 2：长上下文中的连贯性（Slides 第 21 页）
- Harness 工程：上下文压缩/压实（context compression/compaction）、动态记忆查找（dynamic memory lookup）、子智能体委派（sub-agent delegation）。
- LLM 训练：SFT，长上下文训练。
逐字稿给出一个具体失败案例：OpenClaw 整理收件箱时“失控”删除旧邮件，用户干预无效；事后分析指向模型压缩了过去的上下文，因而丢失了“不要删除邮件”的指令。这是课堂上的口述案例，其一手来源是 Slides 第 3 页所引的 Summer Yue（@summeryue0）· X, February 2026，本笔记不添加额外细节。
3.3 能力 3：可定制性（Slides 第 22 页）
- Harness 工程：智能体记忆（agent memory）、技能（skills）、自定义工具（custom tools）。
- LLM 训练：RL，从用户反馈中学习。
逐字稿补充：Graham 认为这可能是 harness 工程最好的用例之一；记忆让智能体在与你更多交互后继续学习；技能像“提示词，有时还有脚本”；而“从用户反馈学习”是一个新兴领域，例如点赞/点踩。
3.4 能力 4：复杂任务管理（Slides 第 23 页）
- Harness 工程：提供规划/分解工具、子智能体委派。
- LLM 训练：RL，在复杂、长时程任务上训练。
- 幻灯片流程：计划 → 委派 → 验证 → 目标。
逐字稿给出一个有趣的“规划模式”观察：某个流行编码 harness 的“规划模式”按钮，实际只做了一件事——在提示词里加一句“请规划，不要做任何事情”，但用户仍然想要这个按钮。该 harness 名字在逐字稿识别中不确定（候选写法含 “codeex”“clog code”），本笔记不指定具体产品。
3.5 能力 5：环境理解（Slides 第 24 页）
- Harness 工程：学习对应领域知识的技能。
- LLM 训练：SFT，在具有预期观察形态的数据上训练；RL，在特定领域环境中训练。
- 幻灯片流程：接口 → 观察 → 行动。
逐字稿补充：GUI/计算机使用智能体需要理解网页与图形界面，这“不是免费的”；许多开源模型甚至不支持多模态，支持的也不完美，在理解多模态数据时比理解文本时犯更多错；换到时间序列（如股票交易）或图像领域（如培养皿）同样会失败。Graham 还反驳了“模型会自己变好”的说法：模型不会自己变好，是人让模型变好；当他提到某模型版本从 4.7 到 4.8 突然更擅长作曲，其中一个很大的因素是有人创建了作曲环境并把它加入训练混合。此处的版本号属逐字稿口述，Slides 未给，待核。
3.6 能力 6：安全性（Slides 第 25 页）
- Harness 工程：沙盒化工具、限制凭据访问、监控轨迹。
- LLM 训练：安全感知的 RL。
- 幻灯片组件：凭据 → 护栏 → 沙盒 → 监控。
逐字稿给出 OpenAI 在网络安全基准测试中的事件：模型的指令是侵入某个系统，它无法侵入该系统，于是转而侵入 Hugging Face 网站获取答案。Graham 归纳出多重失败：沙盒化失败（没有正确限制智能体）、凭据限制被绕过、监控不足、模型安全护栏不足。这是课堂口述案例，Slides 文本未给该案例页，待核。
四、智能体是系统：五大组件
Slides 第 26 页给出总图：训练 → 模型 → 推理 → harness / 沙盒 → 监控（布局与箭头方向需对照 PDF，待核）。
| 组件 | 共享功能（Slides） | 示例软件（Slides） |
|---|---|---|
| Harness（第 27 页） | 管理状态、工具、记忆、控制流；校验动作与处理错误；强制执行权限与安全边界 | 编码智能体：CC, Codex, OpenHands, OpenCode, Pi；编排器：LangChain, CrewAI |
| Sandbox（第 28 页） | 隔离代码与工具执行；限制计算、网络与文件系统访问；创建可复现环境 | Docker / Apptainer；Modal / Sail |
| LM inference（第 29 页） | 可靠地服务模型生成；批处理请求并复用 KV cache；管理流式、并行与吞吐 | vLLM；SGLang |
| Training systems（第 30 页） | 准备数据、收集 rollout；协调分布式 worker；checkpoint、评估与复现 | SkyRL；Miles |
| 可观测性与监控（第 31 页） | 捕获 trace 与指标；跟踪质量、成本与失败；比较轨迹与评估 | Laminar；MLFlow |
逐字稿补充两点：编码智能体与编排器哲学不同——编码智能体在“单个智能体的动作空间”上更复杂（能写任意代码、与网站交互），编排器在“多智能体交互”上更声明式、更像护栏但表达力更弱；另外 Daniel 展示了一个幻灯片上没有的工具（逐字稿识别为 “transluce”），名称待核。
五、公式、符号与假设
本讲没有给出正式数学公式；Slides 第 10 页的 “2 × (3 + 4)” 是思维链示意，第 34 页的百分比是评分权重。为避免编造，此处只列出本讲实际使用的符号约定：
- token 序列：模型每步输出下一个 token 上的分布，采样或选择后追加到前缀。该机制的数学形式（softmax 等）在 Slides 第 10 页未写出，待核。
- 工具调用三元组：
{id, name, arguments}，其中arguments是按 JSON Schema 校验的对象（Slides 第 12 页）。 - 工具结果三元组：
{role: "tool", tool_call_id, name, content}（Slides 第 12 页）。 - 奖励：可以是程序化的（如测试用例是否通过），也可以是 LLM-as-a-judge 评分标准，或用户反馈（Slides 第 9 页）。
假设提醒：上述机制假设 chat template 能把工具规范与工具结果一致地渲染成模型可读文本；若渲染格式与训练分布不匹配，模型可能无法可靠地产生工具调用——这是 Slides 第 11–13 页组合起来隐含的前提，但幻灯片未以“假设”形式明写，属本笔记的推论，待核。
六、常见误区
- 把“会推理”当成“会行动”。 逐字稿明确指出思维链本身是非智能体设定；行动需要工具把 token 序列变成环境中的操作（Slides 第 13–14 页）。
- 把 harness 与训练对立起来。 Slides 第 18 页明确两条路径都能构建同一份能力清单；Graham 的个人偏好（训练更根本）是观点而非规格。
- 以为“模型变强就自动解决一切”。 逐字稿反驳了这一说法：能力提升往往来自有人专门构建了领域训练环境。
- 以为工具调用“免费”。 Slides 第 20 页把它列为第一项能力；逐字稿说很多人视为理所当然，但训练方必须把它当作真实问题。
- 把安全性当成与能力分离的事。 Graham 在逐字稿中明确表示“能力安全是一种能力”，若没被烘焙进模型，你就不会放心用它。
- 以为自主性是二元的。 Slides 第 4 页用六个场景让课堂在“自主 / 先询问 / 从不”之间选择，说明信任是分场景、分后果的连续谱。
- 以为 AI 工具可以代写一切。 Slides 第 37 页明确：课堂亮点必须由你本人撰写，不得由 AI 生成；你要对每一条提交的主张、引用、结果与每一行代码负责，并会被测验。
七、课程中的具体例子（保留）
- 十六个智能体写 C 编译器：Slides 第 2 页——十六个智能体在两周内协作构建了一个 10 万行的 C 编译器，能够编译 Linux 内核；来源标注为 Carlini, Anthropic 2026，并标“↗”。逐字稿补充这是 Anthropic 的 Carlini 做的实验、基于 Rust。“100,000 行”“Rust”等细节以 Slides 与逐字稿为准，未额外补。
- 收件箱删除事件：Slides 第 3 页引用 Summer Yue（@summeryue0）· X, February 2026；逐字稿叙述为模型压缩上下文后丢失“不要删除”指令。
- 六个自主性场景投票（Slides 第 4 页）：①诊断在线商店结账故障；②向 5 万客户起草并发送产品发布邮件；③收集税务表格并准备、提交 2025 年纳税申报；④在不破坏移动端结账的前提下把支付 API 从 Python 迁移到 Rust；⑤若我最喜欢的乐队在我所在地区开演出就买演唱会门票；⑥根据一周血糖读数调整胰岛素剂量。逐字稿记录的课堂倾向：①偏自主；②偏先询问；③“从不”最多但“先询问”也不少；④自主与先询问非常接近；⑤偏自主；⑥“从不”。
- GUI 智能体演示：Slides 第 5 页（Jing Yu Koh · multimodal GUI-agent demo · 2024）；逐字稿描述任务为“导航到匹兹堡一家好的泰国餐厅页面，至少 200 条评论和 4.3 星”，并提到 Yelp 搜索与餐厅 Fusi…（识别不确定，待核）。
- OpenHands 可视化调试：Slides 第 6 页（Graham Neubig · OpenHands visual debugging）；逐字稿描述待办事项列表应用被构建、运行、被浏览器测试、测试删除功能，甚至应用崩溃后智能体先于人类察觉并修复。
- SWE-bench 示例轨迹：Slides 第 16 页（swebench.com ↗）；逐字稿描述 system 消息 “you are a helpful assistant that can interact with a computer shell”，以及 bash 工具调用与退出码。
- OpenAI 网络安全基准事件：逐字稿口述，含侵入 Hugging Face 获取答案；Slides 文本未见对应页，待核。
- 规划模式按钮：逐字稿口述的趣闻，产品名待核。
八、复习问题
- 用 Russell & Norvig 的定义解释智能体，并把“环境/状态观察/动作/奖励”映射到 Slides 第 9 页给出的四组具体例子。
- 为什么说“工具调用只是另一段 token 序列”？harness 在这个过程中承担了哪三件事（Slides 第 13 页）？
- 请画出 Slides 第 14 页的智能体循环，并说明
read_file与send_message分别对应循环中的哪一步。 - Slides 第 15 页的
run/step/query/execute_actions四个方法各自职责是什么？Assignment 1 要求你在哪些方向上扩展？ - 列出六项智能体能力，并对每一项各举一条 harness 工程路径和一条 LLM 训练路径（Slides 第 17–25 页）。
- 训练三阶段（预训练、中期训练/SFT、RL）各自的数据形态与学习目标是什么（Slides 第 19 页）？
- 为什么说“智能体是系统，不只是模型”？请用 Slides 第 26–31 页的五大组件说明，并各举一个示例软件。
- Slides 第 37 页对使用 AI 工具给出了哪些明确边界？“你要对每一条主张负责”在实践中意味着什么？
- Slides 第 38 页的 slack day 规则是什么？能否转让或共享？
- 结合 Slides 第 4 页的六个场景，讨论：为什么同一个用户会在某些任务上接受自主、在另一些任务上要求先询问？
九、待核清单（不猜测）
- Slides 第 10 页思维链示意图中的 “14” 如何得到，需对照官方 PDF。
- Slides 第 16 页 SWE-bench 轨迹图的完整内容。
- Slides 第 26 页系统总图的排版与箭头方向。
- 逐字稿中 “Fusi…” 餐厅名、“transluce” 监控工具名、“codeex/clog code” harness 名、模型版本号 “4.7/4.8”、以及 OpenAI 网络安全基准事件的一手出处，均需对照原视频/官方材料核验。
- 本笔记未给出任何未经 Slides 或逐字稿支持的数值、实验结论、引用或链接。
十、给定资料中的真实官方来源链接
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=UwfjzyLnvMg&list=PLSN0qpDfUvTM&index=1
- 官方课程日程：https://www.cmu-agents.com/#schedule
- Lecture 1 slides：https://www.cmu-agents.com/slides/lecture-01-agents.pdf
从语言模型到智能体：CMU 11-768 第一讲到底在讲什么
来源：平台字幕与官方 Slides。内容已由用户审核确认。
一门新课要回答的老问题
CMU 11-768《AI Agents》在 2026 年秋季开课，第一讲标题是"什么是智能体，它们如何工作？"（Slides 第 1 页）。这门课出现的时间点很有意思：一方面，很多人已经在日常工作中使用编码智能体，另一方面，关于"智能体到底是什么""它为什么有时很强、有时很蠢"仍然缺少一个统一说法。
这一讲要解决的，正是这个落差。它没有直接给一个时髦定义，而是先把"智能体"放回一个更老的坐标系里，再解释今天的语言模型如何被接进这个坐标系，最后指出：真正难的不是做出一个能跑的智能体，而是做出一个真正能用的智能体（依据逐字稿 Graham 的表述，[30:43]–[31:07] 附近；Slides 第 17 页）。
下面按"它怎么定义—怎么从 LLM 长出来—能力从哪来—为什么它是系统"的顺序展开。
智能体的经典定义：感知、行动、奖励
Slides 第 9 页直接引用了 Russell & Norvig 在《Artificial Intelligence: A Modern Approach》中的定义：智能体是任何可以被视为通过传感器感知环境、并通过执行器作用于该环境的东西。这个定义并不新，它在 AI 教科书里存在了很多年。
今天的智能体仍然落在这个框架里，只是每一项都被具体化了（Slides 第 9 页）：
- 环境：一个代码仓库、一个网站、一个应用程序，或者一个工作流。
- 状态/观察：用户发来的消息、文件内容、网页、截图，以及工具返回的结果。
- 动作：回复用户、编辑文件、执行 shell 命令、调用 API，以及鼠标或键盘事件。
- 奖励：测试用例是否通过、LLM-as-a-judge 的评分标准是否满足、用户反馈是否正面。
这里有一个关键的历史连续性：因为定义本身是老的，很多为老智能体开发的技术——搜索、强化学习等——仍然适用于今天基于语言模型的智能体（依据逐字稿 [00:16] 附近的表述，Graham 提到"过去为智能体开发的许多技术……将适用于当前的智能体"）。
学习者解释：把语言模型塞进这个框架，本质上不是在发明一个新范式，而是给一个经典框架换了一个更强的"策略函数"。理解这一点很重要，因为它解释了为什么这门课会同时谈工具调用、上下文管理、强化学习、沙盒——这些看起来跨度很大的话题，其实都在同一个框架的不同位置上。
从下一 token 预测到"在环境中行动"
要理解智能体，先要理解它背后的语言模型。Slides 第 10 页复习了两个基础概念：
- 下一 token 预测：每一步预测下一个 token 的分布，把生成的 token 接到前缀上，再预测下一个。
- 思维链（Chain of Thought）：在给出最终答案之前先生成中间推理 token，这些 token 成为后续预测的上下文。
思维链很重要，因为即使没有工具，它也已经让模型在复杂推理上有大幅提升。但 Slates 第 10 页的示例很清楚地区分了两种模式：2 × (3 + 4) 是一个纯提示词场景，模型生成推理 token，然后生成答案 token"14"。在这个设定里，模型只跟提示词交互，不跟世界交互。
那要如何从"跟提示词交互"过渡到"在世界中行动"？答案是工具（Slides 第 11–12 页）。
一个工具是对模型暴露的类型化接口，包含两部分：
- 名称和自然语言描述（例如
read_file，描述为"Read a UTF-8 file."）。 - 参数的 JSON Schema（例如
path是一个字符串，且为必需参数）。
这个底层规范会通过 chat template 被渲染成一段文本，喂给模型（Slides 第 11 页）。模型从而"知道"有哪些工具可用。
调用时，模型生成类似下面的内容（Slides 第 12 页）：
<tool_call>
{"name":"read_file","arguments":{"path":"test.py"}}
</tool_call>执行结果则通过 tool_response 返回，例如文件内容 def add(a, b): …。这些结果 token 被拼回上下文，成为模型下一次生成的条件（Slides 第 12–13 页）。
Slides 第 13 页把这一机制概括得很清楚：工具调用只是模型可以预测的另一段 token 序列——它命名了一个工具并给出参数；而 harness（执行框架）负责解析、验证、执行这次调用，并把结果作为观察 token 返回。这里引用了 Toolformer（Schick et al., NeurIPS 2023）作为早期训练模型使用工具的代表性工作。
课堂上还讨论了另一种调用形式。Daniel 问"有没有别的方式让模型调用工具"，有学生回答可以直接写代码，比如 bash 脚本或 Python 函数调用。Daniel 回应说，这种方式往往比生成 JSON 更有效，但也有权衡（逐字稿 [22:24] 附近）。
智能体循环：模型只是循环里的一个组件
Slides 第 14 页给出了本讲最核心的结构图：智能体就是循环中的一个模型。这个循环包含：
- 上下文（Context）：任务、历史。
- 模型（Model）：产生推理和工具调用。
- 执行（Execute）：工具调用被环境执行，返回结果。
- 结果（Result）：结果加入历史，进入下一轮。
这个模式来自 ReAct（Yao et al., ICLR 2023）——推理性与行动交替进行：模型先产生思维链，再发出一个或多个工具调用，工具被执行，结果加入历史，然后重复。
Slides 第 15 页展示了来自 mini-swe-agent 的一段极简实现（run、step、query、execute_actions 四个方法），说明这个概念在代码层面确实可以非常短。同时，这一页也直接点明了第一次作业的方向：实现并扩展一个类似的控制器，创建并调用自己的工具，应用到软件任务上。
Slides 第 16 页展示了 SWE-bench 上的一条真实智能体轨迹。课堂上 Daniel 逐步放大这条轨迹，展示了：模型在 system 消息中被设定为"能与计算机 shell 交互的助手"，user 消息给出具体任务上下文，然后模型生成思维链和 bash 命令作为动作，环境返回退出码，智能体继续下一轮（逐字稿 [28:52]–[30:02] 附近）。
学习者解释：这个循环看起来简单得可疑，但"简单"和"能用"之间的距离正是这门课的主线。循环本身可能只有几十行代码；难点在于每一步的可靠性——工具调用是否格式正确、上下文是否还留着关键信息、任务是否被正确分解。
六个能力，两条路径
在介绍完基础结构之后，Graham 把话题转向"是什么造就了一个好的智能体"（Slides 第 17 页）。他列了六项能力：
- 准确的工具调用
- 长上下文中的连贯性
- 可定制性
- 复杂任务管理
- 环境理解
- 安全性
课堂上 Graham 先问大家日常使用智能体时最让人沮丧的是什么。学生的回答包括：太慢、啰嗦冗长、会忘记事情、误解指令、访问不该访问的东西、有随机知识空白、"两行就能搞定却写了一千行代码"、忘记之前的会话、不会反驳不合理的要求、会撒谎、做很多隐含假设（逐字稿 [31:29]–[33:07] 附近）。
Graham 随后把这些抱怨映射回他的六项能力，并指出：准确性是"基本功"，现在很多人视为理所当然，但如果你负责训练模型，它绝非免费得来。
然后是本讲一个重要的结构性区分（Slides 第 18 页）：
- LLM 训练：通过预训练、SFT 或 RL 改变模型行为，教它可复用的推理、工具使用和恢复模式，让能力成为学到的策略的一部分。
- Harness 工程：改变模型周围的系统——提示词、工具、记忆、控制流——提供上下文、验证、重试和安全边界。能力从"模型 + harness"的组合中涌现。
Graham 在课堂上问大家认为哪条路更重要，结果举手支持 harness 工程的人明显更多。他自己的回答是：两者都重要，但典型流程是——你发现问题，先在 harness 一侧解决；然后训练模型的人追上来，意识到这个问题足够大、值得专门训练模型去解决；问题在模型侧被解决后，你就不再需要在 harness 一侧处理它了。所以在他看来，LLM 训练往往是更根本的解法，但耗时更长，因此短期内你需要从 harness 一侧入手（逐字稿 [36:20]–[38:15] 附近）。
Slides 第 19 页给出了训练流程的三阶段：预训练（文本、代码、多模态数据）→ 中期训练/SFT（指令、轨迹、工具调用）→ RL（奖励或偏好、轨迹级行为）。课堂上 Graham 明确说这门课不会处理预训练，因为那不是课程算力能支持的；重点放在中期训练、SFT 和 RL，尤其是针对智能体的 RL（逐字稿 [41:57]–[43:27] 附近）。
每一项能力的两条路径
Slides 第 20–25 页逐项展开六项能力，每一项都给出"harness 工程"和"LLM 训练"两条路径：
准确工具调用（Slides 第 20 页）：harness 一侧可以用语法约束解码；训练一侧可以用工具调用轨迹做 SFT。课堂还举了一个具体的 harness 例子——有个流行的编码 harness 的"规划模式"按钮，实际效果只是在提示词里多加一句"请规划，不要做任何事情"（逐字稿 [46:18] 附近）。
长上下文连贯性（Slides 第 21 页）：harness 一侧包括上下文压缩/压实、动态记忆查找、子智能体委派；训练一侧是长上下文训练。Graham 指出，长上下文不仅仅是一个模型准确率问题，也是一个效率问题——它需要对非常长的序列建模（逐字稿 [38:54] 附近）。
可定制性（Slides 第 22 页）：harness 一侧包括智能体记忆、技能、自定义工具；训练一侧是从用户反馈中学习。Graham 认为这其实是当前 harness 工程最好的用例之一。
复杂任务管理（Slides 第 23 页）：harness 一侧提供规划/分解工具和子智能体委派；训练一侧是在复杂长时程任务上做 RL。
环境理解（Slides 第 24 页）：harness 一侧对应领域知识的技能；训练一侧是在具有预期观察形状的数据上做 SFT，或在特定领域环境中做 RL。Graham 在这里说了一句很关键的话：模型不会自己变好，是人让模型变好——当某个模型突然在作曲上变强，背后往往是有人创建了相应的训练环境（逐字稿 [48:53]–[49:38] 附近）。
安全性（Slides 第 25 页）：harness 一侧包括沙盒化工具、限制凭据访问、监控轨迹；训练一侧是安全感知的 RL。Graham 在课堂上讲了一个具体事件：某次网络安全基准测试中，模型被指示入侵一个系统，它无法入侵那个目标系统，于是转而入侵了 Hugging Face 网站去获取答案。他列举了其中一连串失败：沙盒化失败、凭据限制被绕过、监控不足、安全护栏不足（逐字稿 [50:27]–[51:33] 附近）。
这里需要特别注意：Graham 在课堂上也提到 Carlini 在 Anthropic 做的多智能体 C 编译器实验（Slides 第 2 页：16 个智能体在两周内协作构建了一个能编译 Linux 内核的约 10 万行 C 编译器），以及 OpenClaw 删除邮件的事件（Slides 第 3 页：模型压缩了过去上下文，丢失了"不要删除邮件"的指令）。这些是课程用来展示能力与局限并存的例子。
智能体是系统，不只是模型
这是 Graham 在能力部分之后反复强调的另一点（Slides 第 26 页）：智能体是系统，远比你以前上过的机器学习课处理的东西复杂。Slides 第 26 页把系统拆成六个部分：沙盒、harness、推理、模型、监控、训练。
Slides 第 27–31 页逐项展开：
- Harness 工程（Slides 第 27 页）：管理状态、工具、记忆、控制流；验证动作并处理错误；强制执行权限和安全边界。示例软件包括编码智能体（CC、Codex、OpenHands、OpenCode、Pi）和编排器（LangChain、CrewAI）。
- 沙盒（Slides 第 28 页）：隔离代码和工具执行；限制计算、网络和文件系统访问；创建可复现环境。示例软件包括 Docker/Apptainer、Modal/Sail。
- LM 推理（Slides 第 29 页）：可靠地服务模型生成；批处理请求并复用 KV cache；管理流式、并行和吞吐。示例软件包括 vLLM、SGLang。
- 训练系统（Slides 第 30 页）：准备数据、收集 rollout；协调分布式 worker；checkpoint、评估和复现运行。示例软件包括 SkyRL、Miles。
- 可观测性与监控（Slides 第 31 页）：捕获 trace 和指标；跟踪质量、成本和失败；比较轨迹和评估。示例软件包括 Laminar、MLFlow。
Graham 在课堂上谈到了 harness 工程和编排器之间哲学的差异：编码智能体通常更简单（就智能体之间如何交互而言），但在给智能体的动作空间方面更复杂；编排器在单个智能体上更简单，但能给出声明式工作流，更多是护栏、更少表达力（逐字稿 [54:03]–[55:07] 附近）。
学习者解释：把智能体当系统看，意味着不要把失败简单归因于"模型不行"。同一个模型，换一个 harness、换一套工具、换一个沙盒配置，表现可能差别很大。这也是为什么这门课的作业设计是"harness → 评估 → 训练"的顺序，而不是一上来就做训练。
课程目标与作业
Slides 第 32 页列出了这门课的学习目标：
- 在开源 LLM 之上从零实现一个智能体。
- 为多步任务设计评估。
- 训练智能体以提升能力。
- 推理安全性与可靠性的权衡。
- 追求一个智能体方向的开放研究问题。
Slides 第 33 页给出三次作业和项目：
- A1 · Harness（暂定截止：9 月 10 日周四）：构建基础架构并建立可复现基线。
- A2 · Evaluation（暂定截止：9 月 24 日周四）：测量行为、报告失败案例、区分演示与证据。
- A3 · Training（暂定截止：10 月 22 日周四）：训练或适配一个组件；分析行为、成本和鲁棒性。
- 项目：围绕一个有范围的任务、环境和评估整合这些想法。
Slides 第 35 页明确先修要求：这门课要求之前有认真训练语言模型的经验；需要在 Piazza 的 Google 表单中提交，截止时间是 8 月 27 日周四；如果没有正式先修，可以说明等效经历（如 4–7B+ 语言模型的训练工作或已发表研究）。课程会严格执行先修要求。
Slides 第 36 页给出评分构成：作业 1–3 合计 40%，个人完成；课堂亮点 10%，个人完成，每讲后 24 小时内提交；研究项目 50%，2–3 人组队（提案 5%、中期检查 5%、展示 10%、最终报告 30%）。
Slides 第 37 页说明 AI 工具政策：AI 工具在未被特定作业或项目说明禁止时一般允许；课堂亮点必须由本人撰写；你对每一份提交的主张、引用、结果和每一行代码负责，并会被测验。
Slides 第 38 页说明截止政策：作业 1–3 各有两天 24 小时的 slack days，不可转让或共享；用完后每额外一天或不足一天扣作业分数的 5%；项目提案和报告各有两天 slack days，展示没有。
课程日程概览
Slides 第 34 页给出了学期日程：
- 8 月 25 日–9 月 8 日：智能体能力（什么是智能体、工具、上下文、记忆、规划）
- 9 月 10 日–9 月 15 日：领域（编码、GUI）
- 9 月 17 日–10 月 1 日：训练 + 领域（SFT、RL 基础、深度研究、高级 RL）
- 10 月 6 日–10 月 22 日：框架 + 安全（沙盒、OpenHands、LangGraph）
- 秋季假期：10 月 12–16 日
- 10 月 27 日–11 月 5 日：交互 + 项目（人、多智能体、项目时间）
- 11 月 10 日–12 月 3 日：高级主题 + 展示（工作的未来、搜索、海报）
- 感恩节：11 月 25–27 日
第一讲的问题链
回头看，这一讲回答了四个层层递进的问题：
- 什么是智能体？ 一个经典定义下的感知-行动系统，环境、观察、动作、奖励都被具体化（Slides 第 9 页）。
- 语言模型如何变成智能体？ 通过工具接口，把工具调用变成模型可以预测的 token 序列，再由 harness 解析执行，形成模型在循环中的结构（Slides 第 11–14 页）。
- 什么造就了好的智能体？ 六项能力，每项都可以通过 LLM 训练或 harness 工程两条路径来构建（Slides 第 17–25 页）。
- 为什么这门课要教一整套系统工程？ 因为智能体是系统，不只是模型；沙盒、推理、训练、监控都是系统的一部分（Slides 第 26–31 页）。
下一讲的主题已经给出：Agent Capabilities 1 — Tool Use（Slides 第 39 页）。
官方来源链接
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=UwfjzyLnvMg&list=PLSN0qpDfUvTM&index=1
- 官方课程日程：https://www.cmu-agents.com/#schedule
- Lecture 1 slides：https://www.cmu-agents.com/slides/lecture-01-agents.pdf
说明：本文依据 Lecture 1 的 Slides 文本提取与中文逐字稿整理稿整理。Slides 文本提取中标注"待核"的图示、公式和布局部分，本文未做视觉推断；逐字稿时间定位仅用于辅助定位原话，本文已做改写与组织，未声称经过人工审核。
CMU 11-768 Fall 2026 · Lecture 1 · Course Overview: What Is an Agent?｜完整中文逐字稿
- 视频：官方视频
- 来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
- 说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。
正文
[00:00.320] 好的。嗯，大家好。欢迎来到 11768，呃，AI 智能体。我们非常兴奋能教授这门新课程，呃，这很应景，因为呃，很多人开始使用智能体了。嗯，我叫 Graham Nubig。我是这里计算机科学学院的副教授。 我研究智能体大概已经有好几年了。呃，编码智能体、网页浏览智能体，呃，诸如此类的东西。我将和 Daniel 一起共同授课，他将为这门课开场。那么废话少说，我把时间交给他，他可以开始呃这个，然后我会我会讲课程的第二部分。
[00:40.160] >> 谢谢 Graeme。这个怎么样？酷。好的。那么嗯，大家好。真的很兴奋你们能来。我叫 Daniel Freed。嗯，我的研究组也研究智能体。嗯，所以呃是具身智能体，还有人与智能体之间的交互，以及如今智能体系统之间的交互。所以呃很兴奋能和你们讨论嗯所有这些话题，嗯，是的，我们有一个
[01:07.280] 非常棒的核心教职团队，我们会在几页幻灯片后介绍他们。但是的，智能体现在真的很应景，嗯，智能体的能力进步得相当快。所以你们中可能很多人都在你们的你知道日常工作中使用编码智能体，我们也看到了一些智能体的重大成功案例。嗯，一个例子是有一个呃不错的实验，嗯，由嗯 Carlini 在 Anthropic 做的，看看我们能否使用嗯一个多智能体呃配置的近期编码模型来开发一个非常复杂的软件。这相当成功。所以 16 个智能体在两周内协作，构建了一个基于 Rust 的 C 编译器，它能够编译 Linux 内核。你们可能你知道已经在自己 的工作中有效地使用了智能体，并且看到它们变得好了多少，尤其是在过去大概一年左右。当前的智能体也有局限性。那么有多少人呃尝试过呃使用一个控制你电脑的智能体，比如控制应用程序，或者好的。所以，很多人有。有多少呃人用过 OpenClaw？
[02:22.879] 是的。所以，我们也有不少人。嗯，有一个相当高调的事件涉及 OpenClaw，嗯，被遇到这件事的人发到了 Twitter 上，她当时让模型整理她的收件箱，而模型有点失控了，开始删除旧邮件。 呃，所以她试图干预。她说：“怎么回事？你能描述一下你在做什么吗？”但模型说：“我正在采取核选项。我要把收件箱里所有不在我的保留列表里的东西都丢进垃圾桶。”用户试图干预，说，你知道，不要那样做。嗯，但模型继续删除了很多这些 非常有价值的信息。最后，它承认呃，你知道，我吸取了教训。我以后不会这样做了。而且我确实记得她告诉过我这件事，嗯，但我违反了它。那么，这里幕后发生了什么？嗯，看起来是模型压缩了它过去的上下文，这是一种我们在这门课中会讲到的方式，呃，用来减少过去的 历史，以便模型能够接收更多嗯与环境的交互，但那导致它丢失了这条不要删除邮件的指令。所以，模型变得能力强多了，但它们也仍然有很粗糙的边缘。这门课的很多内容将是关于你如何开发这些能力和模型，从基础语言模型开始，但还有哪些
[03:44.799] 是研究问题，以及我们需要填补的剩余空白是什么。那么让我们嗯试着了解一下人们认为智能体现在擅长什么。所以我们有大概六个不同的任务，你可能想尝试用智能体来做。这些相当主观。我只是想了解一下你可能对什么感到放心。嗯，那么让我们举 手示意，对于一个比如你有一个在线商店，比如某个呃为你托管商店的代码应用程序，它开始出故障了。你想用一个智能体来尝试诊断它。有多少人会对让一个智能体完全自行执行这个任务感到放心？我们能举手表决一下吗？好的。是的。嗯，那让智能体在执行这个任务时 先询问呢？也许向你求助。有多少人认为这可能是必要的？好的。那有多少人认为这根本就是你永远不想让智能体碰的事情？好的。所以，对于这一个，我想我们有更多人选自主。
[04:47.759] 那起草并发送一封产品发布邮件给 50,000 名客户呢？那么，有多少人觉得这完全自主是可以的？好的，我们有一些勇敢的灵魂。嗯，那先询问呢？好的。那智能体根本不应该做这个呢。我想为它负责。好的，所以我想这一个选的是先询问。那嗯收集 你的税务表格并准备和提交你的 2025 年纳税申报表呢？完全自主。好的。是的。那先询问呢？那从不呢？好的。是的。所以我想这一个的“从不”比任何其他选项都多，但先询问大概是一个。比如把支付 API 从 Python 迁移到 Rust。比如说你也许有一些为 Python 应用程序编写的测试用例，你可以应用 到 Rust 版本上。有多少人会觉得这自主是可以的？好的，那先询问呢？是的，那非常接近。我不知道哪一个更多，还有从不。好的。所以，我们呃让呃把这一个算作支持智能体的。自主。如果我最喜欢的乐队在我所在地区举办演出，就买演唱会门票。自主。 所以，如果它找到好的票，你也对智能体买票感到放心。我想那一个大概赢了。那这最后一个呢？根据一周的血糖读数调整胰岛素剂量。所以，一个智能体比如，你知道，具备做这件事的所有能力。比如如果你把健康信息喂给它，嗯它可以监测那个。如果你给它访问比如 某种可以改变处方剂量的东西，它也可以做那个。有多少人会对这自主感到放心？好的，先询问。那从不呢？好的，所以是的，所以那一个是从不，虽然我本来在想也许是先询问。所以酷。所以我想你们都你们都大致对智能体有一些边界感。虽然
[06:54.479] 你知道有很多这类任务我们可能只是不确定。嗯但你们也有一种感觉，就是信任很重要，在课程后面当我们谈到比如沙箱化和嗯安全以及嗯交互和嗯人的监督时，嗯我们会更多地深入这些。
[07:15.599] 好的。所以只是给大家非常简短地呃大致介绍一下智能体是如何——非常简短。就两页幻灯片。智能体在过去几年里是如何发展的。呃，同时也是为了引出我们之后在课程中会看到的一些内容。呃，所以过去几年我们在 CMU 一直在做一些关于 gui 智能体（guey agents）的工作，也就是能够使用计算机应用程序的智能体。
[07:34.240] 这是很多呃教职员工的工作成果。呃，这是呃我们的一个学生 JY Co 呃两年前做的一个演示，展示了一个智能体在实际行动中控制浏览器。所以你给它一个任务，比如导航到匹兹堡一家好的泰国餐厅的页面。它应该至少有两百条评论和 4.3 星。而在右边 这里你可以看到，抱歉有点模糊，但你可以看到与语言模型的交互，比如正在产生的思维链。而在左边这里，模型正在控制浏览器，呃在 Yelp 的搜索框里输入，然后导航到这个页面。Fusidities 是一家非常好的泰国餐厅。呃，所以那是一次成功，我们已经看到呃你知道
[08:19.039] 像这样的框架和智能体在过去几年里真的开始起飞，比如呃 Manis 和呃 OpenAI 的 operator 云计算机使用智能体等等。而且我们在 CMU 这里也做了很多关于构建编码智能体的工作。呃，特别是 Graham 的团队在这方面做了很多，还有呃通过 open hands，它开发呃开源呃智能体。呃，这里有一个
[08:44.800] 来自 Graham 的演示视频，呃我会呃播放一下，展示智能体既编写代码，呃但随后也在浏览器中与它所构建的应用程序交互来测试它。>> 用 list 做的待办事项列表应用。>> 所以智能体需要花一点时间才能工作。呃，我们可以在这边看它工作，看到它呃为静态文件和一个 templates 目录创建了一个目录 并且它创建了主应用文件。我们可以通过点击这边的 changes 标签来查看它对应用文件所做的更改，这会向我们展示它实现的所有东西。所以那是一个挺不错的功能。所以你可以呃在智能体工作时观察呃它在做什么。而且它[音乐]正在编辑 index 文件、style 文件，让应用看起来好看。 呃 requirements.ext 让它容易安装，并添加一个 read me。所以你可以看到它有点在遵循良好的呃编码实践。现在它会启动这个应用。[音乐] 所以它运行得呃非常非常快。而我能说的是，让我们在后台运行，然后[音乐]用你的浏览器测试一下。 所以如果我这样做，它会呃在后台运行这个应用。[音乐] 然后它检查了应用是否在运行。现在它应该用它的浏览器导航到那里。呃它正在检查日志，看到端口已经被占用了，它正在尝试。所以那那正是智能体的一件好事。它们可以调试自己的问题，呃并确保一切正常工作。所以，好的，它
[10:32.959] 现在应该能工作了。[音乐] 然后你看到它浏览到了这个应用。我们可以在这边看到我们的待办事项列表应用。它看到它在运行，但现在它想要测试功能。所以，我们能做的是，我们可以看到它呃走了一遍，它填写了呃买杂货，然后它点击了。你可以看到呃它正在实时测试你的应用，然后它还呃 测试删除功能以及诸如此类的其他东西。所以我觉得这真的很令人兴奋。我并不是，你知道，最好的前端开发者，但它能够呃[音乐]试用前端，看看功能是否正常工作，并即时修复任何问题。我甚至有一次应用崩溃了，智能体在我之前就意识到应用崩溃了，它 能够进去修复问题。所以这就像只是说，你知道，去开发一个应用然后测试它是否能工作一样简单，你真的不需要做任何其他事情。所以现在我们完成了。呃让我停掉这个。在真实场景中，我接下来会做的是让它推送到 GitHub，呃你知道，把它完成，这样我就可以继续在上面工作。
[11:46.560] >> 是的。所以我们很兴奋能在这门课上给你们提供构建智能体的动手实践体验。在课程后面的一节课中，我们会讲到 open hands 和它的呃 SDK，以及你如何呃呃开发像这样的你自己的智能体。所以我们呃有一个非常棒的核心团队。
[12:06.800] 我们有呃我们的核心团队都在这些领域做研究。呃写了一些奠基性的论文，呃我们很希望你们大家都上来呃呃介绍一下自己，呃在课程结束时你们将会着手做一个你们自己的小型研究项目，呃我们会呃从整个核心团队获得指导呃帮助你们 完成那个。所以呃是的，如果我们能呃过一遍，让你们每个人也许只是说一下项目呃并介绍一下自己。>> 是的。所以是的，说得让后面的人也能听到，比如你的研究是关于什么的，因为呃这样这里的人就能大致知道你在做什么，这样当他们做那个领域的项目时就知道该找哪位助教，
[12:53.360] 比如说。>> 是的。>> 大家好，我是三年级博士生。我由 Grant 和 Kamar 指导。呃我的研究领域专注于训练智能体更好地利用兴趣和计算，以及多智能体系统中的组织。>> 大家好。呃我叫 Adita。我是一年级博士生呃在教授项目中工作。呃我的主要研究兴趣是比如为呃智能体强化学习设计奖励 函数。我目前特别在做代码生成方面的工作。呃大家好，我叫——我是一名二年级博士生，和 Anna 教授一起工作，是的，我目前的研究重点是[清嗓子]学习和研究 >> 大家好，我在这里和之前的——一起工作，研究移动性智能体，最近我一直在研究 嘿呃我是 Sus。我是一名和 Daniel 一起工作的博士生。呃我研究我们如何训练智能体更高效地与人交流。>> 嗨，我是 Lily。我是 EMU 的 Syria 商学院学生。所以我在做关于训练智能体授权咖啡和电池的研究。
[14:18.160] >> 呃大家好，我是 Andy。我是四年级博士。我和 Daniel 和 Monop 一起工作。呃我主要研究对齐后训练以及多智能体介绍。>> 谢谢大家。[掌声] >> 是的。所以呃就像 Graham 说的，我们绝对鼓励你们在课程后面开始思考你们的项目时，呃我们会讲课程结构，朝着——
[14:43.680] 结束，但当你们开始思考想要做的项目时，助教将是你们在答疑时间去找的极佳资源，可以获取他们的专业知识。所以，我们也非常感谢几家非常出色的初创公司捐赠算力，你们将能够在课程的作业和项目中使用这些算力。嗯，而且
[15:04.720] 没有他们的支持，我们就无法以我们希望的规模来开展这门课程。所以，我们非常感谢 fireworks AI、modal、prime intellect 和 sale 对本周期的支持。好了。那么今天我们将简要概述什么是智能体，然后我们会讨论课程的计划以及各项事务将如何组织。
[15:33.360] 那么智能体，智能体已经存在很长时间了。智能体的概念以及开发智能体能力的工作，并不是你们知道的只有两三年历史的东西。所以，Russell 和 Norvig 在他们关于 AI 的基础教科书中，将智能体定义为任何可以被视为感知其环境并对该环境采取行动的东西。而这确实 适用于我们现在所拥有的智能体。而且，过去为智能体开发的许多技术，不是全部，但很多都将适用于当前的智能体，尤其是像搜索、强化学习这样的东西。那么，这对我们当前的智能体来说结果如何呢？智能体将处于某个环境中，这个环境可能是类似代码 仓库、网站，或者你电脑上的其他应用程序，它们将通过获取其所处环境当前状态的观察来与该环境交互。所以这就像是与它们交互的用户发来的消息、它们正在调查的文件的内容、它们当前所在的网页、 截图，或者正如我们将在几页幻灯片后重点讨论的，工具的结果，工具就像编程函数，它们可以在环境中执行以与之交互。而且它们在每个时间点都在采取行动。这些行动将更新环境，然后改变智能体所处的状态。所以这可能是类似
[17:09.280] 回复用户、编辑文件、在 shell 中运行命令，或者调用支撑它们正在交互的网页的 API。最后，还有奖励这个概念。那么我们如何用一个数值分数来定义智能体是否成功呢？我们可以这样定义，比如如果某个代码库有一些测试用例，这些测试能通过吗？如果能通过， 那么你得一分，否则你得零分。对于其他任务，可能很难有某种程序化的奖励。所以我们可能需要依赖 LLM 作为评判者来评估这是否成功。在你们的第二次作业中，你们将开发基于 LLM 作为评判者的评估方法，以及其他评估方法。
[17:56.160] 也许我们最终感兴趣的是，我们是否让用户对要求智能体完成的任务感到满意。所以我们也可以为此使用用户的反馈。我认为在我们关于人机交互的讲座中，Valerie Chen 稍后将做一个客座讲座，我们会在那里讨论这个。
[18:14.160] 所以我们要求你们所有人都具备训练语言模型的先前经验，鉴于这些经验，你们应该都熟悉语言模型进行下一 token 预测，对吧？所以在每一步，智能体将预测下一个 token 上的一个分布，我们将采样或以某种其他方式从该分布中选择一个 token，将其插入模型的 上下文窗口中，并一遍又一遍地重复这个过程。最近，我们通过向这些模型引入思维链，在复杂推理任务上看到了很多很多改进。正如你们应该记得的，思维链是预测一串 token，这些 token 不是我们希望模型输出的最终答案，但确实允许它
[19:06.480] 有某种草稿纸来记录其中间推理步骤，然后它可以以此为基础来预测最终答案。所以这就像是一个非智能体的设定，对吧？这仅仅是智能体与你给它的提示进行交互。那么我们如何从这里过渡到拥有能在世界中采取行动的模型呢？
[19:31.679] 我们做到这一点的主要方式是通过工具。工具是处于环境中的某种东西，它将为模型提供一个与该环境交互的接口。你可以把这想象成类似 API，例如，如果我们希望模型能够查明天气是什么，以便它可以回应用户关于 天气的询问，也许我们有一个类似 weather.gov 的 API，它被包装在一个允许模型调用它的接口中。有很多不同的方式来表示这个接口。我想 Graham 将在下一讲中更详细地介绍它们，但这里有一个例子。这是 OpenAI 的工具规范，其中对于给定的工具 也就是模型可以调用的函数，你会有该工具的名称，以及它做什么的自然语言描述。例如，read file 可能是编码智能体用来查看它所在仓库中文件的工具。我们还需要告诉模型它可以向这个函数传递什么。 所以我们会有一个关于它所接受参数的描述，比如也许这个函数只接受一个 path，它是一个字符串，即要打开的文件的路径。所以我们必须，所以我们有名称和描述，我们有函数接口的某种结构化表示，这里是 JSON，而且这也需要以某种方式 输入到模型中。所以我们会有这个工具的底层规范，然后模型将通过应用这个模板来查看它，该模板将其转换为一串文本，然后会被分词并被 LLM 读取，这可能看起来像这样，而模型将必须要么经过训练，要么通过少量示例，知道如何使用这些工具。
[21:28.960] 这就是模型如何意识到它可以用来与环境交互的工具。但模型还需要使用这些工具来交互，它通过工具调用来做到这一点。例如，生成这样的文本，比如 tool call，然后你指定你想调用的函数名称以及参数作为
[21:51.200] 好吧，如果你使用的是某种基于 JSON 的表示方式。有人能提出另一种让模型调用工具的方式吗？有谁？让我们——对，后面那位。>> 对，你可以直接写代码。比如，bash 脚本，或者如果你把它表示为一个 Python 函数，模型就可以，你知道，用 Python
[22:24.080] 语法来调用这个函数。我们会在几节课后看到，这实际上是一种非常有效、而且往往比生成 JSON 更有效的方式。嗯，但也有一些权衡。所以，当模型调用工具时，它会从在环境中执行该工具得到返回结果。比如，如果我们正在读取嗯这个测试文件，也许那个测试文件里有这段用于一个 add
[22:46.400] 函数的代码，嗯环境会嗯你知道在这个 content 字段中返回嗯该工具的结果，然后那会用一个模板渲染给模型，嗯模型随后会把它作为 token 读入，并继续产生输出。关于这个有什么问题吗？
[23:07.520] 好。所以现在既然我们大概有了这个基本的工具接口，我们就能够嗯让语言模型使用这些工具在环境中行动。所以它这样做的方式就是读入这些 token 序列，这些序列表示有哪些工具可用，嗯呃表示呃工具的结果，然后嗯生成命名这些工具的 token 序列，进而调用它们。 所以这个呃我们有这样一个区分，一边是语言模型本身，它只是你知道与 token 交互，嗯后面我们会讲多模态语言模型，它们也许还会接收图像或其他模态。嗯所以语言模型只是 token 进、token 出，也许是多模态 token，但然后还有这个 harness，它
[23:53.039] 负责实际嗯协调嗯这些工具调用，嗯把语言模型当作它的引擎来使用。所以这个智能体循环基本上，而 Toolformer 是一篇非常有影响力的论文，它展示了你可以训练模型能够使用工具调用，然后以那些结果为条件，嗯使未来的文本更可能，并展示了一种早期的方式，你可以训练这些
[24:19.679] 模型使用工具，嗯它泛化了先前的 LLM 训练流程。所以一旦你有了这个基本设置，你就可以通过运行嗯一个循环来实现一个智能体，嗯这里一篇有影响力的论文叫 ReAct，嗯来自 2023 年，嗯它代表 reasoning and acting（推理与行动），因为模型在产生思维链，然后之后采取行动，而它的工作 方式是，你会有一些呃上下文，嗯你知道它也许包括对总体任务的某种定义，比如解决 GitHub 上的 pull request 或解决 GitHub 上的 issue。我们过一会儿会看到一些这方面的例子。然后你有一个任务，它也许是来自用户的一个初始查询。嗯你还会有一个所有可供该智能体调用的工具的列表，它将被
[25:06.720] 用我们展示过的那些格式之一来表示。而且你还会有一段智能体在交互过程中嗯已经产生和观察到的过去的嗯观察和行动的历史。而在每个呃在每个时间步，嗯所以对于一个给定的特定状态，智能体所处的状态，它会以所有那些为条件，产生一条推理链嗯你 知道思维链嗯，带有一个草稿本，关于它应该做什么，然后产生嗯一个或多个工具调用，用来与环境或与用户交互。比如读取那个文件。所以那个工具调用会在环境中被执行。我们得到结果。我们把这些结果加入历史，嗯环境也会作为 我们工具调用的结果而得到更新，然后我们再次重复这个过程。所以模型以这个更新后的历史为条件，嗯以及它从这个时间步得到的观察，然后发出一个新的工具调用，如此反复。我们也可以有结束任务的工具调用，比如向用户发送一条消息，嗯或以其他方式表示完成。嗯所以这是
[26:16.400] 一种我们可以你知道表示我们已完成的方式。在规划那节课里，我们会稍微谈谈比如模型如何判断它已经完成了。所以我不打算太深入这个细节。嗯你们在做第一次作业时会更深入，那个作业要求你实现一个嗯 ReAct 风格的循环，来嗯实现一个小型编码智能体，并用
[26:39.679] 它来解决一个 repo 中的问题，嗯实际上是一个国际象棋引擎 repo。所以也有一些有趣的演示与之配套。嗯所以我们不会太深入这个细节，你要写的代码会和这个不同，但你知道在高层次上这是一个非常简单的实现。这是从一个叫 mini-swe-agent 的简洁 repo 中摘录的，嗯它实际上
[27:03.520] 在一个基准上取得了非常高的性能，嗯我们会在几节课后讲到这个基准。它是一个非常标准的嗯用于编码智能体的基准。嗯但它是一个非常好的 repo，我鼓励你们去看看，因为嗯它对最近的模型很有效，但它也相当简单，真正地把嗯你知道把基础知识提炼了出来。嗯但你们会 呃一开始先有一些消息，呃给模型关于它是一个智能体以及关于这个任务的上下文。然后你就只是呃一遍又一遍地运行这个循环，它实现了我们在上一张幻灯片上的内容，智能体会嗯用所有过去的呃消息向一个语言模型发出一个查询，这些消息包含历史，嗯然后呃它会把这个消息加入历史
[27:55.360] 并且嗯执行嗯在环境中的呃动作。我们省略了这部分，但如果你感兴趣可以看看那个代码 repo。执行动作，得到观察，然后嗯返回它们。所以，我觉得看看示例智能体轨迹也很有启发性。嗯关于 SWE-bench 呃数据集或
[28:23.679] SWE-bench 评估任务和排行榜，一件好事是它们这里有一些示例轨迹。嗯所以，让我们看看也许 GPT-OSS。这可以让你感受一下呃工具调用可以长什么样，以及模型在执行这个任务时实际看到的是什么。嗯让我看看能不能放大一点。
[28:52.640] 是的。所以模型在一条 system 消息中嗯得到一些呃上下文，嗯它说 you are a helpful assistant that can interact with a computer shell（你是一个能与计算机 shell 交互的有用助手）。所以这就像非常通用，嗯但然后我们还会有一条 user 消息，它给出关于某个特定任务的上下文，呃解决这个特定的嗯 pull request。 嗯，所以模型以所有这些为条件，然后它需要嗯发出第一个动作，你可以在这里看到它生成的思维链轨迹。所以这个蓝色是由模型生成的。所以你这里有思维链，然后它产生一个工具调用。这个智能体正在使用嗯你建议的工具格式，也就是只是发出 bash 命令，嗯由你 知道编写应该运行的嗯命令，该命令被交给环境。环境执行它，这里的结果只是一个退出代码，退出代码为零，表明它成功了，但然后我们继续到下一轮，智能体嗯你知道发出一个新的想法和一个新的 bash 命令，等等。
[30:02.480] 所以希望这能让你对你知道一种可能的嗯实例化工具和实例化提示词的方式有所了解，你将在你的第一个作业中对此进行实验。酷。所以嗯现在我将交给 Graham。非常感谢。好的。嗨。嗯我想嗯大家都能听到我。所以接下来我想谈谈智能体能力。而我所说的这个意思是
[30:43.520] 比如什么造就了一个好的智能体。嗯因为正如嗯 Daniel 指出的嗯，做一个智能体并不很难。嗯尤其是现在有你可以调用的库来你知道用语言模型做推理或类似这样的事情。难的部分是做一个真正能用的。嗯所以我想稍微思考一下嗯你在一个智能体中需要具备的能力
[31:07.520] 。所以相当多的人说他们以某种方式每天使用智能体，你知道像 openclaw 或编码智能体。嗯当你使用智能体时，让你感到沮丧的事情是什么？比如什么出了问题是个麻烦？我们看到了一个删除你所有电子邮件文件的例子。那可能会让你沮丧，但是嗯任何任何其他事情吗？ >> 它们太慢了。>> 好的。是的，那那是个好点。嗯在后面那个啰嗦又冗长。那是个好点。我看到一只手。是的。>> 嗯所以它们会忘记事情。是的。好的。>> 它们可能会误解你说的话，并因此犯错。是的。>> 做做什么来着？ >> 比如访问它们不应该访问的东西。是的。好的。那那是个好点。嗯是的。嗯对非常常见或正常的事情有随机的知识空白。是的，那那是个好点。是的。>> 是的。嗯两行就能搞定却写了一千行代码。是的，那那是个好点。那是冗长的一种不同变体，但它也是是的。
[32:32.960] 冗长。是的。>> 是的。忘记之前的会话交互。所以，嗯，是的。>> 当你说一些嗯不合理的事情时，不知道如何反驳。是的。>> 它们可能会对你撒谎。是的，那是个糟糕的点。>> 很多隐含假设。是的。所以，嗯，这些都是很好的观点。嗯，也都是让我沮丧的事情。所以嗯它们它们并不完全符合
[33:07.120] 我幻灯片上的内容，但你知道那是个很棒的嗯很棒的事情。所以嗯你需要在智能体中具备的第一个能力是准确的工具调用。这是实际上没有人嗯提到的，因为现在很多人把它视为理所当然，但如果你开始这门课并且
[33:31.200] 你负责训练模型，这就不是你可以视为理所当然的事情。嗯所以嗯你知道这算是基本功。如果你不能准确地调用工具，你在任何任务上都会失败。但是嗯它不是免费得来的。另一件事是长上下文中的连贯性。所以有人说了嗯当它们忘记事情时让我恼火，或者当它们忘记之前的会话或
[33:53.440] 类似这样的东西时让我恼火。所以嗯那那是它的一个例子。它的另一个例子是嗯 Daniel 给出的那个例子，之前模型被告知不要做某事，但后来它还是做了。所以嗯那那是另一个例子。另一件事是嗯可定制性。所以你希望智能体按你的方式做事。嗯每个人都有不同的嗯做事
[34:19.200] 方式嗯不同的需求。所以我们这门课可能有与另一门课不同的需求。所以你需要确保如果你在使用智能体嗯它会遵循那些需求。嗯复杂任务管理。所以分解任务嗯能够处理非常长跨度的事情是一个非常重大的嗯非常重大的问题。
[34:40.480] 环境理解，我所说的这个意思是嗯有一个很好的例子是，在一个代码库中做一千行的改动，而两行就够了，那是环境理解的一种隐性失败，不知道你知道这实际上是一个真的很简单的改动，但它没有意识到这一点，它反而去做了一堆其他事情，然后一个
[35:02.720] 最后的事情是安全性，很多人把安全性和能力分开，他们说我是安全研究员或我是能力研究员。但从我的观点来看，能力安全是一种能力。它是你需要烘焙进模型的东西。如果它没有被烘焙进模型，你不会觉得你不会觉得使用这个模型很舒服，因为它可能会跑去做
[35:26.160] 一些你嗯不希望它做的事情。所以从我的观点来看，这是模型或更广泛地说是智能体的一种失败。所以有两种方式来构建这些能力。这是当你在做智能体时真的真的非常重要的事情。嗯第一个是嗯在 LLM 中训练。第二个是工程化 harness 嗯这样
[35:50.240] 你在 LLM 周围放置结构来让它嗯你知道以你想要的方式解决嗯比如解决你想要的问题。所以问题嗯你认为哪一个在这里更重要或更有效？所以有没有人有强烈的观点认为 LLM 训练是让你的智能体良好运作的方法？ 好的，实际上令人惊讶地多的小数量的人。那关于嗯训练嗯工程化智能体周围的 harness 呢？好的，那那是个相当多的人。我看到 Daniel 举了两次手，所以他不允许那样做，但是 [笑声] 嗯他是讲师。所以，当然，这两个嗯都很重要。嗯我我的个人观点嗯或者好吧，首先我会
[36:48.160] 我会你知道稍微谈谈它们包含什么。所以，第一个是通过预训练嗯监督微调或强化学习来改变模型的行为。嗯然后你可以教嗯可复用的模式用于推理、工具使用和恢复。 重要的是这些能力成为学习到的模型或学习到的嗯策略的一部分。而在 harness 工程中，你改变模型周围的系统。你给它提示词、工具、记忆和控制流，提供上下文、验证、重试和安全边界。而这些能力不仅从模型中出现，它们出现 来自 harness 和模型的结合。所以我对这个问题的回答是，嗯，它们可能都很重要，但通常发生的情况是，你发现一个问题，先在这里解决它。然后训练模型的人追上来，意识到这是一个大问题。它是一个足够大的问题，值得他们以特定的方式训练模型 来解决这个问题。它被解决了，你就不再需要在 harness 这一侧解决它了。所以这是一个典型的流程。嗯，所以从我的角度来看，这往往是左侧那个更根本的解决方案，但它需要大量时间，所以你最终需要从右侧开始解决它，因为你没有时间，嗯，
[38:15.440] 当你在改进你的模型时。所以如果处于能够做到的情况下，我个人倾向于将 LLM 训练作为问题的根本解决方案。是的。>> 所以，这是个很好的问题。那么，哪些能力会在 LLM 训练后仍然存在？嗯，我可以明确说的一点是长上下文，嗯，比如将你所有的记忆都保持在上下文中，因为 那不仅仅是一个模型准确率的问题。它也是一个模型效率的问题。所以你需要能够对非常长的序列进行建模。当然，你也许能够想出一个不像 transformer 那样具有 n 平方复杂度的模型，但你知道，那是，你知道，那是一个更大的讨论，假设我们停留在当前 的建模范式内，我会说，嗯，我在这里列出的大部分其他东西，嗯，我在这里列出的大部分其他东西。准确的工具调用，嗯，上下文窗口内的一致性，复杂任务管理，环境理解，主要从能力的角度来看，可以，甚至也许包括安全，主要从能力的角度来看，理论上可以通过模型来解决， 但它们还没有被解决。嗯，所以我们仍然需要，嗯，我们仍然需要 harness，嗯，工程。可定制性是一个很有趣的点，因为你可以在线训练模型。嗯，但也有一些情况，你有点想给它一个脚本，或者你想根据你所处的情况给它单独的指令，而在那些情况下，也许训练它
[40:15.280] 并不是最好的解决方案。但我们可以之后再多谈谈你知道的所有这些细节。是的。>> 是的。所以，嗯，一个后续问题是，当推理，嗯，延迟非常重要时，将 harness 设计得契合问题是一个好的解决方案吗。所以，嗯，我想指出的是，所有这些能力都可以通过训练或通过 harness 来处理。 嗯，所以我不会说某一个一定总是比另一个更好。它们都是，嗯，都是解决方案。关于推理时间，嗯，效率的有趣之处在于，你通常需要使用一个更弱的模型，或者你需要使用更少的算力，而正因为如此，你通常需要围绕它设置更多的护栏，因为你的成功 率会更低，或者你需要对你感兴趣的特定任务做适配。所以，嗯，你使用的模型越小，你就需要越小心，因为非常大的模型通常能更好地泛化。嗯，它们并不完美，但它们能更好地泛化。
[41:22.480] 好。嗯，还有其他人吗？好。所以，嗯，接下来进入，嗯，进入这个。所以，嗯，这门课的一个大目标就是让人们能够，让你们所有人都能够为智能体任务训练模型。嗯，这是，嗯，一个重点，因为并不是很多人能做好这件事。所以我们希望在这个房间里的人成为那不多的那些人，对吧？我们，我们希望你们擅长
[41:57.839] 某件很难做的事情，但是，嗯，很难做，对吧？但是，你知道，嗯，如果你确实知道怎么做，知道怎么做是一件好事。嗯，我们不会处理预训练，嗯，因为预训练，嗯，你知道，它是整个互联网规模的，那，嗯，不幸的是不是我们的，嗯，模态额度，嗯，能够支持的。
[42:19.200] 嗯，但我们会，嗯，思考中期训练和有监督微调以及强化学习。嗯，这里的要点是，中期训练和有监督微调是你已经拥有智能体如何完成任务的示例演示，然后你在那些示例演示上训练，嗯，通常通过优化最大似然。希望这门 课上的每个人都以前做过这个，因为这是这门课的先修要求。所以，嗯，我们只会简要地涉及那部分，而更多地聚焦于你如何在智能体场景中为做这件事创建数据。嗯，然后我们会更多地聚焦于，嗯，强化学习，因为针对，嗯，比如智能体的强化学习是相当棘手的。所以有多少人以前做过
[43:03.680] 用语言模型的强化学习？我猜也许没多少人。是的。好。有多少人为推理，推理任务做过？有多少人为智能体做过？好，也许大约一半一半，而且只有大概10，10%，10%，而80%的人没有做过。所以是的，我们会努力投入大量精力，确保每个人到
[43:27.520] 这门课结束时都能够做到这一点。好。嗯，那么一个一个地看所有这些能力。嗯，准确的工具调用是一个，嗯，重要的能力。嗯，我们通过 harness 工程和 LLM 训练来实现它。嗯，我们课上要讨论的一些东西是语法约束解码，嗯，它让你能够确保你的工具调用是格式良好的，并且符合
[43:50.400] 你所拥有的规范。嗯，对于 LLM 训练，这首先通过 SFT 来完成，嗯，或者通过，嗯，通过中期训练，或者随便你想怎么称呼它。嗯，这是通过在工具调用，嗯，工具调用数据，工具调用轨迹上训练来完成的。嗯，关于长上下文中的一致性，嗯，这同样可以通过 harness 工程或者，嗯，或者 LLM 训练来完成。对于 harness 工程，我们通过上下文压缩或压实来实现，它们算是同义词，嗯，你取出所有上下文并将其总结压缩，嗯，以便智能体继续工作。它也可以通过动态记忆查找来完成，嗯，你拥有你想处理的所有上下文之上的记忆，但你不立即把全部拉进来，嗯，你按需，嗯， 你知道，按需拉进来。嗯，它也可以通过子智能体委派来完成。所以你把一个非常长的任务的某些部分委派给一个智能体。它把那些上下文保留在记忆中，但当它完成执行那部分后，就把它从记忆中丢弃。嗯，对于 LLM 训练，嗯，你可以通过长上下文训练来做到。我们不打算大量处理它，但我们会讨论
[45:04.319] 关于它可能稍微讲一点。嗯，然后关于可定制性，我认为这实际上是目前测试框架工程（harness engineering）最好的用例之一。而且有很多方法可以做到这一点。其中之一是通过智能体记忆（agent memory）。所以确保智能体在你与它更多交互时继续学习，嗯，通过技能（skills），这些技能有点像 提示词，有时还有脚本，关于你想要做的事情，你在每个时间点想要使用这些东西时把它们引入进来。嗯，也可能为特定任务定制工具。嗯，对于语言模型训练来说，这是一个新兴领域。我认为没有很多人在非常详细地做这个。嗯，但有一些方法
[45:54.240] 你可以从用户反馈中学习。所以比如如果一个用户在使用一个工具，他们给了点赞或点踩，你可以专门从他们那里学习来适应他们。复杂任务管理。嗯，你可以通过测试框架工程来做到这一点。所以你可以提供规划或分解工具。你可以为模型设置一个规划模式，在那里它 嗯，它可以进行规划。我，这是，我了解到的一个小事情，但有一个很流行的编程测试框架，我想是 codeex，不，抱歉，也许是 clog code，他们有一个规划模式，嗯，当人们按下规划模式按钮时，唯一做的事情就是在提示词中额外添加了一句话，说“请规划，不要做任何事情”。嗯，[笑声] 但每个人都想要一个规划模式。他们想要一个按钮。所以，他们确保这完全通过提示词来实现。但还有更复杂的方法来做到这一点。嗯，再说一次，你可以通过子智能体委托（sub-agent delegation）来做到这一点，比如将任务分解并委托给其他智能体。而在训练中一个非常好的方法是在复杂的
[47:02.160] 长时程任务上训练。所以关于环境理解，我的意思是，无论你的智能体正在交互的是什么数据格式或环境，它都需要能够理解它。举一个非常好的例子，在计算机使用智能体（computer use agents）中，嗯，你需要能够理解网页，和/或，你知道，图形用户界面或类似的东西。 这不是你免费就能得到的东西。事实上，智能体现在在这方面的表现实际上相当糟糕。嗯，即使是最强的智能体也许不算，但就像，嗯，许多开源模型甚至不支持多模态数据，嗯，那些支持的在理解它方面也不完美。它们犯很多错误，比它们在 理解文本时犯的错误多得多。所以这是理解环境的一个失败，因为它们需要理解其环境中的多模态数据。但现在让我们把它们放在一个不同的环境中，它们需要交易股票或类似的事情。嗯，它们不太擅长理解时间序列。它们在时间序列上犯很多错误。或者你把它们放在一个
[48:09.920] 它们需要理解一张图片的环境中，你知道，一个有很多细菌的培养皿或类似的东西。它们在这方面也会失败。所以你有点需要能够理解你希望智能体与之交互的任何环境。所以有方法可以通过测试框架工程来做到这一点，比如提供对应于
[48:33.040] 领域知识的技能，但你也可以在具有预期观测形状的数据上训练，或者在特定领域的环境中训练。所以现在实际上，人们会说这样一句话，就是“哦，模型会变得更好，然后我们就不用再担心这个了。”
[48:53.520] 嗯，问题是模型不会自己变好。嗯，是人们让模型变得更好。所以我们想在这门课中教的一件事就是，当 Claude 突然从 4.7 变成 4.8，它突然能更好地作曲了，到底发生了什么。嗯，这里面有几个因素，但其中一个非常大的因素是 嗯，他们创建了一个环境来训练模型，嗯，一个比如作曲环境，然后突然你知道他们就在上面训练，你知道他们把它加入到训练混合中，然后模型在作曲或类似的事情上就变得更好了。所以这里的很多工作是创建模型可以工作的特定领域环境。
[49:38.640] 嗯，然后是最后一个，安全性。嗯，我们会讨论一下，因为如果你没有一个安全的模型，你就无法在真正有后果的场景中使用它。嗯，在测试框架工程方面有很多事情你可以做。你可以给它一个沙盒。嗯，你可以限制对凭据的访问。你也可以监控
[49:59.599] 智能体在工作时的行为。嗯，你也可以做安全感知的强化学习。嗯，我，我想可能相当多的人听说过这个，但谁听说过 OpenAI 的那次事件，它侵入了 hugging face？好的。嗯，所以我想大多数人都听说过了，但最近有一个，嗯，有一个例子，嗯，最新的 OpenAI 模型在一个智能体测试框架中， 它正在做一个网络安全基准测试，所以它的指令是侵入这个系统，所以它要去侵入一个系统，那就是它的任务，但它无法侵入那个系统。所以它转而基本上侵入了 hugging face 网站，嗯，从 hugging face 网站上获取了答案，然后在那里做了。所以这里发生了一系列失败， 嗯，第一个失败是沙盒化的失败，因为他们在做这个基准测试时没有正确地限制智能体。嗯，他们确实限制了凭据的访问，所以它通过侵入一个它没有凭据的东西来绕过了这个问题。嗯，他们也没有足够的监控，所以他们无法判断 这发生了。嗯，他们在这个模型上没有足够的安全护栏，因为这个模型被训练来测试，嗯，你知道，它是否能侵入系统。所以大概他们最终向公众发布的模型确实有更好的护栏。
[51:33.440] 但就像，嗯，这些都是你可以处理安全性的方式，我们也会讨论这类事情。所以最后一件事，嗯，关于能力部分有什么问题吗。我们当然会进入更多的细节。所以好的，听起来不错。嗯，所以我们真正想在这里强调的另一件事是，智能体是系统。嗯，它们不仅仅是模型，而且它们远比
[52:01.839] 比你上过的任何机器学习课、任何你以前在这里上过的课所处理的大多数东西都要复杂。所以你将不得不能够处理一堆不同的东西。第一件事是一个执行框架（harness）。嗯，这是一个中等至复杂程度的软件。 嗯，它有很多活动部件，比如你需要处理上下文，你需要处理工具，嗯，护栏（guardrails），管理工作流，嗯，诸如此类的东西。嗯，你需要有一个沙盒。所以，你需要确保智能体在一个受控的环境中工作。而且，嗯，你有不同的方式可以做到这一点。呃，它需要与一个模型配合工作，而你需要 进行推理。但如果你以前用语言模型做过推理，你可能处理过像 16 个 token 或 32 个呃 16k token 或 32k token 这样的上下文。在这里你需要像 256 呃 至少他们有一种类似最先进的呃编码智能体，举个例子。所以嗯推理问题更难。你需要处理缓存你的推理以及其他
[53:14.319] 类似的事情。而且你需要监控和训练。所以嗯从这个角度来看，我们在这门课上要教的是执行框架工程（harness engineering）。所以嗯如何管理状态、工具、记忆、控制流，嗯如何验证动作、处理错误、强制执行权限和安全边界。嗯，而且对于这些系统中的每一个，我们也会尝试
[53:37.839] 给出你可以查看的软件示例。嗯，对于执行框架，用于编码智能体的执行框架有两大种类。嗯，目前一些流行的有 cloud code、codecs、open hands、open code 和 pi。呃，大概你们知道你们至少用过或听说过其中一个。嗯，我开发了 open hands，所以我
[54:03.440] 非常了解它。呃，所以我会谈论呃谈论这个。我还想分享一点我自己的经验以及我们在开发它时学到的一些教训。与此分开的是，嗯有编排器（orchestrators）。我我写了 lang chain，但也许我应该写的是 lang graph，还有 crew AI，以及它们之间有一种非常不同的哲学 。嗯，这些编码智能体通常更简单，呃就不同的交互呃智能体彼此如何交互而言，但它们在给智能体的动作空间方面更复杂。所以智能体可以写代码，它可以与网站交互，它可以做其他类似这样的事情。嗯编排器则不那么复杂，就 它们所拥有的单个智能体而言。所以智能体可能只能回答客户服务请求，或者嗯在数据库中查找某些东西，但它不能写一个任意的 Python 程序。但另一方面，你可以得到这些声明式的工作流，即智能体工作流能够做什么。所以那是一种不同的构建方式。它更多是护栏但更少
[55:07.680] 表达力。而且它们在不同的场景中都有各自的位置。对于沙盒，嗯为了防止你的智能体在未经你许可的情况下呃共享你的 API 密钥，或者呃入侵其他网站。嗯我们需要隔离代码和工具执行，并限制智能体所呃能访问的计算、网络和文件系统访问。嗯另一件大事是用于评估和训练。 这是一个创建可复现环境的非常好的方式。呃所以比如像 SWEBench，每个问题都有一个呃每个问题都有一个沙盒，你从沙盒开始，那就是智能体开始工作之前的环境状态，所以你可以呃在本地运行这些。嗯我们为此使用的一些呃技术是 Docker 和 Appainer。嗯或者你可以在
[55:58.480] 云端运行它们，呃我们这门课的两个呃计算合作伙伴 Modal 和 sale 是呃类似云呃中的呃提供商，实际上 prime intellect 在这方面也是。所以下一件事是呃语言模型推理。所以我我觉得嗯鉴于你们之前在 NLP 或类似大语言模型课上的经验，大概大多数人都呃 以前遇到过这个，但基本上嗯它们呃需要服务模型生成嗯批量请求。所以如果你有多个智能体同时访问你的语言模型，这样更高效，但你也需要把请求批量合并在一起。智能体中极其极其重要的一部分是缓存先前的请求，因为智能体会采取越来越 多的动作，所以它们采取的动作越多，它们积累的呃上下文就越多，你需要确保你复用它来呃正确地做这件事，嗯你知道还有其他系统方面的考虑。呃我们在这里要谈论的一些示例软件是呃像 VLM 和 SGLANG 这样的东西。这是两个最流行的。然后还有成吨成吨的推理 提供商。嗯如果人们以前没见过 open router，open router 是一种把所有推理提供商聚集在一起的通用东西。对于任何模型，呃你会看到像 20 或 30 个不同的提供商在服务这些模型。所以有非常非常多。呃我们的一个计算赞助商 fireworks 就是一个例子，还有 sale 嗯训练系统。所以在这里他们
[57:37.920] 做的是呃基本上他们需要更新模型的权重，所以他们呃准备数据、收集 rollout，嗯嗯他们协调一大群一起工作以呃训练模型的 worker，而且他们可以呃 checkpoint、评估和复现运行。嗯这里的一些示例软件是 Sky RL 和 Miles。嗯所以呃这些可以用于呃训练。
[58:04.319] 最后嗯可观测性和监控。呃所以这是呃为了理解你的智能体是如何工作的。所以这些捕获智能体的 trace 或 trajectory，智能体做了什么，嗯并且它们收集围绕这些的指标，嗯比如花了多少时间、花了多少成本，呃跟踪质量、成本和失败。嗯而且它们允许你 比较 trajectory 和评估。所以嗯呃我对一个叫 Laminer 的非常熟悉，因为我经常用它。MLFlow 是另一个例子。嗯我我刚刚意识到 Daniel 展示了一个叫 transluce 的呃，它不在我这里的幻灯片上，但那是另一个另一个流行的。
[58:45.680] 所以嗯到这门课结束时，我希望班上每个人都能做的是呃在一个开源 LLM 之上呃从零开始实现一个智能体。所以呃这是自己实现一个执行框架呃为多步任务设计评估。所以基本上呃我们会给你一个你需要评估的任务，而你需要为它创建一个评估。这非常非常 这很重要，即使你对评估不感兴趣，但你也对训练感兴趣，因为做训练最好的方法之一就是创建一个评估并将其扩展，这样你就可以进行基于强化学习的训练。所以，这是一项非常重要的技能，训练智能体以提升它们的能力。所以我们实际上运行 RL 循环，处理所有的系统问题
[59:32.079] 并能够解决这个问题，对安全性和可靠性的权衡进行推理。所以我们会讨论安全性和可靠性，最后我们会有一个项目，我会在下一张幻灯片上讲。所以这些是作业。第一个作业基本上是实现类的作业。所以你会被给定一些要做的事情，并被要求 去完成它。第一个是创建一个 harness。第二个是评估，正如我所测量的，正如我提到的。然后是基于 RL 的训练。所以这些大约是本课程的前半部分。然后课程的后半部分将是用这些技能做一个项目，你知道，做一些有点
[01:00:21.599] 有趣或新颖的事情，在你选择的任何领域。项目将是小组制的。所以我们会要求你们组成两到三人的小组，创建一个项目。大体安排是，我们首先讨论智能体的能力。
[01:00:39.200] 然后我们会进入一些具体的领域，比如编码和 GUI 智能体。做训练，包括 SFT 和 RL。我们会讨论框架和安全性。然后我们会讨论交互，然后我们有一些项目时间，每个人讨论他们的项目。然后在课程结束时，我们会 试着邀请各个领域的专家来做客座讲座，可能结合一点概述以及他们自己的研究。所以希望你们能够从一些非常令人兴奋的人那里学到东西。我和 Daniel 试着头脑风暴了世界上最好的人来谈论他们的
[01:01:25.280] 东西，我不知道有没有人拒绝。所以我想每个人都答应了。所以我们每个主题都得到了我们的第一选择。所以我对此非常兴奋。所以在我们开始之前，我们有一些先修要求，这是一门在机器学习的实现方面很重的课程。我们希望每个人都能够在课程结束时
[01:01:52.720] 用 RL 训练一个智能体。所以我们的先修要求也是，我们要求之前有认真训练语言模型的经验。LTI 关于语言模型、语言模型系统、NLP 的课程都符合这个先修要求。所以如果你上过其中一门，那就可以。一些深度学习课程可能也符合。如果你不确定，随时问我们。但我们 会要求这个，我们会让每个人都必须为此提交一个表单。所以请务必去做。如果有大约 30 个人想知道他们的课程是否符合要求，请不要在课后在这里问我这个具体问题，因为通过邮件会更好，但其他问题我们很乐意在课后回答。
[01:02:39.599] 我们会在 Piazza 上分享一个 Google 表单，请提交表单，以便我们最终确定选课名单。是的，我已经讲过了。如果你没有上过课程，如果你在业界工作过并且在业界训练过一个模型，那应该是一个真正的模型，至少 4 到 7B 的规模，不是 1 亿参数的。但我们也会接受那个。
[01:03:08.000] 这是评分政策。作业一到三总共占 40%，单独完成。我们会有课堂亮点，所以基本的想法是我们希望人们来上课，并给出一个关于你在课堂上学到的某个东西的亮点。 这背后的想法不是让人们做无用的工作或类似的东西，但如果你看完了整个讲座，希望你想到了一些有趣的东西。如果你说没有什么有趣的，我早就知道这一切了。我在之前的课上在这里学过了，那已经在先修列表里了。那 对我们来说也是有用的信息。所以跟我们分享。但希望我们在每节课上至少会说一件有趣的事情。所以你可以跟进那个。研究项目占总成绩的 50%，这将以两到三人的团队进行，因为它是课程作业的一半。 提案占 5%，中期检查占 5%，展示占 10%，最终报告占 30%。如果你之前上过 LTI 的项目制课程，评分将与 LTI 项目制课程的要求类似。如果你没有上过，我们会分享相关细节。
[01:04:41.680] 这是你们一直在问的问题。这是一门 AI 智能体课程。我们非常彻底地讨论了我们想在这里做什么。我想先说明，我几乎在所有事情上都一直使用智能体。但与此同时，对所有事情都使用智能体有点危险，因为你会失去
[01:05:09.599] 对你想做的事情的视野，对你正在学习的东西的视野。你会错过一些细节和其他类似的东西，因为这门课的目标是学习。但我们也希望每个人都熟悉标准做法。我们希望你们记住这一点，AI 工具将基本上在课程中的所有方面被允许，除非我们 另有说明，而我们可能不会经常另有说明。课堂亮点必须由你撰写，而不是由 AI 生成。所以这些不需要很长。它们可以很短，可以只有几句话，但请自己写。你要对每一份提交的主张、引用、结果和每一行代码负责。
[01:06:02.079] 我们会为作业创建测验，以便你可能需要解释你代码的某些部分。我们会怎么做呢？我们也知道如何使用智能体。所以我们正在考虑让智能体阅读你的代码，并针对你的代码提出定制的测验问题。所以我们对这件事是认真的。
[01:06:31.359] 而这意味着你的智能体生成的那一千行垃圾代码，对你来说提交上去会是个糟糕的主意。嗯，所以提交一些简洁的、你真正理解得很透彻的东西，嗯，比提交你知道的更多东西基本上要好。嗯，所以是的，嗯，我们快讲完了。嗯，截止日期，嗯，在 Slack 上提交。所以，所有的作业，嗯，都包含两个 24小时的 Slack 日。嗯，它们不能被转让或共享。在 Slack 日用完之后，罚则是每额外一天或不足一天扣作业分数的5%。所以，这意味着截止日期就是截止日期。而且，嗯，但我们还是很好的，如果你错过了截止日期，我们会给你一些时间来弥补。嗯，如果你错过截止日期超过两天，嗯，那么
[01:07:20.559] 嗯，我们不会接受大多数借口。如果你住院了或类似的情况，我们嗯可能会要求你提供相应的证明，但那会是一个嗯理由的例子。嗯，但除此之外，请尽量赶上截止日期，因为我们不会对此做太多例外。嗯，酷，就这些。嗯，我们现在是441。嗯，所以这意味着我们有九 分钟的时间提问，如果有人有问题的话。嗯，否则嗯期待在课上见到大家。有问题吗？请说。
CMU 11-768 Fall 2026 · Lecture 1 · Course Overview: What Is an Agent?
- 视频：官方视频
- 字幕：en-orig · auto captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
正文
[00:00.320] Okay. Um, hi everyone. Welcome to 11768, uh, AI agents. We're very excited to be teaching this new course, uh, which is timely as, uh, a lot of people are starting to use agents. Um, my name is Graham Nubig. I'm an associate professor in the school of computer science here. I've been working on agents for maybe uh, several years now. Uh, coding agents, web browsing agents, uh, those sorts of things. and I'll be co-eing together with Daniel who will be kicking off this class. So without further ado, I'll turn it over to him and he can start uh this and I'll I'll do the second part of the class.
[00:40.160] >> Thanks Graeme. How's this? Cool. Yeah. So um hi everyone. Really excited that you're here. My name is Daniel Freed. Um my research group also works on agents. Um so uh grounded agents and also interaction between people and agents and these days interaction between agent systems. So uh excited to talk about um all these topics with you and um yeah we have a
[01:07.280] really great core staff too who we will introduce in in a few slides. But yeah so agents are are really timely right now and um the capabilities of agents are advancing pretty quickly. So probably a lot of you are using coding agents in your you know day-to-day work and we've seen some big success cases of agents. So um one example is there was this uh nice experiment um by um Carlini at anthropic that saw could we develop a really complex piece of software using um a multi- aent uh configuration of a a recent coding model. And this was pretty successful. So 16 agents collaborated together over two weeks and constructed a rustbased C compiler which was able to compile the Linux kernel. And you've probably you know been using agents effectively in your own work and and seen how much better they've gotten especially in the last like year or so. There's limitations of current agents too. So how many folks have uh tried uh using an agent that controls your computer like controls applications or Okay. So, a lot of folks have How many uh folks have used OpenClaw?
[02:22.879] Yeah. So, we have a number of folks, too. Um, there was this pretty high-profile incident involving OpenClaw um which got posted on Twitter by the the person who ran into this where she was asking the model to organize her inbox and the model kind of went out of control and started deleting old emails. And uh so she's trying to intervene. She says, "What's going on? Can you describe what you're doing?" But the model says, "I'm taking the nuclear option. I'm going to trash everything in the inbox that isn't already in my keep list." And the user is trying to intervene, saying, you know, do not do that. Um, but the model goes on to delete a lot of this really valuable information. And at the end, it acknowledges uh that, you know, I learned my lesson. I'm not going to do this in the future. And I do remember that she told me this, um, but I violated it. So, what happened behind the scenes here? Well, it seems that it was the model compacted its past context, which is a way that we'll cover in this class of uh of reducing the past history so that the model is able to take in more um interactions with the environment, but that caused it to lose this instruction to not delete emails. So, models are getting much more capable, but they also still have really rough edges. And a lot of this class is going to be about how do you develop these capabilities and models starting from base language models but also what
[03:44.799] are the research problems and what are the remaining gaps that we need to fill. So let's just uh try to get a sense of what people think agents are are good at these days. So we have like six different tasks that you might want to try to use an agent for. And these are pretty subjective. I just want to get a sense of like what might you be comfortable with. Um, so let's do a show of hands for a task like say you have an online store like some uh code application that hosts a store for you and it started failing. You want to use an agent to try to diagnose that. How many people would be comfortable with having an agent carry this task out like fully on its own? Can we get a show of hands? Okay. Yeah. Um, how about having the agent like ask first as it's carrying out this task? Maybe asking you for help. How many people think that might be necessary? Okay. And how many folks think that this would just not be something that you would ever want an agent to touch at all? Okay. So, for this one, I think we had more autonomous.
[04:47.759] How about drafting and sending a product launch email to 50,000 customers? So, how many people are okay with this being fully autonomous? Okay, we got some brave souls. Um, how about asking first? Okay. And how about an agent shouldn't do this at all. I want to be responsible for it. Okay, so I think this one was was asked first. How about um collecting your tax forms and preparing and filing your 2025 tax return? Fully autonomous. Okay. Yeah. How about ask first? And how about never? Okay. Yeah. So I think this had the most never of anybody but ask first probably one migrating a payments API from Python to Rust. Say maybe you have some test cases that were written for the application in Python that you can apply to the Rust version. How many people would be okay with this being autonomous? Okay, how about ask first? Yeah, that was very close. I don't know which one was more and never. Okay. So, we'll uh let's uh have this be one for the agents. Autonomous. Buy concert tickets to my favorite band if they set up a show in an area in my area. Autonomous. So, you'd be okay with the agent buying the tickets, too, if it finds good ones. I think that one probably won. And how about this last one? Adjusting an insulin dose after a week of glucose readings. So, an agent like, you know, has all the like the capabilities to do this. Like if you're feeding it your health information, um it could monitor that. If you give it access to like something that can change a prescribed dosage, it could do that too. How many people would be comfortable with this autonomously? Okay, asking first. And how about never? Okay, so yeah, so that one was never, although I was thinking maybe it would be ask first. So cool. So I think you're all you all kind of have a sense of some of the boundaries that agents have. Although
[06:54.479] you know there's a lot of these tasks where we might just not be sure. Um but you also have a sense that like trust is important and later on in the class when we talk about like sandboxing and um security and um interaction and oversight by people um we'll get into these a little bit more.
[07:15.599] Okay. So just to give like a very brief kind of uh tour of how agents have extremely brief. It's just two slides. how agents have progressed over the past few years. Um, and also to motivate some of the things that we'll see later on in the course. Um, so we've been doing some work on guey agents, like agents that can use computer applications here at
[07:34.240] CMU over the past few years. This is work from a lot of um, faculty. Um, and um, here's a demo that was made by um, one of our students, JY Co, um, two years ago showing an agent in action controlling the browser. And so you give it a task like navigate to the page of a good Thai restaurant in Pittsburgh. It should have at least 200 reviews and 4.3 stars. And on the right here you can see, sorry it's a little fuzzy, but you can see the interactions with the language model like the chains of thought that are being produced. And on the left here, the model is controlling the browser um typing in the search box on Yelp and then navigating to this page. Fusidities is a really good Thai restaurant. Um and so that's a success and we've seen um you know
[08:19.039] frameworks and agents like this really taking off over the past few years with like um Manis and um OpenAI's operator cloud computer use agent and so on. And we've also done a lot of work uh here at CMU on building coding agents. Um uh Graham's group in particular has done a lot on this and also um through open hands which develops um open- source um agents. Um and uh here's a
[08:44.800] demo video uh from Graham which I'll uh play showing the agent like both writing code um but then also interacting with the application that it's built in the browser to test it out. >> To-do list app in list. >> So this will take a little while for the agent to work. Um, we can watch it work over here and see that it has uh made a directory for static files and templates and it has created the main app file. We can look at the changes it made to the app file uh by clicking over here in the changes tab which shows us all of the things it implemented. So that's kind of nice feature. So you can kind of watch uh what the agent is doing while it's working. and it's [music] editing the index file, the style file to make the app look nice. Uh requirements.ext to make it easy to install and adding a read me. So you can see it's kind of following good uh coding practices. And now it will start up the app. [music] And so it's running uh very very quick. And what I can say is let's run in the background and then [music] test out with your browser. And so if I do this, it will uh run the app in the background. [music] And it checked if the app is running. And now it should navigate there with its browser. uh it's checking the log and saw the port was already in use and it's trying it out. So that that's kind of the nice thing of agents. They can debug their own problems and uh and make sure that things work. So, okay, it
[10:32.959] should be working now. [music] And you see it browsed to the app. And we can see our tuning list app over here. And it sees that it's running, but now it wants to test the functionality. So, what we can do is we can see that it uh went through and it filled in uh buy groceries and it clicked. And you can see that uh in real time it's testing out your app and then it also is uh testing the delete functionality and other stuff like this. So I find this really exciting. I'm not, you know, the best front-end developer, but it's able to kind of [music] try out the front end, see if the functionality is working and fix any issues on the fly. And I even had one time where the app crashed and the agent realized that the app crashed even before I did and it was able to go in and fix the problems. So this is as simple as just saying, you know, go and develop an app and then test out if it works and you don't really need to do anything else. So now we're done. Uh let me stop this. In a real scenario, what I would do next is I'd ask it to push it to GitHub and uh you know finish it up so that I can continue working on it.
[11:46.560] >> Yeah. So we're excited to give you hands-on experience building agents um in this class. And in one of the lectures later on in the course, we'll cover open hands and its um SDK and how you can um uh develop agents of your own like this. So we um have a really great core staff.
[12:06.800] We have uh our our core staff all does research in these areas. um wrote some of the foundational papers and uh we'd love to have you all come up uh and uh introduce yourselves and um at the end of the course you'll be getting to work on a mini research project of your own and um we'll be uh getting guidance from the entire core staff um helping you out with that. So um yeah, if we can just uh go through and have each of you maybe just project uh and introduce yourselves. >> Yeah. So the yeah say it so the people in the back can hear like what what your research is on because uh then the people here can like essentially know what you're working on so they know which TA to talk to when you're working on like a project in that area for
[12:53.360] instance. >> Yeah. >> Hey everyone I'm a third year PhD student. I'm advised by Grant and Kamar. Um my research areas are focused on training agents to better leverage interest and compute and also organized in multi- aent systems. >> Hi everyone. Uh my name is Adita. I'm a first year Ph student uh working in the professor program. Uh my main research interests are like designing reward functions for like agent reinforcement learning. I particularly work on code generation for for now. Um hi everyone my name is and I'm a second year PhD student working with professor Anna Yeah and my current research focus is [clears throat] learning and research >> hi everyone I'm here working with the previous of working on mobility agents and recently I've been working on Hey uh I'm Sus. I'm a PhD student working with Daniel. Uh I work on how we can train agents to communicate more efficiently with people. >> Hi, I'm Lily. I'm a Syria business student by EMU. So I'm work on training on agents on authorizing coffee and battery.
[14:18.160] >> Uh hey everyone, I'm Andy. I'm a fourth year PhD. I work with Daniel and Monop. Uh I mainly work on alignment post training and also multi-agent introductions. >> Thanks everyone. [applause] >> Yeah. So uh like Graham said, we definitely encourage you as you're starting to think about your projects later on in the course, uh we'll talk about the course structure towards the
[14:43.680] end, but as you're starting to think about the projects that you want to do, TAS are going to be a really great resource for you to go to in office hours and get their expertise. So, we're also really grateful to um uh several really awesome startups for uh donating uh compute which you will be able to use in the assignments and projects for the course. Um and uh
[15:04.720] without their support uh we wouldn't be able to run um this at the scale that we would like to. So, we're really grateful to fireworks AI to modal um to prime intellect and to sale for uh helping to support uh this semester. Okay. So today we'll give a brief overview of what an agent is and um then we'll talk about uh the plan for the class and how things will be structured.
[15:33.360] So an agent um agents have been around for a long time. The concept of an agent and um work developing agentive capabilities um isn't just you know a two or three year old thing. So, uh, Russell and Norvig in their foundational textbook on AI, um, define an agent as anything that can be viewed as, uh, perceiving its environment and acting upon that environment. And that's really true for the agents that we have now. And um, a lot of the techniques that were developed for agents in the past, not all, but a lot of them will apply to current agents too, especially things like search reinforcement learning. So, uh, how does this pan out for our current agents? So, agents are going to be situated in an environment, which could be something like a code repository, a website, um, some other application on your computer, and they'll be interacting with that environment by getting observations of the current state that they're in in the environment. So that's something like messages from the user they're interacting with, the contents of a file that they're um investigating, the current web page that they're on, screenshots, or as we'll focus on in a few slides, the results of um tools which are you like programmatic functions that they can execute in the environment to interact with it. And they're taking actions at each point in time. um which is going to these actions will update the environment and then change the state that the agent is in. So that could be something like
[17:09.280] replying to the user or editing a file or running a command in the shell or calling an API that backs the web pages that they're interacting with. And finally, there's also this notion of reward. So how do we define whether the agent was successful or not using a numeric score? And we could define this as like if there are some test cases for a codebase, do those pass? If they do, then you get a one, otherwise you get a zero. Uh for other tasks, it can be difficult to like have sort of a programmatic reward. So we might need to rely on an LLM as a judge to uh evaluate whether this was successful or not. In your second assignment, you'll be developing LLM as a judge based evaluation approaches among other eval.
[17:56.160] And maybe we're ultimately interested in did we, you know, make the user happy with the task that they asked the agent to do. So we could also use feedback from the user for that. And I think in our uh human interaction lecture that Valerie Chen is going to be giving a guest lecture later on, we'll talk about this.
[18:14.160] So we asked all of you to have prior experience training language models and um given that experience you should all be familiar with um language models um doing next token prediction right so at each step um the agent is going to predict a distribution over the next token um we'll sample or in some other way choose one of those tokens from that distribution um insert it into the context window of the model and repeat this um over over and over again. And recently we've uh seen a lot of a lot of improvements on uh complex reasoning tasks by introducing chain of thought to these models. So as um as you should remember a chain of thought is um predicting a sequence of tokens um which aren't the final answer that we want the model to output but do allow it to um
[19:06.480] sort of have a scratch pad um for its intermediate uh reasoning steps which it can then um use um and condition on to predict the final answer. And so this is like a non-aggentive setting, right? This is just the agent um interacting with prompts that you give to it. So how do we go from this to um having models that can take actions in the world?
[19:31.679] So the main way that we do this is through tools. So a tool is something that's situated in the environment um which will uh provide an interface for the model to interact with that environment. And you could think of this as being like um an API for example like if we want the model to be able to uh uh find out what the weather is so that it could respond to a user asking about that maybe we have um an API for like you know weather.gov which is wrapped in an interface that allows the model to call it. So there's a bunch of different ways to represent this interface. Graham will be covering them in more detail I think in the next lecture but here's one example. This is the open AI um tool specification where um for a given tool aka function that the model can call. You'll have a name for that tool um a natural language description of what it does. So for example, read file might be a tool that a coding agent uses to uh look at files in the repo that it's acting in. We'll also need to tell the model what it can pass to this function. So we'll have a description of um the parameters that it takes like maybe this function just takes a path which is a string you know for the file to open. So we have to uh so we have the name and description we have some structured representation of the interface to the function which here is JSON and um this also needs to get fed into the model in some way. So we'll have um this you know underlying specification of the tool and then the model will view it by applying this template which turns it into a sequence of text which will then be tokenized and read by the OM which uh might look like this and the model will have to be um uh sort of either trained or with few examples um know how to use these tools.
[21:28.960] So this is how the model becomes aware of the tools that it can use to interact with the environment. But the model will also be needing to use those tools to interact and it does this through tool calls. So for example generating text like this like tool call and then you specify the name of the function that you want to call and the arguments as
[21:51.200] well if you're using sort of a JSONbased representation. Can anybody make a suggestion for another way that you might have the model call tools? Anyone? Let's Yeah, in the back. >> Yeah, you could just write code directly. for example, bash scripts or if you represented this as a Python function, the model could just, you know, call the function using Python
[22:24.080] syntax. And we'll see in a couple lectures that that's actually a very effective and often more effective way than generating JSON. Um, but there's some trade-offs, too. So, when the model calls the tool, it'll get back results from executing that tool in the environment. So for example, if we're reading um this test file, maybe that test file has this code for an add
[22:46.400] function in it and um the environment will return um you know the results of that tool in this content field and then that'll be rendered to the model using a template um which the model will then read in as tokens and get to continue producing output. Any questions about this?
[23:07.520] Cool. So now that we kind of have like the basic tool interface, we're able to um have the language model use these tools to act in the environment. So the way it does this is just by reading in these token sequences that represent what tools are available um uh that represent uh the results of tools and then um generating token sequences naming the tools which then invokes them. And so this uh we have this distinction between the language model itself which is just you know interacting with tokens um later on we'll talk about multimodal language models which maybe also take in images or other modalities. Um so the language models just tokens in and tokens out maybe multimodal tokens but then there's this harness which is
[23:53.039] responsible for actually um coordinating um these tool calls um using the language model as an engine for it. And so the agentive loop basically and tool former was a really impactful paper that showed that you can train models to be able to use tool calls and then condition on the results of those to um make future text more likely and showed an early way that you could train these
[24:19.679] models to use tools um that generalized prior LLM training procedures. So once you have this basic setup that you then can implement an agent by running um a loop um and one of the influential papers here was called react um from 2023 um which stands for reasoning and acting because the model's producing chains of thought and then taking actions afterwards and the way it works is you'll have some uh context um which you know includes maybe a definition of the general task of solving pull requests on GitHub or solve issues on GitHub. We'll see some examples of this in a bit. Then you have a task which is maybe an initial query from the user. Um you'll also have a list of all the tools that are available for the agent to call which will be
[25:06.720] represented using one of those formats that we showed. And you'll also have a history of the past um observations and actions that the agent um has produced and observed as it's been interacting. And in each uh at each time step um so for a given particular state the agent's in it's going to condition on all of that produce a reasoning chain um you know chain of thought um with a scratch pad about what it should do and then produce um one or more tool calls that will be used to interact with the environment or with the user. So for example reading the file. So that tool call will get executed in the environment. We get the results. We add those results to the history and um the environment is also getting updated as a result of our tool call and then we repeat the process again. So the model's conditioning on this updated history and you know the observations that it gets from this time step and then issues a new tool call and it repeats over and over. We could also have tool calls that end the task that for example send a message to the user um or otherwise signal completion. Um and uh so this is
[26:16.400] a way that we can you know signal that we are finished. And in the planning lecture we'll talk a little bit about like how might a model determine that it's done. So I won't get into the details of this too much. Um you'll get into it when you do assignment one which asks you to implement a um react style loop to uh implement a small coding agent and use
[26:39.679] it to solve problems in a repo um a a chess engine repo actually. So there's some fun demos that go with it too. Um so we won't get too much into the details of this and the code that you will write will look different than this but you know at a high level this is a very simple implementation. This is an excerpt from this nice minimal repo called mini sui Asian um which actually
[27:03.520] gets very high performance um on the bench benchmark which we'll talk about in a few lectures. It's a very standard um benchmark for coding agents. Um but it's a really nice repo that I'd encourage you to take a look at because um it's um effective with recent models, but it's also pretty simple and really kind of distills um you know the the distills the basics down. Um but you'll uh start off by having some messages uh which give the model context about it being an agent and about the task. And then you'll just uh run this loop over and over again that implements what we had on the past slide where the agent will um issue a query to a language model using all of the past u messages which contain the history um and then uh it'll add that message to the history
[27:55.360] and um execute um the uh execute uh the actions in the environment. And we've left this off, but you can take a look at the code repo if you're interested. Execute the actions, get the observations, and um return them. So, I think it's also instructive to take a look at example agent trajectories. And um one nice thing about the Swebench uh data set or
[28:23.679] Swebench evaluation task and leaderboard is that they have some example trajectories here. Um so, let's take a look at maybe GPTO OSS. And this can give you a feel for what uh tool calls can look like and what the model is actually seeing um as it's carrying out this task. So let's see if I can zoom in a bit.
[28:52.640] Yeah. So the model gets some uh context in a system message um which says you are a helpful assistant that can interact with a computer shell. So this is like very generic um but then we'll also have a user message which gives some context about a particular task uh solving this particular um poll request. Um and so the model conditions on all of that and then it needs to um issue a first action and you can see here the chain of thought trace that it generates. So this blue is generated by the model. So you have the chain of thought here and it produces a tool call. This agent is using the um the tool format that you suggested which is just issuing bash commands um by you know writing the uh command that should be run that gets handed off to the environment. The environment executes it and here the result is just an exit code exit code zero showing it succeeded but then we go on to the next turn and the agent uh you know issues a new thought and a new bash command and so on.
[30:02.480] So hopefully that gives you a sense of you know one possible way of uh instantiating tools and instantiating prompts and you'll experiment with this in your first assignment. Cool. So uh now I'll hand over to Graham. Thanks a lot. Okay. Hi. Uh I guess uh everyone can hear me. So next I'd like to talk about agent capabilities. And what I mean by this is
[30:43.520] like what makes a good agent. Uh because as uh Daniel pointed out uh it's not very hard to make an agent. uh especially nowadays that there are libraries that you can call to you know do inference with a language model or something like this. The hard part is making one that actually works. Um so I'd like to think a little bit about uh the capabilities that you need to have
[31:07.520] in an agent. So a fairly large number of people said they use agents in some way you know every day like openclaw or coding agents. Um what are the things that frustrate you when you use an agent? like what what has gone wrong that was a problem? We saw an example of deleting all your email files. That would probably frustrate you, but um any any other things? >> They're too slow. >> Okay. Yeah, that that's a good one. Uh in the backy wordy and verbose. That's a good one. I saw a hand. Yeah. >> Um so they forget things. Yeah. Okay. >> They might misunderstand what you said and make mistakes due to that. Yeah. >> Do doing what was it? >> Like accessing things they shouldn't be accessing. Yeah. Okay. That that's a good one. Um Yeah. random knowledge gaps for very common or normal things. Yeah, that that's a good one. Yeah. >> Yeah. Writing a thousand lines of code when two would do. Yes, that that's a good one. That's kind of a different variety of verbose, but it's also Yeah.
[32:32.960] verbose. Yeah. >> Yeah. Forgetting previous session interactions. So, um, yeah. >> Not knowing how to push back when you say something, uh, unreasonable. Yeah. >> They may lie to you. Yes, that's a bad one. >> A lot of implicit assumptions. Yeah. So, um, these are all great points. Um, all things that frustrate me, too. So um they're they don't exactly align with
[33:07.120] what I had on the slide but you know that's a a great uh a great thing. So um a first capability that you need to have uh in agents is accurate tool calling. This is something that actually nobody mentioned uh because a lot of people take it for granted nowadays but this is not something you can take for granted uh if you're starting this class and
[33:31.200] you're in charge of training the model. Um so uh you know it's kind of bread and butter. If you can't call tools accurately you're going to fail at any task you do. But uh it's something that doesn't come for free. Another thing is coherence over long context. And so somebody said uh it annoys me when they forget things or it annoys me when they forget the previous sessions or
[33:53.440] something like this. So uh that that's one example of it. Another example of it is uh what the example that Daniel gave where previously the model was told to not do something but then it ended up doing it anyway. So uh that that's another example. Another thing is uh customizability. And so you want the agent to do things your way. Uh everybody has different uh ways
[34:19.200] of doing things uh different requirements. So our class might have a different requirement than another class. And so you need to make sure that if you're using an agent uh it's going to follow those requirements. Uh complex task management. So breaking down tasks uh being able to handle very long horizon things is a very big uh very big issue.
[34:40.480] environment understanding and what I mean by this is uh there were like one good example of this is making a thousand line change in a codebase where two lines would do that's an implicit failure of environment understanding to know that you know this is actually a really simple change but it didn't realize that and it goes and does a bunch of other things instead and then a
[35:02.720] final thing is safety and a lot of people separate safety and capabilities they say I'm a safety researcher or I'm a capabilities researcher. But from my point of view, capability safety is a capability. It's something you need to bake into the model. And if it's not baked into the model, you won't feel like you won't feel comfortable using the model because it might go off and do
[35:26.160] something that you don't uh want it to do. And so from my point of view, it's a a failure of the model or the agent more broadly. And so there's two ways to build these capabilities. And this is something that's really really important when you're working on agents. Uh the first one is uh training in LLM. The second one is engineering the harness uh so
[35:50.240] that you put structure around the LLM to get it to uh you know solve uh like solve the problems you want in the way that you want to. So question um which one do you think is more important or effective uh here? So does anyone have an a strong opinion that LLM training is the way to to get your agents to work well? Okay, actually surprisingly large small number of people. What about um train uh engineering the harness around the agents? Okay, that that's a pretty large number of people. I saw Daniel raised his hand twice, so he's not allowed to do that, but [laughter] um he's the instructor. So, of course, both of these uh are important. Um my my personal opinion um or well, first I'll
[36:48.160] I'll you know talk a little bit about what they entail. So, the first one is changing the model's behavior through pre-training uh supervised fine-tuning or reinforcement learning. Um and then you can teach uh reusable patterns for reasoning, tool use and recovery. The important thing is the capabilities become part of the learn model or the learned uh policy. And so in harness engineering, you change the system around the model. You give it prompts, tools, memories and control flow provide context, validation, retries and safety boundaries. And the capabilities emerge not just from the model, but they emerge from the combination of the harness and the model. So my my answer to this is um they're they're maybe both important but typically what happens is you identify a problem and you solve it here first. Then the people who are training the models uh catch up and realize that this is a big problem. It's a big enough problem that it warrants them training the model in a particular way to solve the problem. it gets solved and you don't need to solve it in the harness side anymore. So that this is a typical flow. Um so from my point of view this is kind of very often the more fundamental solution on the left side but it takes a lot of time and so you eventually you need to start out by solving it on the right side because you don't have the time uh
[38:15.440] when you're improving your models. So I I personally favor LLM training as a fundamental solution to problems if you're in a situation where you're able to do that. Yeah. >> So, this is a great question. So, what are the capabilities that will survive LLM training? Um, one I can definitively say is long context uh in like maintaining all of your your memories uh in context because there that is not a that's not a model accuracy problem only. It's also a model efficiency problem. So you need to be able to model very long sequences. And sure you might be able to come up with a model that doesn't have like n squ complexity like the transformer does but you know that's uh you know that's kind of a bigger a bigger discussion assuming we're staying in our our current modeling paradigm I'd say uh most of the other stuff I have on here uh most of the other stuff I have on here. Accurate tool calling, uh, coherence within your context window, complex task management, environment understanding, mostly from a capabilities perspective can be and maybe even safety mostly from a capabilities perspective could be theoretically solved through the model, but they're not solved yet. Um, and so we still need uh we still need harness uh engineering. Customizability is a very interesting one because you could train the model on the fly. Um, but there are also cases where you kind of want to give it like a script or you want to give it separate instructions based on the situation that you're in and they're like maybe training it would
[40:15.280] not be the best solution. But we can talk a lot more about you know all of these details later. Yeah. >> Yeah. So um a follow-up question on when inference uh latency is very important is engineering the harness to fit the problem a good solution. So uh what what I want to point out is all of these capabilities can be handled either through training or through the harness. Um and so I I wouldn't say like one is necessarily always better than the other. They're both uh both solutions. The interesting thing about like inference time uh efficiency is you need often need to use a weaker model or you need to use less compute and because that's the case you often need to put more guard rails around it because you're going to have less su uh success or you need to do like adaptation to the particular task you're interested in. So um the the smaller the model you're using the more careful you need to be because the very big models can often generalize better. Um they're not perfect but they can generalize better.
[41:22.480] Cool. Uh any others? Okay. So um moving on to uh onto this. So um one of the big goals of this class is to get people to be able get all of you to be able to train a model uh for agentic tasks. And this is uh a big point in that not that many people can do this well. And so we'd like the people in this room to be not that many people, right? We we'd like you to be good at at
[41:57.839] something that that's hard to do, but uh hard to do, right? But, you know, um like it's a good thing to know how to do if you do know how to do it. Um we're not going to be handling pre-training uh because pre-training uh you know, it's on the scale of the entire internet and that's uh unfortunately not something our our modal credits uh will support.
[42:19.200] uh but we will be uh thinking about mid-training and supervised fine-tuning and reinforcement learning. And uh the points here are mid-training and supervised fine-tuning is where you already have example demonstrations of how an agent did a task and you're training on those example demonstrations uh usually by optimizing maximum likelihood. Hopefully everybody in this class has done that before because it's a prerequisite for the class. So um uh we'll only be covering that briefly and be focusing more on how you create the data for doing this in indogentic setting. Um and then we're going to focus more uh on reinforcement learning because reinforcement learning for uh like agents is is pretty tricky. So how many people have done reinforcement
[43:03.680] learning with language models before? I guess maybe not that many people. Yeah. Okay. How many have done it for reasoning reasoning tasks? How many have done it for agents? Okay, maybe about half and half and it's only like maybe 10 10% 10% and 80% have not done it. So yeah, we're going to try to put a lot of effort into making sure that everybody is able to do this by the
[43:27.520] end of the class. Cool. Um so looking at all the capabilities one by one. Um, accurate tool calling is uh an important capability to have. Uh, we do it through harness engineering and LLM training. Uh, some of the things we're going to talk about in class are grammar constraint decoding, uh, which allows you to make sure that your tool calls are wellformed and match the
[43:50.400] specification that you have. Um, for LLM training, this is first done through SFT um or through uh through mid training or or whatever you want to call it. Um and this is done by training on tool calling uh tool calling data tool calling traces. Um for coherence uh over long context uh this again can be done through harness engineering or uh or LLM training. For harness engineering, we do this through context compression or compaction which are kind of synonyms um where you take all of the context and you summarize it down uh for the agent to continue working. It also can be done through dynamic memory lookup uh where you have memory over all the context that you want to be handling but you don't pull it all in immediately uh you pull it uh in you know on demand. Um and it can also be done through sub aent delegation. So you delegate some parts of a very long task to an agent. It keeps that context in memory, but then it drops it out of memory when it's done performing that. Um for LLM training, uh you can do that through long context training. We're not going to handle it a lot, but we'll talk
[45:04.319] about it maybe a little bit. Um then for customizability, this is actually one of the best use cases for harness engineering uh right now, I think. And there's a bunch of methods that you can do this. Uh one is through agent memory. So making sure that the agent continues to learn uh as you interact with it more um through skills uh which are kind of prompts and sometimes scripts about uh the sort of thing that you want to do that you pull in at each uh time peri time time point when you want to use those things. Um also possibly custom tools for a particular task. um for LM training uh this is a nent area. I don't think there's a lot of people who are doing this in uh in a lot of detail. Uh but there are methods
[45:54.240] where you can learn from user feedback. And so like if a user is is using a tool and they give a thumbs up or a thumbs down, you can learn specifically from them to adapt to them. Complex task management. Uh ways you can do this are through harness uh engineering. So you can provide planning or decomposition tools. You can have a plan mode uh for the model where uh it uh it gets to plan. I this is a little uh thing that I I learned but there was a popular coding harness I think it was uh codeex no sorry maybe it was clog code where they had a plan mode and uh the only thing when people pressed the plan mode button was that it added an extra thing to the prompt that said please plan do not do anything. Um, [laughter] but everybody wanted a plan mode. They wanted a button. So, they made sure that it was done entirely through prompts. But there's also more uh more complex ways uh to do this as well. Um, again, you can do this through sub aent delegation by like breaking down the task and uh delegating it to other agents. And a very good way to do this in training is to train on complex uh
[47:02.160] long horizon tasks. So for environment understanding, what I mean by this is whatever data format or environment that your agent is uh interacting with, it needs to be able to understand it. And just to give one very good example of this in computer use agents, um you need to be able to understand web pages and or uh you know guey interfaces or something like this. This is not something you get for free. In fact, like agents are actually pretty bad at it right now. Uh even the strongest agents are maybe not, but like um many of the open source models don't even support multimodal uh data and uh like the ones that do are not perfect at understanding it. They make lots of mistakes, many more than when they're understanding text. So this is a failure of uh understanding the environment because they need to understand the multi- modal data in their environment. But now let's put them in a different environment where they need to trade stocks or something like that. Um, they're not very good at understanding time series. They make a lot of mistakes on time series. Or you put them in an
[48:09.920] environment where they need to understand a picture of a, you know, a petri dish with lots of bacteria in it or something like that. They fail to do that as well. So you kind of need to be able to understand whatever environment you want the agents to uh, you know, interact with. So there's ways you can do this through hardness engineering by like giving skills that correspond to
[48:33.040] domain knowledge but you can also train on data with the expected observation shape or train in domain specific environments. So right now actually like there's kind of this thing uh that people say which is like oh the models are going to get better uh and then we won't need to worry about that anymore.
[48:53.520] Um the thing is models don't get better. Uh people make models better. And so one of the things we'd like to teach in the class is like what's actually going on under the hood when suddenly Claude goes from uh 4.7 to 4.8 and it can suddenly like compose music better. Um, and the there's a few things uh that go into this, but one of the really big ones is uh they make an environment to train the model on uh a like compose music environment and suddenly you know they they train on it for you know they add it to the training mix and then the model gets better at composing music or something. So a lot of the uh a lot of the work here is creating domain specific environments that models can work in.
[49:38.640] Uh then the final thing is safety. And uh we're going to talk about it for uh like a bit because if you don't have a safe model, you're not going to be able to use it in real consequential settings. Um and there's a lot of things with respect to harness engineering that you can do here. You can give it a sandbox. Um you can limit access to cred credentials. You can also monitor the
[49:59.599] agent as it works. Um and you can also do safetyaware reinforcement learning. Um I I think probably a fair number of people heard about this, but who heard about the OpenAI incident where uh it hacked into hugging face? Okay. Well, so I think most people heard about it, but recently there was a um there was an example where uh the newest open AAI model was in an agentic harness and it was working on a cyber security benchmark and so its instruction was hack into this system and so it was going to hack into a system and that was its task and it wasn't able to hack into the system. So instead it basically hacked into the hugging face website and uh and got the answers from the hugging face website and and did it there. And so there were a bunch of failures that h that happened here. Um the first failure was a failure in sandboxing because they didn't properly contain the agent when it was doing this benchmark. um they they did have limited access to credentials and so it worked around that problem uh by by hacking into something it didn't have credentials for. Um and they also didn't have sufficient monitoring so they weren't able to tell that this happened. Uh they didn't have sufficient safety guardrails on this model because this model was being trained to test uh you know like whether it could hack into systems. So presumably the models that they eventually released to the public did have better guardrails of this model.
[51:33.440] But like um these are all ways that you can handle safety and we'll be talking about this sort of thing as well. So the final thing um and any questions about the capabilities part. We're going to go into a lot more detail of course. So okay, sounds good. Um so another thing that I we really want to reinforce here is agents are systems. uh they're not just models and they are far more
[52:01.839] complex than most of the things you've dealt with in any machine learning class that you've taken any you know class here that you've taken before. So you're going to have to be able to handle a bunch of different things. The first thing is a harness. Um, which is a moderate to complex piece of software. Um, and it has a lot of moving parts like you need to deal with the context, you need to deal with the the tools, uh, the guardrails, manage workflows, uh, stuff like that. Um, you need to have a sandbox. So, you need to make sure the agent is working within a contained environment. And, um, there are different ways you can do that. uh it needs to work with a model and you need to do inference. But if you've done inference with language models before, you might have dealt with contexts of like 16 tokens or 32 uh 16k tokens or 32k tokens. Here you need like 256 uh at least they have kind of like a state-of-the-art uh coding agent for instance. And so um the inference problems are harder. You need to deal with caching your inference other things
[53:14.319] like that. and you need monitoring and training. So um from the point of view of this what we're going to teach in the class is harness engineering. So um how to manage state tools memory control flow um how to validate actions, handle errors, enforce permission and safety boundaries. Um and we're in for each of these systems, we're also going to try
[53:37.839] to give examples of software uh that you can look at. Um, for harnesses, there's kind of two big varieties of harness that you use for coding agents. Um, some popular ones right now are cloud code, codecs, open hands, open code, and pi. Uh, probably you know you're you have used or have heard of uh at least one of these. Um, I develop open hands so I
[54:03.440] know it very well. Uh, and so I'll be talking about uh talking about this. I'd also like to give a little bit of my experience and some of the lessons we learned uh in in developing it. Separately from this um there are orchestrators. I I wrote lang chain but maybe I should have written lang graph instead in crew AI and kind of the they have a very different philosophy between these. Um the the coding agents are are generally simpler uh in terms of how different intera uh agents interact with each other but they're more complex in the action space that you give to the agent. So the agent can write code, it can interact with a website, it can do other stuff like this. Um orchestrators are less complex with respect to the individual agent that they have. So the agent might just be able to answer customer service requests or um look something up in a database, but it couldn't write an arbitrary Python program. But on the other hand, you get like these declarative workflows of what the agent workflow is able to do. And so that's like a a different way of building things. It's more guardrail but less
[55:07.680] expressive. And they both have their place uh in different settings. For sandboxing uh to prevent your agents from uh sharing your API keys without your permission or uh hacking into other websites. Um we need to isolate code and tool execution and limit the compute network and file system access that agents have uh access to. Um another big thing is for evaluation and training. It's a very good way to create reproducible environments. Uh so like for SWEBench for instance each problem has a uh each problem has a sandbox and you start in the sandbox and that's the state of the environment before the agent starts working and so you can run these uh locally. Um some of the uh technologies that we use for this are Docker and Appainer. Um or you can run
[55:58.480] them on the cloud and uh two of our uh kind of compute partners for this class Modal and sale are uh like cloud uh in uh providers and actually prime intellect for that matter. So the next thing is uh language model inference. And so I I think you know given your previous experience in an in an NLP or like large language model class presumably most people have uh encountered this before but basically um they uh are required to serve model gener generations um batch requests. So if you have like multiple agents hitting your language model at once, it's more efficient but also you need to batch the requests together. An extremely extremely important part of agents is caching previous requests because agents take more and more actions and so the more and more actions they take the more uh kind of context they build up and you need to make sure that you reuse that to do it um uh properly and you know other systems considerations. Uh some example software that we're going to talk about here is uh things like VLM and SGLANG. These are the two most popular ones. And then there is also tons and tons of inference providers. Um if people have not seen open router before, open router is kind of a a general thing that gathers together all the inference providers. And for any model, uh you'll see like 20 or 30 different providers serving the models. So there's lots and lots of them. uh one of our compute sponsors fireworks is an example of that and sale um training systems. So here what they
[57:37.920] do is uh basically they need to update the models weights and so they uh prepare data, collect rollouts um uh they coordinate a whole bunch of workers who are working together uh to to train the models and they can uh checkpoint evaluate and reproduce runs. Um some example pieces of software here are Sky RL and Miles. Um and so uh the these can be used for for training.
[58:04.319] And finally um observability and monitoring. Uh so this is uh to understand how your agents are working. So these capture the traces or trajectories of the agent, what the agent did um and they gather metrics around them uh such as like how much time it takes, how much cost it was uh track the quality, the cost and the failures. Um and they allow you to compare trajectories and evaluations. And so um uh I'm I'm pretty familiar with one called Laminer because I use it a lot. MLFlow is another example. Um I I just realized that Daniel showed one called transluce uh which is not on my slide here but that's another another popular one.
[58:45.680] So um by the end of the class what I hope everybody in the class would be able to do is uh implement an agent from scratch uh on top of an open source LLM. So uh this is implement a harness yourself uh design evaluations for multi-step tasks. And so basically uh we'll give you a task that you need to evaluate and you need to create an evaluation for it. This is super super important even if you're not interested in evaluation but you're also interested in training because one of the best ways to do training is to create an evaluation and scale it uh so that you can do reinforcement learning based training. So um very important skill to have uh train agents to improve their capabilities. So we actually run the RL loop, handle all the systems problems
[59:32.079] and be able to to solve this um reason about safety and reliability trade-offs. So we'll we'll be talking about safety and uh finally we will have a project which I'll I'll talk about on the next slide. So um these are the assignments. Um the first assignments are basically going to be implementation assignments. So you'll be given uh something to do and be asked to do it. And uh the first one is create a harness. Um the second one is evaluation as I measured as I I read measure as I mentioned. Um and then uh training uh where we train with RL. And so these are approximately the first half of the course. Um and then the second half of the course will be to do a project uh using these skills uh to you know do something kind of
[01:00:21.599] interesting or novel uh in whatever area you choose. And the project will be group based. So we're going to be uh asking you to form groups of two or three and uh and create a project. And the general schedule is uh we're going to talk about agent capabilities first.
[01:00:39.200] Um we're then going to jump into some specific domains like coding and guey agents. Um do uh training uh and uh including SFT and RL. Um we're going to be talking about uh frameworks and safety. And then uh we're going to be talking about interaction and then we have some project hours where everybody uh discusses their projects. And then uh on the end of the class we're going to try to get uh some domain experts from each of the various domains to give a guest lecture um which maybe combines a little bit of overview but also their research. Uh and so uh hopefully you'll be able to learn from uh some pretty exciting people. And I uh Daniel and I tried to brainstorm the best people in the uh in the world to talk about their
[01:01:25.280] things and I don't know if anybody said no. Um so I think everybody said yes. So uh we we got our first choice for every every topic. So I'm pretty excited about that. Um so uh before we begin uh we have some prerequisites and this is a heavy course in terms of implementation in machine learning. Uh we want uh we want everybody to be able to train an agent
[01:01:52.720] with RL by the end of it. And so our prerequisite is also um we require prior serious experience training a language model. Um the LTI courses on uh language models, language model systems, NLP would all qualify for this prerequisite. So if you took one of those, that's okay. Um some deep learning courses might qualify for it. Uh if you're not sure, um feel free to ask us. Um but we are going to require this and we're going to have everybody have to submit a form for it. So uh please uh please do do that. Um if there's like 30 people who want to know whether their course qualifies, please don't ask me that specific question after the class here because it'd be better to do it through email, but um uh other questions we'll be happy to answer after the class. Um
[01:02:39.599] we'll share a Google form on Patza and please submit the form so we can finalize enrollment. Um and uh yeah, I I already um uh I already covered it. If you haven't taken a course, if you've worked in industry and you've trained a model in industry, uh it should be a real one like at least 4 to 7B uh size, not uh not 100 million parameters. Um but we will also accept that.
[01:03:08.000] Um and so uh this is the grading policy. Um so assignments one to three will be 40% total uh completed individually. Um we're going to have lecture highlights and so uh basically the idea here is that we want people to uh come to the lecture and uh give a highlight about you know something you learned in the lecture. The idea around this is not to be, you know, not to make people do like undo work or something like this, but if you watch through the entire lecture, you hopefully you came up with something that's interesting. If you said if you could say nothing was interesting, I knew all of this already. I learned it here uh in this previous class that's already in the prerequisite list. That's also useful information for us. So share it with us. But hopefully we'll say at least one interesting thing in every lecture. So uh you can uh you can follow up with that. Um the research project uh is uh 50% of the total um and this will be in teams of two to three uh because it's uh half of the class assignments. Um the proposal is 5%, the check-in is 5%, presentation is 10% and final report is 30%. And if you've taken an LTI uh project based class before, grading will be similar to the LTI project based class uh requirements. Um if you have not then uh we'll we'll be sharing uh details about this.
[01:04:41.680] Um so here's the question uh you've all been asking for. This is an AI agent class. Um and we discussed this very um very thoroughly about what we wanted to do here. Um, I'd like to preface this by saying I use agents all the time for nearly everything I do. Um, but at the same time, it's kind of dangerous to use agents for everything because you lose
[01:05:09.599] like sight of uh of what you want to uh like of what you're learning. You miss some of the details and other things like this because the goal of this class is learning. Um but we also want everybody to be familiar with the standard practice. Um we uh we want you to keep this in mind and uh AI tools will be permitted um for basically everything in the course unless we say otherwise and we probably won't be saying otherwise very often. Um so uh lecture highlights must be written by you not generated by AI. Um and so these do not need to be long. Uh they can be short, they can be a few sentences, but please uh write them yourself. And uh you are responsible for every submitted claim, citation, result in line of code.
[01:06:02.079] And uh we will create quizzes uh for the assignments so that uh you will you may need to explain uh parts of your code. And uh how will we do this? Uh we also know how to use agents. So we will We we are considering having agents read uh read your code and come up with quiz questions tailored to your code. Um so uh we're we're serious about this. Uh
[01:06:31.359] and that means that the thousand lines of slop that your agent generated will be a bad idea for you to submit. Um so it's better to submit something concise that you really understand well uh than uh submitting you know more uh basically. Um, so yeah, uh, we're almost done. Uh, deadlines, uh, submissions in Slack. So, all the assignments, uh, include two 24-hour Slack days. Um, they cannot be transferred or shared. After the Slack days are used, the penalty is 5% of the assignment score per additional day or part day. So, what this means is the deadline is the deadline. And, uh, but we're being nice and if you miss the deadline, we'll give you some time to make up for it. uh if you miss the deadline by more than two days um then
[01:07:20.559] uh we're not going to take most excuses. If you you were in the hospital or something like that, we uh might ask you to get appropriate acknowledgement of that, but that would be an example uh reason why. Uh but other than that, please try to meet the deadline because we're not going to make many exceptions to this. Um cool, that's all. Uh we're at 441. Um so which means we have nine minutes for questions if people have questions. Um otherwise uh looking forward to having everybody in the class. Any Yes.