Computer Use Agents
- 日期
- Tue, Sep 15, 2026
- 讲师
- JY Koh
- 课程资料
- 16 项
- 内容产物
- 4 项
课程简介
CMU 11-768 Fall 2026 · Lecture 7 · Computer Use Agents 学习笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
依据：Lecture 7 slides（官方 PDF 文本提取，标注为待核）与课堂逐字稿整理稿。本文标注的 Slide 页码依据 slides 提取文本；逐字稿仅用于补充讲解语境。图示、公式细节未与官方 PDF 视觉核验，均标注“待核”。本笔记内容已由用户审核确认。
一、学习目标
学完本讲，你应该能够：
- 用自己的话定义 Computer Use Agent（CUA），并说清它与本课程此前讨论的文本型/软件工程型智能体的区别（Slides 第 2 页）。
- 复述 CUA 的核心循环：观察 → 推理 → 行动 → 新观察，直到任务结束并计算奖励（Slides 第 3–8 页）。
- 画出 CUA 建模的上下文结构：目标、截图、动作交错拼接成多模态序列，由 VLM 预测下一个动作（Slides 第 47–56 页）。
- 区分四类评测范式：静态、端到端（网页）、端到端（桌面/移动）、长时程（Slides 第 18、24、34、38 页）。
- 说明每类评测的验证方式：逐动作匹配、程序化状态检查、LLM/VLM 评委、人类评审、评分细则。
- 概述三阶段训练流程：预训练 grounding/控制能力 → SFT 行为克隆 → RL 在模拟环境中按奖励更新策略（Slides 第 59、74 页）。
- 认识 CUA 的残余挑战：速度与成本、个性化、基础设施与 UX、多智能体系统（Slides 第 77–80 页）。
二、概念主线
1. 什么是 CUA
CUA 是通过与人类相同的图形用户界面（GUI）操作计算机的智能体。它可以操作浏览器、桌面或移动设备（Slides 第 2 页）。关键点在于：它接收截图作为输入，产出与人类相同类型的动作——点击、滚动、打字、滑动、轻触。逐字稿中讲者强调，CUA 与文本型智能体的差别在于“在与人类完全相同的空间中操作”。
2. 观察—推理—行动循环
Slides 第 3 页给出循环骨架：用户任务 + 环境观察（通常为截图）进入 CUA 模型；模型产生推理轨迹与一个动作；基础设施校验并执行该动作；环境返回新截图作为下一轮观察。Slides 第 4–8 页用“买一个蓝色马克杯”贯穿演示：点击马克杯 → 向下滚动找配送信息 → 点击邮编框 → 输入“15213”，最后计算奖励 1.0。
逐字稿补充了循环终止条件：模型认为完成，或用尽预算。奖励通常是 0/1 二元，有时是部分奖励。
3. 一段简史
Slides 第 9–15 页与逐字稿给出时间线：
- 2017：MiniWoB，高度简化的玩具网页界面（Slides 第 9 页）。
- 2018：MiniWoB++。
- 2022：WebShop，用真实商品与语言目标做购物（Slides 第 10 页）。
- 2024：WebArena、VisualWebArena，走向真实网页界面（Slides 第 11 页）；OSWorld 转向完整桌面工作流（Slides 第 12 页）。
- 2026：MyPCBench 用个人历史数据（Slides 第 13 页）；CUA-World 用专业化桌面软件与真实数据（Slides 第 14 页）；GPT-6 ASTRA 演示显示能力飞跃但仍昂贵（Slides 第 15 页）。
逐字稿提到 2025–2026 年出现用编码智能体端到端生成环境与合成数据的趋势（如 MyPCBench 中的 Gmail 克隆），以及 CUA-World 覆盖天文学、生物学等长尾专业软件。
三、关键机制
1. 评测的三种“成功”定义
Slides 第 17 页区分：ACTION（是否点中目标）、OUTCOME（正确物品是否在购物车）、PROCESS（是否遵守约束，如不下单）。评测器应匹配你想要衡量的能力。
2. 静态评测
在固定观察上预测一个动作，不做环境 rollout（Slides 第 19 页）。任务示例：“把 20 美元以下的蓝色马克杯加入购物车，不要购买。”评分规则：预测的 (x, y) 落在目标框内得 1 分；对录制动作则匹配目标、操作与输入值。
- ScreenSpot-Pro：1581 张截图、23 个应用、3 个操作系统，逐点/区域匹配（Slides 第 20–21 页）。Blender 示例中真值框为 (417, 132)–(576, 152)，原始分辨率 2560×1440。
- Mind2Web：2350 个任务、137 个网站、31 个领域，离线动作匹配（Slides 第 22–23 页）。示例任务为在 Brooklyn–Central 租车，真值动作为点击 CAR 标签。
逐字稿指出静态评测的主要缺陷是假阴性多：同一截图下可能有多个合理动作，但只有录制的那一个被判为正确。
3. 端到端评测（网页）
代理在环境中真正执行任务，最后用程序化检查验证状态（Slides 第 25 页）。示例检查：cart.item == blue_mug、cart.price < 20、orders.count == 0，全部通过才给 reward = 1.0。
- WebArena：812 个长时程任务、4 个自托管域、功能性成功检查（Slides 第 26–27 页）。示例任务：创建名为 awesome_llm_reading 的空仓库；验证器检查目标 URL 页面 HTML 中是否含该名字，但不检查仓库是否为空。
- VisualWebArena：910 个任务、3 个自托管站点、25.2% 图像输入任务（Slides 第 28 页）。
逐字稿补充：WebArena 的验证器由 CMU 研究生手写；程序化验证器也可能有假阴假阳；可能出现奖励黑客（例如为完成“加入评分最高商品”而把大量商品塞进购物车）。
4. LLM/VLM 评委
当无法为每个任务写验证器时，用另一个 LLM/VLM 依据评分细则判断轨迹证据（Slides 第 29 页）。
- WebVoyager：643 个任务、15 个真实网站、评委与人类一致率 85.3%（Slides 第 30–31 页）。示例任务：找到气候变化数据可视化相关 star 最多的 GitHub 项目。
- Online-Mind2Web：300 个任务、136 个真实网站、WebJudge 与人类一致率 85.7%（Slides 第 32 页）。
逐字稿提醒：真实网站会发生漂移，评委质量本身成为得分的一部分；大量智能体持续访问网站也让站点所有者不满。
5. 人类评审
Slides 第 33 页给出人类评审用于解决模糊结果：示例中 Reviewer A 判 PASS，Reviewer B 判 FAIL，最终因订单确认页显示发生了禁止的购买而判 FAIL。逐字稿称人类审查仍是黄金标准，但非常昂贵，通常只做一两次用于校准。
6. 端到端评测（桌面、移动）
- OSWorld：369 个任务、9 个应用、每个任务自带定制评测器（Slides 第 35–36 页）。示例任务：用文件夹中的收据更新记账表；检查 A1:E8 不变、C9:C13 费用标签、金额与滚动余额、D9 ≈ −186.93、E9 ≈ 603.07。
- WindowsAgentArena：154 个任务、11 个程序、20 分钟并行扫描（Slides 第 37 页）。
- 附录另有 WorkArena（33 个任务模板、19912 个实例、ServiceNow 环境）与 AndroidWorld（116 个任务族、20 个应用）。
7. 长时程评测
Slides 第 39 页给出混合检查示例：动作轨迹为 click(mug) → click(add_to_cart) → click(place_order)；状态检查通过（正确物品），过程检查失败（发生了购买），最终判 FAIL。要点：只看结果可能漏掉违规。
- Odysseys：200 个真实网页任务、平均 6.1 条评分细则、主评测上限 100 步（Slides 第 40–41 页）。示例任务：规划棕榈泉婚礼旅行，8 条细则中满足 7 条 = 0.875 平均分。
- OSWorld 2.0：108 个工作流、中位人类耗时 1.6 小时、平均代理调用 318 次（Slides 第 42–44 页）。示例任务：构建 FreeCAD 支撑支架并导出匹配图纸的 STEP 模型；另一个示例是在 SolveSpace 中按参考视频尺寸建模并保存。
- CUA-World-Long：200 个长任务、常需超过 500 步、8 条质量标准（Slides 第 45 页）。
- 附录另有 MyPCBench（184 个任务、17 个已登录应用、68% 跨应用）与 WeaveBench（114 个任务、中位 16 次切换、中位 76 次工具调用）。
8. 建模：VLM 从交错视觉历史预测动作
Slides 第 47–56 页逐步展示：上下文从“目标 + 截图 0”开始，模型输出动作 0（如 click(x₀, y₀)）；动作被追加到轨迹；下一步上下文变为“目标 + 截图 0 + 动作 0 + 截图 1”，模型输出动作 1；到第 t 步，上下文是目标、截图与动作的交错序列，模型输出 click/type/tool/stop 之一。第 56 页强调：文本经 tokenizer + embedding，截图经视觉编码器 + projector，视觉 token 是连续 embedding 而非词表 ID；具体 token 划分与编码器因系统而异。
Slides 第 57 页列出前沿 CUA 的动作接口差异：GPT-6 Astra 用 Python/PyAutoGUI 或原生 computer tool；Fable/Opus 5 用原生 computer tool calls；Gemini 3.8 Flash 用归一化坐标的函数调用（0–999）；Qwen 3.8、Kimi K3 用函数调用；Muse Spark 用脚本 + 直接 GUI 动作。逐字稿强调输出格式差异让可复现性成为难题，并提到坐标通常归一化到 0–1000，因此跨分辨率迁移较好。
9. 训练三阶段
Slides 第 59、74 页给出总览：
- 预训练：UI 元素与动作的 grounding/控制能力。用 Mind2Web、AITW 等数据学习“屏幕上有什么、如何寻址”，输出 click(x, y)、type(text)、swipe(…)（Slides 第 60 页）。示例：“CAR” ↔ 元素框/屏幕坐标，元素 bookCarTab。
- 后训练 SFT：行为克隆人类演示与合成轨迹（Slides 第 61 页）。MolmoWebMix 为 36K 人类 + 105K 合成任务轨迹；AgentNet 发布 22625 条任务轨迹、3 个操作系统。合成数据常用特权信息（如 AxTree）让 teacher 展开得更好，再过滤保留有用轨迹。
- 后训练 RL：在模拟环境中 rollout，按奖励更新策略（Slides 第 63–64 页）。示例任务：把 20 美元以下蓝色马克杯加入购物车、不要结账；动作序列为点击商品 → 加入购物车 → 验证/停止。
10. RL 为什么需要模拟环境
Slides 第 65 页给出三条理由：真实金钱（重复尝试会产生重复付费预订）、真实的人（预订会派单，探索影响他人）、无法干净重置（取消费用、时间不可恢复）。结论：在可重置的副本中探索，而非反复真实交互。
11. 环境与数据生成
- CUA-Gym：94 个模拟网页应用，计划→实现→测试；注入初始状态、检查最终状态、每次会话重置（Slides 第 66 页）。约 32K 验证元组、110 个环境、38% 跨应用任务（Slides 第 67–68 页）。3578 条 SFT 演示 → 在 10858 条验证元组上做 GSPO，OSWorld 从 62.2% 提升到 72.6%（Slides 第 69 页）。
- Gym-Anything：200 个软件应用、3 个操作系统、CUA-World 中 10K+ 任务（Slides 第 70 页）。创建代理构建、独立审计验证证据（Slides 第 70–71 页）。12103 个任务与环境、覆盖 22/22 个 SOC 职业组（Slides 第 72 页）。约 2000 条成功 SFT 轨迹，平均清单分从 12.7 升到 22.5，完美清单通过率从 1.6% 升到 4.4%（Slides 第 73 页）。
四、公式与符号的假设
本讲未给出以等号写成的数学公式，Slides 中的“公式”主要是伪代码与判分规则。以下为提取文本中可确认的形式及其假设，具体渲染与符号需对照 PDF 核验：
- 静态评测判分（Slides 第 19 页）：对录制动作匹配 target、operation、input value；对点选类，若 (x, y) ∈ target box 则记 1.0。“correct point / step”不等于“completed task”。
- 端到端状态断言（Slides 第 25 页）：
assert all(checks)→ PASS → reward = 1.0。假设环境可 API 访问数据库/文件/应用状态，且检查覆盖了任务的全部关键条件。 - WebArena 验证器（Slides 第 27 页）：
reward = int(name in html.lower())。显式说明“不检查 emptiness”，因此该验证器只覆盖名字存在性。 - OSWorld 记账检查（Slides 第 36 页）：
compare_table(saved_workbook, gold_workbook, rules)；规则包括 A1:E8 不变、C9:C13 标签、金额与滚动余额、D9 ≈ −186.93、E9 ≈ 603.07。≈ 表示数值比较带容差。 - Odysseys 评分（Slides 第 41 页）：7/8 satisfied = 0.875 average；图示另标 “Perfect completion = 0.0”，该处文本提取存在矛盾，待核。
- RL 目标（Slides 第 63–64 页）：滑出轨迹后按 reward 更新策略；正奖励提高该轨迹动作权重，零/负奖励降低。逐字稿表述为“最大化环境中的奖励”。
- 时间成本（Slides 第 77 页）：Task time ≈ turns × time / turn。这是估算式，用于说明减少轮数或每轮成本都能缩短任务时间。
符号说明：o 表示观察，a 表示动作，t 表示步数；x, y 为屏幕坐标；reward 为任务奖励。
五、常见误区
- 把“动作正确”当作“任务完成”。Slides 第 19 页明说：A correct step is not a completed task。静态评测只衡量 grounding/模仿，不衡量端到端成功。
- 以为程序化验证器一定可靠。Slides 第 25 页提示“只覆盖测试覆盖到的条件”；第 27 页的 WebArena 例子明确不检查仓库为空。逐字稿也承认存在假阴假阳。
- 以为 LLM 评委等价于人类判断。WebVoyager 一致率 85.3%、Online-Mind2Web 85.7%（Slides 第 30、32 页），仍有差距，且逐字稿提醒评委容易被欺骗。
- 只看最终交付物。Slides 第 39 页显示，最终物品对了但过程中发生了禁止的购买，应判 FAIL。
- 以为可以在评测集上训练。逐字稿明确说不应在基准上训练；工业界常见做法是生成相似任务在分布内训练，部分基准提供单独的 train/val/test 划分。
- 以为截图表示不如 HTML/无障碍树。逐字稿称基于截图的方法简单得多，且效果出奇地好；桌面场景也不易获得 HTML。但 GPT-6 等模型在可用时仍会使用无障碍表示。
- 忽略坐标归一化。逐字稿称坐标通常归一化到 0–1000，因此绝对分辨率不同也能较好迁移；但这是经验性说法，未见 slides 正式规格。
- 把 RL 当成唯一关键阶段。逐字稿强调预训练与 SFT 为 RL 提供强大基础；RL 是“点睛之笔”。
六、复习问题
- 用“买蓝色马克杯”的例子，完整描述 CUA 的一轮 observe–reason–act，并指出奖励在何时计算。（参考 Slides 第 3–8 页）
- 静态评测与端到端评测各衡量什么能力？为什么前者会产生较多假阴性？（参考 Slides 第 19、25 页）
- WebArena 的仓库创建任务中，验证器检查什么、不检查什么？这说明了程序化验证器的什么局限？（参考 Slides 第 27 页）
- 在什么情况下你会选择 LLM/VLM 评委而不是程序化验证器？它带来什么新风险？（参考 Slides 第 29–32 页）
- Odysseys 为什么使用评分细则而不是单一二元奖励？7/8 满足意味着什么？（参考 Slides 第 40–41 页）
- 长时程评测中“状态检查”和“过程检查”分别捕捉什么？举出 Slides 第 39 页的例子说明二者冲突时如何裁决。
- 描述 CUA 在第 t 步的上下文组成。文本和截图分别如何进入模型？（参考 Slides 第 53–56 页）
- 训练三阶段各解决什么问题？为什么 RL 阶段需要模拟环境？（参考 Slides 第 59、65、74 页）
- CUA-Gym 与 Gym-Anything 在“生成环境与奖励”上分别采取什么策略？它们报告了哪些数据规模？（参考 Slides 第 66–73 页）
- Slides 第 77–80 页列出哪些未解决问题？各自的核心难点是什么？
七、给定资料中的官方来源链接
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=jwGluLrrqjQ&list=PLSN0qpDfUvTM&index=7
- 官方课程日程：https://www.cmu-agents.com/#schedule
- Lecture 7 slides：https://www.cmu-agents.com/slides/lecture-07-computer-use-agents.pdf
（完）
当智能体学会“看屏幕、动鼠标”：CMU 11-768 Lecture 7 计算机使用智能体详解
来源：平台字幕与官方 Slides。内容已由用户审核确认。
本文基于 CMU 11-768 Fall 2026 Lecture 7《Computer Use Agents》的逐字稿与官方 Slides 整理，逐字稿为平台英文字幕的机器翻译清洗稿，状态：用户审核通过；Slides 文本为按页提取，图示和公式需对照官方 PDF 核验。本文已由用户审核确认，涉及图示/公式细节处标注待核。
一、问题：为什么计算机使用智能体是“另一种智能体”
这门课此前讨论的智能体，大多在文本或代码空间中工作。Lecture 7 的讲者 JY Koh 一开场就点出：计算机使用智能体（Computer Use Agent, CUA）与 sui 智能体、更基于文本的智能体“有点不同”（逐字稿 [01:42.880] 前后）。不同在哪里？
学习者解释：文本智能体的“世界”是 token 序列，它可以读写字符串、调用 API；而 CUA 的世界是渲染出来的图形界面——网页、桌面应用、手机屏幕。它接收的输入是截图，执行的动作是点击、滚动、打字、滑动、轻触（逐字稿 [02:49.680]；Slides 第 2 页）。换句话说，CUA 不在“底层接口”上操作，而是在和人类完全相同的界面上操作。
这带来了几个连锁问题：
- 输入是像素：模型必须从截图中理解界面上有什么、在哪里。
- 动作是 GUI 操作：坐标、滚动量、文本输入，而不是函数调用。
- 成功标准模糊：人类自己都未必对“任务是否完成”达成一致（逐字稿 [12:45.600]）。
Lecture 7 的主线正是围绕这三个问题展开：如何建模、如何评估、如何训练。
二、Observe–Reason–Act：CUA 的基本循环
Slides 第 3 页给出 CUA 的核心循环：用户任务和**环境观察（截图）**输入模型，模型产生“推理 + 动作”，基础设施验证并执行动作，环境状态改变后返回新截图。讲者用“买一个蓝色马克杯”贯穿全课（Slides 第 4–8 页）：
- 任务：“买一个蓝色马克杯”；初始截图显示商品列表。
- 模型推理：“我看到屏幕中央有一个蓝色马克杯，我要点击它。”输出
{"mouse": {"left_click": [33, 201]}}。 - 环境执行点击，返回新截图（商品详情页）。
- 模型推理：“配送详情在下面，我要向下滚动。”输出
{"scroll": {"dy": 520}}。 - 继续输入邮编
15213，直到模型认为完成。 - 环境计算奖励，通常 0/1 二元（Slides 第 8 页；逐字稿 [05:15.840]–[05:55.840]）。
这个循环在讲者口中被明确称为 ReAct 循环——推理与行动交错（逐字稿 [50:40.160]）。
学习者解释：ReAct 的关键在于“交错”。模型不是一次性输出完整计划，而是每一步都看到最新的屏幕状态，再决定下一步。这既降低了单步决策难度，也让智能体能从意外结果中恢复。
Slides 第 9–15 页用一条时间线回顾了 CUA 的“史前史”：
- 2017–2022：MiniWoB（Shi et al., ICML 2017）等玩具环境，合成数据、简化界面，用类似游戏智能体的强化学习训练（逐字稿 [06:40.319]–[07:25.520]）。
- 2022–2024：WebShop（Yao et al., NeurIPS 2022）引入真实商品和语言目标；WebArena（Zhou et al., ICLR 2024）用真实数据填充模拟环境；OSWorld（Xie et al., NeurIPS 2024）转向完整桌面工作流。
- 2025–2026：MyPCBench、CUA-World 等开始用编码智能体端到端生成应用和合成数据（Slides 第 13–14 页；逐字稿 [09:35.120]–[10:19.279]）。
讲者提到 GPT-6 Astra 在 CAD 软件中做 3D 建模的演示被加速了 30 倍，实际速度慢得多且昂贵（Slides 第 15 页；逐字稿 [11:23.040]–[11:42.160]）。
三、评估：什么样的“成功”才算成功
Slides 第 17 页把“成功”拆成三层：
- ACTION：点对目标了吗？适合快速的 grounding / 模仿测试。
- OUTCOME：购物车里是正确商品吗？用状态检查或证据判断。
- PROCESS：遵守约束了吗？比如“不要结账”。
讲者强调：评估者应该匹配你想衡量的能力（Slides 第 17 页）。
3.1 静态评估（02A）
静态评估在固定观察上预测动作，不展开环境（Slides 第 19 页）。例子：
- ScreenSpot-Pro（Li et al., 2025）：1,581 张高分辨率专业 UI 截图，23 个应用，3 个操作系统，评估点是 point/region matching（Slides 第 20–21 页）。Slides 第 21 页给出 Blender 中构造 UV 球体网格的任务，ground truth 框在原始像素 (417, 132)–(576, 152)，点在框内得 1 分。
- Mind2Web（Deng et al., NeurIPS 2023）：2,350 个任务、137 个网站、31 个领域，离线动作匹配（Slides 第 22–23 页）。Slides 第 23 页的例子是“在 Brooklyn–Central 租车”，ground-truth 动作是点击
bookCarTab。
讲者指出静态评估的核心缺点：假阴性。同一个截图和任务，很多不同动作都可能是合理的，但因为你只有一组真值动作，其他合理动作会被判错（逐字稿 [17:02.800]–[17:44.320]）。
3.2 网络端到端评估（02B）
端到端评估让智能体在环境中真正执行任务，最后检查结果（Slides 第 25 页）。
- 程序化状态检查：WebArena（Zhou et al., ICLR 2024）有 812 个长任务、4 个自托管域名，验证器检查环境状态而非轨迹“看起来对不对”（Slides 第 26–27 页）。Slides 第 27 页的例子是创建空仓库
awesome_llm_reading，验证器检查目标 URL 页面内容是否包含仓库名——注意它不检查“空”。讲者补充，WebArena 的验证器是 CMU 研究生手写的（逐字稿 [20:32.640]）。 - VisualWebArena（Koh et al., ACL 2024）：910 个任务、3 个自托管网站、25.2% 的图像输入任务，说明视觉理解本身改变了哪些任务可被表达（Slides 第 28 页）。
- LLM/VLM 评判：Slides 第 29 页展示用 LLM/VLM 加评分细则判断证据，例如“蓝色马克杯在购物车、价格低于 20、没有购买记录”三条都满足则 PASS。WebVoyager（He et al., ACL 2024）有 643 个任务、15 个真实网站，评判者与人类一致率 85.3%（Slides 第 30–31 页）。Online-Mind2Web（Xue et al., COLM 2025）扩展到 300 个任务、136 个真实网站，WebJudge 与人类一致率 85.7%（Slides 第 32 页）。讲者提醒：真实网站评估面临“基准漂移”，且网站所有者不喜欢被大量智能体 ping（逐字稿 [27:52.799]–[28:14.159]）。
Slides 第 33 页展示人类审查如何解决模糊结果：两位审阅者对同一轨迹给出 PASS/FAIL 分歧，最终依据“订单确认显示发生了禁止的购买”裁定 FAIL。讲者称人类审查仍是黄金标准，但非常昂贵（逐字稿 [28:37.200]–[29:20.399]）。
3.3 桌面/移动端到端评估（02C）
- OSWorld（Xie et al., NeurIPS 2024）：369 个任务、9 个应用、每个任务自带 setup 逻辑和应用特定状态检查（Slides 第 35–36 页）。Slides 第 36 页的例子是“用文件夹中的收据更新记账表”，验证器比较保存的工作簿与黄金工作簿，检查 A1:E8 不变、C9:C13 费用标签、金额和滚动余额（D9 ≈ −186.93，E9 ≈ 603.07）。
- WindowsAgentArena（Bonatti et al., 2024）：154 个任务、11 个程序、20 分钟并行扫描，用并行 VM 把多日桌面评估变成云规模扫描（Slides 第 37 页）。
- 附录幻灯片还列出 WorkArena、AndroidWorld、MobileWorld 等（Slides 第 89–92 页，待核）。
3.4 长时程评估（02D）
长时程任务可能需要人类数小时（Slides 第 39 页）。Slides 第 39 页展示混合检查：状态检查确认商品正确，过程检查发现发生了购买，最终 FAIL。讲者解释：只评结果会漏掉过程违规；长时程奖励稀疏，需要中间评分细则（逐字稿 [34:29.599]–[36:18.720]）。
- Odysseys（Jang*, Koh*, Fried & Salakhutdinov, 2026）：200 个真实网络任务，平均 6.1 条评分细则，主评估上限 100 步（Slides 第 40–41 页）。Slides 第 41 页的“棕榈泉婚礼旅行”例子有 8 条细则，7/8 满足得 0.875。
- OSWorld 2.0：108 个工作流，中位数人类时间 1.6 小时，平均 318 次智能体调用（Slides 第 42 页）。Slides 第 43 页的例子是构建 FreeCAD 支撑支架并导出匹配图纸的 STEP 模型，展示“文件存在但仍可能错误”。
- CUA-World-Long（Aggarwal et al., 2026）：200 个长任务，常需 >500 步，8 条质量准则，每个软件一个难任务（Slides 第 45 页）。
四、建模：VLM 从交错视觉历史中预测动作
Slides 第 47–55 页逐帧展示：上下文在步骤 t 包含目标、截图 0、动作 0、截图 1、动作 1……截图 t，模型输出动作 t（click/type/tool/stop）（Slides 第 54 页）。Slides 第 56 页进一步说明 tokenization：文本 → tokenizer + embeddings；截图 → vision encoder + projector；视觉 token 是连续 embedding，不是词表 ID；token 数量和编码器因系统而异。
学习者解释：这解释了为什么 CUA 本质上是 VLM（视觉语言模型）。它的“记忆”不是数据库，而是不断增长的、交错的视觉-动作序列。每一步的决策都以前面所有截图和动作为条件。
Slides 第 57 页列出前沿 CUA 的动作接口差异：GPT-6 Astra 用 Python/PyAutoGUI 或原生 computer tool，输出 pyautogui.click(x, y) 或 computer_call: actions=[…]；Fable/Opus 5 用原生 computer tool calls；Gemini 3.8 Flash 用归一化坐标 0–999 的函数调用；Qwen 3.8、Muse Spark、Kimi K3 各有 schema。讲者评论：“输出格式和 schema 因模型而异（可复现性是噩梦）”（Slides 第 57 页）。
关于动作空间，讲者说不同模型大体相似（click、scroll 等），但产生方式不同，可能是自回归输出或 XML（逐字稿 [52:48.240]–[53:23.200]）。关于为什么用截图而非 HTML，讲者认为截图简单得多，桌面没有容易获取的 HTML，且基于截图的表示“效果出奇地好”（逐字稿 [54:09.119]–[54:53.440]）。关于加载屏幕等边缘情况，常见做法是执行动作后 sleep 约 3 秒再截图（逐字稿 [55:50.160]–[56:33.119]）。
五、训练：预训练 → SFT → RL
Slides 第 59 页给出三阶段总览：
- 预训练：UI 元素和动作。boxes/labels/OCR → grounding；click/type/swipe → control（Slides 第 59–60 页）。数据来源包括 Mind2Web、AITW、OS-Atlas 等。Slides 第 60 页的例子是“CAR” ↔ 元素框/屏幕坐标，
element: bookCarTab。 - 后训练 SFT：模仿有用轨迹。人类演示（目标 + 人类轨迹 → 行为克隆）和合成数据（智能体生成轨迹 → 行为克隆）。Slides 第 61 页以 MolmoWebMix 为例：36K 人类 + 105K 合成任务轨迹；AgentNet 发布 22,625 条任务轨迹、覆盖 3 个操作系统。Slides 第 62 页展示演示数据规模增长：从 2023 年 Mind2Web 2,350 任务到 2026 年 MolmoWebMix 36K 人类 + 105K 合成。
- 后训练 RL：Slides 第 63–64 页展示“生成经验 → 评分结果 → 改进策略”。Slides 第 65 页解释为什么需要模拟环境：真钱（重复尝试可能产生重复付费预订）、真人（预订会派车，探索影响他人）、无法干净重置（取消费用、时间不可恢复）。
学习者解释：SFT 学“下一步动作预测”，RL 则评估整条轨迹的结果。讲者提到 RL 阶段把成功率从约 0.5 提升到约 6 或更多（逐字稿 [01:11:46.400] 前后），并强调 RL 对长时程任务尤其重要，因为长任务的人类演示非常昂贵（逐字稿 [01:12:09.360]）。
Slides 第 66–69 页介绍 CUA-Gym：94 个 mock web 应用、约 32K 验证元组、110 个环境（94 mock web + 16 desktop），38% 跨应用任务。Slides 第 69 页给出数据规模消融：3,578 SFT 演示 → 在 10,858 验证元组上做 GSPO，OSWorld 性能从 62.2% 提升到 72.6%。
Slides 第 70–73 页介绍 Gym-Anything：200 个软件应用、3 个操作系统、10K+ 任务；创建智能体构建，独立审计验证证据；约 2,000 条成功 SFT 轨迹；平均 checklist 分数从 12.7 提升到 22.5（满分 100），完美 checklist 通过率从 1.6% 提升到 4.4%。
六、总结与剩余问题
Slides 第 75 页总结四点：
- CUA 直接在 GUI 上操作，观察和行动与人类同一界面。
- 评估应针对正确指标和行为层次：静态动作、端到端任务、长时程工作流测量方式不同。
- 模型从交错视觉历史预测动作；CUA 是发出 GUI 动作的 VLM，有些也用代码和工具。
- 预训练 → SFT → RL。
Slides 第 77–80 页列出剩余问题：速度与成本（长上下文昂贵，Game-TARS 稀疏思考、FDM-1 直接动作预测）；个性化（GUM 推断用户上下文，记忆需可修正，用户控制至关重要）；基础设施与 UX（Codex 后台计算机使用，需要同意、可见性和关闭开关）；多智能体系统（MACU 规划 → 并行 → 重规划，并行只有在协调保持正确性时才有帮助）。
讲者在结尾问答中补充了几个重要观点：无障碍表示（accessibility tree）对鲁棒性有价值，但纯截图表示更通用（逐字稿 [43:42.560]–[45:16.160]）；评分细则通常由 LLM 生成、人类验证（逐字稿 [45:38.720]–[46:42.720]）；不应在评估集上训练，但生成相似任务训练很常见（逐字稿 [47:46.000]–[48:07.200]）；坐标通常归一化到 0–1000，能较好迁移（逐字稿 [01:08:15.280]–[01:08:37.040]）。
官方来源
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=jwGluLrrqjQ&list=PLSN0qpDfUvTM&index=7
- Lecture 7 slides：https://www.cmu-agents.com/slides/lecture-07-computer-use-agents.pdf
说明：本文依据逐字稿与 Slides 文本提取写作，Slides 中的图示、公式和布局细节需对照官方 PDF 核验；逐字稿为机器翻译清洗稿，状态：用户审核通过。文中“学习者解释”段落为补充说明，不代表讲者原话。
CMU 11-768 Fall 2026 · Lecture 7 · Computer Use Agents｜完整中文逐字稿
- 视频：官方视频
- 来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
- 说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。
正文
[00:00.560] 好的，太好了。那么，是的，今天我非常兴奋地邀请到JY Code作为我们的客座讲师。所以，嗯，JY是多模态机器学习领域的专家，尤其是计算机使用智能体方面。呃，所以，JY在学术界和工业界都有经验，他在这里的CMU作为博士生，与我以及Russ Solakutnav合作，但同时也有大量的工业界经验。 呃，所以他在开始博士研究之前曾在Google Research工作，呃，还曾在Meta的TBD实验室领导计算机使用智能体团队一年半。>> 是的。所以可以说他为嗯他们现在发布的那些强大智能体奠定了基础。而且，是的，他还构建了呃这一领域一些最具影响力的基准测试，还做了关于智能体搜索的工作。
[00:58.640] 所以他也会在课程后面做一个讲座，我们已经讲过一些他关于多智能体计算机使用的工作，期待呃今天听到关于计算机使用智能体基础的内容。谢谢。>> 好的。呃，非常感谢Daniel的友好介绍。嗯，所以今天我非常高兴来谈谈一件我个人呃多年来一直非常兴奋的事情， 到现在已经好几年了。嗯，我觉得今天人们已经理所当然地认为我们有呃计算机使用是许多前沿模型的一个核心能力。但是呃，正如我们将通过呃稍微回顾一下计算机使用智能体的历史所看到的，情况并不总是如此。呃，我觉得在很长一段时间里，嗯，人们并不清楚我们 是否想要计算机使用智能体，或者我们将如何构建强大的计算机使用智能体。嗯，所以我觉得计算机使用智能体，或者说CUA，与你们在这门课上已经讲过的智能体有点不同。嗯，所以我觉得它们有点不同，我把它们看作与嗯sui智能体和更基于文本的智能体有点不同。嗯，所以首先
[02:00.880] 什么是CUA？嗯，CUA是一个模型，是一个智能体，它观察一个，抱歉，其实不是。嗯，好了，我们继续。这里你 >> 呃，我我觉得我觉得没问题。是的，抱歉。所以，CUA是计算机使用智能体。嗯，它与你已经见过的智能体有点不同，因为它作为输入接收的实际上是图形用户界面，你在 大多数应用中拥有的那种。嗯，所以你可以想象这可能是一个网页浏览智能体。它可以是一个在你的桌面电脑上操作的智能体，或者它可以是在移动设备比如你的手机上操作的东西。嗯，所以所有这些广义上来说都是计算机使用智能体。嗯，通过人类使用的相同图形用户界面来操作的智能体。嗯，所以它们 接收呃截图作为输入，它们执行呃与人类相同类型的动作。所以点击动作、滚动动作、滑动、轻触等等。嗯，所以这里的关键区别是它们在与人类完全相同的空间中操作，我觉得这是一个非常呃它允许在建模和评估领域解决非常有趣的问题和挑战。
[03:12.000] 所以广义上来说，呃CUA按以下方式运作。你有一个用户任务，嗯，它被提供给你们的CUA。你也有一些环境观察，通常这是一张截图，呃，是你当前电脑屏幕或移动设备上任何内容的截图。嗯，这也可以是一些其他呃比如基于文本的表示，但广义上来说大多数前沿CUA呃 使用呃截图表示。所以这基本上正是用户如果尝试同样任务时会看到的东西。这两个输入都进入你的CUA，然后它产生一个推理呃轨迹，通常是这样。所以它像一个思维链，它产生一个它想要接下来执行的动作。而这个动作可以是呃你在呃你的环境中允许的任何 动作。所以它可以是点击、滚动、打字等等。然后我们处理这个动作，我们在你的环境中拥有的某个基础设施中执行它。嗯，所以如果它是一个点击动作，你实际上在计算机上执行它，你呃在执行这个动作后得到一些新的观察。
[04:15.760] 给你一个具体的例子，嗯，你可能有一个任务，比如给我买一个bloom map，你可能有一些初始环境观察，例如来自你的网页浏览器。所以这些会作为输入进入你的计算机使用智能体模型，你的模型可能会产生某种形式的推理轨迹。我看到屏幕中央有一个blue monk。我将点击它，然后它会产生 呃一些像格式化后的动作来执行。所以在这种情况下，它想要在环境上做一个左键点击。嗯，我们将在环境上执行这个动作。所以我们将实际上在环境上某处点击，我们将得到执行这个动作后发生的新观察。所以例如在这里你得到一张新的截图。嗯，在这种情况下，智能体 已经点击进入了blue mark。所以你有一种新的观察，显示了blue mark的细节。你的模型然后接收这个新观察，它根据这个更新后的观察产生一个新动作来执行。所以它可能会然后说像配送详情在下面。我将向下滚动来找到它们，它会产生这个滚动动作，我们 将再次在基础设施上执行。嗯，然后你得到一个新的观察，你基本上重复这个过程，直到嗯你的模型决定它完成了。对吧？所以这可以持续很多很多次迭代，取决于你的任务有多复杂。呃，所以是的，你从观察开始，你的模型执行一个动作，你得到一个新观察，嗯，这个过程重复 直到结束，嗯，通常你会从环境、人类用户或其他评判者那里得到某种奖励，告诉你模型是否成功完成了任务。嗯，所以通常这个奖励只是0或1的呃二元奖励，但在某些情况下也可以是嗯部分奖励。
[05:55.840] 嗯，所以这广义上来说就是计算机使用智能体的工作方式。就是这种观察、推理、行动的循环，重复直到呃你的模型耗尽它的预算，或者它认为它已经成功完成了任务。嗯，今天这看起来非常简单，呃而且模型实际上非常能干，正如我们将在几张幻灯片后看到的。嗯但这并不
[06:15.360] 总是如此。所以嗯也许我们可以做一个快速投票。那么人们认为计算机使用智能体最早是什么时候呃被发明的？嗯，人们认为是2020年之前吗？好的，有几只手。嗯，说2020到2024年这段时间是我们开始构建CUA的时候。好的。那么谁认为是在2024年之后？
[06:40.319] 嗯，所以答案实际上是，嗯，第一篇 cooler 论文出现在 2017 年。所以按照嗯人工智能的标准来看，这算是相当古老了，对吧？嗯，它实际上是 OpenAI 的一篇论文。所以他们有一个叫嗯 World of Bits 的项目，他们在其中构建了一个非常简化的玩具环境。嗯，正如你在这里看到的，它主要像是合成数据。它 是非常简化的环境，看起来可能有点像是你在移动设备上可能看到的东西，但它非常简化。嗯，完全一点都不真实。嗯，他们在这篇论文中所做的是，他们实际上做了像一堆强化学习训练。嗯，所以这并不太像现代的 LLM 强化学习，而更像是我们过去为嗯游戏智能体所做的强化学习。嗯，但
[07:25.520] 这个我觉得这个项目在当时是超前的。当时所有的模型都完全做不到这个。就像即使是这个简化的嗯 mini world of bits 任务，它们也非常不擅长。嗯，所以这大概就是第一篇 cooler 论文。嗯，在 2022 年，我觉得随着嗯语言模型的出现，更多的兴趣开始涌现。嗯，有另一篇叫 web shop 的论文 嗯，它也做了嗯有点像简化的购物环境，但它比 mini world of bits 要真实得多，正如你在这个可视化中看到的。所以它有点像点击浏览这个模拟的亚马逊网站。嗯，它仍然有点人工化，但比 mini world of bits 好多了。嗯，而在这个时候，像语言模型智能体开始在这些任务上表现
[08:08.080] 好一些了。嗯，而我们当时并没有真正以我们在 2022 年为 mini world of bits 训练它们的那种方式来训练它们。在 2024 年，实际上嗯，有这样一个我认为是开创性工作，嗯，实际上来自 CMU 的 Graham 的实验室，叫做 web arena。所以这是这样一个工作，它创建了很多模拟环境。 嗯，正如你所看到的，所有这些环境都比 web shop 和 mini world of bits 真实得多。嗯，它们包含嗯非常真实的数据，通常是从真实世界网站脚本化而来的。所以这些环境填充了来自像亚马逊和 Reddit 等等的实际数据。嗯，所以这些是嗯非常真实的数据和环境。嗯，我认为它们标志着 一个节点，嗯，模型也开始在这些问题上变得好一些了。在 2024 年，嗯，还有几乎同时发布的这个叫 OS world 的基准。所以我认为这是如今嗯最标准的计算机使用基准之一。而这也使用了嗯真实应用和有点像真实数据，但嗯它解决的是像桌面
[09:12.399] 使用，而不是网页浏览。所以它更通用一些。你可以在 Linux、Windows 嗯等等上使用很多不同的应用。嗯，所以这与 web arena 有点不同，因为 web arena 是网页浏览，但 OS world 是嗯像完整的桌面使用。嗯，我觉得事情在 2025 到 2026 年也发生了一些变化。所以这是像编码智能体变得 好很多的节点，而在这个时候，像很多人开始嗯端到端地生成环境。嗯，所以对于 web arena 和 OS world，他们所做的事情实际上是他们拿了这些应用的开源实现，然后用真实数据填充它，但所有的所有的软件都已经是由人类编写并以某种方式开源的。嗯，my PC bench 和其他 一些嗯基准在 2025 到 2026 时期实际上使用编码智能体来嗯端到端地生成整个应用和网站。所以这里展示的实际上是一个 Gmail 克隆。它不是真正的 Gmail，但看起来非常接近。而这实际上完全是由一个 LLM 一个编码智能体生成的。嗯，这里填充的所有数据也完全是合成的。所以，我们 实际上已经到了这样一个节点，很多像前沿 cooler 环境完全是被生成的，并且完全填充了合成数据，嗯，而我认为它们实际上工作得非常好。嗯，有时它们甚至比真实应用更好，因为嗯它们干净得多，也没有那么多噪声。
[10:37.200] Ka world 也是另一个来自 CMU 的项目，它采用了像真实世界的软件。所以，我认为 Kool World 非常有趣，因为这里使用的软件有点更属于长尾。所以你有一些用于像天文学和生物学以及诸如此类的软件。嗯，所以它有点它有点偏离了像仅仅是面向消费者的应用，进入了非常专业化的
[10:57.519] 桌面软件。嗯，这里的所有数据也嗯我认为是真实数据。所以它是从真实世界用例下载的数据嗯，并被加载到嗯这些环境上。嗯，这里展示的是本月嗯最近的一个来自 GPT6 Astra 的演示。所以 GBT6 Astra 是计算机使用方面的一个巨大飞跃。我认为它基本上像解决了大多数计算机使用基准。嗯，它 在这一点上像比人类基线更好。嗯，它有点比之前的模型快很多，但它仍然非常慢而且非常昂贵。所以这个演示展示的是 GPT6 Astra 操作像 CAD 软件。所以它在做像一些像 3D 建模。嗯，它实际上被加速了 30 倍。所以在实践中，像这个模型实际上比你在这里看到的 要慢 30 倍。嗯，但但重点是它非常有能力，但它仍然非常昂贵而且嗯慢。嗯，所以我们还有一些研究问题剩下，但在能力方面，我认为当前的模型嗯非常好。嗯，所以你可以看到像我们从 miniw walk 的玩具环境嗯进步到今天 9 年后，我们有了 GPT6 extra，它嗯在很多计算机任务上与你中位数的人类相当或更好。
[12:04.399] 嗯，所以非常令人兴奋，非常快速的进步，我们可以思考像我们如何衡量这些问题。嗯，如果你从研究或学术的角度来思考它们，嗯基准是嗯我们衡量这些智能体进步有多快的核心方式之一。嗯，所以我们会过一遍像 几个计算机使用基准的概览，以及它们如何衡量嗯智能体表现的不同维度。嗯，所以如果我们思考计算机使用可能如何被衡量，嗯它实际上有点我觉得它有点更有趣，嗯比我们在这门课中已经涵盖的一些其他智能体。嗯，所以计算机使用在某种意义上比像 软件工程或数学更不像是一个可验证的任务。嗯，在软件工程或数学中，你有某种单元测试或或者嗯符号验证器来告诉你是否成功完成了一个任务。对于计算机使用来说，实际上有点不同，因为你可以想象，嗯成功完成一个问题并不总是嗯人们甚至人们通常并不对此达成一致。所以它可以
[13:07.040] 像程序化验证器那样编写可能很有挑战性。嗯，所以我们有几种方法来评估计算机使用代理在一些这类基准上的表现。嗯，最明显的一种是动作是否正确。所以你可以衡量一次点击动作是否是在正确的项目上完成的。嗯，你也可以衡量结果，这可能是最端到端的方式。所以你给 它一个任务，比如给我买一个蓝色的马克杯，然后你衡量结果，看它是否真的成功购买了该物品。所以这可以通过以下方式完成：如果你有一个模拟环境，就检查状态；或者你只需查看代理的轨迹，并判断它是否提供了足够的证据表明它完成了任务。 嗯，最后，我认为你还想衡量模型是否遵守了任务中可能固有或明确规定的任何约束。所以，例如，你可能希望它导航到结账页面，但不要真正结账。嗯，所以你还有一些其他类似的验证器，以确保它在遵守任何用户定义约束的同时完成了任务。所以，我认为
[14:08.480] 对于计算机使用代理来说，评估通常非常具有挑战性。嗯，我们今天将介绍几大类评估。嗯，所以大致来说，我把它们分成了四个不同的类别。那么我们将讨论的第一个是静态评估。
[14:23.120] 嗯，让我们回到我们的任务。把20美元以下的蓝色马克杯加入购物车，并且不要购买。静态评估衡量代理轨迹上的每一个动作是否正确。所以，例如，你可能从这张截图开始，你的代理会，你知道，尝试点击这里的这个蓝色马克杯。嗯，我们可能有 某种真值参考规则，比如我们希望代理预测的XY位于一个目标框内。例如，如果预测位于一个目标框内，比如这里的这个红色框，我们就给它1分。嗯，否则我们给它0分，这意味着它在这个点上没有执行正确的下一个动作。 嗯，所以在这种情况下，对于静态评估，嗯，如果当前步骤产生了正确的动作，我们就认为它是正确的，否则就不是。嗯，所以这，因为它是静态的，你实际上无法端到端地展开代理。呃，我们只是在每一个点上考虑动作，以判断它是否成功完成了当前动作。
[15:21.839] 这一领域的一个典型基准示例被称为screen spot pro。嗯，screen spot pro是我们所称的定位基准。嗯，它测试模型将动作定位到截图中的能力，它由各种应用程序、操作系统以及大约1,500个不同的截图示例组成。在这种情况下，评估是指逐点 准确率。所以，例如这个构建UV球体网格的任务，嗯，你这里有这张类似Blender的截图。所以这是一张截图。它不是实际环境。所以代理无法访问任何交互式环境。它只是被给予这张截图，目标是判断它是否产生了一个落在这个红色
[16:04.639] 边界框内的动作。所以，如果它预测了一个落在这个边界框内的点击动作，我们就认为这是一个正确的动作。否则，我们就认为这是一个不正确的动作，并且我们不给它任何奖励。静态基准的另一个例子是mine to web，嗯，它测试更多种类的动作。所以，嗯，你可以把这看作下一个动作 模仿，在这种情况下，给定当前截图和一个任务。嗯，例如在布鲁克林中心租一辆车，嗯，我应该预测我的代理将采取的下一个动作。嗯，所以在这种情况下，真值动作是点击汽车标签，嗯，在这个任务中不会执行任何实际动作。你被给予一系列截图，你的代理必须 根据那张截图预测下一个动作。所以一旦它预测了动作，我们就给它打分，然后我们从离线数据集中提供下一张截图给它。所以再次强调，这实际上并不是在环境中执行任何动作。在这种情况下没有环境。它是一系列静态截图和真值动作。
[17:02.800] 嗯，所以你可以想象，这种评估非常快。运行起来非常便宜，因为你实际上不必支持一个环境或模拟。你只需提供截图，并衡量所预测的结果动作是否位于真值集合中。嗯，但你可以想象，静态评估的一个问题是，其中有很多 假阴性，嗯，意思是对于这张截图和这个任务，有很多不同的动作你可以点击、可以执行，而这些都会被认作是正确的，嗯，例如在这种情况下，我其实不一定非得点击汽车标签，我可以先调整日期，那也会被认为是一个有效动作， 但在静态评估中，因为你有一组真值动作，嗯，我们实际上会把那些标记为不正确。嗯，所以这是静态评估的主要缺点之一。它们非常快，但它们并没有真正衡量你关心的东西，嗯，也就是真正的端到端性能。所以我认为我们真正关心的是代理最终是否成功完成了
[18:02.400] 事情。嗯，这就把我们带到了第二种评估类型，也就是端到端评估。嗯，所以再次，让我们看看这个示例任务，比如把20美元以下的蓝色马克杯加入购物车。嗯，我们可能会让一个代理在环境中或在真实网络上执行这个任务，最后我们可以通过使用一些嗯 程序化单元测试之类的检查来验证代理是否正确完成了任务。所以，例如，嗯，一个代理可能尝试过这个任务，在轨迹结束时，我们可以运行这些类似单元测试的检查，看看它是否真的完成了。所以这是在一个模拟环境中进行的。所以我们实际上可以API访问所有后端和环境状态。嗯，所以我们可以检查 蓝色马克杯是否在购物车中，嗯，蓝色马克杯是否低于20美元，以及代理是否没有嗯实际下单。所以我们检查订单，订单数是否为零，并且只有当所有这些单独的单元测试都通过时，我们才给它1的奖励。嗯，所以这实际上是一种程序化检查。在这种情况下，你实际上有一个环境。所以代理 就是在做它需要做的任何事情。嗯，所以我们不会检查智能体在每一步执行的单个动作，但我们确实会根据这些单元测试来检查它最终是否真正完成了任务。所以如果你的单元测试足够全面，你可以想象这是一个非常好的奖励信号，嗯，因为我们并不真正
[19:26.720] 监督模型的每一个单一动作。嗯，我们只是有点像在最后检查它是否正确地完成了任务。所以 Web Arena 是首批这样做的基准测试之一。嗯，Web Arena 是一个模拟的嗯网页导航基准测试，由几个环境组成。嗯，所以你有大约 800 个任务，分布在 Web Arena 嗯托管的四个不同的嗯应用上 ，智能体将执行所有这 800 个任务，最后你有一个程序化验证器，正如我们讨论过的，来检查智能体是否正确完成了任务。所以这是一个示例任务。嗯，目标是创建一个名为 awesome LLM reading 的空仓库。所以智能体将进入这个模拟的 GitLab 嗯仓库。所以重要的是 这是一个模拟环境。它是你在本地托管的东西。嗯，对于 Web Arena，你在任何时候都不会进入真实的网络。嗯，智能体必须尝试在这个模拟环境内完成这个目标，在任务结束时我们运行这个程序化验证器，它基本上检查一系列条件。嗯，所以你本质上检查它 是否成功创建了这个仓库。嗯，所有这些验证器都是由学生嗯由卡内基梅隆大学的研究生实际上手写的。嗯，所以他们嗯他们为 Web Arena 中的每一个任务手写了这些验证器，嗯在最后，假设你知道学生们做对了，嗯在执行结束时，嗯你会有一个验证器告诉你 智能体是否成功完成了任务。嗯，仍然有一些像嗯假阴性和假阳性，因为验证器不够全面，但总体而言它是一个非常嗯我认为它是一个非常好的基准测试。它确实嗯引领了人们更加认真地对待和评估计算机使用智能体。
[21:13.360] 嗯，所以我认为将其与之前的静态评估进行对比很重要，因为这实际上衡量的是端到端的性能。所以你只是检查结果。你不关心单个步骤，只要智能体正确完成了任务。嗯，但你也可以想象这会导致一些奖励黑客行为。所以智能体 可能嗯你知道尝试去完成验证器而不是真正做那件事，这是大多数计算机使用嗯和智能体嗯评估和训练中普遍存在的一个常见问题。嗯，但我认为这在智能体评估中是一个巨大的进步。嗯，所以他们确保你在模拟沙盒中完成它，并且他们嗯衡量的是结果而不是单个步骤。
[21:55.440] 嗯，Visual Web Arena 是 Web Arena 的扩展，它有点将其视为一个多模态问题。嗯，所以在 Web Arena 发布时，大多数模型嗯使用可访问性树来处理计算机使用问题或网页浏览问题，可访问性树有点像 HTML 的简化版本，而 Visual Web Arena 实际上嗯试图说服人们你有点想 使用基于截图的方法来操作。所以 Visual Web Arena 是 Web Arena 的扩展，包含 910 个嗯视觉基础任务。所以 Visual Web Arena 中的所有问题都必须使用多模态表示来解决。嗯，评估有点类似。所以你也有像程序化检查。嗯，但你也有一些基于多模态的嗯
[22:36.720] 程序化检查。所以嗯有些问题如果你纯粹依赖 HTML 是无法解决的。这些基准测试的另一种评估形式是，不使用程序化验证器，而是使用一个不同的 LLM 来判断模型是否正确完成了任务。所以给定模型的轨迹，你用另一个 VLM 或 LLM 来提示。嗯，所以是一个基于文本的 LLM 或多模态嗯视觉语言模型，嗯你用模型的轨迹来提示它，你问它是否成功完成了任务，这也是一个非常流行的评估指标，嗯因为并非总是能够为你关心的每一个任务编写嗯验证器。嗯所以在这种情况下你再次有智能体嗯 完成的轨迹，而不是程序化验证器，你有一堆评分标准，或者也许你甚至只是把任务提供给 LLM。嗯所以你提供智能体的轨迹，你提供任务，LLM 有点告诉你它是否认为智能体成功完成了任务。
[23:39.120] 嗯，所以通常像这些评判者，它们会产生一个简短的思维链，然后是一个判断，例如通过或失败，或者有时嗯你会分配像嗯浮点分数，但通常评判者告诉你你是否通过或失败。嗯所以这更通用一些，因为你不需要为基准测试中的每一个任务编写验证器。嗯但它也有点更 容易出现像假嗯假阳性和假阴性，因为嗯你可以想象你的 LLM 评判者很容易被愚弄，认为一个任务被成功完成了，而实际上并没有。嗯但总的来说这更加嗯灵活，嗯几个使用这种评判形式的基准测试包括 Web Voyager，它是最早测试计算机使用智能体
[24:21.039] 在真实网络上导航能力的基准测试之一。嗯所以这与 Web Arena 非常不同，因为嗯他们所做的是他们实际上想出了 15 个不同的网站和这些网站上的 600 个任务，他们告诉嗯基准测试是实际进入真实网络上的这些网站来完成某些目标。嗯所以实际上如果你 进入基准测试中的一些网站，比如有一个叫 All Recipes 的食谱网站。如果你看看智能体应该做的一些任务，你实际上可以在真实网络上看到之前智能体尝试这个任务的一些痕迹。嗯这实际上不太好，但你可以嗯看到一些证据说明为什么我们可能不想在真实网络上评估智能体。
[24:58.960] 嗯但尽管如此，像 Web Voyager 仍然相当有价值，因为它在非模拟环境中测试了这些智能体的能力。所以它基本上是你所能得到的最接近真实世界评估的东西。嗯它让智能体去在真实网络上做这些事情。嗯例如像找到关于气候变化数据可视化的最多 star 的 GitHub 项目。嗯他们他们 呃 部署 就像 谁 要 去 真正 的 GitHub 上 做 一大堆 搜索 然后 呃 找到 这个 任务 实际 上 最 多 星 的 GitHub 仓库。呃 在 这个 情况 下 因为 它 是 在 真正 的 网络 上 运行 你 其实 没 办法 有 程序化 的 验证器 因为 你 其实 没 办法 呃 查询 真正 的 网络 来 获取 它 的 状态。呃 所以 这里 的 评估 就是 我们 之前 讨论 过 的 VLM 评委。所以 这个 评委 是 一个 不同 的 LLM 它 会 接收 到 智能体 已经 采取 和 观察 到 的 一 系列 动作 和 截图。呃 它 会 接收 到 原始 任务。所以 它 会 看到 这个 任务。它 会 看到 智能体 做 了 什么 然后 它 大致 预测 智能体 是否 成功 了。所以 我们 用 类似 VLM 评委 的 方式 来 评估 这个 问题。呃 没有 程序化 的 验证器。呃 所以 这 有点 像 你 可以 把 它 想 成 相当 呃 蛮荒
[26:07.600] 蛮荒 西部 式 的 评估。你 在 真正 的 网络 上 部署 一些 东西 你 运行 一个 评委 然后 你 呃 你 知道 归根结底 说 它 是否 成功 了。但 我 觉得 很 多 人 喜欢 这个 因为 它 在 某种 程度 上 是 你 能 得到 的 最 接近 真实 世界 评估 的 东西。
[26:22.159] Online Mind2Web 是 一个 是 一个 不同 的 评估 它 是 对 WebVoyager 的 后续 工作。呃 所以 它 更 全面 一些。评委 也 更 强 一些。所以 他们 展示 了 评委 实际 上 与 人类 有 大约 85% 的 一致 性。呃 所以 它 非常 接近 呃 达到 呃 评委 的 人类 水平 一致 性。呃 但 本质上 呃 Online Mind2Web 也 关注 真实 世界 的 评估。呃 所以 他们 扩展 到 了 大约 136 个 网站 和 300 个 任务 然后 他们 让 LLM 去 这 136 个 真实 网站 上 完成 某些 任务。呃 他们 他们 展示 了 呃 它 实际上 呃 使用 这个 评委 比 起 WebVoyager 的 评委 要 更 准确 一些。呃 所以 他们 有 一个 稍微 更 复杂 的 评委 有 不同 的 评分 标准 和 准则。呃 但 本质上 这个 也 是 在 真正 的 网络 上 运行 它 继承 了 它 继承 了 很 多 在 真正 的 网络 上 运行 所 带来 的 问题 类似 于 WebVoyager。呃 但 再次 说 这 是 一个 非常 好 的 评估 因为 它 在 某种 程度 上 是 你 能 得到 的 最 接近 真实 世界 评估 的 东西。呃 所以 这 是 完全 自动化 的 但 你 有 一个 LLM 在 评判 任务。你 有 一个 LLM 在 真实 世界 中 尝试 任务。
[27:31.919] 呃 因为 呃 评委 非常 接近 人类 评估 有点 像 有 这样 一个 自动化 流水线 一个 LLM 在 真实 世界 中 做 某件事 然后 你 有 一个 类 人类 的 评委 来 判断 它 是否 成功 了。所以 我 觉得 这 是 一个 非常 好 的 基准 呃 非常 自动化 呃 但 再次 说 我 觉得 人们 最近 其实 不 太 用 这个 了 因为 它 呃 基本上 被 现代 模型 解决 了 而且 我 觉得 随着 酷 词 变得 越来越 流行 呃 人们 在 把 它们 部署 到 真实 世界 上 也 变得 更加 谨慎 了 呃 因为 这些 网站 所有者 真的 不 喜欢 有 大量 智能体 不断 地 ping 他们 的 网站 来 做 基准 测试 呃 是 的 你 可以 想象 如果 你 的 网站 被 包含 在 这个 基准 里 然后 你
[28:14.159] 开始 获得 更 多 流量 呃 你 大概 也 不会 很 高兴。是 的。呃 所以 我 觉得 一个 重要 的 事情 是 我们 讨论 的 所有 这些 评估 指标 都 非常 呃 它们 是 程序化 验证器 或者 它们 是 LLM 评委。呃 所以 它们 有点 像 代理 指标 呃 来 判断 我们 是否 成功 完成 了 一个 任务。
[28:37.200] 呃 但 人类 审查 意思 是 呃 让 人类 来 判断 智能体 是否 做 了 某件事 可能 仍然 是 黄金 标准。所以 呃 这些 论文 也 跑 了 类似 人类 评估 基线 他们 让 人类 来 审查 智能体 的 轨迹 然后 呃 每个 轨迹 你 会 有 几个 人类 来 判断 智能体 是否 成功 了。呃 最后 你 呃 有点 像 让 人类 来 投票 智能体 是否 成功 了。呃 这 在 某种 程度 上 像 仍然 是 判断 智能体 是否 成功 完成 任务 的 黄金 标准 但 这 运行 起来 非常 昂贵。所以 在 你 的 基准 中 的 每 一个 任务 上 或者 你 想 做 的 每 一次 呃 类似 调试 或 验证 运行 中 都 做 这个 其实 是 不 太 可能 的。呃 但 这 是 大多数 基准 创建者 仍然 会 做 的 事情。呃 让 人类 来 尝试 任务 或 评判 任务 然后 判断 智能体 是否 在 做 正确 的 事情。呃 但 这 非常 昂贵。所以 通常 只 会 做 一 两 次 来 确保 你 的 智能体 或 你 的 基准 是 呃 合理 的。
[29:44.320] 所以 人们 通常 做 的 另 一 种 评估 是 呃 它们 也 是 端到端 评估 但 它们 是 在 类似 桌面 电脑 或 移动 设备 上 做 的 而 不是 在 网络 上。呃 所以 网络 很 有趣 因为 你 可以 在 网络 上 做 很 多 事情。呃 我 个人 觉得 我 大部分 时间 也 是 花 在 Chrome 上。呃 但 有 一 组 不同 的 任务 呃
[30:04.240] 需要 你 操作 一个 实际 的 虚拟机 或 一个 实际 的 桌面 然后 呃 也 有 几个 基准 以 端到端 的 方式 来 衡量 这个。呃 我们 已经 讨论 过 OSWorld 它 是 最 经典 的 CUA 基准 之 一。呃 所以 这 是 一个 你 可以 启动 的 真实 Linux 虚拟机。它 包含 大约 369 个 任务 横跨 九 个 不同 的 应用 比如 呃 呃 LibreOffice 呃 Chrome 等等。呃 它 是 369 个 任务 横跨 九 个 不同 的 应用。对于 每 一个 这些 任务 他们 也 有 程序化 验证器。所以 你 的 智能体 必须 呃 进入 这些 虚拟机 横跨 多个 不同 应用 尝试 任务 然后 在 最后 我们 运行 一些 呃 类似 单元 测试 的 检查 来 看 它 是否 成功 了。
[30:51.600] 所以 OSWorld 中 任务 的 一些 例子 比如 呃 你 可能 想 要 更新 呃 这个 LibreOffice 呃 电子表格 呃 使用 另 一个 文件夹 中 的 收据。所以 在 这个 情况 下 你 的 智能体 必须 导航 到 那个 文件夹 找到 收据 它 必须 输入 类似 合适 的 呃 数值 到 这个 呃 工作簿 中 然后 它 必须 保存 这个 工作簿。所以 它 必须 做 这个 呃 横跨 不同 应用 导航 呃 处理 类似 在 这个 呃 LibreOffice 中 输入 条目 然后 呃 基本上 基于 仅仅 一个 初始 用户 指令 完成 这个 任务。在 智能体 认为 它 完成 之后 我们 运行 所有 这些 类似 单元 测试。所以 我们 检查 电子表格 是否 包含 我们 认为 呃 应该 在 那里 的 条目 然后 我们 给 它 再次 分配 一个 0 1 分数 来 表示 我们 认为 它 是否 成功 了。所以 有点 像 在 精神 上 与 WebArena 非常 相似 但 呃 在 这个 情况 下 它 使用 的 是 呃 真实 世界 的 应用。所以 这些 是 人们 日常 实际 使用 的 应用 然后 它 再次 在 一个 完整 的 虚拟机 中 操作。所以 它 不再 仅仅 局限 于 呃 Chrome 或 网络 了。它 呃 在 这个 之上 操作
[31:56.960] 虚拟机，包含九个不同的应用程序。嗯，所以它的范围要广得多，你可以想象，有了桌面代理，我们可以自动化比只使用网页浏览代理更广泛的任务。嗯，Windows agent arena 是另一个基准测试，它在某种程度上受到了 OS World 的启发。嗯，但它是微软创建的一个基准测试，用来测试 代理在 Windows 机器上的性能。嗯，所以世界上很多很多生产性工作实际上都发生在 Windows 机器上，嗯，我认为这个基准测试相当不错，因为它衡量了代理执行一些在企业工作流或其他公司中非常常见的任务的能力。 是的。所以嗯，同样，就像你启动一个 Windows 虚拟机，让一个代理尝试一项任务，最后你有一些嗯验证器来确定它是否正确完成了任务。而且同样，因为我们使用的是程序化验证器，嗯，其中一些也非常容易出现假阳性或假阴性，但嗯，你知道它们有它们的优点，嗯，就像 LLM
[33:00.559] 评判器一样。嗯，但这些本质上是人们从 2024 到 2025 年期间一直在探索的一些核心端到端评估基准。是的。嗯，这些基准测试中的大多数实际上已经被现代冷却器（coolers）相当好地解决了。所以例如，最近的 GPT6 发布，我认为在大多数这些基准测试上达到了远高于人类的表现。所以嗯，
[33:27.519] 社区最近实际上嗯[清嗓子]嗯对我刚提到的这些基准测试投入的注意力少了一些，嗯，他们开始关注一些更像长时程或嗯呃专门化的基准测试，我想这么说。所以长时程计算机使用是另一个，它它实际上是同样的任务，但我们关心的问题跨越嗯长得多的时程。所以一些 这些任务可能需要真正的人类花费数小时，甚至像两到三个小时甚至更长时间才能完成。嗯，所以现在我们在真正地对 kas 完成人类实际上不想做的事情的能力进行压力测试，或者我们让它们完成越来越长的任务。我们在 OS world、web arena 中看到的很多任务嗯通常需要人类大约 10 分钟或 20 分钟或更少才能完成。嗯，但我们将在接下来的幻灯片中看到的一些任务可能需要几个小时，并且可能操作专门化的软件。所以，我们真的到了试图让它们比专家级人类使用计算机更好的地步。
[34:29.599] 嗯，很多这些长时程基准测试，它们遵循非常相似的评估策略。所以你有一种动作轨迹，它代表了 LLM 在这个环境上采取了什么动作。嗯，你有几个不同的检查，因为它是长时程的。嗯，人们倾向于采用更像评分细则（rubric）的方法。所以那个我们有一个单一结果 奖励来判断它是否成功完成了的版本，对长时程来说不太有用，因为它非常稀疏。嗯，奖励非常稀疏。嗯，代理在中间可能做了很多有用的事情，但未能获得满分。嗯，所以我们对长时程计算机使用有更多像细粒度的奖励。嗯，这里有几个不同的检查。所以状态检查就是 我提到的中间奖励。嗯，这些有点像基于评分细则的奖励，它们检查代理是否成功完成了一个中间任务，然后你有一种过程奖励，嗯，这本质上检查代理是否成功端到端地完成了整个任务。嗯，所以这有点像最终奖励，但我们有一些中间 奖励，以确保我们对长时程计算机使用有一些非零信号。嗯，当然你总是可以有一个最终裁决来判断代理是否成功完成了。嗯，但同样，因为我们在看长时程计算机使用，嗯，这非常稀疏。所以很多时候，很多模型在最终奖励上得到嗯零分，这就是为什么 你有点想要有中间评分细则或嗯呃过程奖励。是的。而且重要的是，嗯，正如我提到的，在最后有一个好的交付物并不总是长时程计算机使用的一个好评估指标，因为它太长了。嗯，所以你有点想要有一些临时奖励，以确保模型在此期间做了有成效的事情。
[36:18.720] 呃，Odyssey's 是我们在 CMU 这里构建的一个基准测试，它衡量长时程嗯网页浏览。所以 Odyssey's 包含 200 个任务。嗯，我们为每个任务都有嗯评分细则。所以，我们有一组嗯我们认为模型应该完成的事情。呃，所以，例如，对于这个任务，嗯计划一次去棕榈泉婚礼的旅行。嗯，有八个评分细则，模型需要 满足才能完美地完成任务。呃，例如，你需要找到一趟从匹兹堡到洛杉矶的嗯直飞航班。嗯，你需要嗯找到在上午 9:00 到下午 4:00 时间窗口内的驾车时间。而且你有一堆这样的标准，你需要满足才能成功完成任务。嗯，所以 Odyssey's 有点像是在许多 不同的评分细则上给表现打分，嗯，让你可以有部分奖励，而不是在最后只有一个单一的二元通过或失败嗯结果。是的。所以嗯这个例子我已经提到了，我们有点像是八个评分细则项，嗯，模型需要满足每个评分细则项才能达到完美的分数，理想情况下。
[37:23.119] 嗯，所以这有点像是介于两者之间，呃，它它大部分仍然是由 LLM 嗯验证的，但你有点像是有了细粒度的 LLM 验证，因为你不只是在最后产生一个单一的通过或失败嗯奖励。嗯，OS World 也提出了一个后续版本，叫做 OS World 2，在这个基准测试中，他们实际上试图处理像长达一小时的工作流。所以 OS World 1.0 你知道，是很多只需要几分钟人类时间就能完成的任务，而 OS World 2 实际上有大约 108 个任务，嗯，这些任务中位数需要人类 1.6 小时才能完成。嗯，所以这非常嗯，其中一些工作流非常复杂。嗯，它们需要嗯进行 CAD 建模或 3D 建模或照片编辑。嗯，所以我们正在进入更 专门化嗯软件使用的领域。嗯呃，平均而言，他们基准测试的大多数前沿代理需要大约 300 次调用。呃，所以这也嗯通常，它是对一个代理进行几个小时的基准测试。呃，所以我认为这到了这样一个地步，它有趣得多，因为你可以看到这些代理嗯甚至像一次高效地工作几个小时。嗯，所以它
[38:35.920] 评估起来非常昂贵，但看到从 Eval 开始、一小时后回来发现智能体仍在其中一些任务上高效地取得进展，这确实很神奇。这是一个来自 OS World 2 的示例任务，嗯，我不知道，我不了解中位数人类水平，但我肯定无法轻松做到这一点。所以我认为我们正在达到 拥有专家级任务的水平。嗯，所以这里的任务是构建一个猫支撑支架，并导出一个与这张图纸匹配的模型。所以你得到了左侧的这个参考图纸，模型的目标是能够通过操作某种计算机辅助设计软件来生成右侧的这个东西。嗯，所以 我认为这实际上非常复杂。嗯，它需要像在三维空间中导航。它需要知道在这个环境中有哪些工具可用，而且你需要能够持续工作大概好几个小时，因为嗯，人类也需要大约一个小时或至少半个小时才能完成这个任务。
[39:33.920] 嗯，所以它有很多非常有趣的东西。嗯，这是一个 GPT6 模型尝试 OS World 2 其中一个任务的可视化。所以再次强调，这是加速了 30 倍的。所以实际上要慢得多，但你可以看到它在终端环境之间导航、回到 CAD 软件，并且实际上自主地进行三维建模 大约，我想大概一个小时的时间。嗯，所以它实际上能够相当好地设计这些三维零件。嗯，这相比我们甚至两年前拥有的模型是一个巨大的飞跃。嗯，所以我们正在达到这样一个点，这些模型非常擅长尝试许多这样的长时程路径。是的。
[40:19.280] 嗯，所以我还会提到几个其他基准测试。所以 Cool World Long 是另一个来自 CMU 的基准测试，嗯，它试图解决长时程计算机使用的问题。嗯，所以再次强调，这是非常专业化的环境，涵盖各种任务，例如医疗保健任务、工程任务，或者只是建筑和设计任务。嗯，包含 200 个这类任务， 其中大多数需要 500 次或更多 LLM 调用。嗯，所以再次强调，这是几个小时的人类工作时间。嗯，当这个基准测试在今年早些时候发布时，大多数模型在这个基准测试上得分非常非常差，但嗯，我认为最近的模型包括 GPT6 嗯实际上在这个基准测试上达到了接近 90% 的性能。所以，呃，长时程计算机使用以及
[41:04.240] 一般的计算机使用非常，呃，非常接近被解决了，或者有些人甚至说已经解决了，因为这些模型在许多这些通用软件使用任务上能够超越人类。嗯，所以希望我已经说服了你们，呃，其中一些基准测试是有用的，呃，或者大多数在某个时间点是有用的。嗯，呃，我认为，我希望我已经说服了你们 它们非常有能力，并且有非常有趣的嗯，呃，能力。呃，它们现在非常擅长自动化许多不同类型的任务。嗯，所以我认为自然的下一个问题是，我们如何，这些模型是如何构建的？我们如何构建这些模型？我们如何训练它们？我们如何向它们引入这类能力？嗯，所以我们会稍微谈谈，
[41:50.319] 建模通常是如何工作的，以及嗯这些模型是如何训练的。但我想暂停几秒钟，看看有没有关于基准测试的问题，因为我知道我们涵盖了很多不同的东西。 >> 是的。所以对象[清嗓子]对于有双星号的东西。
[42:23.040] >> 是的。所以问题是这些基准测试中是否有任何一个对中间步骤分配部分评分奖励。所以例如，如果任务是向购物车添加一些高评分的东西，你确实做到了，但你添加了一些不相关的物品，你会因此被扣分吗？嗯，我认为这是个好问题。呃，我认为奖励设计在很多 这些基准测试中非常棘手。所以嗯，Web Arena 中的一些任务确实有这种惩罚，嗯，但很多时候人们设计这些基准测试是为了让答案是不含糊的。所以问题在某种意义上是有些人工化的。所以例如 Web Arena 可能会说将评分最高的物品添加到我的购物车，如果你这样做了，那么嗯，你 就得到满分，但呃，我见过智能体通过向购物车添加大量东西来奖励黑客，嗯，试图完成目标。所以这很大程度上取决于基准测试。我认为通常人们并不会以如此细粒度的方式惩罚它，但这也是因为在这些基准测试创建的时候，智能体 太差了，所以我认为一些奖励分数一开始就比较宽松。嗯，所以我认为有些基准测试确实有这些东西，但并不是，我不会说它非常普遍。是的，我认为人们倾向于偏向于给出假阳性而非假阴性。
[43:42.560] >> 是的，>> 这不完全是一个基准测试的问题，但你认为观察其他非……有好处吗？ >> 是的。所以问题是使用除基于像素的表示（如截图）之外的其他模态是否有好处。我认为肯定有。嗯，OpenAI 说过，嗯，至少以前的模型，我假设 Astra 也是，它们在可用时使用 计算机的无障碍表示。所以 Mac OS 有这种无障碍表示，本质上是为了视觉障碍用户。嗯，它有点像 XML 的表示，它们确定并告诉你，比如这个区域是某个物品的描述，这个区域是我的 Chrome 窗口，这个区域是我的 Dock，等等。 嗯，所以模型至少在可用时确实使用这些表示，GPT6 是这样的。我认为这有很多价值，嗯，特别是如果你想让你的模型更鲁棒。嗯，我个人真的很喜欢纯粹的基于截图的表示，因为我认为它非常通用。所以你训练一个只接收截图 表示的模型。你可以将它部署在任何设备上，而无需访问设备的底层内部结构。所以你可以在 Windows 上训练一个模型，它大概能更容易地泛化到 Linux 或 Mac 计算机，嗯，因为它们有不同的底层无障碍表示，例如。嗯，但是的，我认为模型
[45:16.160] 肯定会使用替代表示。是的。是的。>> 是的。我的问题是关于评分标准本身的。>> 是的。>> 我假设每个任务点的评分标准是不同的。那么，一个是专家生成的评分标准，还是像你生成的那种合成评分标准，然后在奖励方面，它们是如何——有哪些方法可以缓解来自评分标准的（奖励黑客问题）？
[45:38.720] >> 是的。所以问题是，评分标准是由人类生成的，还是我猜是由LLM合成生成的，以及奖励黑客是否会在评分标准中出现，以及你如何缓解这个问题。嗯，至少对于Odysseies来说，大多数评分标准是LLM生成的，但经过人工验证。嗯，所以生成这些评分标准的一种常见方式实际上与上一个问题有关，即 你给模型一些特权信息。所以你给它比在评估时更多的信息。呃，例如一些提示或一些HTML表示，然后你让它表现得比在现实评估场景中更好，基于此你可以生成更准确的评分标准，因为你有特权信息。嗯，所以对于 Odysseies，而且我认为对于很多工业界的基准测试来说，如今评分标准通常是由LLM生成、由人类验证的。嗯，有时甚至不是人类验证，而是由另一个具有潜在不同信息的LLM来验证。但是嗯，是的，我认为如今做合成生成是相当合理的，但我个人 仍然觉得，在某个时间点，人类必须查看你放入基准测试中的一切，以确保它是合法的。嗯，我认为奖励黑客在训练这些模型时确实经常发生。嗯，是的，我想我们可以在训练部分再讨论一下，但基本上我认为奖励黑客是一个巨大的问题。我会 说，基于评分标准的奖励实际上比基于结果的奖励能更好地缓解奖励黑客。嗯，因为你有一些需要满足的不同标准。嗯，例如，如果你只是想买评分最高的商品，那么你可以通过把网站上每一件东西都加到购物车里来实施奖励黑客。
[47:21.440] 而你可以想象，基于评分标准的奖励会检查这一点。是的。>> 是的。>> 你对这些学术基准是被用于评估，还是人们也会在上面训练，无论是学术界还是工业界，有什么了解吗？>> 是的。所以问题是，人们是否在学术界或工业界在这些基准上训练。呃，嗯，你不应该在这些基准上训练， 因为你基本上是在你训练的东西上测试。但我认为在工业界，生成与你的基准非常相似的任务是相当常见的。嗯，所以人们通常会产生类似OS World或Web Arena的任务，并在那些任务上训练。嗯，这样它就处于分布内，但你并不是真正 在同一个测试集上训练。所以我认为这非常常见。嗯，Cool World实际上也有一个明确的训练集。嗯，所以对于一些基准测试，它们有训练、验证，理想情况下还有测试划分，人们可以在上面训练，但嗯，你不应该在真正的评估集上训练，不过我认为人们确实会找到代理信号来本质上做到这一点。是的，
[48:30.000] 不幸的是，Cool，是的，也许是最后一个问题，然后我们可以继续。是的。>> 我很好奇，表示是什么？动作空间是什么？>> 是的。嗯，>> 是的。所以问题是，对于CUA来说，动作空间实际上是什么。嗯，我我想我们会很快涵盖这一点。
[48:54.079] 所以，也许我们可以用几张幻灯片来做这件事。是的。但基本上，嗯，这取决于每个模型，也取决于你想把模型部署在什么环境中。嗯，是的。所以，我们如何——CUA是如何表示的？正如你现在可能已经猜到的，就像如今AI中的大多数模型一样，一个CUA基本上就是一个LLM，被提示并以某种格式接受输入。 嗯，更具体地说，它通常是一个视觉语言模型，因为大多数模型接受截图表示作为输入。所以你有一个目标，通常用自然语言表示。比如，再次嗯，抱歉多次使用同一个例子，但目标可能是像“给我买一个低于20美元的蓝色Monk”。然后你可能有一个 初始截图，它基于你的环境最初拥有的内容。所以你有一个目标和一张截图。呃，这进入你的策略，本质上是一个LLM。嗯，然后你的LLM产生一个动作。嗯，所以通常这表示为文本。所以你的LLM产生某种结构化的文本表示，表示它想要 产生的动作。所以这可能是一个带有X和Y坐标的点击动作。呃，我们再次执行这个动作。嗯，我们把它添加到下一步喂给LLM的上下文中。嗯，我们在执行这个动作后也会得到一张新截图。所以我们在执行动作一或动作零后得到截图一。
[50:19.119] 我们再次把它拼接到我们已经拥有的上下文中。嗯，我们再次把它喂给LLM，它产生我们接下来应该执行的下一个动作。然后我们再次在截图一上执行这个动作，嗯，得到一张新截图，我们把它拼接到上下文中，我们再次把它喂给LLM。所以，嗯，你重复这个过程，直到你的LLM认为完成了，或者
[50:40.160] 直到，呃，我想，你用完预算为止。是的。嗯，所以你有这个循环。嗯，它通常被称为ReAct循环，即推理与行动。嗯，在每一步，你把你产生的动作、新截图拼接到你不断增长的上下文中，然后把它反馈给LLM。嗯，所以它是一个交错的——像视觉语言交错的——动作历史。你有 自然语言形式的目标、动作。你还有像素输入或潜在的其他元数据。嗯，这本质上被拼接成这个长序列，这个长的多模态序列，你把它喂给你的语言模型。嗯，所以它会不断增长。嗯，人们过去有不同的方式来处理越来越长的上下文，但基本上，呃， 你在每一步把上下文喂给LLM，它在那个时间点生成下一个动作。嗯，当然，我们实际上并不会喂入实际的像素输入。呃，通常对于大多数语言模型，你会以某种方式对它进行分词。所以，你会把你的自然语言分词为文本token。你会把你的截图转换为像 视觉 token，或者类似基于像素的表示。现代语言模型的好处在于，这通常就直接拼接在一起。所以，对模型来说，看起来就是你有一堆 embedding，你在每一步都把它们喂进去。模型产生一个动作，然后你把它 token 化，然后你
[52:04.640] 把它拼接到上下文中。所以你基本上就是重复这个过程，每一步喂进去的 token 数量不断增加。模型接收这些，在每个时间点都以上下文为条件，产生下一个动作，你就这样重复。所以我认为这是一种非常简洁优雅的表示方式，而且效果出奇地好。 是的。所以你通常有文本，你把它转换成 token embedding。你有截图，你把它转换成视觉 embedding。所有这些只是交错和拼接在一起，传给你的语言模型。如今你的语言模型是一个 transformer。显然，它会为下一轮产生动作。
[52:48.240] 是的。所以之前关于我们通常使用什么动作空间的问题。不同的模型有不同的动作空间。如果你眯着眼看，它们大体上看起来相似。所以它们通常有一组可以产生的动作，比如点击动作、滚动动作，或者你在环境中有的其他什么动作，对吧？所以不同的模型产生 它们的方式略有不同。例如，你可能把它们产生为类似 pi 自回归输出，或者你可能把它们产生为类似两次调用或 XML。然后你必须处理这个，并把它传给你实际上能在环境中执行的东西。
[53:23.200] 是的。后面有一个问题。是的。所以问题是为什么我们使用截图而不是 HTML？我认为 HTML 也是有效的——嗯，HTML 非常混乱，但人们做过简化版的 HTML 作为表示。我猜一个原因是，大多数最近的模型不使用——或者说它们主要依赖截图。它们有时仍然使用 元数据，但总的来说，大多数最近的模型使用截图作为输入。我认为原因是它简单得多。所以给你的电脑截个图，比试图处理底层的无障碍元数据要简单得多。你可以在网页上轻松获取 HTML，但你 不太容易对桌面电脑使用做到这一点，对吧？所以出于这个原因，我认为人们更倾向于使用截图，因为它简单得多。而且我认为对某些人来说有些意外的是，基于截图的表示已经达到了非常非常强的性能，所以使用 HTML 或其他 结构化表示似乎几乎是不必要的。所以我想这大概就是——为什么？它非常简单，而且效果出奇地好。>> 是的，好问题。我认为这是甚至在一年前人们还不太容易接受的东西。但是，是的，多模态模型非常强大。所以做这件事出奇地容易。
[55:21.599] >> 是的，还有一个问题。>> 所以如果是要选一个按钮之类的。那么加载屏幕也是。>> 是的。所以关于如何处理这些边缘情况，你可以做很多不同的设计选择。抱歉，问题是——问题是，你是会立即截取加载中的屏幕截图，还是当你 执行一个动作时你会怎么做。所以我认为电脑使用既烦人又有趣，因为与数学或编程模型相比，你会遇到很多这类边缘情况和基础设施问题。这可以通过几种方式解决。我认为人们最常用的方式是，你通常在执行动作后 sleep 或等待几秒钟。 所以你会执行一个动作，然后 sleep 大约 3 秒，再截取屏幕截图并执行下一个操作。所以如果你在那个时间点确实看到了加载屏幕，模型通常被训练为再等待一段指定的时间。是的，还有更复杂的基础设施——有更
[56:33.119] 复杂的模型设计方式，你等待直到屏幕截图发生变化，但我认为最简单和最常见的处理方式就是直接 sleep。是的。非常感谢大家的提问。所以这基本上就是在高层面上，据我所知，几乎所有前沿模型是如何被训练的。
[56:57.280] 抱歉，这不是它们如何被训练的。这是它们如何被建模的，我们很快会看一下它们实际上是如何被训练来做这些相当惊人的事情的。好的。那么关于训练模型，基本上大多数电脑使用 agent 都是基于预训练的视觉语言模型。所以训练发生在几个不同的阶段，比如
[57:25.119] 显然当你部署 GPD6 作为电脑使用 agent 时，你不是在部署——我想这不显然，但你不是在部署一个只能做电脑使用的 GPD6 专用版本。所以它是一个通用模型，电脑使用是它的能力之一，我认为这越来越是人们采用的方式。所以电脑使用特定的训练
[57:45.200] 发生在几个不同的阶段。在预训练阶段，模型用大规模互联网数据训练时，核心特定数据通常以几种不同格式注入。所以你有你的 grounding 数据，这是鼓励模型学习在屏幕截图中 grounding 信息的数据。所以这些数据通常看起来是 这样的形式：你有一张屏幕截图，然后你有一系列边界框后面跟着坐标。所以模型在这里学习的是 grounding 能力。它学习的是，如果我想点击按钮，我必须产生这些 XY 坐标。所以它只是在学习最基本的能力，比如 学习定位、grounding 等等。你可能还有一些以即时动作形式出现的数据。这实际上与我们讨论过的静态评估非常相似。你可能有这些屏幕截图，你可能有某个高层次任务，这里的目标是预测你将产生的下一个动作。所以 这再次像是非常嘈杂的数据。它甚至可能是合成生成的。但预训练的目标只是引入某种形式的定位和接地能力，这是 KUA 之后需要做的，以便能够产生准确的点击和其他类型的动作。是的。
[59:07.280] 在预训练之后，我们通常会有一个后训练阶段。所以如今这通常与其他模型能力混合在一起，但为了简化，我们可以假设我们现在只关心 cooler。在这个后训练阶段，有两个不同的阶段。第一个阶段是你通常做监督微调，来训练 模型模仿人类对计算机使用任务的演示。所以你有一堆可以收集或以其他方式从某处抓取的人类演示。这通常看起来很像我们已经有的轨迹。所以给人类一张截图，人类做某事，然后你记录那个动作。所以它可能是一个点击动作 人类做的，你用某种你有的专门软件来记录。所以你有一堆这样的人类演示，本质上是一些人类记录人类做酷炫任务。如今，制作合成数据也非常常见。也许你以某种方式提示你的智能体，或者你提示一个更强的模型尝试 任务，然后你保留你认为好的轨迹。所以这是由不同模型或同一模型合成生成的。你也把它混合进来。所以这通常比人类数据质量更低、更嘈杂。但你可以大规模生成这个。所以你不需要人类实际生产这个 数据。所以在你的后训练阶段拥有这种合成模型生成的数据通常仍然相当有用。我认为更有趣的部分是在这个后训练监督微调阶段之后，你的模型有了一些 cooler 能力的雏形。所以在 SFT 之后，你通常可以提示你的模型，它会在大多数这些 cooler 任务上做得不错。
[01:00:46.480] 我认为更有趣的阶段是，如今这在一年前还不太常见，但如今 KUA 经历一个强化学习阶段非常常见。所以这里发生的是，你把你的 cooler 放到一个模拟环境或真实世界中，然后给它一件事去做。它会 做那件事。所以它会尝试买一个蓝色马克笔，最后你会给它分配一个奖励，要么来自基于评分标准的奖励、程序化验证器，或者只是 LLM 作为评判者。你以某种方式给它分配奖励，然后你做强化学习来强化成功的轨迹。 所以这有点有趣，因为这些任务没有人类演示。你只是让模型在沙盒中自由行动，它尝试一堆东西，从它的成功和失败中学习，你重复这个直到它能够成功做到这一点。是的，所以我认为如今人们对 RL 非常兴奋。但其他
[01:01:47.520] 阶段对于确保模型有一个强大的基础来做 RL 也非常重要。是的。所以让我们更详细地谈谈各个阶段。在预训练阶段，正如我提到的，你有这些截图，这个预训练数据的目标只是学习非常基本的能力。所以你可能会学习 接地能力。给定这张截图，通常你的模型被训练来预测下一个 token，对于非常长的描述，本质上归结为预测页面上项目的坐标。所以你可能有这个 car 标签，模型必须预测 xy 坐标或这个边界框的 xy 宽度和高度。所以这有点像是 非常嘈杂的数据，但你可以想象以非常大的数量大规模获取的数据。所以例如，像这个截图数据，它来自 United 网站，你可以很容易地通过只是在网站上运行一个简单的 JavaScript 脚本来生成所有的边界框和接地数据。所以你可以提取所有的边界框。所以你可以收集这种 接地数据非常容易地大规模收集，这就是为什么它非常适合预训练，因为你想要大量嘈杂但数量庞大的数据。人们有时收集的另一种数据类型是动作预测。所以你可以例如以同样的方式从 HTML 页面收集，你可以得到元素名称或页面上某些东西的元素元数据， 你可以把这作为 cool 的输入，它必须再次预测这个的 xy 坐标。所以这更像是语义理解，你可以提供一个 HTML 元素，它必须预测它想做什么或它想点击什么。所以这再次像是非常底层的能力。它真的只是
[01:03:37.039] 教 cooler 如何理解截图以及如何接地页面上的各种项目。是的。所以你基本上学习截图理解。你学习定位。所以有时你学习项目在页面上的位置。你也可以学习项目彼此之间的关系。所以我们知道例如 car 按钮在 hotel 按钮的右边，这可能以 Subway 的方式引入到你的预训练数据中。但这通常是大规模生成的，并混合到前沿模型预训练的早期阶段，只是为了让我们的 cooler 在之后学习接地和学习产生动作时更容易一些。
[01:04:16.559] 在预训练之后，下一个阶段是人们通常称之为后训练。所以再次，我们忽略所有进入后训练的其他数据，专注于这里的 cooler 数据。这通常有两种形式。所以你有像人类数据，你付钱给一堆人类来标注轨迹。所以你可能会给 某人这样的任务，比如找到一个我能在三小时内做的 birya taco 食谱，这里的人类实际上通过在 Web 浏览器上导航来尝试这个任务。所以这里这个例子来自 momo，Momo web mix，他们收集了 36,000 条人类轨迹。所以他们收集了 36,000 个像这样的人们做任务的演示。 上网并做各种事情，然后在每一个点上你都会得到截图观察，并且得到人类所做的操作。所以最终你会得到一堆类似截图、动作、截图、动作的轨迹。他们还生成了一堆合成轨迹，我们稍后会讨论。但一些开源的类似 cooler 数据的例子有 Momo web mix，还有 agent net。这两篇论文基本上都收集了这些任务的人类演示。所以这是一种非常好的方式来训练你的 cooler 去模仿人类行为。所以以截图为条件，训练它去预测序列中的下一个动作。
[01:05:37.440] 当然，如今越来越多地，我们也可以为这部分训练生成一些合成数据。所以 Momo，Momo web 实际上做了这件事，他们给模型提供了网站的无障碍树。所以我觉得这在之前的一些问题中已经提到过了，但其中一些模型在你给它们 元数据时表现会好很多。所以他们有一个被训练来使用无障碍树的模型，他们把无障碍树——本质上是一种简化的 HTML 表示——提供给模型，然后让这个模型在无障碍树空间中进行动作的展开，并收集同样的轨迹集合。所以你有点像是在让这个模型使用一种特权 信息，而它在实际评估和推理时可能没有这种信息。所以它的表现会比只给定截图输入时要好得多。所以你展开这个模型，用这种方式收集了大约 105,000 条合成轨迹，你也可以嗯，并不是所有轨迹都会被保留。所以你可以查看这些轨迹，只保留那些 你认为基于某种指标质量更高的，比如用另一个 LLM 来评判，或者用人工来筛选掉一些。所以轨迹是这样收集的，要么通过人类演示，要么通过合成数据。我非常喜欢的一个类比是关于合成数据版本的，也就是你拥有特权信息的那种，那就是嗯，我其实不知道这是不是真的，但我 听说 Tesla 实际上在训练期间给他们的车部署了 LA。所以他们有这些安装了 LA 的车，它们进行自动驾驶，显然如果你有 3D LA 信息，你会做得好很多。他们收集那些轨迹，然后丢弃 LA 信息，训练他们的模型只使用
[01:07:16.960] 基于像素的信息。所以你有点像是在让一个老师使用比你在推理时拥有的好得多的信息。嗯，我觉得对于很多 cooler 应用来说，人们已经尝试过类似的方式来注入元数据或额外信息，以便收集更好的轨迹。所以我觉得这是一种非常好的引入合成数据的方式。是的。
[01:07:38.400] 后面有问题。尤其是关于某些像素值。>> 是的。所以问题是，如果你做预训练，坐标表示是否能泛化？所以如果你在 Mac 上训练它，它能泛化到平板电脑，或者甚至其他分辨率的电脑上吗？ 对吧？嗯，我觉得通常我们收集坐标数据的方式是将其归一化。所以坐标通常不以绝对格式表示。它们通常以零到一千来表示。零零零表示屏幕左上角，一千一千表示屏幕右下角。所以它们是完全归一化的，根据我的
[01:08:37.040] 经验，即使绝对分辨率不同，它也能很好地迁移。是的。酷。嗯，问题 >> 有没有更高效的方式来做这件事，比如也许 >> 是的。所以问题是，是否有更高效的方式，如果你有 >> 而不是通过截图获取，只是把某些项目在某个数据库中的值存储起来，你可以直接
[01:09:18.799] >> 有一些框框，就像对框框功能的描述 >> 是的，我觉得那也是一种合理的方式。我觉得如今的模型并不是被训练成单一的 cooler。比如如果你尝试使用 GPD6 Astro，它并不总是只做电脑操作。 它会穿插 bash 调用，它会写代码。所以我觉得模型通常试图以最高效的方式解决问题。所以它们在训练时和推理时肯定都会使用那种数据。嗯，是的。哦，好的。好的，酷。嗯，好的。那我讲得快一点。是的，基本上我们现在有一堆
[01:10:04.400] 开源数据可以用来训练模型。所以有很多不同的例子，训练的最后一个阶段实际上是训练后 RL。所以正如我在这个阶段提到的，你有点像是把模型部署在一个模拟环境中，或者如果你觉得嗯，你知道，特别刺激的话，你也可以部署在真实的网络上，但如今人们并不怎么 那样做。所以你通常把模型部署出去，它在模拟环境中采取动作，最后它产生——你从环境或从其他某个评判者那里获得奖励，然后你基于奖励更新策略。所以大致来说，如果你得到正奖励，你就把那条轨迹中发生的动作的权重调高一些；如果你得到 零或负奖励，你就把那些动作的权重调低。所以你的模型本质上是被训练来在环境中最大化奖励，而在这种情况下它是完全自主地做的。所以没有人类演示。模型尝试的任务可能来自人类，但嗯，通常它是以自主方式运作的。
[01:11:05.760] 是的。嗯，希望为什么我们需要模拟环境来做 RL 是显而易见的。我们关心的很多任务，我们并不真的想反复在真实世界中执行。比如，你只能在 Uber 上预订机场接送那么多次，然后你就会被封禁或者钱花光了，对吧？所以你有点需要模拟环境来做
[01:11:23.120] 强化学习。嗯，有几个基准嗯，有点像是处理强化学习构建嗯范式的。所以我觉得我不会详细讲这些，但嗯，你可以在幻灯片上参考它们。嗯，但基本上我想展示这张图，它说明训练后中的强化学习部分实际上对于嗯非常 引入像强大的cooler能力。嗯，所以你在SFT之后一开始的成功率大约是0.5，如果你在最后做强化学习，你会爬升到大约6左右或更多。嗯，所以这是ka训练中非常重要的一部分，而且据经验而言，它还会引入一些仅通过人类演示难以实现的能力。所以这要容易得多
[01:12:09.360] 去训练一个模型在长时程上操作，嗯，使用强化学习，因为你通常没有非常长的任务的人类演示。收集这些演示非常昂贵。是的。嗯，所以总结一下，是的，我们讲了cooler是如何训练的。嗯，你通常有几个训练阶段，比如预训练，嗯，后训练，在后训练中你在 人类演示和合成数据上做行为克隆，然后你还有最后的点睛之笔，就是你做模型的强化学习阶段，嗯，在模拟中运行它并在环境中获得奖励，希望到最后你能得到一个强大的cooler。嗯，但在高层次上，这就是如今大多数实验室用来训练cooler模型的流水线。
[01:12:56.400] 所以总结一下，嗯，ka与其他智能体有些不同。我认为它们嗯有点像是非常好的，因为它们直接在与人类相同的界面上操作。嗯，评估cooler非常棘手，人们已经做这件事有一段时间了。嗯，但本质上我们想针对正确的指标和行为水平。
[01:13:14.800] 嗯，cooler模型被设计成从交错的视觉历史中预测动作。所以你有点有这种交错的上下文，包含截图和你之前采取的动作，模型必须以此为条件来产生下一个动作。嗯，如今的模型是分多个阶段训练的。所以你有用大规模噪声数据进行的预训练。你有SFT，你在其中尝试
[01:13:35.280] 行为克隆人类或合成演示，然后你有一个最终的RL阶段，你嗯让模型嗯在模拟环境中自行展开，你奖励它的好行为并惩罚它的坏行为。嗯，本质上这嗯可能是我今天有时间覆盖的所有的总结了，但我鼓励你去看看幻灯片，了解一些嗯剩余的见解
[01:13:59.520] 关于我们还能为Kers做什么工作。嗯，本质上我认为速度仍然是个问题。个性化是一个非常成熟的研究领域。嗯，基础设施和UX是一个很大的关注点。而且，嗯，我也对多智能体系统非常兴奋。是的。嗯，感谢大家的聆听，很高兴在剩余时间里回答问题，或者是的。上帝保佑。
CMU 11-768 Fall 2026 · Lecture 7 · Computer Use Agents
- 视频：官方视频
- 字幕：en-orig · auto captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
正文
[00:00.560] Okay, great. So, yeah, today I'm super excited to have JY Code as our guest lecturer. So, um JY is an expert on um on multimodal machine learning um in particular computer use agents. Uh so, JY um has experience both in academia here as a PhD student CMU uh working with uh me and Russ Solakutnav but um also has a lot of industry experience. Uh so he worked at Google research before starting the PhD and um also he led the computer use agent team at Meta's TBD lab for a year and a half. >> Yeah. And so kind of like laid the foundation for um the strong agents that they're releasing now. And yeah, he also um built uh some of the most uh impactful benchmarks in this area and also did work on um search for agents.
[00:58.640] So he'll also be giving a lecture later on in the course and we've covered some of his work already on multi-agent computer use and looking forward to uh you know hearing about the foundations of KUAS today. Thanks. >> Cool. Uh thanks a lot Daniel for the kind introduction. Um so today I'm very happy to talk about something that I've uh personally been very excited about for for a couple years now. Um I think today it's like taken for granted that we have uh that computer use is a is a core capability in like many frontier models. But uh as we will see by uh kind of stepping a bit through the history of computer use agents, this was not always the case. Uh I think for a long time like um it was not very clear whether we would want computer use agents or how we would build strong computer use agents. Um so I think computer use agents or kas are a little bit different from the agents that you've already covered in this class. Um so I think they're a bit different I think of them as a little bit different from like sui agents and more textbased agents. Um so first of
[02:00.880] all what is a cooler? Um, a coupa is a model is an agent that observes a sorry, it's actually not. Um, okay, there we go. Here you >> uh I'm I'm think I think it's fine. Yeah, sorry. So, AUA is a computer use agent. Um, it's a little bit different from the agents that you've already seen cuz what they take as input is actually the graphical user interface that you have in most applications. Um, so you might imagine this could be a web browsing agent. It could be an agent that operates on your desktop computer or it could be something that operates on a mobile device such as your phone. Um, so all of these are broadly speaking computer use agents. Um, agents that operate through the same graphical user interface that humans use. Um, so they take in like screenshots as inputs and they execute actions uh of the same type that humans do. So clicking actions, scrolling actions, swiping, tapping and and so on. Um so the key difference here is that they operate in the exact same space as humans which I think is a very uh it allows for very interesting problems and challenges to be solved in the modeling and evaluation space.
[03:12.000] So broadly speaking uh kers operate in the following fashion. You have a user task um that is provided to your cooler. You also have some environment observations and typically this is a screenshot uh of whatever you have on your current computer screen or mobile device. Um this can also be some other uh like textbased representations but broadly speaking most frontier course uh use like screenshot representations. So this is essentially exactly what the user would see if they were to attempt the same task. Both of these inputs go into your cool which then produces a reasoning uh trace typically. So it's like a chain of thought and it produces an action that it wants to execute next. And this action can be uh any any action that you allow in your um in your environment. So it can be clicking, scrolling, typing and so on. We then process this action and we execute it in in some infrastructure that you have uh in your environment. Um so if it's a click action, you actually execute it on the computer and you uh get some new observation after executing this action.
[04:15.760] To give you a concrete example, um you might have a task like buy me a bloom map and you might have some initial environment observation for example from your web browser. So these would go into input into your computer use agent model and your model might produce some reasoning trace of the form. I see a blue monk on the center of the screen. I will click it and it will then produce uh some like formatted action to be executed. So in this case it wants to do a left click on the on the environment. um we will execute this action on the environment. So we will actually like click somewhere on the environment and we will get a new observation that happens after we execute this action. So for example over here you get a new screenshot. Um in this case the agent has clicked into the blue mark. So you kind of have a new observation that shows the details of the blue mark. Your model then takes in this new observation and it produces a new action to be executed condition on this updated observation. So it will it might then say like delivery details are below. I will scroll down to find them and it will produce this scroll action that we will again like execute on the infrastructure. Um you then get a new observation and you basically repeat this until um your model decides it's done. Right? So this can go on like for many many iterations depending on how complex your task is. Uh so yeah you start with an observation your model executes an action you get a new observation um and this repeats until the end and um typically you get some reward either from the environment a human user or some other judge telling you whether the model did it successfully or not. Um so typically like this reward is just a 01 uh binary reward but it can also be um a partial reward in some cases.
[05:55.840] Um so this is broadly speaking how a computer use agent works. It's this like observe, reason, act loop that that repeats until uh your model exhausts its budget or it thinks it has successfully completed the task. Um and today it seems like very simple uh and models are actually very capable as as we'll see in a couple slides. Um but this was not
[06:15.360] always the case. So um maybe we can have a quick poll. So when do people think like computer use agents were first uh like invented? Um do people think it was like before 2020? Okay, a few hands. Um say like 2020 to 2024 period is when we started building coolers. Cool. And who thinks it was like after 2024?
[06:40.319] Um so the the answer is actually that um the first cooler paper came out in in 2017. So this was like uh quite ancient in in um AI standards, right? Uh it was actually a paper from OpenAI. So they had this project called um World of Bits where they built this very simplified toy environment. Um as you can see here, it's mostly like synthetic data. It's very simplified environments that look maybe a bit like what you might see on a mobile device, but it's very simplified. Uh not not realistic at all. Um and what they did in this paper is that they actually did like a bunch of RL training. Um so this is not really similar to like modern LLM RL but it's more like similar to RL that we did for um game agents in in the past. Um but
[07:25.520] this was I think this project was like ahead of its time. All of the models back then could not do this at all. Like even this simplified uh mini world of bits task they were very bad at it. Um so this was the first cooler paper probably. Um in 2022 I think a bit more interest started picking up with the advent of uh language models. Um there was this other paper called web shop that um also did uh kind of like simplified shopping environments but it was a lot more realistic than than mini world of bits as you can see in this visualization here. So it's kind of clicking through this like simulated Amazon site. Um and it's still a bit artificial but it's a lot better than mini world of bits. Um and at this point like language model agents started doing
[08:08.080] a bit better at these tasks. uh and we weren't really training them in uh in in the way that we were for many world bits in 2022. In 2024 actually um there was this I think of it as a seminal work uh from actually Graham's lab at CMU called web arena. So it was this work that created a lot of simulated environments. Um and as you can see like all of these environments are a lot more realistic than web shop and mini world of bits. um they contain like very realistic data often scripted from real world websites. So these environments were populated with actual data from like Amazon and Reddit and and so on. um so these are like very realistic data and environments. Um and I think they marked a point where uh the models also started getting a bit better at these problems. In 2024 uh there was also this almost concurrent release of this benchmark called OS world. So I think this is one of the most uh standard computer use benchmarks these days. And this was also using uh real apps and kind of like real data but um uh it tackled like desktop
[09:12.399] use rather than web browsing. So it's a bit more general. You can use a lot of different apps on Linux, Windows uh and so on. Um so this is a bit different from web arena because web arena was web browsing but OS world is uh like fullon desktop use. Um I think things also took a bit of a change like in 2025 to 2026. So this was the point where like coding agents got a lot better and at this point like a lot of people started generating environments uh end to end. Um so for web arena and OS world what they did was they actually took open source implementations of these apps and they filled it with realistic data but all of the all all of the software was already written by humans and open sourced in some manner. um my PC bench and a few other uh benchmarks around the 2025 to 2026 period actually use coding agents to generate uh the entire app and website um end to end. So shown here is actually a Gmail clone. It's not the real Gmail, but it looks very close to it. And this is actually generated completely by an LLM a coding agent. Um all of the data that is seated here is also completely synthetic. So, we've actually gotten to a point where a lot of like frontier cooler environments are completely generated and completely filled with synthetic data and um I think they actually work very well. Uh sometimes they're even better than real applications because um they're a lot cleaner and they're not as as noisy.
[10:37.200] Ka world was also another project that came from CMU that took like a real world software. So, I think Kool World is very interesting because the software used here is a bit more of the long tail. So you have some software used for like astronomy and biology and other things of that sort. Um so it's a bit it it departs a bit from like just consumerbased apps into very specialized
[10:57.519] desktop software. Um all of the data here is also uh I think is realistic data. So it's downloaded data from real world use cases um and is loaded onto onto onto these environments. Uh shown here is a demo very recently from this month of GPT6 Astra. So GBT6 Astra was a huge leap in computer use. I think it basically like solved most of the computer use benchmarks. Uh it's like better than human baselines at this point. Um it's kind of a lot faster than previous models, but it's still very slow and very expensive. So this demo is showing GPT6 Astra operating like a CAD software. So it's doing like some like 3D modeling. Um it's actually sped up 30 times. So in in practice like this model is actually 30 times slower than whatever you see here. Um but but the point is it's very capable but it's still very expensive and and slow. Um so we still have some research problems left but in terms of capabilities I think current models are are very good. Um so you can see like we progress from toy environments with miniw walk uh to today 9 years later where we have GPT6 extra that is uh comparable or better
[12:04.399] than your median human at a lot of computer tasks. Um so very exciting very fast progress and we can think about like how we might measure these problems. Um if you think about them from a research or academic standpoint uh benchmarks are one of the core ways in which we measure how fast these agents have been progressing. Um so we'll go through an overview of like several computer use benchmarks and how they measure different axes of uh agent performance. Um so if we think about how computer use might be measured uh it's actually a bit I think it's a bit more interesting um than some of the other agents that we've already covered in this course. Um so computer use is in some sense a bit less of a verifiable task compared to like software engineering or math. Um in software engineering or math you kind of have unit tests or or um symbolic verifiers that tell you whether you successfully accomplish a task. for computer use is actually a bit different because you can imagine that um successfully accomplishing a problem is not always um people do not even people do not generally agree on it. So it can
[13:07.040] be challenging to write like programmatic verifiers. Um so we have several ways of evaluating how computer use agents do on some of these benchmarks. Um the most obvious one is whether the action was correct. So you can measure whether a click action was uh done on the correct item. Um you can also measure the outcome which is probably the most endto-end fashion. So you give it a task like buy me a blue mark and you measure the outcome of whether it actually managed to um purchase the item or not. So this can be done through either like um checking the state if you have a simulated environment or you can just look at the trajectory of the agent and decide whether it um provided enough evidence that it did the task. Um, and lastly, I think you also want to measure whether the model respected any constraints that might be inherent or explicit in the task. So, for example, you might want it to navigate to the checkout page but not actually check out. Um, so you also have some other like uh verifiers to make sure that it did the task while respecting any userdefined constraints. So, I think
[14:08.480] evaluation is generally very challenging for Kas. Um, there's a few broad categories of evals that we'll go through today. Um, so broadly speaking, I've categorized them into four different categories. So the first one that we'll talk about is static evaluations.
[14:23.120] Um, so let's go back to our task. Add a blue mark under $20 to the cart and do not purchase. Static evaluations measure whether every action along the the trajectory of the agent is correct or not. So for example, you might start off on this screenshot and your agent would, you know, try to attempt to click on this blue mark here. um we might have some ground truth reference rule like we want XY that the agent predicts to be in a target box. If it's if the prediction is within a target box for example this red box here we give it a score of one. Um otherwise we kind of give it a score of zero meaning that it did not perform the correct next action at this point. Um so in this case like for static evals uh we consider it to be correct if the current step produces the correct action or not. Um so this this because it's static you can't really like roll out the agent end to end. Uh we just consider actions at every point um to determine whether it successfully accomplished the current action.
[15:21.839] One canonical example for benchmarks in this area is known as screen spot pro. Um so screen spot pro is what we call a grounding benchmark. Um it tests the abilities of the models to ground actions in the screenshots and it consists of a variety of applications operating systems and about 1,500 different screenshot examples. The evaluation in this case is the perpoint uh accuracy. So for example this task of constructing a UV spear mesh um you have this like uh Blender screenshot here. So this is a screenshot. It's not an actual environment. So the agent does not have access to any interactive environment. is just given this screenshot and the goal is to determine whether it produces an action that falls within this red
[16:04.639] bounding box over here. So if it if it predicts a click action that falls within this uh bounding box, we consider it to be a correct action. Otherwise, we consider it to be an incorrect action and and we don't give it any any reward. Another example of a static benchmark is mine to web um which tests a bit more of a variety of actions. So um you can think of this as like next action imitation in this case given a current screenshot and a task. Um for example rent a car in Brooklyn central um I'm supposed to predict the next action that my agent will take. Uh so in this case the ground truth action is to click on the car tab and um no actual action is executed on this task. You're given a series of screenshots and your agent has to predict the next action condition on that screenshot. So once it predicts the action we score it and we provide it the next screenshot from our offline data set. So again, this is not actually executing any actions in the environment. There's no environment in this case. It's a series of static screenshots and ground truth actions.
[17:02.800] Um, so as you can imagine, this kind of evaluation is very fast. It's very cheap to run because you don't actually have to support an environment or simulation. You just have to provide it screenshots and measure whether the resulting predicted actions are within uh the ground truth set. Um but one problem as you can imagine with static evals um is that there's a lot of uh false negatives uh in the sense that for this screenshot and for this task there's a bunch of different actions that you can click that you can perform that would be considered correct uh like for example I don't really have to click the car tab in this case I could adjust the dates um to begin with and that would also be considered a valid action but in static evals because you have a set of ground troop actions um we we actually would mark those as as incorrect. Um so that's one of the main drawbacks of static evals. They are very fast but they don't really measure the thing that you care about uh which is the true end toend performance. So I think what we actually care about is whether the agent successfully did the
[18:02.400] thing at the end of the day. Um which brings us to our second type of evaluations which are end to end evaluations. Um so again like let's let's look at this example task like add a blue monk under under $20. um we might have an agent execute this task in an environment or on the real web and at the end we can uh check whether the agent did it correctly by using some uh programmatic unit test like checks. So for example um an agent might have attempted attempted this task and at the end of the trajectory we can uh run these like unit test like checks to see whether it actually did it. So this is done in a simulated environment. So we actually have like API access to all of the back end and state of the environment. Um so we can check whether the blue mark is in the cart uh whether the blue mark is less than $20 and whether the agent has not um actually place the order. So we check whether the order is the order count is zero and we only give it a reward of one if all of these individual unit tests pass. Um so this this is actually like a programmatic check. You actually have an environment in this case. So the agent is doing like whatever it needs to do. Um so we don't we don't check for the individual actions that the agent executes at every step but we do a check for uh whether it actually accomplish the task at the end based on these unit tests. So if your unit tests are comprehensive enough you could imagine that this is a very good uh reward signal um because we don't really
[19:26.720] supervise every single action of the of the model. Uh we just kind of like check whether it did it correctly or not at the end. So web arena was one of the first benchmarks to do this. uh web arena was a simulated um web navigation benchmark consisting of a few environments. Um so you have 800 about 800 tasks on four different uh apps that web arena uh hosts and the agent will execute all of these 800 tasks and at the end you kind of have a pro programmatic verifier as we discussed to check whether the agent did it correctly or not. So this is an example task. Um the goal is to create an empty repository called awesome LLM reading. So the agent will go onto this simulated gitlab uh repo. So importantly this is a simulated environment. It's something that you host locally. Uh you do not go on the real web at any point in time for for web arena. Um the agent has to try and accomplish this goal within the simulated environment and at the end of the task we run this programmatic verifier which basically checks for a bunch of conditions. Uh so you you essentially check whether it managed to create this repository or not. Um and all of these verifiers are handwritten by students uh by grad students at CMU actually. Um so they uh they hand wrote these verifiers for every single task in web arena and um at the end of the assuming you know the students did it correctly uh at the end of the execution um you would have a verifier that tells you whether the agent successfully did it or not. Um there there's still a few like uh false negatives and false positives because the verifiers are not comprehensive enough but on the whole it's a very uh I think it's a very nice benchmark. It definitely like led the way to people taking and evaluating coolers a bit more seriously.
[21:13.360] Um so I think it's important to contrast this with previous static evals because this actually measures the end to end performance. So you kind of just check for the outcome. you don't care about the individual steps as long as the agent did the task correctly. Um but you could also imagine that these this leads to some reward hacking. So the agent might uh you know try to accomplish the verifiers rather than actually doing the thing which is a common problem across most cool and and agent um evals and training generally. Um but I think this was a huge step up step up in in agent evals. Um so they made sure that you did it in simulated sandboxes and they uh measured like the outcome rather than the individual steps.
[21:55.440] Uh visual web arena was a an extension of web arena that kind of like treated it as a multimodal problem instead. Um so at this point when web arena was released most models were um handling the cooler problem or web browsing problem using accessibility trees which are kind of a simplified version of HTML and visual web arena actually uh tried to convince people that you kind of want to operate using screenshot based approaches. So visual web arena is an extension of web arena that consists of 910 uh visually grounded tasks. So all of the problems in visual web arena have to be solved using multimodal representations. Um and the evaluation is a bit similar. So you kind of have like programmatic checks as well. Um but you also have some multimodal based uh
[22:36.720] programmatic checks. So um there's some problems that you can't solve if you rely purely on HTML. Another form of evaluation for these benchmarks is instead of using programmatic verifiers, you kind of use an a different LLM to judge whether the model did the task correctly. So given a trajectory of a model, you prompt a different VLM or LLM. Um so a textbased LLM or multimodal uh vision language model uh you prompt it with the trajectory of the model and you ask it whether it completed the task successfully and this is also a very popular evaluation metric um because it's not always possible to write u verifiers for every single task that you care about. Um so in this case you again have the trajectory that the agent um completed and instead of a programmatic verifier you have a bunch of rubrics or or maybe you just even provide the task to the to the LLM. Um so you provide the trajectory of the agent and you provide the task and the the LLM kind of tells you whether it thinks the agent completed the task successfully or not.
[23:39.120] Um so typically like these judges they produce a short chain of thought followed by a judgment for example pass or fail or sometimes uh you assign like um floating point scores but typically the judge tells you whether you pass or fail or not. Um so this is a bit more general because you don't need to write verifiers for every single task in your benchmark. Um but it's also a bit more prone to like false uh false positives and negatives because um you can imagine that your LLM judge can be easily fooled into thinking that a task was successfully accomplished when it was not. Um but in general this is a lot more uh flexible and um several benchmarks that use this form of judging include web voyager which was one of the first benchmarks
[24:21.039] that tested the abilities of coolers to navigate on the real web. Um so this is very different from web arena because um what they did is they actually came up with 15 different websites and 600 tasks on these websites and they told the uh the benchmark is to actually go on the real web on these websites to accomplish certain goals. Um so actually if you go on some of these websites in the benchmark like there's this recipe site called all recipes. If you look at some of the task that the agent is supposed to do you can actually see some traces on the real web of previous agents attempting this task. uh which is actually it's not great but you can um see some evidence of why we might not want to evaluate agents on the real web.
[24:58.960] Um but nevertheless like web voyager was still quite valuable because it tested the capabilities of these agents on non simulated environments. So it's basically as close to a real world evaluation as you can get. Um it gets agents to go and do these things on the real web. Uh for example like find the most start GitHub project for climate change data visualization. um they they uh deploy like who was to go on the real GitHub and do a bunch of search and uh find the actual most start GitHub repo for this task. Um and in this case because it's going on the real web you can't really have programmatic verifiers because you can't really uh query the real web for its state. Um so the eval here is the VLM judge that we talked about. So the judge which is a different LLM gets the uh series of actions and screenshots that the agent has has taken and observed. Um it gets the original task. So it sees this task. It sees what the agent has done and it kind of predicts whether the agent succeeded or not. So we we evaluate this problem with like VLM judges. Um no programmatic verifiers. Um so this is kind of like you can think of it as quite uh wild
[26:07.600] wild west evals. you deploy something on the real web, you run a judge and you um you know at the end of the day say whether it's succeeded or not. But I think a lot of people like this because it's in some ways as close to a real world eval as you can get.
[26:22.159] Online mind to web was a was a different eval that followed up on web voyager. Um so it's a bit more comprehensive. The judge is a bit more stronger. So they show that the judge actually has about 85% agreement with humans. Um so it's very close to uh being uh human level agreement for for judges. Um but essentially uh online Mind2 web also looks at real world evals. Um so they expanded to about 136 websites and 300 tasks and they get LLMs to go on these 136 real websites to accomplish certain tasks. Um and they they show that uh it actually um using this judge is a bit more accurate compared to Web Voyagers judge. Um so they they have a bit more of a complicated judge with different rubrics and and criteria. Um but essentially this also goes on the real web and it inherits it inherits a lot of problems that going on the real web um poses similar to web voyager. Um but again it's a very nice eval because it's [snorts] in some ways as close to a real world eval as you can get. Um so this is completely automated but you have an LLM judging the task. You have an LLM attempting the task on the real world.
[27:31.919] Um and because uh the judge is very close to human eval kind of have this automated pipeline of an LLM going on the real world doing something and you have a human-like judge determining whether it succeeded or not. So I think it's a very nice benchmark um very automated uh but again I think people don't really use this uh recently anymore because it's um essentially solved by modern models and I think as cool words have gotten more popular um people have also become a bit more careful about deploying them on the real world um because these website owners really don't like having like tons of agents ping ping their websites to to do a benchmark um yeah you can imagine that if your website got included in this benchmark and you
[28:14.159] started getting a lot more traffic, uh you probably also wouldn't be very pleased about it. Yeah. Um so one I think one important thing is that all of these evaluation metrics that we talked about are very um they are programmatic verifiers or they are LLM judges. Um so they are kind of like proxies uh to determine whether we successfully accomplish a task or not.
[28:37.200] Um but human review meaning uh getting humans to judge whether the agent did something or not is probably still the gold standard. So uh these papers also ran like a human evaluation baseline where they got humans to review the trajectories of agents and um you get a few humans per trajectory to determine whether the agent succeeded or not. Uh and at the end you uh kind of like get humans to vote whether the agent succeeded or not. Um, this is in some ways like still the gold standard of determining whether an agent successfully accomplished a task, but this is very expensive to run. So, it's not really possible to do this for every single task in your benchmark or every single um uh like debug or validation run that you want to do. Um, but this is something that most benchmark creators still do. Uh, get humans to attempt the task or judge the task and and determine whether the agent is doing the right thing. Um, but this is very expensive. So it's typically just done once or twice for ensuring that your agents or your benchmark is uh reasonable.
[29:44.320] So the other type of evaluation that people typically do is um they're also end to end evaluations but they're done on like desktop computers or mobile devices instead of of the web. Um so the web is quite interesting because you can do a lot of things on the web. Uh I think personally most of my time is spent on Chrome as well. Um but there's a different set of tasks where uh
[30:04.240] require you to operate an actual virtual machine or an actual desktop and um there's also several benchmarks that measure this in an end toend fashion. Uh we've already talked about OS world which is one of the canonical cua benchmarks. Um so this is a real Linux virtual machine that you can spin up. It consists of about 369 tasks across nine different applications like um uh Libra Office, uh Chrome and and so on. Um it's 369 tasks across nine different applications. And for each of these tasks, they also have programmatic verifiers. So your agent has to um go into these virtual machines, attempt the task across multiple different apps, and at the end, we run some uh unit test like checks to see if it succeeded or not.
[30:51.600] So some examples of task in OS world are for example uh you might want to update uh this Libra office uh calcul sheet um using receipts in a different folder. So in this case your agent has to navigate to the folder find the receipts it has to enter like the appropriate uh values in this um workbook and it has to save the workbook. So it has to do this uh navigate across different apps um handle like entering items in in this uh Libra office and uh basically like accomplish this task based off just an initial user instruction. After the agent thinks it's done, we run all of these like unit tests that we have. So we check whether the the spreadsheet consists of the items that we think uh should be there and we assign it again a 01 score of whether we think it succeeded or not. So kind of like very similar in spirit to web arena but um in this case it's using uh real world apps. So these are apps that people actually use on a day-to-day basis and it's operating again in a full virtual machine. So it's not just confined to to Chrome or to the web anymore. It's uh operating over this
[31:56.960] virtual machine that consists of nine different apps. Um so it's a lot more broad and you could imagine that um with desktop agents we can automate a wider variety of tasks than if we just use web browsing agents. Uh Windows agent arena was a different benchmark that is kind of like inspired by OS World. Um but it's a benchmark created by Microsoft to uh test the performance of agents on Windows machines. Um so a lot of a lot of like productive work in the world actually happens on Windows machines and um I think this benchmark was quite nice because it measured the ability of agents to perform some of these tasks that are very common in enterprise workflows or in um other companies. Yeah. So it's uh again like you spin up a Windows virtual machine, you get an agent to attempt a task and at the end you have some uh verifiers to determine whether it did the task correctly or not. And again like because we're using programmatic verifiers um some of these are also very prone to having false positives or false negatives but um you know they have their merits uh as LLM
[33:00.559] judges do um but these these are essentially some of the core end to end eval benchmarks that that um people have been exploring um from the 2024 to 2025 period. Yeah. Um most of these benchmarks are actually quite well solved by modern coolers. So for example the recent GPT6 release I think achieves like much higher than human performance on most of these benchmarks. So um the
[33:27.519] community has actually um recently [clears throat] um put a bit less attention on these benchmarks that I just mentioned um and they've started to look at a bit more like longer horizon or um uh specialized benchmarks I would say. So long horizon computer use is this other it it's actually the same task but the problems that we care about span uh much longer horizons. So some of these tasks can take actual humans hours or even like two to three hours or even longer to complete. Um so now we're really like stress testing the abilities of kas to complete um things that humans don't actually want to do or we're getting them to complete like longer and longer tasks. A lot of the ones that we've seen in OS world the web arena uh typically take a human about uh 10 minutes or 20 minutes or less to complete. Um, but some of the ones that we'll see in the following slides can be several hours operating over potentially specialized software. So, we're really getting to the point where we're trying to get them to be better than like expert human use of the computer.
[34:29.599] Um, a lot of these long horizon benchmarks, they follow a very similar evaluation strategy. So, you kind of have this action trace which represents what actions the LLM has taken uh on this environment. Um and you have a few different checks because it's long horizon. Um people tend to impose like more rubric based approaches. So the the version where we have a single outcome reward of whether it successfully did it or not is not so useful for long horizon because it's very sparse. Um the rewards are very sparse. Um the agent can be doing a lot of useful work in the interim but fail to achieve a perfect score. Um so we kind of have like a lot more fine grain rewards for long horizon computer use. Um there's a few different checks here. So state check is the interim rewards that I mentioned. Um these are kind of like rubric based rewards where they check whether the agent did um an intermediate task successfully and you kind of have a process reward where um this essentially checks whether the agent completed the entire task successfully end to end. Um so this is kind of like the ultimate reward but we have some intermediate rewards to ensure that we have some non-zero signal for long horizon computer use. Um and of course you can always have a final verdict for whether the agent successfully did it or not. Um but again because we're looking at long horizon computer use uh this is very sparse. So often times like a lot of models get uh a zero on the final reward which is why you kind of want to have intermediate rubrics or um uh process rewards. Yeah. And and importantly um as I mentioned like having a a good deliverable at the end is not always a great evaluation metric for long horizon computer use because it's so long. Um so you kind of want to have some interim uh rewards to ensure that the model was doing productive things in the meantime.
[36:18.720] Uh Odyssey's was a benchmark that we built here at CMU that measures long horizon uh web browsing. So Odyssey's consists of 200 tasks. Um we have uh rubrics for every task. So, we have a set of um things that we believe the model should complete. Uh so, for example, for this task, um plan a trip to a Palm Springs wedding. Um there's eight rubrics that the model needs to satisfy in order to perfectly accomplish the task. Uh for example, you need to find a a non-stop flight from Pittsburgh to to Los Angeles. Um you need to uh find a drive time within the 9:00 a.m. to 4 p.m. window. and you have a bunch of these criterias that you need to satisfy in order to accomplish the task successfully. Um so Odyssey's kind of like grades performance across many different rubrics um allowing you to have partial rewards instead of just a single binary pass or fail um outcome at the end. Yeah. So um this example I already mentioned we kind of have like eight rubric items um and the model needs to satisfy like each of the rubric items in order to achieve a perfect score idea.
[37:23.119] Um, so this is kind of an in between of uh like it it's still LLM uh verified for the most part, but you kind of have like fine grain LLM verification because you don't just um produce a single pass or fail uh reward at the end. Um OS World also came up with a follow-up called OS World 2 and in this benchmark they actually tried to tackle like hourlong workflows. So OS World 1.0 know was a lot of tasks that require just like minutes of of human time to complete and OS World 2 actually has about 108 tasks that that on on median require a human 1.6 hours to complete. Um so this is very uh the some of these workflows are very complex. Um they require like doing uh CAD modeling or 3D modeling or photo editing. Um so we're getting into the realm of like more specialized um software use. Um uh on average most Frontier agents that they benchmark required about 300 calls to make. Uh so this also um typically it's it's a few hours of like just benchmarking an agent. Uh so I think this gets to a point where it's a lot more interesting because you can see these agents uh even like productively working for hours at a time. Um, so it's
[38:35.920] very expensive to evaluate, but it's quite magical to like start in Eval, come back an hour later and you see that the agent is still productively making progress on some of these tasks. This is an example task uh from OS World 2 and um I don't know if I don't know about the median human, but I definitely uh would not be able to do this uh easily. So I think we're getting to the point where you have like expert level task. Um, so the task here is to build a a cat support bracket and export a model that matches this drawing. So you're provided with this reference drawing on the left and the goal of the model is to be able to produce this thing on the right uh by operating some like um computer aided design uh software. Um so I think this is this is actually very complex. Uh it requires like navigating 3D space. It requires knowing like what tools you have available in this environment and you need to be able to sustain work over like probably several hours because um a human would also take uh on the order of an hour or at least like half an hour to complete this task.
[39:33.920] Um so it has a lot of very interesting things. Um this is a visualization of a GPT6 a GPT6 model trying to attempt one of the OS world two tasks. So again like this is sped up 30 times. So it's actually much slower in practice, but you can actually see like navigating between a terminal environment, going back to the CAT software and um actually like doing 3D modeling uh autonomously for for about I think an hour of time. Um so it's actually able to design like these 3D parts quite well. Uh and this is a a very huge leap from the models that we had even like two years ago. Um, so we're getting to the point where these models are are very good at attempting many of these like long horizon paths. Yeah.
[40:19.280] Um, so I'll mention a few other benchmarks. So cool world long is another benchmark from CMU uh that tried to tackle like long horizon computer use. Um, so again this is uh very specialized environments across a variety of tasks for example healthcare task, engineering tasks or like uh just architecture and design task. um consists of 200 of these types of tasks and most of them require like 500 or more LLM calls. Um so again this is a few hours of human work time. Um and when this benchmark was released early this year most models scored a very very poorly on this benchmark but um I think recent models including GPD6 uh actually um achieved like close to 90% performance on this benchmark. So, uh, long horizon computer use and like
[41:04.240] computer use in general is very, uh, very close to being solved or some might say even solved because these models can outperform humans at a lot of these, uh, general software use tasks. Um, so hopefully I've convinced you that uh, some of these benchmarks are useful uh, or or most of them were useful at some point in time. Um and uh I think I think hopefully I've convinced you that they are very capable and have very interesting um uh capabilities. Uh they are very good nowadays at automating a lot of different types of tasks. Um so I think the natural next question is like how do we how are these models built? How do we build these models? How do we train them? And how do we introduce these type of capabilities to them? Um, so we'll talk a bit about like how how
[41:50.319] modeling typically works and also how uh these models are trained. But I think I'll pause for a few seconds for any questions about benchmarks because I know we covered a lot of different things. >> Yeah. So object [clears throat] for something that has a twoar.
[42:23.040] >> Yeah. So the question is whether any of these benchmarks assign um partial rubric rewards to intermediate steps. So for example, if the task is to add something highly rated to the card and you did do that but you have some like irrelevant items that you added, do you get penalized for that? Um I think that's a good question. Uh I think reward design is very tricky in a lot of these benchmarks. So um some tasks in web arena do have these kind of like penalization uh but a lot of the times like people design these benchmarks so that the answer is kind of unambiguous. So the problems are a bit artificial in a sense. So for example web arena might say add the top rated item to my cart and if you do that then um you kind of get like a perfect score but uh an agent I I've seen like agents reward hack by adding a ton of things to the cart um to try and accomplish the goal. So it depends a lot on the benchmarks. I think typically people don't really penalize it at such a fine grain manner but it's also because at the point where these benchmarks were created the agents were so bad that I think some of the reward scores were a bit more generous to begin with. Um so I think it's some benchmarks do have these things but it's not I wouldn't say it's like super common. Yeah, I think people tend to air on the side of like giving false positives than false negatives.
[43:42.560] >> Yes, >> this is not exactly like a benchmark question, but do you think there is like benefit in looking at other not? >> Yeah. So the question is whether there's benefits in using other modalities other than just pixelbased representations like screenshots. I think there definitely is. Um open AAI has said that um at least previous models and I assume Astra as well they use like the accessibility representations of the computer when available. So Mac OS has this like uh accessibility representation that essentially is for like visually impaired users. um it's kind of like XML like representations where they determine where they tell you like this area is a description of a certain item like this area is my chrome window and this area is my dock and so on. Um so the models at least GPD6 do use these representations when available. I think there is a lot of value in it um especially for uh if you want your model to be more robust. Um, I'm personally I personally really like the pure like screenshot based representations because it's very I think it's very general. So you train a model that takes in just screenshot representations. You can deploy it on any device without having access to the underlying internals of the of the device. So you can train a model on like Windows and it presumably gen generalizes a bit more easily to like a Linux or Mac computer um because they have different like underlying accessibility representations. for example. Um but yeah, I think models
[45:16.160] definitely use alternative representations. Yeah. Yes. >> Yeah. My question was around the rubrics itself. >> Yeah. >> I'm assuming rubrics are different for each dots. So are they one is that expert generated rubrics or is that like a synthetic rubric that you generate and then in terms of reward are they how what are ways to mitigate that from the rubrics?
[45:38.720] >> Yeah. So the question is um whether rubrics are generated by humans or I guess synthetically generated by LLMs and um whether reward hacking happens with rubrics and how you mitigate that. Um so at least for Odysseies most of the rubrics are LLM generated but human verified. Um so one common way of like generating these rubrics is actually uh related to the previous question where you give a model like privilege information. So you give it more information than it has at evaluation time. uh for example like some hints or some uh HTML representation and you kind of get it to do better than it would in a realistic eval scenario and based on that you can uh generate like more accurate rubrics um because you have like privilege information. Um so for Odysseies and I think for a lot of uh industry benchmarks rubrics are typically these days like LLM generated by by human verified. Um sometimes not even human verified but verified by a different LLM uh that that has like potentially different information. But um yeah, I think I think this is quite reasonable these days to do like synthetic generation, but I personally still feel that a human has to uh look at everything that you put into a benchmark at some point in order to make sure that that it's um uh legitimate. Um I think reward hacking definitely happens a lot for um training these models. Um yeah, I think we can talk a bit more about that in the training section, but essentially I think reward hacking is a hu huge problem. I would say that rubric based rewards actually mitigate reward hacking a bit better than outcome based rewards. Um because you kind of have like these different criteria that you need to satisfy. Um if for example like if you just want to buy like the top rated item then you can reward hack by adding every single thing on the site to your cart for example.
[47:21.440] Whereas you could imagine that rubric based rewards uh check for this. Yeah. >> Yes. >> Do you have any sense about whether these academic benchmarks are used for evaluation or also do people train on them whether academia or industry? >> Yeah. So the question is whether um people train on these benchmarks in academia or industry. Uh well you're not supposed to train on these benchmarks because you're kind of like uh testing on the thing you're training on. But I think uh it's kind of like common in industry to generate tasks that are very similar to your benchmarks. Um so it's very common for people to uh produce tasks that are like OS world like or web arena like and train on those things instead. Um so that it's like in distribution but you're not really training on the same test set. So I think that's very common. Um cool world also actually has an explicit training set. Um so for some benchmarks they have like train validation and and ideally test splits um that people can train on but um you're not supposed to train on the the actual eval but it's uh I think people do find like proxy signals for essentially doing that. Yeah
[48:30.000] unfortunately cool yeah maybe the last question and we can move on. Yeah >> I was curious what representation What is the action space? >> Yeah. Um, >> yeah. So, the question is what the action space is actually for KUS. Um, I I think we'll cover this very quickly.
[48:54.079] So, maybe we can just uh do that in a couple slides. Yeah. But essentially, um, it depends on a model by model basis and it also depends like what environment you want to deploy the model in. Um, yeah. So, how how do we um how is a KUA represented? As you might have guessed by now, as is with most models in AI these days, a cool is basically an LLM prompted and accepting inputs in a certain format. Um, so more specifically, it's a it's typically a vision language model because most models take in um screenshot representations. So you have a goal that's typically represented in natural language. Like again um sorry to use the same example multiple times, but the goal might be like buy me a blue monk that's less than $20. And you might have an initial screenshot that is based on what your environment initially has. So you have a goal and a screenshot. Uh this goes into your your policy which is essentially an LLM. Um and your LLM produces an action. Um so typically this is represented as like text. So your LLM produces some structured representation in text of what action it wants to produce. So this might be a click action with X and Y coordinates. Uh again we then execute this action. um we add it to the to the context that we feed into the LLM at the next step. Um we also get a new screenshot after we execute this action. So we get screenshot one after executing action one or screenshot zero.
[50:19.119] We again concatenate this to the context that we already have. Um we feed this again to the LLM and it produces the the next action that we are supposed to execute. We then execute this action again on screenshot one um and get a new screenshot and we concatenate this to the context and we we we again feed it to the LLM. So, um, you repeat this until until your LLM thinks it's done or
[50:40.160] until, uh, you run out of budget, I guess. Yeah. Um, so you have this loop. Um, it's typically called a react loop, reasoning and action. Um, and at every step, you add in you you concatenate the action that you produce the new screenshot to your ever growing context and you feed this back into the LLM. Um, so it's an interle like vision language interle uh, action history. you have natural language in the form of goals, actions. You also have like pixel inputs or potentially other metadata. Um, and this essentially is like concatenated into this long sequence, this long multimodel sequence that you feed into your language model. Um, so this grows and grows. Um, there's different ways that people used to handle like longer and longer context, but essentially, uh, you feed in like the context, um, to the LLM at every step and it generates the next action at that point in time. Um, of course, like we don't actually feed in the actual pixel inputs. Uh, typically for most language models, you would tokenize this in some way. So, you would tokenize like your natural language to become uh text tokens. You would convert your screenshots into like visual tokens or or some like pixelbased representations. And the nice thing about modern language models is that um this is typically just like concatenated directly. So, what it looks like to the model is that you have a bunch of embeddings uh that you feed into it at every single step. the model produces an action that you then tokenize and you uh
[52:04.640] concatenate it to the context. Um so you kind of just repeat this growing the number of tokens that you feed in at every step. Um and the model takes in that uh conditions on the context at every point. Um produces the next action and you kind of repeat this. So it's a very um nice and simple representation I think uh and and it works surprisingly well. Yeah. So you typically have like text that you convert into token embeddings. You have screenshots that you convert into visual embeddings. And all of this is just interled and concatenated and passed to your language model. Um and and these days your language model is a is a transformer. Um obviously that produces actions for the next turn.
[52:48.240] Yeah. So the question earlier about how like what action space we usually use. Um different models have different action spaces. uh if you squint they like generally look similar. So they typically have a set of actions that they can produce like click actions, scroll actions um or whatever other actions you have in your uh environment, right? Um so different models produce them in slightly different ways. For example, you might produce them as like pi autogy outputs or you might produce them as just like two calls or XML. Um and you would have to like process this and pass it into something that you can actually execute on your environment.
[53:23.200] Yeah. Um, there's a question in the back. Yeah. So the question is why do we use screenshots rather than HTML? I think HTML is also a valid well HTML is very messy but people have done like simplified versions of HTML as representations. Um I guess one reason that most so most recent models do not use um or they primarily rely on screenshots. They sometimes still use metadata but uh to generalize it broadly most recent models use uh screenshots as inputs. And I think the reason for it is that it's a lot simpler. Um so it's it's a lot simpler to like take a screenshot of your computer rather than trying to process the uh underlying uh like accessibility metadata. um you you can like get HTML easily on the web but you can't really do it as easily for like desktop computer use right um so for that reason I think people like prefer screenshots a lot more because it's a lot uh uh more simplified and I think somewhat surprisingly to some the screenshot based representation has achieved like very very strong performance so it seems almost like unnecessary to use HTML or other structured representations um so I guess that that's kind of like um Why? Uh it's very simple and it and it works somewhat surprisingly well. >> Yeah, great question. I think um this was something that you know people did not really accept as readily even one year ago. Um but yeah, the multimodal models are very good. So it's uh surprisingly easy to do this.
[55:21.599] >> Uh yes, another question. >> So if it to pick a button or something. So the loading screen as well. >> Yeah. So um there's a bunch of like different design choices you can do with regards to like how you can handle these edge cases. Um sorry the question is to the question is like whether whether you would take a loading screenshot immediately or what do you do when when you execute an action. So I think computer use is annoying and also interesting because it encounter you encounter a lot of these like age cases and infrastructure problems compared to like math or or coding models. Um it this can be resolved in a couple of ways. I think the most common way that people do this is that you typically like sleep or wait for a few seconds after executing an action. So you would execute an action and you sleep for like 3 seconds before you take the screenshot and execute the next thing. Um so if you do see a loading screen at that point in time, models are typically trained to wait again for another um specified amount of time. Um yeah, there's more complex infra infra um there's like more
[56:33.119] complex ways of designing models where you kind of like wait until the screenshot changes, but I think the most simple and common way to deal with it is you just sleep. Yeah. Thanks a lot for the questions. So um this is essentially at a high level like how almost all frontier coolers are trained uh to the best of my knowledge.
[56:57.280] Um sorry this is not how they're trained. This is how they are modeled and we'll soon like look a bit at how they are actually trained to do these uh um quite amazing things. Cool. So um with training a model uh essentially like most computer use agents are based on pre-trained uh vision language models. So training happens at a few different stages like
[57:25.119] obviously when you deploy GPD6 as a computer use agent uh you're not deploying I guess it's not obvious but you're not deploying a specialized version of GPD6 that can only do computer use. So it's kind of a general model that can have computer use as one of its capabilities and I think this is like increasingly the way that people do it. Um so computer use specific training
[57:45.200] happens at a few different stages. Um at the pre-training stage where models are trained uh with like large scale internet um data uh core specific data is typically injected in a few different formats. So you have kind of like your grounding data which is um data that encourages the model to learn to ground information in in screenshots. Um so this data typically looks at like the form of um you have a screenshot and you kind of have like a series of bounding boxes followed by coordinates. So what the model is learning to do here is that it's learning to um uh just like learning grounding abilities. So it's learning like if I want to click on the button I have to produce like these XY coordinates. So it's just learning like the most basic uh capabilities like learning localization, grounding and so on. Um you might also have some data in the form of like immediate actions. So this is actually very similar to the static evals that we talked about. Uh you might have these screenshots, you might have some like highle task and the goal here is to predict the next action that you will you will produce. Um so this is again like very noisy data. It might even be like synthetically generated. But the goal here in pre-training is just to introduce uh some form of like localization and grounding ability which the kua needs to do later on in order to um be able to produce like accurate click and um other types of actions. Yeah.
[59:07.280] Um after pre-training we typically have a post-training phase. Um so nowadays like this is typically mixed in with other model capabilities but uh to simplify it we can assume that we only care about like cooler for now. Um and in this post- training phase there's there's two different stages. So the first stage is that you typically do supervised fine-tuning um to train the model to mimic a human demonstration of of a computer use task. So you have a bunch of like human demonstrations that you can collect or otherwise like uh scrape from somewhere. Um, and this typically looks a lot like the trajectories that we already have. So the human is given a screenshot, the human does something and you record that action. So it might be a click action that the human does, you record that using some like specialized software that you have. Um, so you have a bunch of these like human demonstrations that essentially uh some humans um record humans doing cool like tasks. Um, nowadays it's also very common to make in synthetic data. So maybe you prompt your agent in a certain way or you prompt a stronger model to attempt the task and you kind of like keep the trajectories that you think are good. Um so this is like synthetically generated by a different model or the same model. Um and you also kind of mix this in. Uh so this is typically a bit lower quality and noisier than human data. But you can generate this at scale. So you don't need humans to actually produce this data. So it's actually generally still quite useful to have um this like synthetically model generated data in your post- training stage. Um the more interesting part I think is after this uh post- training supervised fine-tuning stage your model has some semblance of a cooler uh capability. So you can after after SFT you can typically prompt your model and it will
[01:00:46.480] do a decent job at most of these cooler tasks. I think the more interesting stage is um nowadays it's is it wasn't very common like one year ago but nowadays it's very common for kas to undergo an a reinforcement learning stage. So what happens here is that you kind of drop your cooler in a simulated environment or real world and you give it a thing to do. Um it it it will do that thing. So it will like try to buy a blue mark and at the end you will assign a reward to it either from like rubric based rewards, programmatic verifiers or like um just llm as a judge. You assign a reward to it in some way and you kind of do like reinforcement learning to reinforce the successful trajectories. Um so this is kind of interesting because it's like uh there is no human demonstration for these tasks. You kind of just let the model loose in a sandbox and it um tries a bunch of things uh learns from its uh successes and failures and it you you repeat this until until it's able to do this successfully. Um yeah so I think people are very excited about RL these days. Um but the other
[01:01:47.520] stages are also very important to ensure that the model has a strong foundation for doing RL. Yeah. So let's talk a bit more in detail about the various stages. So in the pre-training phase as I mentioned you kind of have these like screenshots and the goal of this pre-training data is just to learn uh very basic capabilities. So you probably learn like grounding abilities. um given this screenshot uh typically your model is trained to predict the next token uh for very long description that essentially boils down to like predicting the coordinates of items on the page. So you might have this like car label and the model has to predict the xy coordinates or the xy uh width and height of this bounding box. Um so it's kind of like very noisy data but data that you can imagine getting in very large quantities at scale. Uh so for example like this screenshot data it comes from the United website and you can easily generate like all of the bounding box and grounding data by just running a simple like JavaScript uh script on the website. So you can extract like all the bounding boxes. So you can collect this kind of grounding data very easily at scale which is why it's a great fit for pre-training where you want like a a lot of uh noisy but large volumes of data. The other type of data that people sometimes collect is um action prediction. Uh so you can collect for example in the same way from HTML pages you can get like the element name or element uh meta of certain things on the page and um you can provide this as as the input to the cool and has to predict again the xy coordinates of this. So this is a bit more like semantic understanding where you can provide like a HTML element and it has to predict uh what it wants to do or what it wants to click on. Um so again this is like very low-level capabilities. It's really just
[01:03:37.039] teaching the cooler how to understand screenshots and how to ground uh various items on the page. Yeah. So you basically learn like screenshot understanding. Uh you learn localization. So sometimes you learn like where items are on the page. uh you can also learn like the relationship of items with each other. So we know for example like the car button is to the right of the hotel button and uh this might be introduced in your pre-training data in Subway. Um but this is typically generated at scale and mixed into the early stages of frontier model pre-training um just so that our cooler has a bit of an easier time later on in in learning to ground and learning to produce actions.
[01:04:16.559] uh after pre-training the next stage is uh I think people typically call it post- trainining um so again like we are ignoring all the other data that goes into post- trainining and focusing on cooler data here um this typically comes in two forms so you have like human data where you pay a bunch of humans uh to annotate trajectories so you might give someone the task like find a birya taco recipe I could make in under three hours and a human here like actually attempts this task by navigating on a web browser um so over here like this example is from momo Momo web mix um where they collected like 36,000 human trajectories. So they collected 36,000 demonstrations of people doing tasks similar to this like going on the web and doing things and and um at every point you get the screenshot observation and you get the thing that the human did. Uh so you end up with a bunch of like screenshot action screenshot action trajectories. They also generated a bunch of um synthetic trajectories that we'll we'll talk about in a bit. Um but some open- source examples of like cooler data are Momo web mix and also agent net. So both of both of these papers essentially like collected human demonstrations uh of these tasks. So it's a very nice way to um train your cooler to like imitate human behavior. Um so condition on screenshot is trained to predict the next action in the sequence.
[01:05:37.440] Um, of course, increasingly so these days we can generate some like synthetic data for for this part of the training as well. Um, so Momo Momo web actually did this thing where they uh gave a model the accessibility tree of the website. So I think this was um uh addressed in some of the previous questions but uh some of these models do a lot better when you give them metadata. So they have a model that is trained to use like accessibility trees and they provide the accessibility tree which is essentially a simplified HTML representation uh to the model and they let this model like roll out taking actions in the accessibility tree space and they collect the same set of trajectories. Um so you kind of have this model that is using like privilege information that it might not have at actual eval and inference time. So it's uh doing a lot better than it would have if it just had screenshot inputs. Um, so you roll out this model, you collect like 105,000 synthetic traces in this way and you can also um uh not not all traces are retained. So you can kind of like look at the traces and keep just the ones that you think are higher quality based on some metric like using a different LLM to judge it or using a human to filter them out. Um, so traces are collected in this way either with human demonstration or or synthetic data. Um, and one analogy I really like with the synthetic data version where you have privilege information is that um, I actually don't know if it's true, but I heard that Tesla actually deploy deploys their cars um, during training with LA. So they have these cars that have LA installed and they do like self-driving and obviously if you have like 3D LA information, you do a lot better. Um, and they collect those trajectories and they discard like the LAR information and they train their model to just use
[01:07:16.960] like the pixelbased information. So you kind of have a teacher that is using a lot better information than you would have at inference time. Um and I think for a lot of cooler applications, people have attempted similar ways of injecting like metadata or extra information um in order to collect better traces. Um so I think it's a very nice way of introducing synthetic data. Yeah.
[01:07:38.400] Question in the back. Especially with certain pixel values. >> Yeah. So the question is whether um if you do like pre-training does the coordinate uh representation generalize? So if you train it on a Mac, does it generalize to a tablet or maybe even like computers with other resolutions? Right? Um and I think typically the way that we collect like coordinate data is that it's normalized. So uh coordinates aren't usually represented in absolute uh formats. So they're represented in like um zero to a thousand. Zero zero zero meaning like top left of the screen and thousand thousand meaning like bottom right of the screen. So they're like completely normalized and in my
[01:08:37.040] experience it transfers quite well even when the absolute resolutions are different. Yeah. Cool. Um question >> more efficient way of doing it like perhaps a >> Yeah. So the question is whether there's like more efficient ways if you have >> instead of getting the screenshot through it just stoing like values what certain items are in some database you can just
[01:09:18.799] >> there are boxes just like a description of what the box does >> yeah I think that's a That's also a reasonable way to do it. I think models these days are not trained as uh like monolithic coolers. Like if you try using like GPD6 Astro, it doesn't always just do computer actions. It will interleaf like bash calls and it will like write code instead. So I think models like typically try to solve it in the most efficient way possible. Um so they definitely like do use that kind of data at training and also at inference. Um yeah. Oh, okay. Okay, cool. Uh, okay. I'll cover a bit more quickly then. Yeah, basically we have a bunch of like um uh
[01:10:04.400] open- source data that we can train models on these days. So, a lot of um different examples and the final stage of training is actually uh post- training RL. So as I mentioned in this stage you kind of like deploy the model in a simulated environment or if you are feeling um you know extra spicy you can also deploy on the real web but people don't really do that nowadays. Um so you typically like deploy the model it takes actions in a simulated environment and at the end it produces you get a reward from the environment or from some other judge and you update the policy based on the rewards. So uh broadly speaking like if you get a positive reward you upweight the actions that happen in that trajectory a bit higher and if you get a zero or negative reward you um downweight those actions. So your model essentially is trained to maximize the reward in an environment and in this case it's like doing it completely autonomously. So no human demonstrations. Um the tasks that the model is attempting might come from humans but uh typically it's like operating in an autonomous fashion.
[01:11:05.760] Yeah. And um hopefully it's obvious why we need simulated environments for RL. A lot of the tasks that we care about, we don't really want to execute them repeatedly on the real world. For example, you can only book uh an airport ride on Uber so many times before you get banned or you run out of money, right? So you kind of want like simulated environments to do
[01:11:23.120] reinforcement learning in. Um and there's several uh benchmarks that uh kind of like tackle the reinforcement learning building uh paradigm. Um so I think I won't go through these in detail but uh you can reference them on the slides. Um but essentially I like to show this graph which shows that the reinforcement learning part of post training is actually very important for uh introducing like strong cooler abilities. Um so you start off with a success rate after SFT of about.5 and if you do like reinforcement learning at the end you climb to about like 6 or or more. Um so it's a very important part of ka training and uh anecdotally it also introduces some abilities that is hard to do through human uh demonstration alone. So it's much easier
[01:12:09.360] to train a model to operate over long horizons um using reinforcement learning because you typically don't have human demonstrations of very long uh tasks. It's very expensive to collect those. Yeah. Um so to summarize yeah we covered like how coolers are trained. Uh you typically have a few training stages like pre-training um post- training where you do like behavior cloning on human demonstrations and synthetic data and you also have like the final cherry on top where you do the model uh reinforcement learning phase um of running it in simulation and getting rewards in the environment and hopefully by the end of this you get a strong cool. Um but at a high level this is like the pipeline that most labs use to train cooler models these days.
[01:12:56.400] So to summarize um kas are a bit different from other agents. I think they are uh kind of like really nice because they operate directly on the same interface as humans. Um it's very tricky to evaluate coolers and people have been doing it for for a while. Um but essentially we want to target the right metrics and level of behavior.
[01:13:14.800] Um cooler models are designed so that they predict actions from interled visual history. So you kind of have this interleaf context of screenshots and actions that you previously took and the model has to condition on this to produce the next action. Uh models these days are trained in multiple stages. So you have pre-training with large scale noisy data. You have SFT where you try to
[01:13:35.280] behavior clone human or synthetic demonstrations and you have a final RL phase where you uh let the model um roll out itself in simulated environments and you reward it for good behaviors and penalize it for bad behaviors. Um essentially that's uh probably the summary of all I have time to cover off cover today but I encourage you to visit the slides for some uh remaining takes
[01:13:59.520] on what we can still work on for Kers. Um, essentially I think speed is still an issue. Personalization is a very ripe area for research. Uh, infrastructure and UX is a big concern. And, um, I'm also very excited about multi- aent systems. Yeah. Uh, thank you for listening and happy to take questions for remaining time or Yeah. God bless.