CMU 11-768 Fall 2026 · Lecture 6已发布

Domains 1: Coding Agents

日期: Thu, Sep 10, 2026
课程资料: 44 项
内容产物: 4 项

课程简介

编码 Agent 的运行框架、工具接口与软件工程任务。

Assignment 1 · Build an Agent Harness截止

CMU 11-768 Fall 2026 · Lecture 6 · Domains 1: Coding Agents 学习笔记

来源：平台字幕与官方 Slides。内容已由用户审核确认。

依据：Lecture 6 官方 Slides 逐页文本提取（待核）与中文逐字稿整理稿（机器翻译、待审核）。本笔记已由用户审核确认；图示、公式与数值若无法从文本确认，均在文中标注“待核”。关键论点标注 Slides 页码；仅有逐字稿支撑的内容标注为“逐字稿”。

一、学习目标

学完本讲后，应当能够：

区分三类“会写代码”的能力层次：单步补全、修改代码仓库、承担完整软件开发生命周期（Slides 第 2 页）。
说明构建编码智能体的三个基本要素——Prompt、Tools、LLM——并解释本讲为何重点放在 Tools 与 LLM 两部分（Slides 第 3 页；逐字稿说明提示与记忆已在前两讲覆盖）。
描述编码模型训练的三个阶段：预训练、中期训练（含 SFT）、RL 后训练，以及每个阶段中编码特有的细节（Slides 第 6 页）。
解释为什么“编辑（Editing）”与“推理（Reasoning）”是代码模型区别于通用语言模型的关键能力（Slides 第 5 页）。
说明填充（infilling）目标如何把“挖空—预测”转化为自回归任务，并理解其收益与对普通补全的影响（Slides 第 14–17 页）。
区分多种代码评估范式：与参考解比较、结构比较、嵌入比较、执行检查，并说明各自的优缺点（Slides 第 20–24 页）。
描述编码智能体的“定位—编辑—验证”循环，以及工具集设计（尤其文件编辑格式）对性能的影响（Slides 第 33、41–43 页）。
解释 SWE-bench 风格任务如何构造、fail-to-pass / pass-to-pass 的含义，以及通过变异注入 bug 生成训练数据的思路（Slides 第 47、51–52 页）。
概述更广泛的软件工程任务（前端开发、库实现、测试生成、CI 修复、维护等）及其评估方式（Slides 第 56–72 页）。
初步理解“代码行为预测 / 世界模型”的动机、风险与开放问题（Slides 第 74–79 页）。

二、概念主线

本讲的主线可以概括为：从“会写一段代码”到“会改一个仓库”，再到“会做软件工程”，每一步都需要新的数据、新的工具与新的评估方式。

2.1 三个层次（Slides 第 2 页）

单次补全（single completion）：给提示，产出一段代码。
修改仓库（modify multiple files）：在既有代码库中定位、修改多个文件。
完整开发（full software lifecycle）：需求、评审、部署、监控、维护等。

逐字稿中 Graham 明确说，本次讲的重点是 Tools 与 LLM，因为 Prompt/记忆已在前两讲讨论（逐字稿）。这提示我们：编码智能体的“难点”更多在于模型能力和工具接口设计。

2.2 单步代码模型的三种能力（Slides 第 5 页）

语言（Language）：语法、API、惯用法。
编辑（Editing）：需要同时看到缺口两侧（condition on both sides）。
推理（Reasoning）：从规格到行为。

逐字稿中老师强调编辑“不是免费得到的，需要训练进模型里”，并举了 Python 空白字符有意义、不能随意归一化的例子（逐字稿；对应 Slide 第 7–9 页“preserve structure / preserve whitespace”）。

2.3 训练三阶段（Slides 第 6 页）

预训练：大规模含代码语料。
中期训练：代码混合比例、更长上下文。
RL 后训练：推理 + 执行奖励。

这条主线贯穿全讲：数据清洗、分词、填充、评估、RL、智能体循环、多脚手架训练，最后延伸到世界模型。

三、关键机制

3.1 预训练：从代码与文本中学习

Slides 第 7 页给出的损失写法为下一 token 预测（示意）：

L_{L M} = - E_{x \sim D_{m i x}} \sum_{t} \log p_{θ} (x_{t} ∣ x_{< t})

符号与假设（据 Slide 第 7 页文本）：

$D_{m i x}$ ：混合数据分布（网页文本、技术文章、数学、源代码、文档）。
$x_{t}$ ：第 $t$ 个 token； $x_{< t}$ 为其前缀。
$p_{θ}$ ：参数为 $θ$ 的模型。
假设：标准自回归分解；损失对序列中各位置求期望。该公式的完整排版与是否含归一化常数待核（PDF 文本提取可能丢失上下标）。

Slides 第 7 页还列出三条原则：混合来源（Mix sources）、保留结构（Preserve structure：缩进、文件边界、API、文本—代码关系）、训练中评估（按领域留出损失 + 功能性编码任务）。

**数据选择与清洗（Slides 第 8 页）：**按语言与来源过滤、去重（fork、vendored 库、近乎相同的文件、复制的解法）、来源控制（许可证、日期、退出、敏感数据过滤）、防止泄漏（按仓库或问题族切分、去污染评测任务）。逐字稿中提到 StarCoder 论文与 trufflehog 之类的敏感数据工具（逐字稿）。

**分词（Slides 第 9 页）：**字节级 BPE；保留空白；压缩高频空白串。Slide 给出的示意为 if·ready: ↵ ····return·value ↵ 的分词结果，具体切分需对照 PDF 核验。逐字稿讨论了两点：把空白与非空白分开分词是常见做法，理由是避免输出“半个单词”；但如果允许跨空白合并，可显著减少 token 数，代价是自动补全等场景可能出现超长 token（逐字稿）。

**规模（Slides 第 10 页）：**StarCoderBase 15.5B 参数、1T token、80+ 语言，来源包括源码、issue、commit、notebook；并称“多数语言随 token 增加而提升”。坐标轴含义见 Slide 文本（横轴训练 token，纵轴 pass@1）。

3.2 中期训练：加代码、加长上下文

Slides 第 11 页：继续语言模型训练，提高代码曝光同时保留文本与数学；经验性地选混合比例；Code Llama 在 Llama 2 基础上追加 500B token（7B/13B/34B）；随后扩展上下文，用相关文件与更长序列。

Slides 第 12 页给出混合比例对比表（Code/text/math 与 Common code、MATH、MMLU）：

100/0/0：49.8 / 10.3 / 23.8
70/20/10：48.3 / 33.2 / 62.9

逐字稿解释：纯代码配方在代码上略好，但在数学与问答上明显变差；加入文本与数学后代码仅有小降、其他能力保住（逐字稿；数值以 Slide 第 12 页为准）。

Slides 第 13 页：一致序列（相关文件、路径、跨文件依赖）；Qwen2.5-Coder 从 8K 训练到 32K，YaRN 支持 128K；用跨文件补全、填充、短上下文保持来验证。逐字稿补充：把所有文件直接拼接常常超过最长上下文，Google 单体仓库据称有数十亿行（逐字稿，该数字需核）。

3.3 填充（Infilling）

Slides 第 14 页给出经典例子：

python

def is_positive(x: int) -> ____:
    return x > 0

只看左侧无法确定返回类型，而右侧函数体提供了 bool 的证据；正确补全为 bool。

Slides 第 15 页给出机制：训练序列为 prefix [M0] suffix [M0] span [END]；推理为 prefix [M0] suffix [M0] → 生成 span。多个洞用不同哨兵（M0、M1……）关联位置。可零样本执行：补全代码、推断类型、生成注释、重命名变量。

Slides 第 16 页给出通过率对比（HumanEval 派生填充任务）：

推理方法	单行	多行
从左到右，1 个候选	48.2%	24.9%
从左到右，10 个候选 + 重排	54.9%	28.2%
因果掩码填充	69.0%	38.6%

逐字稿强调“这完全不需要重排”，仅靠填充式推理就能接近大幅改进（逐字稿；数值以 Slide 第 16 页为准）。

Slides 第 17 页给出目标函数消融（相同 52B token 训练预算）：

从左到右语言建模：HumanEval 6.0%、MBPP 8.9%
因果掩码：HumanEval 8.0%、MBPP 10.9%

逐字稿评论：这说明加入填充训练并非要与标准补全做取舍（逐字稿；数值以 Slide 第 17 页为准）。

3.4 更多从代码中学习的方式（Slides 第 18 页）

Commit diffs：before + message → after/diff（OctoPack）。
Diagnostics：broken code + error → repair（DrRepair）。
Tests：program + test outcomes → reward（CodeRL）。
Execution traces：program → executed lines + states（CodeExecutor）。

逐字稿补充：CI 测试历史是“几乎免费”的丰富数据来源；测试与执行轨迹相关（逐字稿）。

3.5 评估生成代码

**与参考解比较（Slides 第 20 页）：**exact match、BLEU-4（token n-gram 重叠 + 简短惩罚）。Slide 用整数正负判断举例：return x > 0 与 return x >= 0 只差一个 token，但行为在 0 处不同；return not (x <= 0) 文本差异大但语义等价。

**结构比较（Slides 第 21 页）：**CodeBLEU，基于抽象语法树与数据流匹配。

**嵌入比较（Slides 第 22 页）：**CodeBERTScore，编码提示与每个代码片段，比较上下文 token 向量，最佳匹配得到精确率/召回率/F 值。

**执行检查（Slides 第 23 页）：**对候选代码在隔离运行时中运行测试用例，比较输出；Slide 给出 is_positive 在 −1、0、1 上的失败示例（0 处返回 T 而期望 F）。

**优缺点（Slides 第 24 页）：**可接受不同正确实现、能通过执行检测语义错误、与观测行为一致；但有限测试可能漏掉错误行为、需要可信测试与隔离依赖、需要时间与稳定环境。逐字稿进一步指出：好的单元测试极难写，某些基准的测试存在假阳性或假阴性，例如界面上“了解更多”按钮的测试可能连背景色一起检查（逐字稿）。

**基准（Slides 第 25 页）：**HumanEval（164 个 Python 函数，docstring → 补全，隐藏单元测试）；CodeContests（竞赛题，statement → 完整程序，编译 + 测试）。

**pass@k（Slides 第 26 页）：**k 次采样中至少一次通过所有测试的概率。

**非功能需求（Slides 第 27 页）：**NoFunEval 提出运行时效率（跑测试 + 计时，得分 speedup）、延迟/资源（对比参考编辑，DiffBLEU）、可维护性/安全（静态检查 + 编辑匹配，CodeQL × DiffBLEU）；共 397 个代码编辑任务。逐字稿补充说：对非智能体任务，效率基本没被纳入训练（逐字稿）。

**从测试结果学习（Slides 第 28 页）：**采样 → 推理 + 最终代码 → 运行测试 → 通过/失败作为奖励 → 更新模型，使被奖励的回答更可能。公式（据 Slide 第 49 页类似形式）：

max_{θ} E_{τ \sim π_{θ}} [R (τ)]

符号与假设： $τ$ 为轨迹， $π_{θ}$ 为策略， $R$ 为奖励。此处假设奖励可定义、可计算；探索与信用分配问题在 Slides 第 49 页被点出（“晚失败无法指出早先哪一步做错”）。公式完整形式与梯度估计待核。

**推理示例（Slides 第 29 页）：**计算 $1 + \dots + n$ ， $0 \leq n \leq 10^{12}$ 。直接迭代太慢；配对推导出 $n (n + 1) / 2$ 并检查 0、1；错误记忆为 $n (n - 1) / 2$ 在 $n = 1$ 即错。

RL 后的代码准确率（Slides 第 30 页）：16K → 32K 为 RL 期间回答 token 上限（K = 1,000 token），★/64K 表示评测时给更多 token 但不再 RL；橙色虚线为 o3-mini。具体曲线数值待核。

3.6 智能体编码

**单步与智能体对比（Slides 第 32 页）：**单步是自包含提示 → 推理 + 代码 → 评估答案；智能体是 issue + 既有文件 → 搜索 → 编辑 → 跑测试 ↺ → 补丁 + 证据。

**定位—编辑—验证循环（Slides 第 33 页）：**仓库含 cli.py、config.py、test_config.py；定位“零在哪里变成默认值？”；编辑“仅当值为 None 时用默认”；验证零 + 默认用例 + 调用方回归；失败则修正假设。

**定位（Slides 第 34 页）：**示例命令 rg 'retries' buggy_config.py 得到 return config.get("retries") or 3；用环境变量运行单测复现 AssertionError: 3 != 0。

编辑（Slides 第 35 页）：

之前：return config.get("retries") or 3（所有假值都选默认）
之后：value = config.get("retries") 且 return 3 if value is None else value（显式 0 得以保留）

验证（Slides 第 36 页）：python3 -m unittest -v test_config 输出 missing → 3、None → 3、zero → 0、positive → 2，Ran 4 tests: OK。

**固定工作流替代方案（Slides 第 37 页）：**Agentless（Xia 等），不做迭代式工具调用，而是先定位文件、再定位类与函数、再选行，最后用单步模型生成补丁。逐字稿说明：该方案曾在短期内“令人沮丧地有效”，因为当时模型对单步代码求解训练充分、对智能体工具调用训练不足（逐字稿）。

**仅 shell 工具集（Slides 第 39 页）：**模型选命令，一个 shell 接口；读/搜用 rg、cat、find；编辑用 sed、Python、patch；执行检查用 pytest、构建命令；bash 返回输出 + 退出码 + 超时。mini-SWE-agent 即此类。

用 shell 编辑（Slides 第 40 页）：sed 's/^RETRIES = 3$/RETRIES = 5/' settings.py > settings.new 然后 mv。

**整文件与文本替换（Slides 第 41 页）：**整文件写简单但重复未变代码；搜索/替换紧凑但需无歧义匹配。Slide 给出 OpenHands/Pi/OpenCode 的编辑工具与 Aider 的 SEARCH/REPLACE 格式示例。

**diff 与 patch（Slides 第 42 页）：**统一 diff 标准但含行号与上下文；Aider 省略 hunk 行号；Codex 用文件操作 + 上下文，无行数。

**diff 格式影响（Slides 第 43 页）：**Aider 重构基准（89 个 Python 任务），GPT-4 Turbo（gpt-4-1106-preview）成功率：SEARCH/REPLACE 20%，简化统一 diff 61%。逐字稿补充：GPT-4 Turbo 在统一 diff 上更好是因为训练数据中见过更多；Gemini 曾因自有编辑格式在测试框架上表现较差（逐字稿；数值以 Slide 第 43 页为准）。

**找相关代码（Slides 第 44 页）：**从症状出发（flag、loader、retry loop 都可能匹配），沿值追踪（配置输入 → 默认逻辑），例如 rg 'retries' .。

**依赖图导航（Slides 第 45 页）：**LocAgent 给智能体特殊工具沿依赖路径追踪（config.py → cli.py → client.py → test_config.py），检索相关邻域后用执行验证。逐字稿提醒：让此类方法优于更简单方法在实践中很难（逐字稿）。

3.7 评测与训练

**SWE-bench（Slides 第 47 页）：**从 GitHub issue 构造；取 issue 被解决时的仓库状态（PR 之前）；找到与 issue 相关的测试；智能体生成补丁；应用补丁与测试；全部相关测试通过得 1 分，否则 0 分。两种测试：fail-to-pass（PR 前失败、PR 后通过）与 pass-to-pass（PR 前后都通过，防回归）。

**构建可运行训练任务（Slides 第 48 页）：**起始状态（文件 + 依赖 + 测试命令）、issue（请求行为 + 复现）、检查（bug 修复前失败、修复后恢复）。收集轨迹前先验证环境。

**多步修复的 RL（Slides 第 49 页）：**助手动作（搜索、编辑、测试/结束）为训练目标；文件查找结果、补丁应用、测试输出只作为上下文；终端奖励为“是否修好”。问题：晚失败无法指出早先哪一步决策错误。公式： $max_{θ} E_{τ \sim π_{θ}} [R (τ)]$ 。

训练数据与推理算力（Slides 第 50 页）：SWE-Gym，32B 模型；学习式验证器选择用于推理扩展。具体曲线待核。

**注入 bug 造修复任务（Slides 第 51 页）：**准备可执行基线 → 变异代码（不改测试）→ 保留新的测试失败。SWE-smith。

**注入与检查（Slides 第 52 页）：**基线四个测试全过；把 return 3 if value is None else value 变异为 return 3 if not value else value，零的测试失败；基线 4 过、变异 3 过 1 败、恢复 4 过。

**跨语言（Slides 第 53 页）：**Python（pyproject.toml、pytest）、TypeScript（package.json、npm test）、Rust（Cargo.toml、cargo test）；先读项目配置再决定构建与测试命令。Multi-SWE-bench、SWE-bench Multilingual。

**多脚手架训练（Slides 第 54 页）：**脚手架 = 提示 + 工具 + 智能体循环 + 上下文管理；同一模型权重；示例脚手架 OpenHands、OpenCode、Codex；任务检查（补丁 → 测试奖励）；轨迹 + 奖励更新模型。Nemotron 3 Ultra：每个任务分布 ≥2 个脚手架；Polar：原生脚手架调用共享模型 API 代理。逐字稿补充 Nvidia Neotron 报告在 5 个或更多脚手架上训练（逐字稿；具体数量以 Slide 第 54 页“≥2”为准，逐字稿数字待核）。

3.8 前端开发

**智能体驱动的前端（Slides 第 56 页）：**定位组件与事件处理器 → 编辑逻辑与样式 → 浏览器验证（类型 → 点击 → 检查保存状态；截图 → 视觉理解）→ 失败则修正。SWE-bench Multimodal 提供浏览器与截图工具。

**浏览器智能体 vs Playwright（Slides 第 57 页）：**浏览器智能体由模型选下一步动作（观察 → 选动作 → 输入/点击，适应式探索）；Playwright 脚本由模型写可重复检查（fill、click、expect、screenshot）。截图 → 视觉检查或批准基线比较。

**验证前端修复（Slides 第 58 页）：**编辑 value.length → value.trim().length；行为：0 条记录、有效名仍可保存；外观：错误提示在 Name 旁可见可读。

**评测视觉问题解决（Slides 第 59 页）：**issue + 图片；起点仓库；候选补丁；执行检查（空白 → 错误、无新记录）；回归检查（有效名 → 保存、既有测试通过）。要点：issue 里有图片并不代表用图像相似度作为评分规则。

3.9 更广泛的软件开发

**从需求到维护（Slides 第 61 页）：**需求 → 实现循环（编辑、测试）→ 评审（diff + 证据、发布决策）→ 部署/监控（生产信号、回滚路径）→ 维护（新需求、观测到的失败）。要点：测试通过只是其中一步，不是发布决策。

**开发者时间分布（Slides 第 62 页）：**编码 15%、会议 + 邮件 25%、调试 14%、跑测试 8%、代码评审 5%、需求 + 文档 6%、帮助 + 同步 + 社交 11%、学习 + 行政 + 杂项 8%、休息 8%。来源：Meyer 等 2019，5,928 份微软自我报告工作日。

任务对比（Slides 第 63 页）：

任务	产物	环境/验证器	时间视野
修复	补丁	仓库 / 回归测试	一个问题
库创建	实现	脚手架 / 包测试	多个函数
测试生成	新测试	有 bug + 已修复版本 / 判别	一个行为
CI 修复	代码或配置	工作流 / 所需检查	一次构建运行
演化	累积变更	持久应用 / 里程碑 + 回归	依赖任务

**定位任务（Slides 第 64 页）：**CodeScout；搜索策略：issue + 仓库 → 检查/搜索 → 预测文件（如 django/…/datetime.py，模块 TruncDate，函数 as_sql）；奖励为三个层级上的 F1；与 gold patch 中位置比较，补丁不作为策略输入。

**应用创建与修改（Slides 第 65 页）：**ViBench；需求（产品/功能描述）；Zero-to-One 建新应用 15 个；Vibe-on-Ref 扩展参考 MVP 45 个；Vibe-on-Vibe 扩展智能体早先 MVP 45 个；每个结果用人工编写的浏览器测试计划检查。

**库实现（Slides 第 66 页）：**Commit0 与 ProgramBench；库脚手架中 parse(…)、evaluate(…) 缺函数体，配文档与测试，一起实现并跑集成包测试；ProgramBench 给参考可执行文件，探测输入观察输出，新实现匹配可观测行为，内部设计可不同。

**维护（Slides 第 67 页）：**SWE-Milestone。

**测试生成（Slides 第 68 页）：**SWT-Bench（Mündler 等 2024）。

**CI 修复（Slides 第 69 页）：**观测到构建失败（Node 16 而包要求 Node >=18，npm ci: EBADENGINE）；因果相关改动是把 setup-node 配成 node-version: '20'；重跑安装、构建、测试。数据集 LCA CI Builds Repair。

**学习新任务的技能（Slides 第 70 页）：**找相关代码（搜索/依赖）、实现函数（规格 → 代码）、写 bug 测试（issue → 可执行检查）、训练策略（SFT 或 RL）；留出任务：新仓库上修 issue、建库、检测新 bug；等推理预算。

**生成练习项目（Slides 第 71 页）：**SWE-Playground（Zhu、Gandhi & Neubig）。

迁移到新开发任务（Slides 第 72 页）：32B 模型，Base 与 Hybrid-Gym 对比：SWE-bench Verified 修复 7 → 32.4；SWT-Bench Verified 测试生成 9.01 → 16.86；Commit0 Lite 建库 8.34 → 13.45。柱状图数值以 Slide 第 72 页为准，图形细节待核。

3.10 代码行为预测与世界模型

**选择动作并预测效果（Slides 第 74 页）：**历史 h（已检查代码、工具结果）；策略 $π (a ∣ h)$ 选择“跑测试”；真实环境执行，观测到断言失败；世界模型 $p (o ∣ h, a)$ 预测测试结果；预测可能错；历史也条件化预测。

预测执行：共享引用（Slides 第 75 页）：

python

items = [1]
alias = items
alias.append(2)
result = len(items)

真实执行：items = [1, 2]、alias = [1, 2]、result = 2；错误预测是把别名当作副本。

**预测何时帮助智能体（Slides 第 76 页）：**两个修复 A（对）与 B（错）；模型预测 A 失败、B 通过 → 错误决策选 B 丢弃 A；因此要跑两个候选来检查模型判断；比较任务成功率与总延迟/token/工具调用。

**从程序执行学习（Slides 第 77 页）：**FAIR CodeGen Team 等 2025，CWM。

**构建与使用模拟器（Slides 第 78 页）：**Dainese 等 2024，Code World Models / GIF-MCTS。

**开放问题（Slides 第 79 页）：**更好的决策（在同一批问题上度量预测误差与任务成功率）、更低的成本（含延迟、token、真实环境调用）、新设定（留出程序、依赖、环境）、何时执行（变化回退阈值，同时度量失败与节省）。

四、公式的假设与符号汇总

预训练损失（Slide 第 7 页）： $L_{L M} = - E_{x \sim D_{m i x}} \sum_{t} \log p_{θ} (x_{t} ∣ x_{< t})$ 。假设标准自回归分解、混合分布采样；记号排版待核。
RL 目标（Slide 第 49 页，另见第 28 页思想）： $max_{θ} E_{τ \sim π_{θ}} [R (τ)]$ 。假设轨迹可采样、奖励可计算；信用分配难点见 Slide 第 49 页。
pass@k（Slide 第 26 页）：k 次采样至少一次通过全部测试的概率；具体无偏估计式未在文本提取中给出，待核。
BLEU-4 / CodeBLEU / CodeBERTScore / DiffBLEU（Slides 第 20–22、27 页）：文本 n-gram 重叠、AST + 数据流、上下文嵌入匹配、与参考编辑的相似度。具体公式待核。

五、常见误区

“会写代码 = 会做软件工程”。Slides 第 2、61、62 页反复提示：编码只占开发者时间约 15%，评审、部署、监控、维护同样重要。
“编辑是免费的”。逐字稿明确说编辑需要训练；Slides 第 5 页把 Editing 与 Language、Reasoning 并列为三项能力。
“填充会牺牲普通补全”。Slides 第 17 页消融说明因果掩码反而更好；逐字稿强调不是取舍。
“测试通过就等于正确”。Slides 第 24 页列出有限测试可能漏掉错误行为；逐字稿举了按钮背景色假阴性的例子。
“diff 格式无所谓”。Slides 第 43 页显示 SEARCH/REPLACE 20% 对简化统一 diff 61%（同一模型）。
“一个 shell 工具就够了”。Slides 第 39–41 页与逐字稿指出：文件编辑接口需要专门设计，搜索/替换需无歧义匹配。
“世界模型预测可以直接信”。Slides 第 75、76 页给出别名误判与错误决策的例子；Slides 第 79 页把“何时执行”列为开放问题。
“基准测试是干净的”。逐字稿说每个著名基准都有论文抱怨单元测试质量；Slides 第 24、48 页强调环境与测试可信度。

六、复习问题

Slides 第 2 页的三层能力分别是什么？各需要一个什么样的数据或工具支持？
为什么预训练时要保留缩进、文件边界与文本—代码关系（Slides 第 7 页）？Python 空白字符的例子说明了什么（逐字稿）？
Slides 第 9 页的分词示例中，空白串被合并成 token 的好处与风险各是什么？
Slides 第 12 页的混合比例表说明了什么权衡？如果你是产品团队，会如何选择？
填充任务如何把“挖空预测”变成自回归（Slides 第 15 页）？多个洞如何处理？
Slides 第 16 页三种推理方法的差异是什么？为什么“因果掩码填充”优于“多候选 + 重排”？
比较“与参考解比较”“结构比较”“嵌入比较”“执行检查”四种评估范式（Slides 第 20–24 页），各自最容易被什么错误欺骗？
SWE-bench 的 fail-to-pass 与 pass-to-pass 分别防止什么错误（Slides 第 47 页）？
定位—编辑—验证循环中，Slides 第 35 页的“之前/之后”代码差异为什么重要？它修复的根因是什么？
为什么文件编辑格式会显著影响智能体性能（Slides 第 41–43 页）？搜索/替换格式的主要缺点与缓解方式是什么？
变异注入 bug（Slides 第 51–52 页）相比从 PR 构造任务，优势与劣势各是什么？
多脚手架训练（Slides 第 54 页）试图解决什么问题？为什么模型在不同脚手架上表现会不同？
前端任务为什么需要视觉验证（Slides 第 56–59 页）？Slides 第 59 页说“issue 里有图片并不代表用图像相似度评分”，这提醒我们什么？
Slides 第 63 页的五种任务在“环境/验证器”和“时间视野”上如何不同？这如何影响训练数据需求？
世界模型 $p (o ∣ h, a)$ （Slides 第 74 页）在什么情况下可能误导智能体？Slides 第 76、79 页建议如何评估它的价值？
逐字稿提到模型对 Python 有强偏好（逐字稿），Slides 第 53 页展示多语言配置。这两者之间可能有什么张力？

七、给定资料中的官方来源链接

课程主页：https://www.cmu-agents.com/
课程日程与阅读：https://www.cmu-agents.com/#schedule
作业与项目要求：https://www.cmu-agents.com/#assignments
Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
课程视频：https://www.youtube.com/watch?v=1BWeH1oOM7k&list=PLSN0qpDfUvTM&index=6
官方课程日程：https://www.cmu-agents.com/#schedule
Lecture 6 slides：https://www.cmu-agents.com/slides/lecture-06-coding-agents.pdf
Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1

说明：以上链接为给定资料中列出的官方来源；本笔记未另行检索或添加资料之外的链接。所有数值、图表与公式细节请以官方 PDF 与视频为准；标注“待核”处表示仅凭文本提取无法确认。

从补全到软件工程：CMU 11-768 Lecture 6 如何拆解 Coding Agents

来源：平台字幕与官方 Slides。内容已由用户审核确认。

一、这节课到底在回答什么问题

如果只看标题“Domains 1: Coding Agents”，很容易以为这是一节工具巡礼：把当下流行的编码智能体列一遍，讲讲它们怎么调用终端、怎么改文件。但 Lecture 6 的组织方式不是这样。它更像是在追问一串递进的问题：

一个“会写代码的语言模型”，和普通语言模型差在哪？
把这样的模型放进智能体循环后，仓库级编码要多出哪些环节？
这些环节里，工具接口的设计为什么能显著改变成功率？
当任务从“修一个 bug”扩展到前端、库实现、CI 修复时，训练和评估要怎么变？

Slides 第 2 页把整个领域切成三层：写代码（单次 completion）、修改仓库（多文件修改）、执行完整开发流程（full software lifecycle）。这三层不是并列的，而是难度与自由度的递进。课堂的讲述顺序也基本沿着这条线：先讲单步代码模型（第 4–30 页），再讲智能体式编码（第 31–45 页），然后是评估与训练（第 46–54 页），最后扩展到前端与更广的开发任务（第 55–72 页），并以代码行为预测收尾（第 73–79 页）。

学习者解释：我理解这节课的核心张力在于——单步模型和智能体的“能力来源”不同。单步模型靠预训练和推理能力一次性给出答案；智能体靠工具、循环和验证把不确定性摊开。课程用大量篇幅处理后者，说明在仓库级任务上，工程接口往往比模型规模更早成为瓶颈。

二、单步代码模型：三件事缺一不可

Slides 第 5 页给出写代码的三个要素：Language（语法、API、惯用法）、Editing（同时依赖两侧上下文）、Reasoning（从规格到行为）。这三件事看起来朴素，但每一项都对应一种训练决策。

2.1 预训练：代码不是普通文本

Slides 第 7 页强调预训练要混合网页文本、技术文章、数学、源码和文档，并保留结构——缩进、文件边界、API、文本与代码的关系。第 8 页讲数据清洗：按语言和来源过滤、去重（fork、vendored 库、近似文件）、记录许可证与来源日期、做敏感数据过滤、防止评测泄漏。

学习者解释：分词看似底层，但它决定了模型“看见”代码的粒度。如果缩进被压成一个 token，模型就更容易学到块结构；如果每个空格都是独立 token，序列会变长且结构信号被稀释。这也解释了为什么讨论代码模型时总绕不开 tokenizer。

2.2 中期训练与长上下文

Slides 第 11 页引 Code Llama：在 Llama 2 基础上追加 500B token，并扩展上下文，训练相关文件和更长序列。第 12 页给出 Qwen2.5-Coder 的混合比例对比：100/0/0（code/text/math）时 Common code 49.8、MATH 10.3、MMLU 23.8；70/20/10 时为 48.3、33.2、62.9。课程用它说明——纯代码会牺牲通用能力，而混合能在代码几乎不掉的情况下保住数学与问答。

第 13 页讲长输入：把相关文件、路径、跨文件依赖拼成连贯序列；Qwen2.5-Coder 从 8K 训练到 32K，再用 YaRN 扩到 128K；并用跨文件补全、infilling 和短上下文保持来验证。

2.3 Infilling：让模型看右侧

Slides 第 14 页的例子非常直观：

def is_positive(x: int) -> ____:
    return x > 0

只看左侧函数体，返回类型难以确定；但右侧 return x > 0 强烈暗示是 bool。第 15 页给出机制：训练时把序列写成 prefix [M0] suffix [M0] span [END]，推理时放 prefix [M0] suffix [M0] 让模型生成 span；多个洞用不同 sentinel。第 16 页的实验数字（HumanEval 派生的 infilling 任务）显示：从左到右单候选 48.2%/24.9%，从左到右 10 候选重排 54.9%/28.2%，因果掩码 infilling 69.0%/38.6%。第 17 页进一步说明，因果掩码训练在标准补全上并不吃亏（HumanEval 8.0% vs 6.0%，MBPP 10.9% vs 8.9%）。

学习者解释：Infilling 的价值不只是“填空”。它把“编辑”这件事变成可训练目标：模型学会同时利用改动点的左右两侧。这正好对应真实开发中“在已有代码中间插入逻辑”的常见动作。

第 18 页列出更多学习信号：commit diff（before + message → after/diff）、诊断（broken code + error → repair）、测试（program + test outcomes → reward）、执行轨迹（program → executed lines + states）。

三、评估生成代码：从文本相似到执行

Slides 第 20 页讲参考解比较：精确匹配、BLEU-4。课堂举的例子很能说明问题——return x > 0 与 return x >= 0 只差一个 token，但在 x=0 时行为不同；而 return not (x <= 0) 文本差异大，对整数却等价。第 21、22 页给出 CodeBLEU（比较 AST 与数据流）和 CodeBERTScore（嵌入后做 token 级匹配）作为结构/表示层面的替代。

第 23 页转向执行检查：候选代码在隔离运行时中跑测试用例，比较输出。第 24 页列优缺点：能接受不同正确实现，但有限测试可能漏掉错误行为；能检测语义错误，但需要可信测试、依赖和隔离环境；匹配观测行为，但需要时间和稳定环境。课堂特别强调单元测试的假阳性/假阴性问题：测试不够严格会放过错误实现；而像“右上角好看的按钮”这类需求，测试可能去检查背景色，造成假阴性。讲者说这是当前训练编码智能体的最大问题之一。

第 25 页给出 HumanEval（164 个 Python 函数，docstring → 补全，隐藏单测）与 CodeContests（竞赛题，statement → 完整程序，编译+测试）。第 26 页定义 pass@k。第 27 页补充非功能需求：运行时效率、延迟/资源、可维护性/安全，并提到 NoFunEval 的 397 个代码编辑任务。

第 28 页讲用测试结果做 RL：采样 → 运行测试 → 奖励 → 更新模型。第 29 页用 1 + … + n 说明推理如何从“逐个加”走到公式 n * (n + 1) // 2，并指出 n * (n - 1) // 2 在 n=1 就错。第 30 页展示 DeepCoder 的 RL 后编码准确率曲线，说明 16K→32K 的响应 token 限制与 64K 评估的设置。

四、智能体式编码：localize–edit–verify

Slides 第 32 页对比单步与智能体：单步是自包含 prompt → 推理+代码 → 评估答案；智能体是 issue + 现有文件 → 搜索 → 编辑 → 跑测试 ↺ → patch + 证据。第 33 页给出核心循环：Localize（零在哪里变成默认值？）→ Edit（只在 None 时给默认）→ Verify（零、默认、调用方回归），失败则修正假设。

第 34–36 页是贯穿例子。问题代码是 return config.get("retries") or 3，因为 0 是 falsy，所以显式零被覆盖。定位用 rg 'retries'，复现用单测；编辑改为：

value = config.get("retries")
return 3 if value is None else value

验证跑四个用例：missing→3、None→3、zero→0、positive→2，全部通过。

第 37 页提到一个替代方案 Agentless：把流程固定为定位文件、类/函数、行，再用单步模型生成补丁。讲者说它曾在一段时间内比智能体更有效，因为模型被大量训练做单步解题，而不是工具调用与验证循环。

4.1 工具集：shell-only 够不够？

Slides 第 39 页给出 mini-SWE-agent：只给一个 shell 工具，读写用 rg/cat/find，编辑用 sed/Python/patch，执行用 pytest/build。第 40 页展示用 sed 替换文本的例子。但第 41 页指出问题：整文件写入简单但重复未改代码；search/replace 紧凑但需要无歧义匹配。第 42 页给出 unified diff 和 Codex 的文件操作 patch 语法。第 43 页引用 Aider 的重构基准（89 个 Python 任务，GPT-4 Turbo）：SEARCH/REPLACE 20%，简化 unified diff 61%。

课堂讨论中，讲者提到有些模型在自己的编辑格式上训练，换到别的 harness 就明显变差，这说明 harness 的格式也是模型能力的一部分。

学习者解释：这解释了一个常见困惑——同一个模型在不同编码工具里表现差异很大。不一定是模型“变笨”，而是它的训练分布与工具的输出格式不匹配。第 54 页的 multi-harness training 正是对此的回应。

4.2 找到相关代码

第 44 页讲从症状出发、沿值追踪；第 45 页给出 LocAgent：沿依赖图检索相关邻域，再用执行验证。

五、智能体的评估与训练

第 47 页讲 SWE-bench 的评估流程：用 GitHub issue 和解决时的仓库状态、相关测试，生成补丁后应用并跑测试；fail→pass 表示修复，pass→pass 表示未破坏行为。第 48 页给出可运行训练任务的构成：起始状态（文件+依赖+测试命令）、issue（请求行为+复现）、检查。第 49 页讲多步修复的 RL：搜索、编辑、测试/结束是训练目标；文件、补丁、测试输出是上下文；终局奖励的问题是“晚期失败无法指出早期哪一步错了”。第 50 页提到 SWE-Gym 的 32B 模型与推理期 learned-verifier 选择。第 51–52 页讲 SWE-smith：变异代码而不变异测试，保留新出现的失败，从而程序化生成修复任务。第 53 页讲多语言（Python/TypeScript/Rust）要先读项目配置再决定构建与测试命令。第 54 页讲 multi-harness training：同一权重在 OpenHands、OpenCode、Codex 等多个 harness 上训练，Nemotron 3 Ultra 的说明是每个任务分布至少两个 harness。

六、前端与更广的开发任务

第 56 页讲前端智能体：定位组件与事件处理器、改逻辑与样式、在浏览器中验证（type→click→检查状态，截图→视觉理解）。第 57 页对比浏览器智能体与 Playwright 脚本。第 58 页给出修复例子：把 value.length 改为 value.trim().length，行为检查（0 记录、有效名仍保存）与外观检查（错误可见）。第 59 页特别强调：issue 里带图片，并不意味着用图像相似度评分。

第 61 页给出从需求到维护的完整视图：需求、兼容性、审查、发布决策、部署/监控、维护。第 62 页引用 2019 年微软调查：编码只占 15%，会议+邮件 25%，调试 14%，跑测试 8%，等等。第 63 页用表格比较修复、库创建、测试生成、CI 修复、演化的 artifact、环境/验证器与视野长度。第 64–69 页分别讲定位（CodeScout，F1 三层）、应用创建（ViBench）、库实现（Commit0、ProgramBench）、维护（SWE-Milestone）、测试生成（SWT-Bench）、CI 修复（Node 16 vs >=18 的 EBADENGINE 例子）。第 70–72 页讲跨任务迁移：Base 32B 与 Hybrid-Gym 32B 在 SWE-bench Verified（7→32.4）、SWT-Bench Verified（9.01→16.86）、Commit0 Lite（8.34→13.45）上的对比。

七、代码行为预测

第 74 页提出 policy 与 world model 的区分：策略选动作，世界模型预测结果。第 75 页给出别名例子：items = [1]; alias = items; alias.append(2); result = len(items)，真实执行 items=[1,2]、result=2，而错误预测会把 alias 当拷贝。第 76 页说明预测何时有用：两个补丁中模型误判 A 失败 B 通过，选错；要用真实执行校验，并同时比较任务成功率与总延迟/token/工具调用。第 77–78 页提到 CWM 与 Code World Models/GIF-MCTS。第 79 页列出开放问题：更好的决策？更低的成本？新的设置？何时执行？

八、把课程收束成一张图

第 81 页的结论把六条线并列：代码模型（预训练、中期训练、执行奖励 RL）；智能体编码（localize/edit/verify 与工具）；智能体训练（可运行任务、修复奖励、多样 harness）；前端（浏览器交互与视觉验证）；更广开发（定位、应用、库、维护、测试、CI）；代码行为预测。这六条线共同回答开头那个问题：从“会写代码”到“能交付软件”，中间隔着的不是单一能力，而是一整套接口、验证与训练数据的工程。

学习者解释：如果让我用一句话概括这节课的立场，我会说——编码智能体的进步不是一个纯模型问题。工具格式、测试质量、环境稳定性、harness 多样性，都会直接改变可测量的成功率。这也解释了为什么课程把大量篇幅给了评估与数据构造，而不是只讲模型架构。

九、待核事项

Slides 第 12、16、17、30、43、62、72 页的图示、公式与完整表格需对照官方 PDF 视觉核验；本文引用的数值来自提供的文本提取，标注为“待核”。
逐字稿为英文自动字幕清洗稿的机器翻译，存在错词与时间戳；本文凡“课堂提到/讲者说”均以逐字稿为依据，未冒充讲者原话，也已由用户审核确认。
涉及论文年份、作者与具体页码的引用，均以 Slides 提取文本中的标注为准；未能从给定资料确认的细节未作补充。

十、官方来源链接

课程主页：https://www.cmu-agents.com/
课程日程与阅读：https://www.cmu-agents.com/#schedule
作业与项目要求：https://www.cmu-agents.com/#assignments
Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
课程视频：https://www.youtube.com/watch?v=1BWeH1oOM7k&list=PLSN0qpDfUvTM&index=6
Lecture 6 slides：https://www.cmu-agents.com/slides/lecture-06-coding-agents.pdf

CMU 11-768 Fall 2026 · Lecture 6 · Domains 1: Coding Agents｜完整中文逐字稿

视频：官方视频
来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。

正文

[00:02.710] 好的。嗯，今天我要讲的是用于编码和软件开发的智能体。所以，想必在座的大多数或全部人都在使用编码智能体。因此，嗯，我几乎不需要描述它们的底层功能是什么。嗯，但我想谈谈我们如何构建编码智能体，以及为了构建编码智能体我们需要具备哪些必要的东西。而且嗯，其实我刚才意识到这张幻灯片的标题可能有点不太准确。我想谈的第一件事是，拥有一个语言模型意味着什么。不是一个智能体，而是一个能够写代码的语言模型。嗯，所以你可以做不同的嗯事情。比较明显的一件事是你用英语说，嗯，我想要你写一个程序，嗯，帮我可视化这些数据。嗯，然后你在提示里给它一个CSV嗯文件，嗯，为这些数据创建一个柱状图。嗯，有多少人经常做这种查询？不一定是柱状图，但比如帮我写这个程序来完成这个任务。可能很多人嗯，你在ChatGPT或编码智能体之类的里面写。嗯，然而，还有很多其他涉及生成代码的任务，不仅仅是一次性的嗯提示，你还可以嗯让它修改代码仓库。所以，你可以让它修改多个文件。嗯，所以这更像是，在这个仓库里嗯修复这个问题，或者做这个改进之类的。然后最后，实际上开发软件不只是写代码、实现修复问题和实现功能，还涉及很多其他东西。所以我想

[02:08.720] 也谈谈其中一些其他东西。当然嗯，就像我们所有其他智能体一样，这将涉及我们嗯编写提示、为智能体提供工具，以及嗯拥有一个能够解决这些任务的好的语言模型。在这节课中我主要关注第二和第三部分。原因是我们在过去两节课中

[02:37.120] 讨论了技能和记忆嗯，以及你知道我们如何整合长上下文。所以对于提示，我想很多内容已经在上一节课中涉及了。所以我们这里的第一个要素是一个具有写代码能力的大型语言模型。

[03:02.319] 所以例子就是写代码。嗯，为了从提示写代码，你嗯本质上需要能够涵盖几件事。嗯，抱歉，澄清一下，这是目前是单步的。这不是智能体式的代码编写。这是单步嗯语言模型。所以为了具备在单步中写代码的能力，你需要能够理解语言嗯，你将要用它来编写的编程语言。另一个非常非常重要的嗯话题是编辑，因为通常我们不会写一次代码然后就扔掉，尽管我们有时确实会那样做。通常我们喜欢迭代地改进，而这并不是语言模型免费就能做到的。你需要把它训练到模型中。然后最后一件事是推理。编码一直是推理模型比较成功的应用之一嗯，就像数学是另一个嗯推理模型的成功应用一样。你嗯可以很容易地验证，或者至少相对容易地验证你做得很好。所以嗯这些就是我想涵盖的三件事。

[04:19.509] 所以当你训练时，最近每个语言模型都是编码模型。嗯，所以没有不会写代码的语言模型。原因是编码是语言模型最有价值和广泛使用的用例之一，但这不是你免费就能得到的。嗯，所以也许在我进入嗯实际内容之前，你们认为训练一个擅长嗯生成代码的模型需要做哪些事情？先不一定要是智能体任务，但比如你们认为可能需要做哪些事情？>>有什么想法吗？是的。>>首先，token必须>>好的。是的。所有嗯token嗯都必须存在。嗯，所以什么是，什么是一些对于代码必须存在但对于非代码模型不一定需要存在的token？[笑声]>>是的。嗯，我会给几个嗯这些token长什么样的例子。嗯还有没有其他你可能需要做的事情？嗯，但比如抱歉只是总结一下。所以嗯你需要确保你的tokenizer有那些会出现在代码中的token。是的。 >>你只需要大量的编码。>>你需要大量的编码数据来训练。是的。嗯，那你从哪里获取呢？>>GitHub。>>GitHub。当然。好的。酷。嗯还有其他想法吗？是的。对于编码有像所有操作系统相关的特殊工具>>对于编码有像GP和操作系统相关的特殊工具。所以是的这嗯非常重要。嗯所以那更多是那可能更多是在智能体方面，一旦你开始创建一个实际上嗯在代码库中移动的智能体。但如果你在写比如bash脚本，你肯定需要了解那些，即使在单步设置中也是如此。所以嗯是的，这些都是很好的观点。

[06:30.319] 嗯所以预训练嗯，你需要在大型语料库上预训练。它需要包含代码，但不能只有代码，因为你也需要它理解英语或你用来嗯查询它的任何语言。另外，嗯你需要确保分词是正确的，我稍后会谈到这一点。嗯下一件事嗯是中期训练。我将把中期训练和监督微调当作嗯是一样的，或者中期训练或持续预训练或监督微调都是这一步的一部分来谈。嗯在这里你需要在代码混合数据上训练。对于编码模型另一个非常重要的事情是能够处理长上下文，因为你想要从整个代码库中引入大量上下文。然后嗯另一件事是嗯RL后训练。当然我们稍后会就我们如何训练这些模型的具体机制嗯更详细地讨论嗯每一个。所以我主要想关注嗯这个嗯任务中编码特定的细节，嗯把如何进行监督微调或RL后训练的总体概述留到后面。所以预训练，这基本上是我们需要从代码和文本中学习，当然在这个阶段我们通常只是优化下一token预测，如方程所示。所以当你训练一个编码模型时，与训练一个只需要回答问题的模型相比，你当然想混合网络文本，因为每个模型从网络文本中学习，但也从技术文本中学习。所以这可能是关于代码如何工作的解释。呃，比如用于提升逻辑推理能力的数学、源代码、文档、呃，很多很多来源。另一件事是，这大概是在你们中任何人开始研究语言模型之前，但曾经有一段时间我们会规范化空白字符。呃，基本上

[08:38.159] 我们会把所有的空格都缩减成单个空格之类的。如果你用 Python 编程，那会非常糟糕，对吧？因为空白字符是有意义的。所以你需要小心这类事情。你还需要小心文件边界以及你如何表达文件边界。你需要能够拥有

[08:58.720] 呃 API，你需要能够保留文本和代码之间的关系。比如，如果你在文本中有一个 markdown 呃块，你想确保它仍然保持在 markdown 块中。当然，你在训练时也会进行评估。通常，你只会针对编码任务在一个留出的损失上做这件事，呃，但你需要设计你的评估，使它覆盖呃你

[09:27.279] 想要覆盖的那些内容。所以当你训练一个基于文本的模型时，选择和清理数据很难。当你训练一个编码模型时，它甚至更难，因为代码在其分布上非常呃独特。所以你需要做的第一件事是按语言和来源进行呃过滤。网上有很多代码完全是自动生成的，所以从你想要在其上训练的角度来看，它就没那么有趣。所以你可以移除那类代码。嗯，你可以移除编码错误的内容，呃以及低价值的重复代码。所以嗯，一个例子是 forks 或 vendored 库，或几乎相同的文件，或复制的解决方案。所以如果你在网上看 GitHub，有数千到呃数万，数十万计的 forks。所以如果你只是下载它们并按原样使用，那么你会有很多呃很多麻烦。哦，抱歉。呃，还要提一下，这个在下面写得很小，但我很多内容都取自一篇叫 Start Coder 的论文，它呃是首批编码器之一，呃是首批关于预训练模型以擅长编码任务的论文之一。所以如果你感兴趣这些所有细节，你可以更仔细地看看那篇论文。而且我相信 Daniel，呃，你参与了这个。所以你可以向 Daniel 询问更多呃更多细节。嗯，另一个有趣的事情是人们呃在某些情况下会考虑，而在其他情况下不会考虑，就是记录许可证、呃来源日期、选择退出，以及呃敏感数据过滤。所以只是举一些例子，如果你在专有许可证下的代码上训练，然后发布模型，然后在这个模型上赚了十亿美元，拥有专有许可证的人会在两年后找上门来说：“嘿，你在我的代码上训练了。给我一亿美元，因为我的代码对你们的模型有贡献。”而你

[11:39.920] 可能不希望这种情况发生。嗯，或者你可能不在乎。但你需要决定 [笑声] 呃，实际上美国版权法对你知道呃你是否应该在乎这一点有点模糊。嗯，呃来源日期，所以如果你不想要一大堆旧代码，你可能只想要新代码。嗯，你可以给人们选择退出的能力。还有呃敏感数据。嗯，人们会上传各种他们不该上传的东西，呃比如他们的呃 API 密钥，这个班上没人会这么做，对吧？呃，请请千万不要把你的 API 密钥推送到仓库，但你知道人们会这么做。嗯，而且我，有一些工具可以让你做这种敏感数据过滤，比如我呃我和 Hugging Face 用的是一个叫

[12:28.399] truffle hog 的，呃，它基本上会识别所有这些呃 API 密钥，然后呃如果你想构建一个训练集或测试集，你会呃想要构建一个训练集或测试集，但这也非常困难，因为有 forks 以及你会遇到的所有其他问题。所以嗯，转到下一件事，也就是 tokens。呃，这是呃你知道某种随着代码变得更加重要而需要演变的东西。但基本上呃，这里我写了字节级 BPE，用于代码和技术文本。但 BPE 基本上只是任何子词分词方法。嗯，我想也许大多数人会熟悉呃 BPE，但你可以呃替换成任何你想要的。嗯，你需要小心的事情是保留呃空白字符以及呃压缩非常频繁出现的东西。所以在我们开始大量处理代码之前，空白字符可能就像是一个 token。所以这意味着如果你生成呃 16 个缩进的你知道 Python 代码，你会有 16 个空格之类的。

[13:44.560] 所以通常呃它看起来会是这样，嗯你知道会被压缩成比如三个空格和一个换行，像是一个单独的 token。所以你开始呃以专门对代码有用的方式进行分词。另一件你可以做但不一定需要做的事情是嗯比如规范化代码，使它处于一个更正常呃更标准呃的设置中但呃你并不一定需要那样做。所以另一件你需要考虑的事情是你想要处理的语言。所以 Starcoder 呃他们采取的方式是对所有语言建模，或者不是对所有语言建模，而是对很多非常流行的语言建模，并且由于拥有更多训练数据的那些语言，最终在特定任务上表现更好。另一个呃你需要注意的其他事情是，你知道比如用户会想要使用哪些语言，以及哪些语言更容易编写，嗯或者被你正在使用的任何 agentic 框架支持。所以例如，如果你让模型做某事，而你没有指定语言，它会使用什么语言？比如你有没有说过“给我写个 app 来做 X”？它会为此使用什么语言？是的，>> 主要是 Python。是的，我想主要是 Python。呃，有没有人见过任何相反的情况？通常是 React。[清嗓子] >> 是的，没错。所以，如果你要比如一个仪表盘，它会用 React。如果你要几乎任何其他东西，它会用 Python。嗯，当然，你可以指定别的东西。嗯，我呃有一个很重的数据处理流水线，呃我最初是用 Python 写的，然后我让模型把它转换成 Rust 来让它更快。所以模型做到了，但之后每次它运行另一个程序时，它又用 Python 运行了。所以 [笑声] 就像我不得不说“用 Rust，用 Rust，用”

[16:02.240] 在 Rust 里用 Rust，每次都是直到最终它做到了。所以它们有这些非常强的偏好。嗯，我认为这些偏好是内建的，原因有两个。一是编写语言的便利性。所以，Python 相对来说简洁且易于编写。流行度，然后还有下游的，嗯，模型 RL 是在什么上面训练的？而且它很可能是用类似 Python 解释器进行 RL 训练的。

[16:27.120] 所以，它更，嗯，偏向那种语言。好。关于预训练有什么问题吗？>> 是的。>> 有没有一种情况，你想要增强你的分词器，指定比如空格为一个单独的 token，这样模型在内部就更容易区分比如语言 token 和代码 token，即使它们是一样的？ >> 是的。是的。所以这是个很好的问题。我的意思是，在这个例子里它确实有 return space 这个面目在里面。但是，嗯，一件有趣的事情是，嗯，你实际上不需要把空白字符和其他任何 token 区别对待。嗯，字节对编码字面上就是用来合并字节的。嗯，所以然而在几乎所有情况下，人们确实把空白字符区别对待。嗯，原因是，比如说模型表现得有点差。它输出一个部分单词是很奇怪的。所以他们想要基本上给模型加偏置，让它不要输出部分单词或部分单词 token。所以，嗯，通常他们会把空白字符和非空白字符分开分词。但是，嗯，也有一些例外情况。>> 是的。>> 是的。你可以获得更高效的表示，特别是对于代码，如果你把跨空格的 token 合并的话。所以在 encoder 论文里我们这么做了，嗯，你可以得到大约 40% 的 token 数，但它也可能以奇怪的方式破坏东西，就像说的那样，比如 import numpy as np 会变成一个 token，但模型就无法在 import numpy as 处停下来，你必须回退并做处理，而且你还必须非常小心，如果你改变分词器，它可能在大规模上出问题，你必须对你的做法稍微聪明一些。关于这个有一些论文，我们可以分享。

[18:37.120] >> 是的。所以，所以基本上复述一下，嗯，你可以，两个主要观点是，如果你实际上允许它不分空白字符与否地随意分词，你可以得到紧凑得多的分词。嗯，但它可能以奇怪的方式破坏东西。所以比如如果你写 import numpy is np，那在你的代码

[19:03.840] 模型里可能变成单个 token，但然后你在做自动补全之类的事情时就会遇到麻烦，因为它太长了。所以，嗯，好。还有别的什么事情吗？好的。所以，嗯，现在我想进入中期训练。而且，嗯，我这里有几个例子。一个来自 Code Llama。另一个来自，嗯，Qwen Coder。所以，这些是，嗯，公开论文，解释了它们如何拿一个之前训练好的非代码模型，并将其适配为代码模型。所以这很好地突出了，嗯，一个非代码模型和一个代码模型之间的区别。所以，嗯，你首先可以做的事情是你可以增加代码曝光量。所以如果你从一个更，嗯，你知道，更多样的数据集开始，你可以增加代码的曝光量。嗯，你也可以通过经验来选择混合比例，嗯，以尝试获得代码方面的收益，并，嗯，保持通用能力，而且，嗯，所以例如在 Code Llama 中，他们拿了 Llama 2，然后为额外的模型做了 5000 亿额外的 token，嗯，而且，嗯，上下文扩展在这里也很重要。所以这里有一个来自 Qwen 2.5 Coder 的经验性，嗯，结果，基本上他们说这是一个代码模型，我们要让它非常擅长代码，所以我们干脆加 100% 的代码，然后之后在 100% 代码上训练。但问题是它在，嗯，其他重要任务上得到了相当差的分数，比如数学或问答。嗯，但如果他们继续训练，不是用 100% 代码，而是也用文本和数学，他们就能够让代码

[20:53.360] 性能有微小的下降，同时保持其他性能。而这显然之所以可能或者重要，是因为，嗯，你知道，如今没有人会只把模型用于代码，嗯。所以另一件事是长上下文处理。而且我已经，嗯，讲了很多关于长上下文处理的内容。所以我就只是在之前的一节课里讲到这个，嗯，但我还是就稍微详细地讲一下。但你基本上想要，嗯，想出一致连贯的序列来，嗯，训练模型。所以为代码模型创建数据的一种非常天真的方式是，我就拿这个仓库，然后把所有文件拼接在一起。但把所有文件拼接在一起，实际上经常会超出，嗯，哪怕是最长上下文模型的上下文。比如你通常最多只训练到 100 万 token，而有很多代码库，或者我会说大多数合理的代码库都超过 100 万 token。嗯，比如 Google 的单一仓库据说有数十亿行那么长，对吧？所以那甚至不在同一个，嗯，数量级上。所以你但会想要有一致的文件理解。所以你可以做类似这样的事，取一个目录里的所有内容，或者你可以解析依赖图，顺着依赖关系，把相互依赖的东西拼接在一起。嗯，诸如此类。嗯，所以 Qwen 2.5 Coder 作为一个例子，它开始训练到 8K，然后它扩展，嗯，到 128K，用 yarn，嗯，就像我几节课

[22:38.880] 前讲过的那样。所以这也很重要，而这种上下文操作让你能够做到那一点，你可以通过像跨文件补全和填充以及短上下文保持这样的任务来验证它，我过一会儿会讲到这些。所以接下来我要讲填充。我觉得其实我不该讲这个，因为 Daniel 才是专家而这主要关于他的论文，但无论如何我还是要讲，人们可以问他问题。嗯，但他有一篇论文叫 encoder。嗯，它是最早的关于用填充目标训练强代码模型的论文之一，这样它们就能够执行不仅仅是代码补全的任务，而且还有，嗯，预测代码内部部分的任务。所以这意味着我们会得到类似这样的东西，这是 positive，而我们要预测它的返回类型，嗯，这只是一个例子。所以问题是如果你只是像普通的自回归模型那样往左边看函数体，你实际上是不知道的，对吧？比如没有关于 is positive 的信息。我的意思是，好吧，也许你可以从 is positive 猜到 bool，

[23:51.520] 但它也可能是一个字符串或其他任何东西，或者是一个枚举。所以然后如果你看这个，你可以呃你知道，看出它实际上是返回类型。那么我们怎么做呢？呃，答案出奇地是——这是个问题还是不是？呃，好吧。所以我认为答案是呃，出奇地，你知道，你可以做出奇地简单的事情来让这个工作，也就是基本上你把一个填充任务变成一个自回归任务。所以你有前缀，你有后缀，然后你在这里填充呃填充这个跨度。所以这个 m0 本质上是你填充的第一个呃，第一个跨度，然后这个是呃，在 m0 之后出现的。呃，然后在推理时你只需要把 m0 放在这里，在前缀和后缀之间，然后你再放一次 m0，然后它生成一个跨度。所以这也可以呃推广到一个或多个洞。所以就像如果你有多个洞，你可以有一个 M0 和一个 M1，然后先生成 M0，然后第二个生成 M1。所以这是一个通用的方式，你可以做任何类型的填充任务。并且这允许你从软件工程的角度做各种有用的事情。所以你可以在一个大文件的中间补全代码。你可以推断类型呃，就像我为返回类型所做的那样。你可以生成注释，重命名变量。所有这些都是填充任务的实例。这里是论文中的一些实验结果。所以基本上有呃从左到右推理，在他们评估的若干任务上，在单行和多行上大约是 48 到 24 呃和 25。呃，如果你做从左到右并生成 10 个候选，然后呃使用呃整个文件的上下文对它们重新排序，你会呃得到更好的结果呃但仍然呃

[26:00.880] 不会达到训练一个模型来做这件事所获得的近乎那样的改进。而且你知道这完全不需要任何重新排序。那只是呃使用呃基于填充的推理，并且呃你可以证明呃你可以用填充来训练，但仍然保持呃标准的代码补全。

[26:22.960] 所以呃，如果你做因果掩码，也就是你基本上呃你知道掩码了从左到右，他们仍然能够保持性能。所以这并不像是两者之间真正的权衡。呃，这里有任何问题或任何后续评论吗？>> 是的。你应用一个像是随机选择，类似于一种长尾分布。所以大多数时候你只做一个，但你可以做到我认为像是 100 个的几率。然后对于每一个，你像是采样，而那些就像是——你只是替换模型 >> 所以模型不预测那个掩码。 >> 好的。所以我为后面的人重复一下，但基本上呃问题是，你如何决定采样多少个跨度以及跨度有多大，答案基本上是你可以使用一个长尾分布，这样呃大多数时候你做一个，然后有时做两个，更少的时候做三个，呃，对于呃输出的长度也类似，你可以大概可以根据你在现实生活中实际看到的填充来建模，呃，如果你想那样做的话。所以 >> 是的，让我就按规模来算。是的，酷。好的，所以这是你可以为代码填充做的一件重要的呃事情的一个例子。呃，但你在从代码中学习方面实际上可以变得非常有创意。

[28:08.480] 所以有若干不同的来源。所以，呃，你可以使用提交差异。所以，如果你有差异之前的版本，呃，如果你有差异之前的文件和一条消息，呃，比如来自一个拉取请求或类似的东西，你就可以生成差异或之后的输出。所以，这将允许你基本上学习好的代码编辑模型。呃，另一件事是从诊断中学习。所以，呃，你可以获得损坏的代码加上一个错误，并生成呃修复。所以，有没有人知道你可以从哪里获得大量这种数据，呃，几乎免费？>> Stack Overflow。>> Stack Overflow 呃，是一个不错的来源。但那不是我想的那个。issues。 >> GitHub issues 呃，也许稍微更呃 >> 是的，GitHub 拉取请求呃是不错的，但呃，比如呃如果你是一个软件开发者呃，你在哪里看到代码出错？也许在 GitHub 里或什么的——测试。是的。所以我特别要讲的是持续集成测试，也就是当你向仓库推送时运行的测试，或者类似这样的东西。所有这些历史都包含在 GitHub 中，而且是公开的，所以你可以看到某个特定版本的代码运行并产生了一个错误，然后一个更晚版本的代码最终通过了并被合并进来。所以这是一种非常丰富的呃这种测试数据的版本，你不需要呃自己生成，因为自己生成会很痛苦。呃，当然，如果你开始运行一个智能体，你可以运行一个智能体并生成数据。呃，这是我稍后会讲到的。所以，呃，是的，测试与此相关。呃，另一件事是执行轨迹。所以，这有点有趣。它有点小众，但呃你知道呃它挺酷的，也就是你可以看到

[30:14.720] 程序实际上是如何运行的，通过在测试时对它进行插桩并运行它。你可以呃，例如，让语言模型生成在程序执行的某个特定点的这个呃调用栈，或者类似的东西。呃，或者生成在程序执行中某个特定点的输出。

[30:34.240] 我将在这次演讲的最后稍微谈一下代码世界模型，它有点依赖于此。酷。呃，有任何问题吗？好。好的。那么接下来我想谈谈评估生成的代码。呃，有几种范式来做这个。呃，其中一种最明显的方式，呃，实际上呃，我们在 2017 年的一篇论文中做过，我猜，是与一个参考解决方案进行比较。所以，呃，这意味着你有一个人类生成的输出，你不是比较执行结果，而是只说，机器生成的代码是否与人类生成的代码相似？所以很久以前，呃，有一个著名的指标叫 blue。还有人实际在用它吗？哦，哇。我实际上 [笑声] 对还有这么多人仍然知道呃仍然知道这个感到惊讶，但呃有一个叫 blue 的指标，它基本上衡量呃输出代码之间的呃 token 和 n-gram 重叠。所以这呃这允许你找到表面上看起来相似的代码，但问题是呃它嗯，如果你有类似大于或等于的情况，这只差一个 token，但从程序行为的角度来看，它实际上非常不同。嗯，另一方面，有些东西用不同的表达式，从 token 重叠的角度看非常不同，但实际上并不不同。所以有几种方法来本质上解决这个问题。一种方法是嗯这个叫做 codeblue 的指标。基本上它做的是，不是从 token 的角度比较表面级别的相似性，而是比较程序之间的抽象语法树。所以程序的一个特征是每个程序都可以被转换成一棵树，这棵树

[32:43.440] 本质上指定了嗯模型你知道执行或者程序将如何根据编程语言的逻辑被执行。所以他们嗯他们基于语法树进行比较，他们也基于语法数据流匹配嗯或者语义数据流匹配进行比较，这就像是哪些变量嗯来自嗯哪个位置以及诸如此类的东西。嗯另一种比较代码相似性的方法是嗯使用基于嵌入的方法。所以这是我们嗯在 2023 年创建的叫做 code bert score 的东西。基本思想是你嗯会取代码，这是一个嗯代码的例子。这是另一个代码的例子，然后你嗯嵌入代码中的每个 token。然后你嗯检查以确保每个 token 在嗯在另一组代码中有一个好的、与之对应的其他 token。这嗯这让你能够判断出所有的代码嗯所有的代码本质上都反映嗯在另一个实现中。

[34:10.790] 所以嗯这些在嗯的情况下是好的，或者实际上抱歉我将在几张幻灯片后讨论权衡。所以嗯也许我就切换到另一种评估代码的范式，这是如今到目前为止更流行的。嗯所以如今评估代码的典型方式是实际运行它。所以你要做的是你有一个候选嗯一个候选输出，然后你有一个一堆测试用例，基于候选输出你运行测试用例，然后嗯你评估，如果它通过了所有测试用例你就得到分数一。如果它没有通过所有测试用例你你就得到分数零。嗯所以这些各有嗯优点和缺点。所以如果你看执行检查嗯执行嗯基于执行的评估的好处是它可以接受不同的正确实现。缺点是创建真正好的单元测试真的很难。对于你会找到的每个著名基准，嗯都有一篇论文抱怨单元测试有多糟糕，嗯说它们要么有假阳性嗯从嗯允许不正确实现的角度来看因为测试不够严格，要么有假阴性，也就是给模型的输入嗯给 agent 的输入不足以通过所有所有单元测试。举个这方面的例子，嗯它可能会说我要一个嗯我要一个我界面右上角的好看的按钮，嗯上面写着了解更多之类的。而那可能很棒，但测试可能也在检查按钮的背景颜色之类的。嗯因为那是一个在测试最开始时不明显的要求，但后来变得明显了。所以这这是像如今嗯训练编码 agent 中最大的问题之一嗯因为我们移动我们

[36:20.480] 正在转向越来越困难的任务，嗯创建既有低假阳性又有低假阴性的测试是非常困难的。嗯所以嗯另一个优点是通过嗯执行检测语义错误是好的，但这需要嗯本质上好的测试依赖在一个隔离的环境中，嗯所以例如嗯如果你需要

[36:57.119] 运行测试，如果你只是使用 Python 标准库就很容易，但如果不是你就需要事先安装所有依赖。嗯所以我们做这个的典型方式是我们有类似 docker 容器的东西嗯它安装所有依赖。嗯但你可能会以为你已经安装了所有依赖，但然后你没有考虑到你正在使用的库的版本。所以版本被升级了测试就死了。嗯然后你你给一个更新的模型更低的分数只是因为这些测试过时了。所以嗯这是我们必须处理的另一个另一个问题。然后最后一件嗯你需要在时间上有一个稳定的执行环境。所以对一个非常大的仓库运行测试实际上可能非常耗时。在很多嗯我们处理的基准中嗯包括我稍后会讲到的 Sweetbench，嗯他们你知道想出了解决这个问题的方法，但它确实是一个嗯一个困难。

[38:04.069] 酷。嗯所以就只是给一些嗯像单步代码生成中流行的基准的想法。嗯 human develop string 和嗯你需要生成一个补全嗯并且有隐藏的单元测试。所以，嗯，这实际上，我想，只是由 OpenAI 的人手动创建的。嗯，所以他们想出了一堆像那种棘手的 Python，嗯，函数，嗯来实现。所以它不一定反映真实世界的使用。嗯，一个稍微难一点的是叫做 code contests 的。基本上，这就像 code forces 嗯风格的嗯更像竞争性编程的东西。还有一堆一堆其他的评估基准，但它们从有点像简单到更复杂的。在他们第一次尝试时就能解决一个 code forces 问题，没有错误并 100% 正确。但那有点像我们在这里要求单步语言模型做的。所以嗯这实际上是一个相当相当困难的任务，它需要大量的推理。嗯所以嗯这是来自嗯一篇 2021 年论文的例子，该论文引入了 humanl，如果你知道现在的 codeex 是什么，你会对这也叫 codeex 感到非常困惑嗯但这是 2021 年为嗯 github 实现的原始代码补全模型，基本上嗯这里他们做的是他们评估嗯

[40:13.200] 嗯基本上模型通过多少，所以嗯你可以看到随着他们扩大模型规模，模型通过了更多嗯在 1 上，而 pass one 的意思是你只能生成一个输出并且它通过了，但报告像 pass k 嗯也很常见，就像如果你生成嗯如果你生成五个输出。嗯，其中任何一个正确吗？所以那就像一个稍微

[40:41.280] 还有一个更宽松的指标，你也可以用它来评估。>> 好的。呃，这里有什么问题吗？>> 代码效率或复杂度是如何被纳入训练的？你读懂了我的心。呃，谢谢你。谢谢你替我介绍下一张幻灯片。所以呃，答案是：对于非智能体任务，编码效率基本上没有纳入训练。呃，他们基本上没有呃没有考虑它是更快的代码还是更慢的代码。他们基本上只考虑它是不是正确的代码。呃，这方面有一些呃例外。呃，其中之一是呃这篇呃论文 no fun，呃，其基本思想是，你不仅仅评估呃像功能需求这样的东西，也就是它是否正确完成了它应该做的事情。你还要做呃呃运行时相关的东西，比如运行时效率呃延迟或资源使用，以及可维护性和安全性，呃呃基于像静态检查这样的东西。所以也有工作在评估呃这些其他方面。对于智能体任务，考虑效率呃更为常见，我稍后会讲这一点。

[42:15.040] 好的。那么呃，对于单步代码生成模型，呃，学习这些模型的一种典型方式是通过基于测试结果的强化学习。所以这相当直接，呃，从强化学习任务的角度来看，就是你只需采样回答，对它们运行测试，然后更新模型。所以这里没有什么呃令人意外的东西。呃，但有一件非常重要的事情是，这让你能够训练呃大型语言模呃训练呃大型推理模型。所以呃，正如我提到的，在你知道的单步中解决 codeforces 相当困难，所以它要求你呃真正仔细地思考呃回答，而推理模型在这方面非常非常有帮助，所以呃，这是呃可以呃可以做到的。所以比如一个担心效率的提示是，如果你在做 codeforces 类的题目，很多竞技编程任务会给你一个执行超时，所以如果你超过了执行超时，基本上呃你就不会得到呃不会得到任何分数，但呃是的，这呃基本上就是你可以在这里使用的方法，呃，你可以证明这个强化学习呃风格的东西在中期训练之后可以呃你知道进一步改进你的呃进一步改进你的结果。这只是来自一篇叫 deep coder 的论文中的一个例子，你可以去查一下。但基本上呃你知道他们在 16k 上下文下训练了一段时间，呃在 32k 上下文下训练了呃更长时间，最终呃报告的是呃像 64k 上下文这样的结果。

[44:16.150] 好的。呃，那么是的，有有有什么问题吗？>> 是的 >> 那是为了给单步模型提供上下文，以便为代码更高效地编写代码吗，还是说这就是原因，因为它是一个语言模型，然后 >> 是的，这是个很好的问题。所以呃，为了重复一下问题，对于包含在代码中的注释，为什么我会在代码中得到你知道的注释，而实际上它并不是必需的，呃你知道，为了呃解决任务。我这里只有假设，因为我并不我不百分之百确定比如说我最常用的那些模型。你知道 GPT 在做什么，或者 Claude 在做什么。呃，所以我想回去呃多读一点这方面的内容，但我的猜测是，呃，第一，这些模型是在现有代码上训练的，而现有代码是有注释的，所以它们会把注释填上。第二，它们被训练来做推理。而且我觉得很可能存在某种隐含的激励，让你在代码中添加更多注释以换取更多 token，这样你在生成代码时就可以思考下一步该做什么。呃，所以我认为可能存在一种隐含的激励，促使模型去写注释。然后呃我现在很确定现在呃，也许六个月前还不是这样，但现在也有某种像非功能性的你知道分类器或者像代码质量模型，他们正在把它纳入其中，呃，它会说呃写注释，写适当数量的注释，像不要太多也不要太少，因为对之前的编码模型的一个很大的呃抱怨就是它们会生成像一百万条没人会读的注释，而且极其冗长。所以，呃，我记得当呃更新版本的 Codex 发布时，他们强调的一件事就是，我们不会给你一大堆注释垃圾，我们会简洁地完成它。所以，是的。

[46:24.230] >> 有没有针对代码质量的特定训练方法，使其具有人类可读性？呃，我的意思是，我觉得显而易见的做法是提出某种你知道能评判代码质量的模型，并在强化学习期间将其加入你的奖励中。我不知道这方面的任何细节。你知道任何吗呃 >> 我觉得当人们做后训练时，他们常常会收集可读性作为一项特定的评判 >> 所以是的，好的 >> 所以呃，为了向后排重复一下，当你在做后训练时呃，常常会有一种像 RLHF 的步骤，你训练一个奖励模型来预测输出有多好，而可读性就是你可以从人类标注者那里收集的东西之一，然后它会通过

[47:13.520] 奖励被间接地呃提升。好的。呃，好的，所以我肯定想讲一下智能体编码。那么呃，这是呃单步编码和智能体编码之间的巨大区别，就是我们呃现在在做迭代式的工具调用，呃像你知道的所有智能体一样，让让我们先谈谈最常见的场景，即在 GitHub 仓库或代码仓库中修复 bug 或实现新功能。当你想做这样的事情时，通常你呃你有三个步骤。呃，第一个是定位。呃，所以这就像，如果我们谈论的是呃抱歉，我想我的贯穿例子现在呃已经是很久以前的了，但基本上我们有一个呃一个修复某些代码的例子，呃以便返回值会是呃会是合适的。所以呃这里的定位本质上就是尝试找到需要修改的地方。呃然后呃编辑就是呃修改需要修改的地方。所以例如，如果我们呃只在值为 none 时设置默认值。呃然后验证就是检查你实际上做得对不对。然后呃通常你会在一个循环中做这件事。所以如果你呃如果你发现验证失败了，你会回去迭代，直到呃一切都通过。所以是的，这里这里是那个贯穿例子。呃所以呃，如果你的工作是找到那个当值为零时呃重试没有正确设置默认值的地方，呃你呃会首先定位实现并复现该行为。嗯，然后你去编辑，所以为了修复这些问题，你会进去编辑代码，然后你通过运行所有测试来验证，如果测试失败，那么你就利用这些失败来指导你在这里执行的下一步。所以这就是智能体式编码。嗯，你基本上把它放在一个循环中，使用我稍后会讲到的工具。

[49:45.599] 嗯，我只想指出有一种替代方式，可以在没有任何迭代式智能体工作流的情况下进行仓库级别的编码。嗯，有一篇论文叫，或者说有一篇论文叫 agent list，实际上是由 Steven Sha 写的，他是这里的一位新助理教授。但嗯，基本上他们所做的是，嗯，不是让智能体能够调用工具来进行定位。它基本上把它拆成了一个定位步骤，在那里嗯，他们会先定位，嗯，他们先预测需要编辑的文件，嗯，然后预测需要编辑的嗯类和函数，然后具体挑选需要编辑的行。然后他们会生成一个补丁，嗯，只使用一个单步语言模型。所以这是一个替代方案。这在一段短时间内，令人沮丧地有效，因为我在那个时候正在构建编码智能体，嗯，我们看到相当多的语言模型，这个工作流实际上比使用编码智能体效果更好。原因是嗯，这些语言模型被非常大量地训练，擅长像单步代码解决这样的事情，而它们

[51:06.319] 并没有被非常大量地训练，擅长像智能体式工具调用和嗯以及验证循环这样的事情。嗯，嗯，所以是的，无论如何，这是你可以解决仓库级别代码生成的一种替代方式，但通常现在每个人嗯每个人都使用智能体。那么你如何将智能体应用到这个上面呢？嗯，一个答案是你给智能体一个单一工具。嗯，你给智能体的那一个工具是嗯运行 bash 命令的能力。所以这回到了，你知道，之前的评论，能够使用 bash 命令真的非常重要。而一个 mini suite 智能体是嗯由创建 SWUB 嗯的人创建的东西，它本质上是一个只使用 bash 命令来解决问题的智能体。所以这允许你进行定位。所以嗯你可以定位的方式是使用嗯像 gp 或 rip gp 这样的东西，它算是 gp 的一种更具表达力的版本，嗯 cat 和 find。嗯，如果你想编辑文件，你可以使用像 said 这样的东西，或者你可以运行一个 python 一次性脚本，你可以使用 patch，然后为了执行检查，你可以使用 piest 或构建命令。所以至少在

[52:26.880] 理论上，这就是你实现一个编码智能体所需要的全部。嗯，有人能想到一些为什么这可能不理想的原因吗？所以，所以答案是只有有限数量的工具，像 GDB 和其他嗯，还有像 GDB 这样的其他工具，但你可以直接通过 bash 命令运行 GDB，对吧？所以，就像只是为了澄清，我不是说我只把这些工具提供给智能体。我说的是你只给智能体提供一个工具，它是一个 bash 工具。>> 是的。很好，很好的观点。所以嗯嗯，只是重申一下这个观点是什么，嗯，这些将是大多数的工具调用，如果这些是大多数的工具调用，如果你把它们封装成一个更高效的工具，智能体应该能够在这里做得很好。所以只是举个例子，如果我想编辑文件，智能体每次都需要思考它是要用 said 还是 python 还是 patch。嗯，而如果你做一个标准的文件编辑工具，专门为智能体做好这件事而定制，嗯，那么它基本上就会主要依赖那个文件编辑工具，它会让文件编辑更正确。所以只是举个例子，这就是你用 said 运行时的样子，嗯，所以嗯那会，你知道，匹配行，匹配行之类的。但如果你需要替换一个有很多反斜杠的东西或者类似的东西呢。那么你就会一直像反斜杠反斜杠。

[54:38.160] 你需要把反斜杠的数量弄得完全正确。嗯，所以就有一些东西使得嗯 bash 不一定是处理嗯文件的最佳可能接口，比如说。所以大多数或者甚至可能是所有嗯你知道的有竞争力的编码智能体所做的最大改变是，它们嗯至少也添加了一个用于文件编辑的工具。所以你可以嗯要么写整个文件，要么嗯你可以在文件内搜索和替换。通常这些的实现方式嗯是从拥有一个工具的角度来看的，这个工具允许你嗯以这个工具嗯使用的固定格式来执行这些操作。所以嗯一种已经被研究过的方式是统一 diff 格式。所以基本上你有嗯像行和需要做的更改。但嗯问题在于嗯如果你让智能体去获取行号，它会一直弄错行，然后像不能正确应用。所以相反，嗯，做这些事情最典型的方式是，嗯，嗯，你有一个东西看起来很像 diff，但它不是，嗯，它不是纯粹的 diff，或者它不指定行号。所以，嗯，这可能是做这件事最常见的方式。它在很多嗯，它在很多嗯编码智能体中被使用。有人能想到做右边那个而不是左边那个的一个缺点吗？任何是的，正是如此。所以如果有多个

[56:39.599] 版本的 retries 等于 3 出现在文件中的任何地方，它就不会匹配，然后嗯你就无法区分哪一个应该被修复。所以基本上编码智能体在这种情况下需要做的是，它需要写更多的行直到它变得无歧义。但做右边那个的好处远远大于那个坏处。所以几乎所有的编码

[57:02.160] 智能体都使用这个。嗯，所以 diff 格式可以产生很大的差异，文件编辑工具格式也可以产生很大的差异。嗯，这是一个来自叫做嗯 Ader 代码编辑基准或代码编辑基准的东西的例子，嗯，基本上它评估了嗯 GPT4 Turbo，那是当时的一个模型，在嗯比较像搜索替换和统一 diff 格式，而 GPT4 Turbo 就是像在统一 diff 格式上要好得多，因为它在训练数据中见过更多那种格式。嗯，如今嗯它稍微没那么突出了，但它仍然是一个因素。就像直到最近 Gemini 嗯在一些测试框架上表现得相当差，因为 Gemini 像 Gemini 是在它自己的编辑格式上训练的，那和其他所有人的都不同，所以

[57:52.720] 它运行得不太好。我想他们在过去几个月左右修复了它，或者类似这样的。但如果没有正确处理，这可能会成为让 agent 良好工作的一个非常非常非常重大的瓶颈。好。呃，这里有什么问题吗？好的。嗯，那么接下来是找到相关代码。所以你从搜索你拥有的任何信息开始。所以呃 rip gpap 是做这件事最呃你知道最流行的工具，但你也可以用呃 grap 或者你知道的专用代码搜索工具来搜索，然后呃你沿着值追踪，呃 agent 就去修复东西。还有人们研究过的关于代码定位的更复杂的方法。所以呃仅举一个例子，这是一篇叫 LOK agent 的论文，LOK agent 所做的是它给 agent 提供了一个特殊工具，让它能够沿着依赖路径追踪。所以如果你呃找到了问题，你找到了 config.py，呃然后一旦你找到了 config.py，如果你想的话，你可以用这个工具自动跳转到其他相关的呃文件。

[59:20.000] 所以呃这很有趣。呃我要说的是，根据我的经验，让这类方法比更简单的东西运行得更好是很难的。我稍后会给出一些实际的例子。所以对于编码 agent 的评估和训练，呃评估编码 agent 最流行的呃基准是 bench。可能很多人都听说过这个。这里大多数人。好的，酷。呃，但不管怎样，因为我假设不是每个人都听说过它。基本上，呃这个基准被创建的方式是，它是从 GitHub issue 创建的，他们取 GitHub issue，呃也就是问题。他们取 GitHub issue 被解决时的代码库状态。所以基本上就是 pull request 之前发生的事情。他们还找到了与 GitHub issue 相关的测试。所以这些测试呃是因为它们是与这个 GitHub issue 一起引入的，这些应该是在测试 GitHub issue 中需要修复的功能。所以然后基于此呃他们你知道有一个 agent 生成一个补丁。呃基于这个补丁他们应用这个补丁，他们应用测试，呃他们生成结果，如果所有相关测试都通过，那么你得到一分的分数。如果所有相关测试都不通过，他们得到零分的分数。呃这里的 harness 实际上意味着与你知道的 agentic harness 不同的东西。它指的是像测试 harness。

[01:00:59.760] 呃所以它只是运行测试。但这里要提到的一件重要事情是，在一些 GitHub 仓库上运行整个测试套件会非常昂贵。所以像其中一些仓库是 Django 和 NumPy 以及其他类似的东西，运行测试 harness 对于你想评估的每个例子来说大概要 5 分钟或 10 分钟，呃甚至可能更多。而所以他们在这里所做的是，他们想出了一个更小的测试子集，这些测试需要通过在才能让这个 issue 被认为已解决。这些分为呃几个类别。第一个是 fail to fail to pass。所以这些是在 PR 之前失败、在 PR 之后通过的东西。所以它们在测试实际上在 PR 中实现的功能。然后还有 pass to pass 的，这些是相关的测试，在呃 PR 被提交之前通过，然后也需要在之后通过，以确保你没有破坏任何东西，没有任何回归。好。所以呃要构建可运行的基准任务，本质上，你会在评估呃评估基准的地方呃作业中做更多这方面的事情，但可运行的任务基本上是你需要一个起始状态。所以你需要呃文件，呃你可能需要依赖项，你需要测试命令，issue 是请求的行为，呃检查项就是测试。

[01:02:35.990] 所以呃如今 agent 的好消息实际上是，评估是呃制造更好 agent 的最大问题之一，因为如果你能生成大量评估，呃你实际上可以在这些评估基准上进行强化学习，训练一个 agent 做好工作。所以呃如果你能经历这个过程生成许多许多这样的输出，你就能实际上呃训练一个模型。你这样做的方式基本上就是你运行模型来呃运行模型生成一个输出并计算奖励，如果呃你知道它做得好还是不好。呃我们关于做这件事的最早的呃论文之一叫做 sugeim。所以呃 sui bench 的问题是，呃 swbench 只是一个评估数据集，或者他们只有为评估数据集配置的呃训练环境。所以对于呃 Swedge，呃我们基本上创建了呃额外的训练环境并在其上训练了一个模型，并展示了一个之前你知道相当差、大约 11% 准确率的模型，即使只有 500 个例子，也能扩展到呃好相当多。呃我不打算过多谈论这个因为我们将在后面的一节课中讨论这个。但我们还展示了，如果你生成大量输出并重新排序它们，呃你可以做推理时扩展并做得呃更好。呃所以在这上面最大的努力，我实际上刚意识到我幻灯片里没有它，呃但值得知道，叫做 SU rebench 和 SU rebench，基本上它是什么，就是它不断创建更多的 SWEBench 环境，呃所以他们一直从呃 2024 年 12 月到 2026 年 6 月呃 6 月 30 日的 PR 创建它们。我实际上最近刚和作者讨论过，我想他们会在 10 月左右更新这个。呃所以这只是呃创建越来越多这些，他们有一个

[01:04:49.440] 训练集和一个开发集。所以这可能是大规模呃大规模公开可用的呃编码呃编码 agent 数据以 SWEBench 格式存在的最好例子。但呃还有更轻量级的方式来创建这些数据。所以呃其中一个问题是，你需要为你创建的每一个呃呃训练例子有一个单独的 pull request。所以

[01:05:21.119] 这是来自 Swissmith 的一个例子，呃它不依赖于呃 pull request，而是依赖于人工注入的 bug。你所做的是准备一个可执行的基线，你变异代码但保留测试。所以基本上呃有多少人听说过像变异测试？有几个人。好的。这实际上是一个来自软件工程或软件工程实践的非常有趣的想法，就是如果你不确定你的测试是否足够好——比如你有单元测试，但你不确定你的单元测试套件是否非常全面。你可以进去对代码做小的变异，这些变异应该会破坏代码，然后你看看测试是通过还是不通过。如果测试仍然通过，那些测试就不是好测试。它们没有覆盖真正的破坏。所以你应该扩充你的测试套件。嗯，所以我认为这项工作受到了那个想法的启发，但又不完全相同，因为他们对仓库中的代码进行变异，然后试图找到之后失败的测试。所以这本质上让你能够做的是，嗯，它允许你创建合成的有缺陷数据，然后你可以用这些数据来训练模型。仅仅通过代码库中现有的测试就可以做到。嗯，这里有一个例子。所以嗯，如果你有代码，其中所有四个现有测试都通过，然后你进去对代码进行变异，嗯，变成类似 return three if not value 而不是 if value is none。然后嗯，其中一个测试现在可能会失败。所以这是一个很好的例子，你本质上能够提出某种东西来说“请修复这个”。所以这件事的好处是它非常轻量，你可以通过这种方式生成几乎无限量的数据。这件事的坏处是这些不一定是自然的

[01:07:40.799] 训练样本，即你真正想要处理的那种。嗯，所以你也可以用这个框架扩展到不同的编程语言。所以有一个叫 multisbench 的东西，还有另一个叫 SWEBench multilingual 的东西，基本上可以扩展到许多不同的编程语言。

[01:08:04.880] 唯一的后勤困难是不同语言之间的约定不同，你需要处理它们。嗯，好消息是现在编码智能体可以非常容易地做到这一点。所以改变约定并没有那么难。嗯，我想谈的最后一件事是多脚手架训练。所以正如我之前提到的，嗯，如果编辑格式非常不同 ——在语言模型期望的和脚手架期望的之间——嗯，大多数语言模型在它们没有训练过的脚手架上表现会更差。所以现在语言模型提供商在多个脚手架上进行训练是非常常见的。所以他们不仅会在单个脚手架上训练，还会在像 open hands 和 open code 以及 codecs 和嗯，你知道的，所有其他脚手架上训练。举一个例子，嗯，你可以阅读 Nvidia 的 Neotron 报告，他们有意在嗯，我想是五个或更多，五个或更多脚手架上进行训练，以提供一个能与许多不同脚手架配合工作的模型。

[01:09:21.189] 好的。嗯，我们剩下的时间不多了。我也许我们就快速过一下一些不同的嗯，一些不同的任务，而不仅仅是直接的问题解决。这些在研究中的主流程度较低，但如果你想构建一个功能性的模型，它们非常重要。所以，嗯，前端开发是其中之一。嗯，有一个前端开发基准测试叫 Swebench Multimodal。嗯，这是一个只专注于修复前端嗯，修复前端任务中的 bug 的基准测试。所以它不是嗯，它不是专注于开发新的前端。它更多是关于问题解决的那种。所以在创建多模态模型时有两个区别。第一个区别是通常你会认为你需要一个多模态模型来嗯，来接收输入并理解输入。所以你需要一个被训练为能够摄取图像的语言模型。嗯，另一件事是你需要验证所以你需要——在浏览器中有几种验证方式。一种方式是编写在浏览器中工作的测试。嗯，另一种方式是查看输出。嗯，所以嗯，现在人们做这件事基本上有两种方式。嗯，一种常见的方式是拿一个基于 GUI 的智能体，把它用作你的编码智能体的一个组件，或者有一个能够同时处理基于 GUI 的任务和编码任务的智能体。下一节课将是关于 GUI 智能体的。所以我不会解释我们如何构建 GUI 智能体。嗯，另一种方式是编写一个测试 GUI 的脚本。比如，Playwright 是一个常见的嗯 GUI 测试库，很多人用它来做这类事情。

[01:11:30.640] 嗯，所以基本上，你知道，如果你有一个问题，你可以用长度为 0 的输入提交，嗯，你就能视觉上验证这发生了。或者你能编写一个嗯 Playwright 脚本，确保在你添加零并回车后会出现一个取消链接的名字。嗯，所以是的，然后 su bench multimodal 是一种评估它的方式。几个月前一个非常令人震惊的结果是有一个流行的模型叫 GLM 5，嗯 GLM，嗯 GLM 5.2，嗯是一个发布的模型。它不是多模态模型，但它在 front-end design arena 上拿了第一名。嗯，尽管如此，这有点动摇了我的假设，关于你需要什么才能

[01:12:22.560] 成为一个好的前端嗯开发嗯编码智能体。但嗯，你知道，我仍然认为多模态行为可能相当重要。嗯，所以有一个著名的开发中内循环和外循环的想法。开发的内循环基本上就是到目前为止我一直在讲的内容。它本质上是做 bug 修复、测试等等的迭代过程。然后开发的外循环是所有不是这些的东西。软件开发中涉及很多不是这些的东西。比如，代码审查就是一个例子。嗯，部署和监控是另一个例子。维护嗯，还有嗯规划和提出需求是另外的例子。理想情况下，我们希望有智能体能够帮助我们做所有这些事情。嗯，但困难的是所有这些东西都更难定义，也更难找到训练数据。嗯，所以这是一个来自微软 2019 年的有趣调查。所以这可能和现在嗯不同，但嗯在 2019 年他们对他们的软件开发人员做了一项调查，并且有人问你的时间有多少花在编码上，答案是呃15%。嗯，这就像是新的呃问题开发和功能新呃功能开发以及问题解决。他们花呃25%的时间在会议和邮件上，呃14%在调试上。呃8%在运行测试上，呃等等等等。所以其实这也是软件开发呃工作中很大的一部分。

[01:14:14.470] 所以嗯我们能做的是我们可以比较呃任务，所以嗯你知道如果我们有修复呃问题修复，嗯我已经讲过那个了，但也有嗯从零开始创建呃东西，嗯为你的代码创建好的测试，修复失败的部署之类的，以及呃在很长一段时间内开发你知道的应用程序。所以第一个呃例子是嗯你可以专门训练做定位。嗯，所以你可以有特别擅长识别你知道的，比如在一个编码任务中呃从哪里开始着手的模型。我们有一种做这个的RL呃呃方法叫做code scout。嗯，如果你想在项目中做和RL相关的东西，这可能对你有意思，因为它就像是编码智能体嗯但它更可控一些，而且用更小的模型更容易做。所以如果那听起来对你有意思，你可以看看我们的code scout论文。嗯，第二个任务是呃从零开始创建应用。呃有一个叫做viench的东西，它评估智能体在总体上做vibe coding方面有多好。嗯这非常呃难做，因为你需要嗯你需要能够评估它们恰当地，所以呃你需要想出好的验证器。另一件事是嗯实现完整的呃那种类似后端库。呃有几个基准测试用于这个。一个较老的叫做commit zero，一个最近流行的叫做program bench。在program bench中本质上你需要做的是你需要逆向工程一个二进制文件并重新实现它从零开始。所以这里你能做的是你可以运行二进制文件并根据输入和输出进行探测，然后你需要创建一个新的实现。这很有意思，因为它非常长视野，而且呃你知道随着较短视野的任务变得越来越容易，它现在就像是新的挑战之一。

[01:16:24.390] 另一个例子呃如果你对基准测试感兴趣的话是sui里程碑基准测试。这里的基本想法是你不想实现单个问题。你想在一个时期中实现许多问题。并确保当你呃逐个实现这些问题时，你不会把代码库弄得一团糟，让它更难维护，呃因为那是代码呃一些编码智能体的一个大问题。所以呃这是一个旨在评估那一点的基准测试。如今另一个非常大的问题是呃生成测试，让你能够验证你修复问题是否做得好。因为我觉得现在有了编码智能体，如果你能写出非常好的测试来验证嗯问题被解决了，你差不多就赢了一半，对吧？你就像是明确规定了需要完成什么任务，然后智能体只是非常擅长在它们上面迭代。所以，SWTB是一个呃能让你做到这一点的基准测试例子，它基于我之前提到的变异测试。最后一个嗯是那种更像运维的呃任务，比如部署，我不认为现在有一个真正好的嗯有真正非常好的基准测试用于这个，而且它有点更难，因为你需要实际的基础设施来运行它，但我有一个例子的参考嗯呃是由Jet Brains创建的，如果你感兴趣可以看看

[01:17:54.719] 嗯，所以是的，我们我们已经到50了呃我们已经过50了。嗯，我我可能就非常非常简短地讲一下这个。所以还有嗯呃一些工作呃由CMU这里的人呃包括我呃和Daniel以及另一个叫做SU playground的以及嗯SU playground和hybrid gym，它们基本上是在嗯自动为编码之外的其他东西创建编码基准测试呃，为软件工程中的其他东西。所以，嗯，抱歉我时间用完了，我没法讲code world models了，但我很乐意，呃，在答疑时间或类似的时候聊聊那个。所以，呃，是的，谢谢大家。

CMU 11-768 Fall 2026 · Lecture 6 · Domains 1: Coding Agents

视频：官方视频
字幕：en-orig · auto captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。

正文

[00:02.710] Okay. Um, today I'll I'll be talking about agents for coding and software development. So, presumably many or mo most or all uh people here are using coding agents. So, um I I barely need to describe what their underlying functionality is. Um but I would like to talk about um how we can build coding agents and what the necessary things we need to have in order to build coding agents are. And um actually I just realized that the title of this slide is uh maybe uh a little bit of a misnomer. The first thing that I'd like to talk about is what it means to have a language model. Not a not an agent, but a language model that's able to write code. Um, and so there's different uh things you can do. The kind of obvious thing is you say in English um I want you to write a program that uh visualizes this data for me. Um and you give it like a CSV uh file in your prompt and uh creates a bar chart uh for this data. Um, how many people do this sort of query like frequently? Doesn't necessarily need to be a bar chart, but like write write this program for me to do this task. Probably a lot of people uh and you write it in like chat GPT or a coding agent or something like that. Um, however, there's a lot of other tasks that involve generating code that's not just like a oneoff like prompt, but you can also do um you can uh have it modify repositories. So, you can have it modify like multiple files. Um, and so this would be something more like in this repository uh fix this issue or make this improvement or something like that. And then finally, actually developing software is not just writing code and implementing fixing issues and implementing features, but there's also a lot of other stuff involved. So I'd

[02:08.720] like to talk about some of that other stuff as well. And of course uh like all of our other agents, this will involve us uh writing prompts, providing tools to the agent, and uh having a good LM that's able to solve these tasks. I'm mostly in this lecture going to focus on the second and third parts of this. And the reason why is because we spent the last two lectures

[02:37.120] talking about like skills and memory uh and you know how we incorporate long context. And so for the prompting I I think many of the things were already covered in the previous lecture. So the first element that we have here is a large language model that has the ability to write code.

[03:02.319] And so the example is writing code. Uh in order to write code from a prompt, you uh essentially need to be able to cover a few things. Um and sorry, just to clarify, this is this is single step at the moment. It's not agentic codew writing. It's singlestep uh language models. And so in order to have the ability to write code in a single step, you need to be able to understand the language uh the programming language that you're going to be writing in. Another very very important uh topic is editing because usually we don't write code once and then throw it away although we do do that sometimes. Usually we like iteratively improve and that is not something that comes for free from language models. you need to train it into the model. And then the final thing is reasoning. And coding has been one of the more successful applications of reasoning models uh just because um like math which is another uh successful application of reasoning models. You uh can easily verify or at least somewhat easily verify that you did a good job of it. So uh these are the the three things that I'd like to cover.

[04:19.509] So when you're training recently every language model is a coding model. Um so there there are no language models that don't know how to write code. And the reason why is because coding is one of the most valuable and like widely used use cases for language models, but it's not something you get for free. Um, so may maybe before I go into the like the actual content, what sorts of things do you think are necessary to train a model that is good for uh generating code? Not necessarily for agentic tasks first, but like what what sorts of things do you think you might need to do? >> Any ideas? Yeah. >> First of all, tokens must >> Okay. Yeah. all the tokens uh must exist. Uh so what what are what are the some tokens that must exist for code that wouldn't necessarily need to exist for non-code models. [laughter] >> Yeah. So um there's uh I'll give a few examples of of what these tokens look like. Um any any other things that you might need to do? Um but like sorry just to to summarize. So uh you need to make sure your tokens have the tokens that are going to appear in code. Yeah. >> You just need like a lot of coding. >> You need a lot of coding data to train on. Yeah. Um and so where where would you get that? >> GitHub. >> GitHub. Sure. Okay. Cool. Um any any other ideas? Yeah. For coding there's like special tools like all the OS related >> for coding there's special tools like GP and OS related tools. So yes this is uh very important. Um so that's more that's maybe more important on the agentic side once you start creating an agent that actually like moves around the codebase. But if you were writing bash scripts for instance, you definitely need to know know about those even as in a single step setting. So um yeah, these these are all great points.

[06:30.319] Uh so pre-training um you need to pre-train on a large corpus. It needs to include code, but it can't be only code because you also need it to understand English or whatever language you're quering in it uh it in. As well, um you need to make sure the tokenization's right, which I'm going to talk about in a bit. Um the next thing uh is mid-training. And I'm going to talk about mid-training and supervised fine-tuning as kind of being the uh the same or mid-training or continual pre-training or supervised fine-tuning is all being part of this step. Um in here you need to train on code mixtures. Another really important thing for coding models is to be able to handle long contexts because you want to pull in lots of context from throughout the codebase. And then uh another thing is uh RL post training. And of course we're going to be talking about uh each of these in more detail later with respect to the specific mechanics of how we train these models. So I want to mostly focus on uh kind of like the coding specific details in this uh task and uh leave the overall overview of how you do supervised fine-tuning or RL postraining for later. So pre-training this is basically we need to learn from code and text and of course we're just optimizing next token prediction at this stage typically as is shown in the equation. So when you're training a coding model in contrast to training a model that only needs to answer questions you want to mix web text of course because every model learns from web text but also technical pros. So this could be explanations of how code works. Uh things like math for improving logical reasoning abilities, source code, documentation, uh many many sources. Another thing, this is before probably any of you started working on language models, but there was a time when we normalized whites space. Uh so basically

[08:38.159] we would take all of the spaces and like drop them down to a single space or something like that. That would be really bad if you're programming in Python, right? Because whites space is meaning bearing. So you need to be careful about these sorts of things. You also need to be careful about file boundaries and how you express file boundaries. You need to be able to have

[08:58.720] uh APIs and you need to be able to preserve text and code relationships. Like if you have a markdown uh block within text, you want to make sure it remains in a markdown block. And of course, you will be evaluating while you're doing training. Typically, you'll do this just on a held out loss uh on coding tasks, but uh you need to shape your evaluation so that it covers uh the

[09:27.279] sorts of things that you want to cover. So choosing and cleaning data is hard when you're training a textbased model. It's even harder when you're training a coding model because code is very uh unique in its distribution. So the first thing you need to do is uh filter by language and source. There is a lot of code online that's completely autogenerated and so it's like less interesting from the point of view of you wanting to train on it. So you can remove that sort of code. Um you can remove things that are in wrong encodings uh and low value repeated codes. So um one example of this is forks or vendored libraries or nearly identical files or copied solutions. So if you look online on GitHub, there are thousands to uh tens of thousands, hundreds of thousands of forks. And so if you just download them and use them as is, then you'll have lots of uh lots of trouble. Oh, sorry. Uh also mentioning this is tiny down here, but I'm taking a lot of this from a paper called Start Coder, which was uh one of the first coders on uh one of the first papers on pre-training models to be good at coding tasks. So if you're interested in all of these details, you can you can take a closer look at that. And I believe Daniel uh you were involved in this. So you can ask Daniel for more uh more details. Um another interesting thing that people uh think about in some cases don't think about things in other cases is recording licenses and uh source dates and optouts and uh sensitive data filtering. So just to give some examples, if you train on code that's in a proprietary license and then release the model and then make a billion dollars on this model, the people who have the proprietary license are going to come around two years later and say, "Hey, you trained on my code. Give me a hundred million because my code is contributing to your model." And you

[11:39.920] might not want that to happen. Um or you might not care. But you need to decide [laughter] u uh actually US copyright law is a little bit vague about you know uh whe whether you should care about that or not. Um uh source states so if you don't want a whole bunch of old code you might want only new code. Um you can allow people the ability to opt out. also uh sensitive data. Um people upload all kinds of things they should not be uploading uh like their uh API keys which nobody in this class will do, right? Uh please please do not push your API keys to the repository but you know people do this. Um, and I there are tools that allow you to do this sensitive data filtering like I uh I and hugging face use one called

[12:28.399] truffle hog uh which is uh one that basically identifies all these uh API keys and then uh if you want to build a a training set or a test set you'll uh want to build a training set or a test set but this is also very difficult because of forks and all of the other problems you deal with. So um moving on to the next thing which is tokens. Uh this is something that uh you know kind of needed to evolve as code became more important. But basically uh here I wrote bite level BP on code and pros. But BP is basically just any subword tokenization method. Um I think maybe most people would be familiar with uh with BPE but you can uh substitute in anyone you want. Um the uh thing that you need to be careful about is preserving uh whites space and also uh compressing very frequent things. So up until we started working on code a lot whites space might be like one token. So that would mean that you would if you're generating uh 16 indented you know Python code you would have 16 spaces or something like that.

[13:44.560] And so typically uh what it looks like is it would be um you know compressed into like three spaces and a a return in like a single token. So you start uh tokenizing specifically in a way that's useful for code. Another thing you can do but don't necessarily need to do is um like normalize the code so it becomes in a more normal uh more standard uh setting but uh you you don't necessarily need to do that. So another thing you need to think about is the languages that you want to handle. And so Starcoder uh they took the approach of modeling all the languages or modeling not all the languages but a lot of very popular languages and kind of necessarily the ones that had more training data ended up being better at particular tasks. Another uh other things that you need to be aware of are you know like what languages will the users want to be using and also what languages are easier to write um or supported by you know whatever agentic framework you're using. So for instance, if you ask a model to do something and like you don't specify the language, what language will it it use? Like have you ever said code me up an app to do X? What language will it use for that? Yeah, >> mostly Python. Yeah, I think mostly Python. What uh has anybody seen any other contrary things? typically react. [clears throat] >> Yeah, exactly. So, if you if you ask for like a dashboard, it will do React. If you ask for just about anything else, it will do Python. Um, and of course, you can specify something else. Um, I I had a heavy data processing pipeline uh that I originally wrote in Python and I asked the model to convert it to Rust to make it faster. And so the model did that but then every time it ran another program it ran it again in Python. So [laughter] like I had to say in rust in rust in

[16:02.240] rust in rust every time until finally it did it. So they have these very strong biases. Um I think the biases are built in for two for two reasons. One is ease of writing the language. So, Python is like relatively concise and easy to write. Popularity and then also like downstream uh what was the model RL trained on? And it was probably RL trained with like a Python interpreter.

[16:27.120] So, it it's more uh favoring that language. Cool. Any questions about pre-training? >> Yeah. >> Is there a situation where you want to augment your tokenizer specified like space for example? a separate token so that it's easier for the model to internally separate like language tokens and coding tokens even though they're like the same. >> Yeah. Yeah. So this is a great question. I mean like it it does have return space face in here in the example. But um one one interesting thing is um you actually don't need to treat whites space any differently than every other token. um bite pair encoding is like literally meant to merge together bytes. Um and so however in almost all of the cases people do treat whites space differently. Um and so the reason why is because like let's say the model is performing a little bit poorly. It's very weird for it to put out a partial word. And so they want to like essentially bias the model so that it's not outputting like partial words or partial word tokens. And so um typically they will tokenize whites space separately and non-white space separately. But um there are also kind of exceptions to that. >> Yeah. >> Yeah. You can get a more efficient like representation in particular for code if you like tokens across space. So in the encoder paper we did that um you can get like 40% tokens but it can also break things in weird ways like said so like import nump numpy as np would be one token but like the model wouldn't be able to stop like import numpy as you have to like back off and do like and you also have to be really careful if you change your tokenizer can break at scale and you have to be a little bit smart about how you do it. There's some papers on that we can share.

[18:37.120] >> Yeah. So, so just to um repeat basically uh you can the two of the main points were like you can get a much more compact tokenization if you actually allow it to just tokenize indiscriminately regardless of whites space or not. Um but it can break things in weird ways. So like if you write import numpy is np that could become like a single token in your your code

[19:03.840] model but then you'd have trouble like doing autocomplete or something like that because it's too long. So, um, cool. Any any other things? Okay. So, uh, now I'd like to go into mid training. And, uh, I have a few examples here. One is from Code Lama. Another one is from, uh, Quen Coder. And so, these are, uh, public papers that explain how they take a previously trained non-code model and adapt it to being a coding model. So this kind of is good for highlighting the uh the difference between a co a non-coding model and a coding model. So um the first thing that you can do is you can increase code exposure. So if you start off with a much uh you know more heterogeneous data set you can increase the exposure of code. Um you could also choose the mixture empirically uh to try to get code gains and uh maintain general capability and um so for example in code llama they took llama 2 and then did 500 billion additional tokens uh for the the additional models and uh there context extension is also uh important here. So here's one empirical uh result from Quen 2.5 coder where basically they said this is a coding model and we're going to make it really good at code and so we'll just add 100% code and and train on 100% code afterwards. But the problem was it got uh quite poor scores on uh other important tasks like math or question answering. Um but if they continued training with not 100% code but but also text and math they were able to have like a tiny drop in the coding

[20:53.360] performance while maintaining other performance. And this is obviously possible or important because uh you know like nobody's going to use a model like just for coding uh nowadays. So another thing is long context handling. And I already kind of talked about long context handling a lot. So I'm just going to go into this uh in a previous class, but I'm just going to go into this in a little bit of detail. But you basically want to uh come up with coherent sequences of uh to train models on. So a very naive way to create data for a coding model is to say I'm just going to take this repo and I'm going to concatenate all the files together. But concatenating all the files together actually very often will exceed the long context of even the longest context models. Like you only train up to typically 1 million tokens and there's many code bases or like I'd say most reasonable code bases are more than 1 million tokens. um like the Google monor repo is allegedly like billions of lines long, right? So that's not even on the same uh order of magnitude. And so but you would like to have a coherent file understanding. So you can do things like take everything in a directory or you can parse the dependency graph and follow the dependencies and concatenate together things that depend on each other. Uh that sort of thing. Um so Quen 2.5 coder just as an example it started training into 8K and then it extended uh up to 128K with yarn uh like like I talked about a couple classes

[22:38.880] ago. So this is also really important and this sort of context manipulation allows you to do that and you can validate it through tasks like cross file completion and infilling and short context retention which I'm going to talk about in a bit. So next I'll talk about infilling. I feel like actually I shouldn't talk about this because Daniel's the expert and this is mostly about his paper but I I will talk about it anyway and people can ask him questions. Um but he had a paper called encoder. Um it was one of the first papers on training strong coding models with an infilling objective so that they would be able to perform tasks that are not just code completion but also um kind of predicting the uh inner part of code. And so what this means is we get something like this is positive and we want to predict its return type um would be just one example of this. And so the problem is if you just look to the body to the the left in a normal auto reggressive model, you actually have no idea, right? Like there's no information about is positive. I mean, well, maybe you could guess bool from is positive,

[23:51.520] but it it could also be a string or whatever else or an enum. And so then if you look at this, you could uh you know, tell that it's actually the return type. So how do we do this? Uh the answer is surprisingly is that a question or no? Um okay. So I think the answer is uh surprisingly you know you can do surprisingly simple things to make this work which is basically you turn an infilling task into an auto reggressive task. And so you have like the prefix, you have the suffix and then you infill uh infill the span here. And so this m0 is essentially the the first uh like the first span that you infill and then this is uh that comes after m0 here. Um and so then at inference time you just do you put m0 here between the prefix and the suffix and then you put m0 again and then it generates a span. So this also um generalizes to one or more holes. So like if you have multiple holes, you can have an M0 and an M1 and then generate M0 at first and then M1 second. So this is a a general way that you can do any sort of infilling task. And this allows you to do all kinds of useful things uh from the point of view of software engineering. So you can complete code in the middle of a large file. You can infer types uh like I did for return types. You can generate comments, rename variables. All of these things are like instances of an infilling task. And here are some of the experimental results from the paper. So there are uh basically left to right inference on uh a number of tasks that they evaluated on single lines and multiple lines was like 48 to 24 uh and 25. Um if you did left to right with 10 candidates and reranked them uh using uh the context from the whole file, you would uh get do better uh but still uh

[26:00.880] would not achieve nearly the improvement of training a model to do this. And this is you know not requiring any reranking at all. That's just uh using the um the infilling based inference and uh you can demonstrate that uh you can train with infilling but still maintain uh standard code completion.

[26:22.960] And so uh if you did causal masking where you basically uh you know masked the the left to right they still were able to maintain performance. So it's not like really a trade-off between the two. Um, any any questions here or any follow-up comments? >> Yeah. You apply like a a random selection of like kind of like a long tail distribution. So the majority of the time you're just doing one, but you could do up to I think like 100 chance. And then for each one of those you like sample and those are like you just replace the model >> and so the model doesn't predict the mass. >> Okay. So I I'll repeat that for the back but basically um the question was how do you decide how many spans and how big the spans are to sample and the answer was basically you use a longtail distribution so that uh you're doing one most of the time but then you do two some of the time three less of the time uh and and similarly for the uh the length of the outputs and you could probably model that after the infills that you actually see in real life uh if like you wanted to do that. So >> yeah let me just count on scale. Yeah, cool. Okay, so this is one example of an important uh thing that you can do uh for code in filling. Um but you can actually become very creative in learning from code.

[28:08.480] So there's a bunch of different sources. So, um, you can use commit diffs. And so, if you have the before, uh, if you have the files before the diff and a message, uh, like from a a pull request or something like that, you can then generate the diff or the after output. So, this would allow you to basically learn good code editing models. Um, another thing is learning from diagnostics. So, um, you can get broken code plus an error and generate, uh, repairs. So, does anyone have an idea of where you could get like a ton of this data, uh, nearly for free? >> Stack overflow. >> Stack Overflow uh, is is one good one. That's not the one I was thinking of, though. issues. >> GitHub issues uh maybe a little bit more uh >> yeah, GitHub pull requests uh is is good, but uh like uh if you're a software developer um where where do you see code break? Maybe in in GitHub or something tests. Yeah. So I I specifically was going to talk about continuous integration tests, which are the tests that run when you make a push to a repository or something like this. All of this history is included in GitHub and it's public and so you can see that a particular version of the code ran and generated an error and then a later version of the code eventually passed and got merged in. So this is a very rich uh version of this sort of test data that you don't need to um like generate yourself because generating it yourself would be would be painful. Um, of course, if you start running an agent, you can run an agent and generate the data. Uh, which is what I'll be talking about later. So, um, yeah, the tests is related to that. Um, another thing is execution traces. And so, this is kind of interesting. It's a little bit niche, but there uh you know it it's pretty cool, which is like you can see

[30:14.720] how the program actually acts by instrumenting it and running it at test time. And you could um for example ask the language model to generate this the stack at a particular point in in program execution or something like that. Um or generate the outputs at a particular point in program execution.

[30:34.240] And I'm going to talk a little bit about code world models at the very end of this talk which kind of relies on this. Cool. Um any any questions? Good. Okay. So next I'd like to talk about evaluating generated code. Um there's a few paradigms to do this. Um, one of the most obvious ways to do this, um, which actually, uh, we we did in a paper in 2017, I guess, is comparing with a reference solution. And so, um, what this means is you have a human generated output and you compare not the execution results, but you just say, is the machine generated code similar to the human generated code? And so a long time ago, uh, there was a famous metric called blue. Has anybody actually used this anymore? Oh, wow. I'm actually [laughter] surprised by how many people still know uh still know of this, but um there's a metric called blue, which is basically measuring uh token and engram overlap uh between the uh the output code. And so this is uh this would allow you to find code that looks similar on the surface, but the problem with this is that uh it um if you have like more than or equal, this is only one token different, but it's actually very different from the point of view of the behavior of the program. Um, and on the other hand, there's uh things with different expressions that look very different from the point of view of token overlap, but actually are uh are not different. So there's been a few ways to um essentially account for this. So one way of doing it is uh this metric called codeblue. And basically what it does is instead of comparing the surface level similarity uh from the point of view of tokens it compares the abstract syntax tree uh between the uh programs. So one feature of programs is every program can be turned into a tree uh that

[32:43.440] essentially specifies how uh the model you know exe or how the program will be executed uh according to the logic of the programming language. And so they um they compared based on the syntax tree and they also compared based on a syntactic data flow match um or semantic data flow match which is like which variables are uh kind of come from uh which place and and stuff like that. Um another way that you can compare code similarity is uh using embedding based methods. And so this is um something that we uh created in 2023 called code bert score. And the basic idea is that you uh would take code and this would be one example of uh code. This would be another example of code and you uh embed each token in the code. And then you uh check to make sure that each token has a good other token that corresponds to it in the uh in the other set of code. And this uh this allows you to tell like that all of the code is uh all of the code is essentially reflected um in the other implementation.

[34:10.790] So um these are good in the case um or actually sorry I'm going to talk about the trade-offs in a few slides. So um may maybe I'll just switch to the other paradigm of evaluating code and this is by far more popular nowadays. Um so nowadays the typical way of evaluating code is by actually running it. And so what you do is you have a candidate um a candidate output and then you have a bunch of test cases and based on the candidate output you run the test cases and um you evaluate and if it passes all the test cases you get a score of one. If it doesn't pass all the test cases you you get a score of zero. Um so each of these have uh benefits and disadvantages. So if you look at the execution checks um the benefit of execution uh executionbased evaluation is that this can accept different correct implementations. The disadvantage is this is really hard to create really good unit tests. And for every famous benchmark that you will find, um there is a paper complaining about how bad the unit tests are and uh saying that they either have false positives uh from the point of view of um allowing incorrect implementations because the tests are not rigorous enough or false negatives which is that the input to the model is um the input given to the agent is not enough to pass all all of the unit tests. And just to give an example of this, um it might say I want a uh I want a nice button on the top right of my interface that uh says learn more or something like that. And that might be great, but the test might also be checking the background color of the the button or something. Um because that's a requirement that wasn't obvious right at the very beginning of the test, but became obvious later. So th this is like the problem uh it's like one of the biggest problems in uh training coding agents nowadays uh because we move we're

[36:20.480] moving to more and more difficult tasks and um creating tests that both have low false positives and low false negatives is is very hard. Um so the uh the advantage another advantage is detecting semantic errors through uh execution is good but this needs um a essentially good tests dependencies in an isolated environment and uh so for instance um if you need to

[36:57.119] run the tests it's easy if you're just using the Python standard library but if not you need to install all the dependencies beforehand. Um and so the typical way we do this is we have something like a docker container uh which installs all of the dependencies. Um but you might think you have installed all the dependencies but then you didn't think about the versions of the libraries that you're using. And so the versions get upgraded the tests die. um and uh you're you're giving a newer model a lower score just because the tests are like out of date. So uh this is another another problem we have to deal with. And then the final thing is um you need a stable execution environment in time. And so running the tests for a very big repository can actually be very time consuming. And in a lot of the benchmarks uh that we deal with uh including Sweetbench which I'll talk about in a moment um they you know have come up with ways to fix this but it it's definitely a uh a difficulty.

[38:04.069] Cool. Um so ju just to give some ideas of benchmarks that are popular in uh like singlestep code generation. Um human develop string and um you need to generate a completion uh and there are hidden unit tests. So, uh, this was actually, I think, just manually created by people at OpenAI. Um, so they came up with a bunch of like kind of tricky Python, uh, functions, uh, to implement. And so it's not necessarily reflective of real world usage. Um, one that's a little bit harder is one called code contests. And basically, this is like code forces uh style uh like more competitive programming things. And there's a bunch of a bunch of other evaluation benchmarks, but they go from kind of like simple to to more complex ones. on the fir on their first attempt can solve a code forces problem with no mistakes and get it 100% correct. But that's kind of what we're asking singlestep language models to do here. And so uh this is actually a pretty pretty difficult task and it requires extensive reasoning. Um so uh this is an example of something from uh a paper from 2021 that introduced humanl and everybody if you know what codeex is nowadays you'll be very confused by the fact that this is also called codeex um but this was the original code completion model that was implemented for uh github in 2021 and basically Um here what they're doing is they are evaluating um

[40:13.200] how uh how much basically the model passes and so uh you can see that as they scaled up the model the model passed more um at one and what pass one means is you get to generate a single output and that passes but it's also common to report like pass k uh which is like if you generate uh if you generate five outputs. Um, is any one of those correct? And so that's like a slightly

[40:41.280] more lenient metric that you can evaluate as well. >> Cool. Uh, any questions here? >> How is code efficiency or complexity factored into training? You read my mind. Uh, thank you. Thank you for introducing my next slide for me. So um the answer is coding efficiency is for non-agentic tasks was mostly not factored into training. Um they mostly did not uh did not consider whether it was faster code or slower code. They mostly just considered whether it was right code. Um there are a few uh exceptions to this. Uh one of them is uh this uh paper no fun of um and the uh the existing um basically the idea here is that you don't just evaluate um like functional requirements which are like that it does the thing it's supposed to do correctly. But you also do um uh runtime things like runtime efficiency uh latency or resource use and maintainability and security um uh based on things like static checks. So there is work to evaluate uh this other stuff as well. For agentic tasks it's uh more common to consider efficiency and I I'll talk about that in a bit.

[42:15.040] Cool. Okay. So um for singlestep code generation models uh a typical way to learn the models is through RL over the test results. So this is pretty straightforward uh from the point of view of RL tasks which is you just sample responses you run the tests over them and then you update the model. So nothing nothing too uh surprising here. Um but one really important thing is this allows you to train um large language mo uh train uh large reasoning models. So uh as I mentioned solving code forces in you know a single step is quite difficult and so it requires uh you to really think carefully about the uh response and reasoning models are very very helpful in doing this and so um this is uh possible uh possible to do. So like one hint of worrying about efficiency is like if you're doing a code forces thing a lot of competitive coding tasks give you a execution timeout and so like if you go over the execution timeout basically um you won't get uh you won't get any score for it but um yeah that that's uh basically the method that you can use here and um you you can show that this reinforcement learning uh style thing after mid-training can uh you know further improve your uh further improve your results. This is just one example from a paper called deep coder that you can look up. But basically um you know they trained for a while at 16k context uh trained uh for longer at 32k context and eventually um reported on uh like 64k context here.

[44:16.150] Cool. Um so yeah any any questions? >> Yeah >> is that to provide the single step model context to like code more efficiently for code or is that the reason because it was language model and then >> yeah that's a a great question. So um to repeat the question for the comments that are included in code why do I get you know comments in code when it's actually not necessary um you know to like solve the task. I only have hypothesis here because I I don't I'm not 100% sure about like the models that I use the most. You know what what GPD is doing or what what Claude is doing. Um, so I'd like to go back and and read a little bit more about this, but my guess is that um, number one, the models are trained on existing code and existing code has comments, so they'll fill them out. Number two, they are trained to do reasoning. And I feel like there's probably some implicit incentive to add more comments in your code to buy more tokens so that you can think about what to do next while you're generating the code. Um, so I think there might be an implicit incentive to get models to do comments. And then uh I'm certain now now nowadays uh maybe not like six months ago but now there's also some like non-functional you know classifier or like code quality model that they're incorporating uh that says uh write comments write an appropriate number of comments like not too many not too few because one of the big um complaints about previous coding models was that they would generate like a million comments that nobody would ever read and be super verbose. So, um I remember when uh the more recent version of Codex was released, one of the things they stressed was we will not give you a whole bunch of comment spam and we'll just do it concisely. So, yeah.

[46:24.230] >> Is there specific training methods for code quality that make it human readable? Um, I mean like I think the the obvious way of doing this is to come up with some model that you know judges code quality and add it to your reward during RL. I don't know of any details of this. Do do you know of any uh >> I think like when people do post training they'll often like collect readability as one particular judgement that >> so yeah okay >> so um to to repeat for the back like when you're doing post training um there's often a kind of like RLHF step where you train a reward model that predicts how good the output is and readability is one of the things that you can collect from human annotator ators and then that will be boosted in

[47:13.520] like indirectly through the reward. Cool. Um okay so I definitely like to cover aentic coding. So um this is uh the big difference between uh singlestep coding and agentic coding is that we um now are doing iterative tool calls uh like you know all agents and let let's first talk about the most common setting of uh either fixing bugs or implementing new features in GitHub repositories or in code repositories when you want to do something like this typically you uh you have three steps. Uh the first one is localize. Um so this is like if we are talking about uh sorry I think my my running example is now uh a long time ago but basically we had um we had an example of fixing some code uh so that the return value would be uh would be appropriate. And so um localize essentially here is trying to find the place that needs to be modified. Um then uh edit is uh changing the place that needs to be modified. So for example, if we're um setting a default only when the value is none. Um and then verify is checking that you actually did a good job. And then uh typically you will do this in a loop. So if you uh if you find that verification fails, you'll go back and iterate until uh everything passes. So yeah, here here's the the running example. Um so uh if you uh if your job is to find the place where uh retries is not defaulting appropriately when the value is zero, um you uh will first localize the implementation and reproduce the behavior. Um and so uh then you edit and so to uh fix the issues you will then go in and edit the code and then you verify by running all of the tests and if the tests fail uh then you use the failures to uh guide the next step uh that you're performing here. So this is agentic coding. Um and you basically put this in a loop uh using the tools that I'll talk about in a bit.

[49:45.599] Um I'd just like to point out there's one um alternative way that you could do repo level coding without any iterative agentic workflow. Um there was a paper uh called or there is a paper called agent list which is actually by uh Steven Sha a like new assistant professor here. But um basically what they did is um instead of having the agent be able to call call tools to do localization. It basically split it into a localization step where uh they would first localize um they first predict the files that need to be edited uh then predict the uh classes and functions that need to be edited and then pick specifically the lines that need to be edited. Then they would generate a patch um just using a singlestep language model. So this is an alternative. This was like for a short time like frustratingly effective because I I at this time was building coding agents and uh we saw quite a few language models where this workflow actually worked better than using a coding agent. And the reason why is because um the language models were trained very heavily to be good at like singlestep code resolution and they were

[51:06.319] not trained very heavily to be good at like agentic tool calling and uh and verification loops. Um and uh so yeah anyway this is an alternative way that you could solve repo level code generation but typically now everybody uh everybody uses agents. So how do you apply agents to this? Um one answer is you give the agent a single tool. Um the one tool that you give to an agent is uh the ability to run bash commands. And so this goes back to, you know, the comment before that being able to use bash commands is really important. And a mini suite agent is uh something that was created by uh the people who created SWUB um which is essentially an agent that only uses bash commands to to solve problems. And so this allows you to localize. So um the way you can localize is using uh things like gp or rip gp which is kind of a a more expressive version of gp um cat and find. Um if you want to edit files you can use things like said or you can run a python oneoff script you can use patch and then to execute checks you can use piest or build commands. So at least in

[52:26.880] theory this is all that you need to implement a coding agent. Um, can anyone think of some reasons why this might not be might not be ideal? So, so the answer was there's only a limited number of tools like GDB and other uh and there's other tools like GDB, but you could just run GDB through the bash commands, right? So, like it just just to clarify, I'm not saying you only provide these tools to the agent. I'm saying you provide only one tool to the agent and it's a bash tool. >> Yeah. great uh great point. So um uh just to reiterate what the point was is um these will be most of the tool calls and if these are most of the tool calls if you encapsulate them into a more efficient tool the agent should be able to do a good job here. And so just to give an example if I want to edit files the the agent needs to think every time about whether it uses said or python or patch. Um whereas if you make a standard file editing tool that's specifically tailored to agents doing a good job at it um then it will just mostly rely on that file editing tool and it will get the file edits more correct. And so just to give an example this is what it would look like if you were running with said um and so uh that will you know match the line match the lines and stuff like this. But what if you needed to replace a thing that had lots of backslashes in it or something like that. Then you would be like backslashbackslash all the time.

[54:38.160] You need to get the number of backslashes exactly correct. Um and so there's just a number of of things that make uh bash not necessarily the best possible interface for um working uh with files for instance. And so the biggest change that most or maybe even all uh you know competitive coding agents use is they uh also add at least one tool for file editing. And so you can um either write a whole file or uh you can search and replace within the file. And typically these are implemented uh from the point of view of like having a tool that allows you to um to perform these operations in a like fixed format uh that the this tool uh uses. So um one one way that has been examined is a unified diff format. So basically you have um like the lines and the change that needs to be made. But um the problem with this is uh if you ask the agent to get the lines, it will mess up the lines all the time and like not apply uh properly. And so instead, uh, the most typical way of doing these things is, um, uh, you have something that looks a lot like a diff, but it's not, uh, it's not purely a diff or it doesn't specify line numbers. So, um, this is probably the most common way of doing it. It's done in a lot of, um, it's done in a lot of, uh, coding agents. Can anyone think of a downside of doing the right one instead of the left one? Any Yeah, exactly. So if there's multiple

[56:39.599] versions of retries equals 3 anywhere in the file, it won't match and then uh you you won't be able to disambiguate which one should be fixed. So basically what the coding agent needs to do in that case is it needs to write more lines until it becomes unambiguous. But the advantage of doing the thing on the right is much larger than that disadvantage. So almost all coding

[57:02.160] agents use this. Um so diff format can make a big difference and file editing tool format can make a big difference. Um this is an example from something called the uh the Ader code edit benchmark or code editing benchmark and uh basically it evaluated uh GPT4 Turbo which was a model at the time on uh comparing like search and replace and unified diff format and GPT4 Turbo was just like a lot better at unified diff format because it had seen more of that in its training data. Um nowadays uh it's a little bit less prominent but it still is a thing. Like until very recently Gemini uh performed quite poorly on a number of harnesses because Gemini like Gemini was trained on its own edit format that was different than everybody else's and so

[57:52.720] it was not working well. I think they fixed it in the past like few months or something like this. But this can be a major major major bottleneck uh to getting agents to work well if it's not done properly. Cool. Uh any questions here? Okay. Um so the next thing is finding relevant code. So you start with uh searching for you know whatever information you have. So uh rip gpap is the most uh you know popular tool for uh for doing this but you can also search with uh grap or you know a dedicated code search tool and uh then you follow the values and uh the agent goes and fixes things. There's also more complex methods that people have examined with respect to code localization. And so uh just to give one example, this is a paper called LOK agent where LOK agent what it did is it provided the agent with a special tool that allowed it to follow dependency paths. So if you um found the issue, you found config.py um and then uh once you found config.py, you could automatically jump with this tool to other related uh files if you wanted to.

[59:20.000] So um this is uh interesting. Uh what I will say is from my experience it's hard to make this work this sort of approach work better uh than the simpler stuff. And I'll I'll give some actual examples of that later. So for coding agent evaluation and training um the most popular uh benchmark for evaluating coding agents is bench. Probably many people have heard of this. Mo most people here. Okay, cool. Um, but anyway, like because I assume that not everybody has heard of it. Basically, the way uh this benchmark was created was it was created from GitHub issues where they take the GitHub issue uh which is the problem. They take the codebase state at the time the GitHub issue was resolved. So, it's basically what happened before the pull request. And they also find the tests that were associated with the GitHub issue. And so the tests are um because they were introduced together with this GitHub issue, these are presumably testing the functionality that needed to be fixed in the GitHub issue. So then based on that um they you know have an agent that generates a patch. um based on the patch they apply the patch they apply the tests and uh they generate results and if all of the relevant tests pass then you get a score of one. If all the relevance tests don't pass they they get a score of zero. Um harness here means actually something different uh than you know what an agentic harness means. It means like the test harness.

[01:00:59.760] Um so it's just running the test. But one important thing uh to mention here is that running the whole test suite on some of these GitHub repos would be really expensive. So like some of these repos are Django in NumPy and other things like this and running the test harness would be like 5 minutes or 10 minutes for each example that you wanted to evaluate uh possibly even more. And so what they did here is they came up with a smaller subset of tests that need to pass in order for this issue to be considered resolved. And these fall into uh some categories. The first one is failed to failed to pass. And so these are things that failed before the PR and pass after the PR. So they're testing the functionality uh that was actually implemented in the PR. Then there's also pass to pass ones and these are related tests that passed before uh the the PR was filed and then also need to pass after to make sure you didn't break anything and didn't have any regressions. Cool. So um to build runnable benchmark tasks um essentially what and you'll do more of this in the evaluation uh evaluation benchmark place uh assignment but run runnable tasks are basically you need a starting state. So you need uh the files uh you might need the dependencies and you need the test command and the issue is the requested behavior um and the checks are the the tests.

[01:02:35.990] So um the good news about agents nowadays is actually evaluation is one of the biggest problems in uh making better agents because if you can generate lots of evaluations um you can actually perform reinforcement learning on these evaluation benchmarks and train an agent to do a good job. And so um if you can go through this process of generating many many of these outputs, you can actually uh train a model. And the way you do this is basically you just run the model to um run the model to generate an output and calculate the reward if uh you know the it did a good job or not. And um one of our uh one of our first papers on doing this was something called sugeim. So the thing with uh sui bench is that uh swbench is only an evaluation data set or they only have configured uh training environments for the evaluation data set. So for uh Swedge uh we basically created uh additional training environments and trained a model on it and demonstrated that a model that was you know quite poor before with about 11% accuracy with even 500 examples could scale up to being uh quite a bit better. And um I'm not going to talk about this a whole lot yet because we're going to be talking about this in a later class. But we also demonstrated that if you generate lots of outputs and rerank them uh you can do inference time scaling and do uh even better. Um so the biggest effort on this uh which I actually just realized I don't have it in the slides um but is worth knowing is something called SU rebench and SU rebench basically what this is is it's constantly creating more SWEBench environments and uh so they've been they've created them from PRs from uh December 2024 to June uh June 30 2026 And I actually just discussed with the author recently and I think they're going to up update this in October or so. Um so this is just um creating more and more of these and they have a

[01:04:49.440] training set and a development set. So this is probably the best example of a large scale uh large scale publicly available uh coding uh coding agent data in the format of SWEBench. But um there's also more lightweight ways to create this data. So um one of the problems with this is you need a single pull request for each one of these uh uh training examples that you create. And so

[01:05:21.119] this is an example from Swissmith um which is not relying on uh pull requests but it's instead relying on artificially injected bugs. And what you do is you prepare an executable baseline and you mutate the code but keep the tests. And so essentially um how many people have heard of like mutation testing? A few people. Okay. This is this is actually a really interesting idea from uh kind of like software engineering or software engineering practice which is if you are not sure if your tests are good like if you have unit tests but you're not sure if your unit test suite is very comprehensive. You can go in and make small mutations to the code that should break the code and then you see if the tests pass or not. And if the tests still pass those tests are like not good tests. they they aren't covering like real breakages. So you should augment your test suite. Um so I think this work was inspired by that idea, but it's not exactly the same in that um they mutate the code in the repository and then they try to find tests that fail afterwards. And so what this essentially allows you to do is um it allows you to create synthetic buggy data that you can then use to train models. uh just by just with the existing uh tests in in codebase. Um so here's a example. So um if you have code where all four of the existing tests pass, you then go in and mutate the code um uh into something like return three if not value instead of uh if value is none. And then um one of the tests may fail now. And so this is a good example where you essentially uh are able to come up with uh something that says please fix this. So the good thing about this is this is very lightweight and you can generate a nearly infinite amount of data by doing this. The bad thing about this is these are not necessarily like natural uh

[01:07:40.799] training examples that you might want to tackle for real. Um so you can scale out to different uh programming languages with this framework as well. So there's uh something uh called multisbench and another thing called SWEBench multilingual which basically uh scale out to you know a number of different programming languages.

[01:08:04.880] The only logistic difficulty is that the conventions are different between the languages and you need to handle them. Um, the good thing is now coding agents can do this very easily. So it's not that not that hard to change the conventions. Um, a final thing I'd like to talk about is multi-harness training. And so as I mentioned before, um if the edit format is very different between uh what a language model expects and what the harness expects, um most uh most language models will perform worse on the harnesses that they're not trained on. So now it is very common uh for language model providers to train on multiple harnesses. So they'll not just train on, you know, a single harness, but they'll train on like open hands and open code and codecs and um, you know, all of these other harnesses. And to give one example of this, um, you can read the reports of Neotron by Nvidia where they intentionally trained on um, I think five or more uh, five or more harnesses to provide this to provide a model that works with a lot of different harnesses.

[01:09:21.189] Cool. Um, we don't have a whole lot of time left. I'm maybe we'll just go quickly through a number of different um a number of different uh tasks other than just like straight up issue resolution. These are less these are less mainstream in research, but they're very important if you want to build a functional model. So, um front-end development of one of is one of them. Um there's a front-end development benchmark called Swebench Multimodal. Um this is a benchmark that focuses only on fixing bugs in frontend uh fixing bugs in uh front-end tasks. And so it is not um it's not focusing on developing like new frontends. It's more uh kind of on the issue resolution thing. And so there's two differences in the in creating a multimodal model. The first difference is typically you would think that you would need a a multimodal model to do um uh to like take in the input and understand the input. So you would need a language model that was trained to be able to ingest images. Um the other thing is you need to verify and so you need to there's a couple ways to verify in the browser. One way is to write tests uh that work in the browser. Um another way is to look at the the output. Um so uh there's kind of two ways that people do this nowadays. Um, one common way of doing this is to take a guey based agent and use it as a component of your coding agent or have a have an agent that can handle both guey based tasks and coding tasks. Next class is going to be about guey agents. So I'm not going to explain at all about how we build guey agents. Um, another way you can do it is by writing a script that tests the guey. And so for instance, Playright is a common uh guey testing library that a lot of people use to do this sort of thing.

[01:11:30.640] Um and so basically, you know, you might if you have an issue where you're able to submit with a length of zero input, um you would uh then be able to verify that visually that this happens. or you would be able to write a um a playright script that makes sure that after you add zero it enter an unblink name will appear. Um so yeah and then su bench multimodal is is a way to evaluate that one very shocking result from a couple months ago was there's a popular model called GLM 5 uh GLM uh and GLM 5.2 uh was a model that came out. It was not a multimodal model, but it got like number one on front-end design arena. Um, regardless of this, so that kind of shook my assumptions about what you need in order

[01:12:22.560] to be a good front-end uh developing uh code coding agent. But uh you know, I I still think multimodal behavior is probably pretty important. Um, so there's a famous idea of inner loops and outer loops in development. And the inner loop of development is kind of what I've been talking up about up until this point. And it's essentially the uh iterative process of doing bug fixes, testing, and these sorts of things. Then the outer loop of development is everything that's not this. And there's a lot of stuff involved in software development that's not this. So for example, code review is one example. Um deployment and monitoring is another example. Maintenance um and also uh planning and coming up with requirements are are other examples. And ideally we'd like to have agents that can help us with all of these things. Um but the hard thing is all of these things are kind of harder to define and also harder to find training data for. Um so this is an interesting survey from Microsoft in 2019. So this might be different uh than it is now but um in 2019 they did a survey of their software developers and asked how much of your time do you spend coding and the answer was uh 15%. Um and this is like the the new uh issue development and and feature new uh feature development and issue resolution. They spent uh 25% of their time on meetings and email uh 14% debugging. uh 8% running tests uh etc etc. So like actually this is a big part of the software development uh work uh as well.

[01:14:14.470] So um what we can do is we can compare uh tasks and so um you know if we have repair uh issue repair um I already talked about that but there's also um creating uh things from scratch um creating good tests for your code fixing failing deployments or something and uh developing you know apps over a long period of time. And so a first uh example is um you can train specifically on localization. Um so you can have models that are are particularly good at identifying you know like where to um where to start working on a coding task. We have a RL um uh approach to doing this called code scout. Um, this might be interesting to you uh if you want to do an RL related thing for your project because it's like coding agents uh but it's a little bit more tractable and easier to do with smaller models. So if that sounds interesting to you, you can check out our code scout paper. Um, a second task is uh creating apps uh from scratch. Uh there's something called viench which is evaluating agents for how good they are at kind of vibe coding in general. Um this is very uh hard to do because you need to um you need to be able to evaluate them properly and so uh you need to come up with good verifiers. Another thing is um imp implementing full uh kind of like backend libraries. Uh there's a few benchmarks for this. One older one called commit zero and one newly popular one called program bench. In program bench essentially what you need to do is you need to re reverse engineer a binary and reimplement it from scratch. And so here what you can do is you can run the binary and probe based on inputs and outputs and then you need to create a new implementation. And this is interesting because it's very long horizon and uh you know with shorter horizon tasks becoming easier and easier it's now like one of the new challenges.

[01:16:24.390] Another example uh if you're interested in benchmarks is the sui milestone benchmark. And the basic idea here is that you don't want to implement a single issue. You want to implement many issues in kind of an epoch. And make sure that as you um implement the issues one by one, you're not making a huge mess of the codebase and making it harder to maintain uh because that's a big problem with code uh some coding agents. And so uh this is a benchmark that's uh meant to evaluate that. Another really big problem nowadays is uh generating tests that allow you to validate whether you did a good job of fixing an issue. Because I feel like now with coding agents, if you can write really good tests to validate that um the issue is solved, you've kind of solved half of the battle, right? you've like specified what task needs to be done and then agents are just really good at iterating on them. So, SWTB is an example of a benchmark that uh that allows you to do this and it's based on the mutation testing that I mentioned before. A final one is um kind of like more operational uh like tasks like deployment and I don't think there's a really good um there there are really really good benchmarks for this yet and it's kind of harder because you need actual infrastructure to run it but I have a reference for an example um uh that was created by Jet Brains that you can take a look at if you're interested

[01:17:54.719] Um, so yeah, we're we're at 50 uh we're at 50 past. Um, I'll I'll maybe just cover this one uh very very briefly. So there's also some um uh some work uh by people uh here at CMU including me uh and Daniel and for the other one called SU playground and um SU playground and hybrid gym which basically are um automatically creating coding benchmarks for things other than just coding uh for other things in software engineering. So, um, sorry I ran out of time and I'm not going to be able to cover code world models, but I'm happy to, uh, talk about that in our, uh, in office hours or something like that. So, uh, yeah, thank you everyone.

Domains 1: Coding Agents

课程简介 ​

CMU 11-768 Fall 2026 · Lecture 6 · Domains 1: Coding Agents 学习笔记 ​

一、学习目标 ​

二、概念主线 ​

2.1 三个层次（Slides 第 2 页） ​

2.2 单步代码模型的三种能力（Slides 第 5 页） ​

2.3 训练三阶段（Slides 第 6 页） ​

三、关键机制 ​

3.1 预训练：从代码与文本中学习 ​

3.2 中期训练：加代码、加长上下文 ​

3.3 填充（Infilling） ​

3.4 更多从代码中学习的方式（Slides 第 18 页） ​

3.5 评估生成代码 ​

3.6 智能体编码 ​

3.7 评测与训练 ​

3.8 前端开发 ​

3.9 更广泛的软件开发 ​

3.10 代码行为预测与世界模型 ​

四、公式的假设与符号汇总 ​

五、常见误区 ​

六、复习问题 ​

七、给定资料中的官方来源链接 ​

从补全到软件工程：CMU 11-768 Lecture 6 如何拆解 Coding Agents ​

一、这节课到底在回答什么问题 ​

二、单步代码模型：三件事缺一不可 ​

2.1 预训练：代码不是普通文本 ​

2.2 中期训练与长上下文 ​

2.3 Infilling：让模型看右侧 ​

三、评估生成代码：从文本相似到执行 ​

四、智能体式编码：localize–edit–verify ​

4.1 工具集：shell-only 够不够？ ​

4.2 找到相关代码 ​

五、智能体的评估与训练 ​

六、前端与更广的开发任务 ​

七、代码行为预测 ​

八、把课程收束成一张图 ​

九、待核事项 ​

十、官方来源链接 ​

CMU 11-768 Fall 2026 · Lecture 6 · Domains 1: Coding Agents｜完整中文逐字稿 ​

正文 ​

CMU 11-768 Fall 2026 · Lecture 6 · Domains 1: Coding Agents ​