Agent Capabilities 2: Context Management for Long-Context Agents
- 日期
- Tue, Sep 1, 2026
- 课程资料
- 44 项
- 内容产物
- 4 项
课程简介
CMU 11-768 Fall 2026 · Lecture 3 学习笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
Agent Capabilities 2: Context Management for Long-Context Agents
依据：Lecture 3 slides（官方 PDF 文本提取，标注为“待核”）与 Lecture 3 中文逐字稿整理稿。本文为学习笔记，已由用户审核确认；凡提取文本无法确认的图示/公式细节均标注“待核”。
一、学习目标
学完本讲后，你应当能够：
- 解释智能体上下文为何快速增长：从“线性增长的历史 → 平方级的输入/注意力计算”这一基本事实出发，说明长上下文是智能体系统的结构性问题，而非偶发工程问题（Slides 第 2–4 页）。
- 区分长上下文的两个核心挑战：容量（模型能否真正用上证据）与效率（系统能否负担服务成本），并知道各自对应的技术族：架构、位置编码、训练 vs. 注意力、KV 内存、缓存、压缩（Slides 第 5 页）。
- 描述推理服务的阶段与指标：prefill / decode、KV cache、TTFT、TPOT、吞吐、成本，并说明它们为什么决定智能体体验（Slides 第 7–9 页）。
- 说出四类长上下文架构机制：滑动窗口注意力、线性注意力/循环模型、稀疏注意力、KV 压缩，并解释它们在“可访问历史”上的取舍（Slides 第 17–25 页）。
- 理解长上下文训练的三段式路线：短预训练 → 继续训练 → 长适配，以及位置编码（绝对、RoPE、NoPE、位置插值、YaRN）在其中的作用（Slides 第 27–33 页）。
- 掌握 KV/前缀缓存的工程要点：为什么稳定前缀能省钱、哪些操作会“破坏缓存”、缓存感知路由与逐出策略如何影响命中率（Slides 第 38–44 页）。
- 把上下文压缩理解为状态估计问题：用行为保真、表示有界、可恢复性三条约束来设计压缩策略，并知道如何用“继续式评估”衡量它（Slides 第 46–51 页）。
二、概念主线：一条从“增长”到“压缩”的因果链
本讲的主线可以概括为：
增长 → 推理瓶颈 → 架构选择 → 缓存复用 → 压缩兜底
- 增长：智能体循环不断把指令、动作、环境观察塞回上下文。Slides 第 3 页给出一个极简算例：固定前缀 + 每次新增历史，调用 1–5 的输入从 6K 增到 26K，累计从 6K 增到 80K。逐字稿里讲师用更小的数字（1、6、11、16、21、26）讲同一件事：调用次数线性增长，累计输入平方增长。
- 推理瓶颈：prefill 计算密集、延迟随提示长度增长；decode 顺序执行、受内存带宽限制（Slides 第 8 页）。TTFT 对应 prefill，TPOT 对应 decode（Slides 第 9 页）。
- 架构选择：标准注意力是全局两两比较，成本 O(n²)。局部/循环/稀疏/压缩机制本质上都在给“每次比较的数量”设上限，把 O(n²) 降为 O(wn) 或 O(n)（Slides 第 13–25 页）。
- 缓存复用：同一前缀的 KV 不必重复计算。Slides 第 38 页展示 Call 1/2/3 中“可复用前缀不断增长”；逐字稿强调缓存命中率 90–95% 是常见良好水平（此为讲师课堂说法，非官方规格）。
- 压缩兜底：当上下文超过可承受范围，压缩把历史 Hₜ 映射为有界工作状态 ŝₜ，希望未来动作分布近似不变（Slides 第 47 页）。
三、关键机制
3.1 智能体提示的组成（Slides 第 4 页）
Slides 给出 1,500 个 OpenHands 会话的统计：平均每次会话 77,922 tokens。组成比例为：
| 组成部分 | 占比 |
|---|---|
| 系统提示 + 工具描述 | 23% |
| 用户消息 | 9% |
| 推理轨迹 | 9% |
| 模型回复 | 2% |
| 工具调用 | 20% |
| 工具结果 | 37% |
逐字稿补充：系统提示约 15K–18K tokens（因 OpenHands 对工作方式“很有主见”）；推理 token 约为对用户文本输出的五倍。注意：这些数字是逐字稿中的口头说明，Slides 只给出百分比与均值，具体区间标注待核。
3.2 注意力复习（Slides 第 10 页）
符号：X ∈ ℝ^{n×d}，行 zₜ；qᵢ, kⱼ, vⱼ 为 Q, K, V 的行；d_k 为 key 宽度；掩码 Mᵢⱼ = 0（j ≤ i），−∞（否则）。
公式（待核，提取文本可能有排版错乱）：
- Q = XW_Q，K = XW_K，V = XW_V
- A = softmax(QKᵀ/√d_k + M)
- O = AV
假设：因果掩码保证只能看当前及之前 token；softmax 归一化使权重在 0–1 且和为 1。逐字稿指出，正是这个非线性 softmax 让注意力能“聚焦相关上下文”，也是它难以重排求和的原因。
3.3 局部与混合模型（Slides 第 14–16 页）
- 局部建模：只考虑常数个子集，成本 O(n²) → O(wn)，w 为局部窗口。
- 混合模型：N 次局部计算 → 1 次全局计算 → 重复。Slides 第 16 页列出六个模型的局部/全局配比：
| 模型 | 局部/状态计算 | 全局计算 | 比例 |
|---|---|---|---|
| Qwen3.8-Flash-Next | Gated DeltaNet | QSA 稀疏检索 | 36:12 ≈ 3:1 |
| GLM-5.3-Flash | Kimi Delta Attention | DSA + IndexPool | 34:11 ≈ 3:1 |
| Kimi K3 | Kimi Delta Attention | dense Gated MLA | 69:24 ≈ 3:1 |
| Nemotron 3 Ultra | Mamba-2 | dense GQA | 48:12 = 4:1 |
| Inkling-Small | 512-token window | dense GQA | 35:7 = 5:1 |
| DeepSeek-V4-Pro-0813 | 128-token window branch | compressed sparse / dense 交错分支 | — |
逐字稿补充：如今几乎所有广泛流行的模型都是某种混合模型，“很多局部 + 一次全局”是标准做法。
3.4 线性注意力与 Delta 家族（Slides 第 19–23 页）
线性注意力（Slide 20）：把 softmax 归一化替换为双线性分数，从而可重排求和：
- oₜ = Σᵢ softmax(qₜᵀkᵢ/√d_k) vᵢ → 用 qₜᵀkᵢ 替代
- 重排后 oₜ = Sₜqₜ，其中 Sₜ = Σᵢ kᵢvᵢᵀ，递推 Sₜ = Sₜ₋₁ + kₜvₜᵀ
- 新符号：Sₜ ∈ ℝ^{d_k×d_v} 是固定大小的循环记忆。
- 假设/代价：丢掉 softmax 的非线性 → 表达力更弱，不能做 token 间相对比较；但换来 O(n) 更新。
DeltaNet（Slide 21）：先读出当前关联，再只写入预测误差。
- v̂ₜ = Sₜ₋₁ᵀkₜ，eₜ = vₜ − v̂ₜ
- Sₜ = Sₜ₋₁ + βₜkₜeₜᵀ，oₜ = Sₜᵀqₜ
- 新符号：v̂ₜ 是当前记忆预测；eₜ 是误差；βₜ ∈ [0,1] 是修正强度（类似学习率）。
- 直觉：沿 kₜ 方向修正，无关记忆保持不变。
Gated DeltaNet（Slide 22）：先整体衰减，再做同样的定向修正。
- S̃ₜ₋₁ = αₜSₜ₋₁；Sₜ = S̃ₜ₋₁ + βₜkₜ(vₜ − S̃ₜ₋₁ᵀkₜ)ᵀ
- 新符号：αₜ ∈ [0,1] 是标量保留门；αₜ = 1 时退化为 DeltaNet。
- 直觉：每个时间步衰减一点，等价于对遥远过去打折，防止“很远的怪 key”影响当下。
KDA（Kimi Delta Attention）（Slide 23）：把标量衰减换成逐通道衰减。
- Dₜ = Diag(αₜ)，S̃ₜ₋₁ = DₜSₜ₋₁
- Sₜ = (I − βₜkₜkₜᵀ)DₜSₜ₋₁ + βₜkₜvₜᵀ
- 新符号：αₜ ∈ [0,1]^{d_k} 是向量保留门；Dₜ 是其对角矩阵；I 是 d_k×d_k 单位阵。所有通道保留相同 → 退化为 Gated DeltaNet。
- 直觉：不同特征有不同的记忆寿命，有些长期延续、有些短期。
3.5 稀疏注意力（Slides 第 24 页）
两类：固定模式（输入无关，成本可预测，但不能适应 query）与内容相关（query 特定，自适应检索，但选择器出错会漏掉相关 key）。两者都在保留集合上重新归一化。
- 固定：Sᵢ^fixed = {j ≤ i : i−j < w} ∪ (G ∩ [1,i])
- 内容相关：Sᵢ^dyn = TopK_{j≤i} g(qᵢ, kⱼ)
- Aᵢⱼ = softmax(qᵢᵀkⱼ/√d_k)，oᵢ = Σ_{j∈Sᵢ} Aᵢⱼvⱼ
- 新符号：G 是固定全局位置集合；g 是学习的选择器；|Sᵢ^dyn| = K。
逐字稿举例：OpenAI 早期稀疏注意力约每 512 token 做一次稀疏；DeepSeek 用“廉价注意力先选 top-k，再对 top-k 做完整注意力”。课堂问答中讲师说相似度通常用点积而非余弦，是否预归一化“不知道细节”。
3.6 KV 压缩（Slides 第 25 页）
- GQA：减少 key/value 头数（不是 query 头数）。
- MQA：GQA 只有一组的情形。
- MLA：缓存潜在向量，再重建各头的 K、V：
- cₜ^{KV} = W^{DKV}zₜ，kₜ^{(h)} = W^{UK(h)}cₜ^{KV}，vₜ^{(h)} = W^{UV(h)}cₜ^
- 新符号：cₜ^{KV} 是缓存潜在；h 为头索引；W^D 压缩，W^{UK}/W^{UV} 恢复各头 K/V。
- 逐字稿补充：GQA 通常可减少 4× 或 8× 的 KV 内存。
3.7 训练长上下文（Slides 第 27–35 页）
三段式：短预训练（4K–8K，数据充足、成本低）→ 继续训练（32K–128K，长文档、打包序列）→ 长适配（128K+，任务轨迹、合成课程）。
逐字稿强调：真正难的是数据——互联网上 128K 连贯 token 并不多；打包序列有助效率但不利于学习跨文档注意力。
位置编码：
- 绝对位置（Slide 28）：zₜ = E[xₜ] + pₜ；正弦编码 pₜ,2ℓ = sin(t/ω_ℓ)，pₜ,2ℓ+1 = cos(t/ω_ℓ)，ω_ℓ = 10000^{2ℓ/d}。
- RoPE（Slide 29）：q̃ᵢ = R(i)qᵢ，k̃ⱼ = R(j)kⱼ；q̃ᵢᵀk̃ⱼ = qᵢᵀR(j−i)kⱼ，分数只依赖相对位移。V 不变。
- NoPE（Slide 30）：zₜ = E[xₜ]，无位置向量；顺序由因果计算或循环动态推断。逐字稿解释：因果掩码下，位置 2 的“猫”只看前一个 token，位置 5 的“猫”看前四个，因此它们在注意力第一层就已不同，所以自回归 Transformer 可以不用位置编码；非自回归仍需。
- 位置插值（Slide 31）：R(t) → R(t/s)，θ_ℓ → θ_ℓ/s，s = L′/L。让长位置留在预训练相位范围内。
- YaRN（Slide 32）：θ_ℓ′ = (1−γ_ℓ)θ_ℓ/s + γ_ℓθ_ℓ，0 ≤ γ_ℓ ≤ 1；长波长全插值，短波长保留局部相位。注意力温度 τ = 0.1 ln s + 1（Slides 注明该公式在 LLaMA 上拟合，是配方而非通用常数）。
数据混合（Slide 35）：连贯长来源 + 答案依赖远距离证据的任务 + 渐进长度增长。各模型细节（Kimi K3、GLM-5、Nemotron 3 Ultra、DeepSeek V4）见 Slide 35 表格。
上下文并行（Slide 36）：内存 O(nd_k/P)，总工作量 O(n²d_k)，精确而非稀疏。逐字稿补充：超过约 32K 上下文后变得重要，但新机制（如门控 delta）缺少现成 kernel 是现实痛点。
3.8 提示与 KV 缓存（Slides 第 38–44 页）
- 调用内：decode 时输出 token 扩展 KV 缓存。
- 调用间：工具结果作为新输入返回；prefill 后加入可复用前缀。
- 价格（Slide 40，USD/1M tokens，2026-08-30 核价）：例如 Kimi K3 缓存输入 $0.30 / 常规输入 $3.00 / 输出 $15.00；Claude Opus 5 为 $0.50 / $5.00 / $25.00。逐字稿概括：缓存 token 常为未缓存价格的十分之一。
- PagedAttention（Slide 41）：块表翻译，物理块非连续分配。
- RadixAttention（Slide 42）：基数树前缀匹配、复用前缀追加后缀、分割共享前缀、LRU 逐出叶子。
- 缓存感知路由（Slide 43）：score = 可复用前缀 − 队列惩罚；示例 Worker 1 有 18K 匹配、队列 2，Worker 2 有 2K 匹配、队列 0。
- 提示设计（Slide 44）：稳定前缀（指令、工具、示例）；把变化状态追加到后面；避免时间戳/重排 schema 等无谓抖动；度量复用（缓存 token、TTFT、逐出）。
- 逐字稿警告：每一步改系统消息（如附加当前时间）会彻底破坏缓存；中途切换模型也会让你下次回到贵模型时仍付贵价。
3.9 上下文压缩（Slides 第 46–51 页）
例子（Slide 46，务必记住）：
- Before：CI 任务 30 分钟超时 → 检查日志 → 24K 行日志、反复 “address already in use” → 设
--workers=1重跑 → 套件通过，日志存于/tmp/ci.log。 - After（压缩后）：目标：修 CI 超时；原因：端口冲突；已验证：
--workers=1通过；证据：/tmp/ci.log；下一步：用已测试的 worker 设置开 PR。
形式化（Slide 47）：
- p(A_future | Hₜ) ≈ p(A_future | ŝₜ)
- ŝₜ = C(Hₜ; B)，|ŝₜ| ≤ B
- 符号：Hₜ 完整历史；ŝₜ 压缩工作状态；B token 预算；A_future 未来动作；C 压缩器。
- 三条约束：行为保真、表示有界、可恢复性（复制精确锚点、保留源证据指针）。
什么能存活（Slide 48）：KEEP EXACT（锚点：目标、约束；最近尾部：当前尝试、最新结果）；ENCODE（检查点：决策、进度、ID）；EXTERNALIZE（证据存储：24K 行日志 → 3 个失败 + 命令 + 产物路径）；DISCARD（重复、被取代的尝试）。
策略四决策（Slide 49）：触发（token 阈值 / provider 溢出 / 手动）→ 选择（保护开头、压缩较旧区域、保留近期尾部、在合法回合/工具边界切）→ 替换（可读摘要 / 结构化检查点 / 不透明 provider 项 / 全新上下文）→ 恢复（历史不变、重试或重置、保留可搜索历史、定义失败行为）。
重复压缩漂移（Slide 50）：CUDA 12.4 → “Use CUDA 12.x” → “Use recent CUDA”，约束丢失。公式 ŝ_{k+1} = C(ŝ_k ⊕ R_k)，即每次更新继承上一个检查点。
继续式评估（Slide 51）：植入状态（约束、决策、产物）→ 用生产策略压缩 → 继续 → 测量后续决策。指标：状态召回（约束+标识符）、任务成功、效率（token/延迟/成本）、稳定性（重复压缩）。
逐字稿的真实故事：OpenHands 早期压缩策略在 SWE-bench 上分数不降、省了很多 token，但上线后出现“每次压缩就发一个 PR”“压掉对话中间的用户指令”——因为只在单轮对话上评估过。结论：光做基准不够，必须在真实环境测试。
四、误区清单
- “模型标称 1M 上下文，就能用好 1M”——错。Slides 第 11 页用 GPT-4 大海捞针图说明“标称 vs. 有效上下文”有差距；逐字稿说较弱模型在很长上下文上会变差，RULER、HELMET 是常见基准。
- “注意力成本只是平方，所以优化就是少算”——不完整。还要区分 prefill（计算密集、TTFT）与 decode（内存带宽、TPOT），二者瓶颈不同（Slides 第 8–9 页）。
- “线性注意力全面优于 softmax 注意力”——错。丢掉 softmax 后表达力更弱，无法做 token 间相对比较；Delta/门控/KDA 都是在补这个短板（Slides 第 20–23 页）。
- “缓存只是实现细节，与我设计 harness 无关”——错。改系统消息、附加时间戳、重排 schema 都会让缓存归零，直接放大成本（Slides 第 44 页）。
- “压缩就是摘要”——不完整。压缩是状态估计，要满足行为保真、有界、可恢复三条约束，并保留精确锚点与证据指针（Slides 第 47–48 页）。
- “在基准上不掉分，压缩就没问题”——逐字稿的反例证明不然：单轮评估掩盖了多轮场景的失败。
- “打包序列能训练长上下文”——逐字稿指出：打包有助效率，但不利于学习跨文档/长程连贯注意力。
五、复习问题
- 若固定前缀为 P，每次调用新增 h 个 token，写出第 t 次调用的输入长度与累计输入长度，并说明为什么累计是平方级（参考 Slides 第 3 页算例）。
- TTFT 与 TPOT 分别对应推理的哪个阶段？为什么长输入会推高 TTFT，而 decode 更受内存带宽限制？（Slides 第 8–9 页）
- 用自己的话解释：为什么线性注意力可以重排求和？这个重排换来了什么、又牺牲了什么？（Slides 第 20 页）
- DeltaNet 中 v̂ₜ、eₜ、βₜ 各是什么？为什么说它是“只写入预测误差”？αₜ = 1 时 Gated DeltaNet 退化成什么？（Slides 第 21–22 页）
- KDA 相比 Gated DeltaNet 的关键新增是什么？为什么“逐通道衰减”能同时表达长寿命和短寿命上下文？（Slides 第 23 页）
- 固定模式稀疏注意力与内容相关稀疏注意力各自的失效模式是什么？（Slides 第 24 页）
- MLA 缓存的是什么？它和 GQA 在“减少 KV 内存”上的思路有何不同？（Slides 第 25 页）
- 为什么 RoPE 的“只依赖相对位置”不足以保证长度外推？位置插值和 YaRN 各自怎么处理？（Slides 第 29、31、32 页）
- 列出至少三件会破坏前缀缓存的操作，并说明各自为什么破坏（Slides 第 44 页；逐字稿中“不要每步改系统消息/时间戳”的警告）。
- RadixAttention 的五步操作中，“分割”和“逐出”分别解决什么问题？（Slides 第 42 页）
- 用 Slides 第 46 页的 CI 例子，说明压缩后哪些信息被保留、哪些被外置、哪些被丢弃。
- 写出压缩作为状态估计的三条约束及其形式化表达（Slides 第 47 页），并解释“重复压缩漂移”为什么危险（Slides 第 50 页）。
- 继续式评估与普通基准评估有何不同？四个指标分别测什么？（Slides 第 51 页）
六、待核事项
- Slides 第 10、14、18–25、28–32、36、47、50 页的公式与图示，提取文本存在排版错乱（如 Q=XW_Q 被提取为“Q=XW , K=Q XW”），需对照官方 PDF 视觉核验。
- 逐字稿中所有具体数值（如系统提示 15K–18K、缓存命中率 90–95%、Anthropic 5 分钟逐出、Codex 2500 万用户、模型规模 135/35B 等）均为课堂口头说法，非官方书面规格，本文不将其作为事实断言。
- Slides 第 16、33、35、40 页涉及具体模型版本、价格与内部训练细节，标注日期为 2026 年；引用前请以官方 PDF 与对应厂商页面为准。
- 逐字稿中关于“OpenAI 加密压缩”“YaRN 温度公式拟合于 LLaMA”等，以 Slides 第 32 页注记与原始论文为准。
七、官方来源链接（给定资料中的真实链接）
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=AiwCCvFW1uE&list=PLSN0qpDfUvTM&index=3
- Lecture 3 slides：https://www.cmu-agents.com/slides/lecture-03-long-context.pdf
当智能体的历史越来越长：CMU 11-768 Lecture 3 的上下文管理笔记
来源：平台字幕与官方 Slides。内容已由用户审核确认。
一个编码智能体连续工作一整天，会发生什么？它不会突然变笨，但它会开始忘记你半小时前说过的约束，会重复提交同一个功能，会在压缩上下文之后把“不要删除这个文件”抛到脑后。CMU 11-768 Fall 2026 的第三讲讨论的正是这件事：智能体的历史在自然增长，而我们需要一整套从模型架构到缓存策略再到上下文压缩的机制，让它在这个增长过程中仍然可工作、可负担。
本文依据该讲的官方 Slides（Lecture 3 slides）与逐字稿整理稿整理，Slides 页码均在关键论点后标注；逐字稿为机器翻译整理稿，引用其内容时视为“课堂讲述”，不作为官方结论。本文内容已由用户审核确认。
一、问题：线性增长的历史，平方增长的输入
先看一个最朴素的算术。Slides 第 3 页给出了一张表：如果每次调用大约消耗 5K token，加上固定的初始前缀 1K，那么第 1 次调用输入 6K，第 2 次 11K，第 3 次 16K，第 4 次 21K，第 5 次 26K；而累计消耗分别是 6K、17K、33K、54K、80K。也就是说，单次调用的输入随步数线性增长，累计消耗随步数平方增长（Slides 第 3 页）。
课堂上的口头版本用的是 5 个 token 起步的玩具数字，讲的道理一样：如果完全朴素地把整个历史一遍遍塞回去，成本会很快变得不可承受。讲者提到自己做过持续好几天的编程智能体会话（逐字稿 [02:02] 附近）。
那真实智能体的提示到底由什么组成？Slides 第 4 页给了一份对 OpenHands 1500 个会话的统计：平均每会话 77,922 个 token，其中系统提示加工具描述占 23%，用户消息 9%，推理轨迹 9%，模型直接回复 2%，工具调用 20%，工具结果 37%（Slides 第 4 页）。课堂上的补充是：这些会话大多只是解决单个任务，并不是持续多天的长会话；系统提示本身在 15K–18K token 量级，因为 OpenHands 对“人应该怎么做事情”有很多主张（逐字稿 [03:00]–[03:48] 附近）。
这个构成比例值得记住：真正的“对话内容”占比并不大，大头是系统提示、工具调用和工具结果。也就是说，即便用户只说了一句话，智能体的上下文也可能已经很胖。
Slides 第 5 页把长上下文问题拆成两个挑战（Slides 第 5 页）：
- 容量（Capacity）：模型能不能真正用上所需要的证据？这由架构、位置编码和训练共同决定，并且需要评估来度量。
- 效率（Efficiency）：系统是否负担得起服务它的成本？这由注意力、KV 内存、缓存和压缩共同决定，影响成本和延迟。
本文的结构基本沿着这两条线展开：先讲架构如何降低注意力的二次代价，再讲训练长上下文模型难在哪里，然后是 KV 缓存这条工程主线，最后是上下文压缩——也就是当你已经用尽上述手段之后，还能怎么继续。
学习者解释：可以这样理解“容量 vs 效率”的分工：容量问题问的是“模型看不看得见远处的那根针”，效率问题问的是“让它看一遍要花多少钱”。两者常常互相拉扯——把窗口开大有益于容量，但直接推高服务成本。
二、推理的两段式：prefill 与 decode
要谈效率，先要谈推理本身怎么跑。Slides 第 7 页给出服务栈的最小图景：请求进入调度器，调度器负责排队、组批、路由，同时读写 KV 缓存；随后模型先做 prefill，再逐步 decode，最后流式返回（Slides 第 7 页）。
Slides 第 8 页把两个阶段拆得很清楚（Slides 第 8 页）：
- Prefill：处理全部提示位置，计算量密集，延迟随提示长度增长；输出是每个位置上的 K、V。
- Decode：每步只处理一个新位置，是顺序的、受内存带宽限制的，每个输出 token 一步。
Slide 上的描述是：prefill 是 compute-intensive，decode 是 sequential and memory-bandwidth-intensive。这个区分在后面反复出现——缓存省的主要是 prefill，而 decode 的代价与输出长度挂钩。
Slides 第 9 页给了四个服务指标（Slides 第 9 页）：TTFT（arrival → first token）、TPOT（输出 token 之间的间隔）、吞吐（完成的 token 或请求除以墙钟时间）、成本（输入 + 缓存输入 + 输出）。
学习者解释：TTFT 对应的直觉是“我打完字要等多久才开始看到回答”；TPOT 对应“回答吐字有多快”。课堂上提到交互式编程时用户会同时在意这两个，而后台跑通宵任务时可能完全不在意，并且服务商常常会问客户更在意哪个，以便针对性优化（逐字稿 [10:02]–[10:27] 附近）。同一张 slide 的脚注指向 DistServe（OSDI 2024）作为 prefill/decode 执行模型的来源。
三、标准注意力为何不可持续
Slides 第 10 页是注意力复习：Q、K、V 由 X 做线性投影得到，加因果掩码后做 softmax，得到输出（Slides 第 10 页）。Slides 第 13 页点明问题：标准注意力在整个序列上做两两比较，因此代价是二次的（Slides 第 13 页）。
课堂上的口算版本是：如果目标是 100 万 token 的上下文，两两比较数量级在 10¹²（一万亿）次，而每次比较还要乘大向量，单个序列的总量级会到 10²⁰ 上下（逐字稿 [15:07]–[16:01] 附近）。这里的具体量级属于课堂上口头估算，不是 Slide 上的数据。
Slides 第 11 页展示了那张经典的“大海捞针”图（GPT-4 版本，引用 Kamradt 2023），用来提醒：模型宣称支持的上下文长度，和它实际上能有效利用的长度，是两回事（Slides 第 11 页）。课堂上讲者说这个评估“非常奇怪”：拿一堆 Paul Graham 的文章，塞进“旧金山最好的三明治店在哪”这句话，再问模型这个位置（逐字稿 [12:09]–[12:55] 附近）；类似的基准还有 RULER 和 HELMET。
学习者解释：这张图之所以反复被引用，是因为它把一个容易被忽略的区别可视化了：context window 是接口规格，effective context 是行为表现。Slides 第 11 页的标题就是“Advertised vs. effective context”。
四、架构：把二次比较限制成常数上限
Slides 第 14 页给出核心思路：把 O(n²) 变成 O(wn)，其中 w 是局部窗口大小；对常数 w，工作量对 n 是线性的（Slides 第 14 页）。注意 Slide 的措辞：w 不一定非得是“窗口”，它只需要是比较数量的上限。
Slides 第 15 页描述混合模型的结构：若干次便宜的局部状态更新（local），中间插入一次全局计算，从完整历史中检索，然后重复（Slides 第 15 页）。用课堂上的说法：“如今广泛流行的几乎所有模型，都是某种混合模型”（逐字稿 [17:00] 附近）。
Slides 第 16 页列了六个模型的局部/全局配比（Slides 第 16 页）：
| 模型 | 局部/有状态计算 | 全局计算 | 比例 |
|---|---|---|---|
| Qwen3.8-Flash-Next | Gated DeltaNet | QSA 稀疏检索 | 36:12 ≈ 3:1 |
| GLM-5.3-Flash | Kimi Delta Attention | DSA + IndexPool | 34:11 ≈ 3:1 |
| Kimi K3 | Kimi Delta Attention | dense Gated MLA | 69:24 ≈ 3:1 |
| Nemotron 3 Ultra | Mamba-2 | dense GQA | 48:12 = 4:1 |
| Inkling-Small | 512-token window | dense GQA | 35:7 = 5:1 |
| DeepSeek-V4-Pro-0813 | 128-token window 分支 | compressed sparse / dense 交错分支 | （Slide 未给比例） |
（表内数字直接来自 Slides 第 16 页；具体模型的规格细节不在本讲范围内展开。）
Slides 第 17 页总结了四种反复出现的机制：滑动窗口注意力、线性注意力或循环模型、稀疏注意力、KV 压缩（Slides 第 17 页）。下面逐个看。
4.1 滑动窗口注意力
Slides 第 18 页给出的机制非常朴素：沿用同一个注意力公式，只是把可见范围限制为最近的 w 个因果位置，代价从 O(n²) 降到 O(nw)（Slides 第 18 页）。它的明确缺点是：一个局部层无法直接检索到远处的 token。
课堂上补充了一个重要细节：由于层叠效应，第一层只能从相邻块拿信息，但上一层已经携带了更早块的信息，于是窗口在深层“逐渐扩展”（逐字稿 [19:42]–[20:08] 附近）。
4.2 从线性注意力到 KDA
Slides 第 19–20 页串起了一条推导链（Slides 第 19、20 页）。标准形式的输出是先算归一化的因果注意力权重 a_ti，再加权求和 v_i。线性注意力的关键一步是：把 query 归一化的 softmax 权重换成双线性得分，这样求和就可以重新结合，得到一个固定大小的循环记忆 S_t，并可以写成 S_t = S_{t-1} + k_t v_tᵀ 的增量更新，输出变成 o_t = S_tᵀ q_t。
Slide 的注释写得很直白：这样做“让 softmax 可以重排”，但代价是“additive writes · interference is not explicitly erased”——写入是加性的，干扰不会被显式擦除（Slides 第 20 页）。
学习者解释：可以把它理解为把“每次都要翻遍全部历史做对比”换成“维护一个会不断被写入的状态表”。好处是每步更新时间恒定，坏处是这张表不能像 softmax 那样精确地挑出“最相关的那几个”，它只是把所有写过的键值混在一起。
Slides 第 21 页是 DeltaNet：既然加性写入会累积干扰，就改成只写“预测误差”。幻灯片给出 v̂_t = S_{t-1}ᵀ k_t，e_t = v_t − v̂_t，S_t = S_{t-1} + β_t k_t e_tᵀ，其中 β_t ∈ [0,1] 是修正强度（Slides 第 21 页）。Slide 的原话是“correct along k_t · unrelated memory persists”——沿着 k_t 修正，不相关的记忆保持不动。
Slides 第 22 页加入标量遗忘门：先把整个状态按 α_t 衰减，再做同样的定向修正，α_t = 1 时退化为 DeltaNet（Slides 第 22 页）。Slide 上把 α_t 称为 global retention，β_t 称为 targeted edit。
Slides 第 23 页是 Kimi Delta Attention：把标量遗忘换成逐通道的对角阵 D_t = Diag(α_t)，每个键空间特征有自己的记忆寿命；每个通道保留相同程度时退化为 Gated DeltaNet（Slides 第 23 页）。
课堂上讲 DeltaNet 的直觉是：你在减少“预测当前 value”的误差，β 类似于学习率，控制单次更新不要太大；衰减项则是为了防止很久以前的某个奇怪 key 对未来的预测产生过大影响（逐字稿 [25:30]–[29:17] 附近）。这一段的数学细节在课堂上被明确说成“高层次”的，并指向原始论文。
4.3 稀疏注意力
Slides 第 24 页对比了两种稀疏方式（Slides 第 24 页）：
- 固定模式：与输入无关，可见集合 S_i^fixed 是“最近的 w 个位置”加上固定的全局位置集合 G；代价可预测，但模式无法适配查询。脚注指向 Longformer（2020）。
- 内容相关：S_i^dyn = TopK_j≤i g(q_i, k_j)，由学习到的选择器决定，具有自适应性，但选择器出错就会漏掉相关键。脚注指向 DeepSeek Sparse Attention（2025）。
两者都要在保留集合上重新归一化。
课堂上补充了两种实现味道：一种是每 512 个 token 左右做一次稀疏注意力，仍然是二次的但系数小很多；另一种是先用量非常廉价的注意力（比如把向量降投影得很小）选出 top-k，再对 top-k 做完整注意力（逐字稿 [31:37]–[32:43] 附近）。有听众追问相似度度量是不是余弦距离，讲者回答说通常就是点积，但是否预先归一化“我其实不知道那个细节”（逐字稿 [32:43]–[33:10] 附近）——这是一个明确承认不确定的地方。
4.4 KV 压缩
Slides 第 25 页给出 MLA 的形式：缓存一个潜在向量 c_t^KV = W^DKV z_t，再从它恢复出每个头的 k 和 v（Slides 第 25 页），脚注同时指向 GQA（Ainslie et al., 2023）和 DeepSeek-V2（2024）。
课堂上用更口语的方式解释了这条路线的两种做法（逐字稿 [34:04]–[35:57] 附近）：分组查询注意力（GQA）减少键和值的头数（常减少 4 倍或 8 倍），因为缓存的是 KV 不是 Q；多查询注意力是 GQA 只有一组时的特例；多头潜在注意力（MLA）不减少头数，而是把每个头的维度压小；两者也可以叠加。
五、训练长上下文：数据比架构更难
Slides 第 27 页给出经典的三段式路线（Slides 第 27 页）：短预训练（4K–8K，数据充足、成本低）→ 继续训练（32K–128K，长文档、打包序列）→ 长适配（128K+，任务轨迹、合成课程）。
课堂上强调了一点幻灯片没说的：互联网上并没有那么多连贯的 128K token 数据，这差不多是一本书的量级；而序列打包虽然提升效率，却不是在学跨文档注意力，甚至可能不该学（逐字稿 [36:22]–[37:33] 附近）。
Slides 第 28–30 页讨论位置编码（Slides 第 28、29、30 页）：绝对位置编码把位置向量加在进入第一层之前；RoPE 旋转 Q 和 K，使得分只依赖相对位移；NoPE 则完全不加显式位置。Slides 第 30 页还有一句关键澄清：NoPE 移除了位置外推问题，但并没有移除“训练长程行为”的需要。
课堂上有一个很值得记的问答：为什么现在因果 Transformer 可以不用位置编码？听众回答“因为我们做因果掩码”，讲者确认并展开——位置 2 的“猫”只注意前一个 token，位置 5 的“猫”注意前四个，它们从注意力第一层起嵌入就不同了（逐字稿 [40:50]–[41:45] 附近）。讲者同时指出，非自回归 Transformer 仍然需要位置编码。
Slides 第 31 页讲位置插值：把 RoPE 的每个维度统一拉伸 s = L′/L，让长位置落在预训练见过的相位范围内（Slides 第 31 页）。Slides 第 32 页讲 YaRN：按波长选择性插值，长波长完全插值，短波长保留局部相位，并加一个注意力温度修正；Slide 明确提醒“τ 的公式是在 LLaMA 模型上拟合的，是配方不是通用常数”（Slides 第 32 页）。
Slides 第 33 页列了各模型的具体扩展路径，例如 Qwen3.8-Flash-Next“262K native；完整课程未披露，partial RoPE，YaRN 扩展到 1M”，GLM-5.3-Flash“1M native；精确课程未披露，主注意力用 NoPE，不做 RoPE 重缩放”，Kimi K3“8K → 64K → 256K → 1M，NoPE”（Slides 第 33 页）。这里“未披露”是 Slide 自己的措辞，值得注意。
Slides 第 34 页给出长上下文数据来源：长文档（书、代码库、仓库、多文档语料）、打包样本、智能体轨迹、合成任务（Slides 第 34 页）。Slides 第 35 页对比了各模型的数据混合，并总结出共同的配方：连贯的长来源 + 依赖远处证据的任务 + 渐进的长度增长（Slides 第 35 页）。
课堂上还坦率地讲了为什么这部分细节少：开源模型必须公布架构才能被运行，但数据混合和训练策略是保密的（逐字稿 [47:59]–[48:47] 附近）。
Slides 第 36 页是上下文并行：显存按设备分摊为 O(n d_k / P)，通信与块注意力重叠，总计算量仍是 O(n² d_k)，是精确而非稀疏的（Slides 第 36 页）。课堂上说它大约在超过 32K 上下文之后开始变得重要，但也很麻烦，因为基础设施团队得为非标准注意力实现对应的并行 kernel；讲者举了自己想给一个较小的 Qwen 模型用上下文并行，却发现“互联网上任何地方都没有用于门控 delta 网络的 kernel”的经历（逐字稿 [49:59]–[51:41] 附近）。
六、KV 缓存：智能体 harness 作者最该关心的一段
Slides 第 38 页画出了多次调用之间的复用结构（Slides 第 38 页）：第 1 次调用做 prefill 建立 KV；输出在 decode 期间扩展缓存；第 2 次调用时工具结果作为新输入返回，但可复用的前缀（cached）不必重算；如此往复，前缀不断增长。
课堂上把这个账算得更直白：到第 50 步时，如果不做缓存处理，输出相关的部分会比新输入大 50 倍（逐字稿 [53:49] 附近）。（这是课堂上的量级示意，不是 Slide 上的实测数字。）
Slides 第 39 页给了两条脚注式结论（Slides 第 39 页）：Prompt Cache 那条说“计算增长比加载缓存注意力状态更快，交叉点取决于硬件”；“Can I Buy Your KV Cache?”那条说“一次性 prefill 被多个读者摊薄，单次调用成本趋近复用步的下限”，并明确区分“实测物理成本”和“服务商 API 价格”是两个不同的量。
Slides 第 40 页是价目表：列出多个模型的缓存输入 / 常规输入 / 输出价格（美元每百万 token，价格检查日期为 2026 年 8 月 30 日），例如 GLM-5.2 是 $0.26 / $1.40 / $4.40，Kimi K3 是 $0.30 / $3.00 / $15.00，Claude Opus 5 是 $0.50 / $5.00 / $25.00（Slides 第 40 页）。Slide 注明“cached”指缓存读取/命中，缓存创建可能另行计费。
课堂上据此给了一个经验规则：通常缓存 token 的价格是未缓存 token 的十分之一，所以你必须非常在意你的 token 有没有被缓存；被问到好的缓存命中率是多少时，现场给出的答案是 80–90%，讲者补充说自己觉得 90–95% 就没什么问题（逐字稿 [54:44]–[56:03] 附近）。
接下来是两条开源服务库的技术主线：
- Slides 第 41 页：分页 KV 缓存分配，块表翻译，出自 vLLM / PagedAttention（Kwon et al., SOSP 2023）（Slides 第 41 页）。
- Slides 第 42 页：RadixAttention 前缀共享，包含“匹配复用前缀并追加后缀”“分裂共享公共前缀”“按 LRU 驱逐叶子”三步，出自 SGLang（Zheng et al., 2023）（Slides 第 42 页）。
Slides 第 43 页是缓存感知路由：请求带前缀哈希 A 进入缓存感知路由器，评分 = 可复用前缀 − 队列惩罚，然后在“匹配 18K、队列 2”和“匹配 2K、队列 0”的 worker 之间选择（Slides 第 43 页）。
Slides 第 44 页给出面向缓存复用的提示设计原则（Slides 第 44 页）：稳定前缀（指令、工具、示例）、把变化的状态追加在后面、避免无谓扰动（时间戳、重排 schema）、度量复用（缓存 token、TTFT、驱逐）。
课堂上对“避免无谓扰动”给了一个具体例子：不要把当前时间日期拼进系统消息，因为它每步都在变，会破坏缓存；如果确实需要，就附加到最后一条用户消息或最后一条工具结果上（逐字稿 [01:04:24]–[01:05:18] 附近）。课堂上还提到中途动态更换工具是可以的（那没问题），但频繁换模型来省钱要非常节制，因为下次回到贵模型时该付的还是要付（逐字稿 [01:05:18]–[01:06:09] 附近）。
关于缓存驱逐，课堂上说服务商不会永远保留你的缓存，有人答“接近一小时”，讲者回应说自己知道 Anthropic 以前是 5 分钟，现在至少有延长到一小时的选项但可能略贵一点（逐字稿 [01:01:52]–[01:02:11] 附近）。有听众问是否存在缓存分层（快速设备缓存 + 慢速磁盘缓存），讲者回答“我实际上不知道那个问题的答案”，并表示这是很好的后续研究问题（逐字稿 [01:02:54]–[01:03:17] 附近）；另一位听众补充“他们通常从 GPU 卸载到 CPU”，讲者确认“是的，因为 GPU 内存太宝贵了”（逐字稿 [01:03:54] 附近）。
学习者解释：把这些串起来，harness 作者的实操含义是——缓存只在“整个前缀逐字节相同”时才命中。因此系统提示、工具 schema、示例这些应当逐字不变；任何随时间变化的东西（时间戳、随机 ID、动态重排）都应当往后放，或者干脆不要放进前缀。
七、上下文压缩：当缓存也救不了你
Slides 第 46 页给了一个例子（Slides 第 46 页）。压缩前有四条观察：CI 任务 30 分钟后超时；检查 job log；24K 行日志里反复出现“address already in use”；设置 --workers=1 后重跑；套件通过，日志存在 /tmp/ci.log。压缩后变成：目标、原因、已验证的设置、证据路径，以及下一步动作。
Slides 第 47 页把它形式化为状态估计（Slides 第 47 页）：历史 H_t 经压缩器 C 得到工作状态 ŝ_t，约束是 |ŝ_t| ≤ B；目标是让未来动作的分布近似不变，即 p(A_future | H_t) ≈ p(A_future | ŝ_t)。
Slides 第 48 页列了压缩中各类信息的去向（Slides 第 48 页）：
- 精确保留：锚点（目标、约束）
- 编码：检查点（决策、进度、ID）
- 精确保留：近期尾部（当前尝试、最新结果）
- 外部化：证据存储（24K 行日志 → 3 个失败 + 命令 + 产物路径）
- 丢弃：重复项、已被取代的尝试
课堂上提醒了一件事：完整历史通常还是会保存在磁盘上，所以即使压缩丢掉了内容，你仍然可以让智能体回去读历史文件；而一些好的 agent 框架会有意避免删除会话最开头的内容（逐字稿 [01:09:03]–[01:10:09] 附近）。
Slides 第 49 页把压缩策略拆成四个决策（Slides 第 49 页）：
- 触发：token 阈值 / 服务商溢出 / 手动请求；要为下一次输出预留空间。
- 选择：保护开头、压缩较旧区域、保留近期尾部；在有效的轮次或工具边界处切分。
- 替换：可读摘要 / 结构化检查点 / 不透明的服务商条目 / 全新上下文。
- 恢复：保持历史不变 / 重试或重置 / 保留可检索历史；并定义压缩失败时的行为。
课堂上补充说触发条件里有一种是“服务商强迫你这么做”，那差不多是硬性上限；很多编码智能体有 /compact 这类命令；摘要有时要求某次工具调用必须包含某些字段，有时就是自由文本；讲者还提到似乎只有 OpenAI 会把内容压缩成加密形式，让人读不到它的算法（逐字稿 [01:10:42]–[01:11:53] 附近）。
Slides 第 50 页讲重复压缩的漂移（Slides 第 50 页）：历史里的“Use CUDA 12.4”经过一次摘要变成“Use CUDA 12.x”，再过一次变成“Use recent CUDA”，最后约束消失。Slide 的公式是 ŝ_{k+1} = C(ŝ_k ⊕ R_k)——每次更新都继承上一个检查点。旁边的对策和 Slide 48 一致：精确复制锚点、保留近期尾部、检索源证据。
课堂上讲的真实故事很能说明问题（逐字稿 [01:12:22]–[01:14:03] 附近）：他们做了可能是最早的接近生产级的压缩算法，在 SWE-bench 上花了一个月左右，找到的策略完全不降低分数还省了很多 token；但放进真实系统后，它开始犯特别傻的错误，比如每次压缩就发一个 pull request——它忘了自己已经往 GitHub 发过了，于是同一个功能收到五个 PR；或者忘掉对话中途发来的用户指令，因为评估只在单条用户消息的对话上做过，而压缩时把多条用户消息压掉了。结论是：光做基准测试不够，除非基准测试极其健壮，否则还是需要在真实环境里测试。
学习者解释：这个例子的教训不在于“压缩不好”，而在于“离线指标与在线行为之间存在系统性落差”。SWE-bench 这类基准往往只覆盖单轮或有限交互模式，而漂移恰恰发生在多轮、长周期、有外部副作用（发 PR、删文件）的场景里。
Slides 第 51 页给出基于延续的评估方法（Slides 第 51 页）：种入状态（约束、决策、产物）→ 应用生产压缩策略 → 继续 → 测量后续决策；四个度量维度是状态召回、任务成功、效率（token + 延迟 + 成本）、稳定性（重复压缩）。
八、收束：五层栈
Slides 第 53 页把整讲压成五层（Slides 第 53 页）：
- 增长：智能体循环反复摄入指令、动作和环境观察。
- 推理：prefill、decode、密集注意力和 KV 内存制造了不同的瓶颈。
- 架构：局部、循环、稀疏和压缩机制决定了哪些历史仍然可访问。
- 缓存：稳定前缀避免重复 prefill，但不减少可见历史。
- 压缩：保留意图、精确任务状态、近期工作和恢复路径——然后评估延续性。
第 4 条里的“但不减少可见历史”值得单独强调：缓存解决的是成本，不是容量。一个上下文已经超过有效窗口的智能体，即使缓存命中率 95%，也仍然会忘记东西。
Slides 第 54 页标注下一讲是 Agent Capabilities 3 — Skills and Memory（Slides 第 54 页）。课堂上有人问 sub agent 是否算一种上下文管理方式，讲者回答“是的，但我们后面会讲 sub agent，所以这次没有涵盖它们”（逐字稿 [01:14:52] 附近）。
待核与不确定之处
- 逐字稿为机器翻译整理稿，自动字幕的错词与图示、公式均需对照原视频；本文引用逐字稿处只作为“课堂讲述”的转述。
- Slides 第 10、18、19、20、21、22、23、24、25、28、29、30、31、32、36、42、43、47、50 页包含公式与图示，逐页文本提取无法替代视觉核验，本文涉及的公式均按提取文本转写，可能存在符号位置偏差。
- Slides 第 4 页的饼图、Slides 第 11 页的针尖图、Slides 第 38 页的调用时序图、Slides 第 41–43 页的块表 / 基数树 / 路由图，其具体布局需对照官方 PDF 确认。
- 课堂口头给出的数量级（如一万亿次两两比较、10²⁰ 量级、第 50 步大 50 倍、缓存价格十分之一、命中率 80–95% 等）属于讲述中的估算或经验，不是 Slide 上的规格数据。
- 缓存驱逐时长、缓存分层、相似度度量是否归一化等问题，讲者在课堂上明确表示不确定或未展开。
- Slides 第 16、33、35 页中标注为“未披露”“undisclosed”的模型训练细节，本讲未给出结论。
官方来源
- 课程主页：https://www.cmu-agents.com/
- 课程日程与阅读：https://www.cmu-agents.com/#schedule
- 作业与项目要求：https://www.cmu-agents.com/#assignments
- Assignment 1 · Harness：https://github.com/cmu-agents/assignment-1
- Assignment 2 · Eval：https://github.com/cmu-agents/assignment-2
- 课程视频：https://www.youtube.com/watch?v=AiwCCvFW1uE&list=PLSN0qpDfUvTM&index=3
- Lecture 3 slides：https://www.cmu-agents.com/slides/lecture-03-long-context.pdf
CMU 11-768 Fall 2026 · Lecture 3 · Agent Capabilities 2: Context Management for Long-Context Agents｜完整中文逐字稿
- 视频：官方视频
- 来源：平台英文自动字幕清洗稿的机器翻译；保留原段落时间戳。
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
- 说明：这是逐字稿，不是摘要；保留讲述顺序、例子与课堂互动。
正文
[00:07.120] 好的。呃，大家好。后面的人能听到我说话吗？都没问题。好的。那么，呃，欢迎各位回来。我这里有回声吗？没问题。好的。那么，呃，今天我想谈谈面向长上下文智能体的上下文管理。我一开始想把标题定为类似“面向长上下文LLM的上下文管理”，因为今天我要讲的大部分内容在某种程度上 更偏向通用的LLM内容，但长上下文在我们做智能体时极其极其重要，而且它未必在所有LLM先修课程中都被很好地覆盖。所以，我今天会相当详细地讲这个内容。我也会尽量过一遍许多最先进的开源模型中使用的长上下文架构，
[01:00.640] 这样你们就能看到大家在处理非常长的上下文时所采用的最佳实践。那么首先，我想稍微谈谈智能体的上下文能增长到多长。基本上，输入是平方级的，呃，或者说你所进行的token访问数量是平方级的，基于线性增长的历史记录。所以，呃，如果智能体每次调用 大约消耗，你知道，五个token，假设你一开始有一个token的初始系统提示，那么第一次调用你会得到六个token，11、16、呃、21和26。你知道，这会长啊长啊长。呃，所以等到最后，如果你完全朴素地处理，你就不得不为这五次调用处理累计的token 你知道，这可以持续很长时间。我不知道有谁用过的最长单次编程智能体会话之类的，但我确实做过好几天的，呃，几乎一整天都在用它工作。所以你可以想象这可能会变得非常非常大。呃，实际的
[02:22.959] 计算复杂度，呃，实际上比这个还要大。呃，不对，抱歉。是的，这里的计算复杂度是平方级的。那么为什么智能体的提示会这么快变得这么长呢？我对我们在，呃，编程智能体OpenHands中使用的1,500个会话的提示组成做了一点数据分析。每个会话都相对较短。
[03:00.000] 这些不是那种持续多天的长会话。这更像是解决单个任务。但呃，每个平均使用了大约78,000个token。呃，其中23%是系统提示加上所有的工具描述。呃，这对一个编程智能体来说算是比较大的。我想我们的系统提示大概在15,000到18,000个token左右，因为我们非常有主见。 呃，OpenHands智能体对于人们如何，呃，做事情非常有主见。呃，这些往往在8,000到，呃，8,000到20,000个token之间变化，取决于你，你知道，引入了多少工具以及类似的东西。呃，另外，呃，9%是用户消息。所以这是用户要么说出初始任务，要么之后跟进补充，呃，额外信息。9% 是推理轨迹，呃，你知道，就是模型在思考事情的地方。2%是模型回复。抱歉，这个没有显示在幻灯片上，但2%是，呃，模型直接回复用户。所以，它花在推理上的token大约是花在用文本与用户沟通上的五倍。而20%是工具调用。呃，
[04:10.879] 所以，你知道，呃，你知道，写代码，呃，读代码，诸如此类。而37%是工具结果。所以，这是，呃，你知道，获取，你知道，读取文件之类的结果。所以你知道，每一步都能，都会包含所有这些，而且这往往会，呃，非常快地累积起来。
[04:34.080] 所以当你在处理这种非常长的上下文时，呃，这里有两个大挑战。呃，第一个是容量，也就是你的模型是否能够首先处理这么大，呃，量的证据，这是一个挑战，呃，因为正如我稍后会讲到的，训练模型处理非常长的上下文是很难的，呃，所以为了管理这个，你 需要，呃，像架构、位置编码、训练数据这些东西，呃，而且，呃，你还需要能够评估你是否在这件事上做得好。然后第二件事是效率，呃，也就是系统是否负担得起服务它的成本。也有一堆技术可以做到这一点。而且我，我觉得有些人可能把这当作理所当然，如果你只是用，你 知道，Claude Code或Codex之类的，但它们在做一些相当了不起的工程壮举，把一个，你知道，数万亿参数的模型在几秒内提供服务，呃，一遍又一遍，反复地，数百万，我不知道，Codex有2500万用户。所以你可以想象，呃，2500万日活用户每隔几秒就来访问
[05:43.840] 这个巨大的模型。呃，如果你想的话，你可以在脑子里算算这大概是多少flops，呃，每秒。那是很多flops。好的。呃，那么有什么问题或想法吗？大概挺直白的，我猜。好的。那么，呃，我们进入推理。所以为了实现这个，呃，你基本上需要能够，呃，计算，呃，预填充和解码。所以，呃， 在每一步计算你的输入和输出。而这个，呃，你知道，我相信大家在之前的课上都做过，呃，从，呃，你知道，一个语言模型生成。有些事你们在之前的课上可能想过也可能没想过。如果你上过，比如，呃，语言模型系统或深度学习系统之类的课——实际上比我想的要多。 所以如果你上过系统课，你大概处理过这个。呃，但你还需要考虑的其它事情是，你如何调度，呃，多个并发查询？呃，以及你如何把结果保存在KV缓存中，呃，这样你就可以重用它们，而不必一遍又一遍地重新计算。呃，不必一遍又一遍地计算。
[07:05.599] 所以这里再稍微详细一点，呃，我们在推理中有几个，呃，阶段。第一个叫做预填充。呃，你也可以叫它编码。呃，但我认为现在预填充是，呃，比较受偏爱的专业术语。所以，你把你所有不需要生成的输入token拿进来，呃，然后你从每一个，呃，生成键值对。而 然后，呃，当你在解码时，你基本上，呃，把提示，呃，和键值对拿来，呃，生成一个输出，并追加，呃，你从输出得到的键值对。所以，呃，你是以这种增量方式做这件事，而这可以按批处理序列来完成。
[07:53.520] 所以呃，当我们把它看作一个服务系统时，有很多方法可以量化一个服务系统有多好。呃，第一个是首 token 时间。也就是你需要等待多久才能开始，呃，你需要等待多久，从你开始呃输入一个查询，到你开始看到第一个输出之间。所以，我相信所有使用聊天 GPT 或类似东西的人呃，你输入一个很长的输出，你会注意到它开始得到响应所需的时间比如果你输入，抱歉，输入一个很长的输入，你会看到它开始得到响应所需的时间比如果你输入一个很短的输入要长，这就像你的 TTFT 更高。呃，然后呃，你的每个输出 token 时间，是呃，时间，就像 生成每个 token 之间所花费的时间。如果你去一个像 open router 这样的网站，呃，通常呃，这里这会是你的每个输出 token 时间，我很确定他们这里没有计算 prefill，它只是像是呃每个 token 之间的时间，在黑色背景下这可能有点小，但你可以看到，呃，对于 GLM 5.2，呃，你有像是 63、48、呃 24，呃，那种的。而且你知道，有些提供商比其他提供商好得多。比如这个提供商，不是要针对 Mistl，但他们得到的是，你知道，每秒三三个 token，然后这些其他的，比如 Parasel 和 Coreweave 得到的是，你知道，每秒数百个 token。
[09:32.720] 所以呃，所以基本上呃，你知道，这对你使用呃呃使用智能体的体验有很大影响。呃，另一个有趣的事情是呃，人们使用智能体有不同的方式。所以在某些情况下你可能会真的听说这个。比如假设你正在一起呃和一个智能体在你最喜欢的编程 CLI 中做一个交互式任务。你可能真的 真的关心，你知道，TTFT 和 Tapot。呃，如果你在后台运行一个智能体来做一些通宵运行的背景任务，你可能完全不在乎。呃，而且它们之间有取舍，比如吞吐量和成本，以及呃你能多快得到 token。而且比如我，你知道，也呃用我的公司服务编程智能体 而且当我和提供商交谈时，他们经常会问，你更关心哪一个，这样我们就能为你更关心的那个优化系统，所以呃，有呃，有办法可以做到这一点，我也会，我也会谈到那些。好的。呃，是的，所以抱歉，只是为了讲完这张幻灯片。所以吞吐量是呃，完成的 token 或请求数除以墙上
[10:54.880] 时钟时间。所以这通常是在一个更整体的层面上，在整个系统上看到的。然后呃，成本是，你知道，它花费多少。所以作为复习，呃，我，我希望大家应该都已经知道这个了。呃，但在典型的注意力中，你有呃呃查询、键和值的计算，这是呃，所以你有这个在这里，然后你取 softmax，呃，除以这个。你可能有某种呃，你知道，位置呃偏置，然后你计算你的输出，典型的全注意力呃看起来呃像这样。你把所有之前的 token 都掩码掉，和，或者说你考虑所有当前的 token，然后你掩码掉所有呃未来的 token。
[11:44.720] 呃，所以这当然非常昂贵，因为它在序列长度上是二次的。呃，所以我今天要讲的很多东西都是关于如何解决这个问题。呃，所以在效率方面是这样，然后在有效性方面，仅仅因为你有一个模型，或者仅仅因为你有一个服务系统可以接受，你知道，一百万个 token，并不
[12:09.519] 一定意味着你可以有效地使用那一百万个 token。呃，这是一个非常，像是老派的例子，我猜，这是大海捞针评估，有多少人见过这个图，也许一些人，但不多。好的，所以解释一下这看起来是什么样，基本上这说的是，呃，这是一个非常奇怪的评估。基本上，他们 呃，他们拿了一大堆 Paul Graham 的文章，呃，就是那位著名企业家和 Y Combinator 负责人 Paul Graham，然后他们在里面塞了呃，像是关于在旧金山吃三明治最好的地方是什么什么之类的，然后他们问一个问题，在旧金山吃三明治最好的地方在哪里，然后语言模型必须回答那个 正确地。呃，然后他们让上下文越来越长越来越长，然后他们用更长的上下文评估呃模型，最终较弱的模型在这些非常非常长的上下文上开始变差，而且有很多像这样的例子。呃，ruler 和 helmet 是人们在这里使用的常见的基准。呃，但基本上呃，长的，总结是
[13:21.440] 如果你有非常长的上下文，你的模型在处理它们时可能开始变差。所以有没有人注意到这个，当你长时间使用一个编程智能体时，就像是有没有常见的，或者长时间使用过任何智能体，有没有你看到的常见的像失败案例？ 什么样的，什么样的东西困扰过你？有人吗？>> 是的。>> 忘记之前的上下文。>> 是的。忘记之前的上下文。那有没有给你造成什么像问题？>> 是的。对我来说，告诉它不要在对话中间做某件事，然后它还是去做了。呃，有一个非常糟糕的例子，呃，我
[14:16.160] 想实际上是 Daniel 在这里谈到的，那个人说了不要删除这个，然后它还是去删除了，因为它忘记了上下文。所以是的，这在智能体设置中可能是一个非常大的问题，因为你知道用户是在给出这些增量指令。所以，所以我要谈几个不同的东西。我要
[14:36.079] 谈架构。我要谈训练、推理和压缩。所以首先进入架构。呃，这既是关于效率也是关于准确性。所以正如我提到的，标准注意力是全局的。所以它在呃这里所有的 token 之间做两两比较。呃，这个问题是呃，这显然是二次的。而且 有人知道如今模型的像标准长度吗，这些呃，像最先进的语言模型的？>> 256 到一百万。>> 256 到一百万。是的。而且我，我会说大多数人现在目标是 100 万，因为他们不想成为那个，你知道，在其他人都有 100 万时只有 256 的模型。所以在你脑子里快速算一下，多少个
[15:30.880] 成对比较是？ >> 10 12 >> 10 的 10 的 12 次方，或者用英语说。 >> 一万亿。对，那是一万亿。每次都是一万亿次比较。呃，就像当你在生成一个长度为一百万的序列时，每一次比较都需要你把大向量彼此一遍又一遍地相乘。所以你就开始达到你知道的 10 的 我猜大概是 10 的 20 次方左右，呃，这很多，对于单个序列来说是大量的计算。所以呃，人们有办法解决这个问题，基本上几乎所有这些方法都属于把这种二次方计算，对每个时间步所做的比较数量施加某种常数上限的类别。这样就把你的
[16:30.639] 呃 n 的平方降到了 owl，我这里写 w 表示 window，但它不一定需要是一个窗口。它只需要是你所做比较数量的上限。呃，所以有局部模型和全局模型的概念。局部模型基本上就是这种计算上限发生在一个局部窗口内的模型。然后全局模型，你 对计算可能有也可能没有上限，但那个计算上限不是局部的。所以你可以把信息传递呃贯穿整个序列。而且我会说，如今存在的或如今广泛流行的几乎所有模型，都是某种混合模型，它们有许多局部计算，然后一次全局计算，然后许多局部
[17:21.679] 计算，一次全局计算，具体来说我要讲我们这里的六个呃模型。所以呃，最新的 Quen 模型呃有呃 gated delta net 的局部计算。我稍后会解释所有这些呃或其中大部分。然后呃用于全局计算的稀疏检索，比例是 3:1。 呃 GLM 5.3 是呃也是某种 delta 呃和稀疏注意力呃，3:1。呃呃 Kim K3 是呃某种 Delta 和呃这是密集呃密集注意力 3:1。呃 Neimotron 是 Mamba，呃这是一种呃有点像循环风格的模型，呃和密集注意力 4:1。 呃到呃窗口呃滑动窗口注意力呃密集注意力 5 比 1，而 deepseek 有点不同。它有点有呃窗口窗口化注意力，但然后有一些呃稀疏性呃稀疏的东西，带有交错分支。所以呃基本上做事情的标准方式是混合模型呃带有某种局部的东西，呃以及稀疏或密集的呃全局注意力。
[18:50.000] 所以我要讲这个。我不会对所有这些都深入讲一大堆细节，但呃这应该足够让你以后能够查它们。所以第一个是呃滑动窗口注意力。这也许是概念上最简单的。你要做的是取块，或者呃你取输出的块，基本上你能够 关注你当前的块，呃以及所有之前的块，呃或者不是所有之前的块，而是固定数量的之前块。所以如果这是你当前的块，呃它可能是像 512 个 token 之类的块。你会有额外的 512 和 512 个 token 在之前的块里，你会关注那些。呃所以相当简单，注意力没有 变化，但你有像这样的东西。但关于这个的好处是，在第一层你从之前的一两个块传递信息，但在下一层，呃之前的块也带有它们之前块的信息。所以你有点得到这种逐渐扩展的窗口。呃所以呃这呃这被相当广泛地使用。
[20:08.880] 呃但如今更流行的做法是使用某种线性注意力。所以呃线性注意力的工作方式基本上是，我们拿我们的呃标准注意力，标准注意力包含这个呃 softmax 操作在里面。这就是它难以计算的原因，因为你要取指数，呃然后做除法。而且 因为你这样做，你基本上别无选择，只能展开所有的呃 key 和 query 计算，呃像 key query 乘法计算，并且是呃二次方呃二次复杂度。线性注意力所做的是，你基本上从这个呃那种标准注意力公式开始，但你拿你移除 softmax 和这里的归一化，或者我的意思是 归一化其实不太重要，但你移除 softmax，你就只得到 key query 乘法，然后[清嗓子]这绝对表达力更弱，因为基本上你正在做的是，你从某种接受一堆呃像数字的东西， 总之，就像假装假装下面这个加起来等于一，但你从一个接受一堆实数并把它们归一化成概率类数字的东西，这些数字在零和一之间并且加起来等于一。呃，所以这种非线性让你能够专注于比如说只关注之前呃像 token 中的一些。而这正是注意力有点像的优点。那是
[22:15.600] 注意力的卖点，让你能够专注于相关的先前上下文。而你基本上在拿走那个能力，对吧？你你在移除相对比较，只是得到呃 query 和 key 之间的点积。但它允许你做一件非常呃好的事情，那就是你可以使用呃结合
[22:38.880] 呃你可以使用结合性，基本上重新排列，这样呃你可以把之前所有 key 和 value 呃乘法的和转换成单个呃状态矩阵。所以这个状态矩阵本质上是一个函数，它映射，它是一个从 query 到 value 的线性函数。所以注意力也是一个从 query 到 value 的线性函数。所以如果我回到这里 你基本上在计算，你拿 query，你在所有 key 上计算注意力，你把它乘以 value，你在这里得到一个 value。呃而这个状态矩阵也是那个。但关于它的好的部分是，这个状态矩阵可以被分解。所以你可以更新它。你可以把这个状态矩阵保留着，并在常数时间内更新它，每次 你得到一个新值。所以这本质上变得像一个循环神经网络。所以它它允许你呃在线性时间内更新状态矩阵，而不需要呃不要求二次计算。所以这基本上解决了你的二次问题，而且解决得如此之好，以至于呃 O of N 的平方变成了 O of N。
[24:00.799] 这里甚至没有 W 项，因为呃每次你都是完全基于前一个状态来更新的。呃但这里有一些注意事项。所以呃到目前为止这个清楚吗？好，不错。呃，所以最大的注意事项正是我之前提到的那个注意事项。就是这是一个表达力更弱的函数。呃，它不允许你做像 前一个 token 之间的相对比较之类的。所以有一堆呃变通方法试图让它成为一个表达力更强、但仍然有用的函数，呃在呃做类似注意力计算方面。所以呃其中大多数，如果你深入挖掘其中的数学，它们基本上是在试图成为最终值的更好的预测器，
[24:56.559] 那个你从注意力中会得到的值。呃你可以深入阅读这些论文看看它是如何工作的，但我将主要在这里从机制上解释它们。所以呃第一件事是 Delta。DeltaNet 背后的想法是呃你可以预测，呃你基本上可以预测你会想从呃从 KT 的计算中得到的值 呃和 S。所以你知道这里的 KT 是呃时间步 t 的键，你把它与时间步 t 呃 t 减一时的矩阵进行比较，呃你计算这个 v hat t，另一方面你呃所以基本上这这是你会得到的对 kt 的预测。但另一方面，你知道呃输出 VT 应该是什么，基于呃查询乘以呃 或者基于呃那个，等一下。[笑声] 我不想我不想把这个弄错。所以你计算 KT 相对于这个矩阵。你可以把这个矩阵看作是一种呃计算呃从查询到值的映射的方式。你也可以得到实际的值，然后减去呃当你把它与 前一个呃和前一个矩阵相乘时得到的值。这基本上就是你在这里得到的误差。然后你想减少这个误差。所以为了减少这个误差，你要做的是呃本质上加入这个呃这一项，即 KT 呃乘以误差项。做完这个之后，如果你把 KT 乘到新矩阵上，你就不再会有任何
[27:20.240] 误差了。但你不想让更新太大。所以你加入这个 beta 项。这基本上是一个学习，它类似于学习率。而这个学习率会让更新呃更小。所以呃如果你这样做，这允许你基本上把矩阵朝着呃朝着那个会让看起来像 当前键的查询输出你想要的值的呃方向微调。所以它是减少误差呃减少你想要的值的预测中的误差。所以呃正如我所说，这会有点高层次。呃这些论文是呃对此相当好的呃描述。
[28:10.000] 呃所以在这之后呃这基本上所做的是，这把你推向呃让矩阵成为呃值的更好预测器的方向。但这个问题之一是你可以从之前的位置积累大量大量的上下文，而线性注意力没有任何机制来防止非常遥远的上下文对你的下一个预测产生不好的 影响。所以如果你有一个非常奇怪的键在很远的过去，它可能对你呃在未来产生非常大的影响。所以他们修复这个问题的方式基本上是有一个衰减项来呃让矩阵在每个时间步衰减一点。通过在每个时间步让矩阵衰减一点，他们基本上是在 对过去很远的上下文打折。所以你呃把矩阵乘以 alpha t，然后你把这个呃 st tilda 加到呃你的计算中并更新矩阵。所以你把衰减设得越强，基本上它就会越快呃忘记之前的上下文。所以你可以让它更局部或更不局部。
[29:47.360] 所以这是呃门控 delta，用于 Quen 模型。所以它是现在呃更流行的方法之一。呃但呃 GLM 和呃 KI 这两个基本上是两种最先进的模型，都使用了一个稍微更具表达力的版本，呃它们基本上有一个逐元素的呃衰减项。所以每个元素可以以不同的速度衰减。呃 因为每个元素以不同的速度衰减，你可以有一些呃上下文部分呃你知道延续非常长的时间段，一些上下文部分延续非常短的时间段。不错。所以希望这或多或少呃或多或少是可读的。呃这里有任何问题吗？
[30:39.600] 长话短说，它基本上是一种把呃变成呃它是一种把注意力的想法变成你可以在上下文中计算的呃东西的方式。Mamba 是呃非常相似的东西呃但它在数学上稍微更复杂一些呃呃我我决定跳过它。呃我在呃语言模型呃描述课程中更详细地讲过它，但想法类似但
[31:10.000] 只是在数学上稍微更复杂一点。所以接下来我想谈的是稀疏注意力。所以稀疏注意力是呃基本上查找，不是关注前文中的所有内容，而是只关注前文中的一些东西。本质上你限制了你在前文中查找的东西的数量。有几种方法可以做到 这一点。呃最初有一篇由 OpenAI 发表的稀疏注意力论文，呃在相当久之前，大概 5 年前或类似的时间。呃基本上呃他们所做的是每呃 512 个 token 左右做一次稀疏注意力。这样做的结果是，你仍然有一个固定的，它在序列长度上仍然是二次的，但
[32:03.279] 它比你关注每个 token 时要少 512 倍，而且这足够好，呃你能够在你感兴趣的上下文上计算它。另一个选项是做呃内容相关的稀疏注意力。基本上它做的是，你有一种非常廉价的计算注意力的方式。 比如说，你可以把注意力向量降投影到非常非常小，然后你对这些非常非常小的呃注意力向量做注意力。你选出前 k 个，然后你对前 k 个做完整注意力。所以呃那是呃你可以做到这一点的另一种方式，这就是 deepseek 中使用的方法。
[32:43.600] 是的。>>我们如何定义用来做 top 相似度比较的度量，在这个在这个嵌入空间里我们只是做像余弦距离。>>是的。>>是的。我我想通常不是余弦而是只是点积，但是的 >>我我我觉得虽然也许也许他们事先做了归一化。我其实不知道那个细节。
[33:10.080] 还有其他问题吗？所以基本上，嗯，这是一种非常非常廉价的方式来实现全局注意力，因为你只需要写一个内核就能为你做到这一点。嗯，这边的这个方法需要你进行两遍计算，但你知道它可能更准确。
[33:35.039] 所以嗯，我见过这两种方法都被相当广泛地使用。那么最后一点是 KV 压缩。有多少人在语言建模课上见过这个？也许有些人。好的。嗯，所以我会解释一下这个。基本上，嗯，这是标准的多头注意力。所以在标准的多头注意力中，你有一堆头，每个头都有 值和键，所有这些都有一定的维度。所以它们比如说是维度 D，然后你有一定数量的头。比如说那是 H。所以人们解决或帮助解决 KV 缓存非常大且昂贵这个问题的两种主要方法是分组查询注意力。而分组 查询注意力基本上就是使用一个比 h 更小的数字，通常是用 h 除以某个二的幂次，得到键和值的数量。因为这是 KV 缓存，不是 KVQ 缓存。嗯，如果你减少所使用的键和值的数量，你就减少了保存 KV 缓存所需的内存量。这非常重要
[35:02.880] 因为如果它变得非常大，那么它就会限制你能在一张 GPU 上容纳的数量，所以这通常可以减少四倍或八倍之类的。嗯，多查询注意力就像分组查询注意力只有一组的情况。所以你几乎可以跳过它。你已经知道它是什么意思了。嗯，多头潜在注意力，嗯 本质上是对键和值的尺寸进行降维投影。所以它不是减少头的数量，而是让每个头更小。这和我之前讲的自上而下 K 注意力类似。嗯，当然你也可以把这两种方法结合在一起。你可以用分组查询注意力加上更小的键和值。
[35:57.280] 嗯，这里有什么问题吗？好的，很好。嗯，所以这些就是人们用来处理非常长上下文的架构方法，有了这些嗯，你可以处理多达一百万个 token。所以这就像是，你知道，这让效率成为可能。
[36:22.720] 下一件事是训练长上下文模型。这实际上相当困难。原因是因为嗯，如果我们想要处理 128k 个 token 甚至更多，互联网上实际上没有那么多 128k 连贯 token 的数据。 嗯，实际上有很多很多数据。它比大多数维基百科文章要多，你知道，嗯，你开始进入需要有一本书的量级，而互联网上没有那么多书。所以，通常你做的是从短预训练开始。嗯，你用更长的文档做更多的继续训练，然后你做长 适配。嗯，我在这里写了打包序列。嗯，但打包序列对于效率来说是有用的，但它们对于学习长上下文不是特别有用，因为嗯，每次你比如说把 10 个文档打包成一个序列，你并没有在学习跨文档的注意力，事实上你可能也不应该这样做。所以嗯，你真的
[37:33.440] 需要连贯的长数据集来做非常长上下文的适配。嗯，所以我要讲几个我们在这里做的事情。嗯，我要讲的第一件事是处理位置编码。嗯，所以绝对位置编码，我希望每个人在他们的语言建模课上都见过。嗯，但基本上嗯，每个单独的条目，嗯 每个单独的词的嵌入都会得到一个位置向量，那个位置向量通常是通过某种函数计算的，比如使用正弦和余弦之类的。所以希望大多数人都见过这个。所以我会跳过那部分。也许大多数人也都见过 RoPE。大家对 RoPE 一般熟悉吗？好的，我看到相当数量的人。
[38:32.079] 实际上我看到的人比我预期的要少，但嗯，也许人们只是不想举手。嗯，[笑声] 但嗯，但基本上嗯，RoPE 在我看来在数学上相当优雅。嗯，基本上他们试图解决的问题是他们想要一种位置嵌入，嗯，其中位置编码 其中位置编码完全由两个 token 之间的相对距离决定。他们想要这样做的原因是他们不想让绝对位置对，他们不想让绝对位置影响两个 token 是否会互相注意。 所以这非常有效，很多模型都使用 RoPE。嗯，数学推导我不打算在这里讲，因为这不是重点，但他们用旋转和虚数之类的东西做了一些有趣的事情。
[39:51.760] 嗯，但还有这个非常反直觉的，不对。那就是完全不使用位置编码。嗯，如果你上过关于 Transformer 的课，你可能听过这个可怕的说法，就是 Transformer 需要位置编码，因为否则位置一的东西和位置五的东西会有完全相同的 表示。所以如果你有，这是一只猫，但这不是一只猫，它会以相同的程度注意到两只猫之类的。嗯，理论上当 Transformer 最初被创建时，嗯，这可能是真的，但现在实际上基于我们使用的 Transformer，这已经不再是真的了。嗯，有人知道
[40:50.160] 为什么是这样吗？我在幻灯片里给了一点提示，但你可能也需要了解一点 Transformer 的历史，嗯，那就是在我们使用编码器-解码器模型之前，编码器编码所有输入，然后解码器生成输出。有什么想法吗？>> 因为我们在做因果掩码。完全正确。所以嗯，稍微展开一下， 如果我回到我们的 Transformer 方程。嗯，我们有键，嗯，还有查询。如果我们看查询，即使你在位置二有一隻猫，在位置五有一隻猫，位置二的猫只注意到前一个 token，而位置五的猫注意到前面四个 token。嗯，所以它们两个都是
[41:45.680] 实际上是不同的。嗯，至少从注意力机制的第一层开始就是如此。所以你可以看出，即使它们是完全相同的词，从注意力第一层开始的嵌入也会是不同的。而且有一篇有点反直觉的论文证明了你根本不需要位置编码。这 在自回归Transformer中尤其如此。在非自回归Transformer中，所有token都在相互关注。你仍然需要位置编码。嗯，当我们开始讨论门控delta时尤其如此，因为门控delta有衰减项以及诸如此类的东西，这会导致
[42:30.160] 每个token都变得不同。所以一旦我们开始有这类局部偏置，我们实际上就不再需要位置编码了。嗯，如果你看一下，我们这里前两个有RoPE，或者至少有部分RoPE。后两个没有位置编码。嗯，第四个完全依赖于隐式顺序和
[43:02.079] Mamba，其中一些，还有Inkling有相对位置编码。所以你可以看到，在拥有相对位置编码和完全没有之间大致是一半一半。但假设我们确实有位置编码。嗯，对于长上下文，你需要担心的一件事是位置插值或位置外推。这意味着如果你有一个 预训练范围，嗯，你本质上需要扩展你的模型，使其能够处理更长的序列。所以RoPE的承诺是，RoPE只依赖于输出中不同token之间的相对位置。但只要相对位置只是，嗯，好吧，有两个原因说明这并不 没问题。第一，如果你在长度为L的序列上训练，然后扩展到长度为5L的序列，那么它从未见过两个token在超过L的距离上相互关注。所以距离超过L的token在相互关注时会遇到问题，你无法很好地处理这种情况。嗯，另一个问题是
[44:26.560] 实际上相对位置编码只是故事的一部分，实际上自回归掩码在所有这些其他方面也有影响。所以你不能只是拿RoPE就指望它在更长的序列上表现得很好。所以解决这个问题的典型方法是，RoPE中有一个参数，基本上，让我看看我的幻灯片里有没有。 所以我的幻灯片里其实没有，但RoPE中基本上有一个参数theta，它告诉你在每个时间步旋转多少。然后你用扩展序列的倍数来除这个theta。所以基本上你是在拉伸RoPE位置，使得位置零处的RoPE位置，或者位置
[45:22.960] 五处的RoPE位置将等同于你扩展上下文中位置五。还有一种更复杂的方法叫做YaRN，嗯，它基本上在精神上类似于delta，类似于kimi delta注意力，基本上说的是我们希望其中一些是非常长距离的，而另一些不是非常长距离的。在 YaRN中，本质上你让其中一些被扩展得更多，一些被扩展得更少，所以你不一定以相同的频率对它们全部进行插值。嗯，是的，鉴于时间关系，我不会深入讨论太多细节，但基本上这些就是你可以在这里使用的各种方法。
[46:27.040] 那么，嗯，我现在想谈一谈，嗯，从详细的建模内容退一步，进入你可以用来训练这个的数据。那么，嗯，正如我之前提到的，打包样本，你可以直接把一大堆东西打包进去，嗯，一大堆文档打包进你的上下文长度。这可以训练模型能够 处理上下文窗口接近末尾的内容。但它不能做到的是那种全局连贯性。嗯，所以人们有很多不同的方法来做这个。一种方法是直接找到，嗯，非常长的文档，嗯，比如书籍之类的。这类东西不多，嗯，但确实有书籍。有代码库，代码库往往是连贯的。所以你可以 直接下载一个大代码库，那肯定会有更多的token。某种多文档语料库。所以，你知道，比如你可以获取一大堆关于同一主题的文档，然后至少它们之间会有主题连贯性。嗯，我也见过人们在专利上训练，因为专利有，有很多公开数据，而且你可以 把它们链接在一起。现在一个很大的来源是智能体轨迹，因为你知道如果你和智能体交互一会儿，它们可以产生非常非常长的轨迹。所以这是另一个来源。然后你也可以创建合成任务，很多评估都是合成任务。所以你也可以这样做。
[47:59.599] 嗯，我有一些关于模型如何做到这一点的例子。嗯，很多模型训练公司不喜欢和你分享这些细节，因为这就好比，如果他们发布他们的，嗯，好吧，第一，像Anthropic、OpenAI，你知道那些地方，嗯，嗯，Gemini这样的闭源模型，他们甚至不会发布他们的架构细节。开源模型公司必须发布他们的 架构细节，否则你无法运行他们的模型。那么他们保密什么？他们对数据混合和训练策略保密。所以，嗯，我们能说的详细内容就少得多了。嗯，但是，所有在这里的模型都解释了，都解释了一点关于它们的数据混合，内容有比如清洗过的采样文档和 视频，嗯，文档、仓库文件、issue、PR，嗯，长文档问答、科学论文和技术报告。然后嗯，他们还会创建一堆合成任务。通常他们有一个缩放策略，比如8、64、256、100万，嗯，诸如此类的。
[49:10.640] 哦，是的。嗯，[清嗓子]还有一个你可能需要处理的物流细节，嗯，或者根据你训练的模型大小可能不需要处理。比如，对于你的项目，如果你想开始处理，嗯，嗯，更大的模型，那就是上下文并行，基本上这意味着在训练模型时有一堆，嗯，做并行的方法，我
[49:36.319] 上面写了上下文并行，那在模型训练或服务期间，你还可以通过另外哪两种方式来并行化模型？>> 是的 >> 数据并行 >> 是的，你说了第二种的一个更具体的例子，但数据并行，我会说，一般来说是模型并行，而模型并行可以有很多种类。 它可以有专家并行，即你把不同的专家放在不同的机器上；更一般地，还有张量并行，即你把参数拆分；你还可以有流水线并行，即你把不同的层放在不同的设备上。但上下文并行对智能体来说开始变得更加重要，因为当你 训练智能体时，你想在非常长的上下文上训练它们。所以其工作方式是，你会计算上下文的一部分的键和值。所以你把它发送到下一个设备，为那一部分计算键和值，再把它发送到下一个设备，等等等等。所以这非常有用。如果你开始超过大约32k
[50:47.680] 32k上下文左右，它就真的很重要了。但它也很麻烦，因为你有所有做建模的人说，我想要我的K I delta注意力，然后你有所有做基础设施的人说，但是等等，我需要为K I delta注意力实现一个上下文并行kernel，并确保它真的能工作，而且你能通过它进行反向传播，以及所有其他这些东西。所以 我个人有过一次经历，我当时试图训练Quen，像是较小的Quen模型，大概135、35b，我想对它使用上下文并行，结果就像互联网上任何地方都没有用于门控delta网络的kernel，所以你要么需要进去实现那个kernel，要么决定做点 别的。所以这两者有时会发生冲突。所以我想说，要意识到这一点。好消息是，对于标准注意力和滑动窗口注意力之类的东西，所有这些都会更容易有已实现的kernel。困难的部分在于你做的不是标准注意力操作的时候。
[52:06.240] 好。那么接下来我想谈的是提示和KV缓存。这非常重要，即使你不是实现语言模型的人，但如果你只是一个自己在运行智能体的人，也是如此，不过我会讲一下它是如何实现的技术细节。这样，至少你对 它的工作原理有个好概念。基本上，我把它稍微简化了一下，但当我们生成一次智能体调用时，我们有一个输入，我们有一个输出，所以这可能是消息和工具调用，然后我们有工具结果，所以当我们计算输入时，我们做预填充，当我们计算输出时，我们基本上做解码，对吧？所以
[53:04.960] 在预填充和解码期间，我们必须计算KV缓存。这一切都是在一个批次中完成的。每次我们生成一个token时都会做这件事。但在我们有了这些之后，我们就有了直到输出结束为止每个位置的键和值。所以我们可以做的是，把它扔进缓存里，然后我们就不再需要计算它了。 工具结果是新的，因为你知道我们不会对工具结果进行计算。所以这成为一个新的输入，我们生成一个输出，然后我们得到另一个工具结果。所以第二步，我们需要对新输入计算KV缓存，并且在预填充和输出期间，但我们可以复用缓存，然后我们一次又一次地这样做。所以你可以想象，随着
[53:49.280] 你可以想象，我只展示了第1、2、3步，但当我们到第50步时，如果我们对输出没有做任何取巧的处理，这将会比新的输入、在输出中大50倍，对吧，所以如果我们能正确地缓存这个，那将是相当可观的节省。这里有一些来自实际论文的例子，你知道，它们研究了这一点，并且 基本上我们可以看到，这是不同GPU上的token长度，这是使用缓存和不使用缓存时你需要的计算量。所以同一种颜色的上面那个，或者说同一种颜色的下面那个，是使用缓存，上面那个是不使用缓存。所以基本上，随着序列变长，未缓存的那个变成二次方，
[54:44.319] 而缓存的则基本是线性的。这也体现在运行一个智能体的花费上。所以我从官方定价中提取了所有这些数字。基本上你可以看到，如果你看所有这些数字，包括商业的，通常已缓存的token是未缓存token价格的十分之一。所以这意味着你 需要非常注意你的token是否被缓存了。因为如果它们没有被缓存，你运行你的智能体将会花很多钱。嗯，那么有没有人对，如果你运行一个智能体，什么样的缓存命中率是好的，有想法？是的。
[55:33.440] >> 80到90%。>> 80到90%。>> 是的。>> 嗯，是的。所以我听到了80到90%，我听到了像超过95%。我的印象是90到95就一切都没问题。就像我，我对90到95没问题。如果比这更低，你可能没事，但你可能在某个地方有所损失。 嗯，所以，是的，所以基本上你可以缓存你的大部分token。另一个有趣的事情是，输出往往也是输入的五倍。所以这里输出和已缓存输入之间有50倍的差异，这正好反映了你可以从服务它中获得的成本节省量。
[56:28.880] 那么为什么缓存便宜这么多，也许有点显而易见。你只需要查找，你需要查找输出。你只需要查找这些，而不是实际对它们执行计算。特别是前馈网络计算在transformer中相当昂贵，所以能够跳过它是一个特别的优势。嗯，但
[56:58.880] 我们实际上如何管理这个呢？所以第一件事是，你如何把它保存在内存中？有两个最流行的开源，两个最流行的开源LLM服务库，VLM和SGLANG，它们都基于围绕如何管理KV缓存的创新。所以VLM的主要技术创新，当然他们还有更多 现在里面放的东西。它们就像是通用的服务库，但引领 VLM 的主要技术创新叫做 page detention（页驻留）。基本上它的工作方式是，你有物理 GPU 内存块，你把它们排列成类似内存块或磁盘块的形式，就像在更典型的基于 CPU 的机器上那样。所以你可能有块 零、块一、块二，这些是你实际需要计算的，他们把那个放到比如块一里，然后是块七，然后是块三，或者说抱歉，块七、块一、块三，然后在你生成的时候，你有一个指向其中每一个的索引。
[58:20.960] 他们之所以想这么做，是因为他们想要能够并行地生成许多序列，当你完成生成时，你可以把它从内存中驱逐出去，这样你就可以一次性驱逐整块，这使得计算更加高效。所以这是他们用来让这个并行服务工作起来的一个技术技巧。
[58:44.000] 引领 SG lang 的那个技术创新，具体来说，叫做 RAIX attention（基数注意力），它特别专注于复用已经缓存过的输出输入。所以他们所做的基本上就是形成一棵树，这是一棵基数树，他们用它来查找已经被使用过的前缀。所以这个 当它刚出来的时候，特别专注于像 ChatGPT 这样的聊天机器人，因为你知道，你可以让某人开始一段对话，然后继续这段对话，所以你会缓存已经完成的对话的某一部分。但现在它对智能体来说重要得多得多，因为智能体的对话要长得多
[59:39.839] 也频繁得多。所以我们需要更频繁地进行这种查找。另一个你需要考虑的事情，比如在更新版本的 sglang 中已经处理了，就是假设你在做多机服务。你在不同的机器上提供服务，也许你有一个数据中心，有 64 个节点或 128 个节点之类的 东西。每当一个请求进来时，你想要把它路由到拥有你的 KV cache、拥有对你来说最适合访问的 KV cache 的那台机器，所以你要进行实际上具有缓存感知的路由，把它路由到合适的地方，然后让拥有最佳 KV cache 匹配的工作进程来处理。
[01:00:32.000] 我非常喜欢从 Open Router 获取我所有的服务统计数据，因为他们有那么多数据，而且他们覆盖了我想谈论的所有内容，但我相信他们甚至有一个按缓存命中率排列的提供商排名。[笑声] 抱歉，数据，数据不会说谎，但你可以看到 Mistral 的缓存命中率最差，然后你 还有其他的，比如阿里云或 DeepInfra 或 Silicon Flow，它们的缓存命中率更高，这并不完全反映该提供商很差之类的，可能只是他们运行的工作负载不同，但有很多因素实际上会影响这个缓存命中率，其中之一就是缓存的
[01:01:28.240] 路由有多好。另一个就是缓存驱逐有多快。所以提供商不会永远为你保留你的缓存。他们会在一定时间之后驱逐你的缓存。有人知道对于像流行的提供商来说这是多长时间吗？你们有没有看过这个？ 接近一小时。>> 接近一小时。我认为，长时间的那个，我不，我对这个不是特别了解最新的情况，但我知道 Anthropic 以前是五分钟。所以基本上如果你和他们有一段活跃的对话在进行中，而你不暂停你的对话，你就会留在缓存里。否则，如果你停止了对话，你就会被 从缓存中驱逐出去。我认为你现在至少有一个选项可以延长到一小时。但我不知道你是否，但是延长到一小时的话成本会稍微高一点之类的。但没错，基本上他们不想把这个一直保留着，因为这是他们要保存在内存中的大量内存，所以他们想要
[01:02:30.160] 在它有用的时候使用它，在它没用的时候驱逐它。所以这是另一个可能改变你缓存命中率的策略。作为用户，对我们来说坏消息是，无论提供商在路由或缓存驱逐方面有多好，定价都不会改变。所以你可能会仅仅因为他们驱逐你，驱逐你太快而多付钱，这看起来有点不公平。 >> 嗯，是的 >> 他们是否使用类似缓存分层的方式，你可以把它存储在设备内存中，那种针对设备调整过的，然后在上面 >> 是的，所以那是个好问题。所以问题是，他们是否使用缓存分层，比如你有一个存储在设备中的快速缓存，然后有一个存储在磁盘上的慢速缓存之类的？我实际上 不知道那个问题的答案。那会是一个很好的，很好的后续研究让我们去做。开源库可能也不像闭源库那么精密，因为如果你看看 Open Router 上的所有人，他们的工作就是非常擅长服务，所以你认识，他们中有些人可能在使用开源 库，但他们会在这些库之上构建东西，而这差不多就是他们的，你知道，竞争差异化。所以我认为推理提供商内部的解决方案，在这方面很可能比开源中可用的那些要好一些。
[01:03:54.319] >> 他们通常从 GPU 卸载到 CPU。>> 他们通常从 GPU 卸载到 CPU。是的。>> 是的。因为你的 GPU 内存太宝贵了。所以是的。是的。>> 好。所以，是的。哦，抱歉。我跳过了一个重要的部分。所以，这意味着什么？如果你只是在设计一个 harness，而不是在部署模型之类的，那意味着你需要非常非常小心，不要
[01:04:24.319] 无意中破坏你的缓存。所以记住，你的缓存只有在整个前缀完全相同时才有效。所以有很多我们考虑过要做但决定是非常糟糕主意的事情，因为我们不想破坏缓存。有一件完全被禁止的事情，你不应该做，就是不要每次都改变系统消息 呃，你走一步，因为那基本上意味着没有缓存。呃，你没有任何一条消息是可以缓存的。所以一件很合乎逻辑的事，你可能会想到，就是好吧，我想把当前的时间和日期附加到系统消息上，呃，这样智能体就知道现在几点了。不要这么做，因为时间和日期每一步都在变，所以那会 破坏你的缓存。所以如果你确实需要附加它，就把它附加到最后一条用户消息上，呃，或者你得到的最后一条呃工具调用结果上。呃，还有呃，比如在对话中途呃动态地更改你的呃，更改你的工具。你可以这么做。那没问题。呃，另一件事是，一件很合乎逻辑、你可能会想到去做的事，就是每一次都路由到不同的模型 你呃，每一次你走一步。所以就像是，这一步看起来很适合一个你知道的，一个非常强大，一个非常适合非常强大模型的一步。呃，比如呃，比如我想在 Fable 上做这一步，然后我想把接下的三步放在呃，放在一个更便宜的模型上以省钱。但问题是，你仍然在付，你下次用呃 Fable 的时候，仍然要付你知道的 10 倍的
[01:06:09.760] Fable 成本。所以如果你想要通过切换来节省成本之类的话，你真的需要在你要做的切换量上非常呃节俭。好。呃，那么最后一件事我想谈的是上下文压缩。呃，有多少人知道，知道这个？在日常的编码智能体使用中用它。 有多少人在它说正在压缩上下文的时候真的很兴奋？[笑声] 一个都没有，很好。那么呃，上下文压缩背后的基本想法是，呃，你有一个呃之前，呃，这就是你知道的你可能有过的所有观察和行动。呃，然后是之后，你倾向于做的是，你倾向于把你之前的上下文拿来，把它碾压压缩成仅仅是 之前上下文的精华。呃，所以人们之所以像是对使用这个并不兴奋，呃，也许很明显的原因是，如果你在这件事上做得不好，你的智能体就会忘掉东西。它开始做一些完全不同的事，并且呃，也许会删掉你的文件系统。
[01:07:30.880] 所以呃，我们可以把压缩看作本质上是在试图想出一个之前的表示，使得你未来的行动序列，呃，你把你的历史拿来，你压缩它，你得到一个工作状态，然后你希望你的未来行动序列在压缩之后，呃，与之前相同，或者你知道的相似，或者可能更好。呃，与此同时，你还有一个 约束，那就是你的状态需要是有界的。所以它需要足够小，以至于你乐意继续工作。呃，所以如果我们看看在压缩中幸存下来的是什么，呃，有不同的做法。我会讲人们做这件事的不同方式，但一个常见的策略是取一些锚点，这些可以是比如用户在最开始 开始会话时说的最开头的那些内容。呃，所以，呃，当我问那个问题，智能体会忘掉什么的时候，我听到的答案之一，是有时它们会忘掉会话最开头的东西？而那实际上可能是你所使用的智能体框架里的一个设计缺陷。呃，因为呃，一些好的智能体框架 呃，会有意地避免删除会话最开头的东西，因为它们认为那更重要。呃，但这也可能是，你知道的，取决于情况而无法避免的。呃，另外你希望中间那些被编码并放回上下文里的东西也在。 然后最近的尾部通常也会被保留下来。所以人们经常会保留最近的那些动作。与此分开的是，我不太常见到这个，但呃，你也可以把证据加到一个证据存储里。实际上，不，抱歉。我，我纠正一下这个。呃，证据通常会被加到一个证据存储里。呃，至少，如果你在使用一个 CLI 编码
[01:09:47.279] 智能体，通常你的完整历史会被保存在磁盘上的某个地方。所以，如果你知道你的编码智能体把所有这些历史保存在磁盘上的哪里，即使你因为压缩丢掉了东西，你也可以告诉编码智能体回去读历史，然后它就可以，你知道的，呃，替你读历史，找到之前的那些结果。呃，但也有呃，可能更
[01:10:09.120] 结构化的做法，所以那样的话，呃，你知道的，其他东西就可以被丢弃。呃，所以有几种决定呃压缩策略的方式。呃，第一个是触发条件，呃，基本上有两种做法。呃，一种是 token 阈值。所以你对呃 token 阈值设个限制。呃，所以你可以说我，我只打算用到 200k 个 token。一旦我超过 200k 个 token，我 我就要压缩上下文。另一件事是呃，比如当提供商强迫你这么做的时候。呃，所以那差不多是你能拥有的 token 数量的硬性上限。呃，最后，你也可以有一个手动请求。所以呃，很多呃，比如编码智能体工具包都有一个 /compact 呃，像这样的命令，你可以用它来压缩上下文。
[01:11:08.719] 呃，然后基于那个，呃，你会选择，你可能会保护开头并保留最近的尾部，然后压缩更旧的部分，那个摘要可以是一个结构化的摘要。所以有时呃，它们要求一次工具调用，呃，它说这个、这个、这个和这个都必须出现在压缩摘要里。有时它就只是自由文本。 呃，还有呃，我认为这只有 OpenAI，呃，可能吧。但基本上他们呃，他们压缩成某种加密的东西，呃，你没法读。所以你看不到他们是怎么压缩的，因为他们不想让你偷走他们呃，他们宝贵的压缩算法。呃，所以呃，所以就是这样。
[01:11:53.360] 呃，然后呃，是的，你可以，你可以继续在那上面工作。呃，所以这里的一个风险是，呃，有时你可以一遍又一遍又一遍地压缩，第一次它有效，呃，但当你走过更多的时候，呃，你，呃，你最终会丢掉越来越多。所以再次，这是人们不喜欢压缩的原因之一，因为它可能导致智能体脱离正轨。
[01:12:22.960] 呃，所以呃，基本上我们评估这个的方式通常是在一个整体状态里。所以你可以改变压缩算法，并针对你正在评估的任何基准来评估它。我想警告你呃，关于这个。我有一个，有点像我们最初开发 OpenHands 时的一个故事，就是呃，我们实际上，我想，做了最早的有点像 生产级的呃压缩算法。我们当时把它叫做 compression，然后嗯我们用 bbench 来评估它，这是一个非常常见的、你知道的、用于编程 agent 的基准测试，我们在这上面花了相当多的功夫，大概作为一个研究项目搞了一个月左右，然后我们觉得我们找到了一个很棒的压缩策略。它完全不降低我们的分数。它 帮我们省了很多 token。然后我们实际上在嗯我们实际上在我们的系统里试了一下，结果它犯了这些特别傻的错误，比如嗯每次压缩的时候就发一个 pull request。它忘了自己已经往 GitHub 发过 pull request 了。所以呃你知道我们会因为同一个功能收到五个 pull request，因为它忘了。呃或者你知道它会忘掉 对话中间发来的用户指令，因为我们只在单轮对话上评估过，那种只有一条用户消息的对话，而我们在压缩的时候把多条用户消息给压掉了。所以光做基准测试是不够的，呃除非你有非常非常健壮的基准测试，能覆盖每一种使用场景。你多少还是需要在真实环境里测试它们。
[01:14:03.520] 好。嗯所以我们快接近时间了。呃不过是的，基本上呃我今天讲的是呃你知道 agent 循环增长得非常快。呃我们在这个过程中有很多推理方面的挑战。呃我们可以通过架构来应对呃效率问题，嗯还有缓存，呃而压缩差不多就是我们应对这个问题的方式，呃一旦我们扩展到超出 我们能处理的水平。所以嗯我要讲的就是这些。呃如果有什么问题，很乐意很乐意解答，下次我们要讲的是 skills 和 memory。所以这就像是更长的上下文，贯穿你整个呃你整个时间。嗯，在我们都呃都离开之前有什么问题吗？>> 是的。
[01:14:52.960] 好问题。嗯，问题是，我们是否把 sub agent 看作一种管理上下文的方式？是的，但我们后面会讲 sub agent，所以我这次没有涵盖它们。是的。
CMU 11-768 Fall 2026 · Lecture 3 · Agent Capabilities 2: Context Management for Long-Context Agents
- 视频：官方视频
- 字幕：en-orig · auto captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：用户审核通过；中英文按完整语义段合并，每段保留首条原字幕时间戳。
正文
[00:07.120] Okay. Uh hello everyone. Can you hear me in the back? All good. Okay. So, uh well, welcome back everyone. Am I uh echoing? All good. Okay. So, um, today I'd like to talk about context management for long context agents. And I at first thought about titling this as like context management for long context LLMs because most of the stuff I'm going to be talking about today is kind of more general LLM stuff, but long context is super super important for when we're working on agents and it's not necessarily covered uh, very well in all of the LLM prerequisite classes. So, I'm going to cover it in a fair amount of detail today. I'm also going to try to go through kind of the long context architectures that are used in many of the kind of state-of-the-art open uh
[01:00.640] models so you can kind of see the best practices that everybody is using in order to handle a very long context. So first I'd like to talk a little bit about um how long an agent context can grow. So basically the input is quadratic uh from the uh or the number of token accesses that you do is quadratic uh based on the linear history growth. So um if each call uh you make for an agent takes about you know five tokens and let's say you start out with a an initial system prompt of one token uh you'll get six tokens in the first call 11 16 uh 21 and 26. you know, this will increase and increase and increase. Um, and so by the time you have gotten to the end, you will, if you do things completely naively, you will have had to process ADK tokens uh for these five calls. And you know, this can continue for a very long time. I I don't know the longest anybody has used a single coding agent session or something like that, but I've certainly done them for like multiple days uh where I've been working with it for almost the entire day. So you could imagine this could get uh very very large. Um the actual computational
[02:22.959] complexity of this is uh is actually even larger than this. Um well no sorry. So yeah the the computational complexity is is quadratic here. So why do agent prompts get so long uh so quickly? I did a little bit of data analysis on uh the composition of prompts that I used in um uh 1,500 sessions from our uh coding agent openhands. And each each session was relatively short.
[03:00.000] These are not like the long multi-day sessions that you do. This is more like solving a single task. But uh each one used about 78,000 tokens on average. Um 23% of this was a system prompt in all of the tool descriptions. Uh this is relatively large for a coding agent. I think our system prompt is somewhere on the order of like 15,000 to 18,000 tokens because we're very opinionated. Uh the open ants agent is very opinionated about how people uh do work. Um these tend to vary from like 8,000 to uh 8,000 to 20,000 tokens depending on you know how many tools you uh you pull in and stuff like this. Um separately from this uh 9% were user messages. So this is the user either saying the initial task or following up with uh additional information after that. 9% were reasoning traces um and you know where the model is thinking through things. 2% were model replies. Sorry, this isn't showing on the slide, but 2% were um the model replying directly to the user. So, it's spending about um five times as many tokens freezing as it is spending communicating uh with text to the user. And 20% were tool calls. Uh
[04:10.879] so, you know, uh you know, writing code, uh reading code, that sort of thing. And 37% were tool results. So, this is uh you know, getting the result of, you know, reading a file or something like this. So you know each step can uh will consist of all of these and that tends to add up uh very quickly.
[04:34.080] So when you're handling this very long context uh there's two big challenges here. Uh the first one is capacity and so whether your model is able to handle this large amount of uh evidence in the first place and this is a challenge uh because as I'll talk about in a little bit training models to handle very long context is hard um and so in order to manage this you need uh things like architecture position encoding training data uh and uh you also need to be able to evaluate whether you're doing a good job of this. And then the second thing is efficiency uh and whether the system can afford to serve it. And there's a bunch of techniques to do that as well. And I I think some people might be taking this for granted if you're just using, you know, cloud code or codeex or something like this, but they're doing some pretty monumental feats of engineering to take a, you know, multi- trillion parameter model and serve it within seconds uh to uh over and over and over again for mil I don't know codeex has 25 million users. So you can imagine uh 25 million daily users hitting it every few seconds
[05:43.840] for this huge model. Uh if you want to you can calculate in your head how many flops per uh per second that might be. It's a lot of flops. Cool. Um so any questions or things? Probably pretty straightforward I guess. Okay. So uh let's go into inference. And so in order to make this happen uh the you basically need to be able to um calculate uh prefill and decoding. So uh calculate your inputs and your outputs at each step. And this uh you know I'm sure everybody has done in your previous classes uh for generating from uh you know a language model. some things you may or may not have thought about in your previous classes. If you took how many people took like uh language model systems or deep learning systems or something like that actually more than I thought. So if you took the systems course you you probably handled this. Um but other things you need to think about are how do you schedule uh multiple concurrent queries? Um and also how do you save the results in the KV cache uh so you can reuse them and not have to reuse them over and over again. uh not have to calculate over and over again.
[07:05.599] So going into a bit more detail here, um we have a few phrase uh phases in inference. The first one is called prefill. Um you could also call it encoding. Um but I think pre prefill is kind of the like preferred term of art nowadays. And so you take in all of your input tokens uh that you don't need to generate and uh you generate the key and value pairs uh from each of them. And then uh when you're decoding you basically uh take the prompt uh and key value pairs um generate an output and append uh the key value pair that you get from the output. So um you do this kind of incrementally whereas this can be done in a batched sequence.
[07:53.520] And so uh when we think about this as a serving system, there's a bunch of ways we quantify how good a serving system is. Uh the first one is time to first token. So this is how long you need to start um you need to wait between when you start uh entering a query and when you start seeing the first output. And so like I'm sure everybody you know who uses chat GPT or something like that uh you put in a long output you notice it takes longer to start getting a response than if you put in sorry put in a long input you see it takes longer to start getting a response than if you put in a short input and that's like your TTFT is higher. Uh then uh your time per output token is the uh time like the amount of time it takes uh between generating each token. And if you go to a website like open router um t typically uh here this will be your time per output token there I am pretty sure they're not counting prefill here and it's just like the time between uh each of the tokens that might be a little bit small with the black background but you can see that um for GLM 5.2 to uh you have like 63, 48, uh 24, uh that sort of thing. And you know, some providers are drastically better than others. Like this provider, not to pick on Mistl, but they're getting, you know, three three tokens per second and then these other ones like Parasel and Coreweave are getting, you know, hundreds of tokens per second.
[09:32.720] So um so basically uh you know this makes a huge difference in your experience using um uh using agents. Uh one other interesting thing is um there's different ways people can use agents. And so in some cases you might really hear about this. So like let's say you're working together um with a an agent on a interactive task in your favorite coding CLI. you may really really care about, you know, TTFT and Tapot. Um, if you're running an agent in the background to do some background tasks that run overnight, you might not care at all. Um, and there's trade-offs between them, between like throughput and cost and how fast uh you can get the the tokens. And like I you know also um serve coding agents uh with my company and when I talk to providers they'll often often ask which do you care about more so we can optimize the system uh for the one you care about more and so um there's uh there's ways you can do that and I'll I'll talk about those too. Cool. Um yeah so sorry just to finish the slide. So throughput is um the completed tokens or requests divided by the wall
[10:54.880] clock time. So this is kind of seen usually on a more holistic level over the whole system. And then uh the cost is you know how much it costs. So as a refresher um I I hopefully everybody should know this already. Um but you have uh the uh query key and value calculation in uh typical attention and this is uh so you have this here and then you take the softmax um divide it by this. You might have some sort of uh you know positional uh bias and then you calculate your output and typical full attention uh looks uh like this. you mask all the previous tokens and and or you consider all the current tokens and you mask all the um the future tokens.
[11:44.720] Um so this is of course very expensive because it's quadratic in the sequence length. Uh and so a lot of the stuff I'm going to be talking about today is about how to fix that. Um so this is on the efficiency side and then on the effectiveness side just because you have a model or just because you have a serving system that can take in you know a million tokens doesn't
[12:09.519] necessarily mean that you can use that a million tokens effectively and um this is a very like old school example of this I guess this is a needle in the haststack evaluation how many people have seen this figure for maybe some but not not that many. Okay, so to explain what this looks like, basically what this is saying is um it's a very weird evaluation. Basically, they um they took a whole bunch of Paul Graham essays uh by the like famous entrepreneur and head of Y Combinator Paul Graham and they stuck in uh something about like the best place in San Francisco to eat a sandwich is something or other and then they ask a question where's the best place in San Francisco to eat a sandwich and then the language model has to answer that correctly. um and they made the context longer and longer and longer and then they evaluate uh models with the longer context and eventually the weaker models start getting worse at these very very long contexts and there's a bunch of examples of things like this. Um ruler and helmet are kind of the the common benchmarks that people use here. Um but basically uh the long the summary is
[13:21.440] that if you have very long context your models might start getting worse at processing them. So has anybody noticed this when you've used a coding agent for a long time like are there common or used any agent for a long time are there common like failure cases that you see? What what kind of things what kind of things have bothered you? Anybody? >> Yeah. >> Forget the previous context. >> Yeah. Forgets the previous context. Did that cause any like problems for you? >> Yeah. with me telling it not to do something in the middle of the conversation and then it goes and does it. Um there there was a really bad example uh where I
[14:16.160] think actually Daniel talked about it here where the person had said do not delete this and then it went and deleted it anyway because it had forgotten the context. So yeah, this can be a really big problem in agentic settings because you know the user is giving these incremental instructions. So so I'm going to be talking about several different things. I'm going to be
[14:36.079] talking about architecture. I'm going to be talking about training inference and compaction. So going first into architecture. Um this is both about efficiency and accuracy. And so as I mentioned standard attention is global. So it makes pairwise comparisons across uh all of the tokens here. And um the problem with this is uh like this is quadratic obviously. And does anyone know the like standard length of the of models nowadays of these uh of like state-of-the-art language models? >> 256 to a million. >> 256 to a million. Yeah. And I I'd say most people are aiming for a million now because they don't want to be the, you know, the model that has 256 when everyone else has a million. So very quick math in your head, how many
[15:30.880] pairwise comparisons is that? >> 10 12 >> 10 10^ the 12 or in English. >> A trillion. Yeah, that's a trillion. That's a trillion comparisons every time. uh like when you're uh generating a sequence of a million long and each comparison uh requires you to multiply large vectors by each other over and over and over again. So you're starting to get up into the you know 10 to the I guess it'd probably be 10 to the 20ish uh which is a lot that that's a lot of compute um for a single sequence. So um there there's ways that people solve this and basically almost all of them fall into the category of taking this quadratic computation and putting some sort of constant limit on the number of comparisons that are made uh at each time step. And that brings your
[16:30.639] uh n squared down to owl where I I wrote w for window but it doesn't necessarily need to be a window. It just needs to be the cap on the number of comparisons that you're making. Um so there's the idea of local models and global models. And so local models are are basically models where this cap on computation happens within a local window. And then global models you have a cap on you might or might not have a cap on computation but that cap on computation is not localized. So you can pass information uh throughout the entire sequence. And the I'd say almost all of the models uh that exist nowadays or are widely popular nowadays are some variety of hybrid model where they have many local computations and then one global computation then many local
[17:21.679] computations one global computation and specifically I'm going to talk about six uh models that we have here. So uh the most recent Quen model uh has uh local computation of uh gated delta net. I'll explain all of these uh or most of these in a bit. And then um sparse retrieval uh for the global computation and the ratio is 3:1. Uh GLM 5.3 is uh also a variety of delta um and sparse attention uh with 3:1. Um uh Kim K3 is uh a variety of Delta and um this is a dense uh dense attention 3:1. Uh Neimotron is Mamba uh which is uh kind of like recurrent style model and uh dense attention 4:1. um to uh window uh sliding window attention uh dense attention 5 to one and deepseek's a little bit different. It kind of has uh window windowed attention but then some uh sparsity uh sparse stuff with interle branches. So um basically the the standard way of doing things is hybrid model uh with some sort of local thing and uh either sparse or dense uh global attention.
[18:50.000] So I'm I'm going to be talking about this. I'm not going to go into like a ton of details for all of them, but uh it should be enough that you could be able to look them up later. So the first one is uh sliding window attention. And this is maybe the the simplest conceptually. And what you do is you take chunks or uh you take chunks of the output and basically you're able to attend to your current chunk and um all of the previous chunks and uh or not all of the previous chunks, a fixed number of previous chunks. So if this is your current chunk, um it might be a chunk of like 512 tokens or something like this. You would have an additional 512 and 512 tokens in the previous chunks and you would attend to those. Um so pretty simple, no changes to attention, but you have something like this. But the nice thing about this is on the first layer you're passing information from the previous one or two chunks, but on the next layer the uh previous chunks also have information from their previous chunks. And so you kind of get this gradually expanding window. Um so uh this uh this is pretty widely used.
[20:08.880] Um but the more popular way of doing things nowadays is using some variety of linear attention. And so the way uh linear attention works is basically we take our uh standard attention and standard attention has this um softmax operation included in it. And this is the reason why it's hard to compute because you're taking the exponent and um and dividing. And because you do this there's you basically have no choice but to expand all of the uh key and query computations uh like key query multiplication computations in and be uh quadratic uh quadratic complexity. What linear attention does is you basically start from this uh kind of standard attention formulation but you take you remove the softmax and the normalization here or I mean the normalization doesn't really matter but you remove the softmax and you get just the key query multiplication and [clears throat] this is definitely less expressive because basically what you're doing is you're moving from something that takes in a bunch of uh like numbers. Anyway, like pretend pretend that this bottom one adds up to one, but you're going from a thing that takes a bunch of real numbers and normalizes them into probability like numbers that are between zero and one and add to one. Um, so this nonlinearity allows you to focus on like only some of the previous uh like tokens. And this is kind of the good point of attention. That's the
[22:15.600] selling point of attention that allows you to focus on the relevant previous context. And you're basically taking away that ability, right? You're you're removing the relative comparisons and just getting um the the dotproduct between the queries and keys. But it allows you to do something really uh nice which is that you can use uh the associative
[22:38.880] uh you can use associivity and basically rearrange so that um you can convert the sum of all the key and value uh multiplications before into a single uh state matrix. And so this state matrix is essentially a function that map it's a linear function that maps from a query to a value. So attention is also a linear function that maps from a query to a value. So if I go back to this here you're basically calculating you're taking the query you're calculating the attention over all the keys you're multiplying it by the values and you're getting a value here. Um and this state matrix is also that. But the nice part about it is this state matrix can be decomposed. So you can update it. You can keep this state matrix around and update it in constant time every time you get a new value. So this essentially becomes like a recurrent neural network. So it it allows you to update the state matrix um in linear time without having uh without requiring the quadratic computation. So this basically solves your quadratic problem and it solves it so much that um O of N squ becomes O of N.
[24:00.799] There's no like there's no even W term in this because uh each time you're uh you're updating based entirely on the previous state. Um but there's there's caveats here. So um is this clear though uh so far? Okay, cool. Um, so the big caveat is exactly the caveat I mentioned before. Like this is just a less expressive function. Um, and it doesn't allow you to make like relative comparisons between the previous tokens, that sort of thing. So there's a bunch of uh workarounds around this to try to make a more expressive function that nonetheless is useful in um in kind of uh doing attention-like calculations. And so um most of these are if you dig deep into the math in them, they're basically trying to become better predictors of the final value
[24:56.559] that you would get from attention. Um and you you can dig into the papers to see how that works, but I'm going to explain them mostly uh mechanistically here. And so um the first thing is Delta. And the idea behind deltaNet is um you can predict what um you can essentially predict what value you would want to be getting from the uh from the calculation of uh KT um and S. And so you know KT here is uh the key at time step t and you are comparing it with the matrix at time step t uh t minus one and uh you're calculating this v hat t and on the other hand you um so basically this this is the prediction that you would get of kt But on the other hand, you know what the uh output VT should be uh based on um what the query times uh or based on uh what the Hang on one second. [laughter] I don't want to I don't want to get this wrong. So you calculate KT with respect to the matrix. And you can view this matrix as a way of uh calculating the uh the way of calculating the mapping from a query to a value. And you can also get the the actual value and subtract out uh what the value you get when multiplying it from the previous uh together with the previous matrix is. And this is essentially the error uh that you get here. And then you want to reduce the error. And so in order to reduce this error, what you do is you um essentially add in this uh this term which is KT uh times the error term. And after you do this, if you multiply in KT to the new matrix, you will no longer have any
[27:20.240] error. But you don't want to make an update that is too big. So you add this beta term. This is essentially a learning, it's similar to a learning rate. And this learning rate will make the updates uh smaller. And so uh if you if you do this, this allows you to basically nudge the matrix in the direction of uh the in the direction that will cause a query that looks like the current key to output the value that you want essentially. So it's it's reducing the error uh reducing the error in the prediction of the value that you want. So um as I said this is going to be a little bit high level. Um the the papers are are pretty good uh descriptions of this.
[28:10.000] Um so after this um this basically what this is doing is this is pushing you in the direction of the uh of making the the matrix a better predictor of the uh the value. But one problem with this is you can build up lots and lots of context uh from the previous uh spots and linear attention doesn't have any mechanism to prevent very far away cont uh context from having a bad influence on your next predictions. And so if you have like a very weird key far back in the future, it might have a very big influence on you in the f uh in the future. And so the way they fix this is basically they have a decay term to make uh to make the matrix decay a little bit at every time step. And by decaying the matrix a little bit at every time step, they're basically discounting the the far in the past context. And so you multiply um the matrix by alpha t and then you add this uh st tilda into uh into your calculation here and update the matrix. And so the stronger you make this decay basically the the more quickly uh it will forget the previous context. And so you can make this more local or or less local.
[29:47.360] And so this is uh gated delta is used in the Quen models. So it's one of the more uh the more popular methods now. Um but both uh GLM and uh KI which are kind of two of the state-of-the-art models use a slightly more expressive version of this uh where they basically have a per element uh decay term. So each element can decay at a different speed. And uh because each element decays at a different speed, you can have some uh portions of the context that uh you know carry over very long time periods, some portions of the context that carry over very short time periods. Cool. So hopefully this was more or less uh more or less legible. Um any any questions here?
[30:39.600] Long long story short, it's basically a way to turn uh into um it's a way to turn the idea of attention to something that you can calculate um in context. Mamba is uh something that's very similar um but it's a little bit more mathematically involved um uh and I I decided to skip over it. uh I covered it more in the uh language model uh description course but similar idea but
[31:10.000] just a bit more complex uh in the math. So the next thing I'd like to talk about is sparse attention. And so sparse attention is uh basically looking up a not attending to everything in the previous context but only attending to some things in the previous context. And essentially you limit the number of things that you look up in the previous context. There's a few ways you can do it. Um originally in there was a sparse attention paper by OpenAI uh quite a while ago like 5 years ago or something like this. Um and basically uh what they did was they had sparse attention every you know uh 512 tokens or something like this. And what this does is it means that you still have a fixed it's still quadratic in the sequence length but
[32:03.279] it's 512 times less than if you attended to every token and that's like good enough uh that you can manage to calculate it over the context that you're interested in. Another option is to do um content dependent sparse attention. And basically what this does is you have a very cheap way of calculating attention. Like for example, you can down project the attention vectors to be very very small and then you do attention over these very very small uh attention vectors. You pick the top k ones and then you do full attention over the top k. So uh that's uh another way that you can do this and this is what's used in deepseek.
[32:43.600] Yeah. >> How do we define the metric for which we do the top similarity compared in this in the shrine space we just do like coine distance. >> Yeah. >> Yeah. I I think it's typically not cosine but it's just product but yeah >> I I I think although maybe maybe they're normalizing beforehand. They actually don't know that detail.
[33:10.080] Any other questions? So basically um these th this is a very very cheap way to achieve global attention because you can just uh write a kernel that will do this for you. Um this over here will uh require you to have two passes, but you know it's potentially uh more accurate.
[33:35.039] So um I I've seen both of these used pretty widely. So a final um a final thing is KV compression. How many people saw this in a language modeling course? Maybe some people. Okay. Um so I will uh I'll explain this. So basically um this is standard multi head attention. And so in standard multi-head attention, you have a bunch of heads and each one has values and keys and all of these are a certain dimension. So they're like dimension, let's say they're dimension D and you have uh you have like a certain number of heads. Let's say that's H. Um so the two main ways that people solve uh or help uh this problem of having very uh large and expensive KV caches is uh grouped query attention. And grouped query attention basically uh just uses a smaller number than h um you usually h divided by some uh power of two uh number of uh keys and values. And because it's the KV cache, it's not the KVQ cache. Um if you reduce the number of uh keys and values that are used, you reduce the amount of memory that you need to hold the KV cache. This is really important
[35:02.880] because if it gets very large then um it limits the number you can fit on one GPU and so this can usually be reduced by like four or by eight or something like that. Um multiquery attention is just like grouped query attention with a single one. So you can kind of almost skip over that. You already know what it means. Um multi head latent attention the the um is essentially down projecting uh the size of the keys and values. So it's not reducing the number of heads, it's making each head smaller. And so this is similar to what I talked about with the top K attention. Um and of course you can combine both of these together as well. You could have group query attention with um you know smaller uh keys of values.
[35:57.280] Uh any any questions here? Okay, cool. Um so the these are the method the architectural methods that people use to handle very long contexts and with these um you can handle up to a million uh tokens. So like this is uh you know this makes the efficiency possible.
[36:22.720] The next thing is uh training long context models. And this is actually uh pretty hard. And the reason why is because um if we want something to handle 128k tokens or even more, there's actually not that much data on the internet that is 128k coherent tokens. Um it's actually a lot of a lot of data. it's more than most Wikipedia articles, you know, um you're starting to get in the in the order of like having a book uh that you need to have and there's not that many books on the internet. So, typically what you do is you start out with like short pre-training. Um you do more continued training with longer documents and then you do long adaptation. Um I wrote packed sequences here. Um, but packed sequences are are useful can be useful for efficiency, but they're not super useful for learning long contexts because um every time you like let's say you pack 10 documents into a sequence, you're not learning to attend across the documents and in fact you probably should not be. So um you really
[37:33.440] need coherent uh long sets of data to do adaptation to very long contexts. Um so I'm going to talk about a few things that we do here. Um the first thing I'm going to talk about is uh handling positional encodings. Um so absolute positional encodings uh I I hope everybody has seen in their language modeling class. Um but basically uh each individual entry in the uh each individual word embedding uh gets a position vector and that position vector is typically calculated by uh some sort of function like using signs and cosiness and other stuff like this. So hopefully you know most people have seen this. So I'll I'll kind of skip over that. Also maybe most people have seen rope. Are people generally familiar with rope? Okay, I see a fair number of
[38:32.079] people. I see actually less people uh than I would have expected, but um uh maybe people are just uh not interested in raising your hand. Um [laughter] but uh but basically um rope rope is pretty elegant mathematically in my opinion. Uh basically the problem they were trying to solve is they wanted to have a positional embedding. um where the positional uh or positional encoding where the positional encoding is entirely determined by the relative distance uh between two tokens. And the reason why they wanted to do this is they didn't they didn't want the absolute position to have an effect on uh they didn't want the absolute position to have an effect on whether two tokens would attend to each other. And so this is this has been quite effective and a lot of uh a lot of models use rope. Um the the math I'm not going to go through here because it's kind of not the main point, but they do some interesting stuff with rotations and imaginary numbers and stuff like this.
[39:51.760] Um but there's also this like super counterintuitive uh nope. which is uh don't don't use positional encodings at all. Um and if you took your uh class on transformers uh you might have heard this scary stuff which is like you need positional encodings for transformers because otherwise something in position one in position five will have exactly the same representation. So if you have like this this is a cat but this is not a cat it will attend to both of the cats the same amount or something like this. Um theoretically when the transformers were originally created um this might have been true but now actually it's it's not true anymore based on the transformers we use now. Um anybody have an idea
[40:50.160] why this is the case? I've kind of given a hint in the slide but you might also need to know a little bit of the history of transformers uh which is uh before we were using encoder decoder models uh where the encoders encoded all of the input and then the decoders uh generated the output any idea >> because we're doing causal masking exactly so um to elaborate on that a little bit if I go back to our transformer equation Um, we have the keys um, and the queries. And if we look at the queries, even if you have cat at position two and cat at position five, the cat at position two is only attending to the previous token where the cat at position five is attending to the previous four tokens. Um, and so both of them are like
[41:45.680] actually different. um starting at least in the first layer of attention. And so you can tell like even if they are exactly the same word, the embeddings starting in the first layer of attention will will not be different. And there's kind of uh this counterintuitive paper that demonstrates that you don't need uh positional encodings at all. And this is especially true um this is in auto reggressive uh transformers. In non-auto reggressive transformers where you're attending all the tokens are attending to each other. You still do need positional encodings. Um this is especially true when we start talking about like gated delta because gated delta has like the decay term and uh other things like this which causes
[42:30.160] each uh token to be different. So once we start having those sort of local biases, we actually don't need positional uh encodings anymore. Um and if you uh if you look um we uh the first two here have rope uh or at least partial rope. The second two have nope uh no positional attend encodings. Um the the fourth one uh relies entirely on implicit order and
[43:02.079] mamba and some of them in uh and inkling has relative positional encodings. So you can see that it's kind of like half split between having relative positional encodings and not at all. But like let's say we did have positional encodings. Uh one thing you need to worry about for long context is uh position interpolation or position extension. And what this means is if you have a pre-training range, um, you need to essentially extend your model so that it's able to handle longer sequences. And so the the promise of rope was that rope relied only on the relative position uh between the different tokens in the output. But that that's fine as long as the relative position is only um well okay there there's two reasons why this is not fine. So number one if you're training on sequences of length L and then you extend to sequences uh of length SL then it's never seen two tokens attend to each other over a distance of more than L. So the tokens that are more than L away from each other are going to have problems attending to each other and you're not going to be able to uh manage that very well. Um the other problem is
[44:26.560] that actually relative positional encodings are only one part of the story and like actually the auto reggressive mask has an effect in all these other things. So you can't just take rope and expect it to work uh really well on longer sequences. And so the typical way you fix this is there's a parameter in rope that basically uh let me see if I have it in my slide. So I I don't I don't really have it in my slide, but there's basically a um a parameter in rope theta that uh tells you how much you rotate at each time step. and you divide this theta by the uh multiplier by which you extended the sequence. And so basically you're stretching the rope position so that the rope uh position at position zero would um or the rope position at position like
[45:22.960] five would be 5s uh would be equivalent to the position 5s in your uh extended context. And there's also um a more sophisticated way of doing this called yarn uh which basically is uh similar in spirit uh to like the delta uh the ki delta attention um which basically said that we want some of them to be very long distance and some of them to not be very long distance. In yarn essentially you have some of them be uh extended more and some of them be extended less and so you don't um you don't necessarily interpolate all of them at the same frequency. So um yeah in the interest of time I'm not going to go into like a lot of detail about this but basically these are um uh these are the various methods that you can use here.
[46:27.040] So, um, I'd like to talk a little bit now, um, taking a step back from the detailed modeling stuff and going into the data that you can use to train this. So, um, as I mentioned before, packed examples, you can just pack a whole bunch of stuff in there, um, a whole bunch of documents in into your context length. And this will work at training models to be able to process things near the end of your context window. But what it won't work for is the kind of global coherence. And um so there's a number of different ways that people do this. One way is just to find um very long documents uh like books or something like this. There aren't a lot of these uh but there are books. There are code bases and code bases tend to be coherent. So you can just download a big codebase and that will definitely have more tokens. Um multi-document corpora of some kind. So, you know, like uh you can get a whole bunch of documents that are on the same topic and then at least you'll have topical coherence between them. Um I've also seen people train on patents because patents have like there's lots of public data for them and you can kind of link link them together. A big source nowadays is agent trajectories because you know agents can make very very long trajectories if you just interact with them for a little while. So that's another one. And then you can also create synthetic tasks and a lot of the evaluations are synthetic tasks. So you can do that as well.
[47:59.599] Um and I I have some examples of how the models do this. Um a lot of the model training companies don't like to share these details with you because this is like if they release their um well so number one the closed models like anthropic open AAI you know those places uh uh Gemini they will not release even their architectural details the open model companies have to release their architectural details otherwise you can't run their model so what do they keep secret they keep their data mixes and their their training strategy secret so um this there's a lot less detailed stuff that we can say. Um, but there's uh all of them explain all of the ones in here explain a little bit about their data mixtures and they are things like uh cleaned up sampled documents and video um documents, repository files, issue, PRs, uh long document QA, scientific papers and technical reports. And then uh they also create a bunch of synthetic tasks. And typically they have a scaling strategy of like 8 64 256 1 million uh that that sort of stuff as well.
[49:10.640] Oh yes. Um [clears throat] one other logistical detail that you might need to deal with um or might not need to deal with depending on the size of the model you're training. like for example for your project if you want to start dealing with um uh larger models is context parallelism and basically what this means is there's a bunch of uh ways to do parallelism when training models I
[49:36.319] wrote context parallelism up here what are two other ways uh that you can parallelize models during model training or serving >> yes >> data parallelism >> yeah you you said a more specific example of the second one but data parallelism and I would say model parallelism in general and model parallelism can have lots of varieties. It can have uh expert parallelism where you have different experts on different machines uh tensor more generally tensor parallelism where you split um uh the uh the parameters and you can also have pipeline parallelism where you have different layers on different uh different devices. But context parallelism starts becoming import more important for agents because when you train agents you want to train them on very long contexts. And so uh the way this works is you will compute like one part of the context uh the keys and values. So you send it to the next device calculate the keys and values for the next for that part send it to the next device uh and etc etc. So um this is uh very useful. It's really important if you start getting above like 32k
[50:47.680] 32k context or so. Um, but it's also a pain because you have the um you have all the modeling people saying I want my I want my KI delta attention and then you have all the info people saying but wait I need to implement a context parallel kernel for KI delta attention and make sure it actually works and you can back prop through it and all of these other things. So um I I personally had an experience where I was trying to train uh Quen uh for uh like a smaller quen model like 135 uh 35b and uh I I wanted to go do use context parallelism on it and it's like there's no kernel for gated delta net anywhere on the internet and so um you need to uh you need to either go in and implement that kernel or decide to do something else. So um th those two can conflict some of the times. So be be aware I guess um the the good news is for like standard attention and sliding window attention and stuff like that all of those will have uh kernels more easily implemented. The hard parts are where you're like not doing a standard attention operation.
[52:06.240] Cool. Okay. So the next part I'd like to talk about is prompt and KV caching. This is uh very important even if you're not the person who's implementing the language model, but also if you're just a person who's um like running agents uh yourself, but I'm going to talk about uh the the technical details of how it's implemented. So, you know, at least you you have a good uh idea about how that works. So um basically when we I I simplified this a little bit but when we're generating an agent call we have an input uh we have an output uh so this could be like the the messages and tool calls and then we have the tool result and so so as we calculate the input we do prefill and as we calculate the output uh we do uh basically decoding Right? And so
[53:04.960] during prefill and decoding, we have to calculate the KV cache. This is all done in a batch. Uh this is done every time we generate a token. But after we have that, we have the uh key and value uh for each of the positions up until the end of the output. And so what we can do is we can throw that in a cache and then we don't need to calculate it anymore. The tool result is new uh because you know we don't calculate over the tool result. So this becomes a new input and we generate an output and we get another tool result. And so in step two we need to calculate the KV cache over the new input and we uh during prefill and the output but we can reuse the cache and then we do that over and over and over again. And so as you imagine um as
[53:49.280] you can imagine I just showed steps 1 2 3 but by the time we get to step 50 if we haven't done anything tricky with our output this is going to be 50 times larger than the new input in the in the output right so that's pretty significant savings if we can cache this properly and here are some examples from actual papers uh that you know examine this and basically What we can see is this is like the the token length on uh different GPUs and this is the amount of uh of compute that you need with caching and no caching. And so the upper one of the same color is uh or the lower one with the same color is caching and the the upper one is no caching. So uh basically as the sequence gets longer the non-cashed one becomes quadratic and
[54:44.319] the cache one is is mostly linear. This also shows up in how much you pay to run an agent. And so I I pulled all of these numbers from uh the official pricing. And basically you can see here that if you go through all of these uh all of these numbers including the commercial ones typically cashed tokens are onetenth of the price of uncashed tokens. And so what this means is you need to be very aware of uh whether your tokens are being cached or not. Uh because if they're not being cached uh you're going to be paying a lot of money to be running your agents. Um, so does anyone have an idea of what a good uh cash hit rate is if you're running an agent? Yeah.
[55:33.440] >> 80 to 90%. >> 80 to 90%. >> Yeah. >> Um, yeah. So I I heard 80 to 90% I heard like more than 95%. I my my impression is 90 to 95 is everything is fine. Like I'm I'm good with 90 to 95. If it's less than that, you might be fine, but you might be might be losing out somewhere. Um so, uh yeah, so basically you can you can cache most of your tokens. Um one other interesting thing is the output also tends to be five times more than the input. So um this is uh there's a 50x difference between output and cached input which just reflects the amount of uh uh the amount of cost uh savings that you can get from serving it.
[56:28.880] So why why is caching um so much cheaper is maybe kind of obvious. You just need to look up um you need to look up the outputs. uh you just need to look these up instead of like actually uh performing the calculation over them. Uh especially the feed forward network calculation is pretty expensive in transformers and so being able to skip that is is uh particularly a win. Um but
[56:58.880] how do we actually manage that? So um the first thing is uh how do you hold it in memory? And there's two like most popular open-source uh two most popular open-source LLM serving libraries VLM and SGLANG and both of them are based on an innovation around the how you manage KV caching. Um so VLM's main technical in innovation and like of course they have a lot more stuff in there now. They're like general purpose serving libraries, but like the main technical innovation that led to VLM was something called page detention. And basically the way it works is you have physical GPU memory blocks and you rear you arrange them like kind of memory blocks or or disk blocks on uh like more typical uh like CPUbased machines. And so you might have uh block zero uh block one block two that you actually need to calculate and they put that you know in like block one and then block seven and then block three or sorry uh block seven block one block three and then you have an index to each one of these uh as you're generating.
[58:20.960] And the reason why they wanted to do this is they want to be able to generate in parallel many sequences in parallel and when you finish generating um you can like evict that from your memory and so you can uh evict whole blocks at once and this makes the calculation more more efficient. So um this is uh one technical trick that they used to make this uh you know make parallel serving
[58:44.000] work. Um the technical innovation that kind of led to SG lang in particular uh was something called RAIX attention and this is specifically focused around uh like re reusing already cached uh out uh inputs. And so what they do is they form a a tree basically uh it's a radics tree and they use this to look up prefixes that have been used already. And so this uh when it first came out this was specifically focused around uh chat bots like chatgpt uh because you know you can have somebody start a conversation and then continue the conversation and so you'll be caching some portion of the already done conversation. But for uh now it's like much much more important for agents because agent conversations are much
[59:39.839] longer and much more frequent. So we need this uh looking up more frequently. Um so another uh another thing that you need to think about um which is handled in uh newer versions of sglang for instance is let's say you're doing multi-achine serving. So you're serving on um you know different machines uh you know maybe you have a data center with like 64 nodes or 128 nodes or something like this. um every time a request comes in, you want to route it to the machine that has your KV cache that has the best KV cache for you um to be accessing and so that you actually have cache aware routing uh route it to the appropriate place and then have the worker with the best uh KV cache match uh process set.
[01:00:32.000] And so I love getting all my uh my serving statistics from Open Router um because they have so much data and uh they cover all of the things I want to talk about, but I believe they even have a ranking of providers by cash hit rate. [laughter] I'm sorry mist the data the data doesn't lie but um you can see uh mistrol has the worst cache hit rate and then um you have other one other ones like Alibaba cloud or dicard or silicon flow that have higher cash hit rates um this is not entirely reflective of the provider being bad or something like that it could be just the workloads that they're running but there's a number of things that like actually factor into into this cache hit rate and um one of them one of them is how good is the cash
[01:01:28.240] routing. The other one is how fast is the cash eviction. Um so the providers will not hold your cash for you forever. Um they will uh you know evict your cash after a certain amount of time. Does anyone know how long this is for like popular providers? Have you ever taken a look at that >> yet? Close to an hour. >> Close to an hour. I I think the long like I'm not I'm not like super up to date on this, but I know anthropic used to be five minutes. And so it's basically if you have a an active conversation going on with them and you don't pause your conversation, you'll stay in the cache. Otherwise, if you like stop the conversation, you get evicted from the cache. I think you at least have an option to go up to an hour now. Um but I don't know if you um but like it costs a little bit more if you go up to an hour or something. Um but yeah, ba basically they don't want to keep this around because there this is a huge amount of memory uh that they're keeping in memory and so they want to
[01:02:30.160] use it if it's working and and evict it if it's not working. So um that that's another policy that could change your cash hit rate. The bad news for us as users is the pricing doesn't change no matter how good the provider is at routing or um or cash eviction. So you uh you might be paying just because they evict you uh evict you too quickly which seems a little bit unfair. >> Um yeah >> do they use like cash layering where you can store it on device in memory that's like adjusted to device and then on this >> yeah so the that's a great question. So the question was, do they use a cache layering where you have like a fast cache that's stored in the device and then uh a slower cache that's stored on disk or something like that? I actually don't know the answer to that question. That'd be a great uh like a great follow-up for us to do. Um the open source libraries also are probably not as sophisticated as the closed libraries because um if you look at all the people on open router their job is to be really good at serving and so you know they're some of them might be using open source libraries but they'll be building stuff on top of them and that's kind of their you know competitive differentiation. So um like I I think the internal solutions that inference providers are likely to be you know a bit better than the ones that are available in open source too with respect to that.
[01:03:54.319] >> They usually offload to CPU from GPU. >> They usually offload to CPU from GPU. Yeah. >> Yeah. Because your GPU memory is so precious. So yeah. Yeah. >> Cool. Um so uh yeah. Oh, sorry. I skipped a important part. So, what does this mean? If you're just designing a harness and you're not deploying models or or anything like this, it means you need to be super super careful to not
[01:04:24.319] break your cache unintentionally. So, remember that your cache only works if the entire prefix is exactly the same. And so there's a bunch of things that we've thought about uh doing but we decided are a very bad idea because we don't want to break the cache. So one thing that's completely forbidden you should not do is don't change the system message every time you uh you take a step because that means like no caching essentially. uh you don't have a single message that you're able to cache. So one thing one very logical thing that you might think of is okay I want to append the current time and date to the system message uh so that the agent knows what time it is. Do not do this because the time and date changes every step and so that will break your cache. So if you do need to append it, append it to the final user message uh or the final uh tool call result that you get. um and uh like changing your um changing your tools uh dynamically in the middle of the conversation. You can do this. That's okay. Um another thing is a very logical thing that you might think of doing is routing to a different model every time you uh every time you take a step. So it's like this step looks like a really good model for you know a really power a really good step for a really powerful model. um like uh like I want to do this step on Fable and then I want to do the ne next three steps on um on a cheaper model to save money. But the thing is you're still paying you're still paying the you know 10x
[01:06:09.760] cost for Fable the next time you use uh Fable. So you really need to be very uh frugal in the amount of switching you're going to be doing if you want to be doing switching to save cost for instance. Cool. Um, so the final thing I'd like to talk about is context compaction. Um, how many people know know about this? Use this in your everyday coding agent use. How many people are really excited when it says compacting context? [laughter] That's nobody good. So um the the basic idea behind context compaction is uh that you have uh before and uh this is you know all your observations and actions that you might have had. Um then after what you tend to do is you tend to take your previous context and you crush it down into just the essence of the previous context. Um, and so the reason why people are like not excited about using this, uh, perhaps obviously is if you do a bad job at this, your agent forgets things. It starts doing something completely different and, uh, maybe deletes your file system.
[01:07:30.880] So um we can view compaction as essentially trying to come up with a previous representation where your future action sequence um you take your history, you compact that, you get a working state and you want your future action sequence to be the same or you know similar or possibly better um if uh after compaction than it was before. Um at the same time you also have a constraint which your state needs to be bounded. So it needs to be small enough that you're happy to continue work. Um so if we look at what survives uh compaction um there's different ways of doing this. I'm going to cover the different ways that people do this, but a common strategy is to take some anchors and these can be the very first um like the very first things the user said when they started the session for instance. Um, so, uh, one of the answers to the things that I heard agents forget when I asked the question, what do agents forget was sometimes they forget the things at the very beginning of the session? And that actually might be a design bug in the agent harness that you're using. Um, because uh, some good agent harnesses uh, like kind of intentionally avoid deleting the stuff at the very beginning of the session because they think it's more important. Um, but it could also be, you know, impossible to avoid depending on the situation. Um, also you want the stuff in the middle that's encoded and put back into the context. And then the recent tail is often kept around. So often people will keep around the the most recent actions. Separately from this, I don't see this a whole lot, but um, you can also add the evidence to an evidence store. Actually, no, sorry. I I'll correct this. Um, the evidence is typically added to an evidence store. Um, and at the very least, if you're using a CLI coding
[01:09:47.279] agent, usually your full history is saved on disk somewhere. And so, if you know where on disk, your coding agent has saved all of this history. Even if you lose stuff through compaction, you can tell the coding agent to go back and read the history, and it can, you know, uh, read the history for you and find the the previous results. Um but there there are also uh possibly more
[01:10:09.120] structured ways of doing this and so then um you know other things can be discarded. Um so there's a few ways to decide uh compaction policy. Um the first one is the trigger and there's um essentially two ways of doing this. Um one is a token threshold. So you limit uh the token threshold. Uh so you can say I'm I'm only going to use up to 200k tokens. Once I go over 200k tokens, I'm I'm going to compact the context. Another thing is um like when you get forced to do that by the provider. Um so that's kind of like a hard limit on the number of tokens you can have. Um finally, you can also have a manual request. And so uh a lot of uh kind of coding agent toolkits for instance have a slashcompact uh like command that you can use to compact the context.
[01:11:08.719] Um then based on that uh you'll select you might protect the beginning and keep the recent tail and compact the older that summary can be a structured summary. So sometimes uh they require a tool call uh that says this and this and this and this must be in the compaction summary. Sometimes it's just free text. Um there's also uh I think this is only open AI uh possibly. But basically they um they compact into something encrypted uh that you cannot read. So you can't see how they're compacting because they don't want you to steal their uh their precious compaction algorithm. Uh so uh so there's that.
[01:11:53.360] Um and then uh yeah, you can you can keep on on working on that. Um so one of the risks here is uh sometimes you can compact over and over and over again and the first time it works uh but as you walk through more um you uh you end up losing more and more. So again this is one of the reasons why people don't like compaction because it can cause the agent to go off the rails.
[01:12:22.960] Um so uh ba basically the way we evaluate this is typically in a holistic state. So you can vary the compaction algorithm and evaluate it against whatever benchmarks you're evaluating against. I would like to warn you uh about this. I have a a kind of like story from when we were first developing open hands which is um we actually I think made one of the first like production grade uh compaction algorithms. We called it compression then um and we evaluated it against bbench which is a very common you know benchmark for for coding agents and we worked pretty hard on it like as a research project for about a month or something like that and we were like we found this great compaction strategy. It doesn't reduce our score at all. it's saving us a lot of tokens. And then we actually tried it in our um we tried it in our our system and it made these really silly mistakes like um sending a pull request every time it compacted. So it forgot that it had sent a pull request to GitHub. And so uh you know we would get five pull requests for the same functionality because it had forgotten. Um or you know it would forget user instructions that were sent in the middle of the conversation because we had only evaluated on single turn conversations that had a single user message and we were compacting away the multiple user messages. So it's not just good enough to benchmark uh unless you have really really robust benchmarks that represent every use case. You kind of need to test them in the wild as well.
[01:14:03.520] Cool. Um so we're we're getting close to time. Uh but yeah, basically uh what I covered today is uh you know agent loops grow very quickly. Uh we have a lot of inference challenges in doing so. uh we can handle uh efficiency problems through architectures and um and caching and uh compaction is kind of the way we handle this uh once we expand beyond the level that we can handle. So um that's all I have. Uh if there is any questions happy to happy to cover them and next time we're going to be talking about skills and memory. So this is like even longer context over your entire uh your entire time. Um, are there any questions before we all uh all leave or >> Yeah.
[01:14:52.960] great question. Um, the question was, do we consider sub agents as a way to manage context? And yes, but we're going to talk about sub aents later, so I didn't cover them this time. Yeah.