CS336 2026 · Lecture 3已发布

Architectures, hyperparameters

日期: Mon April 6
讲师: Tatsu
课程资料: 9 项
内容产物: 4 项

课程简介

本讲以现代 Transformer 的共同骨架为起点，比较 RMSNorm、SwiGLU、RoPE、串并联子层和去偏置等常见设计。重点不只是罗列模型差异，而是理解这些选择为何在内存访问、优化稳定性和规模扩展中有效，并形成选择层数、宽度、头数、词表和正则化策略的经验框架。

讲义 / Slideslecture 3.pdf打开链接↗论文Attention Is All You Need基础论文↗论文GLU Variants Improve Transformer课程引用↗论文RoFormer - Enhanced Transformer with Rotary Position Embedding课程引用↗论文Root Mean Square Layer NormalizationRMSNorm↗论文Layer Normalization基础归一化↗论文Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerT5↗论文LLaMA - Open and Efficient Foundation Language Models现代架构案例↗视频课程视频课程视频↗

CS336 Lecture 3：架构与超参数——中文讲义笔记

学习目标

完成本讲后，你应该能够：

理解现代 Transformer 架构的核心设计选择，以及这些选择与原始 Transformer 论文的差异。
解释 LayerNorm 放置位置（pre-norm vs. post-norm）、归一化类型（RMSNorm vs. LayerNorm）对训练稳定性与系统效率的影响。
理解门控线性单元（GLU）家族激活函数的原理、参数调整方法及其流行原因。
掌握 RoPE 位置编码的几何直觉与实现方式，理解其与绝对位置嵌入、相对位置嵌入的区别。
识别关键的架构超参数（前馈维度比例、头维度比例、长宽比、词表大小）的合理取值范围与经验法则。
理解 weight decay 在语言模型训练中的反直觉行为——它可能更多是优化器干预而非正则化手段。
掌握注意力机制的稳定化技巧（z-loss、QK-norm、logit softcapping）的原理与代价。
理解 Multi-Query Attention (MQA) 与 Grouped-Query Attention (GQA) 的动机——KV 缓存与推理效率的权衡。
了解滑动窗口注意力与全局注意力混合架构的兴起及其解决长上下文成本问题的思路。

课程主线

本讲的目标是建立对现代语言模型架构的“经验性共识”的整体认知。核心方法不是从理论推导出发，而是通过系统地回顾近年发布的各类模型（GPT、Llama、Gemma、Qwen、OLMo 等），归纳出所有有效架构共享的固定参数与选择，以及可以灵活调节而不会显著影响性能的部分。

讲义的推进顺序：

LayerNorm 的放置位置与变体选择。
激活函数 的门控化改造。
并行层 vs. 串行层 的取舍。
位置编码：RoPE 的动机与实现。
关键超参数：前馈维度、头维度、长宽比、词表大小的经验法则。
正则化在 LLM 训练中的反直觉行为。
训练稳定性技巧：z-loss、QK-norm、logit softcapping。
推理效率干预：MQA / GQA、滑动窗口注意力。

贯穿全讲的核心理念是：架构是一组复杂的权衡——必须在表达能力（泛化）、GPU 训练效率、推理成本、训练稳定性之间取得平衡。许多看似随意的设计选择，实际上是这几方面压力共同作用的结果。

核心概念详解

1. 归一化：LayerNorm → RMSNorm，pre-norm → post-norm

残差流原则

Transformer 中存在一条贯穿网络的残差流（residual stream），注意力与 FFN 以增量（delta）形式向残差流添加信息。原始 Transformer（Vaswani et al., 2017）将 LayerNorm 放置在残差流内、每个子层末尾，这一做法被称为 post-norm 或 residual norm。

所有现代 LLM 都放弃了这一做法，转而采用 pre-norm：将 LayerNorm 移动到残差流之外、每个子层计算之前。

为什么？

梯度衰减问题：post-norm 意味着每经过一个 Transformer 块，梯度都要经过 LayerNorm 的缩放，梯度范数会逐层改变；pre-norm 则为反向传播提供了“直通”路径，梯度可以无阻碍地从输出传播到输入。实验（Salazar & Nguyen）显示 pre-norm 显著减少了梯度尖峰的大小和频率。
训练稳定性：pre-norm 允许更深的网络稳定训练，部分研究的最初动机是消除 warmup 的需求（尽管现代训练仍需要 warmup）。

归一化变体

LayerNorm：减去均值、除以方差、再重新缩放。表达能力更强。
RMSNorm：只做缩放，不减均值、不加偏置。表达能力略弱，但在实践中无显著性能差异；系统效率显著更高——LayerNorm/RMSNorm 这类归一化操作虽然仅占总 FLOPs 的约 0.17%，但因其低算术强度（memory-bound），在小模型上可能占据高达 25% 的运行时间。RMSNorm 已成为几乎所有现代模型的标准选择。

Grok、Gemma 2、OLMo 2 等模型在两个子层计算结束后放置归一化（仍是 pre-norm 逻辑，但位置在计算之后）。另有一些模型采取 double norm——在计算前后各放一个归一化。

经验法则：遇到稳定性问题时，“在各处撒上 LayerNorm”通常能改善稳定性（例如单独在注意力计算内部添加一个 LayerNorm），这一做法在多个模型中被反复验证有效。

偏置项：原始 Transformer 的所有线性层都带偏置，但大多数现代实现直接省略偏置。偏置项同样是低算术强度、高内存密度的操作，去掉偏置不损失表达能力，同时提升系统效率。

2. 激活函数：门控线性单元（GLU）

从 ReLU 到 GeGLU / SwiGLU

原始 Transformer 使用 ReLU。
GPT-3 改用 GeLU（高斯误差线性单元），与 ReLU 的差异仅在接近零处的梯度形状与底部的一个小凹陷。
真正的性能跳跃来自 GLU（Gated Linear Unit，门控线性单元） 家族。

SwiGLU / GeGLU 结构（以 SwiGLU 为例）：

标准 MLP：输出 = W₂·ReLU(W₁·x)
GLU 变体：输出 = W₂·(Activation(W₁·x) ⊙ (V·x))

其中 V 是一个额外的门控矩阵，V·x 对激活输出逐元素相乘做门控。命名规则：前置激活函数名 + GLU。ReLU + GLU = ReGLU，GeLU + GLU = GeGLU，Swish(x)·x = SwiGLU。

关键细节：参数匹配

GLU 比普通 MLP 多一个矩阵 V，直接替换会增加约 50% 参数。标准做法（由 Noam Shazeer 提出）是：将前馈维度缩小约 2/3，以保持总参数数量不变。此后的实验比较（在参数匹配前提下）显示 GLU 变体几乎总是优于非 GLU 变体，且计算开销增加很小。

Google 系模型（Gemma、T5）偏好 GeGLU。
Llama 系及其后继者偏好 SwiGLU。

非 GLU 模型的例外（如 Nemotron 340B 使用 Squared ReLU）极其少见，且并非刻意推荐。结论：GLU 是“免费的胜利”。

并行层 vs. 串行层

串行形式：先注意力，后 MLP，逐个计算。并行形式（GPT-J 首创、PaLM 采纳）：将注意力和 MLP 的贡献同时加回残差流，可共享 LayerNorm 与融合矩阵乘法，提升系统利用率约 15%，但会损失“一半深度”。现代主流模型已回到串行形式——串行优化的成熟使得并行带来的系统优势不值得表达能力的代价。

3. 位置编码与 RoPE

问题背景

注意力机制是位置无关的（permutation invariant）。没有位置编码，打乱输入顺序不影响输出。原始 Transformer 引入了正弦/余弦绝对位置嵌入。

RoPE：旋转位置编码

核心思想：在二维空间中，将每个单词的嵌入向量根据其绝对位置旋转相应角度。由于“内积对旋转不变”，任意两个位置旋转后的向量的内积仅取决于它们的相对角度差——这实现了“相对位置”的约束：f(xᵢ, i) · f(xⱼ, j) = g(i-j)。

关键点：这个性质要求位置编码能被分解为关于词向量和位置的乘积形式，而旧方法不满足——正弦嵌入存在绝对位置的交叉项，绝对位置嵌入显然非相对，相对位置嵌入（如 T5 风格）只能加到注意力矩阵上，无法分解为两个向量的内积。

高维实现：将 D 维向量切成两两一组的分块，每组做二维旋转，对应不同的角度频率。低频旋转慢，捕捉长距离依赖；高频旋转快，捕捉局部邻近关系。

实现细节：

根据序列位置生成余弦和正弦角度表（通常为每层共享）。
将旋转应用于 Query 与 Key 向量，而非直接加到嵌入上。
每次注意力计算前都要应用，以确保位置不变性成立。
实现方式：等价的复数乘法或稀疏矩阵旋转，也可直接手工实现。

RoPE 成为绝对主流：2024 年之后的几乎所有模型均采用 RoPE。Gemma 4 有一种变体 p-RoPE，只旋转前两个坐标来节省显存（面向小模型）。还有一些模型（如若干混合架构中的局部层）使用 NoPE（No Positional Embedding），完全去掉位置嵌入。

4. 关键超参数：经验法则与合理取值范围

前馈维度比例（ff_dim / model_dim = ratio）

经验法则：非 GLU 模型约 4 倍。
GLU 变体在 2/3 缩小后约为 2.67。
Llama 2 选择 约 3.5（理由：MQA 带来了效率，可给 MLP 更多参数预算）。
T5 是极端例外：64 倍，基于系统效率考虑（更大的矩阵乘法硬件利用率更高）；但 T5 v1.1 又回到了 2.5 倍，暗示极端选择虽可行但计算效率不佳。

经验结论：这是一个宽容度极高的超参数。小规模扫描（Kaplan et al., 2020）显示约 1–10 之间都有一个性能平坦的盆地，在这个范围内选择几乎不影响损失。

头维度（head_dim）

经验法则：使得总维度不变——即 head_dim ≈ model_dim / num_heads。
几乎所有模型都遵循这一准则；显著例外是 T5 和 Lambda 系列。
与 ff_dim 类似，head_dim 在 1 附近有一个很宽的平坦盆地，是最不敏感的超参数之一。

长宽比（模型宽度 / 深度）

大多数模型的 model_dim ≈ 100 × num_layers（约 100 倍比例）。
深层模型难以做张量并行，薄而深需要流水线并行（所有人都不愿意碰）；浅而宽则更易并行。系统与表达能力的权衡最终收敛到约 100。

词表大小

单语英语模型：约 30k。
Llama 系及现代多语言/生产系统：约 100k–200k。
驱动因素：多语言覆盖范围与模型规模（较大模型能更好地支撑大词表）。若模型包含视觉等非文本模态，tokenizer 需额外的大词表。

正则化：一个反直觉的结论

直觉推理：语言模型数据多、单次遍历训练、几乎不可能过拟合，因此 dropout 或 weight decay 可能不需要。但实验数据显示 weight decay 仍然被许多现代高性能模型使用，原因并不是正则化，而是它和优化器/学习率衰减交互，帮助收敛到更好的极小值——本质是优化干预，而非正则化干预。dropout 则已基本淡出主流。

5. 训练稳定性：三大技巧

输出端：z-loss

Softmax 计算中的归一化项 log z 可能爆炸（指数运算）或崩溃。技巧：在损失中加入惩罚项，使 log z 朝向 0。来自 Jacob Devlin 的论文（2014 年）。Baichuan 是最早采用的开源模型，DCLM、OLMo 等随后跟上。

注意力端：QK-norm

在 Q 与 K 点积之前对它们分别做层归一化或 RMSNorm，使 softmax 输入的尺度大致恒定（尺度保持为 1）。源于多模态模型（Idefics、Chameleon）。不会影响性能，但有效防止注意力退化与梯度尖峰——如今已成为大多数 LLM 的标准配置。

强行限制：Logit Softcapping

直接对注意力 logits（进入 softmax 的原始值）施加 Tanh 软上限（如 ±30），硬性防止极端 logits。Gemma 2/3/4 使用。注意代价：作为强干预手段，会损失表达能力与性能表现，只适合作为稳定性优先的策略。

6. 推理成本与 KV 缓存：MQA 与 GQA

推理的场景

自回归生成只能逐个 token 生成。为不重复计算历史内容，需要维护 KV 缓存——存储过去所有 query-key 交互。问题在于生成阶段算术强度很差：内存访问中，序列长度项变为 n²（必须反复读取整个 KV 缓存）。

MQA（Multi-Query Attention）

所有注意力头共享同一组 K 和 V，只有 Query 不同。KV 缓存大幅缩小，算术强度中的关键项从 n/d 变为 h·(n/d)（h 为注意力头数）。代价：表达能力显著下降。

GQA（Grouped-Query Attention）

将头的集合分成若干组，组内共享同一组 K 和 V，组间独立。你可以通过分组数在表达能力和效率之间连续调节。实验显示，分组数只要不要太少（即 KV 头数不要太小），推理成本大幅下降的同时，性能损失几乎可以忽略。这解释了如今几乎所有模型（包括 Llama 3、Qwen 系列、Gemma）都倾向于 GQA 而非完整 MHA 或 MQA。

7. 长上下文：滑动窗口注意力与混合架构

GPT-3 原始论文就有在全局注意力与带状矩阵（带状滑动窗口）间交替的设计。这想法最近重新流行。
示例结构（Command A）：每 4 层有 1 层全局全注意力 + 3 层滑动窗口局部注意力。局部层自底向上聚合信息，最终仍可获得全局感受野。
更激进的变体：局部层用 NoPE，彻底消除全局位置信息，在滑动窗口外退化为“词袋”。
采用者：Cohere Command A→ 随后有 Llama 4、Gemma 4、OLMo 3（用完整 RoPE + 滑动窗口）；Qwen 3.5 在每四层间交替使用状态空间模型门控 Delta 网络与全局注意力层。
核心论点：纯全局注意力的成本不再可行，纯局部注意力无法捕捉长期依赖。混合模型已成为处理长上下文的主流方案。

课程作业关联

A1 要求实现与原始 Transformer 不同的三大改动：pre-norm、RoPE、SwiGLU（非 ReLU）。本讲解释了这些改动的动机与实验证据。
作业中可能需要实现 z-loss 或 QK-norm，作为稳定性保障。
作业要求的 GQA 或 MQA 结构，是为了让你体会 KV 缓存带来的成本差异。
长上下文任务中，滑动窗口注意力的实现与全局注意力交替的取舍，可以作为作业完成后进一步实验的方向。

常见误区

认为 LayerNorm 只能放在子层之前：事实：可以放在计算后（post-norm 但仍在残差流外），甚至 double norm。关键是不要放在残差流内。
认为 GLU 一定优于非 GLU：在相同参数预算前提下，实验数据显示 GLU 更优；但直接替换不缩小维度会带来参数量的非线性增长，且并非绝对。
认为 RoPE 是正弦位置嵌入的变体：RoPE 是乘法（旋转），不是加法（嵌入）。它没有交叉项，因此严格满足相对位置约束。
认为 head_dim 必须等于 model_dim / num_heads：这只是经验法则，并非硬性要求；该超参数宽容度很高。
认为 weight decay 的唯一作用是正则化：在单次遍历 + 大数据条件下，它可以表现为优化器的隐式偏置，通过与学习率衰减的交互改善收敛。因此它依然被现代模型使用。
认为 QK-norm 一定会损失表达能力：多数实验显示它不影响性能，但 logit softcapping 这类强干预通常会使性能下降。
认为滑动窗口会丢失长期依赖：混合架构以层为粒度的交替，使得局部信息可以向上聚合并与全局信息结合，并不是简单的窗口截断。

复习清单

请确认你已能独立完成以下任务：

[ ] 说出 pre-norm 与 post-norm 在残差流上的位置区别，并解释其梯度传播含义。
[ ] 说明 RMSNorm 为什么在表达性与系统效率之间是合理的取舍。
[ ] 自主写出 SwiGLU 的公式与参数匹配逻辑（为什么是 2/3）。
[ ] 解释 RoPE 的二维旋转直觉，以及为什么它能实现真正的相对位置分解。
[ ] 列出 ff_dim 比例、head_dim 比例、长宽比的推荐取值和可接受范围。
[ ] 描述词表大小的两个主要群体及驱动因素。
[ ] 解释 weight decay 在 LLM 训练中的双重身份。
[ ] 描述 z-loss、QK-norm、logit softcapping 的行为、目的与代价。
[ ] 从算术强度角度，解释 MQA 与 GQA 为什么能改善推理效率。
[ ] 描述滑动窗口 + 全局注意力混合架构的工作方式与动机。
[ ] 能识别一篇新模型技术报告中的架构决策是否属于主流共识或属于实验性偏差。

备注：本文依据 CS336 2026 Lecture 3 人工翻译逐字稿编写，保留了关键英文术语，不存在虚构观点或数据。所有引用结论均来自课堂所述内容或讲义代码涉及范围。

从原始 Transformer 到现代语言模型：架构选择的共识与分歧

导语当你打开一份最新的语言模型技术报告，看到的往往是“基于 Llama 架构，做了少量修改”这样轻描淡写的描述。但在这句话背后，隐藏着数百个架构决策：LayerNorm 放在哪里？用 ReLU 还是 SwiGLU？头维度与模型维度的比值应该是多少？为什么几乎所有模型都用 RMSNorm 而不用 LayerNorm？这些选择看似琐碎，却直接决定了模型能否稳定训练、能否高效部署。本文基于 CS336 课程第三讲，梳理现代 Transformer 架构的共识与分歧，帮助你理解这些“默认选择”背后的逻辑。

为什么我们需要谈论架构？

训练语言模型时，你很快会遇到一个现实问题：选择太多了。词表大小、层数、宽度、激活函数、位置编码、归一化方式……每一个维度都有多种选项，而它们之间又存在复杂的相互影响。更令人沮丧的是，我们很难像在传统机器学习中那样，用 VC 维之类的理论工具来指导这些选择。

正如课程中所说：“我们都希望生活在一个只需要了解 VC 维之类简单理论工具的世界里，但现实并非如此。”架构设计本质上是一系列复杂的权衡：模型必须从数据中学习（表达能力），必须在 GPU 上高效训练（系统效率），还必须保持稳定——训练中途损失突然爆炸是绝对不可接受的。

面对这样的复杂性，最好的学习方法其实是亲自动手训练模型，尝试不同的架构。但如果没有足够的算力，第二好的方法就是从别人的经验中学习：观察那些成功训练的大模型，看看人们都做出了什么选择，哪些选择被反复验证，哪些选择只是昙花一现。

这正是本讲的核心方法论：通过统计大量已发布的模型，总结架构选择的趋势与共识。从最初的 Transformer 论文到 Llama 2 的出现，再到最近一年发布的各种新模型，我们可以看到一条清晰的历史脉络。

LayerNorm 的位置：一个出乎意料的共识

如果你去看原始的 Transformer 论文，你会发现作者们大部分事情都做对了，但有一件事被后人普遍认为做错了——LayerNorm 的位置。

在原始 Transformer 中，LayerNorm 被放在残差流（residual stream）上。所谓残差流，是贯穿整个网络的“主干道” x，注意力机制和 FFN 等组件会计算出一个增量（delta），然后加回到残差流上。原始的 post-norm 架构在每个组件之后做 LayerNorm，这意味着梯度在反向传播时要穿过这个归一化层，导致梯度范数发生变化。

替代方案被称为 pre-norm：把 LayerNorm 移到残差流之外，放在注意力或 FFN 计算之前。现在残差流保持“干净”，梯度可以沿着这条直通路径一路传回，不受归一化的干扰。这就是人们常说的“保持残差流干净”原则。

这个看似微小的改动带来了两个重要收益。第一，梯度衰减问题得到改善——反向传播时梯度范数保持稳定，不再因为每次经过 LayerNorm 而改变。第二，训练稳定性提升——梯度尖峰的大小和频率在 pre-norm 下明显低于 post-norm。

事实上，所有现代语言模型都把 LayerNorm 移出了残差流。唯一的例外是 OPT-350M，这个模型因为某种原因保留了 post-norm，而 OPT 系列整体上确实以稳定性问题著称。如果你做 pre-norm 和 post-norm 的对比实验，即使不做 warmup，pre-norm 也能获得好得多的收敛效果。

因此，当我们审视 2023 年到 2024 年发布的 19 个密集（dense）模型时，会发现它们无一例外地采用了 pre-norm 结构。这不是巧合，而是对稳定性和深度扩展能力需求的自然回应。

为什么用 RMSNorm 而不用 LayerNorm？

既然 LayerNorm 的位置已经统一，下一个问题是：用哪种归一化？原始 Transformer 使用完整的 LayerNorm：减去均值、除以方差、再重新缩放。现代模型则几乎全部改用了 RMSNorm，它不减去均值，也不加偏置项。

从表达能力上看，LayerNorm 比 RMSNorm 更强，因为减去均值、除以方差可以消除平移和缩放的影响。但实际上，RMSNorm 在语言建模任务中几乎没有表达能力损失——用 RMSNorm 的模型和用 LayerNorm 的模型表现一样好。

那为什么要舍弃更强的归一化操作？答案很简单：速度。这是一个系统与架构协同设计的典型案例。

LayerNorm 虽然只占模型总浮点运算量（FLOPs）的 0.17%，但这并不意味着它只占用 0.17% 的运行时间。关键在于算术强度（arithmetic intensity）：LayerNorm 是一个内存密集型操作，需要将参数在快速内存和慢速内存之间来回搬动，而算术强度极低。在小型模型上，LayerNorm 可能占据高达 25% 的运行时间。

相比之下，矩阵乘法（张量收缩）大部分工作量在计算本身，数据移动不是瓶颈。因此，移除那些算术强度低而内存密集度高的操作，成为提升系统效率的关键。

这就引出了更一般的经验法则：Transformer 中的偏置项通常没什么用。原始 Transformer 的所有线性层都带偏置，但大多数现代实现直接省略了偏置。偏置项同样是算术强度低、内存密集度高的操作，去掉它们可以在保持相同表达能力的同时，免费获得系统性能提升。更重要的是，偏置项在某些情况下还可能引发稳定性问题。

激活函数：门控机制为何取胜

在课程中，有一句话让我印象深刻：“我曾经把‘永远不知道 SwiGLU 是什么’当作一种骄傲的资本。”然而，如今如果不想知道 SwiGLU，你可能就无法理解现代语言模型。

激活函数的选择经历了清晰的演变：ReLU → GeLU → 门控线性单元（GLU）。ReLU 是最基本的选择，GeLU 在 ReLU 的基础上在零点附近增加了一点平滑的凹陷。但真正带来质变的是 GLU 系列：SwiGLU 和 GeGLU。

门控线性单元的核心想法是：与其对输入 x 做完矩阵乘法后直接施加非线性激活，不如增加一个“门控”矩阵 V，用 xV 的结果逐元素乘以 ReLU 或 GeLU 的输出。这个门控机制相当于在调节激活的输出强度，它被证明是一种非常有效的架构原语。

需要注意的是，门控模型比非门控模型多了参数——本来只有 W1 和 W2 两个矩阵，现在增加了 V 矩阵。为了保持参数数量不变，经验法则是把前馈维度缩小 2/3。这是 Shazeer 在提出这个想法时使用的做法，后来的 GLU 变体几乎都遵循了这条规则。

实验证据很明确：在参数匹配的情况下，GLU 变体几乎总是比非 GLU 变体持续更好。这被描述为“免费的胜利”——以很小的计算成本获得不错的提升。几乎所有可信的现代语言模型都使用某种门控线性单元：Google 的 Gemma 和 T5 用 GeGLU，而 Llama 的后继者们用 SwiGLU。两者之间的性能差异并不显著，但 SwiGLU 已经成为更主流的方案。

值得注意的是，不使用 GLU 的模型确实存在——GPT-3 和 Nemotron 340B（使用 Squared ReLU）都展示了非 GLU 方案也能训练出优秀模型——但这样的模型确实非常罕见。

位置嵌入：RoPE 的崛起

如果说 LayerNorm 和激活函数的选择已经基本统一，位置嵌入则是仍在快速变化的领域。

注意力机制本身是位置无关的——它只是内积运算，你可以随意打乱序列顺序，结果不变。因此，必须通过某种方式注入位置信息。原始 Transformer 使用正弦和余弦函数，这些是绝对位置嵌入。之后，一些模型使用可学习的绝对位置嵌入（每个位置有自己的嵌入向量），T5 和 Chinchilla 则使用相对位置嵌入（将偏移量加到注意力计算本身）。

真正改变格局的是 RoPE（旋转位置嵌入）。2024 年以后的大多数模型都在使用它。RoPE 的来源有些出人意料——它起初是 GPT-J 中的创新，来自一位中国作者，通过一个不太知名的博客文章和论文组合发表。

RoPE 的核心思想非常简洁：我们希望嵌入的内积只依赖于相对位置，而不是绝对位置。实现方法是利用内积对旋转的不变性：在二维情况下，把词向量按位置旋转一定角度；在高维情况下，把向量切成两两一组的分块，每组旋转不同的角度。低频的分块旋转得慢，捕捉长距离依赖；高频的分块旋转得快，捕捉局部关系。

重要的是，RoPE 是乘性操作，而不是加性操作。这意味着它不会有绝对位置嵌入中出现的那种交叉项，可以真正满足“内积只依赖相对位置”的性质。相比之下，即使是正弦嵌入，也存在绝对位置的交叉项，不是纯粹的相对位置。

Gemma 4 最近提出了一种 p-RoPE（比例 RoPE），只旋转前两个坐标，这是针对小型模型的内存优化。这个例子说明，位置嵌入领域仍然充满创新空间。

超参数选择：经验法则背后的逻辑

当我们进入超参数层面，一个重要发现是：看似复杂的超参数选择，实际上遵循着相当清晰的经验法则。

前馈比值：4 倍规则

前馈大小与模型维度的比值，是控制 MLP 丰富度的关键超参数。直觉上，这个比值应该翻遍所有可能值来仔细搜索，但现实中，几乎所有人都遵循“4 倍规则”——前馈维度是隐藏维度的四倍。

有两个主要的例外。第一，GLU 变体按 2/3 缩小，因为门控增加了参数；这就是为什么我们看到 2.67 到 2.5 的比值。第二，Llama 2 由于使用了高效的多查询注意力头，将比值任意调整到 3.5，相当于 4 乘以 1.33。总的来说，GLU 模型的比值在 2.6 到 3.5 之间，非 GLU 模型大约是 4。

Kaplan 等人的缩放定律研究显示，前馈比值这个超参数有很宽的“盆地”：从 1 到 10 的范围都很安全，但如果超过 10 到 100，损失会以二次方的速度急剧上升。

头维度与模型维度的比值

另一个有趣的超参数是头维度。经验法则很简单：头维度等于模型维度除以头数。也就是说，总维度和单头 Transformer 的维度相同。“不知为何，这就是经验法则”——但几乎所有模型都遵循了它。

对各类模型的统计显示，无论是经典模型还是新模型，每个注意力头的维度与模型维度的比值都稳定在 1 左右。显著的例外仍然是 T5（以及 Google 的其他一些模型），但总体上共识是清晰的。这个超参数的宽容度也很高，在 1 附近有一个相当宽的盆地。

长宽比：深度与宽度的权衡

长宽比（aspect ratio）定义为本度与深度之比：每层大约 100 的宽度。GPT-3、Llama 以及几乎所有模型都遵循这个比例。

这个问题背后有系统层面的考量：深度增加意味着必须使用流水线并行（pipeline parallel），这是大多数人不愿碰的；而宽度增加更容易通过张量并行（tensor parallel）来处理。因此，从系统角度看，更宽更好；从表达能力角度看，更深更好。最终的经验折中点是大致在 100 左右。

Kaplan 等人的扫描实验证明了这一点：长宽比在约 100 到 300 的范围内是一个安全的赌注，模型性能主要受控于总浮点运算量（flops），而不一定是长宽比本身。

词表大小：单语与多语言的鸿沟

词表大小展现出两类模型之间的明显区别。单语英语模型的词表规模在三万左右；而多语言或生产系统（如 GPT-4 级别的闭源模型）的词表规模可达十万到二十万。

多语言模型需要更大的词表来覆盖整个语言空间，而更大的模型也能处理更大的词表。Google 的模型词表规模尤其庞大，Llama 衍生模型大约在一万个 token 左右，单语模型约三万。

关于词表大小，有一个有趣的提问：当 tokenizer 不同，如何公平比较困惑度？答案是，只要统一用 bits per byte（每字节位数）这个指标，就可以比较不同的 tokenizer，因为长度归一化是固定的。

正则化：为什么显式正则化仍然存在？

如果你认为语言建模数据量巨大，单次遍历 SGD 不会过拟合，因此不需要正则化，那你会面临一个困惑：为什么许多现代模型仍然使用 weight decay？

事实上，weight decay 在现代建模中扮演着出人意料的角色——它主要不是作为正则化器，而是作为一种优化干预手段。实验显示，在不同的 weight decay 设置下，单次遍历 SGD 训练的模型在训练损失和验证损失上几乎没有区别——因为没有发生过拟合，weight decay 并没有在“控制过拟合”。

但另一个实验揭示了 weight decay 的真实作用：在衰减学习率的同时施加更强的 weight decay，模型最终收敛到了一个更好的极小值，取得了更好的训练结果。这是 weight decay 与优化器相互作用的结果，而不是对过拟合的惩罚。

这解释了为什么研究所有时会“莫名其妙”地使用 weight decay：因为这些干预手段与优化器的交互方式高度反直觉，而且它们确实在优化层面产生了实际收益。当你需要决定是否使用 dropout 或 weight decay 时，你已经进入了语言建模最底层的细节——这些细节往往被技术报告省略，但恰恰是它们决定了训练的实际效果。

稳定性：训练中必须面对的问题

随着模型训练成本越来越高，稳定性不再是边缘问题，而是核心问题。如果你的训练损失中途突然出现可怕的尖峰，你可能会得到质量变差的模型，甚至无法继续训练——当你已经花费了数百万美元时，这是灾难性的。

softmax 是稳定性的头号嫌疑犯。它有两个危险特征：指数运算会快速爆炸，除法可能除以零。模型中有两个 softmax：输出端（计算概率分布）和注意力机制内部。

输出 softmax 的归一化项 log z 是不稳定的来源。一个实用的技巧是 z-loss：在损失函数中加入平方 log z 项，惩罚 log z 偏离零过大，从而控制归一化项保持稳定。这个技巧最初来自 2014 年的论文，Baichuan 是第一个采用它的开源模型，后来 DCLM、OLMo 等模型也使用了它。

注意力机制内部的 softmax 同样危险。QK 归一化（QK-norm）在 Q 和 K 相乘之前对它们做归一化，确保进入 softmax 的数值有稳定的尺度。这最初来自多模态领域，Idefcs 和 Chameleon 模型证明了其有效性，现在已经成为大多数大模型的标准干预手段。

更激进的方案是 logit 软上限：直接对进入 softmax 的 logits 施加 Tanh 函数限制。Gemma 模型使用了这种方法。但 NVIDIA 研究者的系统性比较显示，软上限实际上会导致性能下降——它对模型的“表达能力”造成了过强限制，无法表达过于自信的信号。而 QK 归一化则允许调高学习率，提升了训练效率。

注意力机制的干预：从 MQA 到滑动窗口

当模型从训练转向部署，注意力机制面临的核心挑战是 KV 缓存（KV Cache）带来的内存成本。

训练时（预填充阶段），算术强度相当不错，GPU 能被充分利用。但推理时，我们只能逐词生成。为了高效生成，需要保留所有过去的键和值到 KV 缓存，只需计算新的查询-键交互。但问题在于：每一步都要读取参数，算术强度下降了——当模型很小而序列很长时，这种情况特别糟糕。

这一问题催生了多查询注意力（MQA：Multi-Query Attention）：所有头共享一组键和值，只有查询保持独立。这显著减少了 KV 缓存的大小和内存访问量，但代价是表达能力下降。

分组查询注意力（GQA：Grouped Query Attention）找到了中间方案：将键和值分组，减少它们的数量，但保留多个组以维持表达能力。GQA 在推理成本上接近 MQA，在性能上接近完整的多头注意力——两全其美。这就是为什么现在几乎所有模型都采用 GQA。

最后是滑动窗口注意力。GPT-3 时代就有人用全注意力和带状矩阵交替的做法，但直到最近才真正流行起来。这种方案通常是在大多数层使用滑动窗口注意力，只能“看到”局部结构，少数层使用全注意力来聚合全局信息。

Command A 是每四层中三层用滑动窗口，一层用全注意力。Llama 4、Gemma 4 和 OLMo 3 也采用了类似的结构。这种混合设计在长上下文性能与推理成本之间找到了平衡点，成为当前主流方案。Qwen3.5 甚至更进一步，在廉价层使用状态空间模型（SSM）替代滑动窗口注意力，但保留了同样的混合结构。

这些共识从哪里来？

当我们将所有模型放在一起对比时，会看到清晰的模式：所有现代模型都使用 pre-norm 结构；大多数使用 RMSNorm；几乎所有模型都有 GLU 变体；大多数使用 GQA；许多模型采用滑动窗口加全局注意力混合。

但也存在显著分歧：位置嵌入方案（RoPE 是主流，但也有其他选择）、词表大小（从三万到二十万不等）、长宽比（从 100 到 300 不一而足）。

这些共识并非来自理论推导，而是来自大量实验的积累——正如课程中所强调的：“我们事先并不知道去掉偏置是可行的，但是通过大量实验和如今集体积累的知识，我们大致了解了在典型的语言建模工作负载中该怎么做。”特别是 Llama 2 的出现，将一套经过验证的默认选择推广给了整个开源社区。

展望：为什么这些对你重要？

理解这些共识与分歧，对任何尝试训练或微调模型的人都至关重要。作为 CS336 课程的一部分，本讲（第三讲）在课程中的位置非常明确：第一讲和第二讲讨论了训练基础（如目标函数、梯度下降、损失计算），第四讲及以后将涉及状态空间模型、系统优化（并行计算、混合精度训练）等更深入的内容。

本讲聚焦于模型的静态结构——参数、激活、位置编码——这些构成了你可直接调整的“控制变量”。当你后续进行系统优化或处理实际训练任务时，这些架构选择会直接影响你的收益空间：例如，选择 GQA 而非 MHA，决定了你在推理阶段能获得的效率提升；选择 pre-norm 而非 post-norm，直接关系到训练稳定性。

从教学演进的角度看，本讲内容也反映了行业的变化：随着模型规模扩大和部署需求增加，架构设计正从追求“表达能力的上限”转向追求“表达能力与系统效率的平衡”。正如课程中那句耐人寻味的总结：“架构实际上是一系列非常复杂的权衡。”

在下一讲中，我们将探讨状态空间模型（SSM）等注意力替代方案，以及它们在长上下文场景中的表现。这些内容与今天的讨论密切相关——当我们理解了标准 Transformer 架构的共识与分歧，才能更好地评估替代方案的价值与代价。

课程视频与讲义入口：

课程主页：CS336 · Language Modeling from Scratch
Lecture 3 视频：官方视频
课程讲义：Lecture 3 Slides

CS336 2026 Lecture 3：完整中文逐字稿

视频：官方视频
来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。

正文

[00:05.600] 今天我们要讲的是架构，至少对我来说，这一直是个相当费解的话题。所以我打算采用的方式就是尽可能把所有内容都讲清楚。我会把现代的相关论文全部过一遍，我们一起看看大家都做了些什么。我把这节课的标题定为“你所不想知道的关于架构和超参数的一切”，因为我觉得我们都希望自己生活在一个只需要了解VC维之类简单理论工具的世界里，但现实并非如此。好的。我们要做的事情是，尝试从综述的视角来理解架构。最好的方法，甚至比听这节课更好的方法，是你自己去动手训练模型，尝试不同的架构。这绝对是最佳途径。这也是这门课程的理念之一。但我们不可能覆盖所有存在的不同架构的整个设计空间。我们没有足够的算力和时间去这么做。所以我的看法是，我们能做到的第二好的事情，就是尝试从别人的经验中学习。

[01:10.190] 其他人都做了什么？他们做了哪些选择？通过观察一个更宽广、更宏观的图景，也许我们就能开始理解，哦，原来这些是贯穿所有有效架构的那些固定参数和选择。而另一些参数则是可以变化的，且不会影响模型的表现。所以我基本上会讲Transformer的各种变体，比如，从Vaswani的论文开始的现代Transformer是什么样的？然后当我们看到更现代、更新颖的架构时，它们又有什么共同点？然后是我们被允许或不允许改变什么？但当人们实际去做的时候，又在改变什么？我想你们很多人已经上过某种NLP课程，或者至少见过Transformer。所以你们可能见过来自Vaswani等人的那个非常原始的Transformer。

[01:59.860] 有一些相当标准的选择。你会说，哦，Transformer没有位置依赖性，所以我们要加上位置嵌入。那我们怎么做呢，我们会加上一些正弦和余弦函数。我们会通过ReLU进行信息处理。然后我们会使用后置归一化（post norm）。我稍后会详细解释那到底是什么。当你们看作业A1的时候，你们会注意到标准的或原始的Transformer和我们要你们实现的内容之间有一些差异。嗯，我们会要求你们把LayerNorm移动到每个Transformer块或非残差层的前面。我们会要求你们实现一个叫做RoPE的东西。我们还会要求你们实现一个叫做SwiGLU GeLU的东西，而不是ReLU。

[02:41.660] 我们为什么选这些？一个原因是，我们很多内容是从 Llama 复制过来的，其他所有人也都是这么做的。真的，我觉得，如果你要训练自己的语言模型，你很快会遇到一个问题：哦，选择太多了。所有这些我该怎么选？那么，让我们现在来逐一看看这些不同的模型吧。我思考架构的方式，是去看人们做过的所有不同的事情，然后问，人们都做了些什么？我们能从中挑选组合吗？Percy总是为此稍微取笑我，但我确实会尝试去观察每年出现的所有不同模型，来准备这门课。去年，我以为，哦，只有几篇论文，没问题的。但当我浏览了所有东西，发现论文真不少。有Qwen2、Gemma 3、InternLM2。然后还有更多。

[03:31.130] 还有Nemotron-4、Qwen2。哦，天哪，有19个新的稠密（dense）模型。所以去年，我任务繁重。今年呢，我想，新发布的LLM总不可能还那么多吧。肯定该慢下来了，人们不可能每年都训练20个稠密LLM。严格来说是对的，稠密LLM确实没有那么多。一开始我想，哦，有Qwen3。Gemma 4上周四刚发布，所以我把它也放进去了，还有Olmo 3。就几个而已。当然，我也得提一下Percy自己和Marin一起训练的AP模型。我当时想，哦，我们只需要覆盖几个东西就行。但结果发现，如果你开始找，会有很多不同的模型。而且，我们之所以有这么多不同的模型，其中大多数实际上都是MoE，即专家混合模型。这个我明天会讲，而不是今天。

[04:16.320] 正因为我们有如此多样化的模型，我们实际上能很好地看到我们可以做的所有不同选择。所以我做了这个小表格。我们会在课程结束的时候再回到这个小表格。但基本上，从最初的Transformer开始，实际上已经有不少自回归语言模型在同一类数据上训练过了。你可以问这样的问题：不同的词表大小是什么？或者我们使用什么样的层归一化？或者人们使用什么样的位置嵌入？我们会看到一些相当清晰的趋势。我会在接下来的讲解中谈到这些。所以这里的目标是我们要涵盖几个不同的方面。我们会涵盖常见的架构变体。这些是Transformer的不同构建模块。在我们确定了标准的构建模块是什么之后，比如，我们用什么样的非线性激活函数？

[05:11.810] 或者我们用什么样的位置嵌入？然后我们会讨论超参数。我们会深入到更低的细节层次，比如，ff_dim是什么？我们应该把它设为4的倍数还是把隐藏维度乘以4来得到ff_dim？我应该有多少个词表元素？在那之后，我们会讨论一些非常底层的技巧，关于如何让模型稳定训练。我之所以会在本节课讲这些，是因为这些稳定性技巧与架构变化有着非常紧密的联系。从更高层次上讲，我想让你们牢记的一点是，架构实际上是一系列非常复杂的权衡。架构需要做什么？嗯，它必须从数据中学习，所以它必须能泛化。它必须能在GPU上高效训练。而且它不能崩溃。在训练中途，如果你的训练损失像这样一路下降然后突然爆炸，那完全不行。所以所有这些不同的要求最终都被直接嵌入了架构之中。这也是为什么这些东西有点混乱、有点复杂。

[06:16.155] 但你应该记住这一点。这也是为什么很多事情在很多方面并不那么优雅。所以我们从核心架构部分开始。从一个宏观的视角来看，我认为我审视很多架构相关问题的方式，基本上是历史性地看，从Transformer开始到GPT-3左右的早期阶段。那是大量实验发生的时期。人们尝试了许多不同的东西。没有一个每个人都在上面统一的黄金标准。然后Llama 2出现了。每个人都觉得，哇，Llama 2太棒了，我也想要一个自己的Llama 2。于是每个人都开始训练类似于Llama 2的模型，但带着人们各自的一些小变化。然后最后，去年，我们看到了非常大的差异，或者说是一种趋势，朝向使训练更稳定的架构修改。而今年，我们看到很多趋势是朝向支持更长上下文依赖的架构变化。所以这些就是正在发生的大主题。

[07:16.140] 但实际上，我想大家真正看到这个重要转折点是在 Llama 2 发布的时候，所有人都惊叹，哇，我也想训练这样的模型。然后突然之间，或者说也不算突然，在那之后人们又开始重新探索各种可能性。所以看到所有这些不同的变化是非常酷的。好的。我认为人们可以在架构的很多问题上持不同意见，但有一件事是所有人都认同的。如果你去看 Transformer 那篇论文，我想很多人会说，Transformer 的作者们大部分东西都做对了，除了这一件事。而他们真正没有做对、或者说我认为大多数人公认没有做对的地方，就是你把 LayerNorm 放在哪里。

[07:53.250] 在原始的 Transformer 论文中，LayerNorm 被放在了你所谓的残差路径上。在 Transformer 中，有一个残差流，也就是 x，它贯穿整个网络。然后你要应用注意力机制，它会做一些计算。然后它会将一个增量（delta）加回到残差流中。然后，为了确保这些梯度在各层之间保持稳定，一个 LayerNorm 被放置在每一个这样的组件末尾。现在，与其把 LayerNorm 放在残差流中，还有一种替代方案，我称之为 prenorm，也就是把 LayerNorm放在残差流之外，但在每个计算之前。所以你可以把它放在多头注意力之前，也可以把它放在 FFN 之前。我们称这个为 prenorm。这种命名方式会变得有点混乱。你可以暂时把这个叫做 postnorm，但让我们把这个叫做 residual norm，因为你把归一化放在了残差层中。

[08:56.740] 基本上，所有现代语言模型都把 LayerNorm 推到了残差流之外。这基本上就是每个人都在做的一件事。但有一个有趣的例外，那就是 OPT350M。如果大家对语言模型有所了解，就会知道 OPT 这个系列总体来说就是一个混乱的语言模型。而 OPT 350M 更是如此，因为我不知道为什么只有那个模型在残差流中使用了后置的 LayerNorm。好的。所以这是所有人都认同的事情之一。那么你可能会想，为什么这个做法在这么多不同的模型中如此统一？如果你去看一些早期的工作，研究 LayerNorm 应该放在哪里的这类研究，你会发现这类研究最初的动机很大程度上是，当你训练 Transformer 时，你需要做热身（warm up）。实际上，现代 Transformer 训练仍然需要做热身。

[10:01.390] 但在训练时你确实需要做热身。那么，如果我们能去掉热身步骤岂不是很好？这就是很多这类研究最初的动机。但人们很快意识到，去掉热身在这些模型的稳定性和收敛性方面会带来非常严重的问题。所以如果你做 postnorm 加 LayerNorm，这基本上就是原始 Transformer 的做法，你会得到这条紫色的虚线，哦，它的收敛效果不如做 prenorm 之类的方法，你可以忽略其他项，即使没有热身你也能获得好得多的收敛效果。所以这是最初的动机。但实际上，人们很快意识到的是，把 LayerNorm 移到残差流之外，当你把网络做得更深、开始应对稳定性问题时，会产生一些非常重要的影响。

[10:54.440] 对我而言，我觉得梯度衰减问题是最清晰的。当你和做架构设计的人交流时——我其实并不是那种深入研究这个领域的人——但人们经常说的一件事就是，保持你的残差流干净。在这种情况下，你的 x 从 prenorm 那边进来。这个 x 一路向上传播，一直传播到你的最终输出。这允许梯度在反向传播中直接穿过这里。这使得梯度传播变得非常简单，从而同时改善了稳定性和信号传播。这就是人们非常非常快就意识到的事情，如果你在初始化时做类似 prenorm（图中的蓝色线），梯度的大小保持不变，因为你在反向传播中有这种漂亮的直通传播。另一方面，如果你使用后置 LayerNorm，就会出现这些复杂的影响，因为每一次

[11:55.002] 你经过一个 Transformer 块时都在做 LayerNorm。这会在你向后传播时改变梯度的范数。所以你可以看到，从保持残差流干净这个原则出发，prenorm 是非常有道理的。人们还通过实验意识到，这总体上也会改善稳定性，梯度尖峰的大小和频率在 prenorm 下相比 postnorm 都有所改善。这是一张来自 Salazar 和 Nguyen 的图，我认为他们是最早仔细研究这个现象的人之一。我觉得这就是为什么这个做法一直保留了下来。稳定性和深度扩展能力对现代大型语言模型来说都是非常重要的。

[12:37.920] 所以这个把 LayerNorm 移到残差流之外的想法基本上被所有人采纳了。那么现在，如果把 LayerNorm 放在残差流里不好，为什么 LayerNorm 必须放在开始位置呢？当然，我们有 prenorm，它是在计算之前的。但我们也可以把它放在计算之后。至少在那种逻辑下，那也是一样好的。确实如此。很多最近的模型，比如 Grok、Gemma 2 或 OLMo 2，都有这种结构，它们把 LayerNorm 移到了计算之后。所以这是一种 postnorm，但它在残差流之外。其他一些模型实际上还是在各处都放 LayerNorm。它们在这里放一个 LayerNorm，在后面放一个 LayerNorm。我后面在讨论稳定性时会讲到这个。但另外一个似乎被很好验证的经验法则是，如果你遇到稳定性问题，你可以在各处撒上 LayerNorm。

[13:33.620] 这通常会改善稳定性。说这句话感觉很奇怪，因为这听起来太荒谬了。然而，这个说法实际上已经被证明是正确的。每次人们遇到稳定性问题时，他们就说，哦，但如果我们只是在注意力里面扔一个 LayerNorm 呢？事实证明，那也有效。我们稍后会讲到。好的，这就是 postnorm 或者在这种情况下叫 double norm，因为这里有两个 LayerNorm。好的。你还可以做的另一件事是，在原始 Transformer 中，LayerNorm 就是这里的这个操作。所以你有激活值 x，你减去均值，除以方差，然后重新缩放回去。这当然没问题。这并不是说这是错的。很多模型都基于这个方案成功训练过。但基本上，我认为大多数或所有现代模型使用 RMSNorm，它不减均值也不加偏置项。

[14:23.610] 所以只是缩小再放大回去。你可以在这里的方程中看到。实际上，LayerNorm 比 RMSNorm 具有更强的表达能力。所以在表示能力上，你没有理由必须使用 RMSNorm。但 RMSNorm 很好，因为在实际中，确实没有表达能力上的损失。使用 RMSNorm 的模型和 LayerNorm 一样好。但更重要的是，它更快。这就是系统和架构协同设计开始发挥作用的地方。在上一讲中提到了算术强度（arithmetic intensity）这个概念。我们希望通过做矩阵乘法和其它高强度的计算来让我们的 GPU 保持高利用率。

[15:06.360] 我们不希望GPU在来回搬运极小块内存上浪费时间，那是对我们强大GPU的一种极低效的利用。所以，我们想要做的是移除那些体量小、涉及内存移动但并不能给我们带来多少表达能力的操作。从这个角度看，如果均值减法与加法确实没给我们带来太多帮助，那我们的真正目标就是直接去掉它。你可能会想，这有什么要紧的呢？我们只不过是在优化一个微小的操作，它在此案例中大概只占系统总浮点运算量的0.17%。但正如Percy提到的，关键不在于浮点运算数，也就是FLOPs。浮点运算是我们做的矩阵乘法，但这并不等于运行时间。运行时间要复杂得多。而统计归一化，比如LayerNorm，即便它们只占0.17%的浮点运算量，取决于你的工作负载和具体配置，

[16:11.840] 它们可能占据高达25%的运行时间。这太夸张了。在小型模型上，这个占比可能真的非常大，因为执行这些操作时，你仍然需要将所有参数在快速内存和慢速内存之间来回搬动。因此，数据移动真的非常重要。正因为如此，RMSNorm依然可能很重要。你可以看到这里的区别。算术强度用白色表示，然后你可以看到涉及的浮点运算量用黑色表示。你会看到LayerNorm的算术强度非常低，这是我们希望尽可能移除的那种操作。好的，那边有提问吗？归一化的内存移动与计算密度相比，失衡得如此严重[听不清]。所以对于像张量收缩（这里指矩阵乘法）这样的操作，大部分工作量在于乘法计算，而对于统计归一化，大部分工作量在于内存移动。

[17:07.210] 而内存移动是相当慢的。所以想象一下，如果移动数据几乎占据了全部计算，那么你这里仍然要付出相当大的代价，因为激活值可能非常大。是的，我认为这种情况下运行时百分比是相当极端的。这像是在矩阵尺寸并不真正符合现代工作负载的小型模型中的情况，但这会让你理解为什么这是一个免费的优化。你确实会看到这种情况。这是另一篇论文，Narang等人2020年的工作，人们在其中评估了不同的架构干预措施。我觉得这是谷歌的一篇论文。他们展示了，对于一个两亿参数的超小型Transformer，当你切换到RMSNorm时，每秒处理步数增加了。

[17:45.220] 那是这里的第三列。实际上，你还能获得更好的性能，虽说这并非一定得到保证，但无论如何都是一个不错的额外好处。所以通过转向RMSNorm，你在系统层面就白白赢得了一分。所以基本上，现在所有人都决定转向这个方案了。而且一般来说，这里还有一个更普遍的版本。我所说的一切都没有保证。但Transformer和神经网络中的偏置项通常不是那么有用。在最初的Transformer中，所有线性项都带有偏置，但大多数实现实际上直接省略了偏置。这再次是另一个例子，说明某种操作在算术上强度不高，但相对而言内存密集度却相当高。

[18:31.050] 因此，你可能不如直接去掉它们，从而免费获得系统性能提升。还有一些情况，我顺便提一下，偏置项也可能引发稳定性问题，所以它们在其他方面是有用的。但我认为，这些偏置被去掉的主要原因其实是为了简化系统层面上的事儿。好了。我认为关于层归一化，事情比较简单。说它简单，是因为大家做法相当标准化。我们的理解虽然没有达到深层次的理论高度，但对于LayerNorm的作用已有相当清晰的认识。每个人都将层归一化移出残差流，通常采用prenorm结构。但我认为，部分原因可能是Llama 2这么做了。我们大致也知道如何使用LayerNorm来控制诸如梯度尖峰之类的现象，并保持信号传播良好。与之相关的是，现在我们基本上也总是使用RMSNorm。希望你们能理解这里的基本原则，也就是直接去掉偏置项。

[19:34.920] 这能让我们系统在保持相同表达能力的同时，拥有更高的算术强度。我觉得很多架构让人不太满意的一点是，你无法事先对其进行推理，但我们事先并不知道去掉偏置是可行的。但是通过大量实验和如今集体积累的知识，我们大致了解到，在典型的语言建模工作负载中，去掉线性和RMSNorm上的偏置项是没问题的。这种结论是我们基于对各种不同模型的观察而得出的。对于LayerNorm还有问题吗？好的。那么现在，我来讲讲激活函数。有各种各样的激活函数，数目众多——ReLU、GeLU、Swish、ELU、GeGLU、SeLU、SwiGLU、LiGLU。这些都是什么呢？我记得在我之前更偏统计和机器学习的训练阶段，我曾对自己说，我永远也不会去学这些东西。

[20:36.090] 我曾把“永远不知道SwiGLU是什么”当作一种骄傲的资本。但现在，我们确实很有必要对这些对象有一个总体认识，并了解这些名称中的哪些部分真正影响性能。因为你可以仅使用一个相当普通的激活函数来构建和训练语言模型。我猜，Chinchilla可能是那一组模型中最好的了。但即便你只想用ReLU，你也能仅凭这个激活函数训练出性能还不错的语言模型。这没什么不对。如果我们换成GeLU，即高斯误差线性单元，实际上唯一的区别就是底部这里一个小小的凹陷，这在大部分激活区域里并不会改变什么，但会改变接近零处的梯度。然后你就能训练出像GPT-3这样的模型。那是一个非常棒的大型语言模型，虽然按现代标准不算最先进，但完全够用。但接着我们来看门控线性单元，比如SwiGLU和GeGLU。

[21:36.240] 这些才是真正关键的所在。这和LayerNorm非常相似，我认为几乎所有可信的现代语言模型都会使用某种门控线性单元。好了，那么什么是门控线性单元？这些是门控激活函数。如果我们要看一个类似前馈层的结构，我们可以先看第一部分。这是一个非常标准的ReLU前馈网络。我有输入x，用W1处理它，然后逐元素与0比较取阈值（即ReLU操作）。接着再用W2处理得到输出。非常直接的ReLU网络。我不把这说成是我个人经验，但架构设计中经常提到的一点是，门控机制往往非常有帮助。如果你应用这个非常通用的启发式规则，你可能会想，好吧，与其只有这个逐元素ReLU，我们为什么不加一个门控呢？第二个门控，这里的第二项，它只是用来逐元素地乘以我ReLU的输出。

[22:39.070] 我有第二个矩阵V。现在，这只是在调节我ReLU的输出。然后其余步骤保持不变。所以，不再是只有xW1然后W2，而是用xW1，并用xV来门控它。这是另一个大小相同的激活。然后我会用 W2 将它投影回去。好的。那么这是什么？现在，这是一个 ReGLU。命名方式是加上第一个激活函数，这里就是 ReLU 加 GLU，所以是 ReLU 门控线性单元。而门控机制一直是架构设计中另一个非常有效的原语。事实证明，这在语言建模中也非常有效。所以如果你用类似 GeLU 的东西，我们已经讨论过了。那就是 ReLU，但在底部有一个小凹槽，你会得到 GeGLU。如果你用 SwiGLU，也就是 x 乘以 sigmoid，那你就会得到 SwiGLU。

[23:39.440] 所以这是一个压缩函数乘以其余部分。这确实涵盖了很多现代模型。一般来说，Google 的人用过 GeGLU，比如 Gemma 模型、T5 模型都是这类。而所有 Llama 的后继者都使用 SwiGLU。所以 PaLM 和 Llama 的后继者都是 SwiGLU 模型。我会说 SwiGLU 可能是更主流的那个，但说实话，在门控单元之间，这真的没什么关系。现在，这里有一个附注，稍后会是半重要的 trivia。如果你往上看，你会注意到门控模型有更多参数，因为我有这个参数 V。所以如果你稍微算一下，我现在有三个矩阵而不是两个矩阵。你应该做的是，也许应该把前馈维度缩小 2/3 倍，以保持参数数量相同。所以这大致就是这个想法：我想要保持与我原始 MLP 相同的总参数数量，但我想让它变成门控的。

[24:48.412] 所以我要把前馈维度，也就是这个 W 的输出维度，缩小 2/3。这是人们通常遵循的经验法则，但并不是硬性规则。最初提出这个的 Noam Shazeer 的论文里有一些非常小的差异，但那些差异是一致的。我认为归功于他，我觉得他的很多论文都有这些误差棒评估，训练多个复现并检查它们是否更好。如果你去看，GLU 变体几乎总是比非 GLU 变体持续更好。这是一个参数匹配的比较，因为 Noam Shazeer 总是做这个 2/3 调整，以确保所有模型具有相同的总参数数量。这相当不错。在某些方面，这是一个免费的胜利。几乎每个人都在用 GLU。

[25:46.870] 还有其他更受控的系统性比较。这是我之前提到过的同一篇论文，Narang 等人在 2020 年发表的。Google 在 2020 年代其实做了不少不错的大规模架构比较论文，虽然用的是 T5 架构而不是自回归语言模型。他们基本上全面地比较了 GLU 这类东西。你会再次看到，如果我们看 SwiGLU、GeGLU 或者一般意义上的 GLU，它们在损失或其他下游指标上表现明显更好。这些论文相当有说服力，而且现在很多模型训练运行中也清楚地表明 SwiGLU 和 GLU 是好的。所以门控有很多变体，但真正需要知道的单一重要维度是，这些非线性函数的门控实际上相当重要，能以很小的计算成本给你带来不错的提升。这并不是说门控线性单元是必需的。

[26:47.400] 我的意思是，GPT-3 就是那样——我记得 Nemotron 340B 模型用了 Squared ReLU，这是一个疯狂的选择，但那个也有效。这两个模型都表现良好，但事实上，不使用门控线性单元训练的模型确实非常少见。所以证据指向使用这些门控技巧能持续获得收益。所以我认为，这些是我们在架构方面可以做的更有共识性的选择。现在，这个我觉得是个很有趣的想法，但经过时间的考验，可能已经证明它没那么好，或者可能没那么受欢迎。通常，我们是串行处理 transformer 块的。我们先计算注意力，然后计算 MLP，一个接一个。如果你很有系统思维，你可能会说，这引入了瓶颈。我必须等一个计算完成才能做另一个。

[27:42.130] 如果它们是并行的，我可能就能引入一些新的、很酷的系统优化。所以你可能会问，我们能并行化 transformer 块吗？这最初是 GPT-J 中的一个想法，GPT-J 是 GPT-3 的一个开源复现尝试。非常有趣的是，我认为 GPT-J 在传播很多想法方面出人意料地有影响力。我的意思是，PaLM 也是。Google 在架构方面实际上出人意料地大胆。但 PaLM 中的描述，你可以在他们的报告中看到，是这样的——不是嵌套这个，也就是上面那个串行形式，你只是把 MLP 和注意力层的输出加在一起，然后把两者都加回残差流中。如果你实现得当，你实际上可以共享很多组件。你可以共享 LayerNorm。你可以融合矩阵乘法。

[28:40.140] 这让你有可能获得额外的系统优化。我认为很多受 Google 影响的人。比如 Cohere，我知道，是由前 transformer 作者之一创立的。他们做了很多受 Google 启发的优化。他们遵循了这个架构，但没有太多其他人这样做。这种方法在过去两年里确实已经失宠了，我认为主要是因为串行形式的优化已经变得足够好，以至于第二种形式带来的系统收益不值得你在表示能力上付出的小小代价，就是从并行到串行你会得到的那种代价。实际上，你可以这样想，你已经失去了一半的深度。这对你的模型来说可能是有害的。所以从架构方面来看，事实上这部分这么短应该让你意识到，原始的 transformer 公式在多大程度上经受了时间的考验，因为我在这里真正谈论的改变只是 norm 放在哪里，或者是否有偏置项，或者 MLP 怎么处理。但相比你可以做的所有事情，这些实际上是相当小的变化。

[29:56.530] 现在，那些仔细观察的人可能会说，等等，有很多 transformer 替代方案改变了注意力机制。是的，你得等到下一讲，因为今天我只打算讲核心的基于注意力的方法。下一讲我会加入一些状态空间模型的内容。但只要你还在密集注意力这个领域，实际上原始 transformer 论文中的架构与我们今天所做的非常接近。所以你会看到很多这样的情况。现在回到这里，蓝色是 RMSNorm，黑色是 LayerNorm。你会看到大多数现代模型都是 RMSNorm 模型，串行与并行层。蓝色的是并行的，其余是串行的。你会看到大多数是串行层，PreNorm 与 PostNorm。有些我标记为 PostNorm 的实际上是 Pre 和 PostNorm 都用。

[30:46.180] 然后右边这些是 GLU，几乎总是如此，除了像 Falcon 这样的例外，它用的是门控线性单元。但几乎所有现代模型的这些其实都是门控线性单元。所以你可以从我告诉你的内容中很直观地看到这些趋势。好的。所以实际上，不同实现之间真正差异很大的地方，而且我认为很多架构相关的东西仍在不断变化的地方，是你如何处理位置依赖性，以及如何融入来自其他位置的信息。这在一定程度上就是注意力机制的核心部分。所以有很多不同的方法可以将位置编码到Transformer中。并且提醒一下，这非常非常重要，因为注意力机制本身是位置无关的。它们只是内积运算，所以你可以直接打乱它们的顺序。如果你没有位置嵌入，注意力计算的结果会是相同的。

[31:39.990] 最初的Transformer使用了正弦和余弦嵌入，这是一种傅里叶变换的思路，即如果你有正弦和余弦，那么无论如何你都可以从中恢复出位置信息。在它之后不久出现的许多其他大型模型使用了绝对位置嵌入，即每个位置都有自己不同的嵌入向量。然后其他几个谷歌的模型喜欢使用相对位置嵌入。在这种情况下，你不是将嵌入加到词向量嵌入中，而是将一个向量加到注意力计算本身。所以，如果你偏移了三个位置，注意力矩阵就会加上一个不同的偏移量。像T5和Chinchilla这样的模型就使用了这种方案。在位置嵌入方面，真正变得非常主流的是一类叫做RoPE的嵌入，你们中的一些人可能对此很熟悉。2024年以后的大多数模型都使用这种类型的嵌入。

[32:37.540] 考虑到RoPE在某种程度上是突然出现的，这真的很了不起。最初，我认为这也是GPT-J的一个创新，来自一位中国作者，我觉得是通过一个不太知名的博客文章和论文组合提出的。但实际上，它确实有一些非常有趣的想法，解释了为什么要做类似RoPE的事情。所以RoPE是一种相对位置嵌入。对于相对位置嵌入，让我们持有一个明确的观点：我不应该关心任何单词的绝对位置。所以如果“A”和“apple”一起出现，无论它出现在开头还是结尾，在RoPE嵌入中，它们应该得到相同的结果。我们想要以这种方式来表示它。所以我有一个嵌入f，我有另一个嵌入f。这些将接收单词x和y的身份信息以及i和j的绝对位置。我想要这个等式成立：如果我对这些嵌入取内积，它应该等于一个只依赖于相对差异的函数。

[33:50.530] 而之前存在的所有嵌入并没有真正满足这个等式。正弦嵌入不是相对的，因为它有一些绝对的交叉项，这些项不是相对的。绝对位置嵌入，从名字上就能看出，显然不是相对的。然后是相对嵌入，从技术上讲，这些是相对的，但它们不是嵌入，因为它们只是加到注意力矩阵上。所以从这个东西中无法提取出内积结构。鉴于此，你可能会问，有没有一种好的方法可以让我们真正拥有这种相对嵌入？这个想法非常酷。它实际上只是利用了角度和余弦的其他性质。所以我们希望我们的嵌入对绝对位置是不变的。我们知道任何类型的内积对任意旋转都是不变的。所以这个想法是，我要取我的语义词向量，即那些与任何位置无关的向量。

[34:47.000] 这是我的出发点。然后我要根据单词出现的位置，在二维情况下，旋转这些向量中的每一个。举一个简单的例子，假设我们有这个句子“We know that.”。“We”出现在位置0，所以我完全不修改它。我就让它保持在原来的位置。单词“know”在位置1，所以我要将它旋转某个角度。这就是我的一个位置的旋转。现在，如果我对下面的序列应用同样的想法会怎么样？“Of course we know”。在这种情况下，“we”和“know”仍然是相邻的。它们紧挨着彼此，但是它们的绝对位置已经移动了。“Of course”出现在“we know”之前。在这种情况下，我要将单词“we”旋转两个位置，因为它的索引是0, 1, 2，所以单词“we”在第二个位置，编号2，所以我旋转两次。

[35:43.880] 我将“know”旋转三个位置，因为它在编号3的位置。你知道的，这两个向量之间的相对角度仍然是相差1。所以这就是一个非常非常简单的想法，仅仅利用旋转来表示位置。如果我们这样做，那么任何时候我们取内积，这些内积将对绝对位置保持不变。现在，你可能会说，在二维情况下，这非常简单，因为你真的只有一种选择，你有顺时针和逆时针。但在高维空间中，旋转向量的方式有无限多种。那么在D维情况下你该怎么做呢？嗯，你做最简单可行的事情，而且它有效。最简单可行的事情就是反复将其简化为二维的情况。你有一个D维向量，只需把它切成两两一组的分块。每一对二维维度都会被旋转。这些旋转的theta角度是变化的。

[36:38.450] 其中一些频率非常低，所以它们旋转得非常慢，因此可以捕捉长距离的依赖关系。有些旋转得非常快，所以它们捕捉的是像彼此是否为邻居这样的信息。然后最后，在我旋转了每一对向量之后，我就得到了最终的嵌入。这就是RoPE方法。如果你读这篇论文，它有一个关于复数的非常复杂的动机，但我觉得，至少对我来说，理解它的直观方式是，你想要通过简化为二维情况来进行旋转，你只是在旋转每一对坐标。Gemma 4刚在周四发布。他们还有另一种不同的有趣做法，他们称之为，我想是比例RoPE或p-RoPE，这其实是一种很奇怪的说法，实际上他们只旋转前两个坐标，但这也是你可以做的另一种有效方法。

[37:31.560] 所以在这个领域里，有很多不同的事情可以做，而且最终都奏效了。好的。在实践中，你最终会做的是，你可以取你的向量，然后用正弦和余弦做一个稀疏乘法。这将给你某种方式来旋转你的输入向量x。所以x乘以w乘以r，这将是你得到的最终嵌入。最后，这是一个正弦和余弦，看起来有点像正弦嵌入。但非常重要的是，我是与这些正弦和余弦相乘，而不是将它们用作嵌入，因为这意味着没有交叉项。这纯粹是相对的。你从内积中得不到任何绝对位置信息。如果我们真的想深入底层细节，你问，我到底该如何实现这个东西？你将需要这样做。你有你通常的注意力计算部分。

[38:24.730] 然后你要做的是，根据你序列位置的位置ID生成余弦和正弦角度。然后你要将这些余弦和正弦应用到你的查询和键上，用于你的注意力计算。你可以将它们作为矩阵乘法应用，或者你也可以直接手动应用它们，就像一次旋转操作那样。相当直接。而且你会在注意力层级上做这件事，而不是在最底层，以强制实现位置不变性，每次进行注意力计算时都是如此。所以那就是RoPE。它有点令人困惑，但一旦你理解了只是旋转东西的几何原理，实际上它就相当直接了。好的。我在这里暂停一下，以防有人对各种架构细节有疑问。我们接下来会讨论关于超参数的更底层细节，所以是的。你们有谁知道有没有论文做过更高维度的？更高维度的旋转。

[39:22.120] 这是个好问题。我认为没有。但更高维度的旋转，比如空间中任何二维旋转，都只是这个的一个变体。你当然可以在任何闭合环路的流形上做。我没见过那种做法。是的。[听不清] 你认为从论文和技术报告中提炼这类知识的最佳方式是什么？这是个好问题。我不知道有没有什么方法能超越某种组合——既要有足够广泛的阅读来找出模式，这正是我在本讲中试图做的流程；另一种方法是你自己亲自尝试，即使是在更小的规模上，也能形成对这些东西如何组合在一起的直觉和理论。我认为这两种才是真正正确的方法。我认为孤立地阅读任何单一论文都是非常非常困难的，尤其是现在，因为就如今的许多语言模型而言，似乎没有哪篇单一论文能给出任何完整的细节。哦，现在问题很多了。好的，很好。

[40:19.550] 我们继续。好的。所以我对并行层和[听不清]层有个问题。是的，我理解现代模型考虑的是资源效率，所以它们使用并行层。但这两种模式在准确性上是有差异的。我想知道准确性的程度是多少。这个差异是大到足以让当前的模型训练者无法忽视，还是小到可以忽略，或者[听不清]？是的，我认为这实际上相当复杂。如果你读原始的PaLM论文，我想他们对并行层的使用非常自信，比如没有性能下降，15%的系统利用率提升。所以如果你只读那篇，你会说，哦，它就是一样好。但我认为后来的很多Google模型已经停止使用这种方法了，你可以将此视为一个隐含信号，即实际上可能存在一些损失。而且再说一次，这个问题有点难以获得精确的数字，因为据我所知，没有人做过消融实验，

[41:20.340] 至少在并行与串行的受控良好消融实验方面没有。是的。如果你可能读过[听不清]，P-RoPE和RoPE之间的区别是什么？是的。我的意思是，这个区别真的只是在于你旋转哪些坐标？你不会旋转大部分坐标，因为很多——我的意思是，最初的论证，我认为，是低频部分不会旋转太多。所以如果你确实空间非常紧张，你可以丢弃它们。这实际上是对非常小的模型的优化，在这种情况下你没有太多隐藏维度来容纳激活值。是的。那么相对嵌入，没有内积。那是因为它基本上只适用于键吗？我试图理解。是的。所以他们把相对嵌入同时应用于键和值，这就是为什么你会从你所处的位置得到这种相对效果。你想要避免交叉项。

[42:23.150] 所以如果你看正弦和余弦嵌入，那么你不仅会得到原始向量，还会得到这些位置嵌入和词嵌入本身之间奇怪的交叉项，等等诸如此类。然后你可以反推出绝对位置是什么。所以即使是正弦和余弦嵌入也不是纯粹的相对位置嵌入。你必须接受相对嵌入是你想要的东西这一前提。但一旦你接受了，你最终会相当自然地得出RoPE解决方案。[听不清] 是的，是的。那么问题是什么呢？问题在于它不能被分解为一个乘积。这更多是一个美学上的问题。如果你的约束是，我需要它是相对的，并且我需要它能分解为f(xi)和f(yj)的形式，那么这就不是那个类别中的解决方案。公平地说，有很多嵌入以这种方式工作而且确实有效，比如l of i

[43:22.200] 和其他类型的方法都是这样注入到注意力矩阵中的。它们表现得相当好。它不一定是成为主导方法的那一个，这是我能说的。酷。好的。很好。现在，我们将讨论超参数。我认为超参数确实是当你真正需要训练一个模型时才开始接触的东西。当你的语言模型知识是抽象的时候，你不必关心这些。但一旦你必须实例化它，你开始问这样的问题：前馈大小应该多大？我应该有多少个头？我的词表大小应该是什么？你可能还会问：我的权重衰减或丢弃率应该是什么？我甚至需要正则化吗？我有很多词元。所以我需要正则化吗？我需要非常深的模型还是非常宽的模型？在这里做哪些事情才是正确的？

[44:19.150] 所有这些，如果你在没有任何知识的情况下开始，实际上非常令人生畏，因为你需要搜索这个非常大的高维空间。但人们尝试的东西的空间实际上相当小。从中，也许你可以开始思考更聪明的搜索过程，知道你想在哪些方面进行变化。其中一个真正共识性的超参数是前馈大小与模型维度之间比率的这个概念，前馈大小是MLP中第一次矩阵乘法的输出。所以这实际上是你的W1矩阵以及W2矩阵两个维度的比率。这似乎是一个非常重要并且控制着MLP丰富度的事情。那么它应该是什么呢？嗯，不管出于什么原因，它可能应该是你隐藏维度的四倍。这是一个非常有效的经验法则。我将向你们展示一些数据，说明为什么这也许是一个不错的选择。

[45:18.520] 有几个例外。而且有趣的是，真正极端的例外会回溯到那个规则。好的。例外情况之一是门控线性单元的变化。我已经告诉过你们这个了。所以如果你在考虑这件事，这大概已经记在你的脑子里了。如果你保持相同的维度，GLU有更多的参数。所以如果你想保持MLP的参数规模相同，那么你需要按2/3的比例缩小。所以大多数GLU变体来说，这意味着你最终会得到大约2.67这样的数字。所以所有在下面的这些，2.67到2.5，这大致是在应用这个2/3的修正。然后不管出于什么原因，Llama 2的人们决定，嗯，我们实际上有非常高效的多查询注意力注意力头，这个我稍后会谈到。正因为如此，我们可以将这个比率乘以一个任意的1.33。我们会得到大约3.5。所以Llama的人们任意地选择了一个稍有不同的比率，这本质上是对MLP稍微更强调一些。

[46:28.720] 但实际上，如果你真的翻阅所有论文，你会发现对于GLU来说，要么是2.6左右，要么是3.5，如果你是非GLU模型，则是4。好的。还有另一个例外，我觉得非常有趣，但也非常非常酷，那就是，纵观全局，当你阅读这些技术报告时，你会发现大多数人在架构选择上非常无聊。他们会说，我们做了 Llama，但改了一个地方。但谷歌的人有时候胆子很大。T5 是我最喜欢的模型之一，因为他们在设置上非常大胆。他们决定不遵循 4 倍这个经验法则，而是要用 64 倍的乘数，这比 4 大得多。他们对此也有一个合理的论证，这也是一个基于系统层面的论证。他们说，我的矩阵乘法越大，我的硬件利用效率就越高。所以如果我把这个乘数设得很大，那么我的矩阵乘法就有可能被更高效地利用。

[47:36.920] 还有一些其他模型，比如 Gemma 2，也试图在这个数值上再往上推一点。但说实话，T5 的 64 倍是一个惊人的例外。我不认为还有其他模型在前馈乘数上真的达到过那么高。而从经验上看，如果你去看其他试图对这个比值做更受控比较的工作，我引用的这个是 Kaplan 在 2020 年的研究。这是那篇经典的神经规模缩放损失（neural scaling loss）论文，他们对语言模型做了各种受控研究。你会看到，这其实不是那项研究的重点。那是一项规模定律（scaling law）研究。但你会看到其中一个面板里，他们实际上做了消融或扫描实验，改变前馈比值，然后观察一个非常小的模型的损失。

[48:25.520] 但他们在论文中发现的是，有一个盆地，从大约 1 开始，到大约 10 结束，在这个范围内，这个超参数表现很好，而且非常平坦。相对于这里的最优损失，你几乎不会损失什么。如果你把这个值搞得很离谱，比如超过 10 到 100 之类的，那么你的损失就会开始以二次方的速度急剧上升。所以很多这些选择，范围在 2.6 到 4 之间的，它们都落在这个相对良好的盆地里，所以你选这些数值就没问题。那关于这个超参数我们能学到什么呢？嗯，这些默认选择在几乎所有现代语言模型上都表现得非常好。所以你可以放心地选那个。T5 是个不错的模型，或者说 T5 第一版是个不错的模型。它不是个差模型。所以即便是激进的选择，在技术上也行得通，但很可能计算效率不高。而我觉得这个故事里最好笑的部分，对我来说是 T5 系列的包袱，就是他们有一个后续模型 T5 v1.1，本应是

[49:29.560] T5 的改进版本。然后他们又回到了标准的 2.5 倍乘数。这里没有任何明确的说明。但很明显，当他们试图更新 T5 时，他们决定要回到一个更标准的乘数，我觉得这有点好笑。这就是前馈比值，现在你对正确的数量级有了一个大致的概念。接下来，我们来讨论另一个有共识的超参数。我在教 224n 以及教学生这些东西的时候，一直觉得这个很奇怪，那就是如果你有一个多头注意力，在你的 Transformer 里有多个注意力头，那么最规范的做法，几乎是所有人都会做的事情是，如果你有多个头，你要确保这些头的大小，也就是头维度，是这样一个值：使得总维度和单头 Transformer 的维度相同。所以你总是要确保把隐藏维度除以头数，然后基本上再乘以 h。

[50:31.350] 所以在这种情况下，你有 h 个头数，每个头的维度是 d 除以 h。把两者相乘就得到 d。不知为何，这就是经验法则。当然，这不一定要成立。我们可以任意改变头维度和模型维度之间的比值。但大多数模型确实遵循这个准则。而且事实证明这样做效果很好。我们可以看看各种各样的模型，经典的和新的。我这里也有最新的模型。你会发现，是的，比值大致在 1 左右，按每个模型头来算的话，显著的例外是 T5，还有 lambda，那也是谷歌的另一个模型。但真的，所有人都保持在 1 左右。我觉得这是一个很有意思的点。关于头维度，我在这里要总结的是，我认为这是另一个宽容度很高的超参数。有人做过一些消融实验。同样，在 1 附近有一个相当宽的盆地，你可以在这个范围内随意选择。

[51:28.020] 但这个可能不是最关键的超参数。我觉得其中一个最关键、也最有趣的，在概念层面，是长宽比（aspect ratio）这个概念。在这里再补充一点，当你把模型放大或缩小时，通常做法是固定一个长宽比，比如模型宽度和深度的比例。然后你把模型做大。所以长宽比在某种意义上控制着随着模型变大时深度与宽度之间的整个权衡。你可能想知道我的模型应该有多深。如果你一直关注推理等方面的研究，你可能会觉得我需要一个非常深的模型，或者如果你想要系统利用效率，可能需要一个非常浅的模型。你可能会觉得这里面有很多变化。确实有很多变化，比其他超参数的变化要多得多。但实际上有一个相当清晰的甜区，大多数现代模型都落在这里。你不会真的看到模型变得太深。你也不会看到模型在任何一个方向上变得太宽。

[52:29.110] 你会看到，大多数模型的比例大约是 100 倍的模型维度除以层数。也就是说，每有一层，就有大约 100 的宽度。这对于 GPT-3、Llama 或者其他任何一个模型来说都是成立的。我认为，这里的考量部分是在表达能力和硬件之间的权衡。如果你有一个非常深的模型，从系统角度来看，处理起来会非常非常麻烦。模型越深，你有哪些并行化的方式呢？嗯，你可能不得不切分层。如果你切分层，我们会在系统讲座中讨论这个。一旦你开始在深度方向上切分层，你就会遇到非常严重的并行化问题。流水线并行（pipeline parallel），也就是这个名称所指的技术，是大多数人真的、真的不想碰的东西。而宽度则更容易并行化。

[53:19.750] 如果你有一个非常宽的模型，你可以很容易地在你的 GPU 上切分它。这就是张量并行（tensor parallel）。处理起来要简单得多。所以从某种意义上说，有系统层面的理由去做得更宽。也许有表达层面的理由去做得更深。最后你大概会落在 100 左右。我觉得关于 Transformer 超参数的一个真正有趣的地方是，有很多超参数看起来相当重要，但它们也有相当高的宽容度。人们大致收敛到了最小值附近。这是 Kaplan 等人的另一张图，展示了对不同规模模型的超参数扫描结果。你又一次看到，无论你的模型规模如何，大致来说，最优长宽比是相当接近的，它们在大约几百左右，也许稍微少一点，取决于你怎么算。

[54:11.660] 但说真的，接近 100 的范围对于长宽比来说是一个相当安全的赌注。EK 等人做了一系列非常有趣的架构变化实验，他们在这方面的总体结论是，我们看上面的面板，你可以有各种不同类型的模型，它们在深度与宽度的权衡上各不相同。但当你扫描深度与宽度的权衡时，你会发现，从某种意义上说，唯一重要的就是浮点运算量（flops）。随着浮点运算量的增加，模型变得更好。这真正控制了大部分效果，而不一定是长宽比本身。所以我认为，从这些研究中真正浮现出来的认知是，存在一个人们普遍会选择的一般宽容超参数区间。然后你真正主要担心的是系统的利用率，而不是那些难以推理的表达能力问题。

[55:08.350] 好的。然后也许我想提的最后一个超参数问题是词表大小。这个问题对我来说很有意思，因为两类模型之间有一个非常明显的区别。我认为，在开源模型训练的早期，有很多单语模型，它们唯一的目标就是在英语上表现得好。对于这些模型，词表规模要小得多，在三万左右。而在 Llama 之后，很多人对多语言或生产系统非常感兴趣，这些包括像 GPT-4 这样的闭源模型。所有这些模型的词表规模都大得多。这些大致在十万到二十万的词表范围内。你可以看到，一般来说，Google 的模型词表要多得多。Llama 的衍生模型大致在一百 k 个 token 左右。而单语模型大约在三万。这在一定程度上是很清楚的。

[56:12.130] 多语言模型确实需要更大的词表来覆盖整个语言空间。一般来说，右边的模型也更大。已有缩放定律研究表明，模型越大，它能处理的词表就越大。所以这部分也是由现代缩放趋势驱动的，右边的模型通常更大。现在已经没有人训练大规模的单语模型了。所以是的。[听不清] 关于风格的那些东西。如何使用那些词表大小？抱歉，问题是，如果你有多语言模型或者——多模态。是的。所以我想，这取决于你的 token 是如何编码的。但如果你正在对图像之类的东西做 tokenization，那么你确实需要更多的 token 来覆盖这些内容。如果你看看各种开源发布，它们会有一个不同的图像 tokenizer，有自己相当大的词表。是的。这种比较有多有效，我想，这是为不同的东西[听不清]提供的？

[57:21.810] 比较 bits 有多有效？哦，这是个好问题。好的。是的。这不是一个超参数问题，但这是个好问题。那么正确的方式是什么？让我退一步，把我们带入正确的思维框架。如果我们考虑语言建模，语言建模是一个生成式建模任务。我们在对序列的概率进行建模。现在，只要你的序列是固定的，你已经以某种方式处理过它了。而且你对所有字符串提供了一个概率分布，那进行比较总是有效的。在那个层面，总是有效的。那么，当你问这个问题，比较两个任意 tokenizer 的 bits per byte 是否有效时——实际上，有两件事在起作用。一件事是，你有没有改动序列本身？如果你看看 subword tokenizer 之前的一些历史 tokenizer，它们会丢弃一些 token 或丢弃一些词。那会使比较无效。但现代 tokenizer 是完备的。

[58:17.640] 它们可以对任何序列进行建模，所以这不是问题。另一件你需要担心的是，我们是否以某种方式做了长度归一化？但对于 bits per byte，你总是用同一个数字来归一化，也就是字节数。所以这总是一个有效的比较。所以这就是如何看待 tokenizer 比较的方式。比如说，我记得有研究结果显示，对于固定 tokenizer，比较困惑度总是能对应到下游任务上更好的表现，就像同一个指标在[听不清]上提升一样。困惑度和 BPP 是相互对偶的。所以是的，如果你问的是这个。答案既是是也是否，因为如果你在比较——你可以训练并比较困惑度[听不清]，但你改变的是另一个东西。[听不清] 好的，我们得稍后再聊，因为我不确定我完全理解了这个问题。

[59:15.562] 但我觉得这是一组很有意思的问题。好的，很好。好的，行。那么我们正在深入语言建模最底层的细节，我觉得当我们讨论这些的时候，确实会暴露很多有趣的想法。而我认为 dropout 是端到端正则化的一种。我认为这是另一类非常有趣的思想，也是我觉得从你们机器学习 101 的直觉来看非常反直觉的一类。让我们来看看我认为关于正则化的标准论证。如果我做语言建模，我有大量的数据。大多数时候我拥有的数据多到处理不完。除非你在 Google，也许即使在那里，互联网数据也多于算力。所以我很可能不会两次看到同样的数据。所以我只会在一个语料库上做单次遍历。

[01:00:10.540] 有很好的理由和论据可以相信，单次遍历的 SGD 或其他优化器基本上永远不会真正记住我的数据。这意味着在计算受限的语言建模中，过拟合几乎从来都不是一个真正的问题。有些人实际上甚至只看训练损失，因为他们非常坚信单次遍历 SGD 不会发生过拟合。那么，基于这一点，你可以坐下来想一想。在语言模型训练中，我应该使用 dropout 或 weight decay 吗？你可以稍微想一想。一个不幸的事实是，很多最近的模型完全不提这些东西。这确实是比技术报告愿意披露的更低层的细节。但如果你去看，实际上你会发现很多模型两者都用，尤其是 weight decay，即使是对于现代高性能语言模型，它实际上也是一个相当流行的干预手段。

[01:01:11.970] 这是非常非常令人惊讶的。我的意思是，一些 dropout 的东西可能已经失宠了，但 weight decay 实际上仍然相当流行。这非常令人费解。为什么会这样？这就是为什么我认为深度学习很难的原因之一。而这节架构课也非常奇怪和困难。这是因为这些东西以非常奇怪的方式相互作用。所以有一些论文已经论证并展示了很好的证据，表明 weight decay 有时实际上并不是一个正则化器。它实际上是与优化器相互作用，本质上使优化变得更好。所以如果你看单次遍历 SGD 语言模型训练中不同 weight decay 设置下的训练损失与验证损失，你实际上看不到什么区别。weight decay 并没有在移动什么，所以验证损失并没有变好。本来就没有过拟合。

[01:02:08.652] 这里画一条 x 等于 y 的线。所以它并不控制过拟合，但如果我们看不同水平的 weight decay，而且不仅仅是不同水平的 weight decay，我们还看 weight decay 与学习率衰减的组合。我们发现，更强的 weight decay 运行，底部那些蓝色虚线，表现明显更好，因为它们起步很慢，但它们最终收敛到了一个更好的极小值。这在我们衰减学习率时通常是成立的，但在恒定学习率下不一定成立，而后者可能更接近你的直觉来源。所以这就是为什么很难先验地、或者从头开始推理所有这些不同选择的行为，也是为什么我和 Percy 设计这门课的方式是让你们去实际操作这些东西，因为你可能会遇到这样的情况：基本上，weight decay 实际上是一种优化干预手段，而不一定是正则化干预手段。

[01:03:12.050] 这里正是你预期的结果。所以要时刻记住，这类意外的效应在这些场景下真的会开始显现。好，那我们把超参数的所有内容整合起来。实际上，对于那些看起来可能比较棘手的超参数，其实已经有一些对大家都行之有效的标准选择了。4倍规则是个经验法则：保持你的注意力头维度（head_dim）与模型维度（model_dim）的比值等于头数，即让头维度等于模型维度除以头数。选择一个宽高比，大概在100左右。如果你问的是正则化，你可能想尝试几件事情，因为正则化实际上与优化器的交互方式相当反直觉。所以这是一些人仍然会做的事情，即使你根本不需要正则化。实际上，也许我在这里停一下，以防万一。是的，对于扩散模型，架构上有什么显著的不同吗？扩散模型。说实话，这个我还没有深入研究过。

[01:04:16.995] 训练大型扩散模型的人不多，这是一个问题。而且许多已训练的模型都是改造而来的。所以我认为架构实际上和 Llama 类模型是一样的。但如果你问的是，如果要从头训练，最优架构是什么，我一时还真不知道。是的。对于正则化主要影响优化过程这一点，你有什么解释吗？嗯，我想，这并不是说正则化本身会影响优化。我认为人们不再使用 dropout 了，因为它与优化的配合确实不太好。但举个例子，权重衰减（weight decay），也就是向0收缩，这可能允许你使用更高的学习率，或者可能允许你更快地衰减。这些项之间有很多相互关联的方式。好的。现在，我已经谈了很多关于如何通过研究那些已训练的模型来设计表达力强的模型。

[01:05:10.220] 我现在要强调的一点是，在过去的几年里，一个非常重要的重点不仅仅是性能，而是稳定性。随着你的模型训练成本越来越高，这个问题变得越来越重要。我们看到很多这样的选择都是有容错性的。大家都在做类似的事情。所以你可以调整这些，但你不会得到巨大的性能差异。这没问题。但如果你的模型在训练过程中突然在某些部分爆掉，到处出现这些可怕的尖峰，你最终可能会得到一个质量实际上不太好的模型。或者可能是无法恢复的。你可能已经花费了数百万美元在训练上。然后你到了一个模型无法再继续训练的地步。如果你有大量想要投入的计算资源，那将是件非常可怕的事情。

[01:06:00.390] 所以你不会想训练出像那条蓝色曲线那样到处是尖峰、梯度范数巨大的模型。那么我们如何解决这些稳定性问题呢？我的意思是，这真的是，我想说，一个核心中的核心问题。如果你在语言模型，或者一般来说，神经网络中遇到稳定性问题，有几个常见的嫌疑人是你需要开始检查的。其中之一是 softmax。而 softmax 有两个对稳定性都非常不利的因素——一个是指数运算。你可以看到它会非常快地爆炸。你还要除以两个数。这也是一个潜在非常危险的操作。所以 softmax 是一个你需要格外、格外小心的地方。那么在语言模型中 softmax 在哪里？嗯，有两个。一个在输出端，当我们输出概率分布时。然后在注意力机制中，当我们归一化注意力时，还有另一个 softmax。

[01:06:56.080] 所以我们可以把这两个都看作是我们模型真正的危险区域，尤其是注意力部分。让我们从考虑输出 softmax 开始。输出 softmax 可能会对我们爆掉。我们可以做的一件事是，尝试控制归一化项的问题。让我们考虑一下 softmax 的计算。我们想要计算对数概率来计算损失。那么，对数概率是什么？嗯，它是你模型的输出 u。然后你还有这个对数归一化项。这个 u 表现良好，因为从某种意义上说，这是你模型的输出。这只是你的残差流（residual stream）的输出，包含了所有加进去的东西。所以如果 u 表现良好，那么第一项就是表现良好的，如果模型状态正常的话。现在，第二项，这个 log z，可能就没那么正常了。如果 z 非常大或非常小，即使你模型的输出还算正常，它也可能会爆掉。

[01:07:55.750] 那么 z 是什么？嗯，它是一个指数。所以它可能会非常快地爆掉。或者如果这是0，它也可能爆掉。所以这两种情况都非常非常糟糕。现在，我们理想情况下希望我们的 z 在1附近，或者 log z 在0附近。我们能做什么？嗯，你可能会注意到的一点是，如果你思考 softmax 的作用，就会发现这整个结构是过度参数化的。我可以把东西推入或推出，所以如果我给 u 加上一个常数，我就可以在不真正影响 softmax 输出的情况下操纵 z。它可以在归一化项和我模型的输出之间抵消。所以由于这个性质，我可以做的一件事是添加一个正则化项。这来自 Jacob Devlin 的论文，2024年——抱歉，2014年，他在论文中加入了平方 log z 项。这所做的是惩罚你的 log z 距离0有多远。

[01:08:55.120] 如果 log z 接近0，那很好，因为整个表达式在数值上是稳定的。这被称为 z loss 技巧。许多论文都使用过它。Jacob Devlin 等人在2014年最初开创了这种方法。然后通过许多开源模型，它又变得流行起来。我认为 Baichuan 是第一个采用这种方法的开源模型，但后来 DCLM、OLMo 等也在使用这个技巧来稳定它们的输出 softmax。所以这是一个出奇有效的东西。那么假设我们已经处理了输出 softmax 的不稳定问题。现在，我们必须把注意力转向另一个潜在问题，那就是注意力机制本身。这是许多退化（degeneracies）发生的地方。已经开发出许多技术来控制注意力操作产生的不稳定性。实际上，我要说的核心要点是，如果你遇到不稳定，如果你

[01:09:54.630] 能在某个地方加入层归一化（LayerNorm），它可能会控制住这种情况。这实际上在某种程度上就是 QK 归一化（QK norm）这个想法背后的设计理念。那么你要做的是，记住我们的 Q 和 K 将要相乘，然后进入 softmax。所以在标准的注意力操作中，我会进行层归一化作为预层归一化（pre-layer norm），与 QKV 相乘。然后我会得到我的 Q 和 K。它们会通过矩阵乘法相乘，然后我进行 softmax，再与 V 相乘得到加权平均值。然后我输出后面跟着的任何东西。对。所以这是我们通常的注意力机制。现在，如果我们在 Q 和 K 相乘之前只是加入一个层归一化，会发生什么？如果我们这样做，那么这个矩阵乘法（以及因此的 softmax）的输入，大致具有相同的尺度。

[01:10:46.210] 通过使用 RMSE 范数来归一化 Q 和 K 的规模，它们的尺度将始终大致保持为一，从而让 softmax 运算保持稳定。许多不同的模型都采用了这种做法，它最初源自多模态领域。一些最初构建多模态模型的研究人员发现了 QK 归一化，Idefcs 和 Chameleon 模型确实使用了这一技术，并证明了其有效性。随后，许多其他开源语言模型也意识到，同样的技巧完全适用于稳定语言模型中的注意力机制。我认为这种做法如今已经非常、非常标准了。QK 归一化实际上是现在大多数大模型都会引入的一种非常标准的干预手段。从许多不同的训练运行来看，它似乎不会影响性能，但它确实能够有效防止各种注意力退化问题。

[01:11:43.400] 就我所见，我们最初在前置归一化中有层归一化。现在，我们在每个块的非线性变换之后也添加了它们。而且，我们现在把它们同时用在了 Q 和 K 上。实际上，我认为这触及了人们应用在这个领域中的各种稳定化技巧。现在，我要讲的最后一类稳定性干预措施——坦白说，这一招不那么流行，更多是我见过的谷歌特有技巧——就是对数几率软上限。这是一种更为强硬的干预手段，有些人会采用。在 QK 归一化中，我们控制的是 softmax 的输入，并期望其输出表现良好。如果我们真的、真的想要强制输出表现良好，我们可以直接取对数几率——也就是直接进入 softmax 的那些值——然后给它们设一个上限，让它们永远不会过大或过小。这几乎是一种硬性约束。

[01:12:45.863] 当然，它被称为软上限，不过 Tanh 函数本身是有界的。这出现在 Gemma 模型中。我记得 Gemma 2、3 和 4 都使用了对数几率软上限的技巧。它们所做的是，取出所有注意力层的对数几率，然后在某个值处对它们进行软上限处理。英伟达的一些研究人员确实做了相当不错的工作，对这些稳定性干预措施进行了系统性的比较。他们发现，如果从一个基线模型开始，你可以尝试各种各样不同的干预措施。QK 归一化就在其中，它表现略好一些，因为它允许你稍微调高学习率。但如果你只做软上限处理，实际上最终会导致性能下降。所以这里会发生质量退化。这是一种非常强硬的干预手段。你在 softmax 中永远无法表达超出某个程度的非常自信的信号。所以它确实有一些负面影响，但这是稳定注意力输出的一种非常安全的方式——或者抱歉，

[01:13:48.755] 是稳定注意力的输入，也就是进入 softmax 的那些对数几率。这就是稳定性组件的全部内容。我可以在这里稍作停顿。之后我会讲各种注意力相关的内容。好了，行，没问题。那么，今天我想讲的最后一部分是你可以对注意力机制做的各种干预。正如我在本讲开头所说的，今天我只会谈所有你能对注意力机制做的各种干预。所以如果你是想听状态空间模型或者线性时间注意力的话，很遗憾，今天不适合你。我确实想讲的，也是如今真正普遍实施的注意力干预措施，包括分组查询注意力，它通过减少头数来真正节省推理成本，以及稀疏注意力或滑动窗口注意力，它最初来自 GPT-3 那一类模型家族，但现在已经被大多数追求长上下文的模型广泛采用，

[01:14:55.690] 除非它们在用那些比较特别的 SSM 技术。所以我先从分组查询注意力开始，也就是 GQA 或 MQA。我要先说明这类方法的需求来源。然后你们应该就能明白这个技巧是什么，以及为什么它相当自然。目前，我们一直在讨论训练、建模等等这些内容，但让我们先暂停一下。现在，让我们想想部署。你已经训练好了这个非常大的模型。现在，你需要向大量用户提供服务。而你要为服务支付成本。在抽象意义上，你需要为两种不同的资源付费。你需要为你的浮点运算量付费，也就是你执行的计算。但你还需要为另一件事付费。你得为内存访问付费。因为内存访问也会影响你的系统特性，比如延迟、利用率。

[01:15:50.280] 所以你希望这两者都尽量小。现在，让我们想想在训练期间，或者换句话说，在预填充阶段，当人们给你提示词、你查看提示词时会发生什么。在这种情况下，你所需的总算术运算量是批次大小乘以序列长度乘以隐藏维度平方这个量级。这大致就是你所面对的计算规模。当然，我们做的是二次方注意力，所以有 d 平方项。再看总内存访问量。我们这里的内存访问是怎样的呢？我们有批次乘以序列长度乘以隐藏维度，再加上 softmax 的成本，其中有一个 n 平方的项。然后对于投影操作，我们还有一个 d 平方的项。所以这里的算术强度是相当不错的。它是 1/k。这里的 k 是头的数量。所以你需要——抱歉——头维度足够大，这样才能在乘一些大小合理的矩阵。

[01:16:46.090] 而你还有一个 1/bn 项，所以你的序列需要足够长，或者你的批次大小需要足够大。只要这两点都满足，你的 GPU 就能被充分利用。很好。你正在使用你所有的资源。现在，我们已经完成了训练。而现在，我们要为我们的用户服务。我们怎么为用户服务呢？我们要生成词元并把它们发送给用户。如果我们这样做，我无法并行化生成过程。我要做的是，生成一个词元。我以它为条件。然后生成下一个词元。我要一个接一个地重复这个过程。这就是自回归语言模型的宿命。我们必须这样做。要做到这一点，高效的方式是把我所有过去的键和查询都保存在所谓的 KV 缓存中。

[01:17:32.050] 所以 KV 缓存会维护过去那个 Q 点乘 K 的矩阵。然后每当我需要计算新内容时，我可以复用过去已有的那些子矩阵。我实际上只需要计算新的查询-键交互，来填充这个矩阵的其余部分。所以每个我之前计算过的子矩阵，我都可以保留。我只需要计算我新的那部分。这大大节省了计算量。这很棒。但问题在于，现在我们的算术强度就没那么好了。你可能也猜到了，这种 KV 缓存方法会一直读取参数。每进行新的一步，我都需要读取我的参数。我需要做这些点积运算。而且每一步我都只做一次。那么现在，我得到的是什么呢？嗯，我的总内存——哦，抱歉。我的总算术运算量是一样的。

[01:18:30.500] 我仍在乘同样的矩阵，只是增量式地进行，而不是一次性完成。但正因为我是增量式地做，现在我的内存访问模式是批次乘以序列长度的平方再乘以隐藏维度，加上序列乘以隐藏维度平方。这第二项就不那么令人愉快了。过去只需要 d 的平方，但现在是 n 乘以 d 的平方。如果我们计算算术强度，也就是这两者之比，现在得到的是 n 除以 d 再加上 1 除以 b。所以现在我们需要的是大批量加上短序列长度，或者我们需要非常大的模型维度。所以如果我们想高效地部署一个小模型，这就不太妙了。这真的很难处理。这个 n 除以 d 项，也就是这里的第一个项，即序列长度除以隐藏维度，在我们做这种增量计算时非常难以降低。

[01:19:28.850] 这确实是个很难解决的问题。这就引出了 MQA（Multi-Query Attention，多查询注意力）的想法。通常，在注意力操作中会有多个头，每个头有各自的键、值和查询。这通常就是标准的做法。但我们可以做的一件事是，也许可以让所有头共享相同的 K 和 V，每个头之间唯一不同的只有查询。如果我们这样做，就能大幅减少需要在内存中移入移出的数据量。因为 KV 缓存现在显著变小了，这些在所有头之间是共享的。这显著减少了总内存访问量，同时也改变了算术强度。我们之前讨论的关键项是 n 除以 d 这一项。现在，我们有了 h 乘以这一项。如果我们有很多头，这个 h 项能让我们显著减少——抱歉，是提高算术强度。这是相对于之前的一个重大改进。这给我们带来了显著的效率提升，但 MQA 的问题在于，右边这里，所有查询共享一个值和键，

[01:20:48.850] 这样做确实会丧失相当大的表达能力。所以在系统效率和表达能力之间存在权衡。你可能会想，是否存在一个最佳点，让我们不必在表达能力和计算之间做出太显著的取舍？这就是 GQA（Grouped Query Attention，分组查询注意力）发挥作用的地方。原始 Transformer 是多头的，每个头都有查询和键。在多查询注意力中，所有头只有一个键和值。在分组查询中，我们减少了键和值的数量，但保持查询数量不变。所以我们有了一个可以调节的比例，即键头或值头的数量，同时让总的头数远大于这个值。这让我们能够非常简单地控制表达能力和推理效率之间的权衡。

[01:21:40.920] 还有来自 DeepSeek V2 的其他技巧，多头潜变量注意力，我下次会简要提到，它有不同的分解结构和不同的权衡方式。但 GQA 真正的好处在于，在实践中，这种权衡非常有利。所以如果你用多头注意力，你的性能——这我记得好像是 T5 时代的数据——这是下游模型表现，这是每个样本的时间。你想尽可能降低后者。使用多头注意力，你有最好的性能，但代价非常高。使用 MQA，代价更低，但性能也低得多。同样，如果你缩小模型以试图达到性能目标，性能会差得多。GQA 确实能做到两全其美：推理成本很低，同时性能几乎和完整的多头注意力一样。

[01:22:36.970] 你会看到这种 GQA 分组结构，如果头数减少不多，基本上就能获得大部分性能收益，这让你在保留大部分表达能力的同时获得显著的推理改进。Percy 稍后会详细讲解推理机制，但这应该让你对为什么如今几乎所有模型都采用 GQA 结构有个概念，因为它能在几乎不损失表达能力的情况下，带来大量的推理成本优势，这非常关键。好的。对 GQA 或 KV 缓存有什么问题吗？嗯。既然你有这么多关于超参数的经验法则，你在多大程度上还在搜索超参数，而不是直接利用这些经验法则？我觉得是两者结合。我认为每次模型训练都有一些关于什么可以调整的假设。你在很多报告中会看到这一点，超参数通常不是人们改动太多的地方，但在很多报告中你会看到每次只改一个架构变化。

[01:23:50.360] 但要一次性改掉所有东西是非常罕见的。我认为谷歌是为数不多似乎确实会大幅变动配置的机构之一。Gemma 系列做了一些相当有趣的事情。例如最近的 Gemma 4 版本，现在每一层都有独立的嵌入，以此来控制内存使用和浮点运算之间的权衡。他们做了一系列非常有趣的事情。哦，对，后面那位同学。有没有在训练过程中动态改变这些参数的实验？在训练过程中。让我想想。权重衰减，是的。权重衰减，人们会结合学习率来调整。这实际上是一种人们会用的启发式方法，效果很好。除此之外，我不太确定还有什么其他超参数会在训练中经常改变，尤其是架构相关的参数只会在训练不兼容的情况下改变，所以训练过程中没法真正去改它们。

[01:24:49.800] 所以我觉得权重衰减是我能想到的一个。其他的一般都是固定的。对。所以 MQA 不只是推理时的问题，它是……没错。对，你训练时也要用一定数量的重复。好的。最后我要讲的是滑动窗口注意力，这是一个非常古老的想法。GPT-3 实际上用了这个——如果你读那篇论文，他们会说，我们在全注意力（每个位置都能注意到过去所有位置）和带状矩阵（只能注意固定窗口内所有位置）之间交替。OpenAI 在早期对不同注意力模式有一些研究。但实际上，在过去一年里，这个想法变得非常非常流行。在大的全注意力和更局部的注意力之间交替，这个想法确实找到了一个最佳点，可以管理长上下文性能，同时推理成本不至于太高。

[01:25:57.090] 我认为在开源模型中更近期的复兴，我可能会说，Cohere 的 Command A 是我见到的第一个这么做的，它的结构是每四层有一层全注意力处理所有位置，中间三层使用滑动窗口注意力，只能看到局部结构。当然，随着你往上走，抱歉，在这个情况下是往下走，因为他们把图倒过来画了，当你沿着这些块往下走时，你是在把局部信息聚合成全局信息。所以底部的局部注意力当然能获取更全局的信息，但这样能让你管理长上下文的成本，而不必去使用状态空间模型或更特殊的干预手段。这个方法效果很好。还有一些创新是在改变长距离信息的嵌入格式，比如去掉 RoPE 这类位置编码。这样一来就完全没有位置嵌入了。

[01:26:57.080] 所以实际上，你几乎可以说是在看“袋子”，其中短距离信息仍然能获得位置信息。因此人们会做各种各样的干预，涉及这两种嵌入，以及局部和全局结构的交替。我要说的是，这就是注意力机制，而总体而言，如何管理长上下文成本与性能之间的权衡，仍然是一个活跃的研究领域。这也是绝大多数架构工作和变更仍然在发生的地方。我们基本上看到许多其他模型采纳了这一思路——最引人注目的是 Llama 4，Gemma 4，OLMo 3，它们都采用了滑动窗口注意力和全局注意力的组合，而在它们的方案中，使用的是完整的 RoPE 而不是 NoPE 作为嵌入方式。所以正如我所说的，这种做法正变得非常非常流行。Qwen3.5，我把它放在了右边，它们实际上有一点不同，因为它们每四层交替使用一个名为“门控 Delta 网络”的状态空间模型和一次全局注意力层。

[01:28:04.630] 所以这是同样的交替结构，但它们在廉价层上使用了不同的方案。就它们而言，它们使用的是状态空间模型——我将在下一讲中解释那是什么——来代替滑动窗口局部注意力。但你可以看到，我认为这是过去一年中一个新的主题：开放模型确实在努力应对长上下文性能的问题。而解决这个问题的方式，至少到目前为止，就是采用那些混合模型——它们不只是纯全局注意力，也不只是纯廉价注意力，而是某种介于两者之间的混合体。而这种方法在迄今为止的许多模型中似乎都表现得非常好。好，不错。所以正如我想强调的，当你纵观所有这些模型时，你开始看到很多共同的模式，并且希望能形成一种整体的理解，了解你可以做哪些事情，以及哪些是好的默认选择。我们也看到了在如何处理上下文和位置嵌入方面存在很多差异。甚至分词方式也有一些不同。所以这些模型之间确实存在差异，但也有共性，希望这些共性能给你

[01:29:02.800] 一些直觉，当你在做作业、在排行榜上尝试各种调整的时候，能有所依据。谢谢。

CS336 2026 · Lecture 3 · Architectures, hyperparameters

视频：官方视频
字幕：en-US · manual captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：原语言字幕稿，待校对

正文

[00:05.600] So today, we're going to talk about architecture, which, at least to me, has always been pretty inscrutable. And so I'm going to take the approach of just telling you everything. I'm going to go through all of the modern papers. And we're going to just look through what has everyone done. And so I've titled this everything you didn't want to know about architectures and hyperparameters, because I think we all wished we lived in a world, where the only things you had to know were like VC dimension or something, very simple theoretical tools, but that's not really where we are. OK. What we are going to do is, we are going to try to understand architecture from a survey lens. The best thing to do, better than listening to this lecture, even, is for you to go out, and train your own models, and try different architectures. That's, by far, the best thing to do. That's part of the philosophy of the course. But we're not going to be able to cover the whole design space of all the different architectures that are out there. That's not something that we have the compute or the time to do. So my opinion is, the second best thing that we could do is to try to learn from the experience of others.

[01:10.190] What has everyone else done? What are the choices that they are making? And by looking at a broader, somewhat zoomed out picture, maybe we can start to understand, oh, these are the kinds of parameters and choices that are fixed across all effective architectures. And these other ones can be varied without impacting how the model perform. So I'm going to talk about, basically, transformer variants, like, what is the modern transformer starting with the Vaswani paper? And then as we go to more modern, more recent architectures, what do they have in common? And then what are we allowed to vary or not allowed? But what do people vary as they go through this. So I think many of you have taken an NLP course of some kind or at least seen a transformer. So you've probably seen the very vanilla transformer from Vaswani, et al.

[01:59.860] There's some fairly standard choices that you make. You say, oh, transformers don't have positional dependence, so we're going to add a position embedding. And what do we do, we're going to add some sines and cosines. We're going to have information processing through ReLU. And then we're going to have a post norm. I'll talk about what exactly that is later. And when you look at your assignment, your A1, you're going to notice some differences between the standard or the vanilla transformer and what we've asked you to implement. Well, we're going to ask you to move the LayerNorm to the front of each transformer block or the nonresidual layers. We're going to ask you to implement something called RoPE. And we're going to ask you to implement something called SwiGLU GeLU and not ReLU.

[02:41.660] Why do we pick these? One reason is, we've copied a lot of this over from but Llama, so did everyone else. Really, I think, if you were to train your own language model, I think you'll quickly run into this question, oh, there's so many choices. What do I choose for all of these things? And so let's now walk through all of these different models? The way I think about architectures is to look at all the different things people have done and say, what are the things that people have done? Can we pick and choose from those? Percy always makes fun of me for this a little bit, but I try to look at all the different models that come out each year to try to make this lecture. And last year, I thought, oh, there's just a couple papers. It's going to be fine. And then I look through all the things, and there's a lot of papers. There's Qwen2, and Gemma 3, and InternLM2. And then there were even more.

[03:31.130] There's Nemotron-4 and Qwen2. And oh, my goodness, there were 19 new dense models. And so last year, I had my work cut out for me. And this year, I thought, well, there can't be that many new LLM releases. It's got to be slowing down. People can't keep training 20 dense LLMs per year. And that's technically right, there aren't as many dense LLMs. Initially, I was like, oh, there's Qwen3. Gemma 4 just came out last Thursday, so I put that in there, and Olmo 3. There's only a couple. And of course, I have to give a shout out to Percy's own AP model trained with Marin. And I was like, oh, we'll just have a few things to cover. But it turns out, if you start looking, there's a lot of different models. And so the fact that we have so many different models, most of these actually are MoEs, mixtures of experts. And I'll be talking about that tomorrow rather than today.

[04:16.320] Because we have such a big diversity of models, we actually get a pretty good picture of all the different choices that we can make. So I made this little table. We'll come back to this little table at the end of the lecture. But basically, at this point, starting with the original transformer, there's been actually quite a few autoregressive language models trained on the same class of things. And you can ask questions like, what are the different vocabulary sizes? Or what kind of layer norms do we use? Or what kind of position embeddings do people use? And we see some fairly clear trends. I'll be talking about this as we go. So the goal here is that we're going to cover a couple of different things. We're going to cover common architecture variations. So these are different building blocks of the transformer. And after we've established what the standard building blocks are, like, what do we use for the nonlinearities?

[05:11.810] Or what do we use for position embeddings? Then we're going to talk about hyperparameters. We're going to go down even lower detail and say, what is ff_dim? Should we make that a multiple of four or multiply the hidden by four to get ff_dim? How many vocab elements should I have? And then after that, we're going to talk about very low-level tricks of how to get models to train stably. And the reason why I'm going to talk about that in this lecture is because these stability tricks have a pretty close connection with the architecture variation. One of the things that higher level I want to impress upon you is that architectures are actually a very complex set of trade offs. What does the architecture have to do? Well, it has to learn from data, so it has to generalize. It has to train efficiently on GPUs. And it has to not blow up. Halfway through training, if your training loss is just go down like this and then suddenly blow up, that's no good at all. So all these different requirements end up getting baked straight into the architecture. And that's why these things are a little bit messy and a little bit complex.

[06:16.155] But you should keep that in mind. And that's why things are, in many ways, not so elegant. So we're going to start with the core architecture piece. And as a high-level view, I think the way that I see a lot of the architecture stuff, looking, basically, historically, is in the early days of starting with a transformer until GPT-3 or so. There's a lot of experimentation that happens. People try lots of different things. There's no gold standard that everyone has unified on. And then Llama 2 comes out. And everyone's like, wow, Llama 2 is great, I want my own Llama 2. And so everyone starts training Llama 2 alikes with minor variation that people have. And then finally, last year, we saw really big differences or a trend towards architecture modifications that make training more stable. And this year, we see lots of trends towards architecture variations that enable longer context dependence. So there are these big themes that are happening.

[07:16.140] But really, I think you see this big point when Llama 2 comes out, and everyone's like, wow, I want to train some of that. And then suddenly, or not suddenly, but after that, people are starting to explore once again. So it's cool to see all of these different changes. OK. I think people can disagree about a lot of things on architectures, but there is one thing that everyone agrees on. If you take the transformer paper, I think a lot of people say like, the transformer people got most of the things, except this. And the thing that they really did not get right or I think, most people agree, they did not get right is where you put the layer norm.

[07:53.250] So in the original transformer paper, the layer norm goes in what you would call the residual path. So in the transformer, you have the residual stream that's x, that runs through the whole network. And then you are going to be applying attention, which does some computation. And it will add back a delta back into the residual stream. And then, in order to make sure that these gradients are stable across layers, a layer norm is placed at the end of each of these components. Now, instead of putting the layer norms in the residual stream, there's an alternative, I'll refer to this as prenorm, in which you can put the layer norm outside of the residual stream, but before each of the computations. So you can put it before the multihead attention, you can put it before the FFN. We'll call this prenorm. The nomenclature will get a little bit confusing. You can call this postnorm for now, but let's call this residual norm, because you're putting the norm in the residual layer.

[08:56.740] Basically, all modern language models push the layer norm outside of the residual stream. This is just like a thing that basically, everybody does. There is one funny exception, but it is OPT350M. And if you all are familiar with language models, we know, OPT, in general, was a mess of a language model. And OPT 350M is even more so because I don't why only that model has a postlayer norm in the residual stream. OK. So this is one of the things that everyone agrees on. And so you might wonder, why is this such a unified thing across all of the different models? And if you look at some of the early work, studying, where do you place the layer norm style research, what you really see is that the early motivation for a lot of this was, when you train a transformer, you need to do a warm up. Actually, modern transformer training still does warm up as well.

[10:01.390] But you definitely need to do warm up when you train. Now, wouldn't it be nice if we could remove the warm up. So that was the initial motivation for a lot of this research. But people quickly realized that removing the warm up had very serious issues in terms of the stability and convergence of these things. So if you did postnorm plus layer norm, which is basically the original transformer thing, you got this purple dashed line, oh, you just don't converge as well compared to doing something like prenorm, you can ignore the other terms, you would get much nicer convergence even without warm up. So this was the original motivation. But really, what people quickly realized is that moving the layer norms outside the residual stream has some pretty important implications as you make your network deeper and as you start to grapple with stability issues.

[10:54.440] To me, I think the gradient attenuation issues are the most clear. When you talk to people who do architecture design, I'm not really one of the people that deeply engages in this. But one of the things that people often say is, keep your residual stream clean. So in this case, you have your x's coming in from the on, the prenorm side. And this x propagates all the way up to the top, all the way up to your final output. And that allows gradients to propagate in the backward pass, straight through this. That makes gradient propagation very simple, which improves both stability and signal propagation. And that's what people realized very, very quickly, that if you do something like prenorm in blue at initialization, the gradient sizes remains the same, because you have this nice, straight through propagation in the backward pass. On the other hand, if you have post LayerNorm, you have these complicated effects that happen, because your LayerNorming, each time,

[11:55.002] you're going through a transformer block. And that's going to change the norm of your gradients as you go backwards through. So you can see, from the principle of keep your residual stream clean, that prenorm makes a lot of sense. People also realized through experimentation that this also improves stability, in general, that the sizes and frequencies of gradient spikes were improved under prenorm compared to postnorm. And this is a figure from Salazar and Ngyuen, who were one of the first ones, I think, to study this phenomena carefully. I think this is the reason why it's stuck around. Stability and the ability to go deep are both very, very important for modern large language models.

[12:37.920] And so this idea of moving your LayerNorm outside of the residual stream is one that basically everyone has adopted. So now, if putting LayerNorms in residual streams is bad, why does LayerNorm have to be at the start? Of course, we have prenorm, which is before our computation. But we could have it after computation as well. That's equally good, at least under that logic. And that's exactly right. Many recent models, like Grok, or Gemma 2, or OLMo 2, have this structure where they've moved the LayerNorm after the computation. So it's a postnorm of a kind, but it's outside the residual stream. Other models still actually just put LayerNorms everywhere. They put a LayerNorm here. They put a LayerNorm after. I'll get to this later as we talk about stability. But one of the other lessons that seems to have held up very well is, if you have stability issues, you can sprinkle in LayerNorms everywhere.

[13:33.620] And that will generally improve stability. It's almost very strange to be saying this, because it's so ridiculous. And yet, that statement has actually been proven right. Every time people have encountered stability issues, they say, oh, but what if we just throw a LayerNorm into attention? Turns out, that works too. We'll get to that later. OK, that's postnorm or double norm, in this case, where you have two LayerNorms here. OK. The other thing that you can do is, in the original transformer, you have the layer norm, which is this operation right here. So you have your activations x, you're going to mean subtract, divide the variance, and then scale it back up. And this works just fine. It's not like this is wrong. And many models have successfully trained on this scheme. But basically, most or all modern models, I think, use RMSNorm, which doesn't subtract the mean or add a bias term.

[14:23.610] So it's just a scaling down and scaling back up. So you can see this in the equation here. And really, LayerNorm is more expressive than RMSNorm. So there's really representationally no reason why you have to use RMSNorm. But RMSNorm is nice, because in practice, there's really no expressiveness loss. RMSNorm models just as well as LayerNorm. But more importantly, it is faster. This is the part where the systems and architecture codesign starts to come in. Firstly mentioned in the previous lecture, this idea of arithmetic intensity. We want to keep our GPUs hot by doing matrix multiplies and other very intense computations.

[15:06.360] We do not want to be wasting our GPUs by having them move little tiny bits of memory back and forth. That's a very inefficient use of our very powerful GPU. And so what we want is to remove operations that are small and involve memory movement, but don't give us much expressive power. So by that view, what we really want to be doing here is, if the mean subtraction and addition isn't really doing much for us. Just get rid of it. You might think, OK, why does this matter? We're just optimizing this teeny tiny operation that accounts for, in this case, something like 0.17% of the total floating point operations of our system. But as Percy mentioned, it's not really about the flops. The flops are the floating point operations we do, that's multiplying matrices, but that's not runtime. Runtime is a much more complicated object. And statistical normalizations, things like LayerNorm, even though they're only 0.17% of the flops, depending on your workload and depending on the setup,

[16:11.840] can be up to 25% of the runtime. That's crazy. On tiny models, this can be really, really big, because you're still having to move all these parameters back and forth from fast to slow memory and vice versa when you're doing these operations. So data movement is really, really important. And RMSNorm can still matter a lot because of this. So you can see the difference here. The arithmetic intensity is in white. And then you can see the flops involved in black. And you see that LayerNorm has a very low arithmetic intensity, which is the operation we try to want to remove as much as possible. Yeah, question over there? The movement for normalization is so disproportionate compared to density [INAUDIBLE]. So for something like tensor contraction, which is, in this case, matrix multiplies, the majority of the workload is multiplying, whereas for stat normalization, the majority of the workload is memory movement.

[17:07.210] And memory movement is quite slow. So imagine the case, where moving something is almost all the compute. Then you're still paying quite a bit here, because activations can be quite large. Yeah, I think the percent runtime in this case is quite extreme. This is like in the tiny models with matrices that don't really generally make sense in modern workloads, but this is giving you a sense of why this is a free optimization. And you do see this. This is another paper in which people were evaluating different architecture interventions, Narang et al in 2020. I think this was a Google paper. And they show, for teeny tiny transformer of a 200 million parameters, you get more steps per second.

[17:45.220] That's the third column over here when you switch to RMSNorm. And in fact, you actually get better performance, which I don't think is something that you're guaranteed, but it's a nice bonus, regardless. So you get a free systems win by just moving to RMSNorm. And so basically, everyone has decided to move over to this now. And in general, there's a more general version of this. There's no guarantee to any of the things I'm saying. But biased terms in transformers and neural networks are generally not that useful. So in the original transformer, the linear terms all have biases, but most implementations actually just drop the biases entirely. Once again, this is another example of something that's not very arithmetically intense, but fairly memory intensive, relatively speaking.

[18:31.050] And so you might as well just drop these and get the free systems win. There's also some cases, I'll just mention this offhand, where the bias terms can also induce stability issues, so they're useful in other ways. But really, I think the primary reason these are dropped is just to simplify things from a systems perspective. Cool. OK. So I think layer norms, the story is pretty easy. It's easy in the sense that what people do is fairly standardized. Our understanding, not like a deep theoretical level, but our understanding of what layer norm does is fairly good. Everyone moves the layer norm outside the residual stream, often prenorm. But I think this might partially be because Llama 2 did that. And we roughly have a sense of how to use LayerNorm to control things like gradient spikes and keep signal propagation nice. Related to that, we also now basically always use RMSNorm. And you hopefully understand the general principles here of basically, just dropping bias terms.

[19:34.920] And that allows us to keep our system more arithmetically intense while keeping the expressive power the same. I think the unsatisfying thing about a lot of architectures is that, you can't really reason about this beforehand, but we don't beforehand that dropping the bias terms is OK. But from a lot of experimentation and now, collectively acquired knowledge, we roughly that dropping the bias terms on both the linear and RMSNorm is OK for typical language modeling workloads. This is the kind of statement that we can make on the basis of what we do when we look at a variety of different models. Any questions for LayerNorm stuff? Good. OK. So now, I'm going to talk about activations. And there's a whole zoo of activations. There's just a lot-- ReLU, GeLU, Swish, ELU, GeGLU, SeLU, SwiGLU, LiGLU. And what are these things? I think, at one point of my more stats ML training, I thought to myself, I will never learn these things.

[20:36.090] I will make it a point of pride to never know what a SwiGLU is. But now, it's actually very important for us to actually have a general sense of what these objects are and which parts of these names actually matter for performance. So you can build and train a language model on just a fairly vanilla activation. I guess, Chinchilla is probably the best model out of that group. But even if you just want a ReLU, you can train a reasonably performant language model using just that activation. There's nothing wrong with that. And if we move to GeLU, which is a Gaussian error unit, and really, the only difference is this tiny divot at the bottom here, which really, for the most of the activation doesn't change anything but changes the gradients right near zero. Then you can train models like GPT 3. That's a perfectly good large language model, not modern by modern standards, but perfectly fine. But then we get to the gated linear units, like SwiGLU and GeGLU.

[21:36.240] And these are really where most of the action is. This is very similar to LayerNorm in that, I think almost all credible modern language models use a gated linear unit of some kind. OK. So what is gated linear unit? So these are gated activations. So if we want to look at something like a feed forward layer, we can just look at this first part. This is a very standard ReLU feed forward. I have my x. I hit it with a W1. I entrywise threshold at 0. And then I hit it with another W2. I get my output right. Very straightforward ReLU network. I don't say this as my personal experience, but another thing that is often said in architecture design is that gating is often very helpful. So if you apply that very general heuristic, what you might get is to say, OK, well, instead of just having this entrywise ReLU, why don't we also have a gate? And the second gate, the second term here, this is just going to multiply the output of my ReLU, entrywise.

[22:39.070] And I have a second matrix V. OK. Now, this is just going to modulate the output of my ReLU. And then I'm going to do everything else the same. So instead of just having xW1, W2, I have xW1, and I'm going to gate that with xV. This is another activation the same size as this. And then I'm going to down project it back with W2. OK. So what is this? Now, this is a ReGLU. You make these names by adding the first activation, in this case, ReLU and GLU, so the ReLU gated linear unit. And gating has been a very effective other primitive in architecture design. And it turns out that this is very effective in language modeling as well. So if you take something like a GeLU, we've already talked about that. That's the ReLU, but with a little divot at the bottom here, you will get a GeGLU. And if you take a SwiGLU, which is x times a sigmoid, then you will get a SwiGLU.

[23:39.440] So this is a squish times the rest of it. And this really covers a lot of the modern models. Generally, the Google folks have used GeGLU, so like the Gemma models, the T5 models are those. And everything that's a Llama descendant uses a SwiGLU. So PaLM and the Llama descendants are all SwiGLU models. I would say that SwiGLU is probably the more dominant one, but honestly, amongst the gated units, doesn't really matter. Now, here's a side note that will be a semi-important piece of trivia later. If you look up here, you will notice that there are more parameters for the gated model, because I have this parameter V. And so if you do a little bit of math, I now have three matrices instead of two matrices. What you should do is, you should maybe use a smaller feedforward dimension by a factor of 2/3, in order to keep the parameter count the same. So this is roughly the idea of, well, I want to keep the same number of total parameters as my original MLP, but I now want to make it gated.

[24:48.412] So I'm going to make the feedforward dimension, which is the output dimension of this W, a little bit smaller by 2/3. So this is a general rule of thumb that people have followed, but it's not really an iron rule. The original Noam Shazeer paper that proposed this had some very small deltas originally, but they're consistent deltas. And I think to his credit, I think a lot of his papers have these error bar assessments of training multiple replicates and checking to see if they're better. And if you look, the GLU variants are almost always consistently better than the nonGLU variants. And this is a parameter matched comparison, because Noam Shazeer is always doing this 2/3 adjustment to make sure that all of the models have the same total number of parameters. So this is quite nice. It's, in some ways, a free win. Almost everyone uses a GLU.

[25:46.870] There have been other more controlled systematic comparisons. This is the same paper I was talking about before, Narang et al in 2020. Google, actually, in the 2020s, did quite a few nice large scale architecture comparison papers, although with a T5 architecture and not an autoregressive language model. And they basically comprehensively compare things like GLU. And you see, once again, if we look at the SwiGLU, or the GeGLU, or the GLUs in general, they do significantly better at loss or the other downstream metrics. Fairly compelling on these papers, also clear from now a lot of model training runs that SwiGLU and GLU are good. So there's a lot of variations in gating, but really, the important single axis to know is that gating for these nonlinearities is actually quite important, gives you a nice boost without much of a computational cost. That's not to say that gated linear units are necessary.

[26:47.400] I mean, GPT 3 was that-- I think, the Nemotron 340B model used a Squared ReLU, which is a crazy choice, but that works too. Both of these models are perfectly performant, but it's actually quite rare to see anything that's not trained on a gated linear unit. So evidence is pointing towards consistent gains on using these gating tricks. So those are, I think, the more consensus choices for things that we can do in architecture. Now, this one, I think, is a really fun idea, but one that, I think, now, the test of time has shown maybe is not quite as good or maybe not as popular of an idea. Normally, we do our transformer blocks serially. We compute our attention, then we compute the MLP, one after the other. If you're very systems-minded, you might say, well, this introduces a bottleneck. I have to wait for the computation of one to do the other.

[27:42.130] If they were, instead, in parallel, I could bring to bear some new and cool systems optimizations, potentially. So you might ask, could we parallelize the transformer block? And this was originally an idea that was in GPTJ, which is an open source attempted replication of GPT 3. And very interestingly, I think GPTJ has been surprisingly influential in propagating a lot of ideas. I mean, PaLM as well. Google is actually surprisingly bold with the architectures that they do. But the description in PaLM, which you can see in their report, is the following-- instead of nesting this, which is the sequential firm at the top, you're just going to add together the output of the MLP and the attention layer and just add both of those back into the residual stream. If you implement this right, you can actually share a lot of the components. You can share the LayerNorms. You can fuse the matrix multiplies.

[28:40.140] This allows you to potentially get additional systems optimizations. And I think a lot of the people that have been influenced by Google. So Cohere, I know, was founded from one of the former transformer authors. They do a lot of Google-inspired optimizations. They followed this architecture, but not very many others. This has been an approach that has really fallen out of popularity over the past, I think, two years, I think mainly because optimization of the serial form has gotten sufficiently good that the system's gains from the second one just isn't worth the small hits to representation power that you end up getting, going from parallel to serial. Effectively, you can think about it, as you've lost half of your depth. And that can be a deleterious thing to do to your model. So in terms of the architecture things, actually, the fact that this is so short should suggest to you how much the original transformer formulation has somewhat stood the test of time, because the only thing I'm really talking about changing here is where the norms go, or whether we have bias terms, or whether we get the MLPs. But those are actually pretty minor changes compared to all the things that you can do.

[29:56.530] Now, those of you that are carefully paying attention might say, but wait, there's a lot of transformer alternatives that change the attention. Yes, you'll have to wait until next lecture, because today, I'm just only going to cover core attention-based methods. And next lecture, I'll throw in a little bit of state space model stuff. But as long as you're in this dense attention land, actually, the architecture from the original transformer paper is pretty close to what we do. So you see quite a bit of this. So just now going back to this, blue here is RMSNorm, black is LayerNorm. You see most of the modern models are RMSNorm models, serial versus parallel layers. The blue one is parallel, the rest is serial. You see mostly serial layers, prenorm versus postnorm. Some of these ones that I marked as postnorm are actually pre and postnorm.

[30:46.180] And then these ones on the right, these are GLUs, almost always, with the exception of things like Falcon, which use a gated linear unit. But almost all of these are really gated linear units for modern models. So you can see the trends quite visually from what I'm telling you. OK. So really, the thing that is very different across implementations, and I think a place where a lot of the architecture stuff is still in flux is how you do position dependence and incorporate information from other positions. So the core attention component, in some sense. So there are lots of different ways that you can encode position into a transformer. And just to remind you right, this is very, very important, because attention is positionally independent. They're just inner products, so you can just shuffle them. And attention would be the same if you don't have a position embedding.

[31:39.990] The original transformer had sine and cosine embeddings, a Fourier transform intuition that if you have sines and cosines, then you can recover position from that, no matter what. A number of other large models that followed soon after that used absolute embeddings, where each position had its own different embedding. And then several other Google models like to use relative embeddings. And here, you're not adding embeddings into the word vector embeddings, but instead, you're adding a vector to the attention computation itself. So if you're three positions off, the attention matrix gets a different offset added to it. And models like T5 and Chinchilla use this scheme. The thing that has really become pretty dominant in terms of position embedding is this class of embeddings called RoPE, which some of you may be familiar with. Most models past 2024 use this type of embedding.

[32:37.540] And it's remarkable, given that RoPE, in some ways, came out of nowhere. Originally, I think this was also a GPTJ innovation from, I think, not very well-known blog post and paper combination from an author in China. But really, it has some really interesting ideas for why you would do something like RoPE. So RoPE is a relative position embedding. And a relative position embedding, let's make an opinionated stance that I should not care about the absolute position of any words. So if you A and apple appear together, even if it appears at the start or at the end, in RoPE embeddings, they should get the same result. And we want to represent it in this way. So I have an embedding f, and I have another embedding f. And these are going to take in the identity of the words x and y and the positions absolute of i and j. And I want this to be equal. If I take the inner product of these embeddings to be equal to a function, that only depends on the relative difference.

[33:50.530] And every existing embedding before it didn't really fulfill this equality. Sine is not relative, because it has these absolute cross terms that are not relative. Absolute position embeddings, just by the name of it, is obviously not relative. And then relative embeddings, technically, these are relative, but they're not embeddings, because they're just adding to the attention matrix. So there's no inner product structure that you can extract out of this guy. So given this, you might ask, is there a nice way that we can truly have this relative embedding? And the idea is very cool. It's really just looking at other properties about angles and cosines. So we want our embeddings to be invariant to absolute positions. And we know that inner products of any kind are invariant to arbitrary rotation. So the idea is to say, I'm going to take my semantic word vectors, the ones that are independent of any position.

[34:47.000] So this is my starting point. And then I'm going to rotate each of these vectors, in this case, in 2D, based on the position that the words appear. So just as a simple example, let's say we have the sentence, "We know that." "We" appears as at position 0, so I'm not going to touch that at all. I'm just going to keep that where it is. The word "no" is at position 1, so I'm going to rotate it by some angle. And that's my one position rotation. Now, what happens if I apply the same idea to the following sequence? Of course, we know. In this case, "we" and "no" are still adjacent. They're right next to each other, but their absolute position has shifted. "Of course" comes before "we know now." In this case, I'm going to rotate the word "we" by two positions, because it's two index, 0, 1, 2. So the word "we" is in the second position number 2, so I rotate by two.

[35:43.880] I rotate "no" by three positions, because it's in position number 3. And what do you know the relative angle between these two is still separated by 1. So this is a very, very simple idea of just using rotations to represent position. And if we do that, then anytime we take an inner product, those inner products are going to be invariant of absolute positions. Now, you might say, well, in two dimensions, that's pretty easy, because you've only really got one choice, you've got clockwise and counterclockwise. But in high dimensions, there's an infinite space of ways that you can rotate vectors. So what do you do in D dimensions? Well, you do the simplest possible thing, and it works. The simplest possible thing is to reduce it to the 3D case repeatedly. So you have a D dimensional vector. Just cut it up into chunks of two. And each pair of two dimensions gets rotated. And the theta at which these things rotate vary.

[36:38.450] Some of them are very low frequency, so they rotate very slowly, so they can capture long range dependence. Some of them rotate very quickly, so they capture things like, are they neighbors to each other? And then at the end, after I've rotated every pair of vectors, I get my final embedding. So this is the RoPE approach. The paper, if you read it, has a very complex motivation about complex numbers, but really, I think the intuitive way, at least to me to think about it, is to just, you want to rotate by reducing to the two-dimensional case, and you're just rotating every pair of coordinates. Gemma 4 just came out on Thursday. And they have another different kind of fun thing that they do, which they call, I think, proportional RoPE or p-RoPE, which is a really strange way to just say that the only thing they rotate is the first two coordinates, but that's another valid thing that you can do as well.

[37:31.560] So there's a lot of different things that you can do in this space that end up working. OK. In practice, what you're going to end up doing is, you can take your vector, and you can make a sparse multiply with sines and cosines. And this is going to be giving you some way of rotating your input vectors x's. So x times w times r, this is going to be your final embedding that you get. And finally, this is a sine and cosine, which looks a little like sine embeddings. But it's really important that I'm multiplying with these sines and cosines rather than using them as embeddings, because that means that there are no cross terms. And this is purely relative. There's no absolute position information that you'll get out of inner products. If we really, really wanted to get into low-level details, and you ask, how do I actually implement this thing? You're going to have to do that. You have your usual attention stuff.

[38:24.730] And then what you do is you generate cosine and sine angles based on the position IDs of where your sequence is. And then you're going to apply those cosines and sines onto both your queries and keys for your attention computation. And you can either apply them as a matrix multiply, or you can go through and apply them manually, just as a rotation. Fairly straightforward. And you would do this at the attention level rather than at the very bottom, to enforce position invariance, every time you're doing attention computations. So that was RoPE. It is a little bit confusing, but once you understand the geometry of just rotating things, it's actually fairly straightforward. OK. I'm going to pause here for one moment, in case anyone has questions about the various architecture bits. We're going to then talk about even lower level details about hyperparameters, so yes. Do you know of any papers that do a higher dimensional? Higher dimensional rotation.

[39:22.120] It's a good question. I don't think so. But a higher dimensional rotation, like any 2D rotation in the space would just be a variant of this. You could certainly do any one manifold that is a closed loop. I have not seen that. Yes. [INAUDIBLE] What do you think is the best way to distill this kind of knowledge from papers and technical reports? It's a good question. I don't know if there's a way beyond some combination of looking broadly enough to get a pattern, which is what the procedure I'm trying to do in this lecture here. And then the other one is to try it yourself, even a much smaller scale, to form an intuition in a theory for how these things come together. I think those two are really the right ways. I think reading any single paper in isolation is very, very difficult, especially now, because no single paper seems to give any full detail for a lot of language models these days. Oh, lots of questions now. OK, good.

[40:19.550] We'll go in. Yeah. So I have a question on the parallel layers and [INAUDIBLE] layers. Yeah, I understand, the modern models are thinking of the resource efficiency, so they use the parallel layers. But there's a difference between accuracy for these two patterns. I want to know what's the degree of accuracy. Is it big enough or also small enough to allow the current model trainers to ignore that, or [INAUDIBLE]? Yeah, I think that's actually really mixed. So if you read the original PaLM paper, I think they're very confident about the use of parallel layers, like no performance drop, 15% systems utilization improvement. So if you read just that, you'll say, oh, it's just as good. But I think a lot of the later Google models have stopped using this, which you can take on as an implicit signal that actually, there might be some losses. And once again, this one is a little bit hard to get precise numbers on, because no one's done the ablations, as far as I know

[41:20.340] on parallel versus serial, controlled nice ablations, at least. Yeah. What's the difference between P-RoPE and RoPE if you might have read [INAUDIBLE]? Yeah. I mean, this difference is really just, which of the coordinates you rotate? You don't rotate most of them, because a lot of the-- I mean, the argument originally, I think, is that the low frequency parts just aren't rotating very much. And so you can drop them if you're really strapped for of extra space. And this is really an optimization for teeny tiny models, where you don't have very much hidden dimensions to have activations for. Yeah. So the relative embeddings, not having an inner product. Is that because it only applies to keys basically? I was trying to understand. Yeah. So they applied both to the keys and values, which is why you get this relative effect from where you are. You want to not have cross terms.

[42:23.150] So if you look at the sine and cosine embeddings, then you'll not only get the original vectors, you'll get these weird cross terms between the position embeddings and the word embedding themselves, and so on and so forth. And then you can back out what the absolute position is. So even sine and cosine embeddings are not like pure relative position embeddings. You have to accept the premise that the relative embedding is what you want. But once you do, you end up at the RoPE solution somewhat naturally. [INAUDIBLE] Yeah, Yeah. So what's the issue? So the issue with this is that it just can't be factorized as a product. That's more of an aesthetic problem. If your constraints are, I need it to be relative, and I need it to factorize as f of xi and f of yj, then this is not a solution in that class. To be fair, there's a lot of embeddings that work this way that do work, like l of i

[43:22.200] and other kinds of approaches do this inject into the attention matrix. And they do reasonably well. It's not necessarily the one that's become the dominant approach, is what I can say. Cool. OK. Great. Now, we'll talk about hyperparameters. And I think hyperparameters are really something that you start to engage with once you actually have to train a model. When your knowledge about language models are abstract, you don't have to care about any of these. But once you have to instantiate it, you start to ask questions like, well, how big should the feedforward size be? How many heads should I have? What should my vocab size be? And you might also have questions of, what should my weight decay or dropout be? Do I even need to regularize? I have a lot of tokens. So do I need regularization? And do I need very deep models or very wide models? What are the right kinds of things to do here?

[44:19.150] And all of these, if you start out with no knowledge, it's actually very daunting, because you have to search this very big high-dimensional space. But the space of things that people try is actually pretty small. And from that, maybe you can start to think about smarter search processes of where you want to vary things. One of the things that's a really consensus hyperparameter is this idea of the ratio between the feedforward size, which is the output of your first matrix multiply in an MLP and the model dimension. So this is really the ratio of the two dimensions of your W1 as well your W2 matrix. This seems like a thing that's very important and controls the richness of your MLPs. So what should it be? Well, for whatever reason, it should maybe be four times your hidden dimension. And this is a rule of thumb that works remarkably well. And I will show you some data on why maybe this is a fine number to choose.

[45:18.520] There's a few exceptions. And funnily enough, the really extreme exceptions backtrack on that. OK. Exception number 1 is variance of the gated linear unit. I already told you about this. So if you were thinking about it, this is probably cached in your head right. GLUs have more parameters if you keep the same dimensions. So if you want to keep the parameter size of your MLPs the same, well, you need to scale down by 2/3. So most GLU variants, this means that you're going to end up with something like 2.67-ish. So everyone that's down here, 2.67 to 2.5, this is roughly applying this 2/3 correction. And then for whatever reason, the Llama 2 folks decided, well, we actually have very efficient attention heads with MQA, which I'll talk about later. And because of that, we can multiply this ratio by an arbitrary 1.33. And we'll get roughly 3.5. And so the Llama people arbitrarily chose a slightly different ratio, which essentially emphasizes the MLPs a little bit more.

[46:28.720] But really, if you actually look through all the papers, you'll find either 2.6-ish or 3.5 for GLUs, or 4 if you're doing non-GLU models. OK. There's another exception, which I find to be very funny, but also very, very cool, which is, throughout, as you read these technical reports, you'll find that most people are just very boring in their choice of architectures. They're like, we did Llama, but we changed one thing. But folks at Google are very bold, sometimes. And T5 is one of my favorite ones, because they have some really bold settings. They decided that instead of following this 4x rule of thumb, they decided that they want to have a 64x multiplier, which is way bigger than 4. And they have a reasonable argument for this as well, this another systems-based argument. They said, well, the bigger my matrix multiplies, the more efficient I can keep my hardware. So if I make this multiplier really big, then my matrix multiplies can potentially be more efficiently utilized.

[47:36.920] And some others, like Gemma 2, have also tried to really push a little bit higher on this. But really, T5 is astounding exception at 64. I don't think any other model has really gone that high in the feed forward multiplier. And empirically, if you look at other works that try to do more controlled comparisons of this ratio, I've taken this one from Kaplan in 2020. This is the classic neural scaling loss paper, where they do various controlled studies on language models. You'll see, this wasn't the point of the study. It was a scaling law study. But you'll see in one of the panels that they actually have ablation or sweep, where they change the feed forward ratio, and they look at the loss for a very small model here.

[48:25.520] But what they find in this paper is, there's a basin, where you start at about one, and you end about maybe 10, where this hyperparameter is pretty good and very, very flat. You lose very little relative to the optimal loss down here. And then if you get it really wrong, you get above 10 to 100 or something like that, then your loss starts really shooting up quadratically. And so a lot of these choices that range between 2.6 to 4, they're all falling into this relatively nice basin, so you're fine choosing those numbers right. So what can we learn about this hyperparameter? Well, the default choices have worked very well for nearly all modern language models. So you can safely choose that. T5 was a fine model or the version 1 T5 was a fine model. It wasn't a bad model. And so even radical choices can technically work, but it's probably going to be compute-inefficient. And I think the funniest part of this saga, this is the punchline of the T5 saga to me, is that they have a follow up model T5 v1.1 that's supposed

[49:29.560] to be the improved version of T5. And they go back to the standard 2.5 multiplier. So there's nothing explicitly stated here. But clearly, when they tried to update T5, they decided that they wanted to go back to a more standard multiplier, which I find to be a little bit funny. So that's the feed forward ratio, which now, you have a rough sense of what the right order of magnitude is. Now, let's talk about a different consensus hyperparameter. I always found this to be very strange when teaching 224n and just teaching students about this, which is, if you have a multihead attention, where you have multiple heads for your attention in your transformer, the canonical thing to do, the thing that almost everyone does is, if you have multiple heads, you make sure that the size of those heads, the head dimension is such that you have the same dimension as a single head transformer. So you always make sure that you divide the hidden dimension to basically, multiply with h.

[50:31.350] So in this case, you have h the number of heads, and the dimension of each head is d over h. So you multiply the two and you get d. For some reason, this is the rule of thumb. Of course, this doesn't have to be true. We can arbitrarily change the ratios between head dimensions and model dimensions. But most models do follow this guideline. And it turns out to work pretty well. And we can look at a variety of different models, classic and new. I have the latest and greatest coin as well. And you find, yeah, the ratios are roughly around one a model head, notable exception of T5 and even lambda, which is another Google model. But really, everyone sticks around one. And I think this is an interesting one. I think the thing about head dimensions that I'll end with here is, I think this is yet another forgiving hyperparameter. There's a couple ablations that people have done. There's, once again, a pretty wide basin around one that you can get away with.

[51:28.020] But that one's maybe not the most critical hyperparameter. I think maybe one of the most critical and interesting ones, I think, conceptually, is this idea of an aspect ratio. And to add an extra point here, when you scale models up or down, the way you usually do that is you fix an aspect ratio, like how wide your model is versus how deep it is. And then you make the model bigger. So the aspect ratio, in some sense, controls the entire depth to width trade off as you make models bigger. Now, you might wonder how deep should my model be. If you've been following all this stuff on reasoning and so on, you might think I need a really deep model or really shallow model if I want systems utilization. You might think that there's a lot of variation. And there is a lot of variation, much more so than other hyperparameters. But there's actually a fairly clear sweet spot that most modern models fall into. You don't really see models go too deep. And you also don't see models go too wide in either direction.

[52:29.110] You see, most models have a ratio of about 100 D model over n layers. So about 100s width for every layer that you have. I mean, this is true for GPT 3, or Llama, or any one of these models. And really, I think the considerations are partly a trade off between expressiveness and hardware. If you have an extremely deep model, they get very, very annoying to deal with, systems-wise. The deeper your model, what is the ways that you have for parallelizing them? Well, you might have to cut up your layers. If you cut up your layers, we'll talk about this in the systems lecture. Once you start cutting up your layers depth-wise, you have very serious issues in parallelization. Pipeline parallel, which is what this is called, is something that most people really, really do not want to deal with. Whereas width is much easier to parallelize.

[53:19.750] If you have a really wide model, you can cut that up very easily in your GPUs. Tensor parallel is what it's called. It's much, much simpler to deal with. And so in some sense, there are systems reasons to go wide. And maybe there's expressiveness reasons to go deep. And you end up at roughly 100. And I think one of the really interesting things about transformer hyperparameters is, there are a lot of hyperparameters that seem quite important, but they're also fairly forgiving. And people have converged roughly on the minimum. This is yet another plot from Kaplan, et al, which shows another sweep over hyperparameters for differently sized models. And once again, you see, regardless of the size of your model, roughly speaking, the optimum aspect ratio is fairly similar, and they live at about hundreds, maybe a little bit less, depending on how you want to do the accounting.

[54:11.660] But really, anywhere near 100 is a pretty safe bet for aspect ratios EK and others did a number of really interesting architecture variation experiments, in which their general conclusion on this was that, let's look at the top panel here, you have a lot of different kinds of models that you can have in terms of depth to width trade offs. But as you sweep the depth to width trade offs, you find that really, the only thing that matters, in some sense, is flops. As you increase the flops, the models get better. And that's really controlling the majority of the effects, not necessarily the aspect ratio. And so I think what has really emerged from this is the sense that there's a general forgiving band of hyperparameters that people tend to choose. And then you really worry about primarily, your system's utilization rather than expressiveness concerns, which are hard to reason about.

[55:08.350] Cool. And then maybe the last hyperparameter thing that I want to mention is vocabulary sizes. And this one's interesting to me, because there's a really clear difference between two classes of models. I think, in the early days of open source model training, there were a lot of monolingual models, whose only goal was to be good on English. And for those models, you had these much smaller vocab sizes, in the 30,000 range. And then post Llama, a lot of people were really interested in multilingual or production systems, these include closed source models like GPT 4. All of these have much, much larger vocab sizes. And these are roughly in the 100,000 to 200,000 vocab range. And you see, generally, that Google models have a ton more vocab. Llama derivatives roughly range at about 100,000 tokens. And then the monolingual models are about 30,000. This is somewhat clear.

[56:12.130] The multilingual models really do need much larger vocab to cover the whole space. Generally, the models on the right are also bigger. There have been scaling law studies, showing that the bigger your model, the larger the vocab it can handle. And so this is also partially driven by modern scaling trends, where the models on the right are generally bigger. No ones training large monolingual models anymore. So yeah. [INAUDIBLE] the style stuff. How to use those vocab sizes? Sorry, the question was, if you have multilingual models or-- Multimodal. Yeah. So I guess, it depends on the way that your tokens are encoded. But if you're tokenizing your images and things like that, then you need to have many more tokens to account for those. If you look at various open source releases, they'll have a different image tokenizer with its own vocab, which is quite large. Yeah. How valid is it to compare, I suppose, this provides for different [INAUDIBLE]?

[57:21.810] How valid is it to compare bits? Oh, that is a great question. OK. Yeah. That's not a hyperparameter question, but that's a good question. So what is the right way? So let me step back a moment and put us in the right mindset. So if we think about language modeling, language modeling is a generative modeling task. We are modeling the probability of a sequence. Now, as long as your sequence is fixed, you have adulterated it anyway. And you provide a probability over all strings, that's always valid to compare. At that level of things is always valid. Now, when you ask the question, is it valid to compare the bits per byte of two arbitrary tokenizers? Really, there's two things at play. The one thing is, did you touch the sequence at all? If you look at some tokenizers in the past before subword tokenizers, they would drop some tokens or drop some words. That makes the comparisons invalid. But modern tokenizers are complete.

[58:17.640] They can model any sequence, so that's not a concern. The other thing that you have to worry about is, are we length normalizing in any way? But for bits per byte, you're always normalizing with the same number, which is the number of bytes. And so this is always a valid comparison. So that's how to think about tokenizer comparisons. So for example, I think there have been results showing that comparing perplexity for a fixed tokenizers always leads to better downstream performance on downstream tasks, as the same thing improving for [INAUDIBLE]. Perplexity and BPP are dual to each other. So yes, if that's what you're asking. It's only yes and it's only no, because if you're comparing-- you could train compared to perplexity as [INAUDIBLE], but you're changing a different thing. [INAUDIBLE] OK. We'll have to talk later, because I'm not sure I understand the question.

[59:15.562] But I think that's an interesting set of questions. OK, good. OK. All right. So we're going through, really, the lowest levels of details of language modeling, which I think really exposes a lot of interesting ideas while we talk through this. And I think dropout is one of the end regularization. I think it's another very interesting class of ideas, also one that I think is very counterintuitive from your Machine Learning 101 intuition. So let's go through what I think is the standard argument for regularization. Well, if I'm doing language modeling, I have a lot of data. I have more data than I can process most of the time. Unless you're at Google, maybe even then, There is more internet data than there is flops. So I'm probably not even going to see the same data twice. So I'm only going to do a single pass on a corpus.

[01:00:10.540] And there's very good reasons and arguments to believe that a single pass of SGD or other optimizers is never really going to memorize my data very much. So this means overfitting is not really a problem, almost ever during compute constrained language modeling. Now, some people even actually only look at training loss, because they believe so strongly that overfitting doesn't happen in single pass SGD. Now, given this, you can sit and think about this. Should I use dropout or weight decay in language model training? You can think about it a bit. One unfortunate thing is that a lot of recent models don't talk about this stuff at all. It's really lower level details than tech reports are willing to expose. But if you look, actually, you find a lot of models do both, especially weight decay actually is a fairly popular intervention even for modern high-performance language models.

[01:01:11.970] This is very, very surprising. I mean, some of the dropout things maybe have gone out of favor, but weightdecay actually remains fairly popular. And this is very mystifying. Why is this? And this is one of the reasons why I think deep learning is hard. And this architecture lecture is very strange and hard. It's because these things interact in very strange ways. So there have been papers that have argued and shown nice evidence that weight decay is actually not a regularizer, sometimes. It actually interacts with the optimizer to essentially make optimization better. So if you look at the training versus validation loss across different weight decay settings on language model training for single pass SGD, you don't really see any difference. Weight decay isn't shifting things, so the validation loss is better. There's already no overfitting.

[01:02:08.652] Run the x equals y line here. So doesn't control overfitting, but if we look at different levels of weight decay, and not only just different levels of weight decay, we look at weight decay combined with learning rate decay. What we find is that the stronger weight decay runs, these blue dashed lines on the bottom do significantly better, because they start out slow, but they essentially end up converging to a much better minimum later. And this is generally true when we decay learning rate, not necessarily true when we're in constant learning rate, which is maybe somewhat more of where your intuition is coming from. So this is part of why it's very difficult to reason, a priori or from scratch, the behavior of all these different choices and why I think Percy and I have designed this class, so that you interact with stuff, because you might come upon this thing where basically, weight decay is actually an optimization intervention and not necessarily a regularization intervention,

[01:03:12.050] which is what you would expect here. So always keep that in mind that these kinds of unexpected effects can really start to kick in for these kinds of settings. Cool. All right. So to put everything together for hyperparameters, there's actually, for a lot of the maybe more hairy looking hyperparameters, actually just fairly standard choices that have worked well for everybody. Factor of 4 rule of thumb, keep your head dim and your number of heads equal to the model dimension. Pick an aspect ratio, roughly around 100. And if you ask about regularization, you want to maybe try a couple of things, because regularization actually does interact with optimizers in ways that are quite counterintuitive. So this is the thing that some people still do, even though you don't need the regularization at all. Actually, maybe I'll stop here, in case. Yeah. Are there any significant architectural differences maybe for diffusion model? Diffusions. That, I have not looked into enough, to be honest.

[01:04:16.995] There aren't that many people training. Big diffusion is one issue. And many of the models that have been trained are retrofitted. So I think the architectures are actually the same as a Llama-like model. But if you're asking the question, what's the optimal architecture if you were to train from scratch, I don't know what that is, actually, off the top of my head. Yeah. Do you have any explanation for why regularization mostly affects [INAUDIBLE]? Well, I guess, it's not that regularization, in general, affects optimization. I don't think people do dropout anymore, because it doesn't really interact well with optimization. But for example, weight decay, which is shrinkage to 0, that might allow you to use a higher learning rate, or it might allow you to decay faster. There are lots of ways in which all of these terms are interrelated. Cool. OK. Now, I've talked a lot about how to design expressive models by looking at all these other models that have been trained.

[01:05:10.220] One of the things that I'll highlight now is, over the last few years, a really big emphasis has not been on performance alone, it has actually been on stability. And this becomes an increasingly important concern as your models get more and more expensive to train. We've seen that a lot of these choices are forgiving. Everyone's doing similar stuff. And so you can mess with these, but you're not going to get a big performance difference. That's fine. But if your model suddenly blows up some part into training, you get these horrible-looking spikes all over the place, you might end up with a model that is actually not very good quality. Or it might be unrecoverable. You might have spent millions of dollars in training. And you get to a point where the model is no longer able to be trained any further. That would be a horrible thing to happen if you have a lot of compute that you want to spend.

[01:06:00.390] So you don't want to train models that look like this blue curve with spikes everywhere and these big gradient norms happening. So what do we do to fix these stability issues? I mean, this is really, I would say, a core, core issue. And if you have stability issues in language models, or in general, neural networks, there's a few usual suspects that you've got to start looking at. One of them is the softmaxes. And the softmax has two things that are both really bad for stability-- one of them is an exponential. You can see how that blows up very quickly. You also divide two numbers. And that's also a potentially very dangerous operation. So softmax is one place where you got to be extra, extra careful. And where are the softmaxes in a language model? Well, there's two of them. There's one on the output side, when we output our probability distribution. And then in attention, when we normalize the attention, there's going to be another softmax.

[01:06:56.080] So we can think of both of those as really danger zones for our model, especially the attention. Let's start with thinking about the output softmax. The output softmax can blow up on us. And one of the things that we can do is, we can try to control the normalizer problem. So let's think about the softmax calculation. We want to compute a log probability to compute the loss. Now, what is the log probability? Well, it's the output of your model u. And then you've got this log normalizer. This u is well-behaved, because in some sense, this is the output of your model. This is just the output of your residual stream with all the things that are added in. So if u is well-behaved, then the first term is well-behaved, if the model is being OK. Now, the second term, this log z, this might not be so OK. If z is really big or really small, even if the output of your model is somewhat well-behaved, it could blow up.

[01:07:55.750] And what is z? Well, it's an exponential. So it could potentially blow up very quickly on you. Or if this is 0, it could also blow up on you. So both of those directions are very, very bad. Now, we would ideally like our z to be somewhere near 1 or log z to be somewhere near 0. What can we do? Well, one of the things that you notice, if you thought about the action of the softmax, is, this whole thing is overparametrized. I could push things in and out, so if I add a constant to you, I can manipulate the z's without really affecting the output of the softmax. It can cancel out between the normalizer and the output of my model. So because of that property, one thing that I could do is I could add a regularizer. This is from Jacob Devlin's paper, 2024-- sorry, 2014, in which he adds this squared log z term. And what this is doing is it's just penalizing how far away your log z is from 0.

[01:08:55.120] And if log z is near 0, that's nice, because this whole expression is numerically stable. This is called the z loss trick. It's been used by a number of papers. Jacob Devlin and others initially pioneered this back in 2014. And then it's become popular again through a number of open source models. Baichuan, I think, was the first open source model to do it, but then DCLM and OLMo, and others have been using this trick to stabilize their output softmaxes. So this is a surprisingly effective thing. So let's say, we've handled the instability issues on the output softmax. Now, we have to turn our attention towards the other potential problem, which is attention. And this is a place where lots of degeneracies happen. Lots of techniques have been developed to control the instability that attention operations generate. And really, the high level thing that I'll say is, if you have instability, if you

[01:09:54.630] can throw a layer norm in there, somehow, it might control it. And that's really, in some sense, the design philosophy behind this idea called the QK norm. So what you do is remember that we have our Qs and Ks that are going to be multiplied together, and then they're going to go into the softmax. So in the standard attention operation, I'm going to layer norm as a prelayer norm, multiply with a QKV. And I'm going to get my Qs and Ks. Those will get multiplied by a matrix multiply, I'll softmax them, and I'll multiply that with a V to get the weighted average. And then I'll output whatever comes after right. So this is our usual attention. Now, what happens if we just throw in a LayerNorm before we multiply the Qs and Ks? If we do that, then that the inputs to this matrix multiply and therefore, the inputs to the softmax, roughly have the same scale.

[01:10:46.210] They're always going to have a scale of roughly one, because we've used the RMSE norm to divide the size of those Qs and Ks. If we do that, then we're going to keep this softmax operation stable. Tons of different models do this. It's originally from the multimodal world. Some folks who were making multimodal models initially discovered QK norm, Idefcs, and Chameleon really use this and proved it out. And then a number of other open source language models realize that the same tricks are entirely applicable to stabilizing attention for language models. And I think this is now very, very standard. QK norm is actually a very standard intervention that most of the large models now introduce. It doesn't seem to affect performance from lots of different training runs, but it does definitely prevent the kinds of attention degeneracies.

[01:11:43.400] And really, the way that I've seen this is, we have layer norms, initially in the prenorm. Now, we add them after the nonlinearities in each block. And now, we're throwing them in both the Qs and the Ks. And really, I think this is getting at the stabilization tricks that people apply to this world. Now, the final set of things that I'll talk about as a stability intervention, and frankly, this one is not as popular and more of a Google-specific trick that I've seen, but logit soft-capping is a much harder intervention that some people apply. So this one, in QK norm, what we're doing is we are controlling the inputs to the softmax and hoping that the outputs are well-behaved. If we really, really want to enforce well-behaved outputs, what we can do is we can take the logits, the things that go straight into the softmax, and we can just cap them off, so they can never be too large or too small. This is almost a hard constraint.

[01:12:45.863] It's called a soft cap, of course, but a Tanh is bounded at some value. And so this is in the Gemma models. I think Gemma 2, 3, and 4 all use the logit soft cap trick. And what they do is they take all of their logits for the attention layers, and then they soft cap them at some value. Some NVIDIA folks have done actually quite nice work doing systematic comparisons of these stability interventions. And what they find is, if you start with a baseline model, you can do all sorts of different interventions. And QK norm is here. And it does slightly better due to the fact that you can crank up the learning rate a little bit. But if you do soft-capping alone, you actually end up losing performance. So there is a quality degradation that happens. This is a very strong intervention. You can never express very confident signals in your softmax beyond a certain point. So it does have some negative consequences, but this is a very safe way of stabilizing the outputs of your attention-- or sorry,

[01:13:48.755] the inputs to your attention, the logits that go into the softmax. So that's the end of the stability components. I can pause for a moment here. And I'll talk about various attention things after that. Cool. OK. All right. So the last thing I want to talk about today is various interventions that you can make to your attention. And as I was saying at the beginning of this lecture, I'm only going to talk about all the things that you can do to dance all by all attention today. So if you were interested in hearing about state space models or linear time attention, sadly, today is not the day for you. The things that I do want to talk about, which are really commonly implemented attention interventions today, are group query attention, which really saves inference costs by reducing the number of heads and sparse or sliding window attention, which really originally came from the GPT 3-ish family, but have now really been adopted widely by most models that are

[01:14:55.690] looking to do long context, unless they're doing exotic SSM stuff. So I'll start with group query attention, or GQA, or MQA. I'm going to first set up the need for these kinds of things. And then you'll hopefully see what the trick is and why it's fairly natural. So for the moment, we've been talking about training, and modeling, and all these things, but let's take a pause. And now, let's think about deployment. You've trained this very big model. And now, you need to serve it to lots of users. And you're going to pay a cost for serving. And you're going to have to, in abstract sense, pay for two different resources. You're going to have to pay for your flops, the computation that you're performing. But you also have to pay for another thing. You have to pay for your memory accesses. Because the memory accesses are also going to impact your systems characteristics, your latency, your utilization.

[01:15:50.280] So you want both of these things to be small. Now, let's think about what happens during training or alternatively, prefill, when you're looking at your prompt, where someone gives you the stuff. In this case, the total arithmetic operations you have is order of magnitude, batch size, sequence length, hidden dim squared. That's roughly the size of things that you get. And of course, we're doing quadratic attention, so we've got d squared. We've got total memory accesses. What is our memory access that we have here? We have batch times sequence length times hidden dim plus the cost of the softmax, which has an n squared component. And then we've got a d squared component for the projections. So the arithmetic intensity here is pretty good. It's going to be 1 over k. This is the number of heads. So you need to have, sorry, head dims. So your head dims need to be big enough that you're multiplying some reasonably sized matrices.

[01:16:46.090] And you've got a 1 over bn, so your sequences need to be long enough or your batch sizes need to be big enough. As long as both of these are true, your GPUs are going to be fully utilized. Great. You're using all of your resources. Now, we have finished training. And now, we're serving our users. How do we serve our users? We're going to generate tokens and send it to them. Now, if we're doing that, I can't parallelize the generation process. What I'm going to do is, I'm going to generate a token. I'm going to condition on it. I'm going to generate the next token. And I'm going to repeat this process one by one. This is just the curse of autoregressive language modeling. We have to do this. In order to do this, the efficient way to do it is to maintain all of the passkeys and queries that I've had in what's called a KV cache.

[01:17:32.050] So the KV cache is going to maintain this matrix of Q dot K over the past. And then whenever I need to compute something new, I can reuse the submatrices that I've already had from the past. And I only really need to compute the new query key interactions that I need to fill out the rest of this matrix. So every submatrix I've computed before, I can keep. I only need to compute my new one. So this saves a lot on compute. That's great. But the issue here is, now, our arithmetic intensity is not so good. As you might intuit, this KV cache approach is going to be reading parameters all the time. Each time I have a new step, I'm going to have to read in my parameters. I'm going to have to take these dot products. And I'm going to do this once every step. And so now, what do I have? Well, you my total memory-- oh, sorry. My total arithmetic operations are the same.

[01:18:30.500] I'm multiplying the same matrices still, just incrementally rather than all at once. But because I'm doing this incrementally, now, I have a memory access pattern of batch by sequence, squared by hidden dim plus sequence by hidden dim squared. And the second term is not so pleasant. It used to be that it was just d squared, but now, we've got n times d squared. And if we compute the arithmetic intensity, which is the ratio of these two guys, now, we have n over d plus 1 over b. So now, what we need is large batches plus short sequence length or we need really big model dimensions. So if we want to serve a small model efficiently, this is not so good. This is really difficult to deal with. This n over d term, this first term over here, which is sequence length over hidden dim, is very difficult to reduce if we're doing this incremental computation.

[01:19:28.850] This is just a hard thing to deal with. So this leads to this idea of MQA, or Multi-Query Attention. Normally, you have multiple heads in your attention operation. And you're going to have different keys, different values, and different queries. That's normally how you do things. But one thing that we could do is, maybe we can keep the Ks and the Vs the same across all the heads. And the only thing that's different across the heads are the queries. If we do this, then this drastically removes the amount of items that need to be moved in and out of memory. Because the KV cache are now significantly smaller. These are all shared across all the heads. This significantly reduces the total memory access as well as the arithmetic intensity. And the key term that we were talking about here, we had the n over d term. Now, we have h multiplying this. And so this h term allows us to significantly reduce the-- sorry, increase the arithmetic intensity if we have a lot of heads. This is a significant gain over what we had before. So this gets us significant efficiency improvements, but the issue with MQA, this is on the right here, you have one value and one key for all these queries,

[01:20:48.850] you do, in fact, lose significant expressive power if you do this. And so there's this trade off between systems efficiency and expressiveness. And you might wonder, is there a sweet spot in which we can avoid trading off quite significantly expressive power and computation? And that's where GQA, or Grouped Query Attention, comes in. The original transformer is multihead. We have queries and keys for each head. In multi-query, we have one key and value for each. For all the heads in grouped query, we reduce the amount of keys and values, but we keep the number of queries the same. So we now have this ratio that we can play with, which is the number key heads or the number of value heads while keeping the total number of heads much larger than that. So this allows us to very simply control the trade offs between expressiveness and inference efficiency.

[01:21:40.920] There are other tricks from DeepSeek V2, multihead latent attention that I'll mention briefly next time, which have a different kind of factorization structure and a different set of trade offs. But really, the nice thing about GQA is that, in practice, the trade off is quite favorable. So if you have multihead, your performance, this was in, I think, T5 days, if I remember right. This is your downstream model performance. This is your time per sample. You want to reduce this as much as possible. With multihead attention, you have best performance but very high cost. With MQA, you have lower cost but much lower performance. Similarly, if you make your model smaller to try to hit your performance targets, you get much worse performance, GQA really does get the best of both worlds. Very low inference cost, nearly the same performance as your full multihead.

[01:22:36.970] And you see this GQA group structure, where if you have a small reduction in the number of heads, you basically have most of the gains in your performance, which allows you to keep most of the expressive power while getting significant inference improvements. And Percy will talk a bunch more about the inference mechanics later, but this should give you a flavor why models today, almost all, adopt this GQA structure, because it gives you a lot of this inference cost, which is really critical, without very much of a expressiveness hit. Cool. Any questions for GQA or KV cache? Yeah. [INAUDIBLE] given that you have so many rules of thumbs for what hyperparameters are [INAUDIBLE] to what extent are you still searching over hyperparameters versus exploiting these rules of thumb [INAUDIBLE]? I think it's a mix of both. I think every model training run has some thesis about what can be varied. And so you see this in a lot of the reports, where I think the hyperparameters are often not where people are touching too much, but you see architecture changes one at a time in a lot of these reports.

[01:23:50.360] But it's very rare to go and change everything up. I think Google is one of the only orgs that seems to really spice things up in a significant way. The Gemma series has done some pretty interesting things. The most recent Gemma 4 release, for example, now has an individual embedding for every layer in a way to control the trade offs between memory use and flops. It's a very interesting set of things that they've done. Oh, yeah. Back there. [INAUDIBLE] experiments with dynamically altering some of these parameters during training. During training. Let me think. Weight decay, yes. Weight decay, people change in concert with learning rate. That is actually a heuristic that people do. That works very well. Other than that, I don't know if there's a lot of different hypers that people change during training, especially because the architecture ones just make training incompatible. So you can't really change them while you're training.

[01:24:49.800] So I think weight decay is probably the one that I can think of. The others are usually fixed. Yeah. So [INAUDIBLE] MQA, it's not just inference time, it's a [INAUDIBLE]. That's right. Yeah. You train with a certain number of repeats. Cool. The last thing I'll talk about is sliding window attention, which is a really old idea. GPT 3 used, actually, this-- if you read the paper, they'll say, we alternate between full attention, where every position can attend to everyone in the past, and a banded matrix where you can attend to everyone within a fixed window. And OpenAI has some early work on these different kinds of attention patterns that you can use. But actually, this has become really, really popular over the past year. This idea of alternating the big, full attention and a more local attention actually hits a sweet spot for how to manage long context performance while not paying too much for inference.

[01:25:57.090] I think the more recent revival in open models, I would maybe say, Cohere Command A was the first one I saw do it, where they had this structure, where every four layers, they would have a full attention that attended to everything. The three layers in between would use a sliding window attention that would only be able to look at local structure. And of course, as you go up, sorry, in this case, down, because they ordered the diagram the other way, as you go down these blocks, you're aggregating local information into global ones. So the local attentions at the end can, of course, access more global information, but this allows you to manage the cost of having a really long context without having to go for something like a state space model or a more exotic intervention. And that's worked quite well. There's also some innovation, where people change the embedding format for the long range information, where they get rid of things like RoPE. So you have no position embeddings at all.

[01:26:57.080] So you're really looking, almost, at bags, where the short range information still gets position information. So people do all sorts kinds of interventions involving these both embeddings and alternating local and global structure. I'll say that this is attention and in general, how to manage the trade off between long context cost and performance is still an active area of investigation. It's a place, where the most architecture work and changes are still being done. We see, essentially, a bunch of other models adopt this idea-- Llama 4, most recently, Gemma 4, OLMo 3, they all do this combination of sliding window attention and full attention, in their case, using full RoPE instead of NoPE as the embedding. So as I said, this is becoming really, really popular. Qwen3.5, which I've put on the right, they're actually a little bit different, because they alternate a state space model called gated delta net and one full attention for every four layers.

[01:28:04.630] So it's the same alternating structure, but they're using a different cheap layer. In their case, they're using a state space model, I'll explain what that is next lecture, instead of a sliding window local attention. But you see, this is like, I think, a new theme over the past year, where open models are really trying to grapple with long context performance. And the way to do that, at least, so far, is to have these hybrid models that aren't just global attention, aren't just cheap attention, they're some of mix in between. And that seems to have worked very well so far in a lot of these models. OK. Cool. So as I was trying to emphasize, when you look across all of these models, you start to see a lot of patterns, and hopefully, a sense of general understanding about what things you can do and what things are good defaults. We also see a lot of differences in how we handle context and how we handle position embeddings. Even tokenization, there's some differences. So there are differences across these models, but there's also commonalities that hopefully, now, give you

[01:29:02.800] some intuition as you go out, and do your assignments, and mess with the leaderboard and so on. Thanks.

Architectures, hyperparameters

课程简介 ​

CS336 Lecture 3：架构与超参数——中文讲义笔记 ​

学习目标 ​

课程主线 ​

核心概念详解 ​

1. 归一化：LayerNorm → RMSNorm，pre-norm → post-norm ​

残差流原则 ​

归一化变体 ​

2. 激活函数：门控线性单元（GLU） ​

从 ReLU 到 GeGLU / SwiGLU ​

关键细节：参数匹配 ​

并行层 vs. 串行层 ​

3. 位置编码与 RoPE ​

问题背景 ​

RoPE：旋转位置编码 ​

4. 关键超参数：经验法则与合理取值范围 ​

前馈维度比例（ff_dim / model_dim = ratio） ​

头维度（head_dim） ​

长宽比（模型宽度 / 深度） ​

词表大小 ​

正则化：一个反直觉的结论 ​

5. 训练稳定性：三大技巧 ​

输出端：z-loss ​

注意力端：QK-norm ​

强行限制：Logit Softcapping ​

6. 推理成本与 KV 缓存：MQA 与 GQA ​

推理的场景 ​

MQA（Multi-Query Attention） ​

GQA（Grouped-Query Attention） ​

7. 长上下文：滑动窗口注意力与混合架构 ​

课程作业关联 ​

常见误区 ​

复习清单 ​

从原始 Transformer 到现代语言模型：架构选择的共识与分歧 ​

为什么我们需要谈论架构？ ​

LayerNorm 的位置：一个出乎意料的共识 ​

为什么用 RMSNorm 而不用 LayerNorm？ ​

激活函数：门控机制为何取胜 ​

位置嵌入：RoPE 的崛起 ​

超参数选择：经验法则背后的逻辑 ​

前馈比值：4 倍规则 ​

头维度与模型维度的比值 ​

长宽比：深度与宽度的权衡 ​

词表大小：单语与多语言的鸿沟 ​

正则化：为什么显式正则化仍然存在？ ​

稳定性：训练中必须面对的问题 ​

注意力机制的干预：从 MQA 到滑动窗口 ​

这些共识从哪里来？ ​

展望：为什么这些对你重要？ ​

CS336 2026 Lecture 3：完整中文逐字稿 ​

正文 ​

CS336 2026 · Lecture 3 · Architectures, hyperparameters ​