CS336 2026 · Lecture 9已发布

Scaling laws

日期: Mon April 27
讲师: Tatsu
课程资料: 9 项
内容产物: 4 项

课程简介

本讲把 Scaling Laws 作为预测与决策工具：观察损失如何随参数量、数据量和训练计算呈现近似幂律，并学习在对数坐标下拟合趋势。课程进一步讨论架构、优化器、批量大小和下游任务的尺度效应，以及 Kaplan 与 Chinchilla 对计算最优模型和数据配比的不同结论。

讲义 / Slideslecture_09.pdf打开链接↗论文Scaling Laws for Neural Language ModelsKaplan Scaling Laws↗论文Training Compute-Optimal Large Language ModelsChinchilla↗论文Chinchilla Scaling - A Replication Attempt复核阅读↗论文Deep Learning Scaling is Predictable, Empirically早期 Scaling Laws↗论文A Constructive Prediction of the Generalization Error Across Scales联合缩放↗论文Scaling Laws for Autoregressive Generative Modeling跨模态缩放↗论文Scaling Data-Constrained Language Models数据受限场景↗视频课程视频课程视频↗

CS336 Lecture 9：Scaling Laws 中文讲义

学习目标

完成本节课后，你应该能够：

理解 Scaling Laws 的基本思想：如何从中小规模训练结果外推大规模行为，以及为什么这是一种强大的工程范式。
掌握数据 scaling laws 的直觉来源（参数估计与非参数回归），并知道实际观测到的指数（约 −0.1）意味着什么。
理解“斜率稳定、截距可变”这一经验规律，并能解释它为什么对数据混合、架构比较、优化器选择等工程决策有用。
掌握临界批次大小（critical batch size）的概念、估计方法及其与计算规模的幂律关系。
理解 IsoFLOP 分析、下包络线法和曲面拟合法三种计算最优（compute-optimal）训练配置的估计方式，并了解 Kaplan 与 Chinchilla 结论分歧的原因。
认识到 scaling laws 不是自动成立的，需要精心选择 x 轴、超参数和参数计数方式，否则预测可能严重偏移。

课程主线

本节课从一个实际问题出发：假设你有大量计算资源（例如一万块 B200 使用一个月），需要训练一个前沿开源语言模型。你不可能在百万美元级别的大规模运行上反复调试超参数。你需要在小的规模上完成优化，并通过某种可靠的规律外推到大规模行为。Scaling Laws 就是这种“简单、可预测的规则”。

课程结构如下：

历史背景：缩放定律不是新事物，其思想可追溯到统计学习理论和 1990–2010 年代的 NLP 工作。
数据 scaling laws：最简单的单变量关系，从均值估计和非参数回归的直觉推导其函数形式。
模型与超参数层面的 scaling laws：架构比较、优化器选择、宽高比设计。
临界批次大小：一个关键的工程量，随目标损失呈幂律增大。
计算最优训练配置：Kaplan 与 Chinchilla 之争，以及为什么看似微小的选择会导致结论的巨大分歧。
实践启示：生产级模型通常不需要 Chinchilla 最优，而应“过度训练”以利于推理服务。

核心概念与原理

1. 背景：缩放定律的历史根源

Scaling laws 并非深度学习的全新产物。其思想与以下经典脉络一脉相承：

泛化界（Generalization Bounds）：理论机器学习一直关注“模型表现如何随样本量变化”。泛化界给出的误差上界通常依赖于样本量，这本质上是一种关于数据量的性能预测。
经验样本复杂度（Empirical Sample Complexity）：早在 1993 年，Bell Labs 的研究者就提出在小子集上拟合分类器、用错误率衰减曲线估计大数据集上的性能，这是最早的数据缩放定律实践。
NLP 领域早期工作：Banko 和 Brill 关于数据量与系统性能的研究；Kolachina 等人（2012）在机器翻译中拟合了幂次为 3 和 4 的幂律关系——这与今天讨论的幂律形式相同。
Hestness et al. (2017)：课程讲师认为这是神经 scaling laws 的起点。该工作覆盖语音识别、机器翻译、语言模型等多个系统，证明了数据规模与性能之间存在优美的多项式趋势，并提前讨论了涌现（emergence）、计算量 scaling 等话题。

解释：以上历史脉络说明，scaling laws 不是“纯神经网络时代”的新范式，而是从统计学习和经验机器学习中自然生长的思想。

2. 数据 Scaling Laws

2.1 经验观察

固定模型训练流程（模型大于数据集），以规律方式增加数据集大小，观察测试损失下降。在对数-对数坐标下，损失与数据量呈清晰的线性关系，即幂律（power law）衰减：

loss (N) \propto N^{- α} + c

其中 $c$ 是不可约误差（irreducible error）， $N$ 是数据集大小。双对数图中的线性关系意味着远离渐近线；接近不可约误差时曲线会偏离直线。

2.2 幂律的统计直觉来源

参数估计：若用经验均值估计高斯分布的均值，期望误差为 $σ^{2} / n$ 。对数后即为 $\log error = - \log n + \log σ^{2}$ ，斜率为 −1。这就是最简单的 scaling law。

非参数回归：若在 $d$ 维空间中估计任意光滑函数，将空间划分为大小为 $n^{- 1 / (d + 2)}$ 的小格，则估计误差约为 $n^{- 1 / d}$ 。对应的对数-对数斜率为 $- 1 / d$ 。

神经网络的类比：实际观测的指数约为 −0.1 到 −0.3（Kaplan 2020、Hestness 2017），大致对应 $d \approx 10$ 左右的非参数回归器。这是一个有用的心智模型：神经网络的行为类似一个内在维度约为 10 的非参数平滑器。但课程讲师提醒，这个论证部分依赖于内在维度的估计，证据未必完全可靠，可按接受程度采纳这种直觉。

解释：均值估计和非参数回归是课程用来推导 scaling laws 函数形式的 toy model，帮助理解为什么多项式衰减是自然的选择。神经网络的实际指数更慢，反映了其灵活性和高维输入的本质。

2.3 数据缩放定律的工程应用

数据缩放定律本身主要用于预测，但更有趣的是工程决策支持：

数据混合优化：对于两个数据源（如新闻和维基百科），可以训练小模型、拟合不同混合比例下的性能函数、找最小值、外推到大规模。这种方法成立的隐含前提是：混合比例只影响截距，不改变斜率。因此小规模下的最优混合在大规模下仍是最优。Data Decide（大规模实证研究）发现，不拟合缩放定律、直接选择小规模最佳混合方案，效果也很好——这与“斜率不变”论点一致。
数据重复使用（epoch 数）：标准训练下约可达 4 个 epoch 而不受损害，超过后实际缩放曲线会明显差于新鲜数据预测的曲线。修正的函数形式可以预测重复条件下的行为。
无限计算量的极限：即使允许无限次 epoch 和无限大的模型，性能提升也会递减，最终需要转向集成等方法。但注意：正则化、集成等干预措施改变的是截距，斜率几乎不变。
数据过滤的规模依赖：计算资源少时应激进过滤、只保留高质量数据；资源多时应放松过滤、利用更多数据以避免重复。最优过滤器不是规模的固定函数。

3. 模型与超参数层面的 Scaling Laws

3.1 架构比较

训练一组小规模的 Transformer 和 LSTM，绘制各自的对数-对数扩展曲线。如果 LSTM 的斜率（甚至截距）更差，就有证据支持扩展 Transformer 而不是 LSTM。这正是 Mamba、门控 DeltaNet 等论文的标准做法：展示新架构的扩展曲线优于普通 Transformer。

Ek 等人在 Google 的 T5 架构扩展研究捕捉到了许多后来被前沿模型采纳的架构趋势：高效注意力机制（如 Performer）扩展不佳；GLU（门控线性单元）在扩展趋势上持续优于其他选择；Switch Transformer 也有良好的扩展趋势。虽然当时的计算规模远小于今天，但这些规律仍然有效。

解释：这解释了为什么许多人把 scaling laws 视为“验证干预是否真正有效”的范式性工具：如果某个架构改动在扩展趋势中不占优，它很可能不是好的干预。

3.2 优化器选择

SGD 与 Adam 的对比（Hestness 的工作，语言建模 + recurrent highway network）：截距不同，但斜率非常相似。即使像优化器这样大的变化，也很少改变扩展趋势的斜率。这是一个反复出现的经验规律。

3.3 宽高比与深度

极少的层数（如一层）严重不利于扩展；大于 1 层后各深度竞争激烈，但总体上更多层在每个计算量水平都更好。
宽高比（每层宽度与深度之比）近似为尺度不变量（scale invariant）：不同规模下最优宽高比几乎不变（约在 100 维附近）。这让你可以放心采用“固定宽高比、放大规模”的策略。

3.4 参数计数的影响

并非所有参数都平等。Kaplan 等人在拟合扩展定律时发现，如果包含嵌入参数，曲线变得非常奇怪；排除嵌入参数后得到干净的趋势。但一个关键问题是：他们也排除了最后一层（softmax 层）的参数，因为嵌入矩阵和输出矩阵形状互为对偶。包含或排除这些参数对缩放定律的形状有重大影响。

课程原意强调：Scaling laws 是工程出来的，不是自动发生的。需要选择正确的 x 轴、确保超参数正确调优，才能获得可预测的扩展。这是使用 scaling laws 时最核心的元认知。

4. 混合专家模型（MoE）的 Scaling

MoE 是当前训练大模型的主流方式，其扩展规律可以通过 IsoFLOP 类分析量化：

给定计算预算，最小化损失时，总参数量越大，最优稀疏度越高（激活参数占比越小）。
增加“空转参数”（增加总参数量但保持激活参数不变）仍有助于降低损失——未激活的参数仍有价值。
稀疏度、总参数量、激活参数量之间存在可预测的扩展关系。不同的 Q 型概率：
- 需要区分“参数”这个概念在 MoE 语境下已被解耦为总参数和激活参数。

5. 临界批次大小（Critical Batch Size）

5.1 概念

噪声受限区间（noise-limited regime）：批次大小增加，梯度噪声降低，每个额外样本带来的收益接近完美——存在几乎线性的缩放。
偏差受限区间（bias-limited regime）：达到梯度的噪声尺度后，局部下降方向与全局最优方向之间总有偏差，增加批次大小的收益递减。
临界批次大小：从近似完美缩放到收益递减的转折点，是一个工程上的经验法则。

5.2 估计方法（McCandless / OpenAI 的方法）

选择一个目标损失值。
训练几个小模型，在不同批次大小下测量达到目标损失所需的步数和样本数。
归一化后，步数需求与样本数需求呈反比关系，形式上为 $S_{min} / B + B / S_{min}$ 类型的权衡。
解平衡方程，临界批次大小约为 $B_{crit} \approx S_{min} / E_{min}$ （所需最小样本数除以所需最小步数）。它平衡了收敛速度和总样本消耗。

5.3 与计算规模的幂律关系

临界批次大小随目标损失降低（即模型质量提升）而增大，且呈幂律关系。直觉：越接近最优，噪声的影响越明显，方差缩减的价值越大。因此大规模训练运行可以采用较大的批次大小。

6. 学习率的缩放

简单的宽度缩放心智模型：模型越大，同时改变的参数越多，学习率应越小。经验法则：学习率大致按宽度的倒数缩放。
另一种思路是重参数化（如 μP，Maximal Update Parametrization），通过调整初始化大小和优化器步长来使不同规模下最优学习率保持不变。
两种哲学（估计最优学习率随规模的变化 vs. 让最优学习率不变）都在大规模训练中有成功实践。

7. 上游与下游性能的差异

困惑度（perplexity）与参数量高度线性相关，非常可预测。但下游任务表现不一定与困惑度完全一致。课程给出了一个极端例子：在可预测的困惑度趋势中排名靠后的模型在下游任务上反而超过了困惑度更好的模型。Scaling laws 通常对困惑度（或训练损失）适用良好，但从困惑度到下游任务的迁移远不如表面看起来可信。

解释：预训练者应关注迁移问题，不能只报告困惑度。这是对“上游干净、下游可靠”这一直觉的警告。

关键公式与原则

内容	公式 / 原则
数据 scaling law（一般形式）	$loss (D) \propto D^{- α} + c$ ，双对数图为直线
均值估计	误差 $= σ^{2} / n$ （斜率 −1）
非参数回归	误差 $\approx n^{- 1 / d}$ （ $d$ 为输入维数）
观测的指数	约 −0.1 到 −0.3，对应 $d \approx 10$
联合缩放（Rosenfeld 形式）	损失 = 数据项 + 模型项（两个幂律之和）
计算最优配置	固定 FLOPs 时在数据量与模型规模之间取最优权衡
Kaplan 结论（近似）	数据规模 $\propto C^{0.73}$ ，模型规模 $\propto C^{0.27}$ （训练参数增长快于数据）
Chinchilla 结论	数据规模和模型规模基本同步增长，约 20 tokens / 参数
临界批次大小	$B_{crit} \approx S_{min} / E_{min}$ ，随目标损失降低而增大

核心原则：

对数-对数图上的线性关系 = 幂律 = 多项式衰减。
远离渐近线时，幂律成立；接近不可约误差时偏离直线。
斜率通常非常稳定，绝大多数干预只改变截距。
Scaling laws 只在正确的 x 轴、正确的超参数配置下成立——它是“工程出来的”。

课程作业关联

作业 3（预训练实践）中，你在训练小规模模型时会看到类似的数据缩放趋势。课程中展示的“对数-对数线性”现象应在自己的实验中复现。
作业中你需要在固定计算预算下选择模型规模和数据量，这直接对应到 Chinchilla 与 Kaplan 的最优配置讨论。先做小规模试探、拟合缩放趋势，再外推，正是课程推荐的做法。
课程中提到的“模型大于数据集”是指你处于“基本不重复数据”的幂律区间，这个条件也适用于作业的预训练实验设计。

常见误区

认为 scaling laws 是自然规律，不需要调参。实际上，不合理的预热方案、批次大小或参数计数方式都会导致缩放定律严重偏移。Scaling laws 描述的是“如果我继续用这个配方放大，结果会怎样”，而不是“所有放大都会这样”。
认为 Kaplan 或 Chinchilla 的系数是普适常数。二者的差异来自超参数、参数计数、批次大小等细节。任何一组系数都需要在自己设定的训练配置下重新验证。
在接近不可约误差的数据规模上拟合双对数直线。一旦进入渐近线区域，曲线会弯曲，幂律不再成立。必须确认自己处于幂律区间，或显式拟合常数项。
直接在下游基准上依赖缩放定律。下游任务迁移远不如困惑度干净。Scaling laws 的可靠预测对象是训练/验证损失（困惑度），下游性能不一定服从同样的规律。
认为增大批次大小总是无代价。超过临界批次大小后收益递减；批次大小的最优值随目标损失提高而增大，不是固定的。
把 Chinchilla 的 20 tokens/参数当作生产级模型的黄金比例。如果你要部署推理服务，你会倾向于“过度训练”（即训练更多 token、使用更小模型），因为推理成本往往超过训练成本。20:1 是训练计算最优，不一定是总成本最优。
忽略重复数据的后果。超过约 4 个 epoch 后性能明显受挫，且不能通过无限增加模型大小来弥补——需要采用其他方法（如集成）。

复习清单

[ ] 能说明数据 scaling laws 的幂律形式及其统计来源（均值估计、非参数回归），并解释神经网络的指数约为 −0.1 意味着什么。
[ ] 能解释“截距可变、斜率稳定”这一经验规律，并至少举两个应用例子（数据混合、优化器比较）。
[ ] 能说明为什么参数计数（是否包含嵌入/输出层参数）会显著影响缩放定律的形状。
[ ] 能描述临界批次大小的定义、估计方法（目标损失、步数/样本数权衡），以及它与计算规模的幂律关系。
[ ] 能区分噪声受限和偏差受限区间，并解释为什么越接近最优时批次大小可以越大。
[ ] 能说明 Kaplaan（约 0.73 数据 / 0.27 模型）与 Chinchilla（约 0.5 / 0.5）分歧的三个来源（参数计数、学习率预热、批次大小欠优），以及这些差异说明了什么。
[ ] 能概述 Chinchilla 的三种方法（下包络线、IsoFLOP、曲面拟合）各自的思路和优缺点。
[ ] 能解释 IsoFLOP 为什么是研究中的默认方法之一，并举例其在 MoE 扩展分析中的应用。
[ ] 能解释为什么生产级模型往往需要“过度训练”，而不是严格采用 Chinchilla 最优配置。
[ ] 能在自己的预训练实验中判断是否处于幂律区间，并正确选择 x 轴和超参数以获得可靠的缩放预测。

CS336 Lecture 9 深度解读：缩放定律——从经验规律到大规模训练的工程基石

导语

当你被要求在有限预算和时间内训练一个前沿语言模型时，最棘手的问题往往不是基础设施的搭建，也不是数据管线的设计，而是：如何在昂贵的大规模训练运行前，做出正确的架构、超参数和数据配比决策？ 你不可能在目标规模上反复调试；一次失误的代价可能非常高。

这正是缩放定律（Scaling Laws）登场的时刻。它提供了一种范式：在小规模下完成所有优化，通过简单可预测的规律外推到大规模行为。本文基于斯坦福 CS336《Language Modeling from Scratch》第九讲，系统梳理缩放定律的核心思想、历史脉络、实证方法，以及它在现代语言模型工程中的真实角色。我们将看到，缩放定律不仅是一种工具，更是一种接近信仰的工程哲学——而理解它的力量与局限，是每一个大模型实践者的必修课。

从理论到工程：为什么我们需要缩放定律

想象这样一个极端场景：你的基础设施团队已经就位，高质量的预训练数据集也已备好，现在你需要决定模型的架构、超参数、数据配比。这些选择在文献中往往有"最佳实践"可循——但如果你想构建超越现有技术水平的模型，照搬是不够的。你必须自己优化这些选择。

最天真的做法是什么？在大型运行上反复训练、反复调优。这显然不可行。于是，缩放定律给出了一个优雅的替代方案：让一切优化都在小规模下完成，然后用可靠的规律外推。

这种思路并非横空出世。实际上，它与机器学习理论中的经典问题——"我的模型表现会如何？"——紧密相连。理论家对此的回答是泛化界（Generalization Bound）：对于某个有限假设类，误差最多比训练集表现差多少。这些界通常依赖样本量，本质上描绘了损失随训练集增长的曲线。你可以说，缩放定律就是泛化界的经验版本——只是它更贴近真实世界，也更实用。

缩放定律的历史：不是新鲜事的"新范式"

很多从业者把缩放定律视为深度学习的专利，但事实远非如此。这门课的脉络清晰地表明，缩放定律有着深厚的历史积淀。

1993年，贝尔实验室。Cortes 和 Vapnik 等人提出了一个在今天看来无比熟悉的问题：在大数据集上训练分类器很昂贵，能否在小样本上拟合错误率衰减曲线，然后外推到大样本？这本质上就是数据缩放定律——而且发生在深度学习革命之前。

2001年，Banko 和 Brill。他们在NLP领域指出，与其在算法开发上投入，不如收集更多数据——因为系统性能随数据量增长呈现出惊人的可预测性。

2012年，Kolachina 等人。他们在机器翻译任务上研究模型性能随数据量增加的函数形式，最终得出与今天相同的幂律关系——幂次为3或4。这些多项式函数，正是我们在后续讲座中反复遇到的。

2017年，Hestness 等人（这门课教授反复强调被引用不足的一篇论文）。他们系统地建立了跨领域的缩放定律：语音识别、机器翻译、语言模型，均遵循优美的多项式趋势。他们甚至讨论了"涌现"现象和按计算量缩放的思想——这些概念在多年后的LLM时代才广为人知。教授感叹："很多我们今天看到的系统缩放现象，在2017年就能知道，如果当时认真读了这篇论文的话。"

这段历史告诉我们：缩放定律并非神经网络时代的意外发现，而是经验机器学习中一个长期演进的脉络。它从理论中吸收养分——物理学家对极限行为的思考、统计学家对收敛速率的分析——最终成为大模型时代的核心方法论。

有一个关键问题：这些缩放定律是纯经验拟合，还是有理论根基？教授的回应很诚实：缩放定律主要是曲线拟合的练习。理论提供的是一些候选函数形式——比如渐近行为分析、误差衰减速率——但并不能断言"幂律就是唯一真理"。这为后文的"缩放定律的不确定性"埋下了伏笔。

数据缩放定律：最简单、最自然的起点

幂律关系的自然涌现

数据缩放定律是最基础的一类：固定模型训练流程，规律性地增加数据集大小，观察错误如何减少。在双对数坐标轴上，损失与数据量的关系呈现出一条清晰的直线——这就是经典的幂律（Power Law）关系。

为什么幂律是自然的缩放形式？如果误差可以写成 1/n^α 加上某个常数项，那么在对数-对数图上减去常数项后，就会得到一条直线。在经典参数估计中（如估计高斯分布的均值），误差正好是 σ²/n——这就是指数为1的缩放定律。但神经网络的缩放指数要差得多：大约在 -0.1 到 -0.3 之间，远慢于参数估计的 -1。

这说明了什么？神经网络表现得像高维空间中的非参数回归器。如果你试图非参数化地估计一个任意光滑函数，误差大致以 n^(-1/D) 衰减，其中 D 是维度。神经网络的指数接近于 D≈10 的非参数估计器——它们从数据中学习的速度，大致相当于在10维空间中的最近邻估计。这个直觉虽然吸引人，但教授提醒：它依赖于"内在维度"的估计，证据可能有点站不住脚。不过，指数确实量化了模型的学习速率——这一点本身就是有价值的信息。

数据缩放定律的工程应用

数据缩放定律本身只告诉你"模型学习得有多快"，这对外推有点用，但对工程决策帮助有限。更实际的问题来了：

第一，数据混合比例。假设你有新闻数据和维基百科数据，应该用多少新闻、多少维基百科？传统的做法是用小模型做扫描，拟合一个描述不同混合比例如何影响性能的函数，然后外推到大规模训练。但现实往往更嘈杂——据教授的经验，在很多情况下，人们训练一堆小模型，直接挑选最优混合方案，然后信心十足地规模放大，根本不拟合什么缩放定律。这背后的合理性在于：缩放定律的斜率通常不变，只有截距改变——所以小规模下的最优混合方案，往往也是大规模下的最优方案。

第二，数据重复使用。随着计算量增长超过数据量增长，重复使用数据成为必然。经典研究表明，最多四个epoch不会造成明显损害，但超过这个点，实际缩放曲线会明显偏离新鲜数据的预测。更有趣的是，即使尝试各种干预（正则化、集成），斜率依然惊人地相似——你做的干预往往只是改变截距。

第三，数据过滤是规模依赖的。如果你只有有限的算力，激进地过滤数据是正确的——反正你不可能在网络全部数据上训练，不如只保留最高质量内容。但如果算力充足，你会希望训练更多数据（哪怕质量稍低），以避免在高质量数据上做太多重复。最优过滤器不是规模的固定函数——这是一个容易被忽视、但影响深远的洞见。

一个令人警醒的坐标轴陷阱

在讨论"无限计算下的预训练"时，教授插入了一个有趣的提醒：那张看似线性关系的图，实际上y轴是对数刻度，"线性"只是观测区间太小造成的假象。如果你只拥有计算量范围中的极小一段，很难区分多项式缩放和指数缩放——泰勒近似是真实存在的，放得足够细，一切看起来都是线性的。这个提醒在我们评估任何缩放曲线时都适用：保持怀疑，注意范围。

模型缩放定律：架构、优化器与超参数的量化权衡

架构选择的缩放视角

"Transformer真的比LSTM更好吗？"——这个问题经典的暴力解法是训练一个大规模LSTM，和同级别的Transformer对比。缩放定律方法则优雅得多：训练一堆不同规模的小模型，观察它们在计算量序列上的性能趋势。LSTM可能具有不同的截距，甚至不同的斜率——如果斜率更差，那么随着规模扩大，LSTM最终会输得更惨。

如今很多架构论文都会精心绘制这样的缩放趋势图：普通Transformer vs. 新提出的酷炫模型。新模型要么在上，要么在下——因为这才是"更好"的证明方式。教授特别提到Ek等人在Google的工作：他们对T5风格模型做了大量缩放研究，发现门控线性单元（GLU）在全部趋势上优于其他变体，Switch Transformer 的扩展性也不错——这些趋势在当时运行的计算规模远小于今日，却准确预测了后来驱动前沿模型的架构选择。这解释了为什么许多人把缩放定律当作一种"范式"：如果某个干预在缩放趋势中不成立，那它可能就不是一个好干预。

优化器与超参数的缩放分析

缩放定律可以扩展到优化器：SGD和ADAM对比，截距不同但斜率几乎相同。这再次印证了那个"神秘现象"——重大干预改变截距，却几乎不动斜率。对宽高比（aspect ratio）的分析则更精细：不同层数下，最优宽高比的位置大致不变（大约在100d附近），这意味着架构设计可以遵循"固定宽高比、扩大规模"的策略。类似地，注意力头维度也可以做同样的缩放分析，帮助你在不训练巨型模型的情况下建立信心。

一个重大陷阱：并非所有参数都是平等的

Kaplan等人的工作发现：如果把嵌入参数计入参数计数，缩放趋势会变得很怪；排除嵌入参数后，趋势变得漂亮。他们的处理方式是排除所有嵌入参数，只计算"非嵌入参数"——理由是"这些才是真正在做计算的参数"。但这直接影响了缩放定律的结论有效性和后续推断。教授强调：缩放定律不是魔法，可预测性是工程出来的——你需要选择正确的x轴，需要确保超参数设置正确。这是一门手艺活。

混合专家模型（MoE）中的参数价值

MoE已经成为训练大模型的主流方式，它让"总参数"和"激活参数"解耦。苹果和MIT的研究者做了漂亮的分析：当你追求最小损失时，更大的总参数预算会引导你选择更稀疏的模型（激活参数更少）。有趣的是，增加"空转参数"（同样的激活参数，更多总参数）依然能降低损失——未激活的参数仍然在帮助模型。这些关系同样具有可预测的规律性，并且可以写出函数形式。

临界批次大小与学习率：真正需要重新设计的两个量

当你准备训练全新的巨大模型时，大多数架构决策都会沿用经过验证的选择——但批次大小和学习率是你大概率需要重新设计的。批大小之所以关键，还因为上节课强调的系统原因：大批次是数据并行的基础。

临界批次大小：噪声受限到偏差受限的转折点

临界批次大小（Critical Batch Size）的思想很直观：达到某个点之前，增加批次大小有"完美回报"——因为每个额外样本都在减少SGD步骤中的梯度噪声，而你处于噪声受限区间。超过这个点后，你不再受方差限制，而受偏差限制——无论梯度噪声多低，你的局部下降方向与全局最优之间总是存在偏差。这时收益递减。

如何估计临界批次大小？课程给出了实际步骤：在不同模型规模下，测量达到特定目标损失的步数和样本数。通过观测数据拟合"最小步数"和"最小样本数"，然后让两者的归一化值平衡，解出临界批次大小。这个量的关键性质是：它随模型规模呈幂律缩放——模型越大，临界批大小越大。这在直觉上也说得通：越接近最小值，方差缩减的价值越高，因为你在优化越来越微小的差异。

学习率的两种哲学

学习率的选择，存在两个思想流派。第一种是经验缩放：模型越大，学习率应该越小——因为你在同时改变更多参数，或许应当移动得少一些。一个常见的经验法则是按宽度的倒数缩放学习率。第二种是重新参数化：通过改变初始化大小、调整优化器对不同部分的步长，迫使模型在不同尺度下具有相同的最优学习率——这就是 μP（Maximal Update Parametrization）的核心思想。两种方法都有大规模训练的成功案例，教授将在进阶讲座中详细展开。

上游与下游的断裂

最后，教授以一个警示收束这一部分：上游性能（如困惑度）和下游任务表现可能差异巨大。Yi Tay的架构论文展示了这个问题：某个模型的困惑度排名很好，但在实际任务上却被另一个困惑度更差的模型超越了。缩放定律是针对困惑度这类干净、规律的目标建立的——但从困惑度到下游任务的迁移，其确定性远不如乍看起来那么高。做预训练的人不该把"困惑度不错"当作万能的通行证。

Chinchilla 之争：为什么同样的方法会得出截然不同的结论？

两种对立的联合缩放定律

驱动性问题很简单：我需要更多数据，还是更大的模型？ 给定算力预算（FLOPs），如何在参数量和数据量之间分配？

Kaplan等人通过联合缩放定律（数据量 × 参数量 ≈ FLOPs）求解，得出结论：当你增加算力，应该训练越来越大、越来越大的模型（数据量按C^0.27增长，参数量按C^0.73增长）。这直接催生了GPT-3时代的巨型模型风潮——几千亿参数、甚至万亿参数的稠密模型，都是Kaplan的预测在推动。

但2022年，DeepMind的Chinchilla作者用几乎相同的方法得出了截然相反的结论：模型应该相对更小，数据应该相对更多。他们提出了著名的"每个参数20个token"经验法则。问题来了：为什么两份同样合理的分析会得出如此不同的结论？

Chinchilla的三种拟合方法

Chinchilla论文提供了三种不同的缩放定律拟合方式，这个设计本身就很优秀——对建模假设保持稳健：

方法1是下包络线法：取所有训练曲线的下包络线点，对每个FLOPs预算找出最佳性能，然后绘制"最佳模型规模 vs. FLOPs"的散点图，拟合线性趋势。这给出670亿参数的建议。

方法2是IsoFLOP：固定FLOPs预算，扫描参数与数据的权衡，画出每个预算下的损失曲线，取最小点的轨迹。这成为后来最流行的方法——因为它简单、稳健，且对"最优权衡"的读出非常直观。结果是630亿参数。

方法3是假设函数形式 + 曲线拟合：先提出一个联合缩放定律的函数形式（如 Rosenfeld 的简洁形式：两个倒数项之和），然后对训练数据进行曲线拟合，估计系数。这是最"蛮力"的方法，也得到了与Chinchilla相近的建议（约670亿参数）。

三种方法一致指向同一个方向：Kaplan的预测错得离谱。无论是下包络线、IsoFLOP还是函数拟合，都认为Kaplan时代的大型模型被过度放大了。

为什么会有分歧？

这是最精彩的部分。Chinchilla 和 Kaplan 的分歧告诉我们缩放定律世界的混乱现实。教授和多位研究者的分析揭示，一系列看似微小的决策累积成了巨大差异：

第一个差异是参数计数方式。Kaplan排除了嵌入参数，这是常见的学术惯例；但问题在于，他也排除了最后一层（softmax层）的参数。在许多模型中，嵌入层和输出层是互为对偶的（形状相同），因此将它们同时排除是合理的——但事实证明，包含或排除这些参数，对缩放定律的形状有重大实质性影响。

第二个差异是模型收敛程度。Kaplan训练的许多模型太小了，以至于在完成学习率预热时还没有真正收敛——他们的学习率设置欠优。

第三个差异是批次大小。Kaplan固定使用大批次，而这对小模型来说是次优的。

后续研究者通过调整参数计数方式、学习率预热方案、优化器调优，逐步重现了Chinchilla的结果。每一小步看起来都不起眼，但叠加起来，就是Kaplan缩放定律和Chinchilla缩放定律之间的巨大鸿沟。

Yair等人的分析还有另一个重要论断：缩放定律在某种意义上是一种下界——如果你放大一个预热方式很疯狂、批次大小很疯狂的配方，你得到的缩放定律就会很差。因此，你要尽可能接近正确的完整运行方式，再谈外推。

有趣的是，Chinchilla论文本身也有一个未解之谜：方法三（曲线拟合）与另外两种方法在指数上有细微差异，而这会导致渐近结论不同（方法三认为token应该比参数增长更快）。Epoch AI的独立研究通过提取论文数据、重新拟合，证明了原论文的方法三其实欠拟合了原始数据——重新拟合后，方法三与另外两种方法完全一致。也就是说，Chinchilla作者比他们自己意识到的更加正确。

你真的想要Chinchilla系数吗？

教授在这里给出了一个反直觉但极具洞察力的观点：虽然Chinchilla的"20 token per参数t"是最优训练计算方案，但如果你在训练生产级模型，你大概率不想要这个系数。

为什么？因为训练算力只是总成本的一部分。前沿实验室的算力分配中，训练只是冰山一角——大量算力花在了研发和推理服务上。如果你的模型要大规模部署，推理成本和模型大小直接相关。此时，你宁愿训练一个比Chinchilla最优更大（也就是相对更小）的模型——课堂上称为"过度训练"（over-trained）模型：用更多的token训练，让模型更强大，同时保持较小的规模以便高效推理。

看看实际的历史轨迹：GPT-3每参数3 token（严重训练不足）、Chinchilla每参数20 token（训练最优）、当今的生产型模型（过度训练+MoE）。这背后正是推理服务的现实驱动力——如果只是做研究，20比1可能成立；如果要生产，你需要小但强的模型。

缩放定律作为工程范式：总结与反思

通过这门课的完整梳理，可以看到缩放定律的核心思想：资源投入量与性能之间存在对数线性规律，并且适用于各种资源维度——参数量、数据量、算力、MoE稀疏度。这让你可以把本来需要大规模运行才能确定的决策，转化为在小规模下通过拟合和预测就能完成的工程任务。

缩放的流程很朴素：训练一堆小模型，拟合双对数坐标上的直线，验证拟合质量，然后外推到大规模运行。这门课强调，在这个过程中你会遇到几个关键教训：

数据缩放是最自然、最干净的对象——指数虽然令人失望（-0.1级别），但规律性无可置疑。
模型缩放让你能做所有酷炫的工程事——比较架构、优化器，预测超参数。
缩放是工程出来的，不是自动发生的——你需要选择正确的x轴（是否计入嵌入参数）、正确的超参数设置（批次大小、学习率预热），才能获得可预测的缩放。
斜率几乎不变，截距负责所有变化——这是一个反复出现的经验规律，它的普遍性本身就是一个巨大的谜。

最后，教授提醒我们缩放定律不是魔法。它需要谨慎对待、需要验证、需要不断审视假设。它的可预测性和不可靠性并存——但即便如此，它依然是我们在面对"如何在未知的大规模区域做出正确决策"这一难题时，最有力的武器。

本讲在课程中的位置

这门课位于课程体系的中段，紧跟在系统层面的讲解之后，属于"从工程到深度学习的过渡"。它直接回应了讲座二（架构选择）遗留的问题：架构选择和超参数决策应该如何在真实规模下做出？同时，它为后续的进阶缩放课程（涵盖μP、参数初始化、优化器）奠定了方法论基础。课程安排有些特殊：下一讲是推理，随后再回到进阶缩放——但今天的基础内容，是理解后续所有高级话题的前提。当你真正面对大规模、成本高且难以反复试错的训练任务时，缩放定律将是重要的决策工具。

官方资源：

视频：CS336 Lecture 9 官方视频
讲义：课程主页 cs336.stanford.edu（Lecture 9 幻灯片及参考资料）

CS336 2026 Lecture 9：完整中文逐字稿

视频：官方视频
来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。

正文

[00:06.040] 我们开始吧。嗯，暂时离开系统层面的话题，来聊一些更偏向深度学习的东西。我们会有两节课来讲缩放定律（Scaling Laws）及其相关现象。今天讲的是基础内容，我们只讨论相当基础的缩放定律知识。一些经典的工作。缩放定律的基本思想如何与你们可能在机器学习入门课上学到的知识联系起来。然后在两节课之后，我们会讲更进阶版的缩放定律课程，届时我们会深入探讨一系列现代开源模型技术报告，比如 μP（Maximal Update Parametrization）和其他参数初始化方法，今年我还会加入一些关于优化器（Optimizer）的内容。所以会有一个进阶版本，试图真正触及前沿领域。今天，我们先从所有真正基础的内容开始。由于一些日程安排上的限制，下一讲会是推理（Inference），所以你们会在两个主题之间来回切换。

[01:04.129] 你们会先学缩放定律，然后是推理，接着又是缩放定律。所以，缩放定律在某种程度上实际上是以下场景，或者说一旦你遇到这个场景，你会很快碰到它。假设你有一位非常有钱的朋友，给了你一万块B200显卡用一个月，他或她请你构建一个非常好的开源语言模型。他们希望这些B200能产出些成果。你已经组建好了你的基础设施团队。那是你分配给伙伴们的任务。而且，假设你有一个很好的预训练数据集。那会是这之后的任务，但让我们假设你已经有了。然后你准备去训练一个大模型，但训练大模型有很多选择——所有的作业，我们前面早期讲座二的内容，你会遇到诸如“我应该用什么架构？”、“我的超参数应该是什么？”这样的问题。

[01:56.730] 在一个可能耗资数百万美元的运行上去调试这些，是非常吓人的。那么，你究竟如何认真对待这种规模放大的想法呢？你如何确保你的大规模运行真正成功？在这么多选择中，有些你可能只是从文献中照搬。我之前在架构讲座中讲到的，是关于那些被广泛采用的最佳实践，你可能只是随便挑一些用。那没问题，但真正关键的是，假设你处于前沿领域。你想构建一些真正比当今最好的模型还要好的东西。你必须要开始真正优化这些选择。你不能仅仅复制别人的选择就得到超越现有技术水平的东西。那我们该怎么做？这要求我们真正深入接触缩放这个概念。缩放定律一直是一个非常强大的工具。

[02:52.260] 你也可以称之为一种范式（Paradigm）。如果你和一些在大型实验室真正从事缩放工作的人交流，这几乎是一种生活方式。他们会说，我们真的相信缩放定律。这几乎成了一种信念，而且你会明白为什么缩放定律有时可能是相当难以捉摸的东西。但说到底，缩放定律是这些简单、可预测的规则，告诉我们如何从模型在小规模下的性能和表现出发，并尝试将它们外推到大规模行为。所以，处理模型训练最基本、最天真的方法可能是说，我预算充足，我将在这些非常大型的运行上进行多次训练，然后在这些大型运行上调优超参数————但那非常浪费。所以相反，我想做的是，把所有的优化都在小规模下完成，并有一些简单的规则让我能够从小规模行为外推到大规模行为。如果小规模和大规模之间通过某种非常简单、稳健的联系（比如某种规律性）连接起来，那么你

[03:52.100] 就会对这种新的缩放方法来优化你的系统有信心。这就是今天的主要内容——从工程视角看待缩放定律。首先，我会讲一点背景知识，因为缩放定律非常有趣，它们不仅是最新的范式，许多模型都是这样设计和思考的，而且它们与非常经典的思想紧密相连。所以，如果你是机器学习理论研究者，实际上，有些方面你会感到非常亲切，比如说经验样本复杂度（Empirical Sample Complexity）。然后我会讲解各种缩放定律的历史，向你们展示这并不是什么真正新鲜的事物。这背后实际上有相当多的历史和研究积累。所以，我真的、真的不得不在我的幻灯片里加入一些数学内容。

[04:47.270] 我觉得确实有必要这样做，但非常有趣的一点是，在很长一段时间里，机器学习一直在思考“我的模型表现会如何”这个问题，特别是从理论角度来看：我有这个模型类别，我的模型能有多好？泛化界（Generalization Bound）就是理论家对这个问题的回答。对于某个有限假设类（Finite Hypothesis Class），你有你的误差，你会说，哦，我的误差最多比模型在训练集上的表现差这么多。这是经典的泛化界。你可以为更复杂的类别写下来，比如光滑密度（Smooth Densities）之类，但重要的是，这些界不仅告诉你一些性能信息，注意这些界通常依赖于样本量。所以这告诉你的是，这是我的损失值随着训练集增长而变化的一个理论上限。

[05:39.220] 在某种程度上，这与“当我扩大数据量时会发生什么”这个想法非常密切相关。这些是紧密相关的问题，你们或许会相信我说的是真的。如果你试图找出有史以来第一篇缩放定律论文是什么？事实证明，你可以追溯到相当久远。你可以追溯到贝尔实验室（Bell Labs）的 Corina Cortez 和 Vladimir Vapnik 等人，他们在机器学习理论方面做着非常严肃的工作，并提出了这个问题：在非常大的数据集上训练分类器通常非常昂贵。我们希望避免这样做，为此，也许我们可以在较小的样本上拟合分类器，拟合一条关于它们错误率如何衰减的曲线，并用它来估计它们的性能。这几乎就是字面意义上的数据缩放定律，而且是在1993年。

[06:29.990] 所以这个想法，至少是思考模型性能如何随数据集大小变化，是一个非常经典的想法，并且有很多工作支持这种观点。我只是想把它和这些较老的想法联系起来，这样你们就不会把缩放定律看作是这种全新的、纯粹神经网络的玩意儿。它是一个在经验机器学习中有很多根源的想法，并且与理论机器学习也有联系。如果你开始看这方面的历史，有很多人都说过，难道不是这样吗——也许与其在算法开发上花钱，不如我们去收集更多数据，因为如果我们绘制系统性能随数据量变化的图像，事实证明，它们都以可预测的方式在提升或改进？Banko 和 Brill 的工作是关于数据量和缩放思考确实是提升系统性能的有效方法的经典NLP参考文献之一。

[07:30.790] 所以关于这个问题有很多思考方式。其他NLP研究者也有相关工作，比如Kolachina等人在2012年的研究。这是关于scaling laws不同函数形式的最早工作之一，他们同样是增加训练所用的数据量。在这个案例中，是机器翻译系统或BLEU分数。他们问的问题是：模型性能随数据量增加而变化时，正确的函数形式是什么？结果可能令人惊讶也可能不令人惊讶，他们最终得到了与今天相同的幂律关系，即幂次为3和4的形式。在接下来的讲座中，你会不断看到这些多项式函数，幂次为3和4，在我讨论scaling laws时。我们在2010年代就已经知道了很多这方面的内容，即这些函数形式在模型规模扩大时对模型行为具有很强的预测能力。

[08:27.940] 最后，我非常喜欢提到这篇论文，因为出于某种原因，Hestness等人的工作没有被引用到应有的程度。神经scaling的起源，我会追溯到2017年的Hestness。那在很多方面都是一项非常有前瞻性的工作，他们建立了作为数据集规模函数的scaling laws并覆盖了广泛的系统，包括不同类型的语音识别系统、机器翻译系统、语言模型，他们证明了这些东西在大量领域中都遵循这些优美的多项式趋势。这远远领先于它的时代，因为2017年比2020年代早期的OpenAI scaling laws等工作要早得多。他们还讨论了我们今天讨论的内容，比如涌现，即模型可能突然展现出某些能力，因为准确率比损失函数是一个更加不连续的度量。以及按计算量进行scaling，如果我们有能够可预测地扩展到大规模训练数据的系统，那么计算量将变得非常重要。

[09:32.950] 最后，既然我们按计算量进行scaling，那么速度和系统优化就会转化为准确率。所以在某些方面，我们今天看到的很多现象，比如系统scaling等现象，我本可以在2017年就知道，如果我当时阅读并深入思考了这些scaling laws论文的话。回去更认真地思考它们。所以在我每次讲scaling laws的讲座中都会提到那篇论文，就是为了说明，很多这些现象在现代时代之前就早已为人所知，或者说在现代语言建模时代，在见到这些大型语言模型之前，我们是有可能知道我们所处的这个范式的。这很酷。关于这段半历史性的内容有什么问题吗？是的。[听不清]，我想知道这些是纯粹凭经验得出的，还是有某种方式可以验证这一切？你是在问这些scaling laws，以及它们从何而来吗？

[10:33.680] 这些纯粹是曲线拟合的练习。我会稍后讨论scaling laws可能来自哪里，以及为什么多项式可以说是非常自然的scaling laws类别，但并没有金科玉律说只有这些。不过我要说的是，在某些方面，理论提供了一个非常丰富的来源，scaling laws的函数形式可以从理论中产生，因为理论经常大量思考错误率如何作为各种对象的函数而衰减？它们会给你各种候选的函数形式。物理学家也非常擅长这个，因为他们思考极限，比如极限的行为是怎样的？这就给了你不同的scaling laws函数形式。很好。好的，还有其他问题吗？好。那么现在我要开始讨论更现代的神经语言模型scaling，我将从数据scaling laws开始，因为对我来说，它们是最简单和最自然的对象。

[11:34.320] 讨论数据scaling laws也能让我非常循序渐进地带领你们了解，我们如何猜测scaling laws的函数形式可能是什么，以及为什么多项式形式是自然的选择。然后我会开始讨论超参数和架构，以及其他各种更特殊的对象，这些都更难从直觉上证明合理性，但仍然显示出非常好的scaling law行为。关于这种scaling law风格分析，一个显著的特点是，这些幂律关系似乎在各种不同的因素中都成立。所以我们看到的基本的那些在预训练中，也是我今天要讨论的那些。通常你会看到的是横轴为计算量。这是对数计算量。抱歉，横轴上是对数计算量，纵轴是对数测试损失，或者横轴是对数数据集，纵轴是对数损失，横轴是对数参数量，纵轴再次是对数损失。

[12:33.610] 所以我们可以得到各种不同的横轴，你可以把它们认为是在对数轴上的资源。所有这些都在这里使对数测试损失线性化。你也可以看到更奇特的scaling laws。你可以看到基本上等同于这些的下游基准测试上的scaling laws。在这些情况下，你会预期看到sigmoid函数形式，作为计算量或类似计算量的量的函数。最近甚至有很多工作在预测方面，人们做的事情比如把日期放在横轴上。而且很奇怪的是，至少在這些图中——这些曲线的上包络线通常在各种能力的不同形式下是线性的。没有理由认为在底部这些更奇特的scaling laws必须成立。但事实证明，我想语言模型性能作为规模和资源的函数，通常比最初看起来要规则得多。

[13:33.860] 那么首先，我将从数据scaling laws开始。数据scaling laws是一种非常简单的单变量关系，我们将要思考它。在这个世界中，我们将固定我们的模型训练流程。一般来说，除非我另外说明，模型通常会比数据集大小更大。我将以规律的方式增加我的数据集大小，然后观察错误如何减少。我们对这种关系有什么预期？一般来说，如果我们很好地调优超参数，它将是近似单调的。因为我们有更多数据，更多数据应该帮助我们在任务上做得更好。错误应该下降。如果你在做分类之类的任务，甚至对于下一个词元预测，你应该真正从随机猜测过渡到某种熵类的对象，在那里你无法做得比任务的不可约误差更好。

[14:22.350] 那就是噪声下限。所以我们这里会有一个类似S形曲线的对象。这通常是我们对系统的预期。所以我们从实证观察开始，你应该自己做这个实验。你会在作业3的部分内容中看到大致类似这样的结果，即如果我们有一个非常大的模型，远大于我们的数据集，然后我们增加数据集大小，把它放在对数轴上，纵轴是对数测试损失。再次强调，你会发现你绘制的点会形成非常清晰的线性趋势。这是我从Kaplan 2020年那篇论文中取来的。我会反复引用这篇论文。这是OpenAI的"Neural Scaling Laws"论文。他们有很多很好的消融实验和大量优美的数据。现在，这种关系，我在双对数图上画了一条线，这是一种无尺度或幂律关系。这意味着什么？如果我在双对数图上有一条线？

[15:26.258] 你们中的许多人——对你们中的许多人来说，这可能是显而易见的。但只是为了真正强调这一点，如果某个东西在双对数图中是线性的，那意味着什么？嗯，这意味着我所拥有的误差是多项式衰减的。而且，这通常也意味着我离我的渐近线还很远。因为一旦我接近渐近线，我就会开始逐渐偏离直线，而不是保持一条直线。所以让我们看一个例子，向大家展示缩放定律是如何在一个非常简单的估计问题上非常自然地产生的。暂时忘掉语言建模。这本来是一门语言建模课，但在接下来的一页幻灯片，或者实际上是两页幻灯片里，它现在是一门统计学课。不，也许三页。那么在种情况下，我想做的是从一堆数据中估计均值。我有一个输入，它来自一个高斯分布。我要估计这个对象的均值。我直接写下经验均值。然后，你可以写下误差是什么。

[16:18.220] 那么期望误差是多少？嗯，这些值每一个都是高斯的，而且 μ 帽是高斯的，所以我可以写出误差。那就是 sigma 平方除以 n。那么，如果你看一下这个，这就是一个缩放定律。我写下了误差，这个误差，如果我对它取对数，是 log n 的函数，log n 是对数 x 轴。所以一般来说，如果你能提出任何本质上形如 1 除以 n 的 alpha 次方加上某个常数的东西，当你把它画在双对数图上，减去常数项后，你会得到一条缩放线。所以任何这种形式的东西，只要你有多项式衰减和误差，最终都会给你一个缩放定律。当然，在参数估计中，比如均值估计或回归，你当然会期望一个多项式速率。你不会真的期望别的。如果你上过统计学估计课程，你会知道基本上所有经典模型你都会期望有 1 除以 n 的缩放。

[17:17.349] 如果你拟合一个回归或类似的，你可能会得到 D 除以 n 或者其他什么。所以如果这是真的，这就是缩放定律的起源，如果我们画出我们的线，我们应该期望看到大致一个负 1 的斜率，因为那是 x 上的指数。Y 等于负 x 加 C。如果我们拟合我们的神经缩放定律，我们会发现我们的指数是非常不同的。这些指数——左边的两个我取自 Hestenes，右边的那个我取自 Kaplan——大约是 -0.1，-0.3，-0.1，大致来说。那么这意味着什么呢？这意味着这比估计均值或估计线性回归要慢得多。所以这是非常非常慢的收敛速率，仍然是多项式的，但要慢得多。现在这很有趣。这表示我们得到了这些漂亮的多项式速率，但指数不是那么好。

[18:11.320] 它们是 -0.1。你可能会从哪里得到这样的估计速率？嗯，如果你的模型更加非参数化，你就会得到这个。我保证这将是本讲最后一页统计学幻灯片。假设我现在要估计的不是均值，而是一个任意的光滑函数。那么我可以做的一件事是，如果我在做一个回归问题，我有我的输入，x1 到 xn。假设它们在单位盒子里。我有一点高斯噪声，我有一个未知函数 f，我想估计它。对此一个平凡的估计器是，我可以取我的空间，把它切成长度为 n 的负四分之一次方的小二维盒子。那么我们的估计误差是多少？我会有 n 的平方根个盒子。每个盒子会得到 n 的平方根个样本。误差将大致是 1 除以 n 的平方根之类的。一般来说，如果我有一个 D 维函数，我想用这种非常非参数化的、任意的方式去估计，我的误差将大致等于 n 的负 1 除以 d 次方。

[19:11.040] 所以我的缩放将是 y 等于负 1 除以 D 乘以 x。所以现在这些更灵活的函数，那些比线性类别更灵活的函数，现在它们的缩放定律中的速率不仅仅是 1，而是 1 除以 D。所以你可能会有的一个心智模型是，我在这些幻灯片中展示给你的神经网络表现得像在 10 维空间中的非参数回归器，比如在 10 维空间中的最近邻或其他类型的估计器。这大致就是它们从数据中学习的速度。仅仅通过观察一些这样的缩放定律就能知道这一点，这挺酷的。有些人更强烈地论证了这一点。Bari 等人和其他一些缩放定律理论研究者认为，缩放定律指数实际上直接告诉我们，神经网络表现得像一个非参数平滑器，等等诸如此类。

[20:02.568] 这是一组有趣的论证，你可以按你喜欢的程度去接受这种直觉。我不太确定我自己到底有多相信这个论证。有一些证据可能有点站不住脚。它依赖于内在维度的估计，但想到这些指数在某种程度上告诉我们这些神经网络真正学习的速率有多快，这是一件有趣的事情。好的，那么实际上，我在这里暂停一下。这真的是非常基础的数据缩放定律的结尾，即一个分布，你从中采样，然后从中学习。这些例子等等都清楚了吗？很好。哦，是的，有问题。当你说模型总是比数据集大小大时，你的意思是指参数数量与 token 数量是一一对应的关系吗（听不清）？是的，这是个好问题。所以问题是这样的：当我说模型比数据集大小大时，我是什么意思？

[20:52.580] 我的意思是，当模型相对于数据变得很小时，比如你拥有的数据远远超过模型大小，你就进入了这个不可约误差区域。你基本上已经为你的模型类别尽可能好地拟合了数据，无论你再添加多少数据，你都无法做得更好。所以我们想要——当我们考虑缩放定律时，我们通常想要考虑这个幂律区域，这是更简单的情况。要么那样，要么你明确地拟合渐近线，然后对数据进行校正。就是这两种做法。所以当我说模型比数据大时，基本上可以是任何比数据大的量，只要它处于这个幂律区域，通常比如比数据大 10 倍之类的。很好。好的，现在我们回到其他数据缩放定律。数据缩放定律本身并不像你希望的那样有用。它们只告诉你我的模型学习得有多快。

[21:48.450] 这对于预测有点用。它对于其他很多事情没什么用。如果你想做工程，你可能对其他问题感兴趣，比如我的最佳数据混合是什么？对我来说，为我的模型挑选训练数据的最佳方式是什么？这是非常有用的工程问题。我应该重复使用我的数据，还是干脆不重复使用，把计算节省下来用于其他事情？或者也许我应该重复一些高质量的数据，而不重复低质量的数据。我可以用重复使用做各种其他事情。对于数据，我可以做出很多不同的工程决策。那么缩放定律如何帮助我们做出其中一些决策呢？我认为有趣的事情之一是，如果我们稍微思考一下经典模型的行为方式。数据缩放定律，如果我们把它们的起源基本上看作是泛化的经验版本，这些上界告诉我们，对于许多模型来说，数据集组成实际上主要影响这些缩放定律的截距，而它们的斜率实际上是由模型类别决定的。它们并不是真正由数据分布本身决定的。所以斜率可能保持不变，只有截距可能改变。

[22:57.333] 而截距实际上可能呈现出非常有趣的形状。这是一个玩具模型的例子，一个线性回归，有两种不同的采样来源。如果你只获得一种类型的数据，无论怎样，误差都会很高。你实际上需要两者的混合。所以如果你写下这个截距的行为方式，你实际上会开始获得一些非常有趣的见解，了解为什么拥有更多的数据多样性非常有帮助。但这在实际中意味着什么呢？实际上，这意味着你可以在混合数据上拟合数据缩放定律，这可以帮助你进行数据——抱歉，预训练的数据优化。这意味着什么呢？假设你有两个数据源，为了简单起见。你有新闻数据，还有维基百科数据。应该用多少新闻，用多少维基百科呢？你如何确定这个比例？你可以用非常少量的数据训练非常小的模型。

[23:52.795] 然后你基本上可以训练或拟合一个函数，描述不同混合比例如何影响性能。你可以在逐步扩大模型规模的同时做这件事。你可以观察趋势如何变化。然后你可以尝试以外推数据缩放定律的同样方式将其外推出去。然后你可以问，如果我继续将其扩展到完整的训练运行，哪一个将是最优的？所以这是一个非常简单的思路，本质上就是在少量计算下拟合一个函数形式，找到最小值，然后将那个结果扩展出去。这就是数据混合损失论文。不幸的是，我觉得如果你和任何做过大量这类数据混合工作的人交谈，他们会告诉你现实比这个理想世界所暗示的要嘈杂得多。而且据我所知，在很多情况下实际发生的是，你最终训练了一堆小模型。

[24:45.430] 你从小模型中挑选出最佳的数据混合方案，然后直接将其扩展，不需要什么缩放定律。如果你对这类事情感兴趣，Data Decide 是一个关于这类数据混合问题的很好的大规模实证研究。他们发现的是，实际上，如果你不拟合缩放定律，而是仅仅在小规模下选择最佳的数据混合方案，那效果也很好。顺便说一下，这与截距不同但斜率不变的论点是一致的。因为如果斜率不变，小规模下的最佳混合方案也是大规模下的最佳混合方案。好的，另一组非常有趣的问题，我也会在数据的话题下提到，就是重复。我认为越来越普遍的情况是计算量在增长，而我们拥有的数据量并没有增长。所以有很多人对这样一个问题感兴趣：当你越来越多地重复数据时会发生什么？几年前有一项很不错的研究叫做“Scaling Data-Constrained Language Models”，他们大致展示了

[25:48.000] 类似在使用标准训练方法的情况下，最多到四个 epoch，你不会受到任何损害。但如果超过了那个点，那么你实际实现的缩放定律，这条深色曲线，会比你拥有新鲜数据时预测的缩放定律差得多。那是指这里的虚线。而且你也可以同样地写出一个修正的函数形式，通常能够预测这种缩放定律在重复条件下的行为。我们可以把这个想法更进一步推向极端。这是最近的工作，和 Percy 以及我们共同指导的一位学生做的，我们问的是，好吧，如果你把这个想法推向极端并且你只考虑无限的计算量会怎样？所以你被允许任意多次 epoch，比如无限次。你能从这个系统中获得的最好的东西是什么？事实证明，你不能只是不断地重复遍历数据。

[26:37.422] 你不能不断让你的模型变得更大。这些都有递减的回报。所以你最终会转向其他方法，比如集成你的模型，试图从你的数据中挤出越来越多的价值。但我觉得真正有趣的一点是你们在这里看到的红色标准数据缩放定律。随着数据量的增加，性能有着非常漂亮、可预测的提升。我们进行了各种干预，比如正则化和添加集成。你获得了性能提升，但斜率看起来惊人地相似。这是你开始拟合自己的缩放定律后会学到的经验，那就是非常非常频繁地，你的斜率不会改变。通常，你做的干预只是改变了缩放定律的截距。最后我要提一件事，因为我很喜欢这篇论文，在缩放定律和整体扩展的背景下，那就是我们在数据方面关心的很多现象实际上是非常依赖规模的。

[27:31.950] 其中一个例子是数据过滤。如果你或我明天决定为我们正在训练的模型过滤一些数据，我们可能会非常激进地过滤。我们只会保留最高质量的内容，因为你和我都没有太多的计算资源。所以反正我们也不可能在整个互联网上训练。我们不如过滤到最高质量的内容。另一方面，如果你有大量的计算资源，那么你会想要在越来越多的数据上训练，因为你不会想要在这些高质量数据上做太多重复。所以随着你拥有越来越多的计算资源，你的过滤器变得越来越宽松，可能你会开始在越来越低质量的数据上训练。所以我们经常静态思考的很多事情，比如数据过滤或数据质量，实际上要动态得多。

[28:14.870] 随着规模扩大，你必须思考我要从哪里获取其余的数据。过滤器不能作为数据的函数保持不变，而且最优过滤器确实被证明不是规模的固定函数。好了，数据缩放定律部分就到这里。这是我认为最直截了当、最容易理解的部分。即使作为一个有统计/机器学习背景的人，这些东西对我来说也都是非常简单直接的。你增加数据，性能就以非常可预测的方式变好。我认为这是没有争议的。对于这么大的模型，它能以多项式方式如此表现仍然很酷，但也没有真正令人惊讶的地方。所以希望你们都能认同，当你增加数据量时存在如此多的可预测性。哦，是的。所以在这张关于无限计算下预训练的幻灯片上，右边那个图在两个轴上都是线性的，损失和数据集大小，

[29:16.450] 而我之前给的例子是对数-对数坐标。我想知道为什么这里两个轴都是线性的。是的，这是一个糟糕的坐标轴。Herman 和我刚才还谈到像这样的糟糕坐标轴。这实际上是翻倍。所以看起来是线性的，但实际上是对数刻度。[笑声] 我现在已经修好了。你现在修好了。而这里的 y 轴，看起来不像是对数的，但这个范围太小了，线性对数和实际对数基本上是一样的。如果你看这里拟合的函数形式，这些仍然是多项式。所以我们拟合的是相同的函数形式，这里画出来的就是那些。这是另一个要点，我会在课程结尾再讲到它。如果你只拥有计算量范围中的极小一段，你很难判断某个东西是按多项式规模扩展还是按指数规模扩展。因为泰勒近似是真实存在的。

[30:02.510] 如果你放大得足够细，一切看起来都是线性的。所以无论这是多项式还是其他函数形式，你都要时刻保持一点谨慎和怀疑。好的，好。行，那这部分就是关于数据的规模定律，在很多方面都直截了当。现在我想谈谈更特殊形式的规模定律，以及针对各类模型工程的规模定律。我们现在要做的是尝试设计大语言模型。也许你是个激进派，你相信LSTM才是未来。比如，为什么我不能用LSTM而不是Transformer？我想把B200的算力都砸在我的LSTM上。或者你是个不同类型的激进派，你想用SGD来训练。比如，为什么我不能用SGD训练？你可能也会想，如果你资源有限，你有不同的花钱方式。也许你应该让模型训练更久，也许你应该训练更大的模型，也许我应该去收集更多数据。总是有各种权衡在起作用，而规模定律希望能给我们提供量化这些权衡的方法。

[31:10.040] 我先从一些简单的超参数问题开始，比如如何选择我的架构或优化器，这些我在第二讲中已经谈过。在那时我给出的视角是你也许可以研究别人在做什么。现在我要尝试用一套略有不同的权衡来给你这些问题的第一性原理答案。所以你们在这节课中必须相信规模定律，但只要你们相信了，就能自己尝试各种方案，自己去发现答案。好，我们可能从的第一个问题是：Transformer真的比LSTM更好吗？LSTM。它们也可以工作得不错。它们能拟合分布。随着Mamba这类东西的出现，我们知道状态空间模型（SSM）也能工作。所以它们更好吗？回答这个问题的暴力方法是训练一个大规模的LSTM，达到GPT-3或者更大级别的模型。

[32:02.400] 而规模定律的方法则是训练一堆较小的模型，然后说：我要在不同计算量范围内训练我的Transformer，我还要用不同层数的LSTM，同样在多种不同的计算量范围内进行训练。那么这说明了什么呢？这说明LSTM的截距确实不同，甚至斜率也可能与Transformer不同。正因为如此，你可能不倾向于选择LSTM。这张图可以证明应该扩展Transformer而不是LSTM。如果你看看如今很多架构论文，比如你去翻Mamba论文或者门控DeltaNet的论文，它们总会有一张像这样的图，其中包含普通Transformer，以及我们那个很酷的新模型。而我们那个很酷的新模型通常要么在上面，要么在下面，因为那才能证明你的模型表现更好。

[32:51.510] 你当然不想要斜率更差的情况，因为那意味着随着你越扩越大，你的模型最终会表现得更差。所以这是在不用训练巨型模型的情况下真正推理架构优劣的一种方式。我认为，近年来出色的架构规模定律论文并不算很多。我觉得相对较新且我非常喜欢的一篇是Ek等人在Google时做的，那还是他们公开发布这类工作的时候。他们对T5风格模型做了大量的规模研究，在许多不同架构上训练了越来越大T5模型变体。我认为一个非常酷的地方，也是我至今仍然会提起这篇论文的原因之一，是他们恰好捕捉到了我们今天在前沿模型中实施的架构改动。比如Performer，这是一种高效注意力机制，它扩展得不太好。

[33:52.680] 我们没有采用那些方案。我们采用的是门控线性单元（GLU）。在整个规模趋势中，红线一直优于绿线。Switch Transformer——我不知道他们最大规模那次运行发生了什么，但总的来说它有很好的扩展趋势。我忘了，哦，还有Softmax混合（MoS）。实际上我们现在不再用这个了，但根据规模趋势来看，这大概是一个有效的干预措施。所以你可以看到，通过这些规模趋势，尽管当时这些论文中的很多工作所运行的计算规模比我们今天要小得多，他们仍然能够看到当时驱动前沿模型发展的那些架构趋势。我认为这就是为什么很多人把规模定律当作一种近乎范式性的方式，说“哦，如果它在规模定律中没有体现出来，那就不是一个好的干预措施。”因为在某种程度上，过去很多工作确实捕捉到了我们今天所看到的。

[34:47.370] 好。好了，在架构之外，我们可以把这类分析扩展到我们可能想要理解的各种其他事物上。我们可能会问：SGD比ADAM更好还是更差？如果你做规模定律分析——这篇是Hestness的工作——你会看到非常清晰的趋势。这又是那个非常神秘的现象：截距不同，但斜率非常相似。如果你固定了数据，固定了模型，斜率通常非常非常相似。我每次看到这一点都感到非常惊讶。然而，这确实是真的。即使像从SGD换成ADAM这样大的干预，也很少能看到斜率发生变化。如果你是那个用SGD或ADAM训练的人，你会觉得这是巨大的变化。但规模趋势大致保持不变。在这个案例中，这是使用循环高速公路网络（recurrent highway net）做语言建模。

[35:38.170] 类似地，在第二讲中，我讲到了宽高比（aspect ratio）。你如何选择深度和宽度之间的权衡？我当时告诉你，你可以随便选一个大致合理的数字，比如大概是合理倍数的四倍。但如果你做这种规模分析，你可以更精确地知道发生了什么。所以你可以做的第一件事是，观察规模趋势如何随层数变化，你会立刻发现层数极少对你的模型非常非常不利。如果你只有一层，那你哪儿也去不了。那是一个非常非常糟糕的规模趋势。大于1层之后，实际上竞争性出乎意料地强，不过你确实会看到在每个计算量水平上，更多的层数——至少在这种情况下——是更好的。如果你能做更细粒度的研究，就像Kaplan论文中做的那样，你试图找出那些可以称为尺度不变量的量，比如宽高比。

[36:32.000] 层数不是一个尺度不变量。随着你的模型变得更大，你确实需要更多的层。但随着你的模型变大，也许最佳宽高比应该保持不变。这个量是尺度不变的。所以如果我们看宽高比，我们会发现在不同的模型规模下，每层模型在大约100d附近都获得基本相同的最小值——或者可能稍微少一点。而且它确实会稍微向右偏移。对于更深的模型，最优宽高比会向较小的方向偏移，但大体上，最小值保持在差不多的位置。你可以对类似注意力头维度等事物做类似的分析。如果你去看Kaplan的论文，他们在这方面做得相当出色。这是你可以对自己的扩展策略建立信心的方式之一。所以如果你的扩展策略是“我要固定宽高比，然后扩大规模”，你可以画出这样的图，并说服自己我们大概没问题，因为随着我走向越来越大、越来越大、越来越大的模型，我们的优化偏移不会太大。

[37:29.790] 有一件很重要的事，而这是一个巨大的麻烦堆，大到我会用不止一张幻灯片来讲它，我稍后会回来再谈——并非所有参数都是平等的。正因为并非所有参数都是平等的，你的扩展定律看起来是好是坏，取决于你如何定义什么是参数。所以在Kaplan论文中，他们观察到，如果他们把嵌入参数画进深度的扩展定律里，得到的是这些看起来非常怪异的扩展曲线，然后他们说：“哦，这完全不行，完全不行。”正因为完全不行，他们决定把所有嵌入参数排除掉，只计算非嵌入参数，因为你可以这样说服自己：这些才是真正在做计算的参数，或者其他什么理由。我们后面会看到，这将对某些结果的有效性产生不可忽视的影响。

[38:14.300] 但这里的重点是，扩展定律并不是魔法。我觉得Percy喜欢强调这一点：扩展定律以及所有这类跨尺度可预测性，是工程出来的。它们不会自动发生。我们需要选择正确的x轴来观察。我们需要确保这些事情的超参数设置正确，只有在这些条件下，才可能在跨越多个数量级的计算规模上获得可预测的扩展。所以这就是很多扩展定律研究者做的事情。他们思考的是，什么是获得非常可预测扩展的正确方式。与这相关，我认为有一些近期很有趣的论文，比如关于混合专家模型的扩展。既然混合专家模型可以说是训练大模型的主流方式，那么在这个参数数量和参数量某种程度上解耦的新世界里，思考“什么是参数的价值”或“什么是参数”就非常重要。

[38:57.520] 苹果和MIT的一些研究者做了一个漂亮而干净的分析，展示了大模型扩展趋势与总参数和激活参数之间的关系。你看到的一个很酷的现象是，当你开始训练越来越大的模型时，如果你在这里的x轴上增加总参数量，而你想最小化损失，你最终会得到越来越稀疏、越来越稀疏的模型。你看颜色变得越来越深、越来越深。类似地，在这边，激活参数的数量会随着MoE稀疏度的变化而减少。然后你会得到一个我认为在这里比较直观的结果，那就是你可以画出这些曲面。对于每个计算量，你可以问：好吧，我宁愿把这些计算花在激活参数上，还是说我想要多少稀疏度？我可以在不真正改变计算量的情况下改变我的稀疏度。

[39:45.460] 既然这样，你可以问：好吧，如果我加更多空转参数会怎样——同样的激活参数，但总参数更多？当你从图的后方移到图的前方，你会看到损失在改善。换句话说，这说明了在MoE中那些没有被激活的参数仍然在帮助你降低损失，这挺酷的。你可以写出这些关系的函数形式。我不会在细节上深入这部分，但这里的重点是，我们在优化MoE时关心的所有这些量，都具有可预测的扩展性和规律性。而我现在想谈论的两件事实际上是真正的——是的，我现在想谈的两件事是——当你准备训练一个新的巨大模型时，其实我前面提到的大部分东西都是固定的。你们可能都没那么激进，不会真的换成LSTM。

[40:46.510] 你们大概会选一个，我不知道，DeepSeek-V4风格的MoE或者类似的东西。而且你们大概会选择那些经过检验的、稳妥的架构选择。你不会真的把所有东西都推翻重来。但有两件事，你在某种意义上可能确实需要重新设计，那就是批次大小和学习率。如果你训练过深度学习模型，你就知道你必须非常小心批次大小和学习率的组合。你改了一个，就得改另一个。而批次大小，从上一讲来看，因为另一个原因也非常棘手。我们希望批次大小尽可能大，因为大批次给了我们并行化的机会。记住，数据并行需要大的批次大小。那么，说了这么多，我们真正想知道的是：在我们开始遭受损失之前，批次大小能做多大？

[41:36.600] 这才是相关的系统问题。你还想知道这如何随模型规模的变化而变化。在这个前提下，有一个非常重要的概念，被多篇论文使用并且经常被讨论，叫做临界批次大小。临界批次大小大致是这样一种想法：在达到某一点之前，增加批次大小会有完美的回报，直到你开始遇到收益递减。所以我可以把这叫做噪声受限区间。在噪声受限区间中，你往批次里加的每一个额外样本都减少了SGD步骤中的梯度噪声。既然你是方差受限的，方差的减少就非常非常有帮助。这会给你巨大的回报。所以这就是你有完美扩展性的区间。给定你处理的额外样本数量，你做得已经尽可能好了。

[42:24.990] 当你达到某个点，已经到达了梯度的噪声尺度时，你就不再是方差受限了。你变成了偏差受限。这意味着什么？记住，在梯度下降中，你看的是目标的局部结构。你并没有一个全局视角看到最小值在哪里。所以不管我的噪声多低，我只能指向这里的浅蓝色方向。我的局部下降方向和全局最优位置之间总是存在分歧。正因为如此，在某个点上，我会开始得到递减的回报，因为现在我的限制因素不再是方差了。而是这个偏差项。所以过了这一点之后，我就变成了偏差受限。因此临界批次差不多是一个经验法则，你也可以这样叫它，或者一个方便的权衡点，你说这是我们从完美扩展区间开始过渡到低效扩展区间的点。

[43:15.480] 所以它给了你一个很好的经验法则，告诉你如果想在不承受巨大效率损失的情况下尽可能把批次大小设大，应该设在什么位置。希望这个想法非常清楚。现在，临界批次大小是一个有点奇怪的东西。它来源于对各种关于目标局部二次近似的复杂计算。但我大概会给你讲讲实际用来估计这个量的一些机械步骤。所以你要做的第一件事是，选一个目标损失值。然后，你要做的就是训练几个小模型，在你选择的损失值附近测量临界批次大小。你测量的是，随着批次大小增加，你的损失改善速度有多快。然后你把这个过程重复到多个不同的模型规模上。你最终会得到这样的东西。在x轴上你有模型规模，在y轴上有临界批次大小，你会看到临界批次大小随模型规模呈幂律缩放，而指数大约是一个具体的数值。

[44:36.180] 目标损失可能是你想要达到的某些数值。所以你的目标是尽快达到这个目标损失。这就是我现在的目标。接下来我要做的是，遍历所有不同的批次大小或若干种不同的批次大小，记下达到目标损失所需的步数，以及所需的样本数达到该目标损失所需的样本数。当然，第一点和第二点之间通过批次大小这个因子相关联。步数乘以我的批次大小等于样本数。现在，通过一些计算，OpenAI 的研究人员，McCandless 等人，论证了这是我们应当遵循的大致函数形式，即你所需的步数和你所需的样本数是反比关系，用这个估计值——即可能的最小步数和最小样本数——来归一化这两个量。所以这两者之间存在一个权衡。如果你想最小化样本数，你就需要很多步数。

[45:37.550] 如果你想最小化步数，你就需要很多样本。这是一个非常自然的权衡。如果你增大或减小批次大小，你就会得到偏向某一侧的权衡。现在，他们论证说，现在的目标应该是平衡这两侧。我要确保我的步数项和样本数项大致彼此平衡。如果你解这个方程，你会得到这个特定的解，即临界批次大小等于我所需的最小样本数除以我所需的最小步数。而这两个量是通过将这拟合到我已有的观测数据中来估计的。这将平衡方程的两侧。它会让我的步数比最优情况稍微多一点，也会让我需要的样本比最优情况稍微多一点，但它平衡了两侧。你可以把这理解为在优化收敛速度的同时不让所需步数爆炸。还有一点补充，我不想过多展开，因为这会更深入地涉及细节。如果你对此感兴趣，你真的应该去读一下那篇关于临界批次大小的论文，他们的一个论点是估计这个值非常复杂。

[46:48.690] 你可以通过观察梯度协方差与梯度平方范数之比来做到这一点。那这为什么会出现在缩放定律的讲座里呢？到目前为止，这似乎和缩放没什么关系。它出现在缩放讲座里的原因是，首先，临界批次大小是一个合理的批次大小选择。那是一个在收敛速度和批次大小之间取得平衡的批次大小。所以如果我想把我的批次大小设为等于临界批次大小，你可能会问，这个东西如何作为我的目标损失的函数来缩放？目标损失可以被看作计算量。如果我想要一个越来越好、越来越好、越来越好的模型，我应该使用多大的批次大小？这之所以没有完全脱离缩放定律讲座的范畴，是因为随着你在损失尺度上往下走——也就是说你的损失在改善——临界批次大小的数值会增大。

[47:44.430] 而且它以这种非常可预测的方式增大，又一次，这是一个幂律关系。这就是他们论证的关系，大致是批次大小的一个逆多项式函数。基于这一点，这其实相当不错，因为这意味着如果你有一个超大规模的訓練运行，一直跑到这个图的右端，你的批次大小可以相当大。这是有道理的。当你越来越接近最小值时，方差缩减的价值就增加了。因为你在尝试优化越来越精细的对象，当你在最小化非常非常微小的差异时，噪声的影响就重要得多。这就是关于批次大小的部分。另一方面——记住，你总是需要调整的两个量是批次大小和学习率。学习率这边的故事也有点复杂。我会在进阶缩放讲座中更详细地讨论这个问题。

[48:42.390] 但学习率通常会移动。我认为这是一个很好的思维图景来思考在你开始时所处的标准世界中的学习率，也就是说，假设你有一个多层感知机（MLP），就是某种标准神经网络。我不打算做深度缩放。我只做宽度缩放。现在，如果我在做宽度缩放，我的模型越大——模型越大——我的学习率就应该越小。我的学习率应该更小的原因是因为我有更大的、更多的参数，我在同时改变更多的东西。也许我应该移动得少一些。这有点道理。事实上，有一个经验法则，也许你想按宽度的倒数来缩放。这是一个非常常见的经验法则，你只需要随着宽度的函数有规律地降低学习率，这会给你一个缩放学习率的经验方法。

[49:32.340] 还有另一种思想流派，或者说人们使用的另一套技巧，就是人们也会以各种方式重新缩放网络。他们会改变初始化的大小，他们会改变优化器针对网络不同部分的步长，以便迫使模型在不同尺度下具有相同的最优学习率。这个想法被称为 mu p（最大更新参数化）以及其他类似的方法。有些人在这类方法上报告了巨大的成功，其他人则报告了不太成功的经验。基本上，设置大规模训练运行学习率的两种方式确实都体现在这张图中。一种方式是尝试估计这些最小值是什么，并尝试预测你的最小值会是多少。而最小值的变化方式是非常可预测的，所以这不是一个疯狂的想法。或者你可以重新参数化你的模型，试图让最优学习率保持不变，然后直接选择你最好的学习率并开始训练。

[50:31.390] 这就是选择学习率的两种哲学。两者都已在规模化训练中成功应用。有大规模训练的运行使用了这两种方法。我觉得从经验上看，似乎更多人可能倾向于缩放定律的方法，但两者当然都是可行的方法。我将在下节课详细讨论这两种方法。所以在我进入这节课的最后一部分之前——我想讨论的不是那种陷阱，而是缩放定律的各种细节——我想以这个想法来结束这一部分，即上游和下游性能可能差异很大。所以如果你看像对数似然这样的东西，这来自 Yi Tay 的架构论文，你会发现困惑度（perplexity）和负困惑度在这种情况下与参数数量高度相关。你的参数越多，你的模型越好，非常可预测。

[51:33.260] 这是一个非常漂亮的线性趋势。现在你会觉得，好的，这太棒了。现在我们要发布我们最好的模型了。NL12。结果发现 NL12 并不是你最好的模型。实际上是 NL32XL，一个在困惑度上差得多的模型。这是确实会发生的一个非常常见的现象。上游和下游通常是相关的。这可能是我见过的从上游到下游最差的关联之一了。但我认为非常非常重要的一点是记住，缩放定律一般来说，是你想要应用在困惑度侧的对象。它们非常干净、非常规律、非常可预测。但从困惑度到下游任务的迁移，其确定性远不如乍看起来那么高。这是一个你需要格外谨慎的地方。举个例子，我以前的学生去各个地方做后训练，他们总是抱怨。

[52:20.380] 他们会说：“哦，那些做预训练的人，他们把这个模型交给你，说困惑度很好，现在都是你的问题了。”但问题往往出在预训练这边。所以你不想成为那样的人。你要真正去思考迁移问题，而不仅仅是盯着困惑度。好了，这也许令人惊讶，也许不。但我想在这里强调的重点是，在你进行大规模训练之前，你应该对自己将看到的结果有一个大致的预期。不仅仅是它能不能训练起来。事实上，如果你做尺度定律，你应该能够相当精确地预测出你的训练效果的数值。如果我选择一种优化器而不是另一种，收益会是多少？你应该能够预测这一点。所以基于尺度定律的流程是：我要训练一堆较小的模型，通过在双对数坐标图上拟合这些直线来建立某种尺度定律。

[53:14.210] 然后如果拟合效果足够好，我就有信心这个差距会持续下去，然后我就把它直接应用到我的大规模训练中。这是尺度定律流程的朴素版本，但我认为它合理地反映了你在实践中思考这些问题的方式。你还能怎么做？直接凭空部署一次训练？大概不会。好了，我可以在进入这节课的最后部分之前暂停一下，回答一些问题。请讲。对于尺度定律上的每个数据点，你如何看待方差？你做多少次训练？聚合通常是取平均，还是根据情况取最小值或最大值？是的，我认为在大多数情况下，如果不是我见过的所有情况，这些都是单次运行的。之所以是单次运行，是因为当你看困惑度的时候，困惑度是非常非常干净的。

[54:08.480] 这些指标，如果你重新跑一遍，它们的方差可能在小数点后第二位左右，因为训练数据非常同质化，数据量很大，评估集也非常大。你基本上已经把这些指标的方差压到了非常非常低的水平。但如果你在做学习率或临界批量大小的尺度定律，你会看到一些真正糟糕透顶的东西。我确实认为人们会做一些方差消减，但实际上这并不像应该有的那么普遍。请讲。人们会直接使用下游指标而不是困惑度吗？是的，人们确实直接在下游指标上做尺度定律，但这有同样的问题。所以我认为思考尺度定律的方式是，真正可靠地建立至少一件事情。这样想：假设我看到类似这样的东西，我有很强的信念这条transformer的线会继续下去。

[55:09.710] 这是一个非常规律的走势。它可能会有一点弯曲，但我们确实认为这里可能存在某种规律性。但反过来，假设这是一个非常嘈杂、锯齿状的线。你就不会真正知道发生了什么。你可能会说，我或许相信它会继续，但也许是一条线，也许是一条曲线。所以我认为总体而言——如果我要解释这些东西的哲学的话——就是说，好吧，让我们从低方差的测量开始，在那里建立尺度的规律性，然后依靠某种关于迁移的信念，或者在其他地方建立迁移，来试图表明这可以迁移到下游任务。是的。所以其中一些是训练损失，一些是验证损失。你什么时候用哪一种？我明白了，是的。当你做良好的科学实践时，比如Kaplan 2021，你会拟合测试损失。但实际上，除了极少数模型之外，比如无限计算的那个和重复数据的那些，除非你在做那些，否则你都是在做单遍SGD。

[56:15.730] 所以所有这些模型的泛化差距都非常非常小。训练和验证非常接近。我见过一些预训练代码库甚至没有验证损失。因为如果你在做单遍训练，很可能你的训练和验证非常非常接近。这就是为什么很多这些图表会交替使用这两者，因为它们都处于单遍训练的状态。不过问得好。好了，我们继续讲——抱歉，我刚才往回走太远了。好的，现在我要讲尺度定律最著名的应用之一。即使你可能不会亲自使用它，你肯定在某个时候用过这个经验法则。而且，我认为因为这组东西会告诉我们一些关于尺度定律实际执行的有趣内容。好的，驱动性的问题是：我需要更多数据还是更大的模型？

[57:11.020] 你拥有的资源，你需要花费的资源，是算力。有人会象征性地给你一定数量的FLOPs。你要拿着这些FLOPs，把它们花在某个地方——如果你还记得Percy的第一节课，数据量乘以参数量大约等于你的FLOPs，它们是线性关系。那我该要更多数据还是更多模型？我们知道，如果我们拿一个极小的模型，然后往里面倒入大量的数据，那只会被浪费掉。这就是最上面那条深紫色的线。我拿了一个极小的语言模型，然后给它传入了大量的数据，这个东西在很长一段时间内都是平的。这是对算力的完全浪费。你宁可把这个大的黄色模型训练到同样的token数量，甚至更早，如果你的算力是相同的话。现在，如果你能理解数据量和模型大小如何相互作用来产生性能，如果你理解了这种关系，你就可以非常精确地进行优化。

[58:16.010] 所以有一些更高级的尺度定律，它们关联的是各对象之间的联合行为。差不多同时，Kaplan和Rosenfeld都提出了这些函数形式，它们大致等价，描述了模型大小和数据量与误差之间的关系。Rosenfeld的非常简洁。实际上就是两个倒数项之和。这是两个加在一起的尺度定律。Kaplan的稍微复杂一点，但思路大致相同。如果你看一下极限情况，想想这些极限，这是非常直观的。如果我有无限量的数据，那会发生什么？那么这一项趋近于0。你基本上受限于模型大小。你变成了一个纯粹的模型大小尺度定律。如果我把模型大小推向无穷大，那我就会被数据所限制。我就会有一个纯粹的数据尺度定律。

[59:10.000] 所以对这两个公式取极限是有道理的。通常，如果有人告诉你一个尺度定律，你对所有变量取极限来理解系统的行为是个好主意。那么，很酷的一点是，Rosenfeld和其他许多提出这类联合尺度定律的人都表明，这些东西拟合得非常好，即使你进行外推也是如此。所以你要做的是，在这些绿色点上进行拟合，这些点处于低计算量和低数据量的区间。你从未见过同时具备高数据量或高模型规模的情况。然后你可以基于这个角落里的这些小型运行结果进行外推，推算出高计算量或高数据量、高模型规模的区间。你可以非常准确地根据你的预测来预测误差。预测效果非常好。因此，一旦你有了这个，现在你可以做的是说，好，我有固定的FLOPs数量，在FLOPs约束条件下，最小化这个目标。这是一个简单的非线性优化问题。

[01:00:09.830] 你可以用这个来求解。现在，Kaplan和Rosenfeld都提供了这种形式的方程。他们说了什么呢？如果你看Kaplan，他们求解这个方程，他们会说类似这样的话。嗯，数量是——他们总是把这个倒过来。这让我很困惑。数据量应该是C的0.73次方，模型规模应该是C的0.27次方。这反了吗？我觉得反了，抱歉。N应该是参数量，D应该是数据量。所以每个参数的token数随着C的增加而减少。因此，当你增加计算量时，Kaplan的这个建议告诉你要训练越来越大、越来越大、越来越大的模型。所以如果你经历过GPT-3的时代——想到有人没有经历过这个时代，我有点震惊——你会知道有一段时间每个人都在训练这些巨型模型。

[01:01:06.730] 比如几千亿参数的模型、万亿参数的稠密模型，等等。部分原因就是被这个推动的。他们看到了GPT-3，看到了这些Kaplan的预测，然后他们说：“哇，我要在我庞大的模型规模上做我的大规模训练。”但几年后，在2022年，DeepMind的一些人，Hoffman等人来了，他们说，实际上，这些预测全都错得离谱。你们一直在训练这些巨型模型，就是这边的这些三星标记，但实际上，那些模型太大了。我们应该训练的是一条非常不同的趋势，就是这边的这条蓝线，我们应该训练的是这个青绿色的星形标记，那是一个好得多的模型。

[01:01:48.640] 所以他们主张，我们应该训练相对小得多的模型。我想，当然，你们中很多人或者所有人可能都知道Chinchilla论文，你们知道token倍数是多少。每个参数对应20个token。但我认为深入细节是有用的，因为Chinchilla和Kaplan之间的分歧以及所有这一切告诉我们一些关于我们应该如何思考scaling laws的重要信息。这不是一个你转动曲柄就能得到答案的事情。这实际上是一个需要谨慎和敏感对待的事情，我们必须尊重这个过程。所以我将讲解Chinchilla论文。在我讲解完Chinchilla论文之后，我会讨论为什么Kaplan和Chinchilla

[01:02:27.910] 论文存在分歧，以及这对你意味着什么。所以Chinchilla作者提出了三种不同的拟合scaling laws的方法。我非常喜欢这一点，因为它是一种让你对可能做出的建模假设保持稳健的方式。而且这些方法中的每一种实际上都是估计模型规模和数据之间权衡的相当不同的方法。除了方法3有些不同之外，它们大多是一致的。方法1和方法2都同意数据和模型规模的系数大致相同。0.5，0.5，0.49，0.51。方法3确实不同，0.46和0.54，但我们稍后会讨论为什么。Kaplan等人，当然，两者之间有很大的差异，这就是为什么他们在非常大的计算规模下训练大型模型。Chinchilla的方法1是我可能会称之为下包络线法的方法。这也是Kaplan部分使用过的方法，试图找出正确的scaling是什么。

[01:03:29.500] 所以你要做的是，取你所有的训练运行。所以这些彩色线条中的每一条都是一个训练曲线。而你要做的是，实际上取这些训练曲线的下包络线。因为这条训练曲线上的一个下包络线点意味着，对于特定的FLOPs，这是我在任何训练运行中为该FLOPs达到过的最佳损失。所以我可以取所有下包络线点，在底部这里的高计算量区域，然后我可以绘制所有这些点，看看每个FLOPs点对应的模型规模有多大。因为每一个点都是一个具有特定模型规模的训练运行。如果我把它做成散点图，我会得到一个漂亮的线性趋势。那会给我一种估计FLOPs和参数之间最优权衡的方法。这给出了一个答案，即对于他们的计算预算，他们应该训练一个670亿参数的模型。

[01:04:19.580] 这是一种简单且不错的方法，但定义什么是下包络线等等这些事项非常棘手。肯定有一些棘手的部分。方法2是IsoFLOP。这在现在非常流行。它也非常简单且非常稳健。这可能是我个人最喜欢的方法。你要做的是，为此，你选择一组不同的FLOPs预算。所以我有不同的FLOPs预算范围，在这种情况下以3的倍数递增，或者1、3、6。然后你要做的是，对于每个FLOPs预算，你基本上要扫描整个空间。所以这里我有一个变量，即参数与数据之间的权衡。所以对于每个FLOPs预算，我要在固定FLOPs的情况下，扫描参数与数据的权衡。所以我将数据规模加倍，模型规模减半，以此类推。那会产生一系列训练运行，它们的最终损失会描绘出某种曲线。

[01:05:11.040] 所以这些彩色点集中的每一组都是一个固定FLOPs的运行。每个点都在扫描不同的参数，当然，它也在隐式地扫描数据。现在，好处是我可以取这些运行中每一个的最小值，或者甚至拟合一个二次曲线并取二次曲线的最小值，然后我可以画一条线穿过所有二次曲线的底部，或者取二次曲线的底部并将它们绘制为FLOPs与参数的函数关系。那会给我另一个预测，结果非常接近。对于我的计算预算，这是630亿参数。现在，我们能做的最后一种方法实际上可能是最自然的方法，正如Rosenfeld论文以及Kaplan论文所提出的，因为它们两者都提出了一个假设的函数形式，用来描述模型规模和数据如何与损失相关联。所以如果你有这样的一个联合函数形式，你能做的就是，取你已完成的一大批训练运行，然后拟合你的损失景观，也就是你假设的函数形式，然后你只需要做曲线拟合

[01:06:19.560] 来估计你的scaling law中的系数是什么。这是最蛮力的方式。你有一个关于损失、数据规模和模型规模之间关系的假设，训练一堆模型，做曲线拟合。希望这是相当直接的——你如何做这件事。当然，如果你使用这种方法，你如何进行曲线拟合非常重要。现在，你在拟合这些曲面，有很多变量，这有点棘手。这个方法也有点棘手，就像方法1一样。所以它不像FLOPs方法那样直接明了。现在我们讲完了Chinchilla的做法，它和Kaplan的做法在本质上似乎没什么区别。并不是说Kaplan做了一件疯狂的事情，然后Chinchilla的作者做了什么更聪明的事，让你说"哇，为什么我们当初没想到这么做？"他们做的工作都很合理。

[01:07:10.520] 但如果我们看他们的缩放预测，差异非常大。而事后看来，我们通常倾向于认为Chinchilla作者的缩放在某种程度上更正确。如果你自己做一次Chinchilla式的分析，你很可能会发现缩放结果更接近Chinchilla。那么，当两者都在拟合这些联合缩放定律时，为什么会有这么大的差异呢？我想，这就是我们开始进入缩放定律背后混乱现实的地方。缩放定律真的会随着你实现缩放定律的具体方式、使用的超参数设置，甚至x轴是什么而发生很大变化。

[01:07:53.870] 有一篇非常好的论文。哦不，我把作者名字挡住了。我之后可以发到课程Slack里。论文叫《解决计算最优语言模型缩放的差异》（Resolving Discrepancy to Compute-Optimal Scaling of Language Models）。Yair是最后一位作者，他以前是这里的学生。他们展示了——基本上，首先他们用大致Kaplan的设置复现了Kaplan的结果。然后他们说，好的，让我们改变一下参数计数的方式。这会让曲线偏移一点。然后他们说，好的，让我们改变学习率预热的方式。这又让曲线偏移一点。然后让我们再对优化器多做一点调优。

[01:08:29.090] 然后他们就得到了和Chinchilla一模一样的结果。所以他们的论点在这里是，有一系列看似非常小的决策，最终却导致了Kaplan缩放定律和Chinchilla缩放定律之间的巨大差距。我会解释每一步是什么，这不代表这是最重要的事情，但同时也是为了向你们强调这里的细节有多重要。所以第一个箭头，从Kaplan到蓝色线的那一步，是参数计数方式。记得在Kaplan中我说过，如果他们包含嵌入参数，会看到一些奇怪的现象。所以他们排除了嵌入参数。这通常是可以接受的做法。但有一件事确实给他们带来了麻烦，那就是他们也排除了最后一层参数，也就是softmax层的参数。因为在很多模型中，嵌入层和最后一层线性层是互为对偶的。

[01:09:22.580] 它们的形状相同。嵌入层是词表大小乘以隐藏维度。输出矩阵是隐藏维度乘以词表大小。因为它们的形状相同，他们说，那就把这两个都排除掉。事实证明，包含或排除这些参数对缩放定律的形状有重大的实质性影响。他们还发现——这更多是一个疏忽，而不是真正的观点分歧——Kaplan训练的很多模型都非常小。事实上，它们小到在完成学习率预热时还没有真正收敛。所以他们的学习率设置得非常欠优。正因为如此，他们的模型没有完全收敛。然后最后，他们发现Kaplan等人固定使用一个大的批大小，而这些大批大小对于较小的模型来说是欠优的。

[01:10:11.670] 因此，如果你正确地进行调优，并使用不同的批大小，最终会得到与Chinchilla完全一致的结果。所以我认为这些差异中的每一个看起来都非常微小，但如果你改变这些非常微小的差异，缩放定律就会发生很大的偏移。我想强调的一点是，缩放定律在某种意义上是一种下界。它们有点像在说，如果我继续这个配方并放大它，这就是我会得到的结果。但如果你放大的是一个你本来不应该放大的配方，比如预热方式很疯狂，或者批大小很疯狂，你得到的缩放定律就会很差。所以你要尽可能接近正确的完整运行方式。还有另一篇论文做了另一件非常有趣的事情来研究Kaplan和Chinchilla缩放定律之间的差异。作者是Pearce和Song。他们做的，我认为非常巧妙，是他们不训练任何模型。

[01:11:07.620] 他们只是取Chinchilla的训练曲线或隐含的函数形式。他们直接把曲线画出来，然后说，基于这些隐含的训练曲线——这些曲线完全是从Chinchilla模拟出来的——他们生成如果忽略嵌入参数并且大幅缩小规模，Kaplan的隐含训练曲线会是什么样子，以及如果使用总参数数量又会发生什么。他们发现的实际上有两个效应。一个是Kaplan在更小的计算规模上运行，所以他们对小的变化非常敏感，而且忽略非嵌入参数确实是一个足够大的变化，以至于相对于Chinchilla来说搞乱了结果。所以他们的论点与Yair等人的略有不同，他们认为这里实际上是计算规模的大小加上非嵌入参数引入的轻微非线性导致了这种不一致。这确实开始进入很细节的东西了。

[01:12:01.580] 但我希望你们能体会到注意细节的重要性。我认为在建立和依赖缩放定律过程中，很多工作都是在弄清楚正确的区间是什么，这样你才能在不同计算规模上获得可靠的缩放。最后一点，这更多只是我觉得有趣的事情，就是Chinchilla这篇论文在某种程度上推翻了Kaplan的论文。就像是说，"哦，你们错得很离谱。"但有一个剩余的谜团，那就是Chinchilla的方法三从来没有和方法一和方法二一致过。数字看起来差别很小，作者们对此非常淡定，但我认为影响其实不小。如果你看这个，方法一和方法二基本上说参数和token的数量应该一起缩放。所以两者之间应该有一个固定的比例。

[01:12:50.540] 这就是Chinchilla因子20的来源。如果你看方法三，指数是不同的。这意味着如果你扩大计算规模，最终在这种情况下，你的token会比参数多得多。所以它在渐近意义上会有一个与其他方法非常不同的缩放结论。事实证明，我肯定不是唯一一个对此感到不安的人，只是Epoch AI的一些人真的采取了行动来应对这种不安。他们拿不到原始数据也拿不到代码。所以他们做的是从论文的图中提取数据，然后重新拟合法三的回归。他们从论文中提取了数据。然后基于这些数据，他们拟合了那个被提出的二维曲面或三维曲面。

[01:13:37.710] 然后他们展示的是，原论文实际上欠拟合了原始数据。如果他们重新做曲线拟合，相对于Chinchilla论文中提出的缩放定律，他们可以得到低得多的损失。而且实际上，如果你遵循重新拟合的模型，你得到的几乎是精确的"D除以20等于20"这个经验法则，也就是来自Chinchilla论文的那个规则。我觉得这真的是Chinchilla论文一个很有趣的结论，最后事实证明，作者们比他们自己意识到的更加正确。实际上，他们犯了一个错误，而这正是方法3与方法1和方法2不一致的唯一原因。所以这是一个很好的结论，让我们知道这个方法终究是稳健的。好了，从某种程度上说，这就是Chinchilla故事的结尾。但我想用一句话来结束这一部分：你大概不想要Chinchilla系数。在这一点上，我觉得这已经算是相当普遍的常识了。

[01:14:35.220] 但如果你在训练一个生产级模型，你在大多数情况下并不真正关心节省训练算力。你可以看看人们做的各种外部调查，看看前沿实验室是怎么分配他们的算力的。大部分算力并不是用在训练运行上的。大部分算力实际上花在了研发和推理服务上。而且考虑到这一点，特别是对于推理服务，你想要的是小而强的模型。你不想要又大又臃肿、服务成本很高的模型，即使那能最小化训练成本。所以你真正想要的是那种你会称之为“过度训练”的模型。我加了引号，因为实际上，过度训练正是我们想要的。那才是正确的训练量。如果你看看早期发布的所有模型，比如GPT-3——这个绝对是训练不足的，每个参数只有三个token。

[01:15:21.830] Chinchilla达到了每个参数20个token的现代比例。然后有一段时间，人们都停在每个参数20个token。我觉得那个时代，这些模型还没有那么多推理服务。模型很酷，但还没有大规模地部署服务。然后当你开始进入推理服务变得非常现实的阶段，现在你就在过度训练这些模型，而且你已经转向了MoE，你在做所有这些权衡来优化推理服务。所以我认为Chinchilla很重要，不是因为我认20比1的比例就是那个黄金比例。如果你是在做研究，那也许成立。但我认为它重要的原因在于，它告诉了我们很多关于应该如何拟合缩放定律以及如何思考缩放定律的信息。好的，关于Chinchilla以及缩放方面，我要说的最后一件事是，IsoFLOP是我认为作为研究工具一直延续下来的东西之一。

[01:16:12.290] IsoFLOP非常容易执行。你固定一个FLOPs预算，然后扫描所有其他自由度，看看曲面是什么样的。这通常会给你相当可靠的读数，让你了解不同参数如何随着你的自由参数变化。人们已经做过——实际上，我的一个学生为扩散模型做过这个。我讲过的MoE研究就是IsoFLOP风格的设计。如果你遇到这样的情况，在想我该怎么决定这些权衡，IsoFLOP总是一个好的默认选择。好的，总结一下今天的所有内容，有这样一个想法：我们投入的资源量和我们获得的性能之间存在一种对数线性规律。而且它适用于很多不同的东西：模型参数量、算力、MoE的稀疏度。

[01:17:05.880] 这让你可以做很多我之前提到的那些事情——那些本来像是任意选择的事情——但希望是以一种更基于证据的方式来做。这样我们就能够在不明确依赖大规模模型训练运行的情况下进行大规模工程。希望你们还记得这里所有的不同组成部分。数据缩放是一个非常自然的对象，只是指数有点奇怪。模型缩放让你能做所有这些很酷的工程事情。最后，缩放是一个很酷的、稳健的预测器，让我们这么说吧，它是尝试在大规模下做工程的一个指导工具。谢谢。下一讲，我想是Percy讲推理。但之后，我们会回到更进阶的细分缩放话题。[掌声]

CS336 2026 · Lecture 9 · Scaling laws

视频：官方视频
字幕：en-US · manual captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：原语言字幕稿，待校对

正文

[00:06.040] We will get started. Well, temporarily leaving the land of systems to talk about more deep learning stuff, which we're going to have two lectures on scaling law and scaling law adjacent phenomena. So today is going to be the basics. We're just going to talk about pretty basic scaling law stuff. Some of the classic works. How does the basic idea of scaling laws connect to maybe things you know about from machine learning 101. And then in two lectures, we're going to do the more advanced version of the scaling laws lecture, where we're going to basically go into a bunch of modern, open model tech reports, things like mu p and other parameter initializations, and this year I'm going to throw in some more stuff for optimizers. So there will be an advanced version of this that will try to really get to the cutting edge. Today, we're going to start with all the really basic stuff. And due to some scheduling constraints, next lecture will be inference, so you're going to go back and forth.

[01:04.129] You're going to do scaling laws and inference and then scaling laws again. So scaling laws is really the following scenario in some ways, or you run into it very quickly once you run into this scenario. So you have your very wealthy friend, who has given you 10,000 B200s for a month, and he or she has asked you to build a very good open source language model. They would like to get something out of those B200s. And so you've already put together your infra team. That's your assignment to buddies. And you have, let's say a good pre-training data set. That's going to be the assignment after this, but let's say you already have that. And then you're going to go and train a big model, but there's lots of choices in training a big model-- all the assignments, the earlier lecture two stuff that we had earlier, where you have questions like what architecture should I use? What should my hyperparameters be?

[01:56.730] And it's very scary to run these things on a run that is basically, potentially millions of dollars in cost, so how do you really take this idea of scaling up very seriously? How do you make sure that your big run is actually successful? So of these many choices, some of which you're going to just copy out of the literature. The stuff that I talked about earlier in the architecture lecture is really about, well adopted best practices, and you will probably just pick some of these out of a hat. That's fine, but really, let's say you're at the frontier. You would like to build something that's actually better than the best available models today. You have to start really optimizing these. You can't just copy the choices of others and get something that's better than the state of the art. So how do we do that? Well, that's going to require us to really engage with this idea of scaling. So scaling laws have been one very powerful tool.

[02:52.260] You might also call it a paradigm. Scaling laws, if you talk to some people who are really in these big labs doing scaling work, it's almost kind of a way of life. They're like, we really believe in the scaling laws. It's almost a belief, and you'll see why scaling laws can sometimes be quite tricky objects. But really, scaling laws are these simple, predictive rules of how to go from small scale model performance and behavior, and to try to extrapolate them up to large scale behavior. And so the basic naive way to approach model training might be to say, I have a big budget, I'm going to do multiple training runs on these very big runs, and I'm going to tune hyperparameters on those big runs, but that's very wasteful. So instead, what I would like to do is I would like to do all of my optimization at the small scale and have some simple rule that allows me to extrapolate small scale behaviors to large scale behaviors. And if the small scale and large scale are connected by some very simple, robust connections like regularities of some kind, then you

[03:52.100] would have confidence in this kind of new scaling approach to optimizing your system. And that's really what today is all going to be about-- this engineering view of a scaling law. And to start with, I'm going to talk a little bit about background, because scaling laws are very interesting in that not only are they the newest paradigm, that's how a lot of these models are designed and thought, but also they connect very closely to very classical ideas. So if you're a person that's a machine learning theorist, actually, there are some things that you will really feel at home with in some ways, so they're like empirical sample complexities. And then I'll talk through the history of scaling laws of one kind and show you that this is not something that's really new. It's something with actually quite a bit of history and research done to it. So I really, really had to put some math into my slides.

[04:47.270] I really feel the need to do so, but one of the things that is very interesting is that for the longest time, machine learning has thought about this question of how will my model perform, especially from a theory perspective? I have this model class, how good is my model going to be? And generalization bound is the theorists answer to this question, so you have your error, over some finite hypothesis class, and you say, oh, my error is going to be at most this much worse than the model's performance on my training set. This is classic generalization bounds. You can write this down for more complex classes like smooth densities or whatever, but importantly, not only does this tell you some of performance, notice that these bounds are often dependent on the sample size. And so this is telling you, well, this is a theoretical upper bound on my loss values as a function of how my training set grows.

[05:39.220] And in some ways, that's very closely related to this idea of what happens as I scale up the amount of data that I have? Those are closely related questions, where you might believe me that that's true. And if you try to figure out what is the very first scaling law paper that exists? It turns out you can go back quite far. You can go back to Bell Labs and Corina Cortez and Vladimir Vapnik and these folks that were doing very serious theory work on machine learning, ask this question, well, training classifiers on a really large data set is often very expensive. We would like to avoid doing that, and to do so maybe we can just fit classifiers on a smaller sample, fit a curve to how their error rates decay, and use that as a way to estimate their performance. That's almost literally a data scaling law back in 1993.

[06:29.990] So this idea, at the very least, thinking about how your model's performance changes as a function of data set size is just a very classical one, and there's a lot of work that supports this kind of idea. I just want to connect this to some of these older ideas so you don't really think of scaling laws as this very new, purely neural thing. It's an idea that has a lot of roots in empirical machine learning, and also connections to theoretical machine learning. And if you start to look at the history of this, there's a lot of people that have said, isn't it the case that maybe instead of spending money on algorithm development, maybe we should just go collect more data, because if we plot how the performance of our systems behave as a function of the amount of data that we have, it turns out that they are all increasing or improving in predictable ways? Banko and Brill, is one of the canonical NLP references in how the amount of data and thinking about scaling is a really valid way of improving system performance,

[07:30.790] so there's lots of ways of thinking about this. And there's also work by other NLP folks like Kolachina et al. in 2012. This is one of the earlier works on different kinds of functional forms for scaling laws, where once again, they're increasing the amount of data that they're using to train. In this case, machine translation systems or BLEU scores. And they're asking like, what is the right functional form for how a model's performance behaves as a function of increased data amount? And they maybe surprisingly or not surprisingly, end up with the same power laws, which is pow 3 and 4 as really we do today. For the rest of the lecture, you're going to keep seeing essentially these polynomial functions, pow 3 and pow 4 as I talk about scaling laws. And we knew about a lot of this back in the 2010's that these kinds of functional forms were very predictive of model behavior as we scaled them up.

[08:27.940] And then finally, I really like mentioning this paper because for whatever reason, they don't get, or Hestness et al. don't get cited quite as much as they should. The origins of neural scaling, I would point back to Hestness in 2017. Really a very forward looking work in many ways that built these scaling laws as a function of data set size for a wide variety of systems, different kinds of speech recognition systems, machine translation systems, language models, and they showed that these things follow these nice polynomial trends across a really large number of domains. And it was really ahead of its time, because 2017 was long before some of the earlier OpenAI scaling laws and so on in the 2020s. And they talked about stuff that we talk about today, like emergence, where suddenly models might show certain kinds of capabilities because accuracy is a much more discontinuous measure than losses. And really, scaling by compute, that if we have systems that scale predictably to large training data, then really compute is going to be really important.

[09:32.950] And then finally, since we're scaling by compute, well, speed and systems optimization is going to turn into accuracy. So in some ways, a lot of the things that we see today, like the phenomena of system scaling and so on, we could have really known, or I could have known in 2017 had I read and thought about a lot of these scaling laws papers back and thought about them more seriously. So to mention that paper in every one of these scaling law lectures I give, just to say, a lot of these phenomena were known far before the modern era, or the Modern Language modeling era in that it's possible to have known the regime that we're in even before seeing these large language models. That's very cool to know. Any questions about this semi-historical part? Yeah. [INAUDIBLE], and I'm wondering if those are just stored empirically, or is there some way we can justify all of this? You're asking about these scaling laws, and where they come from?

[10:33.680] These are pure curve fitting exercises. I will talk a little bit about where scaling laws can come from and why polynomials are arguably a very natural class of scaling laws, but there's no golden rule that these are the only ones. I will say, though, that in some ways, theory provides a very rich place, where the scaling law functional forms can come from because theory is often thinking a lot about how do error rates decay as a function of various objects? And they give you candidate functional forms of various kinds. Physicists are also very good at this because they thinking about limits, like how do the limits behave? And that gives you different scaling law functional forms. Cool. All right, any other questions? Good. So now I'm going to start talking about more modern neural language model scaling, and I'm going to start with data scaling laws because to me, they're the most simple and natural objects.

[11:34.320] Talking about data scaling laws also allows me to walk you through very step-by-step on how we might guess what the functional form of a scaling law might be and why the polynomial form is a natural one. And then I'm going to start talking about hyper parameters and architectures and all sorts of other more exotic objects that are going to be harder to justify from intuition, but still turn out to have very nice scaling law behaviors. One of the things that's remarkable about this scaling law style analysis has been that these power law relationships seem to hold for a huge variety of different factors. So the basic ones that we see in pre-training, and those are going to be the ones that I'm going to talk about today. The usual ones that you see are compute on the x-axis. This is log compute. Sorry, on the x-axis, log of test loss on the y-axis, or log of the data set on x-axis, log of loss on the y-axis, log of parameters on x-axis, log of loss, once again, on the y-axis.

[12:33.610] So we can get a variety of different x-axes that you might think of as resources in the log axis. All of them linearize log test loss here. You can see also much more exotic scaling laws. You can see scaling laws that are basically the equivalent to these in downstream benchmarks. In those cases, you expect to see a sigmoid functional form as a function of compute or compute-like quantities. There's even been many works recently, more on the forecasting side, where people are doing things like putting dates as the x-axis. And strangely enough, at least for these plots-- like the upper envelope of these curves are often linear in various forms of capabilities. There's no reason that on this bottom side, the more exotic scaling laws need to be true. But it turns out to be the case that I guess language model performance is often much more regular as a function of scale and resources than it initially might appear.

[13:33.860] So to begin with, I'm going to start with data scaling laws. And data scaling laws are a very simple kind of univariate relationship that we're going to think about. So in this world, we're going to fix our model training procedure. And in general, unless I say otherwise, the models are going to generally be bigger than the data set size. And I'm going to increase my data set size in a regular way, and I'm going to see how the error reduces. And what do we expect out of this relationship? In general, it's going to be monotone-ish if we tune our hyperparameters well. Because we have more data, more data should help us do better at our task. The error should decrease. And if you're doing something like classification, or even for next token prediction, you should really go from random guessing to some of entropy-like object where you can't do any better than the irreducible error of your task.

[14:22.350] That's the noise floor. So we're going to have some sigmoidal-like object over here. So this is generally what we expect out of our system. So we're going to start with an empirical observation, and should make this on your own. You'll make something that looks roughly like this in parts of assignment 3, which is if we just have a very big model, much bigger than our data set, and we increase the data set size, and we put that on the log axis, and on the y-axis, we have log test loss. Once again, you will find that the two points that you plot will form a very clear linear trend. I've taken this from Kaplan in 2020. I'm going to refer repeatedly to this paper. This one's OpenAI's "Neural Scaling Laws" paper. They have a lot of lovely ablations and a lot of lovely data. Now, this relationship, where I've plotted a line on a log-log plot, this is a scale-free or power law relationship. And what does that mean? If I have a line on a log-log plot?

[15:26.258] Many of you-- to many of you, this is probably obvious. But just to really emphasize this, if something is linear in a log-log plot, what does that mean? Well, it means that the error that I have is decaying polynomially. And also, it usually means that I'm very far away from my asymptote. Because once I approach my asymptote, I'm going to taper off rather than have a line. So let's go through an example just to show you how scaling law arises very naturally on a very simple estimation problem. So forget language modeling for the moment. This was a language modeling class, but now it is a statistics class for the next one slide or actually two slides. No, maybe three slides. So in this case, what I want to do is I want to estimate the mean from a bunch of data. And so I have an input, it's drawn from a Gaussian. And I'm going to estimate the mean of this object. I just write down the empirical mean. And then, you can write down what the error is.

[16:18.220] So what's the expected error? Well, each of these guys is Gaussian, and that the mu hat is Gaussian, so I can write down the error. That's going to be sigma squared over n. So well, if you look at this, is a scaling law. I've written down the error, and the error, if I log that, is a function of log n, log n being the log x-axis. So in general, if you can come up with anything that is essentially 1 over n to the alpha plus some constant, when you plot that on a log-log plot, subtracting out the constant term, you're going to get a scaling log. So anything of this form, where you have polynomial decay and errors, is going to end up giving you a scaling law. And of course, in parametric estimation like mean estimation or regression, you of course, expect a polynomial rate. You don't really expect anything else. And if you've taken your stats estimation courses, you know that essentially all the classical models you are expected to have a 1 over n scaling.

[17:17.349] If you fit a regression or so on, you might get D over n or whatever else. And so if this is true, and this is the origins of scaling laws, if we plot our lines, we should expect to see roughly a slope of minus 1 because that's the exponent on the x. Y equals minus x plus C. If we fit our neural scaling laws, we will find that our exponents are quite different. These exponents-- I've taken from these two on the left from Hestenes, I've taken the one on the right from Kaplan. They're about -0.1, -0.3, -0.1, roughly speaking. And so what does that mean? That means this is much slower than estimating a mean or estimating a linear regression. So this is much, much slower convergence rate, still polynomial, but much slower. Now this is interesting. It's saying that we're getting these kind of nice polynomial rates, but the exponents are not so great.

[18:11.320] They're -0.1. Where might you get an estimation rate like this? Well, you would get this if your model was more nonparametric. I promise this will be the last statistics slide of this lecture. Let's say that I want to estimate now, instead of a mean, an arbitrary smooth function. So one thing I could do is, if I'm doing a regression problem, I have my inputs, x1 through xn. Let's say they're in the unit box. I have a little bit of Gaussian noise, and I have an unknown function f that I would like to estimate. A trivial estimator for this is I can take my space, I can cut it up into little 2D boxes of length n to the negative 1 to the fourth. And then what's our estimation error? I'll have square root n boxes. Each box is going to get square root n samples. And the error is going to be roughly 1 over square root of n or something. And in general, if have a D-dimensional function I would like to estimate in this very nonparametric, arbitrary way, my error is going to be roughly equal to n to the negative 1 over d.

[19:11.040] So my scaling is going to be y equals negative 1 over D times x. And so now these kinds more flexible functions, functions that are more flexible than the linear class, now have rates in their scaling laws that are not just 1, they're 1 over D. And so one mental model that you could have is that the neural networks I was showing you in these slides are behaving kind of non-parametric regressors in 10 dimensions, like a nearest neighbor or other kinds of estimators in 10 dimensions. That's roughly the rate at which they learn from data. That's kind of a cool thing to be able to know just by looking at some of these scaling laws. Some people have argued this more strongly. Bari et al. and some other scaling law theory folks have argued that the scaling law exponents are actually literally telling us that the neural network is behaving like a non-parametric smoother, and so on and so forth.

[20:02.568] It's an interesting set of arguments, and you can by that intuition to whatever extent you'd like. I don't quite how much I truly, truly buy this argument. There's some of the evidence that might be a little sketchy. It relies on estimators of intrinsic dimension, but it's an interesting thing to think about that somehow these exponents are telling us how fast are these neural networks really learning. OK, so actually, I'll pause for the moment here. That's really the end of the really basic data scaling law of one distribution, and you sample from it, and you learn from it. Is that all clear, like the examples and so on? Good. Oh yes, question. When you say that the model is always larger than data set size, do you mean one-to-one number of parameters to number of tokens [INAUDIBLE]? Yeah, that's a good question. So the question was something like, What do I mean when I say the model is larger than the data set size?

[20:52.580] What I mean is when the model gets small relative to the data, like you have way more data than you have model size, you enter this irreducible error regime. You basically fitted the data as well as you can for your model class, and you can't do any better no matter how much data you add. So we want to-- when we're thinking about scaling laws, we usually want to think about this power law regime, which is the simpler case. Either that or you explicitly fit the asymptote and then you correct for it. Those are the two things. So when I say the model is larger than the data, it can be basically any amount that is larger than the data, as long as it's in this power law regime, usually like 10 times bigger than the data, what have you. Cool. Good. OK, now we are back to other data scaling laws. So data scaling laws by themselves are not quite as useful as you'd like them to be. They just tell you how fast does my model learn.

[21:48.450] And that's kind of useful for forecasting. It's not useful for very much else. If you want to do engineering, you are probably interested in other questions, like what is my optimal data mixture? What is the best way for me to pick my training data for my model? Very useful engineering question. Should I repeat my data, or should I just not repeat my data, save the compute for something else? Or maybe I should repeat some high-quality stuff and not repeat the low-quality stuff. I could do various other things with repetition. There's a lot of different engineering decisions I can make with data. So how can scaling laws help us make some of those decisions? And I think one of the things that's interesting is if we think a little bit about how classical models behave. Data scaling laws, if we think about their origins as essentially empirical versions of generalization bounds, they tell us, well, data set composition for many models really affect the offset of these scaling laws, but their slopes are actually determined by the model class. They're not really determined by the distribution themselves. So the slopes might remain the same, and only the intercepts might change.

[22:57.333] And the intercept might actually be really interestingly shaped. So this is a case of a toy model with a linear regression with two different things you can sample from. If you get only one kind of data, it's the errors are high no matter what. You actually want a mixture of both right. So if you write down how this intercept behaves, you actually start to get some very interesting insights about how having more data diversity is very helpful. But what does this practically mean? Well, practically speaking, what this means is you can actually fit data scaling laws on mixtures, and that can help you do some data-- sorry, pre-training, data optimization. So what does this mean? Let's say you have two data sources for simplicity. You have news, and you have Wikipedia. And how much news to use and how much Wikipedia to use. How can you determine that? Well, you can train really small models on a really small amount of data.

[23:52.795] And then you can basically train or you can fit a function of how different mixture levels affect your performance. You can do this while you scale up your models little by little. And you can see how the trends change. And you can try to extrapolate that out the exact same way as you would extrapolate out a data scaling law. And then you can ask which one will be optimal if I keep scaling it out to my full training run. So this is a very simple idea of essentially fitting a functional form at a small amount of compute, finding the minimum, and then scaling that guy out. So this is a data mixing loss paper. Unfortunately, I think if you talk to anyone who's done a lot of these data mixture work, they will tell you reality is a lot more noisy than this ideal world would suggest. And really, what has, as far as I know, happened in many cases is you end up training a bunch of small models.

[24:45.430] You pick the best data mix from the small model, and you just scale that guy up, no scaling law required. And data decide if you're interested in this kind of thing is a nice, large-scale empirical study of these kinds of data mixture things. And what they found was actually, if you don't fit a scaling law, and instead you just pick the best data mix at the small scale, that works well. For what it's worth, that's consistent with the argument that the intercepts differ, but the slopes don't change. Because if the slopes don't change, the best mixture at small scale is also the best mixture at large scale. OK, another really interesting set of questions that I'll also mention in the context of data is repetition. I think it's increasingly the case that compute is growing, the amount of data that we have is not growing. And so there's a lot of people interested in questions of what happens when you repeat data more and more times. And there was a nice study a few years back called "Scaling Data- Constrained Language Models," where they roughly show

[25:48.000] something like up to four epochs with standard training recipes, you just don't get hurt at all. But if you go past that point, then actually your realized scaling law, this dark curve is much worse than the projected scaling law if you had fresh data. That's the dashed line over here. And you can also, once again, write down a modified functional form that's generally predictive of the behavior of this scaling law under repetition. And we can take this idea even more to the extreme. This was kind of recent work with Percy in one of our co-advise students, where we were asking, like, OK, what happens if you take this idea to the extreme and you just consider infinite amounts of compute? So you're allowed to epoch as many times as you want, like an infinite amount of times. What is the best thing that you can get out of this system? Well, it turns out you can't just keep repeating passes over the data.

[26:37.422] You can't keep making your models bigger. Those have diminishing returns. And so you end up reaching for other things, like ensembling your models to try to squeeze more and more out of your data. But really, I think the one thing that I'll point out that's really interesting is your standard data scaling laws here in the red. It's a very nice, predictable, improvement in performance as you increase the amount of data. We do all sorts of interventions like regularizing and adding ensembles. You get improvements in performance, but the slopes look actually surprisingly similar. This is a lesson that you'll learn once you start fitting your own scaling laws, that very, very often your slopes don't change. Often, the interventions that you do just change the intercept of the scaling laws. One last thing that I'll mention, because I like this paper in the context of scaling laws and scaling in general, is that a lot of the phenomena we care about in data is actually very scale-dependent.

[27:31.950] And one example of this is data filtering. So if you or I tomorrow decide to filter some data for the models we're training, we would probably filter very aggressively. We would only keep the highest quality stuff because both you and I don't have very much compute. So we can't train on all the internet anyway. We might as well filter down to the most high-quality stuff. On the other hand, if you have a lot of compute, then you want to train on more and more stuff because you're going to not want to repeat on this very high-quality data. So as you get more and more compute, your filter has become looser and looser, and potentially you start training on stuff that's lower and lower quality. So a lot of the things that we often think about statically, things like data filtering or the quality of data, are actually much more dynamic.

[28:14.870] As you increase scale, you have to think about where am I going to get the rest of the data. The filters can't stay fixed as a function of the data, and certainly the optimal filters turn out to not be fixed at a function of scale. OK, so that's it for data scaling laws. This is the part that I think is most straightforward to understand. Even as a stats/ML-minded person, this stuff is all very simple and straightforward to me. You increase data, performance gets better in a very predictable way. I think this is uncontroversial. The fact that it does so polynomially for such a big model is still cool, but nothing really crazy. So hopefully this is reasonable to all of you that there's so much predictability when you increase the amount of data. Oh, yes. So on this slide for the pre-training under infinite compute, so that graph on the right is linear in both the loss and data set size,

[29:16.450] whereas the previous example I gave was log-log scale. And I was wondering why it's linear in both. Yeah, so this is a bad axis. Herman and I just talked about bad axis like this. This is actually doubling. So it looks linear, but it's actually log scale. [LAUGHS] I fixed it now. You fixed it now. And the y-axis here, it doesn't look logged, but the range of this is so small that linear logs are basically the same. If you look at the fitted functional forms here, these are still polynomials. So we're fitting the same functional forms, and those are the ones that are plotted here. This is another point. I'll get to it at the end of the lecture. If you only have a tiny slice of a compute range, it's very, very difficult to tell if something is scaling polynomially or if something's scaling exponentially. Because Taylor approximations are a thing.

[30:02.510] Everything looks linear if you zoom in enough. And so whether this is polynomial or some other functional form, you want to be always a little bit careful and skeptical. Good. OK. All right, so that was scaling laws for data, in many ways straightforward. Now I want to talk about more exotic forms of scaling and scaling laws for model engineering of various kinds. So what we're going to do now is we're going to try to design large language models. Maybe you're a radical, and you believe that maybe LSTMs are the future. Like, why can't I use LSTM instead of a transformer? I want to blow my B200 run on my LSTMs. Or maybe you're a different kind of radical and you want to train on SGD. Like, why can't I train on SGD? You might also wonder. If you have a limited amount of resources, you have different ways to spend it. Maybe you should train models longer, maybe you train bigger models, maybe I should go collect more data. There's all sorts of trade-offs always at play, and scaling laws give us quantitative ways of making these trade-offs, hopefully.

[31:10.040] I'm going to start off with some simple hyperparameter questions, things like how do I choose my architecture or optimizer, things that I talked about in the second lecture. And there I gave you the perspective of maybe you can study what other people do. Now I'm going to try to give you a first-principal answers to those questions with a little bit of a different set of trade-offs. So you have to believe in scaling laws in this lecture, but as long as you do, you get to try things on your own and find out. OK, the very first question we might start with is to say, Are transformers really better than LSTMs? LSTMs. They can work OK, too. They will fit distributions. With the advent of things like Mamba, we know that SSMS can work. So are they better? The brute force way to answer this question is to train a big LSTM GPT-3 or even bigger class of models.

[32:02.400] The scaling law way would be to train a bunch of smaller models and say, I'm going to train my transformer across a different variety of compute ranges, and I'm going to train different LSTMs with different numbers of layers on, once again, a variety of different compute ranges. Now, what does this show? This shows LSTMs have definitely different intercept, maybe even different slope than the transformer. And because of this, probably you don't want to pick out LSTM. This plot would justify scaling up the transformer instead of the LSTM. And if you look at a lot of the architecture papers today, like you go look at the Mamba paper or the gated delta net papers, they will always have a plot that looks like this, where you have vanilla transformer, our really cool model. And our really cool models, usually either on top or below, because that's what would prove that your model is doing better.

[32:51.510] And you certainly don't want something where your slopes are worse because that means as you scale up larger and larger, your models will eventually do worse. So this is one way of really reasoning about architectures without training gigantic models. There haven't been, I think, quite as many recent and good architecture scaling law papers. I think the one that is relatively recent and that I like quite a lot is by Ek and others at Google, back when they were publishing this kind of work, where they did a whole bunch of scaling studies on T5-style models, where they trained larger and larger variants of T5 models on many different architectures. And I think one thing that's really cool, and one of the reasons why I still bring this paper up, is they capture exactly the architecture changes that we implement today in our frontier models. So things like performer, which is an efficient attention, that doesn't scale very well.

[33:52.680] We do not implement those. The things that we implement are the gated linear unit. Red line better than green line throughout the scaling trends. Switch transformer. I don't what happened with their biggest run, but generally speaking, it's got good scaling trends. And I forgot, oh, mixture of softmaxes. Actually, we don't do this anymore, but this is probably an effective intervention given the scaling trends. So you see that through this scaling trends, even though a lot of these papers were operating on much, much smaller compute scales than what we operate today, they're able to see the kinds of architecture trends that are driving frontier model development back then. And I think this is why a lot of people use scaling laws as a really, almost paradigmatic way, saying, "Oh, if it doesn't show up in the scaling law, it's not a good intervention." Because in some ways, a lot of the past work has captured what we see today.

[34:47.370] Good. OK, going beyond architectures, we can extend this kind of analysis to all sorts of other things that we might want to understand. We might ask, OK, is SGD better or worse than ADAM? If you do the scaling law analysis, this one's by Hestness, you'll see very clear trends. This is once again, the really mysterious thing of the intercepts are different, but the slopes are very similar. If you fix the data and you fix the models, slopes are often very, very similar. I am very surprised by this every time I see it. And yet, it's very true. It's rare to get different slopes, even with an intervention as big as SGD to ADAM. If you were the one training with SGD or ADAM, you would think that's a huge change. Scaling trends remain roughly the same. In this case, this is a language modeling with a recurrent highway net.

[35:38.170] Similarly, in lecture 2, I told you about aspect ratios. How do you pick your depth-to-width trade-off? And I told you, well, you can just pick some random number that's somewhat roughly reasonable. It's like four times the reasonable multiplier. But if you're doing this kind of scaling analysis, you could much more precisely what is happening. So the first thing you could do is you could look at scaling trends as a function of layers, and you would immediately see that having extremely few layers is very, very bad for your model. If you have one layer, you're not going anywhere. That's a terrible, terrible scaling trend. Greater than 1 layer, actually, it's surprisingly more competitive, although you do see that at every compute level, more layers, at least in this case is better. And if you can do a much more fine-grained study, which they do in Kaplan, where you try to identify what you might call scale-invariant quantities, like the aspect ratio.

[36:32.000] The number of layers is not a scale-invariant quantity. As you make your model bigger, you do want more layers. But as you make your model bigger, maybe the aspect ratio, the optimal one should stay the same. This one is scale invariant. And so if we look at the aspect ratio, we see that at different model sizes, we get basically the same minima around 100d model for every layer, or maybe a little bit less. And it does shift a little bit right. There is a shift towards aspect ratio that is smaller for deeper models, but for the most part, the minima stays roughly similar. You can do similar kinds of analysis for things like attention head dimension and others. And if you look at the Kaplan paper, they do quite a good job at doing all of this. And this is one way that you can be confident about your scaling strategy. So if your scaling strategy is "I'm going to fix my aspect ratio, and I'm going to scale up," you can make plots like this and convince yourself we're probably good because our optimization shifting too much as I go

[37:29.790] to larger and larger and larger models. One thing that's important, and this is a huge can of worms, and a sufficiently big can of worms that I will talk about it on more than one slide, I will come back, not all parameters are created equal. And because not all parameters are created equal, your scaling laws may look good or bad depending on how you define what a parameter is. So in the Kaplan paper, they made this observation that if they drew their scaling laws for depth with the embedding parameters, they got these very funky-looking scaling laws, and they said, "Oh, this is no good at all, no good at all." Because it's no good at all, they decided that they were going to exclude all the embedding parameters, and they were only

[38:14.300] going to count the non-embedding parameters because you could justify it to yourself, saying these are the parameters that are doing computation or whatever. We're going to see later that this is going to have non-trivial implications on the validity of some of the results. But the point here is scaling laws aren't magic. I think Percy likes to make this point, that scaling laws and all these kinds of predictability across scales is engineered. They don't happen automatically. We need to pick the right kinds of x-axes to look at. We need to make sure that the hyperparameters for these things are set right, and only under those conditions does it become possible to get predictable scaling across many orders of magnitude compute. So this is what a lot of scaling law researchers do.

[38:57.520] They think about what the right way is to get very predictable scaling. Now, related to this, I think are very interesting recent papers on things like scaling for mixture of experts. Now, that mixture of experts is, you would say, the dominant way of training large models, it is quite important to think about what is the value or what is a parameter in this new world where the number of parameters and the number of parameters are kind of decoupled. And there's a nice, clean analysis by some folks at Apple and MIT that show the scaling trends as a function of both parameters and active parameters. And the cool thing that you see is, as you start to train bigger and bigger models, if you increase the amount of total parameters

[39:45.460] on the x-axis over here, and you want to minimize your loss, you're going to end up with more sparse and sparse models. See how the color becomes darker and darker and darker. Similarly, here, if you like the number of active parameters decreases as a function of MoE sparsity. And you get the result that I think is somewhat intuitive here, which is you can draw these surfaces. For every compute amount, you can ask, OK, would I rather spend that compute on active parameters and how much sparsity do I want? I can vary my sparsity without really changing the amount of compute that I'm spending. Now, given that you can ask, OK, what happens if I add more of empty parameters, same active parameters, but more total parameters? As you move from back on the plot to in the front of the plot, you see the losses are improving. So in other words, this is saying that the parameters in an MoE that are not active are still helping you reduce your loss, which is kind of cool.

[40:46.510] And you can write down what the functional forms of this are. I won't go into that part in more detail, but the point here is to say all of these kinds of quantities that we care about in optimizing MoE are things that have predictable scaling and regularities. And the two things I want to now talk about are actually really-- yeah, two things that I now want to talk about. When you're going to train a new big model, actually, most of the things that I talk about are fixed. You're probably all not radical enough that you're going to switch to a LSTM. You're probably going to pick like, I don't know, a DeepSeek-V4-inspired MoE or something. And you're probably going to pick architecture choices that are tried and tested. You're not going to really throw everything out. But two things that you really do have to maybe reinvent in some sense is the batch size and the learning rate.

[41:36.600] If you train your deep learning models, you know that you have to be very careful about the combination of your batch size and learning rate. You change one, you have to change the other. Now, batch size, from the previous lecture, is very tricky for another reason. We want the batch size to be as big as possible because a big batch size gives us opportunity for parallelization. Remember, data parallel requires large batch size. So now, having said all that, what we really want to is how large can we make our batch size before we start to suffer? That's the relevant systems question. And you also want to how does that change as a function of model size. So given that preface, there is this very important idea that is used in a variety of papers and talked about very frequently, called the critical batch size.

[42:24.990] The critical batch size is roughly this idea that up until a certain point, when you start to get diminishing returns, there is returns, perfect returns, to increased batch size. So I might call this the noise-limited regime. So in the noise-limited regime, every additional element that you throw in your batch reduces the gradient noise in your SGD step. And since you're variance-limited, that reduction in variance is very, very helpful. That gives you big returns. So this is the regime in which you have perfect scaling. You're doing about as well as you can, given the amount of extra examples that you're processing. As you get to a certain point where you've reached the noise scale of your gradients, you get to a point where you are no longer variance-limited. You're now bias-limited. What does that mean? Well, remember, in gradient descent, you're looking at the local structure of your objective.

[43:15.480] You don't have a global view of where the minimum is. So no matter how low noise I am, I'm only going to point in this light blue direction over here. And there's always a disagreement between my local descent direction and where the global optimum is. Because of that, at a certain point, I'm going to get diminishing returns because now my limiting factor is no longer variance. It is this bias term. So now I'm bias-limited past this point over here. And so critical batch is kind of a rule of thumb, you might call it, or a convenient trade-off point where you say this is the point at which we're starting to cross over from our perfect scaling regime to our ineffective scaling regime. And so it gives you a good rule of thumb of where to be setting your batch size if you want your batch size to be as big as possible without suffering these huge efficiency losses. Hopefully, that idea is very clear. Now, critical batch size is kind of a strange object. It is derived from a variety of complicated calculations about the local quadratic approximations to the objective. But I will maybe give you the mechanical steps of what is actually used to estimate this quantity. So what you do is first, you pick a target loss.

[44:36.180] Target loss might be some numbers that you want to hit. So your goal is to hit this target loss as quickly as possible. That's my goal right now. What I'm going to do is I'm going to sweep over all the different batch sizes or number of different batch sizes, and I'm going to write down the number of steps that I needed to reach my target loss, and the number of examples that were needed to get to that target loss. Of course, number one and two are related by a batch size factor. The number of steps times my batch size is equal to the number of examples. Now, through some calculations, OpenAI folks, McCandless and others, argue that this is the rough functional form that we should follow, that the number of steps that you need and the number of examples that you need are inversely related, with this estimate, which is the minimum number of possible steps and the minimum number of examples, normalizing these two quantities. And so there's a trade-off between these two. If you want to minimize the number of examples, you're going to have a lot of steps.

[45:37.550] And if you want to minimize the number of steps, you're going to need a lot of examples. That's a very natural trade-off. And if you increase or decrease your batch size, you're going to have trade-offs that go one side or the other. Now, they argue the goal now should be to balance these two sides. I want to make sure that my number of steps term and the number of examples terms are roughly balanced with each other. And if you solve for that, you're going to get this particular solution, which is that the critical batch size is the minimum number of examples that I need divided by the minimum number of steps that I need. And these two are estimated by fitting this to the observed data that I have. This is going to balance both sides of the equation. It's going to give me a little bit more number of steps than optimal. It's going to require a little bit more examples than optimal, but it balances the two sides. And you can think of this as optimizing the convergence rate without blowing up the number of steps that you need. There's another added point that I won't really belabor because this gets even more into the weeds. You should really read the critical batch size paper if you're interested into this, where one of their arguments is estimating this is quite complicated.

[46:48.690] You can do this by just looking at the ratio of the gradient covariance in the squared norm of your gradient. So why is that even in the scaling lecture? This is a thing that is not really related to scaling so far. The reason why this is in the scaling lecture is because, first of all, the critical batch size is a reasonable batch size to pick. That's a batch size that balances convergence rate and the size of the batch. So if I want to pick my batch size equal to my critical batch size, you might ask, how does this thing scale as a function of my target loss? Target loss can be thought of as compute. If I want to have a model that is better and better and better, what is the batch size that I should use? And the reason why this isn't the scaling law lecture is as you go down in scale in the loss, so your loss is improving, the amount of critical batch size increases.

[47:44.430] And it increases in this very predictable way, which is once again, a power law. So this is the relationship that they argue, which is roughly inverse polynomial in batch size. So given this, this is actually quite nice because what this is saying is if you have a really large-scale training run where you go all the way to the right of this plot, your batch sizes can be quite large. And that makes sense. As you get closer and closer and closer to the minimum, the value of variance reduction increases. Because you're trying to optimize finer and finer and finer grained objects, the noise matters a lot more once you're minimizing really, really tiny differences. So that's the batch size story. The other side-- so the two quantities, remember, that you're going to always have to tune is batch size and learning rate. The learning rate side of the story is also a little bit complicated. I'll go into this much more detail in the advanced scaling lecture.

[48:42.390] But learning rates generally shift. I think this is a good mental picture for how to think about learning rates in the standard world that you start with, which is, let's say, that you have just like a MLP. Just a standard neural network of some kind. I'm not going to do depth scaling. I'm only going to do width scaling. Now, if I'm doing width scaling, the bigger my model, the bigger my model, the smaller my learning rate should be. And the reason why my learning rate should be smaller is because I have bigger, more parameters, I'm changing more things at once. Maybe I should move less. It kind of makes sense. And in fact, there's a rule of thumb that maybe you want to scale by 1 over the width. That's a very commonly known rule of thumb that you want to just decrease the learning rate regularly as a function of your width, and that will give you a rule of thumb to scale your learning rate.

[49:32.340] There is another school of thought, or another set of tricks that people do, which is that people also rescale the network in various ways. They will change the initialization sizes, and they will change the step sizes of the optimizer for various parts of the network in order to force the model to have the same learning rate minimum across scales. And this is an idea that is called mu p and others like it. Some people have reported great success with these kinds of approaches, others have reported less success. The two ways, basically, of setting a learning rate for large-scale runs really is captured in this plot. In one way, you can try to estimate what these minima are and try to predict what your minimum will be. And the way in which the minimum changes is pretty predictable, so this is not a crazy idea. Or you can reparameterize your model to try to keep the learning rate minimum the same, and then just pick your best learning rate and go.

[50:31.390] Those are the two philosophies to picking your learning rate. Both of them have been applied successfully at scale. There are large-scale training runs that have used both approaches. I think anecdotally it does seem more people are maybe favoring the scaling law approach, but both are certainly viable approaches. I'll talk in detail about both of these next lecture. So before I get into the last part of this lecture, where I want to talk about not quite pitfalls but various details of scaling law things, one thing I want to end this section with is the idea that upstream and downstream performance can be quite different. So if you look at something like log likelihood, this is from the Yi Tay architectures paper, you find that perplexity and negative perplexity in this case parameters are very correlated. The more parameters you have, the better your model, very predictably.

[51:33.260] This is a very beautiful linear trend. Now you're like, OK, this is great. Now we're going to ship our best model. NL12. Turns out NL12 is not your best model. It was actually NL32XL, which was a much worse model in perplexity. This is a very common thing that does happen. Upstream and downstream are often correlated. This is probably one of the worst correlations that I've seen from upstream to downstream. But I think it's very, very important to remember that scaling laws, generally speaking, are objects that you want to apply on the perplexity side. They're very clean, they're very regular, they're very predictable. But transfer from perplexity to downstream is a lot less certain than it might initially seem. This is a place where you want to be very cautious. And as an anecdote, I have former students that have gone and done post-training of various places, and they always complain.

[52:20.380] They're like, "Oh, those pre-training people, they hand you this model. They are like, the perplexity is good. It's all your problem now." But it's often the problems have started at the pre-training side. So you don't want to be one of those people. You want to really think about transfer as well, not just focus on perplexities. OK, so maybe this is surprising, maybe this is not. But the point that I want to drill in here is that before you do your big training run, you should have a rough idea of what you're going to see. Not just is it going to train at all. In fact, if you do scaling laws, you should be able to actually predict fairly precisely the numerical values, almost, of how good your run is going to be. If I choose one optimizer versus the other, what will the gain be? You should be able to predict that. So the scaling law-based procedure is to say I'm going to train a bunch of smaller models, I'm going to establish some of a scaling law by fitting these lines onto log-log plots.

[53:14.210] And then if the fit is good enough, I'm going to have some confidence that the gap will persist, and then I'm just going to deploy that onto my large-scale training run. This is the naive version of a scaling law procedure, but I think one that is reasonably accurate for how you might think about these things in practice. What else are you going to do? Just deploy a run out of nowhere? Probably not. OK, I can pause here and take some questions before I get into the last part of this lecture. Yes. For each of the data points on the scaling law, how do you think about variance? How many runs do you do? And is aggregation usually like the average, or do you take min or max depending on [INAUDIBLE]? Yeah, I think in most cases, if not all cases that I've seen, those are singletons. And the reason why it's a singleton is because when you're looking at perplexities, perplexities are very, very clean.

[54:08.480] These objects, if you rerun them, they're probably like the variances in the second decimal place or something because the training data is very homogeneous, you've got a lot of it, your eval sets are very big. You've really pushed the variance very, very low on these things. But if you're doing something learning rate or critical batch size scaling laws, you will see some truly horrendous stuff. I do think people do some variance reduction, but it's actually not that common as common as it should be, maybe. Yes. Do people use just downstream metrics directly instead of using perplexity? Yeah, people do scaling laws directly on downstream metrics, but that has the same problem as this. So I think the way to think about scaling laws is to really reliably establish at least one thing. So the way to think about it is, let's say I see something like this, I have a strong belief that this transformer line is going to continue.

[55:09.710] This is a very regular trend. And it might curve a little bit, but we do think there's probably some regularity here. But instead, imagine this was a very noisy jaggedy line of this form. You would not really know what is going on. You would say, I maybe believe that it will go on, but maybe it's a line, maybe it's a curve. So I think the general-- if I'm trying to explain the philosophy of this stuff, is to say, OK, let's start with the low variance measurement and then establish scaling regularity there and then rely on some of belief about transfer or establish transfer or elsewhere to try to show that would transfer over to downstream. Yeah. So some of these are [INAUDIBLE], and some are just [INAUDIBLE] loss. When do you do either? I see, yes. You fit to test loss when you're doing good scientific practice, like Kaplan 2021. But realistically, all the models, except for a very few, like the infinite compute one and the repetition stuff, unless you're doing some of that, you're doing one-pass SGD.

[56:15.730] So the generalization gap for all of these models is very, very small. So train and val are very similar. I've seen some pre-training code bases that don't even have validation loss. Because if you're doing one pass, most likely your train and val are very, very close. That's why a lot of these plots exchange those two, because they're all in the one-pass regime. Good question, though. OK, all right, we will continue to move on to-- sorry, I moved on too far backwards here. OK, so now I'm going to talk about one of the most well-known uses of a scaling law. Even though you might not personally be using this, you've certainly used the rule of thumb at some point. And also, I think because this set of things will tell us something really interesting about the actual execution of a scaling law. OK, the motivating question is, do I want more data or bigger models?

[57:11.020] So the resource that you have, the resource that you have to spend, is compute. Someone's going to hand you metaphorically a certain amount of FLOPs. And you're going to take those FLOPs, and you're going to spend them somehow on-- if you remember Percy's very first lecture, it's going to be data times the parameters is roughly your FLOPs, linear in that. So do I want more data or more models? Well, we know that if we take a teeny tiny model and we dump tons of data into it, that will just be wasted. That's the dark purple line at the very top here. I've taken a teeny tiny language model, and I've passed a huge amount of data into it, and this thing has been flat for a very long time. This is a complete waste of compute. You would have much rather trained this big yellow model to the same amount of tokens or even earlier on, if you're having the same amount of compute. Now, if you could understand the interaction of how data amount and model size come together to give you performance, if you understood that relationship, you could optimize this very precisely.

[58:16.010] And so there are scaling laws that are more advanced and relate kind of joint behaviors of objects. So almost simultaneously, Kaplan and Rosenfeld both proposed these functional forms, which are roughly equivalent, describing how the size of the model and the amount of data relate to the error. Rosenfeld is very simple. It's really just the sum of two inverse terms. These are two scaling laws that are added together. And Kaplan is a little bit more complicated, but it's roughly the same idea. And if you look at, think about the limits, this is very intuitive. If I have an infinite amount of data, then what happens? Then this goes to 0. You're basically model-size limited. You become a pure model size scaling law. If I send my model size to infinity, then I'm going to be bounded by data. I'm going to have a pure data scaling law.

[59:10.000] So it makes sense if you take the limits of both of these. It's often a good idea if someone tells you a scaling law to take the limits of all the variables to understand the behavior of the system. Now, the thing that's kind of cool is Rosenfeld and many others that have proposed these kinds of joint scaling laws show that these things fit quite well, even if you extrapolate. So what you're going to do is you're going to fit on these green points, which is a low compute regime and low data regime. You never see things that have either high data or high model sizes. And then you can extrapolate on the basis of these tiny runs in this corner over here to essentially this high compute or high data, high model size regime. And you can very accurately predict the error based on your predictions. Great predictions. And so once you have that, now what you can do is to say, well, I have a fixed amount of FLOPs subject to my FLOPs constraints, minimize this. That's a simple nonlinear optimization problem.

[01:00:09.830] You can do that to solve for this. Now, both Kaplan and Rosenfeld provide equations of this form. What do they say? Well, if you look at Kaplan, and they solve for this equation, they say something like this. Well, the amount of-- they always flip this. It confuses me. The amount of data it should be C to the 0.73, and the amount of model size should be C to the 0.27. Is that reversed? I think it's reversed, sorry. N should be the amount of parameters and D should be the amount of data. And so tokens per parameters decreases with C. So as you increase the compute amounts, this prescription by Kaplan tells you to train bigger and bigger and bigger models. And so if you were around in the days of GPT-3, it kind of shocks me to think that there are people who were not, you would know that there was a period where everyone was training these gigantic models.

[01:01:06.730] Like hundreds of billions of parameters of models, trillion-parameter dense models, what have you. And part of that was driven by this. They saw GPT-3, they saw these Kaplan predictions, and they said, "Wow, I'm going to do my big training runs on these gigantic model size models." But a few years later, in 2022, some folks at DeepMind, Hoffman et al., came in and they said, well, actually, these predictions are all terribly off. You've been training these giant models, these three stars over here, but actually, those are way too big. What we should be training instead is a very different trend, this line on the blue over here, and we should be training this teal star, which is a much better model.

[01:01:48.640] So they argued that we should be training much smaller models relative to this. And I think, of course, many of you or all of you probably the Chinchilla paper, you know what the token multipliers are. It's 20 tokens to every parameter. But I think this is useful to go in detail because the Chinchilla-Kaplan disagreement and all of this tells us something important about how we should be thinking about scaling laws. It's not just something where you turn the crank, and you get the answer. It's something that's actually careful and sensitive, and we have to respect the process. So I'm going to walk through the Chinchilla paper. And after I walk through the Chinchilla paper, I'm going to talk about why the Kaplan and Chinchilla

[01:02:27.910] papers disagree and what that means for you. So the Chinchilla author suggests three different ways of fitting scaling laws. I really like this because it's a way of robustifying yourself to modeling assumptions you may have made. And each of these actually is quite a different approach to estimating the trade-off between model size and data. And minus method 3, which does differ a little bit, they mostly agree. All 1 and 2 both agree that the coefficients for data and for model size are roughly the same. 0.5, 0.5, 0.49, 0.51. Method 3 does differ, 0.46 and 0.54, but we'll talk about why later. Kaplan et al., of course, have a very big difference between the two, which is why they were training big models at big, big compute sizes. Method number 1 from Chinchilla is what I might call the lower envelope method. And this is the method that was also being used partially by Kaplan to try to figure out what the right scaling was.

[01:03:29.500] And so what you do is you take all of your training runs. So each one of these colored lines is a training curve. And what you want to do is you want to actually take the lower envelope of these training curves. Because a lower envelope point on this training curve, that means that for a particular FLOP, this is the best loss that I ever achieved for that FLOP on any training run. So I can take all of the lower envelope points at the very bottom here at the high compute regime, and then I can plot all those points to see how big were the models that each of these FLOP points intersected. Because each of these points is a training run with a model size. And if I scatter plot that, I will get a nice linear trend. And that will give me one way of estimating the optimal trade-off between FLOP to parameters. And that gives one answer, which is for their compute budget, they should train a 67 billion parameter model.

[01:04:19.580] This is a nice and simple way, but very tricky defining things like what is the lower envelope, and so on and so forth. There's definitely some tricky parts. Method 2 is IsoFLOP. This is now very popular. It's also very easy and very robust. This is probably my personal favorite method. And what you do is, for this, you pick a bunch of different FLOPS budgets. So I have a different range of FLOPS budgets going in factors of 3 in this case or 1, 3, 6. And what you do is for each FLOP budget, you're going to essentially sweep the space. So here I have one variable, which is the parameter-to-data trade-off. So for each FLOP budget, I'm going to sweep over the parameter to data trade-off, fixing my FLOPS. So I double my data size, I half my model size, so on and so forth. And that will generate a sequence of training runs whose terminal loss is going to sweep out some of curve.

[01:05:11.040] So each one of these colored sets of points is a fixed FLOP run. Each point is sweeping across different parameters, and of course, it's implicitly sweeping over data as well. Now, the nice thing now is I can take the minimum over each of these runs, or even fit a quadratic and take the minimum over the quadratics, and then I can draw a line through all of the bottoms of the quadratics, or take the bottom of the quadratics and plot them as a function of FLOPS to parameters. And that will give me another prediction, which agrees very closely. These is $63 billion parameters for my compute budget. Now, the final approach that we can do is actually the maybe most natural one, as suggested by the Rosenfeld paper as well as the Kaplan paper, because both of them are propose a hypothesized functional form for how model size and data relate to loss. So if you have such a joint functional form, what you can do is you can take a whole bunch of training runs that you've done, and you can fit your loss landscape, which is your hypothesized functional form, and you can just do curve fitting

[01:06:19.560] to estimate what the coefficients are in your scaling law. This is the most brute-force way. You have a hypothesized relationship between loss, data size, and model size train a bunch of models, do curve fitting. Hopefully, fairly straightforward-- how you do that. Of course, if you're doing this method, how you do the curve fitting is very important. Now, you're fitting these surfaces, there's many variables, it's kind of tricky. This thing also has some trickiness, just like method 1. So it's not a straightforward way the same way that maybe FLOPS is. So now that we've gone through what Chinchilla does, it doesn't seem materially different than what Kaplan was doing. It's not like Kaplan was doing one crazy thing and then the Chinchilla authors did something way smarter and you said, "Wow, why didn't we do that from the start?" They both did pretty reasonable stuff.

[01:07:10.520] But if we look at their scaling predictions, they are very different. And in hindsight, we generally tend to agree with the fact that the Chinchilla authors had more of the correct scaling. If you do your own Chinchilla-style analysis, you will find probably scaling is much closer to Chinchilla. Now, so why is there such a big difference when they're both fitting these joint scaling laws? OK, so this is, I think, where we get into the messy realities of how scaling laws are made. Scaling laws can really change depending on the precise ways in which you're implementing the scaling laws, the hyperparameter settings that you're using, and even what the x-axis is.

[01:07:53.870] So there's a really nice paper. Oh, no, I covered the authors. I can paste this in the lecture Slack later. --called "Resolving Discrepancy to Compute-Optimal Scaling of Language Models." Yair is the last author. He was a former student here. And they show-- basically, first they replicate the Kaplan result using roughly the Kaplan settings. And then they say, OK, well, let's change how we count parameters. That shifts the curve a little bit. And then they say, OK, well, let's change how learning rate warmup is done. That shifts the curve a little bit. And then let's tune the optimizer a little bit more.

[01:08:29.090] And then they get exactly the Chinchilla result. So their argument here is that there is a sequence of what seems very minor decisions that in the end, gives you this big gap between the Kaplan scaling law and the Chinchilla scaling law. And I'll explain what each of these steps is, not that that's the most important single thing, but also to emphasize to you how important the details are here. So the very first arrow, where we go from Kaplan to the blue line, that is how we count parameters. So remember, in Kaplan, I said, well, they saw this weird stuff if they included embedding parameters. So they excluded embedding parameters. That's generally an OK thing to do. But one thing they also did that really messed them up was they also excluded the last layer parameter counts, the softmax ones. Because in many models, the embedding and the last layer linear are dual to each other.

[01:09:22.580] They have the same shapes. The inside has vocab size by hidden dim. The output matrix has hidden dim by vocab size. And because they're the same shape, they said, well, let's just exclude both of those. It turns out whether you include or exclude those has a big material impact on the shape of the scaling law. They also found-- this is much more of an oversight, more than a genuine difference in opinion, that a lot of the models that were being trained in Kaplan were very small. And in fact, they were so small that they were not actually converging by the time that the learning rate warm up was done. So their learning rates were set very suboptimally. And because of that, their models weren't fully converged. And then finally, they found that Kaplan et al. was fixing one big batch size, and those big batch sizes were suboptimal for the smaller models.

[01:10:11.670] And therefore, if you tune that correctly and you use varying batch sizes, you end up getting this thing that agrees exactly with Chinchilla. So I think each of these might seem like a very minor difference, but if you change up these really minor differences, you can get big shifts in the scaling law. And one thing I like to emphasize is that scaling laws are lower bounds in some sense. They're kind of saying, if I continue this recipe and I scale it up, then this is what I will get. But if you're scaling up a recipe that you don't want to scale up, your warmup is kind of crazy, or your batch sizes are crazy, you're going to get bad scaling laws. So you want to be as close to the proper full run as possible. There's another paper that does another very interesting thing to try to study the difference between Kaplan and Chinchilla scaling laws. It's by Pearce and Song. And what they do, and I think this is very clever, is they don't train any models.

[01:11:07.620] They just take the Chinchilla training curves or the implied functional form of the Chinchilla training curves. They just plot them out and they say, well, based on these implied training curves, which are simulated purely from Chinchilla, they generate what the implied training curves will be for Kaplan if you ignore the embedding parameters and you scale way down, and what happens if you use the total parameter count instead. And what they find is really two effects. One is that Kaplan is operating at a much lower compute scale, so they're very sensitive to small changes, and actually ignoring the non-embedding parameter is a sufficiently big change to mess up the result relative to Chinchilla. So their argument is slightly different from Yair and others, which is that here, actually, it's the size of the compute scale plus the slight non-linearity induced by non embedding parameters that's causing this disagreement. So this is really starting to get into the weeds.

[01:12:01.580] But I hope you appreciate the importance of paying attention to detail. And I think a lot of the work that goes on in establishing, relying, scaling laws is about figuring out what the right regime is so that you can get reliable scaling across compute scales. The final thing, and this is really just more of something that I just find amusing more than anything else, which is that Chinchilla is this paper that takes down the Kaplan paper in a way. It's like, oh, you guys were very wrong. But there was this one remaining mystery, which is that Chinchilla method three never agreed with Chinchilla method 1 and 2. And the numbers look small. And the authors are very unperturbed about this, but I think the implications are quite non-trivial. If you look at this, methods 1 and 2 basically say the amount of parameters and tokens should scale together. So there should be a fixed ratio between the two.

[01:12:50.540] This is where the Chinchilla factor of 20 comes from. If you look at number 3, the exponents are different. This means if you scale up your compute, eventually, actually, in this case, you're going to have way more tokens than you are going to have parameters. So it's going to be actually a very different scaling conclusion asymptotically than the other ones. And so it turns out that I was certainly not the only one who was perturbed about this, except some folks at Epoch AI did something about being perturbed. They couldn't get either the raw data or the code. So what they did was they extracted the data from plots in the paper, and they refitted the method 3 regressions. They extracted the data from the paper. And given that data, they fitted that 2D surface or 3D surface that was proposed.

[01:13:37.710] And then, what they showed is actually the original paper must have underfitted the original data. And if they redid the curve fitting, they could get much lower losses relative to what the proposed scaling law in the Chinchilla paper. And in fact, if you follow the refitted model, you got almost exactly D over 20 is 20 rule of thumb from the Chinchilla paper, which is really a funny conclusion, I think, to the Chinchilla paper, where in the end it turns out that the authors were more right than they knew. Actually, they had made a mistake, and that was the only reason why method 3 disagreed from methods 1 and 2. So it's a nice conclusion to know that the method was robust after all. OK, so that's the end of the Chinchilla saga in some ways. But I will end this section by saying you probably don't want the Chinchilla factor. At this point, I think this is a pretty common knowledge thing.

[01:14:35.220] But if you're training a production model, you don't really care about saving training compute for the most part. You can look at all sorts of external surveys that people have done of how Frontier Labs are spending their compute. Most of the compute is not going into training runs. Most of the compute is actually going into R&D and serving. And given that, especially for serving, what you want to do is you want small models that are capable. You do not want big, bloated models that cost a lot to serve, even if that minimizes training cost. And so what you really want is something that you would call as "overtrained." I put that in quotes because, really, overtraining is what we want. That's the right amount of training. And if you look at all the models that were released in the early days, like GPT-3-- this was under-trained for sure, three tokens per parameter.

[01:15:21.830] Chinchilla gets to the modern ratio of 20 per parameter. And then for a while, people were at 20 per parameter. And this is the era, I think, where there wasn't that much serving of these models. The models were cool, but it wasn't being served at scale. And then as you start to move to eras where serving becomes very real, now you're overtraining these models, and now you've moved into MoEs, and you're making all these trade-offs to optimize inference serving. So I think Chinchilla is important, not because I think the 20-to-1 ratio is the one golden ratio. If you're doing stuff for research, it might be. But I think the reason why it's important is because it tells us quite a bit about how we should be fitting scaling laws and how to think about scaling laws. OK, one final thing I'll end with Chinchilla-wise and otherwise for scaling is IsoFLOP have been one of the things that, I think, has a research tool have endured.

[01:16:12.290] IsoFLOP are very easy to execute. You fix a FLOPS budget, and then you sweep over all of the other degrees of freedom to see what the surface looks like. And this gives you, in general, fairly reliable readings of how different parameters vary as a function of your free parameters. People have done this for-- or actually, one of my students did this for diffusion models. The MoE study I talked about was a IsoFLOP-style design. If you're ever in a situation where you're thinking, how am I going to decide all these trade-offs, IsoFLOP is always a good default. OK, so to wrap up everything for today, there's this idea that we have this log-linear regularity between the amount of resources we put in and the performance we get out. And it extends to a wide variety of things, model parameters, compute, sparsity level of your MoE.

[01:17:05.880] And that lets you do a lot of the things that I was talking about earlier, as arbitrary choices, but do it in a much more evidence-driven way, hopefully. And so this allows us to have engineering at scale without explicitly relying on doing big model training runs. And hopefully, you remember, I think all the different components here. Data scaling is this very natural object, just the exponent is a little bit strange. Model scaling allows you to do all these cool engineering things. And then finally, scaling is this very cool, robust predictor, let's call it, for how to try to do engineering at a large scale. And thanks. The next lecture, I think, will be Percy on inference. But after that, we will return to more advanced niche scaling topics. [APPLAUSE]

Scaling laws

课程简介 ​

CS336 Lecture 9：Scaling Laws 中文讲义 ​

学习目标 ​

课程主线 ​

核心概念与原理 ​

1. 背景：缩放定律的历史根源 ​

2. 数据 Scaling Laws ​

2.1 经验观察 ​

2.2 幂律的统计直觉来源 ​

2.3 数据缩放定律的工程应用 ​

3. 模型与超参数层面的 Scaling Laws ​

3.1 架构比较 ​

3.2 优化器选择 ​

3.3 宽高比与深度 ​

3.4 参数计数的影响 ​

4. 混合专家模型（MoE）的 Scaling ​

5. 临界批次大小（Critical Batch Size） ​

5.1 概念 ​

5.2 估计方法（McCandless / OpenAI 的方法） ​

5.3 与计算规模的幂律关系 ​

6. 学习率的缩放 ​

7. 上游与下游性能的差异 ​

关键公式与原则 ​

课程作业关联 ​

常见误区 ​

复习清单 ​

CS336 Lecture 9 深度解读：缩放定律——从经验规律到大规模训练的工程基石 ​

导语 ​

从理论到工程：为什么我们需要缩放定律 ​

缩放定律的历史：不是新鲜事的"新范式" ​

数据缩放定律：最简单、最自然的起点 ​

幂律关系的自然涌现 ​

数据缩放定律的工程应用 ​

一个令人警醒的坐标轴陷阱 ​

模型缩放定律：架构、优化器与超参数的量化权衡 ​

架构选择的缩放视角 ​

优化器与超参数的缩放分析 ​

一个重大陷阱：并非所有参数都是平等的 ​

混合专家模型（MoE）中的参数价值 ​

临界批次大小与学习率：真正需要重新设计的两个量 ​

临界批次大小：噪声受限到偏差受限的转折点 ​

学习率的两种哲学 ​

上游与下游的断裂 ​

Chinchilla 之争：为什么同样的方法会得出截然不同的结论？ ​

两种对立的联合缩放定律 ​

Chinchilla的三种拟合方法 ​

为什么会有分歧？ ​

你真的想要Chinchilla系数吗？ ​

缩放定律作为工程范式：总结与反思 ​

本讲在课程中的位置 ​

CS336 2026 Lecture 9：完整中文逐字稿 ​

正文 ​

CS336 2026 · Lecture 9 · Scaling laws ​