CS336 2026 · Lecture 1已发布

Overview, tokenization

日期: Mon March 30
讲师: Percy
课程资料: 8 项
内容产物: 4 项

课程简介

本讲从“如何在固定资源下训练出更好的语言模型”出发，串起数据、模型架构、系统、Scaling Laws 与对齐等课程主线。后半部分聚焦文本表示：比较字符、字节、词级切分的局限，并通过 BPE 理解词表大小、压缩率、可逆性和多语言处理之间的取舍。

课程资料lecture_01.py打开链接↗论文Neural Language Models are Effective Priors for Scene Text Recognition课程引用↗论文SentencePiece - A simple and language independent subword tokenizer推荐阅读↗代码OpenAI tiktoken实践工具↗论文Neural Machine Translation of Rare Words with Subword UnitsBPE 经典论文↗论文Language Models are Few-Shot Learners模型规模背景↗文档UTF-8 Standard文本编码↗视频课程视频课程视频↗

Assignment 1: Basics已发布

CS336 语言模型从零开始 · 第一讲笔记

学习目标

完成本讲后，你应该能够：

理解 CS336 课程的核心理念：通过从零构建来理解语言模型
区分可跨规模迁移的知识（机制、思维方式）与可能无法迁移的知识（直觉）
正确理解“苦涩的教训”的含义，并掌握“准确率 = 效率 × 资源”的框架
了解语言模型从 N-gram 到现代大模型的历史发展脉络
理解分词（tokenization）在语言模型中的作用，以及各类分词方法的优劣
掌握 Byte-Pair Encoding (BPE) 算法的完整流程，包括训练和推理
了解课程五个作业的定位和核心内容

课程主线

本讲分为两部分：

课程导论：为什么开设这门课、能学到什么、课程结构
第一个技术单元：分词（Tokenization）

贯穿全课的主线是效率（efficiency）：在给定的数据和计算预算下，如何构建最佳模型。无论是系统优化、数据过滤、还是模型架构设计，都可以从效率角度来理解。

第一部分：课程导论

1.1 为什么开设这门课

核心问题：研究人员正在与底层技术脱节。

2016 年：研究人员自己实现并训练模型
2018 年：下载预训练模型（如 BERT）进行微调
现在：直接通过 API 提示（prompt）模型

虽然提升抽象层次能提高生产力，但这些抽象是有漏洞的（leaky）。当你想做基础研究时，仅靠提示模型会极大限制你探索的设计空间。理解语言模型如何工作的方式是通过构建（understanding via building）。

1.2 工业化带来的挑战

前沿模型（frontier models）极其昂贵且细节不公开：

2023 年 GPT-4 据称训练花费约 1 亿美元
GPT-4 技术报告明确表示不分享构建细节
小规模模型可能无法代表前沿模型的行为

两个例子说明规模差异：

FLOPs 分配随规模变化：小规模时注意力层约占 44% FLOPs，大规模（175B 参数）时 MLP 层升至约 80%
涌现行为（emergence）：小模型上 0-shot/few-shot 学习几乎无效，达到临界规模后才突然大幅提升

1.3 可迁移的知识类型

知识类型	内容	能否跨规模迁移
机制（Mechanics）	Transformer 是什么、模型并行如何工作	✅ 可以
思维方式（Mindset）	如何最大化硬件利用、认真对待扩展	✅ 可以
直觉（Intuitions）	哪些数据/建模决策带来好性能	⚠️ 不一定

关于直觉：有些设计决策无法从理论上论证。SwiGLU 激活函数的提出者 Noam Shazeer 在论文中直言：“我们不提供解释……将成功归因于上天的恩赐。”

1.4 苦涩的教训（The Bitter Lesson）

错误理解：规模就是一切，算法不重要
正确理解：能够良好扩展的算法才是重要的

核心公式：

准确率 = 效率 × 资源

效率在大规模下更重要——小规模实验运行时间翻倍可以等待，但大规模运行涉及数亿美元成本，效率提升即使只有 5% 也是巨大进步。经验数据：2012-2019 年 ImageNet 上算法效率提升约 44 倍。

课程框架：在给定数据和计算预算下，什么是最佳模型？即最大化效率。

1.5 语言模型历史脉络

前神经时代：Shannon（1950s）用语言模型测英语熵；N-gram 模型用于机器翻译和语音识别。

神经时代的关键组件：LSTM（1997）、首个神经语言模型（Bengio，2003）、Seq2Seq（2014）、Adam 优化器（2014）、注意力机制（2015）、Transformer（2017）、混合专家（2017）、模型并行（2018-2019）。

基础模型崛起：ELMo、BERT（预训练 + 微调范式）、T5（文本到文本统一框架）。

规模化的拥抱：GPT-2（1.5B，流畅文本）、缩放定律（Kaplan 2020）、GPT-3（175B，上下文学习）、PaLM（540B，训练不足）、Chinchilla（70B，计算最优缩放）。

开源生态：EleutherAI（GPT-J）、Meta OPT（175B，硬件问题多）、BigScience BLOOM（176B）——这些早期复制品都不强。此后 Meta 的 Llama 系列引领潮流，加上 Mistral、DeepSeek、Qwen 等，开源权重模型已接近闭源模型。更进一步的开放包括 AI2、NVIDIA 参与的 MARIN 项目，提供权重 + 论文 + 代码 + 数据。

1.6 语言模型定位的演变

2018（BERT）：用于微调的东西
2020（GPT-3）：用于提示的东西
2022（ChatGPT）：可以对话的东西
2026（智能体）：可以自主行动的东西

基本面未变：仍依赖 GPU 和内核、梯度方法、Transformer 和注意力。变化在于规格：需要更长上下文，推理效率更关键。

第二部分：分词（Tokenization）

2.1 基本概念

分词器（Tokenizer） 在原始输入（字节）和整数序列（token）之间转换：

encode(string) -> list[int]：将字符串编码为 token 索引
decode(list[int]) -> string：将 token 索引解码回字符串

往返性质（roundtrip）：decode(encode(s)) == s 必须成立。若实现的分词器不能往返转换，则存在问题。

压缩率 = 字节数 / token 数。压缩率越高，序列越短，对注意力机制（二次复杂度）越有利。

2.2 朴素分词方法的缺陷

字符级分词器（CharacterTokenizer）：

每个 Unicode 码点作为 token
约 15 万个 Unicode 字符 → 词汇表过大，许多字符罕见（如 🌍），利用率低
压缩率差

字节级分词器（ByteTokenizer）：

按 UTF-8 编码后的字节划分
词汇表仅 256，但压缩率为 1，序列过长
无法利用注意力机制的上下文窗口

词级分词器（WordTokenizer）：

按正则表达式（如 \w+）切分
每个 token 有语义，压缩率不错，但词汇表可能无界
遇到未见过词需使用 UNK token，会干扰困惑度计算

2.3 BPE 算法详解

背景：BPE（Byte Pair Encoding）由 Philip Gage 于 1994 年为数据压缩提出，2016 年被引入 NLP 用于神经机器翻译，GPT-2 首次将其用于大语言模型。

核心思想：基于训练数据构建词表，常见的字节序列合并为一个 token，罕见序列分解为多个 token。

训练算法（以 "the cat in the hat" 为例）：

将字符串编码为字节序列作为初始 token
统计所有相邻 token 对的出现次数
找到出现频率最高的对
将该对合并为新 token（编号从 256 开始递增）
更新序列和词表
重复直至达到预设的合并次数

示例过程：

初始：t,h,e, ,c,a,t,...
第 1 次合并：t,h 出现 2 次 → 合并为 256
第 2 次合并：256,e → 合并为 257
第 3 次合并：257, → 合并为 258

编码（encode）：对新文本，按训练好的合并顺序依次应用合并操作。

解码（decode）：按 token 的字节值拼接后解码为 UTF-8 字符串。

2.4 实际 BPE 的优化方向

讲义中的实现是完整但非常慢的版本，作业 1 中需要优化：

稀疏合并遍历：不应遍历所有合并操作，只遍历对当前序列有效的合并
特殊 token 处理：如 <|endoftext|> 需要检测并保留
预分词：使用 GPT-2 的正则表达式先切分文本块，再对各块应用 BPE
实现语言：Python 不够快时，可以考虑 Rust/C 等

2.5 有效的分词器应满足的性质

即使未来摆脱分词，任何替代方案也应满足：

模型应在某种序列抽象（chunk）上操作：不仅文本，视频、DNA 序列等领域的单字节信噪比很低，需要提升建模层次
块的大小应可变：支持自适应计算，将更多模型容量分配给输入中更有趣的部分

关键原则与公式

效率框架

最佳模型 = f(数据预算, 计算预算)
最大化效率 = 给定资源下取得最好结果

缩放经验法则

Chinchilla 缩放定律（将在作业 3 中深入）：

计算最优时，训练 token 数 ≈ 参数量的 20 倍
例如：70B 参数模型 ≈ 1.4T token
注意：未考虑推理成本，实际中常使用比计算最优更小的模型

训练稳定性原则

平衡表达力（expressivity）、稳定性（stability）、效率（efficiency）
参数和梯度范数应处于“金发姑娘区”（既非爆炸也非消失）

课程作业概览

作业	主题	核心内容
1	基础	实现 BPE 分词器、Transformer、损失函数、优化器、训练循环；资源核算；在 TinyStories 和 OpenWebText 上训练
2	系统	Triton 实现融合内核、分布式数据并行、优化器状态分片；基准测试和性能分析
3	缩放定律	定义训练 API（超参数→损失），拟合缩放定律并外推预测
4	数据	从 Common Crawl 原始 HTML 开始，完成转换、过滤、去重
5	对齐	实现 DPO 和 GRPO

作业特点：不提供脚手架代码，但提供单元测试；可在本地验证正确性，在集群上做实际训练和基准测试；多数作业有排行榜。

核心术语表

术语	含义
Tokenizer	在字符串和整数序列（token）之间转换的组件
BPE	字节对编码，数据驱动的分词方法
Compression ratio	压缩率，字节数/token 数
FLOPs	浮点运算次数，衡量计算量
Scaling laws	缩放定律，预测不同规模下的损失
MoE	混合专家（Mixture of Experts）
Prefill / Decode	推理的两个阶段：预填充和解码
Roofline analysis	屋顶线分析，判断计算受限还是内存受限

常见误区

“苦涩的教训 = 规模至上”：这是错误解读。正确理解是：能够良好扩展的算法才重要。
“缩放定律是自然法则”：缩放定律不会自动实现，必须通过精心构建缩放方案（scaling recipe）才能获得可预测的外推。
“小规模上的直觉可以迁移到大规模”：注意力和 MLP 层的 FLOPs 占比随规模变化，涌现行为在小规模上不可见——某些直觉无法跨规模迁移。
“分词器只是细节问题”：分词影响压缩率（即序列长度，直接影响注意力计算量）和自适应计算能力，是效率优化的重要环节。
“推理时用更大的模型更准”：实际中常训练小于计算最优的模型，因为推理成本会纳入考量。

复习清单

概念理解

[ ] 本课程“从零开始”理念的含义和边界
[ ] 三类知识（机制、思维方式、直觉）的跨规模可迁移性
[ ] 为什么小规模模型不能完全代表前沿模型的行为
[ ] “准确率 = 效率 × 资源”的推导逻辑
[ ] 分词器在语言模型中的功能定位

BPE 算法

[ ] 字符级、字节级、词级分词各自的缺陷
[ ] BPE 训练流程：统计配对 → 找最频繁 → 合并 → 更新词表
[ ] 编码和解码的实现思路
[ ] 压缩率计算：字节数 / token 数
[ ] 有效分词应满足的两个性质

效率思维

[ ] 从效率角度理解：分词、架构设计、数据过滤、缩放定律
[ ] 为什么预测性（predictability）至少与最优性（optimality）同等重要

术语与公式

[ ] 能解释以下术语：token、vocab size、merges、pre-tokenization
[ ] 记住计算最优缩放的经验法则：D ≈ 20N
[ ] 知道训练 FLOPs 的粗算公式：6 × 参数 × token 数

从零构建语言模型：CS336 第一讲深度解析

导语：为什么要从零开始？

当我们打开 ChatGPT 或其他大语言模型时，很容易把它当作一个黑盒——输入提示词，得到答案。但在当今 AI 研究中，有一个日益严重的问题：研究人员正在与底层技术脱节。十年前，所有 AI 研究人员都会自己实现并训练模型；八年前，人们还会下载 BERT 这样的预训练模型进行微调。而今天，很多人仅仅通过提示（prompting）模型就能应付过去。

CS336 课程的核心信念是：只有通过从底层一步步构建一切，你才能真正理解所有东西是如何运作的。这正是本课程设计的基本出发点。

课程背景：为什么基础研究需要底层技术

抽象是有漏洞的

提示模型本身并没有问题，从抽象层次提升上看，这是一件很好的事情。但抽象是有漏洞的（leaky）——当模型无法完成你需要的任务时，没有其他补救办法。如果你真正对基础研究感兴趣，仅仅通过提示模型，你就在极大地限制你的设计空间。要从事基础研究，你需要拆开整个技术栈（stack）。

工业化带来的挑战

语言模型的工业化已经深刻改变了研究格局。前沿模型（frontier models）极其昂贵——GPT-4 的训练据说花费了 1 亿美元（2023 年），而现在的成本可能已经达到 10 亿美元量级。更关键的是，这些模型的具体构建方式没有任何公开细节，GPT-4 的论文明确表示不会分享任何相关信息。

当无法研究前沿模型时，我们可以构建小型语言模型。但必须清醒认识到：小模型可能并不能代表真正的前沿模型。

两个关键例子

例子一：FLOPs 分布随规模变化。如果我们计算不同模型规模下 MLP 层消耗的浮点运算数（FLOPs）占比，会发现一个显著变化：在 1.75 亿参数规模时，MLP 层占比约 44%；当扩展到 1750 亿参数规模时，这个比例会上升到 80%。这意味着小规模研究中的注意力优化可能在大规模下没有同样的收益。

例子二：行为的涌现性。即使在比较早的时期，小模型在零样本（0-shot）或少样本（few-shot）学习上的表现也几乎看不出与随机猜测的区别，只有达到某个临界规模时，能力才会突然大幅提升。

三类可迁移的知识

尽管存在上述挑战，我们依然可以从这门课中学到三类知识：

机制（Mechanics）：事物如何运作，什么是 Transformer，模型并行（model parallelism）如何工作——这些可以完整传递。
思维方式（Mindset）：如何着手构建语言模型，如何最大化利用硬件资源，如何认真对待扩展（scaling）——这些也可以传递。
直觉（Intuitions）：哪些数据和建模决策会带来好的性能——这些不一定能跨规模迁移。

关于直觉，有一个极其坦诚的例子：提出 SwiGLU 激活函数的论文作者 Noam Shazeer 在结论中写道：“我们不做解释。我们将这些架构的成功，归因于上天的恩赐。”这说明有些设计决策纯粹来自实验经验，无法从理论上完全论证。

苦涩的教训的正确解读

关于“苦涩的教训”，常见的误解是“规模就是一切，算法不重要”。但正确的解读恰恰相反：能够良好扩展的算法才是重要的。可以这样理解：

准确率 = 效率 × 资源

效率是产出除以投入，资源就是投入。当规模变大时，效率的重要性远超过资源——如果你在小规模实验时运行时间翻倍，或许只需多等一倍时间；但如果在大规模上进行，那可能涉及数亿美元的成本，即使仅 5% 的效率提升也是巨大进步。OpenAI 2020 年的一篇论文显示，2012 年到 2019 年间，ImageNet 上的算法效率有 44 倍的提升。算法与硬件的双重改进相互叠加，才带来了我们今天看到的巨大飞跃。

语言模型的演进：从数据压缩到智能体

早年：从 N-gram 到神经网络

语言模型有着悠久的历史。早在 50 年代，Shannon 就使用语言模型来测量英语的熵。在很长一段时间里，N-gram 模型被用于机器翻译和语音识别系统——它们不是系统全部，但确保了输出文本的流畅性。

现代语言模型的血统来自神经架构。90 年代出现了 LSTM；2003 年 Yoshua Bengio 在《神经网络》期刊上发表了第一个神经语言模型（一个查看小范围上下文的的前馈网络）；随后，Seq2Seq 建模、注意力机制（attention，为机器翻译开发）、Transformer 架构（也为机器翻译设计）相继出现。2010 年代涌现了混合专家（mixture of experts）模型、模型并行等多种架构、系统和优化器思想。

2010 年代末：预训练时代的开启

ELMo 和 BERT 标志着范式转变：在大量文本上预训练语言模型，然后在特定下游任务（如问答）上微调，会带来巨大改进。当时的标准流程是：取一个预训练模型，在特定任务上微调。

Google 的一篇论文预示了“提示输入、响应输出”的观点，但真正打开闸门的是 OpenAI——他们系统性地拥抱了规模化理念。2018 年左右发布 GPT，然后扩展到 GPT-2，2019 年训练出 GPT-3（1750 亿参数，比当时最大模型大了近 10 倍），展现出诸如上下文学习这样的涌现行为。

此时发生了三件值得一提的事：Google 训练了巨大的 PaLM 模型但训练不足；尚未与 Google 整合的 DeepMind 已经找到了计算最优规模法则；GPT-3 的发布对许多人来说是一个警钟。

开源模型生态的兴起

GPT-3 之后出现了一系列复现尝试：EleutherAI 创建了开源数据集（The Pile）和模型（GPT-J）；Meta 训练了 1750 亿参数的 OPT 模型但遭遇大量硬件问题；Hugging Face 的 BigScience 项目推出了 BLOOM。这些早期模型都不够强大。

过去三年中，开源模型生态系统发生了显著变化。Meta 的 Llama 系列引领潮流（Llama、Llama 2、Llama 3），Mistral、DeepSeek、Qwen 等一批中国模型也纷纷加入。现在的开源权重模型已经接近闭源模型，在工业界被广泛使用。除此之外，还有一条更彻底的开源路线——AI2、NVIDIA 和 Marin 项目不仅提供权重，还提供论文、代码和数据，让研究人员能深入理解模型的构建过程。

这些开源生态的存在意义重大：如果没有它们，这门课可能根本无法开设——正是来自这些开放模型的论文让我们能窥见前沿模型是如何构建的。

语言模型定义的演变

在过去十年中，语言模型的概念发生了深刻变化：

2018 年（BERT 时代）：一个需要微调的东西
2020 年（GPT-3 时代）：一个需要提示词（prompt）的东西
2022 年（ChatGPT 时代）：一个可以与它对话的东西
2026 年（智能体时代）：一个能够自主行动的东西

尽管要求不断提高，但基本原理并没有改变：我们依赖 GPU 和内核，使用梯度或类随机梯度方法优化，仍然用 Transformer 和注意力机制。变化的只是规格——需要更长上下文意味着推理效率更加重要。

课程结构：五个模块五个作业

课程分为五个部分，与五个作业对应：基础系统、系统、扩展、数据、对齐。

作业一：基础系统

目标是能够从零开始训练并构建一个语言模型。组成部分包括：

分词（Tokenization）：模型操作的原子单位是什么？分词器在原始输入（字节）和代表标记的整数序列之间转换。从效率角度看，分词能将长序列减少为更少标记——不仅因为注意力机制是二次方复杂度的，更因为这允许自适应计算：也许有些地方包含很多字节但应压缩为一个标记，而输入中一些更稀有或更有趣的部分则应保留为多个标记。

模型架构（Transformers）：你可能会见到很多重要的改进：激活函数的演变（ReLU 到 SwiGLU）、位置编码的演变（正弦到 RoPE）、归一化方式的演变（LayerNorm 到 RMSNorm、QK-norm、pre-norm 与 post-norm）、注意力计算的简化（稀疏注意力、GQA、MLA）、MLP 层的混合专家（Mixture of Experts，已成为构建计算高效 Transformer 的主导范式）。

模型形状：有多少层？多少个注意力头？隐藏维度是多少？专家数量是多少？这看起来只是琐碎的超参数设置，但在扩展和语言模型背景下，其实有着非常重大的影响。

训练：损失函数（默认下一标记预测，但预测多于一个标记似乎有帮助）、优化器（AdamW 仍常用，但 Muon 等新优化器在最新的开源模型中越来越流行）、初始化（对更大模型的训练稳定性有巨大影响）、学习率调度、正则化、批量大小，以及 MoE 特有的问题。这些看起来只是超参数，但真正以原则性的方式设置它们，将决定一次运行是彻底失败还是达到最先进的水平。

在作业一中，你要实现 BPE 分词器、Transformer、损失函数、优化器和完整训练步骤，在 TinyStories 和 OpenWebText 数据集上训练模型，并做大量的资源核算（flop 计数）以了解计算资源的分配。

作业二：系统

目标：最大化硬件性能。覆盖内核（kernels）、多 GPU 并行化、推理等话题。

核心原则是：组织计算以最小化数据移动。从 HBM 到计算单元的数据移动代价高昂。算子融合（fusion）就是对此原则的一种经典应用：只读取一次数据，完成两个计算，写回一次结果。

推理有两个阶段：预填充（prefill）和解码（decode）。预填充阶段可以并行处理所有 token，计算受限；解码阶段逐个生成 token，很快成为内存受限环节。加速解码的方法包括：剪枝（pruning）更大模型、量化（quantize）、蒸馏（distill）、推测解码（speculative decoding）——用一个便宜模型抢先运行并猜测一批 token，然后用完整模型并行验证。

作业三：扩展定律

设定一个极端场景：如果你有 1e25 FLOPS 的计算量（数千万美元成本），你会训练什么模型？在这种规模下，无法进行典型的超参数调优，因为你只能训练一个模型。

关键的概念转变是：不应该只考虑正在训练的单个模型，而应该考虑一种缩放方案（scaling recipe）——一个从 FLOPs 预算到一组超参数的映射。通过运行一系列小规模实验，拟合缩放定律（scaling law），预测在目标规模下的损失。

这里的一个重要提醒是：缩放定律并不是自然法则，不会自动发生。必须通过精心构建缩放方案来达到可预测性。预测性与最优性同等重要——如果在每个规模都改变超参数，就无法在更大规模时可靠外推。

经典结论：训练应该使用的数据点数量大约是参数数量的 20 倍（D = 20N）。一个 70B 参数的模型应该在大约 1.4 万亿个 token 上训练。

Marin 项目正在实际测试这一方法论：他们预注册了不同计算预算下的预测，正在训练一个外推到 1E22 FLOPS 的模型，本周可能完成后就能检验预测的准确性。

作业四：数据

这是最重要部分之一——数据质量基本决定了模型会有多好。数据反映了模型想要什么：你希望它说多种语言吗？擅长对话吗？能运行长时间智能体式编码任务吗？

评估是深入话题：有内部评估（指导开发过程，要求跨尺度平滑性、相对表现）和外部评估（向客户或评审者报告，关注生态效度）。困惑度（perplexity）仍是捕捉模型内在质量的很好方式——能在私有文档上运行以避免数据污染。

数据不会凭空出现。需要积极的整理：从网络爬取网页、书籍、arXiv 论文和 GitHub 代码、去重、过滤。存在法律问题（合理使用、许可数据），数据处理需要转换、过滤、去重、数据混合、合成数据等多个步骤。

作业五：对齐

基本模板是：从模型生成回复，用人工或验证器评分，更新模型偏好更好的回复。可通过 PPO、GRPO 或用 DPO 实现。

挑战在于强化学习算法不稳定且难以调参。大规模进行强化学习需要推理服务器和训练服务器的编排，尤其在涉及代码执行的环境时。还需要在策略一致性和最大化吞吐量之间不断权衡。

一切皆效率

把这些模块放在一起看，核心思维是：在给定数据和计算预算下，构建最佳模型。资源包括数据、硬件（计算核心、内存、通信带宽），目标是最大化效率。

系统层面：显然关乎计算效率
分词：直接处理原始字节很优雅，但对当前架构计算低效
模型架构：许多变化出于减少内存或浮点运算次数
数据过滤：不浪费宝贵计算在冗余或差质量数据上
扩展定律：用更少计算在小规模做超参数调优

深入分词器：从字符到 BPE

现在我们进入第一讲的核心技术单元——分词。

为什么需要分词器？

原始文本是 Unicode 字符串，而语言模型在通常表示为索引的 token 序列上放置概率分布。分词器就是完成这种往返转换的程序：能将字符串编码成 token 序列，也能将 token 序列解码回字符串。

一个直观认识：压缩率（每 token 的字节数）越大，意味着句子越短。这很重要，因为注意力机制是二次方复杂度的，你需要保证序列足够短。一个 20 字节的字符串如果被切成 8 个 token，压缩率就是 2.5。

糟糕的方案：字符级分词

字符级分词器把每个 Unicode 字符映射为一个整数码点。看似简洁，但问题很明显：

有大约 15 万个 Unicode 字符，词汇表巨大
许多字符很罕见（如 🌍），词汇表利用效率极低
压缩率很差

也更糟糕的方案：字节级分词

把字符串转成 UTF-8 字节系列。词汇表很小（只有 256 个可能值），但序列变得很长，压缩率恒为 1。

有语义但受限的方案：词级分词

把按空格或正则表达式切分出的每个“块”称为一个 token。好处是每个 token 都有稳定语义，词汇表可能十分庞大（且无界），还有一个尴尬问题：测试时遇到没见过的 token 需要 UNK token，这很丑陋且会搞乱困惑度计算。

BPE：字节对编码

这是 GPT-2 开始在大语言模型中使用的方案。基本想法：在原始文本上训练分词器，为数据定制词汇表。常见序列表示为一个 token，且全都能被分词——如果罕见，就分解成更小的单元，不需要 UNK token。

算法在概念上很简单：

从字节序列开始，每个字节是一个 token
统计相邻 token 对的出现频率
合并出现频率最高的 token 对，赋予新索引（256 起）
反复迭代直到达到目标词典大小

具体例子：假设有字符串 "the cat in the hat"。初始状态是一串字节索引。统计所有相邻对，发现 (116, 104) 出现两次（对应 "th"），是最频繁的。将它们合并为新 token 256。继续迭代，将 (256, 101) 合并为 257（对应 "the"），依此类推。序列在缩小，词汇表在增长。

编码新文本时，只需遍历所有合并操作，将合并应用到字符串上。解码时，用词汇表将索引映射回字节序列。

在作业一中，需要做几项改进：

编码时只遍历有用的合并，而不是全部
检测并保留特殊 token（如 <|endoftext|>）
使用预分词（如 GPT-2 的正则表达式）
尽可能加速实现——如果需要，可以用 Rust 或 C 重写

分词器的未来

课程坦诚地指出：理想愿景是实现端到端字节操作，已有相关探索（如 Byt5、Megabyte、BLT 等）看起来很有前景，但目前还没有扩展到前沿规模。所以今年仍然需要教授分词器。

任何替代方案都必须满足两个性质：

模型应在某种序列抽象（chunk）上操作——不仅适用于文本，也适用于视频或 DNA
块应该是可变的，允许自适应计算——并非所有字节都被同等对待

结语：从理解到效率

第一讲的核心信息清晰连贯：语言模型构建的本质是在给定资源下最大化效率。无论是选择分词策略、设计模型架构、优化训练流程，还是决定数据过滤标准，所有决策都可以从效率这一视角审视。

课程结构也体现了清晰的递进逻辑：先掌握基础（分词、Transformer、训练循环），再追求系统效率（内核、并行化、推理），通过扩展定律实现规模预测，最终以数据整理和对齐完成端到端的模型构建体验。

本周的作业一将让你亲手实现一个完整的、可运行的 BPE 分词器和语言模型训练流程。下一讲的主题将是资源核算（resource accounting）——计算模型的浮点运算量和内存消耗，以及这些数字如何指导我们的设计决策。

准备好从零开始吧——这正是理解一切的最好方式。

官方资源：

课程网站：cs336.stanford.edu
视频：CS336 2026 Lecture 1 官方视频

CS336 2026 Lecture 1：完整中文逐字稿

视频：官方视频
来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。

正文

[00:00:04.880] 欢迎大家来到 CS 336，从零开始的语言模型（Language Models from Scratch）。这是我们的教学团队。我是 Percy。这位是 Tatsu、Marcel、Herman 和 Steven。我们为大家带来的是 336 课程的第三版。我们快速做一下介绍。我想说我从事语言模型研究已经 20 年了，但其中大部分时间做的是小语言模型。实际上，从宏观角度来看，现在的模型也仍然算小的。我记得两年前 Tatsu 和我刚开始教这门课的时候，我们并不太确定会有什么样的预期，但我们非常惊喜地发现，有这么多人想学习如何从零构建语言模型，尤其是在如今这个时代，一个编码智能体（coding agent）可能已经能直接零样本（zero-shot）生成一个语言模型了。

[00:00:49.760] 我非常高兴看到大家来到这里，真正想了解它们是如何工作的。是的，谢谢。我是 Tatsu，我是联合授课老师之一。等我们讲到架构（architectures）和扩展（scaling）以及其他有趣的内容时，我会和大家深入交流。我真的很兴奋。这是我在任教期间教过的最有趣的一门课。我觉得每年 Percy 都会取笑我，因为他说你必须把所有讲义重做一遍，因为你负责架构部分，每次内容都会变。但对我来说，这其实非常有趣。而且这是我第一次有这种体验。所以我很期待和大家一起再次经历这个过程。大家好，我是 Marcel。我之前上过这门课一次，这次回来是因为上次实在太有趣了。[笑声] 不过工作量确实很大。在我的研究中，我涉及架构方面的工作，也研究高阶梯度（higher order gradients）和训练（training）。

[00:01:38.080] 是的，期待与大家一起合作。大家好，我是 Herman。一年前，我其实并不真正了解大语言模型（LLMs）是怎么工作的，所以对我来说，token 是你在电子游戏里收集的东西，而注意力（attention）是你拥有的东西，比如注意力经济（attention economy）。在去年花了大量时间学习这门课之后，我现在在做大语言模型研究，并且非常高兴今年能担任助教。大家好，我是 Stephen。我是这门课的第一次担任课程助理（CA），我非常兴奋。我觉得这会非常有趣。总的来说，我的研究涉及语言模型、理论以及一些数据效率（data efficiency）方面的内容，我很期待与大家见面。好的，那我们开始进入正题。

[00:02:23.440] 这是我们第三次开设这门课。去年，我们决定把所有讲座视频放到 YouTube 上。所以有些人可能已经看过了。那么有什么新内容呢？没有变的是“从零开始”的理念。我们仍然坚信，通过从底层一步步构建一切，你才能真正理解所有东西是如何运作的。当然，我们并不是真的从零开始构建所有东西，因为那在学期内是完不成的。所以在过去的两年里，我们一直在完善我们的方案，弄清楚哪些内容从零构建最有价值。最后，正如 Tatsu 提到的，即使只过了一年，很多东西也已经变了。我觉得今年我们可能会花更多时间在混合专家模型（mixture of experts）上。当然，智能体（agents）如今也非常流行。因此，掌握长上下文（long-context）以及实现它所需要的技术将会很重要。

[00:03:23.930] 那么我们为什么开设这门课？我认为两年前的问题是，研究人员正在与底层技术脱节。大约十年前，所有 AI 研究人员都会自己实现并训练他们的模型。甚至八年前，人们还会下载像 Bert 这样的预训练模型并进行微调（fine tune）。而我认为如今，很多人仅仅通过提示（prompting）模型就能应付过去。当然，提示模型本身没有什么问题。我认为你可以用它做出非常了不起的事情。从抽象层次上提升总体上是一件很好的事情，但抽象是有漏洞的（leaky），而且有时候，我相信你们所有人都提示过模型，你们会遇到这样的情况——你想做某件事，但它就是做不到，而且没有其他办法。

[00:04:21.490] 我想说的是，如果你真正对基础研究感兴趣，仅仅通过提示模型，你就在极大地限制你的选项集合，也就是你正在探索的设计空间。而要从事基础研究，你真的需要拆开整个技术栈（stack）。所以我认为，充分理解语言模型如何工作对于基础研究来说是真正必要的。而我们获得这种理解的方式就是通过构建。这就是这门课的理念。但有一个小问题，那就是语言模型的工业化已经发生了。前沿模型（frontier models）真的非常非常昂贵。即使在三年前，据说训练 GPT-4 花费了 1 亿美元，而现在成本可能已经达到 10 亿美元的量级，尽管这只是推测。各大实验室正在建设的 GPU 数量也极其惊人。此外，这些模型是如何构建的，没有任何细节公开。即使在 2023 年，GPT-4 的论文也明确表示，由于竞争环境和安全影响，我们不会分享关于这些模型是如何构建的任何信息。

[00:05:33.650] 所以这些前沿模型在某种意义上是我们无法企及的。现在，我们可以构建小型语言模型，而且我们确实会构建语言模型。但我认为需要记住的是，这些小模型可能并不能代表真正的前沿模型。我会举两个例子来说明为什么可能是这样。第一个例子，这其实是很久以前的事了，我想是 2021 年。我们之后会花更多时间在浮点运算数（flop）计数和计算资源分配上。但如果你看小规模模型，MLP 层消耗的浮点运算数占比大约是 44%。当你扩展到 175B 参数规模时，这个比例会上升到 80%。所以你在小规模时优化什么、什么重要，会与大规模时不同。

[00:06:25.470] 所以如果你在小规模上做了很多关于注意力的工作，你可能在大规模上不会体验到同样的收益。第二个例子是，我们知道行为会随规模出现涌现（emergence）。小模型，这也是很久以前的数据了，但即使在那个时候，如果做各种任务的零样本（0-shot）或少样本（few-shot）学习，基本上看起来什么都没用，只有当你达到一个临界规模时，才会突然看到大量的提升。所以同样地，如果你在小规模上工作，你可能不会看到某些类型的现象，而如果是在完整规模上工作就会看到。好的，这可能会让人有点沮丧，但不要担心，我们在这门课里还是会学到一些东西的。问题在于，我们能学到什么真正可以迁移的东西？我认为很重要的一点是要把知识分成三类。首先是机制（mechanics），也就是事物是如何运作的，什么是 Transformer，模型并行（model parallelism）是如何工作的。

[00:07:31.500] 然后是思维方式（mindset），也就是你如何着手构建一个语言模型？我们会讨论如何最大化利用你的硬件资源，认真对待扩展（scaling）。最后是直觉（intuitions），哪些数据和建模决策会带来好的性能？在这门课里，我认为我们可以在教授机制，也就是事物如何运作，以及思维方式方面做得很好。我们会非常强调对一切进行分析（profile）和基准测试（benchmark），并努力优化效率。这些东西是可以迁移到更大规模的。但是，关于哪些建模决策和数据决策有效的直觉，可能不一定能跨规模迁移。对于这一点，你实际上需要去一个可以在大规模上做实验的地方。关于直觉这一点，值得一提的是，有些设计决策就是无法完全从理论上论证的，它们纯粹来自实验。

[00:08:35.039] 而力学，你可以通过构造直观地看到并行性和核函数将如何加速计算等等。但对于哪些建模上的改变会奏效的直觉，我认为你必须通过运行实验来获得。有一篇著名的 Noam Shazeer 论文，提出了 SwiGLU 激活函数，我们稍后会讲到。在结论部分，他非常坦诚地写了最后一句话：“我们不做解释。我们将这些架构的成功，归因于上天的恩赐。”所以从某种意义上说，这确实是只能从经验中获得的。最后关于“苦涩的教训”，我觉得这个观点流传甚广，大家都在讨论。我认为这里有一个常见的误解，关于它的含义。错误的解读是，规模就是一切，算法不重要。但这不对。正确的解读是，能够良好扩展的算法才是重要的。

[00:09:39.580] 你可以非常简单地把模型准确率看作是效率乘以资源。效率就是产出除以投入，而资源就是投入。实际上，从某种意义上说，效率在更大规模下重要得多，对吧？如果你在做小规模实验，运行时间翻倍了，也许你只需要多等一倍的时间，过后再回来看结果。但如果你在大规模上进行，那可能涉及数亿美元的成本。你绝对不想那样做。即使仅仅是5%的提升也可能是巨大的进步。所以实际上，效率真的、真的非常关键。我希望把这一点融入你们的思维方式，作为这门课的成果之一。从经验上看，如果你看看OpenAI在2020年的一篇论文，它显示在2012年到2019年间，ImageNet上的算法效率有44倍的提升。

[00:10:40.950] 当然硬件确实进步了很多，但这也伴随着算法的改进。当然，当这两者相乘时，你就会看到效率和准确率的巨大飞跃。综上所述，框架就是：在给定的数据和计算预算下，能构建的最佳模型是什么？对于预训练，我们主要关注计算预算，因为我们假设数据量远多于我们拥有的计算能力。但如果你处于数据受限，或者你实际上存有海量 B200 GPU 的情况下，那么你可能是数据受限的。换句话说，就是最大化效率。我们会在整个课程中反复看到这个主题。好的。接下来，我想花一点时间谈谈语言模型，以及一些历史背景，在我们深入更技术性的细节之前。语言模型已经存在一段时间了。

[00:11:52.870] 早在50年代，Shannon 就使用语言模型来测量英语的熵。在很长一段时间里，N-gram 模型被用于机器翻译和语音识别系统。它们不是系统的全部，但它们是确保你生成流畅文本的重要部分。我认为现代语言模型的血统来自神经架构。并且我认为，有一系列想法对这项发展至关重要。在90年代，有 LSTM。Yoshua Bengio 实际上在2003年的《神经网络》期刊上发表了第一个神经语言模型。这实际上不是 LSTM，那只是一个查看小范围上下文的的前馈网络。然后是 Seq2Seq 建模，它大胆地提出我们可以将整个句子压缩成一个向量。接着是 attention 优化器注意力机制，它是为机器翻译开发的。Transformer 架构建立在它之上，也是为机器翻译开发的。

[00:13:03.470] 然后扩展到专家混合模型并行。你会看到在2010年代涌现出许多不同的架构、系统和优化器思想。然后到2010年代末，我认为事情开始变得真正有趣。出现了 ELMo 和 BERT。这些是在大量文本上训练的语言模型，然后你可以在某些下游任务（如问答）上对它们进行微调，并会看到巨大改进。当时的模式是，取其中一个模型然后微调。然后Google有一篇论文，我认为，它确实预示了这种“提示输入，响应输出”的观点。但我认为真正打开闸门的是 OpenAI，因为他们拥抱了规模化。他们在2018年左右发表了 GPT 论文。他们将其扩大到 GPT-2，然后他们真正搞懂了或者说拥抱了规模法则的思想，我们马上就会讲到，这使他们能够训练 GPT-3，这是一个规模大得多的模型，比当时最大的模型大了近10倍以上。

[00:14:26.972] 它展现出诸如上下文学习这样的涌现行为。那时，Google 说，好吧，我们也得做点什么。所以他们训练了一个巨大的模型。结果发现它训练不足。并且当时尚未与 Google 整合的 DeepMind 已经找到了最优计算规模的法则。所以这一切都在发生。GPT-3 发布后，我认为对许多人来说，这算是一个警钟。当时，有很多早期尝试，想要复现这个结果。有一个名为 Eleuther 的草根组织创建了一些开源数据集和模型。它们的规模不算很大，因为他们没有太多算力。Meta 的第一个 LLM，你能看出它是个复现品，因为它大约有1750亿参数，但它不是一个很好的模型。

[00:15:23.950] 他们遇到了很多硬件问题。然后还有另一个项目，Hugging Face 的 BigScience 项目。所以，我要说，这些模型当时都不是很强。然后在过去三年里，我认为开源模型生态系统发生了相当大的变化，Meta 似乎以 Llama 系列模型引领潮流。Llama、Llama 2、Llama 3。Mistral 也加入了竞争。然后是一整套中国模型。我想我漏掉了一些。比如，字节跳动有发布一些东西，然后我认为腾讯可能也有一些其他的。所以很难跟踪所有东西，但每个人都听说过 DeepSeek 和 Qwen。所以我想主要的就是这些。但我认为这些模型有趣和令人兴奋的地方在于，现在我们有了接近闭源模型的开源权重模型。

[00:16:14.540] 所以取决于你问谁以及如何进行基准测试，它们可能稍微落后一点或不相上下。但它们绝对是非常、非常可信的模型，在工业界被广泛使用。现在还有另一条工作路线，不仅仅是发布开源权重模型。AI2、NVIDIA 和我参与的 MARIN 项目，我们努力不仅提供权重，还提供论文、代码和数据，以便我们能更深入地理解这些模型是如何构建的。为什么我如此强调这个开源生态系统？主要是因为如果没有这些模型，这门课程可能就无法开设，因为有大量关于这些大型 MoE 和 RL 系统如何工作的论文发表，使我们至少能够窥见这些前沿模型是如何构建的，并试图拼凑出其中的各个部分。

[00:17:17.440] 现在，我想，当然，即使在这些 Qwen 论文中，很多细节也缺失了。我认为它们遗漏了大量细节，所以无法复现。值得注意的是，数据混合是我们不了解的，但我认为这比什么都没有要好得多。好的。所以在过去的十年里，我认为，对语言模型的概念已经发生了变化，对吧？它曾经是你微调的东西，然后变成了你通过提示词（prompt）来使用的东西。现在在 ChatGPT 时代，它是你可以与之交谈、可以进行对话的东西。而现在——好吧，我猜我没有网络了。[笑声] 没关系。现在，我们处于智能体（agents）的时代。如果你点击这个链接，它基本上会展示一个巨大的智能体轨迹。我仍然觉得——这些模型强大到令人难以置信。

[00:18:10.940] 你给它一页文本，它就能完成一些非常复杂的智能体编码任务。所以，我们今天对语言模型的要求，可能超出了十年前任何人的想象。话虽如此，我认为基本原理并没有改变太多。很大程度上，我们仍然依赖 GPU 和内核。我们仍然使用梯度或类随机梯度方法来优化。我们仍然使用 Transformer 和注意力机制，我们稍后会更多地讨论架构，但它并没有太大变化。我认为规格不同了。现在我们需要更长的上下文长度，这意味着推理效率变得更加重要。所以对我们来说，好消息是我们不必完全改变这门课。只有 Tatsu 负责的最新中国架构部分变了。[笑声] 但基本原理，我认为，至少目前会保持不变。

[00:19:10.960] 好的，也许我先在这里停一下，以防有任何问题或想法。好的。那我们继续。简短插曲。这个程序是什么？这就是我所说的“可执行讲义”。如果你觉得它看起来像一个 Python 程序，那是因为它确实是一个 Python 程序，但为了你的观看体验，它被渲染过了。所以，当我逐步讲解时，实际上是在执行这个讲义。这使我们能够逐步浏览代码，这希望以后会很有趣也很重要。你也可以看到这个讲义的层级结构。例如，我们完成了这个函数。现在，我们回到主函数。好的，让我们谈谈课程安排和教学大纲，我想这会花费不少时间。所有信息都在这个网站或 cs336.stanford.edu 上在线发布。这是一门 5 学分的课程。我认为这门课可能有一定的名声，所以我不需要过多赘述，但——我们有五个作业。

[00:20:32.430] 它们相当繁重。根据这份评论，即使是第一个作业，实际上也相当于 CS 224n 的五个作业。有人告诉我这有点夸张，但我想，还是保守估计为好。那么，你为什么要——为什么你应该选这门课呢？好的，首先，你有一种执念，想要理解事物是如何运作的。这应该是你的首要目标。我认为这纯粹是对语言模型如何工作的好奇心。然后通过这门课，你会锻炼出更强的研究和工程能力，并有信心进入一个新的环境，有能力处理出现的任何情况。所以，当我开始在斯坦福任教时，我创建了一门叫做统计学习理论的课程，它基本上教授机器学习的理论方面。那很好，因为它武装了人们。

[00:21:34.790] 当你阅读论文时，你能理解所有数学。而现在，由于该领域更多地转向系统和实证方面，这类似于一门同类课程，它给人们足够的深度，让他们觉得其他一切似乎都变得容易了。那么，你为什么不该选这门课呢？这很重要，因为确实有不选这门课的理由。首先，你这学期真的想做一些研究。你可能应该和你的导师谈谈。他们应该知道你选了这门课。否则，可能会有一些惊喜。你对学习人工智能领域最新的热门技术感兴趣。我认为有很多其他优秀的课程、研讨课、专题课适合这个目的。我们不做很多事。我们不做多模态。我们不深入探讨智能体。所以，如果你想学这些东西，这门课不适合你。

[00:22:32.010] 如果你进来说，我有一个应用领域，我想在上面取得好结果，这可能不是正确的课程，至少开始时不是。我总是建议直接提示一个模型，微调一个模型。然后作为最后的手段，你才预训练自己的模型。因为那很痛苦，也很昂贵，但确实很有趣。好的。如果你没有选修这门课，你可以在家自学。所有的讲义材料都会发布在网站上。哎呀！而且它们也通过 CGOE 录制。所以，感谢 CGOE 做这件事。之后，它们会被发布到 YouTube 上。当然，在家跟着看视频很棒，但你真的要通过做作业来学习。所以你必须自己想办法激励自己去做。好的，说到作业，我们有五个作业。作业的理念是，我们如何从头开始做，但不仅仅是，比如说，构建一个语言模型，然后这就是作业？

[00:23:42.940] 我们不提供脚手架代码，但我们确实提供了一堆单元测试，以确保你构建的任何东西实际上是正确的，这样你就不会遇到这种稀疏奖励的情况——提交作业后要么全对要么全错。我建议的是——作业的结构是这样的，大部分作业实际上可以在你的笔记本电脑上本地完成。你可以实现并检查正确性。然后我们提供一个集群，这样你可以进行实际的训练运行，看看准确率是多少，或者获得一堆 GPU 来实际基准测试某个内核的性能。然后为了好玩，我们至少会为大部分作业设置一些排行榜。它们看起来会像这样：好吧，既然你已经了解了这个主题，试着在给定的预算内最小化困惑度（perplexity）。

[00:24:42.380] 当年，Marcel 和 Herman 都是排行榜的大师，那是在去年或者前年。所以，如果你想在这方面得到建议，我相信他们会很乐意——实际上，我不知道他们是否会告诉你他们的秘密，但你可以试试。好的，去年，我们在想，AI 能做什么？就像，好吧，我是说，是的，每个人都能用 AI，但尽你所能做到最好。我认为现在，编码智能体已经变得非常强大，它们可以解决所有作业，对吧？但说句显而易见的话，显然，如果你只是把作业一的 PDF 输入到一个云代码工具（Cloud Code）里，你什么也学不到。与此同时，AI 在回答问题和辅导方面可能非常有用。

[00:25:35.160] 所以我们必须找到一种利用 AI 的方法。我们决定做的是，我们提供一个 AGENTS.md 文件，或者等效地，一个提示词，要求 AI 具有教学意识。你可以在我们的 AI 政策指南中阅读更多相关信息。要求是，如果你要使用 AI，就要使用这个提示词。这样，它可以回答关于代码的问题，澄清你的任何理解，但当作业任务是实现 Transformer 时，它不会意外地为你生成 Transformer。这是我们第一年尝试这样做，所以请试用一下，并告诉我们它是否有效。关于算力。今年我们有——感谢 Modal，他们在他们的平台上为我们提供了计算 Credits。

[00:26:33.860] 实际上这相当不错。与去年不同，你可以获得一些——好吧，我想你们去年并没有上这门课。去年，我们有一个需要 SSH 登录的集群。这次更多地是使用 API 的方式——我一开始对此持怀疑态度，但实际用起来之后，发现确实挺愉快的。所以再次请大家试一试，并给我们反馈使用体验。我们写了一份关于如何访问和使用计算资源的指南。好，关于课程安排还有什么问题吗？好的。那么，让我们来谈谈这门课将要涵盖的内容。基本上有五个部分，与你们将要完成的五次作业相对应：基础系统、扩展、规模定律、数据和对齐。我现在会逐一介绍每个部分，让你们初步了解将学到什么。在基础知识部分，这基本上是前两周的内容。目标就是能够从零开始训练一个语言模型并构建它。这里的组成部分是：我们要对数据进行分词，定义架构，然后实现优化器并进行训练。

[00:28:00.500] 那么你们会想，剩下的课程是做什么的呢？我们稍后会讲到。让我们先从分词开始。分词实际上关乎的是：模型操作的原子单位是什么？正式地说，分词器在原始输入（仅仅是字节）和代表标记的整数序列之间进行转换。从概念上讲，它是对文本的一种切分。我们将讨论字节对编码（Byte-Pair Encoding，BPE）分词器，直观地说，它会把输入分解成频繁出现的块。然后请记住，这门课的核心是最大化效率，所以从效率的角度来看，分词是有益的，因为它能将一个长的序列（如果只考虑原始字节流的话）减少为数量更少的标记。但更微妙、或许也更重要的是，它允许进行自适应计算。也许有些地方包含很多字节，但实际上应该被压缩成一个标记，而输入中一些更稀有或更有趣的部分则应保留为多个标记。

[00:29:15.510] 好的，我们稍后会详细讨论。我只想提一下，每年我都希望自己不用教分词，因为梦想是真正有一种端到端的方法，直接对字节进行操作。已经有很多相关工作，包括最近一些看起来很有前景的网络，但到目前为止，这些方法还没有被扩展到前沿规模。由于前沿模型仍然在使用分词器，我们认为教授分词器仍然是明智的选择。好的，在你对输入进行分词之后，你会得到一堆标记。接下来你要在其上定义模型。我想大家对 Transformer 都有所了解。如果你上过 224n（MLP 课程），那你已经见过 Transformer 了。从那以后，我认为 Transformer 有了很多重要的改进或精化。Tatsu 稍后会详细谈谈这个。但这里只是快速过一遍你可能需要考虑的各种事项：激活函数已经演变。

[00:30:31.750] 如何进行位置编码已经演变。如何在不同的层中进行归一化已经演变。Blow up 已经演变。除了做完全注意力之外，有很多方法可以基本上减少注意力计算量，因为注意力是 n 平方复杂度，其中 n 是序列长度，这会变得非常昂贵。所以围绕这一点有很多想法。如果你更有雄心，可以看看这些状态空间模型，或者等效的线性注意力模型，比如 Mamba 和 Gated DeltaNet。这些在过去几年变得越来越流行。通常，这些模型和注意力之间的某种混合似乎效果很好。所以我们将探索其中一些。而在 Transformer 的 MLP 层内部，最初的 Transformer 只是一个密集的 MLP，而现在混合专家（Mixture of Experts）已经成为构建计算高效 Transformer 的主导范式。

[00:31:34.970] 我们将讨论这个。当然，对于混合专家模型，不仅仅是定义架构，我们还会看到，训练模型也需要不同的技术。最后，可能有点无聊但很重要的问题是：你的 Transformer 的形状是什么？有多少层？多少个注意力头？隐藏维度是多少？专家数量是多少？这在讨论规模定律时可能会更多地涉及，但设置这些参数其实——看起来几乎有点琐碎。它只是超参数，但实际上在扩展和语言模型的背景下，它有着非常非常重大的影响。一旦你定义了模型架构，如何训练模型呢？这里围绕着损失函数有很多设计决策。有下一标记预测，这是默认选项。但人们发现，预测多于一个标记似乎有助于改进模型。还有优化器。人们过去常用 AdamW，但越来越多地使用 Muon，尤其是在一些最新的开源模型中，比如 Kimi K2 模型。

[00:32:57.920] 初始化，这听起来也有点无聊，但事实证明它对你在更大模型上的训练稳定性能力有着巨大影响。学习率调度。正则化。批量大小。然后是 MoE 特有的东西。所以你看看这个列表，可能会想，这些只是超参数罢了。我要尝试一堆不同的选项。但事实证明，真正以原则性的方式非常小心地设置这些超参数，将决定一次运行是完全爆炸且毫无用处，还是一次运行达到最先进的水平。好的，当我们讨论规模损失时，我会再回到这一点。那么在作业 1 中，你们要做的是：实现 BPE 分词器，实现 Transformer、损失函数、优化器以及完整的训练步骤。我们会让你们做大量的资源核算，以便你们了解浮点运算量去了哪里。你们将在 TinyStories 和 OpenWebText 等数据集上训练一些模型。

[00:34:06.280] 然后会有一个排行榜，你们要尽可能快地降低困惑度。对于那些熟悉 NanoGPT 速通（speedruns）的人来说，这有点类似。好的，所以在作业 1 结束时，你们应该能够独立地从零开始构建一个语言模型。这是非常令人兴奋的。如果我要在这里给出一个高层次的要点，那就是虽然分词器、建模和训练被呈现为不同的部分，但实际上，一切都关乎于平衡以下几点。你们想要具有表现力的模型，因为你们想要表示数据的复杂性，但同时，你们希望训练是稳定的。我们将大量讨论如何让参数和梯度范数处在这个黄金地带，使它们既不会爆炸也不会消失。事实证明，训练语言模型的很多工作都关乎稳定性。最后是效率，这多少更直接一些。你只需让它在硬件上运行得更快。但你会看到一些有趣的事情，比如，如果我们改变架构，很多架构决策实际上是，我们能否通过（比如说）减少到低维空间的投影来使其更快。

[00:35:41.860] 但接下来的问题是，它是否同样有效？而做出这些权衡取舍，正是这里的关键所在。因此，在作业2中，我们将更深入地探讨系统层面。这里的目标仅仅是充分利用你的硬件性能。我们将讨论内核（kernels）、如何跨多个GPU进行并行化，以及如何执行推理。至于基础知识，我们实际上会在下一讲开始涉及。我提到了资源核算（resource accounting），我想你们可能都以前构建过模型。但这实际上是关于追踪所有浮点运算（flops）的去向以及内存消耗在哪里。所以，我们会花一些时间，基本上就是进行资源核算。我们会看到这个经常出现的公式：在1万亿个token上训练一个SMDB模型需要多少次浮点运算？粗略计算是6乘以N乘以D。这个公式是怎么来的呢？

[00:36:53.850] 然后我们会看看硬件。这里有一幅我心目中的示意图来描述硬件，那就是：你的内存并不在你的计算单元旁边，你必须将参数或激活值从内存移动到计算单元，进行计算，然后再移回去。而这往往是瓶颈所在。例如，B200（我们将有机会使用它）在bf16精度下每秒有2.25千万亿次浮点运算（petaFLOPS），并且有每秒8TB的内存带宽。这意味着什么呢？我想在下一讲中——我打算在下一讲中详细拆解这一点。我们会利用这些信息进行计算，看看不同类型的算法需要花费多长时间。我们将讨论屋顶线分析（roofline analysis），它帮助我们理解一个计算任务究竟是受计算限制还是受内存限制。通常来说，是受内存限制。然后我们会稍微谈谈基准测试（benchmarking）和性能分析（profiling）。

[00:38:08.896] 这就是DGX B200的样子。你有8个GPU，它们通过NVLink连接。如果你有更多——比如有一千个GPU，那么你会有多台这样的机器，它们通过InfiniBand或以太网连接。接下来，当我们谈论系统时，有两个部分：内核。内核基本上就是在GPU上运行的函数。当你使用纯PyTorch时，所有PyTorch的原始操作实际上都对应于启动特定的、内置的内核。所以，无论你是否意识到，你其实已经在使用内核了。但关键是，对于某些类型的计算，如果你仔细观察，你可以编写自定义内核来让GPU运行得更快。这里的主要原则是组织计算以最小化数据移动。记住这张图，从内存移动数据代价高昂。所以你要尽量最小化这种移动。

[00:39:21.970] 举个简单的例子，假设你想计算A和B。通常，你可能需要从高带宽内存（HBM）中读取数据，计算，写回，然后再读取，计算，再写回。这样，你基本上把数据来回发送了两次。有一种叫做融合（fusion）的方法，你只读取一次数据，完成两次计算，然后写回一次。这会节省很多时间。这就是算子融合（operator fusion）。分块（Tiling）是围绕同一思想的更复杂变体。此外，GPU也变得越来越复杂。我不确定我们是否有时间深入探讨所有这些细节，但至少我们希望让你们了解一些GPU的特殊性，并让你们体会到，为了尽可能榨取它们的性能，需要考虑哪些类型的事情。我们会用Triton编写一些内核。那么，如果你有数千个GPU会发生什么？最小化数据移动的原则仍然是一样的。唯一的不同是，在不同GPU之间移动数据的成本甚至更高。

[00:40:44.420] 我们将讨论这些非常经典的集合操作（collective operations），如gather、reduce和all-reduce，它们基本上是思考分布式训练（distributed training）的方式。整个游戏规则是这样的：我们有模型参数、激活值、梯度和优化器状态，它们需要在多个GPU之间进行分片（sharded）或分割。当然，你需要把正确的数据送到正确的节点进行计算，然后再写回去。所以这涉及到一整套编排，而如何高效地做到这一点将是本单元的主题。有多种分片方式。你可以通过分割数据、分割模型、分割模型中的不同层、分割序列、在专家之间进行分割来实现分片。我们将讨论每种方式所带来的权衡。好的。最后，我们将讨论推理（inference）。正如我提到的，它的重要性日益增加。推理的目标是实际使用模型。

[00:41:49.640] 这里有个小细节。推理，当然，当你与模型聊天时需要用到它，但它对于强化学习（reinforcement learning）也很有用。它用于执行展开（rollouts）、测试时计算（test-time compute）、生成合成数据、评估。所以推理是语言模型工作中非常关键的一部分。我们可能不会花太多时间在这个主题上，因为我希望课程内容已经很满了，但我们会看看能覆盖多少。关于是否应该让你们从头编写推理代码有一些讨论，但我们再看看吧。思考推理的方式是，它有两个阶段：预填充（prefill）和解码（decode）。在预填充阶段，你获取提示词（prompt），然后将所有token前向传播，并构建键值对（key value pairs）。这很像训练中发生的事情。然后在解码部分，token被逐个生成。

[00:42:55.380] 这部分很快会成为内存受限的环节，这也是推理困难的原因。有很多方法可以加速推理。你可以尝试通过剪枝（pruning）一个更大的模型来使用更便宜的模型，或者你可以量化（quantize）、蒸馏（distill）。你可以使用一种叫做推测解码（speculative decoding）的技术，用一个更便宜的模型抢先运行并猜测一批token。然后，你用完整的模型并行地处理这些token，看看猜测是否准确。如果你运气好，那么你可以接受所有这些token，这比逐个token生成要快得多。当然，你也可以进行系统层面的优化。有很多专门为推理设计的内核。关于推理，一个有趣的点是，如果你在运行一个服务，查询可能会在不同时间点到达，你必须想办法将它们批次化处理。

[00:43:54.780] 而在训练中，你基本上已经定义好了批次，一切都要可预测得多。在作业2中，将涉及用Triton实现内核以及进行一些并行训练。这里的细节实际上可能会有所变化。因为课程助教（CAs）有改革系统部分的宏伟计划。所以作业可能与去年的看起来略有不同，但它会涵盖大致相同的材料。我想提一点的是，谷歌的一些人写了一本很棒的书，叫《如何扩展你的模型》（How to Scale Your Model），我认为它对于提供对屋顶线分析、Transformer数学以及概念上理解LLM非常有帮助。所以我强烈推荐你们看一看。唯一的问题是它来自谷歌，所以主要涉及TPU。但很多高层次的概念是相似的。而且，他们现在有一个关于如何思考GPU的新章节。

[00:45:09.510] 好，那么第三次作业是关于缩放定律的。到目前为止，你们已经训练了一个语言模型。你们可以通过并行优化内核让它运行得非常快。现在，你们想要扩大规模。那么如何扩大规模呢？想象一下下面的场景。如果你有1e25 FLOPs的计算量，这相当于数千万美元的计算成本，你会训练什么样的模型？我认为这是一项艰巨的任务，因为如果你搞砸了，那可真是一大笔钱打了水漂。而且在这种规模下，你可能无法进行典型的超参数调优，因为你只能训练一个模型。这就是在大语言模型训练中你必须处理的关键问题，而如果你只是微调一个模型或者做小规模的事情，你就不太需要面对这个问题。

[00:46:11.870] 这里关键的概念转变是，我们不应该只考虑我们正在训练的单个模型，而应该真正考虑一种缩放方案。缩放方案是一个从FLOPs预算（比如说1e25或1e24）到一组超参数的映射。基本上就是一个配置文件。对于给定的缩放方案，我们要做的是运行一系列实验来计算在较小规模下得到的损失，然后拟合一个缩放定律，这使你能够预测在目标规模下的损失。所以也许你运行一些小型实验，拟合一个缩放定律，然后预测出在更大规模下你会得到什么结果。这就是基本方法。利用这一点，你现在可以优化缩放方案，通过小规模实验来瞄准更大规模的目标，这非常棒。

[00:47:09.910] 其次，你可以在真正运行实验之前，预测你理论上将要达到的损失。这让你可以去筹集资金。你可以说，看，我做了小规模实验，我认为我能得到一个非常——比如，一个GPD-5级别的模型。请给我很多钱，这样我就能训练那个模型。但我觉得另一个可能的误解是，缩放定律并不是自然法则。它们不会自动发生。你必须通过努力让它们成为现实。这需要通过精心构建一个缩放方案来实现。而且记住，缩放方案必须能够外推。这通常意味着你有一系列超参数，随着规模增大，比如说，学习率可能保持不变，也可能下降，批量大小会增大多少等等。这些都是缩放方案需要解决的问题。所以你要进行参数化。那么，为了获得这些可预测的缩放定律，你实际上需要考虑的一件事是，如何以能够实现所谓的超参数迁移的方式对模型进行参数化。

[00:48:31.080] 也就是说，你在小规模使用的超参数，要么就是你在更大规模使用的那些，要么就是其可预测的函数。因为如果在每个规模，你的学习率有时是1e-5，有时是1e-4，那么当你在更大规模时，你就无法神奇地猜出正确的学习率。所以思维上的一个转变是，可预测性实际上至少与最优性同等重要。你通常会想，哦，我们在这里是试图优化效率，我们想要进行超参数调优并让一切达到最优。是的，你确实想这样做，但你也想要这种可预测性，这样你就不会在更大规模时感到意外。这就是背景设定。我们要看的实际缩放定律是相当经典的。这些是——你们中有些人可能见过这个想法：如果我给你一个FLOPs预算，你应该训练——你应该如何平衡训练更大的模型与在更多token上训练？

[00:49:36.000] 这就是来自Kaplan等人以及所谓的Chinchilla缩放定律的经典计算最优缩放定律的用武之地。基本思想是，对于每一个FLOPs预算，比如说从6e18一直到3e21，你遍历不同的模型大小并选择最好的那个。可以想象成对每个预算进行最小化。然后你拟合一条曲线，使你基本上能够根据FLOPs预算预测参数数量。如果幸运的话，数据点会大致落在一条线上。如果不幸运，数据点就会分布得很散，这意味着你没有信心能够可靠地进行预测。所以结果就是，虽然这相当粗略，但有一个经验法则：训练应该使用的数据点数量大约是参数数量的20倍。所以一个70B参数的模型应该在大约1.4万亿个token上进行训练。当然，根据数据集和架构的不同，这个数字实际上会有所变化。另外，这也没有考虑到推理成本。

[00:51:01.040] 如今很多模型都很小，但它们在比计算最优所需多得多的大量token上进行训练，因为出于推理原因你想要一个更小的模型。我们Marin项目一直在做的一件有趣的事情是预注册我们的结果。我们拟合了一系列不同计算预算下的缩放图。我们拟合了一个缩放定律，基本上做出了这些外推到1E22 FLOPs的预测。这个实际上正在训练中。如果你去Marin网站，你可以跟进进度。它实际上可能最早在今晚就完成了。所以也许周三我会回来报告我们的表现如何，看看我们的预测损失与预注册的损失匹配得怎么样。这个想法是，如果我们训练这个我们从未训练过的大型模型，我们做出了一个预测。

[00:51:53.640] 如果我们能预测它的表现会有多好，那就太好了。所以关于作业3，我觉得这要么有趣，要么实际上——这确实是个有趣的——我们就说这是个有趣的作业吧。[笑声] 所以我们将定义这个训练API，基本上就是，你给我们超参数，然后我们会给你返回一个损失值。我们要尝试做的是模拟如果你能进行大量训练运行会发生什么。当然，我们没有足够的计算资源让每个人都真正训练他们自己的API模型什么的。所以基本上，我们在离线状态下训练了一堆模型，然后基本上提供了一个缓存。看起来就像你正在训练一样。你要做的是提交训练任务。你给我们一个配置，基本上，我们给你返回一个损失，然后你可以做任何你想做的事情。

[00:52:50.630] 我们建议你根据这些数据点拟合缩放定律。然后进行外推。然后我们给你一个预算，基本上评估你的模型最终落在哪里。所以这旨在——我本来想说，这旨在模拟如果你真的有一笔比如1亿美元的预算需要花费时的高压力场景。你必须非常小心地决定如何花这笔钱。当然，这是低风险的。所以在这个时间点，你已经会训练一个模型，知道如何让它变快，知道如何扩大规模。那么，还缺少什么呢？你用什么东西来训练模型？这将是数据部分要讨论的主题，可以说这是最重要的部分之一，因为数据质量基本上决定了你的模型会有多好。另一种表述方式是，你希望你的模型做什么？数据基本上反映了你的模型想要什么。所以，你希望它说多种语言吗？擅长进行对话吗？

[00:54:13.590] 你们想运行长时间、智能体式的编码任务吗？那么这部分也涉及到——我们将从讨论评估开始，这基本上定义了你们希望模型具备的能力。我们要谈的一点是，评估是一个相当深奥的话题。它不仅仅是在某些基准测试上运行。还有用于模型开发的内部评估指标。这里重要的是跨尺度的平滑性，这样就能——记住，我们希望事情是可预测的。相对表现很重要。你未必关心它在绝对意义上的表现如何，因为，比如说，一个困惑度数值，在某个保留数据上达到1.2的困惑度到底意味着什么呢？然后是外部指标。这些是你向客户、评审者或任何你展示成果的人报告的内容。在这里，生态效度才是真正重要的。

[00:55:26.170] 我认为有时候这两类指标会被混为一谈，但我认为它们确实服务于两个不同的目的。所以你可以把困惑度看作是对内部开发非常有用的东西。直到今天，困惑度仍然是捕捉模型内在质量的很好方式，而不必担心基准测试的问题。现在，还有一个单独的问题，就是你用什么来运行你的评估。推荐的做法是，如果你有一些不在互联网上的数据，那会很好，因为你可以避免数据污染。好的。然后还有更高级的用例，这些用例更能代表面向外部的使用场景。还要注意的一点是，语言模型据称是非常通用的。所以我们也确实需要一套非常多样化的评估来匹配这一点。我始终建议设置多个评估——你可以把它们平均成一个数字，但那个平均值往往会混淆很多不同的东西。

[00:56:40.530] 那么在我们设置好评估之后，我们就知道我们在构建什么了。那你如何获取数据呢？首先，数据不会凭空掉下来。它需要被主动地策划和整理。我经常觉得，尤其是在课堂和研究中，有时候你只是被给了一个数据集，然后就是，好吧，现在我要在这个数据集上做研究。但很多语言模型的工作，特别是如果你想收集这些大规模数据集，你必须主动去审视它。网页是从互联网上爬取的。还有书籍——我想这在这个时间点上是有争议的——还有arXiv论文、GitHub代码，等等。这是2021年The Pile的一个旧数据分布图。你可以看到，语言建模数据集是相当多样化的。尤其是现在，我觉得围绕在受版权保护的数据上进行训练是否属于合理使用，存在很多争议。也许有时候你需要获得数据许可，等等。

[00:57:44.920] 所以围绕数据存在法律问题，我认为这些问题是相当重要的。例如，很多GitHub代码没有许可证。那你如何解读这一点？你是假定它是宽松许可的，还是采取保守态度，假定它不是宽松许可的？还有一点——事实是，数据甚至不全是文本。它要么是HTML或PDF，要么代码是目录结构。这需要经过处理才能变成可用于训练的实际文本。这就是数据处理的主题。这里需要几个步骤。转换，将一些非文本内容转化为文本。过滤，只保留好的内容。如果Common Crawl中某个随机的互联网文档质量极差，你很可能不想在其上训练。你需要去重。有多个数据源，或者你如何组合不同的来源？最后，最近这方面也有很多工作，就是生成合成数据，这可能意味着获取真实数据，然后将其改写为更符合下游任务的内容，或更像维基百科风格的内容，或者任何你想要的形式。

[00:59:09.580] 所以这是一个活跃的研究领域。此外，数据既可以用于预训练。还有一种叫做中期训练的东西，通常是在预训练阶段最后放置的高质量数据。这包括长上下文数据，比如可能是BigCode的代码仓库或书籍。最后，还有后训练数据，可能是对话或带有工具调用的智能体轨迹。所以在作业4中，我们会让你们从一个非常原始的语料库开始，比如一份粗糙的网络爬取数据，然后完成所有的工作来过滤、去重，让数据变得干净。所以这是，我会说——我不知道我会不会称之为有趣，但它肯定是，我想，很多人会称之为“脏活累活”的一部分。但我认为这是从零开始构建语言模型的一个重要部分，所以你们必须获得完整的体验。最后，对齐。到目前为止，我们基本上是使用完全监督来训练模型。

[01:00:30.000] 预测下一个词元或接下来的几个词元。现在在这一点上，模型应该已经表现合理了，但我们可以通过使用弱监督来进一步改进它。为什么是弱监督？因为有时候批评比生成更容易。所以你并不总能拥有这样的数据，明确指出这是该提示词的正确回复，但也许你可以有一种方式来指定什么是“好”的样子。那么，基本模板是：你从模型生成回复。你用人工或验证器或语言模型裁判来给它们评分，然后你更新模型，使其偏好更好的回复。这可以通过各种强化学习算法来实现，如PPO或GRPO，或者对于偏好数据，也可以用更简单的DPO。强化学习面临的挑战在于，强化学习算法不稳定且难以调参。

[01:01:32.510] 你们中有些人可能对此有亲身经历。就我个人而言，我更喜欢尽可能保持在监督学习的框架内，这样我们就能在监督环境下处理尽可能长的时间。然后最后，好吧，行，我确实得做强化学习了。但有些人，不管出于什么原因，就是喜欢做强化学习。另外，我们希望今年能讨论的一点是，如果你大规模地进行强化学习并试图最大化吞吐量，实际上会有很多系统层面的挑战。你需要有一个推理服务器和一个训练服务器。然后推理服务器必须生成这些轨迹。特别是如果你针对涉及代码执行的环境做强化学习。这整个就是一个编排的游戏。然后如果你的工作节点落后了，或者你遇到了偏离策略的问题，那么你就在不断地在坚持策略一致性和最大化吞吐量的欲望之间进行平衡。

[01:02:28.710] 这是一片庞大而美妙的混乱之地，希望我们讲到那部分的时候能更多地讨论。那么作业5，我们还在决定具体要做什么。去年是实现了DPO和GRPO，并让它们在某个数学基准上跑通，但我们看看今年能在现实性维度上再推进多远。所以再说一次，记住关键在于效率。效率可以是数据效率，也可以是计算效率。所以思考方式是，你拥有所有这些资源。你有数据，你有硬件，其中有计算核心，你有内存，有通信带宽。你只是在试图弄清楚，在给定一组固定资源的情况下，如何根据某个评估标准构建最好的模型。所以从这个角度来看，我认为你实际上可以把很多这些设计决策看作是为此进行优化。

[01:03:29.490] 所以，系统方面，显然，这关乎计算效率。分词技术，就像我之前提到的，你不能直接处理原始字节，但那样做计算效率会非常低，至少就当今的模型架构而言是这样。因此，分词在很大程度上是为了提高计算效率。模型架构方面，我们将看到的许多变化都是出于减少内存或浮点运算次数的动机。事实上，其中很多是受到更快推理需求的影响。数据过滤，你也可以从效率的角度来看待它。我们不想浪费时间去在一堆冗余的、质量差的数据上更新梯度，即使它可能不会直接损害你，但它确实有损害，因为如果你的计算预算是固定的，花在差数据上的时间多了，花在好数据上的时间就少了。

[01:04:21.270] 最后，缩放定律明确地是关于如何有效地在更小的模型上进行超参数调优。而现在，明天，我们可能会受到数据限制，你应该采取什么设计决策的考量可能会改变。但我认为我们试图教给你们的整体思维方式是，要思考你所采用方法的效率。好的，我先讲到这里。关于这些作业或主题还有什么问题吗？好的。那么现在我们来学习分词。这是我们进入的第一个单元。Andrej Karpathy 有一个关于分词的很棒的视频。你们应该去看看。起点是原始文本，什么是文本？它是 Unicode 字符串。另一方面，语言模型在通常表示为索引的 token 序列上放置一个概率分布。所以我们需要一个程序，能将这些字符串编码成 token，也需要一个程序能将 token 解码回字符串。

[01:05:44.800] 所以，分词器基本上就是能完成这种往返转换的东西。这里有一些例子，可以让你们对分词器的工作原理有个初步了解。实际上，我之前应该试着早点进入那个界面。这样没法演示了。如果你访问那个网站，你可以尝试不同的分词器。这里有一些观察。你们会体会到为什么分词器有点烦人，以及为什么人们想要摆脱它们。一个词，以及它和它前面的空格组成的复合体，是不同的 token。所以你会看到很多 token 实际上是“空格+单词”，这没问题，但有点奇怪。所以这个 “hello” 和这个 “hello” 实际上是两个完全不同的、彼此毫无关联的索引。

[01:06:40.560] 有时，取决于你使用的分词器，数字会被表示为——每几个数字是一个 token。有时这是可预测的，有时则不是。一些分词器试图让每个数字都成为一个 token，但那样你会爆炸式地增加 token 的数量。所以这中间有一些权衡。这是 GPT-5 的分词器。你可以把这个字符串转换成这些索引。然后你可以把它解码回字符串。所以分词器应该能够往返转换。如果你实现了一个分词器，但它不能往返转换，那你就遇到问题了。这里的压缩率是每 token 的字节数。在这个例子中，这个字符串的字节数是 20。token 数是 8。你用 20 除以 8。所以压缩率是 2.5。好的，也就是每 token 2.5 字节。压缩率越大，意味着句子越短，这很好，因为注意力机制是二次方复杂度的，你想要确保输入序列更短。

[01:07:56.200] 现在，你显然可以通过增加词汇表大小来提高压缩率，但那样你就会遇到稀疏性问题，因为词汇表中的每个元素都被视为一个独特的元素。所以现在，分词器，尤其是多语言分词器，有 10 万或 20 万个不同的 token。你可以看看 GPD token 词汇表。我想为了节省时间，我们跳过这部分。那么你如何构建一个分词器呢？我会很快地过一遍。你可能想到的第一件事是，嗯，Unicode 字符串，那是一系列 Unicode 字符。每个字符是一个整数，你可以在 Python 中调用 ord，然后得到一个数字。这可以转换成字符。那么让我们构建一个字符级分词器，基本上就是把每个字符分解开，并将其编码为一个 token。然后这可以解码回来。

[01:08:59.720] 生活很美好。但现在，有 15 万个 Unicode 字符。所以你的词汇表大小可能是 15 万，这很多。我的意思是，这不算疯狂，但我认为更大的问题是许多字符实际上很罕见，这意味着这确实是词汇表的低效利用。而且，能够反映这一点的压缩率也不是很好。所以大多数时候，你实际上是用很多 token 来表示你的序列。而且许多索引实际上并没有被怎么使用。所以这不是一个很好的分词器。这里有另一种尝试。你可以把字符串转换成字节。Unicode 有 UTF-8 编码，这意味着你可以——有时像 “a” 这样的字符串只有 1 个字节，有时，一个字符串是多个字节。

[01:10:05.930] 那么让我们围绕这个来构建一个分词器。我们可以把这个字符串转换成一系列字节。注意现在这个序列变长了，但所有的数字都在 0 到 255 之间，因为这就是字节的含义。压缩率是 1，这不太好。所以字节序列可能非常长，但词汇表大小很小。好的。所以这两种方法都很糟糕。那么让我们尝试取得一些进展。这就是人们过去在 NLP 中实际做的事情，如果有人记得的话。如果你拿一个字符串，我可以把它切分——按空格或某个正则表达式来分割。让我们就把每一块称为一个 token。这种方法的好处是每个 token 都是有意义的，因为单词是人类发明的，而且单词往往具有稳定的语义。但你的词汇表大小是训练数据中不同块的数量，这可能非常多。

[01:11:25.010] 而且，你的压缩率，我的意思是，它相当不错，但词汇表可能非常庞大。实际上，情况比这更糟，因为词汇表实际上可能是无界的，对吧？因为在测试时，你可能会得到一些序列并进行分词，然后你会得到一个从未见过的 token。人们过去会分配这种 UNK token，但那真的很丑陋，而且会搞乱你的困惑度计算。所以这也不是很好。好的，那么我们实际要做的事情叫做字节对编码。这是很久以前为数据压缩引入的。远在语言模型真正登上舞台之前。它最初被引入 NLP 是为了做神经机器翻译。第一篇将 BPE 用于大语言模型的论文是 GPT-2。所以基本想法是，你将在原始文本上训练分词器，以构建一个针对数据定制的词汇表。

[01:12:36.072] 并且你还会拥有这个特性，即所有内容都能被——都可以被分词。如果它是罕见的，那么它就会分解成更小的单元，而不是有这个 UNK token。所以常见的序列将被表示为一个 token。罕见的序列将被分割成多个 token。这就是这个想法。所以这个算法在概念上相当简单。你基本上从你的语料库开始。假设它是一个长序列。你得到一个字节序列。每个字节一开始是一个 token。然后我们将合并出现频率最高的相邻 token 对。让我们通过代码来看这是如何工作的。这里有一个简单的字符串，“the cat in the hat”，这里是 BPE 算法的实现。我们将把它转换成一个字节序列。然后我们将——首先，我们会统计连续 token 对出现的次数。

[01:13:42.320] 所以116、104出现了两次，我们就得到了这个结果。然后我们要找出出现次数最多的配对，也就是116、104。我想这里有几个并列的情况，但我们直接取第一个。然后我们要合并这个配对。通过合并这个配对，我们创建了一个新的token。在这种情况下，这个token将被命名为256。它代表这个配对，我们将它添加到词汇表中。所以256代表th这个序列，也就是说t和h已经被合并了。每当我们看到th，就用256来表示。然后我们遍历索引，将每一处116、104替换为256。所以那两个位置已经被替换了。然后我们继续迭代。下一次我们就会找到256和101，将它们合并，现在我们有了257。然后再合并一次，就得到了258。

[01:14:58.360] 随着时间推移，序列在缩小，词汇表在增长。好的。让我——这里的压缩比，我是说，在这个玩具例子中，压缩比是1.5。好的。现在你有了tokenizer，如何对新文本进行tokenize呢？你拿一个新字符串，然后对它进行编码。概念上讲，基本上就是遍历你做的所有合并操作，然后把合并应用到你的字符串上。好的，我就不逐步讲解那段代码了。这样你会得到一个序列。这就是对"the quick brown fox"的序列编码。当你解码时，会得到相同的结果。为了节省时间，我讲得有点快。我可以保证这个实现是能用的，这是一个完整的BPE实现。但它非常慢。所以在作业1中，我们会要求你们基本上把它做得更快。

[01:16:22.980] 目前，encode遍历所有合并操作，这非常慢，因为可能会有——你拥有的合并数量基本上就是词汇表大小减去256。所以你只需要遍历那些有用的合并。你必须建立一些索引来实现这一点。还有一些关于特殊token的细节，概念上并不深奥，但对构建现代tokenizer很重要。另外，为了简单起见，我展示的tokenizer是直接处理整个字符串的。实际上，在处理时，你会将文本分成多个块，然后在每个块上应用tokenizer。这样会快得多。然后尽量让它更快。在某个时点，你可能会意识到Python并不是很快。如果你想用你喜欢的语言来实现，比如Rust或C或别的什么，那就放手去做吧。

[01:17:29.190] 好的，快速总结一下。Tokenizer在字符串和token或索引之间进行转换。之前基于字符、基于字节、基于词的方法各自都有很大的不足。BPE是一种有效的启发式方法，而且是由数据驱动的，看起来相当有效。正如我之前说的，也许明年我就不用教这个了，但今年我们还是得用tokenization。不过即使我们摆脱了tokenization，我认为任何替代它的解决方案也必须满足以下性质。如果你有模型，transformer需要在某种序列抽象上操作。这一点最明显的体现是，你不仅考虑文本，还考虑视频或DNA序列，在这些领域中，单个字节或单元的信噪比实际上非常低，你必须做某种抽象，把它提升到一个可以进行建模的层次。

[01:18:41.750] 最后，正如我提到的，块应该是可变的。你想要自适应计算。并非所有字节都被同等对待。如果你不这样做，我认为你会得到次优的结果。所以任何端到端的解决方案也必须具备这些性质。好的，我就讲到这里。下周三，我们将开始关于资源核算的单元，我会说这是一个入门级的系统。在那之后，我们将回到架构方向，从这里继续推进。好的。

CS336 2026 · Lecture 1 · Overview, tokenization

视频：官方视频
字幕：en-US · manual captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：原语言字幕稿，待校对

正文

[00:00:04.880] Welcome, everyone to CS 336, Language Models from Scratch. This is the teaching staff. I'm Percy. This is Tatsu, Marcel, Herman and Steven, and we're bringing you the third edition of 336. So we'll just do a quick round of introductions. So I'd like to say that I've been doing language models for 20 years, but most of that time was small language models. Actually, this is still small in the grand scheme of things. I think when Tatsu and I started teaching this class two years ago, we weren't really sure what we were going to expect, but we were very pleasantly surprised that so many people wanted to learn how to build language models from scratch, especially in these days when a coding agent could probably zero shot a language model.

[00:00:49.760] I'm really glad to see all of you here to actually want to learn how they work. Yeah, thanks. I'm Tatsu, I'm one of the co-instructors. I'll be talking to you once we get to architectures and scaling and all these other very fun things. I'm really excited. This is the most fun class I've taught in my time here. I think every year, Percy makes fun of me because he says you have to redo all your lectures because you took on architectures, and everything changes every time. But it's actually pretty fun for me to do it. And it's the first time that I've had that experience. So I'm looking forward to going through this experience again with you all. Hello, I'm Marcel. I've seen this class once before, and I'm returning because it was so much fun last time. [LAUGHTER] It's a lot of work though. In my research, I do architecture stuff, I do higher order gradients, and I do training.

[00:01:38.080] Yeah, looking forward to working with you guys. Hello, everyone, I'm Herman. A year ago, I didn't really how LLMs worked, so I think for me, tokens were the things you collect in video games, and attention was the thing you had, like the attention economy. Then after spending a lot of time on this course last year, I'm now doing LLM research, and I'm really excited to TA this year. Hi, everyone. I'm Stephen. I am a first time CA for this course and I'm very excited about it. I think it will be a lot of fun. Broadly, in my research, I work on language models, theory and some data efficiency stuff, and I'm excited to meet you all. All right, so let's go into things.

[00:02:23.440] So this is the third time we're offering it. Last year, we decided to put all our lectures on YouTube. So some of you have seen it. So what's new? Well, what hasn't changed is 'from scratch' philosophy. We still believe strongly that by building everything from the ground up, you really learn how everything works. Of course, we don't actually build everything up from scratch because that wouldn't fit in the quarter. So over the last two years, we've been refining our recipe and figuring out what are the things that you build up from scratch that are the most high value? And then finally, as Tatsu alluded to, even in a year, a lot has changed. I think this year we're going to spend maybe a bit more time on mixture of experts. And of course, agents are very popular these days. So getting a handle on long-context and what is needed for that are going to be important.

[00:03:23.930] So why did we make this course? And I think the problem two years ago was that researchers were becoming disconnected from the underlying technology. It used to be the case may 10 years ago, all AI researchers would just implement and train their own models. And then even eight years ago, people would download pre-trained models such as Bert and fine tune them. And I think a lot of today, you can get by by simply prompting a model. And of course, there's nothing wrong with prompting model. I think you can do amazing things with it. I think moving up the abstraction in general is a great thing, but abstractions are leaky, and sometimes, I'm sure all of you have prompted models, you run into situations where something is-- you wanted to do something, but it just can't do it and there's no recourse.

[00:04:21.490] And I would argue that if you're really interested in fundamental research, by simply prompting a model, you're vastly constraining the set of options, the design space you're looking at. And by looking at fundamental research, you really need to tear up the whole stack. So I would argue that full understanding of how language models work is really necessary for fundamental research. And the way that we're going to get an understanding is by building. That's the philosophy of the class. But there's one small problem, which is that industrialization of language models has happened. Frontier models are really, really expensive. Even, this is three years ago. GPT-4 was supposedly costing $100 million to train, and now costs probably are on the order of $1 billion, although that's speculative. And the number of GPUs that all the big labs are building is just immense. Furthermore, there's no details on how any of these models are built. So even back in 2023, the GPT-4 paper explicitly says that due to a competitive landscape and safety implications, we're not going to share anything about how the models are being built.

[00:05:33.650] So these frontier models are in some sense out of the reach for us. Now, we could build small language models, and we will build language models. But I think it's important to remember that these might not be representative of the actual frontier models. And I'll give you two examples why this might be the case. So here's one. This is from actually quite a long time ago. I think back in 2021. If we're going to spend more time on flop counting and looking at where the compute is being spent. But if you look at small scales, the fraction of flops spent in the MLP layers is around 44%. And if you scale up to 175B, then it goes to 80%. So what you optimize and what matters at large scale is going to be different from small scale.

[00:06:25.470] So if you did a bunch of small scale stuff here on the attention, you might not experience the same benefits at large scale. The second example is that we know of emergence of behavior with scale. So small models, this is, again, for a while ago, but even back then, if you have a 0 shot or a few shot learning of various tasks, it basically, seemed like nothing was working, and only when you reach a critical scale do you suddenly see a lot of improvement. So again, if you're working on small scale, you might not see certain types of phenomenon compared to if you were working at full scale. OK, so that might be a little bit disheartening, but fear not, we're going to learn something in this class. And the question is, what can we learn that actually transfers? And I think it's important to break this down into three types of knowledge. First, there's the mechanics of how things work, what a transformer is, how model parallelism works.

[00:07:31.500] There's mindset, which is, how do you go about approaching building a language model? We're going to talk about how you want to squeeze the most out of your hardware, taking scaling seriously. And then finally, intuitions. Which data modeling decisions are going to yield good performance? So now, in this class, I think we can do a pretty good job of teaching the mechanics, which is how things work and the mindset. And we're going to really emphasize that you profile and benchmark everything and try to optimize for efficiency. These things do transfer to a larger scale. Now, the intuitions about what modeling decisions and what data decisions do work might not necessarily transfer across scales. For that, you actually have to go somewhere where you can do things at scale. So on the note of intuitions, it is worth remarking that some design decisions are just not justifiable, and just purely come from experimentation.

[00:08:35.039] Whereas mechanics, you can, by construction, see how the parallelism and how kernels are going to speed things up and so on. But for intuitions about what modeling changes work, I think you just have to run experiments. There's this famous Noam Shazeer paper that introduces the SwiGLU activation, which we're going to look about. And in the conclusion section, he very honestly has this final sentence which says, we offer no explanation. We attribute their success, of these architectures, all else, to divine benevolence. So that, in some sense, is something that you just have to gain from experience. Final note about the bitter lesson, which I think has been circulating and people talk about it. I think there's a common misconception here, what it means. I think the wrong interpretation is that scale is all that matters. Algorithms don't matter. But that's not correct. The right interpretation is that algorithms that scale are what that-- or what matters.

[00:09:39.580] And you can think about very simply that the accuracy of your model is basically your efficiency times the resources. So efficiency is output over input, and resources is the input. And efficiency is actually, in some sense, way more important at larger scale, right? If you're doing a small scale experiment, if your run takes twice as long, maybe you just wait twice as long and then you come back later. But if you're doing things at scale, that could be hundreds of millions of dollars. And you definitely don't want to do that. Even like a 5% improvement might be a big deal. So in fact, efficiency is actually really, really critical. And I hope to bake that into your mindset as one of the consequences of this class. And empirically, if you look at-- there's this paper from OpenAI in 2020 that showed there's a 44x algorithmic efficiency on ImageNet between 2012 and 2019.

[00:10:40.950] And it's surely the case that hardware did get a lot better, but that also comes with algorithmic improvements. And of course, when you multiply them together, that's when you see a huge bump in efficiency and accuracy. So the framing with all that said is, what is the best model one can build with a certain data and compute budget? For pre-training, it's mostly, we're going to talk about compute budget, because we're going to assume that we have a lot more data than we have compute. But if you're in a setting where you're data limited or you have stashed away, actually, tons of B200s, then you might be data bound. So in other words, maximize efficiency. And we're going to see this theme come up throughout the class. OK. So next, I want to spend a bit of time talking about language models and a bit of a history and just contextualization before we jump into the more technical details. So language models have been around for a while.

[00:11:52.870] Shannon, back in the '50s was using language models to measure the entropy of English. And for a long time, N-gram models were used, actually, in machine translation and speech recognition systems. And they weren't the whole system, but they were an important part of making sure that you generated fluent text. I would say that the lineage of the Modern Language models comes from neural architectures. And so there is a bunch of ideas, I think, which are important to this development. So in the '90s, there was LSTMs. Yoshua Bengio, actually, wrote the first neural language model in paperback in 2003. This is actually not an LSTM. This was just a feedforward network that looked at the small context. And then there was a Seq2Seq modeling, which boldly said, we can actually compress a whole sentence into a vector. The atom optimizer attention mechanism, which was developed for machine translation. The transformer architecture, which built on top of that, which was also developed for machine translation.

[00:13:03.470] And then scaling up to a mixture of experts model parallelism. You see a lot of different architecture and also systems and optimizer ideas developed in the 2010s. And then by the late 2010s, I think, things were starting to get really interesting. So there was the ELMo and BERT. These are language models that were trained on lots of text, and then you could fine tune them on some downstream tasks like question answering, and it would show a huge improvement on that. So the model there was, take one of these models and then fine tune. And then Google had a paper that was really, I think, foreshadowing this view of prompt in, response out. So it was really, I think, OpenAI that really opened up the floodgates here by embracing scaling. So they had GPT paper back in, I think, 2018 or so. They scaled it up to GPT-2, and then they really figured out or embrace the idea of scaling laws, which we're going to talk about in a second, which enable them to train GPT-3, which was a much more massive, almost more than 10x, the largest model at the time.

[00:14:26.972] And it could show emergent behavior like in-context learning. At that point, Google was saying, OK, we need to do something too. So they trained a massive model. It turned out to be undertrained. And it turned out that their DeepMind, which was not integrated with Google at the time, had figured out optimal-compute, optimal scaling laws. So all of this was happening. And after GPT-3 came out, I think for many folks, this was kind of a wake up call. And at that time, there were a lot of early attempts to, let's try to replicate this. So there was a grassroots organization called Eleuther that created some open data sets and models. They weren't very large because they didn't have much compute. Meta first LLM, you could tell that it's a replication because it was like 175 billion parameters, but it was not a very good model.

[00:15:23.950] They ran into a lot of hardware issues. And then there was another, Hugging Face BigScience project. So these models were, I would say, not very strong. Then in the last three years, I think, the open model ecosystem has changed quite a bit, with Meta kind of leading the way with the Llama series of models. Llama, Llama 2, Llama 3. Mistral got in the game. And then a whole set of Chinese models. I think I'm missing some. Like, ByteDance has something, and I think Tencent probably has some other stuff. So it's hard to keep track of everything, but everyone's heard of DeepSeek and Qwen. So I think I got the main ones. But I think what's interesting and exciting about these is now we have open weight models that are approaching closed models.

[00:16:14.540] So depending on who you ask and how you benchmark, they might be a little bit behind or comparable. But they are definitely very, very credible models that are being widely used in industry. Now there's another line of work, which is going beyond just releasing open weight models. AI2, NVIDIA and the Marin project, which I work on, we try to provide not just the weights, but the paper and the code and the data so that we can understand how these models are built in a more thorough way. Why do I emphasize this open ecosystem so much? Mostly because this course would not be possible, I think, without these models by the fact that there are still many papers that are being published about how these big MOEs and RL systems are working enables us to at least glimpse into how these frontier models are being built, and trying to triangulate the pieces.

[00:17:17.440] Now, I think, of course, a lot of the details, even with these Qwen papers, I think they're missing a lot of details. So can't reproduce them. Notably, the data mixture is something that we don't, but I think it's much better than nothing. OK. So in the last decade, I think, the idea of what a language model has changed, right? It used to be something that you fine-tuned, and then it was something you prompt. And now in the ChatGPT era, it was something you talked to and that you can have a conversation with. And now-- OK, I guess I don't have internet. [CHUCKLING] That's fine. And now, we're in the era of agents. If you click on this link, it basically shows you a giant agent trace. And I'm still-- it's mind boggling how strong some of these models are.

[00:18:10.940] You give it like a page of text and it does some really complicated agentic coding task. So what we demand of our language models today is probably beyond the imagination of any one back 10 years ago. That said, I think the fundamentals, I think, haven't changed that much. Largely, we still built on GPUs and kernels. We still optimize using gradient or stochastic gradient-like approaches. We still have the transformer in attention, And we'll talk a little bit more about architectures, but it hasn't changed that much. I think the specs are different. Now we demand greater context lengths, which means that inference efficiency matters even more. So the good news for us is that we didn't have to completely change this class. Only Tatsu's section on the latest Chinese architectures. [LAUGHTER] But the fundamentals, I think, are here to stay, at least for now.

[00:19:10.960] OK, maybe I'll pause there in case there's any questions or thoughts. OK. So let's go on. So a brief interlude. So what is this program? So this is what I call a executable lecture. If you think it looks like a Python program, it is because it is actually a Python program, but it has been rendered for your viewing pleasure. So when I step through it as actually executing the lecture. And it makes us possible to basically step through code, which will be, hopefully, interesting and important later. And you can also see the hierarchical structure of this lecture. For example, we're done with this function. Now, we go back to main. OK, so let's talk about the course logistics, and the syllabus, which I think is going to take a good chunk of time. So all the information is online at this website or cs336.stanford.edu. This is a 5-unit class. I think this, probably, class has a certain reputation, so I don't need to belabor the point too much, but this is-- we have five assignments.

[00:20:32.430] They are pretty intense. Even the first assignment according to this one. Review was actually equivalent to the 5 assignments. First, CS 224n. I've been told, also, this is exaggerated, but better to, I guess, be conservative in your estimates. So why should you-- so why should you take this course OK, so first you have an obsessive need me to understand how things work. That should be your primary objective. I think just pure curiosity on how language models work. And then in doing the class, you'll develop a much stronger research, engineering muscles and have the confidence to go into a new setting and be equipped to deal with whatever situation comes up. So I think when I started at Stanford, I created this class called Statistical Learning Theory, which was, basically, teaching the theoretical side of machine learning. And that was nice because it equipped people.

[00:21:34.790] So when you read a paper, you can understand all the math. And now, since the field has shifted a lot more towards this more systems and empirical side of things, this is the kind of analogous class, which gives people enough depth that they feel that everything else seems kind of easy. So why should you not take this course? This is important because there's reasons you shouldn't take this course. First, you actually want to get some research done this quarter. You should probably talk to your advisor. They should know you're taking this course. Otherwise, there'll be-- probably, there might be some surprises. You're interested in learning the hottest new techniques in AI. I think there are many other great courses, seminar courses, topics courses that are good for that. We don't do many of the things. We don't do multimodality. We don't talk about agents in any depth. So if you want to learn about that stuff, this is not the right course for that.

[00:22:32.010] If you come in and say, I have an application domain, I want to get good results on it, probably this is not the right course, at least to start. I always recommend just prompt a model, fine tune a model. And then as a last resort, you pre-train your own model. Because it is a pain and it's also expensive, but it's a lot of fun. OK. So if you are not taking the class, you can self-follow along at home. So all the lecture materials will be posted on the website. Oops! And they are also recorded through CGOE. So thank you, CGOE, for doing that. And later, they will be published to YouTube. Now, of course, following at home and watching the lectures is great, but you learn, really, by doing the assignments. So you'll have to figure out how to motivate yourself to do that. OK, so speaking of assignments, we have five assignments. The philosophy assignment is, how do we do from scratch, but not just, say, build a language model, and that's the assignment?

[00:23:42.940] So we don't provide scaffolding code, but we do provide a bunch of unit tests to make sure that whatever you're building is actually correct, so that you don't get this sparse reward setting where you submit a homework and it's either correct or not. What I recommend is that you can-- the assignments are structured so that much of the assignment can actually be done locally on your laptop. You can implement it and check for correctness. And then we are providing a cluster so that you can do an actual training run to see what the accuracy is, or get a bunch of GPUs to actually benchmark the performance of some kernel. And then for fun, we will have, some leaderboards for most of the assignments, at least. And they will look something like, well, now that you've learned about this topic, I'll try to minimize the perplexities given some of budget.

[00:24:42.380] And both Marcel and Herman were masters of leaderboards back in the day, which was last year, or the year before. So if you want tips on that, I'm sure they would be happy-- actually, I don't know, if they would tell you their secrets, but you can try. OK, so last year, we were thinking about, well, what can AI do? It was like, OK, well, I mean, yes, everyone can use AI, but just try to do your best. I think now, I think, coding agents have gotten so good that they can just all solve all the assignments, right? But to state the obvious, obviously, you're not going to learn anything if you just feed in the PDF Assignment 1 into a Cloud Code. At the same time, AI can be very useful for answering questions and tutoring.

[00:25:35.160] So we have to find a way to leverage AI. So what we've decided to do is, we provide you a AGENTS.md file, or equivalently, a prompt which asks the AI to be pedagogically minded. You can read more about it in our AI policy guide. And the requirement is that if you're going to use AI, use it with this prompt. And so it will answer questions about code, it will clarify any understanding, but it won't accidentally generate the transformer for you when the homework is to implement the transformer. And this is the first year we're trying to do this, so please try it out and give us feedback if it's working or not working. So compute. So this year we have-- thanks to Modal, they have provided us with compute credits on their platform.

[00:26:33.860] It's actually quite nice. You can get a number of-- unlike last year-- well, I guess, you guys didn't take the class last year. Last year, we had a cluster you SSH into. This is using more of an API to-- which I was initially skeptical of, but looking at it, actually, is pretty pleasant to use. So again, try it out and give us feedback on how it is. We've written a guide on how to access and use the compute. OK, any questions about course logistics? OK. All right, let's talk about what we're going to cover in this class. So there's, basically, five parts mirroring the five assignments that you'll have. Basic system, scaling, laws, data, and alignment. So I'm going to now go through each part and just give you a taste of what you will learn. So in the basics, this is basically the first two weeks. The goal is to just be able to train a language model and build it from scratch. So the components here are, we're going to tokenize the data, we're going to define architecture, and then we're going to implement optimizer and train it.

[00:28:00.500] So then you wonder, what are the rest of the classes for? Well, we'll get to there. So let's start with tokenization. The tokenization is really about, what are the atoms that the model operates on? And formally, a tokenizer converts between raw inputs, which are just bytes, and a sequence of integers, which represent the tokens. Conceptually, it's a segmentation of the text. We're going to talk about the Byte-Pair Encoding, BPE tokenizer, which intuitively, breaks the input into frequently occurring chunks. And then remember, this class is about maximizing efficiency, so through an efficiency lens, tokenization is good because it takes a long sequence, if you just think about the raw byte stream, and reduces it into a smaller number of tokens. But more subtly, but maybe more importantly, it allows you to do adaptive computation. So maybe some places are actually a lot of bites, but actually, it should be compressed into a one token, whereas some of the more rare or interesting parts of input should be left as multiple tokens.

[00:29:15.510] OK, we'll talk more about this. I just want to mention that every year, I'm hoping that I don't have to teach tokenization, because the dream is to really have an end-to-end way that directly operates on bytes. And there's been a number of work, including recently, there's been these networks that seems promising, but so far, these have not been scaled to the frontier. And since the frontier models are still using tokenizers, we felt like it would be still wise to teach tokenizers. OK, so now after you tokenize your input, you have a bunch of tokens. Now you define a model on top. And everyone, I think, has a familiarity with the transformer. And if you've taken 224n, the MLP class, then you've seen transformers. Since then, I think, there have been a lot of improvements or refinements to transformers, which I think are important. And Tatsu is going to talk more about this in a bit. But just to run through a set of types of things that one might have to think about, so the activation functions have evolved.

[00:30:31.750] How do you do positional encodings has evolved. How you normalize has the different layers. Blow up has evolved. Instead of doing full attention, there's many ways to basically reduce the attention computation, because attention is n squared and where n is a sequence length, and that gets really expensive. So there's a bunch of ideas around that. If you're more ambitious, you can look at these state-space models or equivalently linear attention like Mamba and Gated DeltaNet. These have become more popular in the last few years. And usually, some hybrid between these models and attention seems to work quite well. So, we'll be exploring some of that. And then within the MLP layers of the transformer, the original transformer was just a dense MLP, and now a mixture of experts has become the dominant paradigm for building compute efficient transformers.

[00:31:34.970] So we're going to talk about that. And of course with a mixture of experts, it's not just defining architecture, but we'll see that we'll also need different techniques for training the model. And then finally, perhaps somewhat boringly, but an important question is, what is the shape of your transformer? How many layers? How many heads? What is hidden dimension? Number of experts? This might come in more as we talk about scaling laws, but setting these is actually-- seems kind of almost trivial. It's a hyperparameter, but actually in the context of scaling and language models, it has a huge, huge implication. So once you define your model architecture, how do you train the model? And here there's a bunch of design decisions around the loss function. There's next token prediction, which is the default. But people have found that predicting more than one token seems to be helpful for improving the model. And there's optimizers. People used to use AdamW, but increasingly, Muon has been used, especially with some of the latest open models, such as the Kimi K2 models.

[00:32:57.920] Initialization, which, again, sounds kind of boring, but turns out to have a huge impact on how your ability in the training stability of larger models. Learning rate schedule. Regularization. Batch size. And then MoE specific things. So you look at this list and you might think, well, these are just hyperparameters. I'm going to try a bunch of different options out. But it turns out that really being very careful about setting these hyperparameters in a principled way will make the difference between a run that just blows up and is useless between a run that is achieving state of the art. OK, I'll come back to that point when we talk about scaling loss. So then in Assignment 1, what you're going to do is you're going to implement the BPE tokenizer, implement the transformer, the loss function, the optimizer, the whole training step. We're going to make you do a bunch of resource accounting so you understand where your flops are going. You're going to train some models on these data sets like TinyStories and OpenWebText.

[00:34:06.280] And then there's going to be a leaderboard where you're going to try to drive down perplexity as fast as you can. So for those of you familiar with NanoGPT, speedruns, it's kind of similar to that. OK, so by the end of Assignment 1, you should be able to walk away and build a language model from scratch. So that's very exciting. If I have a high level takeaway here, it's that while the tokenizer and modeling and training are presented as distinct pieces, it is actually, everything is about balancing the following. So you want expressive models because you want to represent the complexities of the data, but at the same time, you want your training to be stable. And we're going to talk a lot about how do you keep the parameter and gradient norms in this goldilocks zone so they don't blow up and don't vanish. Turns out, a lot of training language models is about just stability. And then finally, efficiency, which is somewhat more straightforward. You just make it run fast on hardware. But you're going to see interesting things like, if we change the architecture, a lot of the architecture decisions are, well, we can make it faster by, let's say, reducing the projecting of a low dimensional space.

[00:35:41.860] But then the question is, does it work as well? And so making those trade offs is something that-- is the name of the game here. So in Assignment 2, we're going to dive more deeply into systems. And the goal here is just to get most out of your hardware. So we're going to talk about kernels, how you parallelize across multiple GPUs, and how you do inference. So the basics, which we're actually going to start on next lecture, I mentioned resource accounting, and I think you've all probably built models before. But this is really about keeping track of where all the flops go and where all the memory is being spent. So we're going to spend some time, basically, doing the resource accounting. We're going to see this formula that comes up, how many flops does training SMDB on model on 1 trillion tokens? Well, it's six times n times D roughly. And where does that come from?

[00:36:53.850] And then we're going to look at the hardware. And here's a cartoon I picture of what to remark about hardware, is that, your memory is not where your compute is, and you have to move either your parameters or activations from the memory to compute, do the compute, and move it back. And that often is the bottleneck. So for example, B200, which we'll have the opportunity to play with, it has 2.25 petaFLOPS per second if bf16. And has 8 terabytes second of memory. So what does that mean? I think when we-- I think I'll do this next lecture. We're going to break this down and use this information to do some calculations and see how long different types of algorithms will need to take. We're going to talk about roofline analysis, which allows us to understand whether a computation is bottlenecked by either a compute or memory. In general, it is memory. And then talk a little bit about benchmarking and profiling.

[00:38:08.896] So here's what a DGX B200 looks like. You have 8 GPUs. They're connected via MV link. And then if you have many-- if you have a thousand GPUs, then you would have multiple of these and they're connected either InfiniBand or ethernet. So the next two parts when we talk about system is kernel. So a kernel is basically a function that runs on the GPU. And when you're just using plain PyTorch, all the PyTorch primitives actually correspond to launching particular kernels, which are built in. So you're already using kernels, whether you it or not. But the point is that, for certain types of computation, if you look at it, you can actually write custom kernels to make the GPUs go faster. And the main principle here is organizing the compute to minimize data movement. So remember, this picture moving data from memory is expensive. So you want to try to minimize that.

[00:39:21.970] So just as a simple example, suppose you wanted to compute A and B. So often, you would have to read from your high bandwidth memory, HBM, compute it, write it back, and then read again, compute it, write it back. So then, you're basically sending the data back and forth twice. And there's this idea called fusion, where you read it once, do both of the computations, and you write it back. And that will save you a lot of time. So that's operator fusion. Tiling is a more sophisticated variant around the same idea. There's also-- GPUs have also gotten a lot more complicated. And I'm not sure how many of these details we'll have time to get into, but at least we want to expose you to some of the peculiarities, I would say, of GPUs and give you an appreciation for the types of things that one has to consider in order to squeeze as much juice out of them as possible. And we'll write some kernels in and Triton. So what happens if you have thousands of GPUs? So the principle of minimized data movement is still the same. The only thing is that moving data between different GPUs is even more expensive.

[00:40:44.420] We're going to talk about how these very classic collective operations, gather and reduce and all-reduce are the way to think about, basically, distributed training. The general game is that we have these model parameters, We have activations, gradients and optimizer states, and they need to be sharded or split across multiple GPUs. And of course, you need to bring the right data to the right nodes to make the compute and write it back. So there's a whole kind of orchestration, and how to do that efficiently is going to be the topic of this unit. And there's multiple ways to shard. You can shard by splitting up your data, splitting up your model, splitting up different layers in the model, splitting up the sequences, splitting up between experts. And we'll talk about the trade offs that come with each of these. OK. And then finally, we're going to talk about inference. Which, as I mentioned, is growing in importance. So the goal of inference is to actually use the model.

[00:41:49.640] So minor detail here. So inference is, of course, you need to use inference when you're chatting with a model, but it also is useful for reinforcement learning. It's useful for doing the rollouts. Test-time compute, generating synthetic data, evaluation. So inference is a very critical part of what it means to do language modeling work. So we're probably not going to spend as much time as I like on this because the course is already kind of filled up, but we'll see what we can do. There was some discussion around whether we should make you write inference from scratch, but we'll see. So the way to think about inference is that there's two phases, a prefill and a decode. In the prefill, You take the prompt and then you feed all the tokens forward and build key value pairs. This is very much like what happens in training. And then in the decoding part, tokens are generated one at a time.

[00:42:55.380] And this is the part that becomes quickly memory bound, and this is why inference is hard. And so there's many things you can do to speed up inference. You can try to use a cheaper model by pruning a larger model or you can quantize, you can distill. You can use this technique called speculative decoding, where you use a cheaper model to run ahead and guess a bunch of tokens. And now, you use the full model, which can operate on those tokens in parallel to see if it's good. And if you got lucky, then you can accept all those tokens and you're much faster than if you're doing one token at a time. And of course, you can do systems optimizations. There's a bunch of kernels that are designed specifically for inference. And then one of the interesting things about inference is that, if you're running a service, queries are coming at you at potentially different times, and then you have to figure out how to batch them up.

[00:43:54.780] Whereas in training, you basically already define your batches and everything is much more predictable. So in Assignment 2, there's going to be implementing kernels in Triton and doing some of parallel training. The details here might actually change. As the CAAs have grand plans for revamping the systems. So the assignment might look a bit different from last year's, but it will cover roughly the same material. One thing I will mention is that there's this wonderful book out of some Google people called How to Scale Your Model, and I think it's really nice for providing a conceptual understanding of roofline analysis, and transformer math, and doing LLMs conceptually. So I highly recommend you take a look at that. Now, the only thing is that it's from Google, so it's about TPUs. But a lot of the high level concepts are similar. And now, they have a new chapter on how to think about GPUs.

[00:45:09.510] OK. So the third assignment is about scaling laws. So by now, you've trained a language model. You can make it go really fast by optimizing kernels in parallel. Now, you want to scale up. So how do you scale up? So imagine the following setting. If you had 1e25 FLOPs, so this is tens of millions of dollars of compute, what model would you train? So this is, I think, a daunting task because if you mess up, well, that's a lot of money down the drain. And you can't do your, probably, typical hyperparameter tuning at that scale because you only get to train one model. And so this is the key problem that you have to deal with in large language model training that you don't have to really deal with if you're just fine tuning a model or doing small scale stuff.

[00:46:11.870] And so the key conceptual shift here is that we shouldn't think about a single model that we're training, but really think about a scaling recipe. And a scaling recipe is a mapping from a FLOP budget, let's say 1e25 or 1e24, to a set of hyperparameters. Basically a config file. And for a given scaling recipe, what we will do is run a bunch of experiments to compute the loss that you get at smaller scales, and then you fit a scaling law, and that enables you to predict the loss at a target scale. So maybe you run some small experiments, you fit a scaling law, and then you predict out what you're going to get at a larger scale. So that's the primitive. Now using this, what you can do is now you can optimize the scaling recipe, targeting a larger scale using smaller scale experiments, which is wonderful.

[00:47:09.910] And second of all, you can predict the loss that you're going to, in theory, achieve before actually running the experiment. Which allows you to go raise money. And you say, well, look, I ran the small scale experiments, and I think I can get really-- like, a GPD-5 level model. Please give me a lot of money so I can train that model. But one thing I think is maybe another misconception is that scaling laws are not laws of nature. They don't just happen automatically. You have to will them into existence. And this happens by careful construction of a scaling recipe. And the scaling recipe, remember, it has to extrapolate. So what this typically means is that you have a sequence of hyperparameters which, say, as the scale increases, maybe the learning rate is a constant, maybe it drops, maybe the batch size increases by how much. And these are things that a scaling recipe has to figure out. And so you parameterize. So then in order to get these predictable scaling laws, one thing that you actually have to think about is how do you parameterize the model in a way to get what's called hyperparameter transfer?

[00:48:31.080] Meaning that the hyperparameters you use at small scale are either the ones that you use at larger scale or are predictable functions of it. Because if every scale, your learning rate is sometimes 1e-5, and sometimes 1e-4, then you're not going to be able to magically guess the right learning rate when you are at larger scale. So one shift in thinking is that predictability is actually at least as important as optimality. So you normally think, oh, we're trying to optimize for efficiency here, and we want to hyperparameter tune and make things optimal. And yes, you do want to do that, but you also want this predictability so that you don't get surprised at larger scale. So this is state setting. The actual scaling laws we're going to look at are fairly classic. So these are-- some of you might have seen this idea of, well, if I give you a FLOPs budget, should you train-- how should you balance training a larger model versus training on more tokens?

[00:49:36.000] And this is where the classic compute optimal scaling laws from Kaplan et Al and the so-called Chinchilla scaling laws comes in. The basic idea is that you for each FLOPs budget, so let's say 6e18 all the way to 3e21, you sweep across different model sizes and you choose the best one. So think about minimizing each of these. And then you fit a curve that allows you to, basically, predict the number of parameters given a FLOPs budget. And if you're lucky, it will rely roughly on the line. And if you're unlucky, it's going to be all over the place, which means that you should have no confidence that you're going to be able to predict reliably. And so the upshot of this, this is quite crude, but a rule of thumb is, the 20 times the number of parameters is the number of data points you should train on. So a 70B parameter model should be trained on roughly 1.4 trillion tokens. Of course, depending on the data set and architecture, this number will actually vary. Also, this doesn't take you into account the inference cost.

[00:51:01.040] A lot of models these days are small, but they're trained on way more tokens than is compute optimal because you want a smaller model for inference reasons. So one fun thing that we've been doing in the Marin project is pre-registering our results. So we fit a bunch of scaling plots at different compute budgets. We fit a scaling law, and we basically made these predictions out to 1E22 FLOPs. This one is actually training. If you go to the Marin website, you can follow along. It should actually be done maybe as early as tonight. So maybe on Wednesday I'll report back on how we did and see how we measure-- how we match the pre-registered loss. So the idea here is that we made a prediction on if we were to train this large model, which we've never trained before.

[00:51:53.640] If we can predict how well it's going to do, then that's really nice. So in Assignment 3, I think this is either fun or actually, it's a fun-- let's just say it's a fun assignment. [LAUGHTER] So we're going to define this training API, which basically, you give us hyperparameters, and we're going to give you a loss back. So what we're going to try to do is simulate what happens if you could do a lot of training runs. And of course, we don't have enough compute to actually allow everyone to train their own API model or anything. So basically, we trained a bunch of models offline, and we, basically, provide this cache. So it looks like you're training. And what you're going to do is submit training jobs. You give us a config, basically, we give you a loss back and then you could do whatever you want.

[00:52:50.630] We would recommend you fit scaling laws to these points. And then you extrapolate. And then we give you a budget, and we basically evaluate how well your model landed. So it is meant to-- I was going to say, it's meant to replicate the high stress scenarios if you actually had a budget like $100 million that you need to spend. You have to be very careful about how you spend this. Of course, this is low stakes. So at this point, you will have, you train a model, how to make it fast, you know to scale up. Now, what's missing? What do you train the model on? And that's going to be the subject of the data section, which is arguably one of the most important things because data quality basically specifies how good your model is going to be. One way to also frame it is, what do you want your model to do? Data, basically, reflects what your model wants. So do you want to speak multiple languages? Be good at having a conversation?

[00:54:13.590] Do you want to run long agentic coding tasks? And so part of that is also going to-- so we're going to start by talking about evaluation, which, basically, defines the capabilities that you'd like your model to have. One thing we'll talk about is, evaluation is a fairly deep topic. It's not just about running on some benchmarks. There are internal evaluation metrics for model development. And what matters here is that smoothness across scales so that there-- remember, we want things to be predictable. Relative performance matters. You don't care, necessarily, how well this does in absolute terms, because, let's say, a perplexity number is just like, what is a perplexity of 1.2 on some held out data really mean? And then there's external metrics. These are things that you report to your customers or your reviewers or whoever you're presenting your thing to. And here, ecological validity really matters.

[00:55:26.170] And I think sometimes these two things get conflated, but I think they really serve two distinct purposes. And so you can think about perplexity as something that is really helpful for internal development. And still to this day, perplexity is a very good way of capturing the intrinsic quality of a model without getting-- worrying about benchmarking. Now, there's a separate question of what you run your evals on. And the recommended thing is, well, if you have some data that's not in the internet, that would be good because you can avoid contamination. OK. So and then there's advanced use cases which are more representative of external facing use cases. So one thing to also note is that language models are purportedly very general purpose. So it's only fitting that we actually need a very diverse set of evaluations. And I would always recommend having many evaluations that look at-- you can average them into a single number, but often that average conflates a lot of different things.

[00:56:40.530] So now after we set up the evals, we know what we're building. How do you get the data? Well, first thing is that data does not just fall from the sky. It has to be actively curated. Often, I think, especially in classes and also in research, sometimes you're just given a data set, and then it's like, OK, well, now, I do stuff on the data set. But a lot of language models, especially if you want to collect these large data sets, you have to go and actively look at it. So web pages are crawled from the internet. There's books, I guess, which is controversial at this point, but arXiv papers, GitHub code, and so on. This is an old figure from the pile from 2021. And you can see, language modeling data sets are fairly diverse. There's, especially these days, I think, a lot of contention around, is it fair use to train on copyrighted data? Maybe sometimes you have to license data and so on.

[00:57:44.920] So there's legal issues around data which, I think, are quite important. For example, a lot of GitHub code doesn't have a license. So how do you interpret that? Do you assume it's permissive, or do you be conservative and assume it's not permissive? So there's also-- the fact is that data is not even text. It's either HTML or PDFs, or a code is directories. And this requires processing to turn it into actual text to be usable for training. So that's the topic of data processing. There's a few steps that has to happen here. Transformation, converting some nontexting in a text. Filtering, keeping only the good stuff. If the random internet document in Common Crawl is extremely bad and you don't want to train on it, most likely. You want to deduplicate. There's multiple sources, or how do you combine the different sources? And then finally, more recently, there's been a lot of work on generating synthetic data, which could mean taking the real data and just rewriting it into things that are more the downstream task or just more Wikipedia like, or whatever you want.

[00:59:09.580] So this is an active area of research. Also, data can be used both for pretraining. Something called mid training, which is usually the high quality data that you put at the end of the pretraining step. And this includes long context data such as maybe BigCode repositories or books. And then finally, there's post-training data, which are maybe conversations or agentic traces with tool calling. So in Assignment 4, we're going to make you start with a very raw corpus, like a rock web crawl, and do all the work to filter and to dedupe, and make the data clean. So this is, I would say-- I don't know if I would call it not fun, but it is certainly, I think, a lot of what people would call dirty work. But that is, I think, an important part of building a language model from scratch so you have to get the full experience. So finally, alignment. So far we've, basically, trained a model using full supervision.

[01:00:30.000] Predict the next token or the next few tokens. Now at this point, the model should already be reasonable, but we can improve it further by using weak supervision. And why weak supervision? It's because sometimes it's easier to critique than it is to generate. So you can't always have data that says, this is the right response to this prompt, but maybe you can have a way of specifying what good looks like. So then, the basic template is that you generate responses from the model. You score them either with a human or verify or a LM judge, and then you update the model to prefer better responses. This can be instantiated either through various RL algorithms such as PPO or GRPO, or in a simpler, for preference data, DPO. So the challenges around RL are that RL algorithms are unstable and hard to tune.

[01:01:32.510] Some of you probably know this from firsthand experience. Personally, I prefer to keep things as much in the fold so we supervise the case as long as possible. And then finally, OK, fine, I have to do RL. But some people, for whatever reason, like doing RL. Also, what we'll, hopefully, talk about this year is that if you do RL at scale and try to maximize your throughput, there's actually a lot of systems challenges. You have to have an inference server and a training server. And then the inference server has to generate these rollouts. Especially if you do RL against environments that involve code execution. It's a whole kind of orchestration game. And then if your workers lag behind or then you get into off policy issues, and then you're constantly kind of juggling this on-policyness with a desire to maximize throughput.

[01:02:28.710] It's a big, wonderful mess, which, hopefully, we'll talk more about when we come to that. So Assignment 5, we're still deciding what exactly we want to do. Last year, it was implemented DPO and GRPO, and get it working for some math benchmark, but we'll see how much farther we can push it on the realistic dimension this year. So again, remember it's about efficiency. And efficiency can be either data efficiency or compute efficiency. So the way to think about it, you have all these resources. you have data, you have the hardware, which has compute cores, you have memory, communication bandwidth. And you're just trying to figure out how do you build the best model according to some evaluation given a fixed set of resources. So through this lens, I think, you can actually think about a lot of these design decisions as optimizing for this.

[01:03:29.490] So systems, clearly, that's about compute efficiency. Tokenization, as I mentioned before, you can't just work with raw bytes, but that's going to be very compute inefficient, at least with today's model architectures. And so a lot of tokenization is about improving compute efficiency. Model architecture, many of the changes that we'll see are motivated by reducing the memory or flops. In fact, a lot of them are influenced by the need to have faster inference. Data filtering, you can also view as-- through efficiency lens. We don't want to waste time updating gradients on a bunch of redundant bad data, even if it might not hurt you, but it hurts you in the sense that if you have a fixed compute budget, more time on bad data means less time on good data.

[01:04:21.270] And then finally, scaling laws is explicitly about how you can, essentially, do effective hyperparameter tuning on much smaller models. And now tomorrow, we might become data-constrained, and the calculus of what design decisions you should take might change. But I think the overall mindset which we're trying to teach you is think about the efficiency of your approach. OK, let me stop there. Are there any questions about any of these assignments or topics? OK. So now let's do tokenization. So this is-- we're jumping into our first unit here. So Andrej Karpathy has this really good video on tokenization. You should check it out. So starting point is raw text is what is text? It's Unicode strings. And on the other hand, the language model places a distribution over sequences of tokens usually represented as indices. So we need a procedure that encodes these strings into tokens, and also a procedure that decodes tokens back into strings.

[01:05:44.800] So a tokenizer is basically something that can do this round trip. So here are some examples to give you a flavor for how tokenizers work. Actually, I should have tried to get in there earlier. So this is not going to work. If you go to the site you can play around with different tokenizers. So some observations here. And you'll appreciate why tokenizers are kind of annoying and why people want to get rid of them. So a word and its-- a word conglomerate with its preceding space are different tokens. So many tokens you'll actually see are space a word, which is fine, but kind of strange. So this hello and this hello is actually two completely different indices that have nothing to do with each other.

[01:06:40.560] And sometimes, depending on the tokenizer you use, numbers are represented with-- every few digits is a token. Sometimes, it's predictable, and sometimes, it's not. Some tokenizers try to make every digit a token, but then you're blowing up the number of tokens you have. So there's some trade off there. So here's the GPT-5 tokenizer. So you can take this string and convert it into these indices. And then you can decode it back into the string. And so tokenizers should round trip. If you implement tokenizer, it doesn't round trip, you have a problem. So the compression ratio here is the number of bytes per token. So in this case, we have, the number of bytes of this string is 20. The number of tokens is 8. And you do 20 divided by 8. So the compression ratio is 2.5. OK, so 2.5 bytes per token. The larger the compression ratio, that means the shorter the sentence, which is good because attention is quadratic, and you want to make sure the sent sequence is shorter.

[01:07:56.200] Now, you could obviously increase the compression ratio by increasing the vocab size, but then, you get into sparsity where more and more-- because every element of vocab is treated like a distinct element. So these days, tokenizers, especially multilingual tokenizers, have 100k or 200k tokens, distinct tokens. So you can look at the GPD token vocabulary. I think we'll skip it in the interest of time. So how do you build a tokenizer? So I'm going to go through this fast. So the first thing you might do is like, well, Unicode string, that's a sequence of Unicode characters. And each character is an integer, which you can call ord in Python, and you get some number out. And this can be converted into characters. So let's just build a character level tokenizer which basically breaks up each character and encodes it in a token. And then this can decode back.

[01:08:59.720] Life is good. So now, there are 150k Unicode characters. So your vocab size could be 150k, which is a lot. I mean, it's not crazy, but I think the bigger problem is that many characters are actually rare, which means that it's really an inefficient use of a vocabulary. And also, the compression ratio, which reflects this is not that great. So most of the time, you're actually using a lot of tokens to represent your sequence. And many of the indices are actually not being very used. So this is not a very good tokenizer. So here's another attempt. So you can turn strings into bytes. So Unicode has a UTF-8 encoding, which means that you can-- sometimes a string like "a" is just 1 byte, and sometimes, a string is multiple bytes.

[01:10:05.930] So let's build a tokenizer around that. So we can take this string and convert it into a sequence of bytes. And notice that this is a longer sequence now, but all the numbers are between 0 and 255, because that's what a byte means. And the compression ratio is 1, which is not great. So byte sequences can be very long, but the vocab size is small. OK. So both of these are really bad. So let's try to make some progress. So this is what actually people used to do in NLP, if people remember. So if you take a string, I can just chunk it up into-- break it up by spaces or some regular expression. And let's just call each of these chunks a token. So what is good about this one is that each token is meaningful because humans invented words, and words tend to have a stable semantic meaning. But your vocab size is the number of distinct chunks in the training data, which could be a lot.

[01:11:25.010] And also, your compression ratio, I mean, it's quite good, but the vocabulary can be huge. Actually, it's worse than that because though the vocabulary could be actually unbounded, right? Because at test time, you might get some sequence and you tokenize, and then you have a token you've never seen before. And people used to assign these UNK token, but that's really ugly and can mess up your perplexity calculations. So this is also not great. OK, so what we're actually going to do is called byte pair encoding. And this was introduced a long time ago for data compression. Way before language models were really on the scene, really. It was first introduced to NLP for doing neural machine translation. And the first paper that used BPE for LLMs was GPT-2. So the basic idea is that you're going to train the tokenizer on raw text to construct a vocabulary that's tailored to the data.

[01:12:36.072] And you're also going to have this property that everything becomes-- can be tokenized. If it's rare, then it just breaks up into smaller units rather than having this UNK token. So common sequences are going to be represented as one token. Rare sequences are going to be split into multiple tokens. That's the idea. So the algorithm is fairly simple, conceptually. So you start, basically, with your corpus. Let's assume it's one long sequence. You get a byte sequence. Each byte starts as a token. And then we're going to merge successive pairs of adjacent tokens that occur the most frequently. So let's step through how this is going to work in code. So here's a simple string, "the cat in the hat," and here's the implementation of the BPE algorithm. So we're going to turn that into a sequence of bytes. And then we're going to-- first, we're going to, basically, count the number of times successive tokens appear.

[01:13:42.320] So 116, 104 shows up twice, so we get this. And then we're going to find the pair that happens the most number of times. So that's 116, 104. I guess there's a few ties, but we'll just take the first one. And then we're going to merge that pair. And by merging that pair what we do is we create a new token. In this case, this is going to be called token 256. It's going to represent this pair, and we're going to add it to our vocabulary. So 256 is going to represent the sequence of th. So t and h have been merged. And we're going to call-- every time we see th, we're going to use 256 to represent that. And then we go through indices, and then we replace every occurrence of 116, 104 with 256. So those two places have been replaced. And then we iterate. So the next time we do this, we're going to find 256 and 101. We're going to merge that. And now we have 257. And then we're going to merge that one more time, and we're going to get 258.

[01:14:58.360] So over time, the sequence is shrinking and the vocabulary size is growing. OK. So let me-- and the compression ratio here is-- that we get for this, I mean, this toy example is 1.5 OK. So now that you have a tokenizer, how do you tokenize new text? Well, you take a new string and you encode it. And conceptually, what happens is that you basically go through the set of merges that you've made, and then you just apply the merges to your string. OK, let me actually not step through that code. So that will give you a sequence. So this is a sequence encoding of "the quick brown fox." And then when you decode it, you get the same thing back. So I went through this a bit fast, just in the interest of time. I will say that this implementation works. This is a full-blown BPE implementation. It's extremely slow. So in Assignment 1, we're going to ask you to, basically, make it faster.

[01:16:22.980] So currently, encode loops over all the merges, which is very slow because you might have-- the number of merges you have is essentially the vocab size minus 256. So you only want to loop over the merges that matter. And you have to build some indices to make that happen. There's some details around special tokens. Conceptually not deep, but important to building a modern tokenizer. Another thing is that I've presented the tokenizer just for simplicity. As you take an entire string and then you try to tokenize it, really, what happens is that you break it up into-- your text into chunks, and then you apply tokenizer on each chunk. So that's going to be much faster. And then try to make it as fast as possible. At some point you might realize that Python is just not very fast. And if you want to implement it in your favorite language, Rust or C or something, then go for it.

[01:17:29.190] OK, so quick summary. Tokenizers convert between strings and tokens or indices. The previous character-base, byte-base, word-base are highly suboptimal in their own way. BPE is effective heuristic that is data driven. So it seems to be pretty effective. Now, like I said before, maybe next year, I don't have to teach this, but for this year, we're stuck with tokenization. Even if we get rid of tokenization though, I think whatever solution replaces it, I think, has to satisfy the following properties. If you have the model, the transformer needs to operate on some sort of abstractions of the sequence. And this is most evident if you think about not just text, but video or DNA sequences where the individual bytes or units are actually quite low signal to noise, and you have to do some sort of abstraction to lift it into a place where you can do modeling on that.

[01:18:41.750] And then finally, as I mentioned, chunks should be variable. You want adaptive computation. Not all bytes are treated the same. And if you don't do that, I think, you're going to be suboptimal. So any end-to-end solution also, I think, has to have these properties. OK. So with that, I will end. Next time, on Wednesday, we're going to start the unit on resource accounting. Which is sort of a baby system, I would say. And then after that, we're going to go back into architectures and go from there. All right.

Overview, tokenization

课程简介 ​

CS336 语言模型从零开始 · 第一讲笔记 ​

学习目标 ​

课程主线 ​

第一部分：课程导论 ​

1.1 为什么开设这门课 ​

1.2 工业化带来的挑战 ​

1.3 可迁移的知识类型 ​

1.4 苦涩的教训（The Bitter Lesson） ​

1.5 语言模型历史脉络 ​

1.6 语言模型定位的演变 ​

第二部分：分词（Tokenization） ​

2.1 基本概念 ​

2.2 朴素分词方法的缺陷 ​

2.3 BPE 算法详解 ​

2.4 实际 BPE 的优化方向 ​

2.5 有效的分词器应满足的性质 ​

关键原则与公式 ​

效率框架 ​

缩放经验法则 ​

训练稳定性原则 ​

课程作业概览 ​

核心术语表 ​

常见误区 ​

复习清单 ​

概念理解 ​

BPE 算法 ​

效率思维 ​

术语与公式 ​

从零构建语言模型：CS336 第一讲深度解析 ​

导语：为什么要从零开始？ ​

课程背景：为什么基础研究需要底层技术 ​

抽象是有漏洞的 ​

工业化带来的挑战 ​

两个关键例子 ​

三类可迁移的知识 ​

苦涩的教训的正确解读 ​

语言模型的演进：从数据压缩到智能体 ​

早年：从 N-gram 到神经网络 ​

2010 年代末：预训练时代的开启 ​

开源模型生态的兴起 ​

语言模型定义的演变 ​

课程结构：五个模块五个作业 ​

作业一：基础系统 ​

作业二：系统 ​

作业三：扩展定律 ​

作业四：数据 ​

作业五：对齐 ​

一切皆效率 ​

深入分词器：从字符到 BPE ​

为什么需要分词器？ ​

糟糕的方案：字符级分词 ​

也更糟糕的方案：字节级分词 ​

有语义但受限的方案：词级分词 ​

BPE：字节对编码 ​

分词器的未来 ​

结语：从理解到效率 ​

CS336 2026 Lecture 1：完整中文逐字稿 ​

正文 ​

CS336 2026 · Lecture 1 · Overview, tokenization ​