Data (filtering, deduplication, mixing, synthetic data)
- 日期
- Wed May 13
- 讲师
- Percy
- 课程资料
- 10 项
- 内容产物
- 4 项
课程简介
CS336 Lecture 14：数据（过滤、去重、混合、合成数据）
学习目标
完成本讲学习后，你应该能够：
- 理解数据流水线的完整流程：从原始 HTML/PDF 到可训练文本的转换、过滤、去重与混合
- 掌握基于模型的数据过滤框架，并能区分生成式方法与判别式分类器的适用场景
- 理解 MinHash 与局部敏感哈希（LSH）的原理，以及它们如何使近似去重在网络规模下做到线性时间
- 识别数据混合中"重复轮次（epoching）"这一关键陷阱，并掌握两种规避策略
- 了解后训练阶段合成数据（尤其编码领域）的生成范式与代表性工作
课程主线
本讲是"数据"主题的第二部分。上一讲讨论了数据的来源：实时服务（如 GitHub）→ 转储/爬取（如 GitHub Archive）→ 处理后的数据（如 The Stack），以及相关的服务条款、版权等社会考量。本讲聚焦数据进入训练之前的技术流水线，分为四个阶段：
- 转换（Transformation）：原始 HTML/PDF → 文本
- 过滤（Filtering）：从海量原始数据中筛选高质量子集
- 去重（Deduplication）：移除精确与近似重复内容
- 混合（Mixing）：确定不同数据源之间的权重分配
最后简要讨论后训练阶段（mid-training + SFT）的合成数据。
核心概念
一、数据转换（Transformation）
原始数据并非文本
即使你爬取了网页，原始数据也不是以文本形式呈现的。Common Crawl 内部存放的是 HTML，也有 PDF；GitHub 数据则是目录结构。因此第一步是将这些格式转换为模型可用的 token 序列。
HTML 到文本的处理
这是转换的主要关注点，因为互联网大部分内容是 HTML。处理方式基本是基于规则（rule-based） 的启发式方法：
- 去除样板内容（boilerplate）：导航栏、广告、页脚、页眉等
- 提取主要内容：页面的核心文本部分
关键认知：
- 有损过程：HTML 本质上是层级化的（或渲染后可视化的），而文本是线性的。将 HTML 线性化为 token 序列必然丢失信息
- 表格处理困难：简单表格可用 Markdown 渲染，但嵌套表格极具挑战，必须近似处理或放弃
- "内容"的定义模糊：某些导航元素可能有助于语言模型学习网页结构，因此"什么是内容、什么不是"并不总是清晰
为什么用规则而非模型？ 两个原因：
- 规则方法速度快，能处理海量数据
- 这项任务不需要太多智能，规则通常足够
但基于规则的方法有一定失败率，所以数据中会存在瑕疵。工具选择会影响准确性——例如在 DCLM 的扩展评估中，Resiliparse 的表现优于其他工具。
PDF 处理
PDF 在 Common Crawl 中存在，但只占一小部分。处理有几个额外难点：
- 许多 PDF 因文件过大会被截断，需要重新爬取
- 扫描版 PDF 本质是图像，需要用视觉语言模型（VLM）做 OCR——这比纯文本处理昂贵得多
- PDF 从设计上讲是排版格式，缺乏语义结构（如 HTML 的
h1、p标签），布局信息大量丢失
但 PDF 有独特价值：能费心制作 PDF，通常意味着内容比普通网页更有趣，所以 PDF 的平均质量高于 HTML。这项工作（FinePDFs）由 Hugging Face 完成，从 Common Crawl 提取 PDF，使用 OCR 工具（如 RolmOCR、Docling）转换，并进行大量清理。
二、数据过滤（Filtering）
一般框架
过滤是一个通用算法构件：
- 给定目标数据 T（通常是一小部分高质量数据）和海量原始数据 R（上一步转换得到的全部爬取内容）
- 目标是找到 R 中与 T 相似的子集 T′
通用方案：
- 基于 R 和 T 估计某个模型
- 推导出评分函数 score(x)
- 根据得分保留 R 中的样本（可设阈值，有时随机保留）
两类分类器
| 类型 | 方法 | 示例 |
|---|---|---|
| 生成式模型 | 直接在目标数据上估计模型 | KenLM（5-gram 语言模型） |
| 判别式分类器 | 预测文档属于 T 的概率 | fastText（线性分类器/词袋模型） |
判别式方法更常见：以 T 中的样本为正例，从 R 的随机子集取负例（可做平衡），训练分类器，然后对每一篇新文档打分并设阈值。
关键要求：
- 能从目标数据泛化：希望 T 和 T′ 不完全相同
- 极快：要运行在整个互联网规模的数据上（可达约 100 万亿 token）
过滤类型
1. 语言识别（Language Identification）
- 目标：判断文本是否属于特定语言（如英语或德语）
- Meta 的 fastText 语言识别模型是现成工具，支持 176 种语言
- 训练数据来自多语言网站（维基百科、翻译网站、不同语言的新闻站）
- 这是一个相对简单的问题——看几个词就能判断语言，但语码转换、方言等仍存在微妙之处
- Dolma 项目采用 p(English) ≥ 0.5 作为保留标准
2. 质量过滤：OpenMathText 实例
该工作目标是从 CommonCrawl 构建大规模数学文本语料库，流程分为多步：
- 用规则过滤（例如：是否包含 LaTeX 命令）
- 用 KenLM 在已认证的数学数据集 ProofPile 上训练，保留困惑度低于阈值的样本
- 训练 fastText 分类器预测是否为数学文本
- 含 LaTeX 的样本：阈值较低（0.17）
- 不含 LaTeX 的样本：阈值较高（0.8）
结果：构建了 147 亿 token 的数学语料。论文表明，用这种针对性过滤得到的数据训练模型，数学能力优于用 20 倍量但未过滤数据训练的模型。
结论：质量过滤是工具——你可以按任何方式定义质量。若想擅长数学，就收集数学数据。
3. 质量过滤：GPT-3 与 LLaMA
- GPT-3：正样本来自维基百科、WebText（高赞 Reddit 帖子链接的页面）、书籍；负样本从 CommonCrawl 采样。训练线性分类器，按分数随机保留（
keep = np.random.pareto(9) > 1 - score） - LLaMA：正样本是被维基百科页面引用的网页（而非维基百科文章本身），负样本从 CommonCrawl 采样。保留被分类为正类的文档
4. 质量过滤：phi-1
- 原始数据：The Stack 的 Python 子集
- 用 GPT-4 定义了"教育价值"的 prompt，只对 R 的 10 万子集做分类（昂贵但只在子集上做）
- 得到正类集合 T 后，用随机森林训练便宜的替代分类器（基于预训练 CodeGen 模型的输出嵌入）
- 最终在 HumanEval 上，用过滤数据训练的 1.3B 模型（36K 步达到 17.68%）胜过直接用原始 Python 子集训练的模型（96K 步仅达 12.19%）
启示：你可以用昂贵分类器在小子集上定义"好"数据，然后训练廉价分类器外推到全部数据。
5. 毒性过滤
- Jigsaw Toxic Comments 数据集：维基百科讨论页的评论，标注了毒性类型
- 同样的正负样本 + 分类器的框架
过滤的规模依赖效应
不存在单一最优阈值。数据过滤的取舍取决于训练规模：
- 训练 token 数较少时：想要更高质量的数据（低质量数据无法通过更长训练来弥补）
- 训练 token 数较多时：可以容忍更低质量的数据，因为算力充裕不浪费
实验证据（约 1.57 亿参数模型，数据池仅约 100 个词条的极小规模）：在尚未重复使用数据的阶段，高质量数据（DCLM）明显优于无过滤数据（Resiliparse）。但随着训练推进、数据开始重复遍历（epoch），高质量数据的优势缩小——因为每个数据集都有边际收益递减。如果数据有限，训练再久也只是在重复相同的 token。
实践结论：无限算力下过滤非必需；现实中大多数人是算力受限的，必须在低质量内容上浪费算力与过滤成本之间做明智权衡。
三、去重（Deduplication）
重复的类型
精确重复（Exact Duplicates）：
- 镜像网站（镜像的意义就是复制）
- GitHub fork：即使修改，可能只有几个文件不同
近似重复（Near Duplicates）：
- 服务条款和许可证（MIT 许可证可能出现在无数页面）
- 同一模板生成的页面，仅替换实体（如"加拿大"→"美国"）
- 排版差异（如逗号的有无）
- 页眉页脚相同
极端案例：C4 数据集中某个产品描述出现了 61,036 次——防毒面具描述在 Common Crawl 被反复收录。
为什么去重
- 训练效率：减少数据规模且不丢失信息
- 避免记忆化：被大量重复的受版权保护内容或隐私信息如果进入训练，模型会记住它们
关键挑战
过滤是单条目判断——好坏与否，可线性时间并行化。但去重本质是条目间比较。若做 O(n²) 两两比较，无法在网络规模下扩展。因此需要线性时间的哈希方法。
设计空间
- 去重单元：句子？段落？文档？
- 匹配判据：精确匹配？公共子项存在？公共子项比例？
- 处理动作：移除所有实例？保留一个？
精确去重
概念简单：取字符串做哈希，保留每组哈希的一个，删除其余。
- 优点：简单明确、高精度
- 缺点：无法处理近似重复
- 实现可用 MapReduce 风格，便于并行扩展
C4 的做法：单元是三句话片段，做精确匹配，保留一个删除其余。注意：从文档中间删除三句话会破坏文本连贯性——这是当时的权衡。
近似去重
Jaccard 相似度：J(A,B) = |A ∩ B| / |A ∪ B|
- 取值 [0,1]，0 为不相交，1 为完全相同
- 两文档是近似重复当且仅当 J > 阈值（如 0.99）
MinHash 的核心性质：
随机哈希函数 h 使得 Pr[h(A) = h(B)] = Jaccard(A, B)
直觉：随机哈希函数诱导元素的随机排列，看排列后 A 与 B 各自第一个元素是否相同。若排列第一个元素属于交集（概率 = J/|A∪B| 的累积），则两者相同；否则不同。
为什么需要 LSH？ MinHash 的碰撞概率恰好等于 Jaccard——这很好，但这太随机，方差大。我们需要锐化概率：相似度低于阈值时不碰撞，高于阈值时碰撞。
局部敏感哈希（LSH）
设置：n 个哈希函数分成 b 个波段，每波段 r 个哈希函数（n = b·r）
碰撞定义：A 和 B 碰撞 ⟺ 某个波段内所有 r 个哈希值都相同（与-或结构）
概率计算：
P(某固定波段匹配) = sim^r
P(碰撞) = 1 - (1 - sim^r)^b这是一个 S 形曲线（相变）：相似度 < 阈值时碰撞概率低，> 阈值时碰撞概率趋近 1。
调整 b 和 r 的效果：
- 增大 r（波段内哈希函数更多）：阈值变尖锐，曲线右移（更难匹配）
- 增大 b（波段数量更多）：曲线左移（更容易匹配）
相变阈值：threshold = (1/b)^(1/r)。若想按 Jaccard > 0.9 过滤，需让此值等于 0.9。
实际参数（去重论文）：n = 9000, b = 20, r = 450。
为何碰撞概率在阈值处约为常数？ 在阈值处，P(固定波段匹配) = 1/b，碰撞概率 ≈ 1 - (1 - 1/b)^b ≈ 1 - 1/e ≈ 0.64。
去重的注意事项
- 数据源不断到来——可能单个数据集内部做了去重，但跨数据集去重同样必要
- 有时不同数据集之间高度冗余，应做全局去重
四、数据混合（Data Mixing）
问题定义
语言模型在多个数据源上训练（如 Wikipedia、Common Crawl、GitHub、书籍等）。数据混合即确定各数据源的采样分布 p(s)。
基线方法
| 方法 | 定义 | 问题 |
|---|---|---|
| 拍脑袋（Vibes） | 凭直觉手动设置 | 主观性极强，但比想象的更常见 |
| 均匀采样 | p(s) ∝ 1 | 忽视数据量差异 |
| 按比例混合 | p(s) ∝ num_tokens(s) | 巨大低质量数据源会吞噬预算；应上调高质量源权重 |
上调高质量源权重时需要记住两件事：
- 多样性：文学、代码、论文不可直接比较，不应把所有权重放在单一类型
- 数据源有限性：对小数据源权重过高会导致重复轮次过多 → 持续训练完全相同的 token
关键陷阱：重复轮次（Epoching）
示例：低质量数据源 10T token，高质量数据源 10B token。均匀混合（各 0.5），训练 1T token：
- 低质量源：
(0.5 × 1T) / 10T = 0.05轮——只有 5% 的 token 被看到 - 高质量源：
(0.5 × 1T) / 10B = 50轮——每个数据点重复训练 50 次！
50 轮不一定必要——最坏情况下过拟合，最好情况下浪费算力。关键教训：必须检查各数据源实际训练轮数。
混合的实现方式
按 token 级采样？不。通常做法：采样每个元素所属的混合组件，然后从该组件抽序列填满一个 batch。每个序列属于单一组件，batch 是混合的——目标是减少方差。
UniMax（硬上限方案）
- 场景：多语言模型，低资源语言必须特殊处理
- 之前的工作：在按比例混合上取幂 α（使分布更平坦），但仍不够显式
- UniMax 方案：均匀采样数据源，但对每个数据源的训练轮数设置硬上限 C
- 约束条件：
p(s) × num_training_tokens ≤ C对所有 s - 达到上限后无法再获取该源的 token，转向下一个数据源
- 这是安全网
- 约束条件：
基于回归的混合（Regression-based Mixing）
核心思路（RegMix、Olmix 等）：
- 在小规模（如 3 亿参数）尝试不同数据混合方案
- 每种混合训练一个小模型，在某目标指标（下游评估、困惑度）上得到损失
- 用这些数据点拟合回归模型（输入：混合权重 → 输出：损失）
- 优化该回归模型，找到最小化损失的数据混合
- 将此混合用于大规模训练
这与缩放定律（scaling laws）的研究范式类似——用廉价实验推断大规模设置。
设计决策：
- 采样混合方案：用 Dirichlet 分布（关于分布的分布）
- 回归方法：线性模型或梯度提升树
- 优化目标：通常基于下游评估——但必须小心过拟合！若评估集全是代码任务，权重会全部偏向代码——当你想生成诗歌时就暴露了。均匀/按比例混合无此风险，因为它们不看下游评估
两个"信仰跃迁"：
- 回归模型在极小值处仍然准确：随机采样的混合在分布内预测没问题，但优化时会走向极值区域，可能缺少覆盖。需要小心
- 最优混合从小规模迁移到大规模：目前开放社区的经验表明大致成立，但存在规模相关效应——例如更长的训练会使低质量数据变得可容忍
模拟重复轮次（Simulated Epoching）
问题：如果小规模训练（token 数少）时没有重复轮次，而大规模训练时有，则小规模实验的最优混合严重偏向高质量小数据源，不适用于大规模。
解决方案：让"小规模看起来像大规模"（这是整个课程的主题——如 muP 也在做类似的事）。
具体做法：按下采样比例缩小各数据源。如小规模 10B token、大规模 1T token，比例 1:100。在缩小后的混合上，反复"只训练维基百科"的方案会暴露为高重复轮次、差损失，于是优化自然偏向更平衡的方案。在小规模模拟大规模会遇到的数据稀缺性。
数据混合的粒度
除了跨数据源混合，还可对单一数据源（如 Common Crawl）细分：按领域（用网路组织工具分组）和按质量形成二维网格，每格是一个混合组件。Nemotron 论文正是这样做的。
五、后训练数据与合成数据
一般配方
- 定义一组环境（如代码场景中的 GitHub 仓库）
- 定义一组任务/提示
- 从强模型（教师） 收集回复
几乎所有开源社区的后训练数据大部分是合成生成的。前沿机构可混合使用人工标注，但成本极高。教师模型的能力是瓶颈。
OpenThoughts（推理/数学科学）
- 背景：o1 模型激发了对推理能力的关注
- 用 QwQ-32B（非最强模型！）生成 120 万个示例（约 7.5 万个问题 × 16 个采样）
- 来源：27 个人类与合成来源（Stack Exchange、NuminaMath、化学等）
- 关键发现：
- 更强≠更好的教师：QwQ-32B 作为教师优于当时最强大的 DeepSeek-R1
- 多采样有帮助：每个 prompt 采样 16 个回复比少采样好
- 答案过滤无用
- 少而精的来源优于大而杂：利用较少的高质量来源（如 OpenMath-2-Math）胜过用所有来源
编码智能体数据
SWE-smith：
- 给定仓库，用语言模型生成任务（通过修改代码引入 bug）
- 128 个 GitHub 仓库生成 5 万个任务
SWE-Zero（NVIDIA）：
- 观察：SWE 任务依赖极重（要安装依赖、环境过时），但强模型即使不执行代码也能解决约 70% 的任务（有执行时约 80%）——模型对代码有内部语义理解（"world model"）
- 不用 Docker 镜像，直接生成 30 万条智能体轨迹（来自 15 万真实 GitHub PR）
- 使用 OpenHands 脚手架；防止作弊：移除未来 git 提交防止 agent 通过 git 历史获取答案；禁止执行 Python，只能用 grep/set 等
- 从 Qwen3-Coder-480B 蒸馏，并做过滤（Qwen 有时忽略指令仍执行）
- SWE-Hero：1.3 万条需要执行反馈的轨迹
SWE-rebench：
- 从 3.4K GitHub 仓库、45 万 PR 构建 2.1 万交互式 Python SWE 任务
- 用 Qwen 2.5-72B-Instruct 帮助安装依赖和评估 PR 质量
SWE-ZERO-12M：
- 基于 SWE-rebench-v2 任务（3.2 万可执行 + 12 万不可执行）
- 用极小模型（mini-coder-1.7b，pass@100 为 50.4）和 mini-swe-agent 脚手架将 SWE-Zero 扩展到 1200 万条轨迹
合成/半合成/真实数据的权衡
- 完全合成：任务和回复均来自模型
- 半合成：真实环境 + 合成任务（SWE-smith）
- 真实：真实 GitHub PR（SWE-Zero）
关键公式与原则
MinHash 核心性质
Pr[h(A) = h(B)] = Jaccard(A, B)LSH 碰撞概率
P(固定波段匹配) = sim^r
P(碰撞) = 1 - (1 - sim^r)^bLSH 相变阈值
threshold = (1/b)^(1/r)数据混合的重复轮次检查
source_epochs = (p(s) × train_tokens) / num_tokens(s)如果该值远大于 1，意味着该数据源被反复训练——需警惕过拟合。
UniMax 约束
p(s) × num_training_tokens ≤ C （对所有数据源 s）课程作业关联
- 作业中可能需要实现或评估过滤/去重流程——记住先用小子集做原型，再扩展到全量
- LSH 参数（b、r）的选择取决于你想要锐化的相似度阈值：
threshold = (1/b)^(1/r) - 若你的作业涉及多源数据训练，务必在合并前计算各源的有效训练轮数
- 小规模实验的混合方案推导到大规模型时，考虑模拟重复轮次（按比例下采样各源）
常见误区
- 认为存在普适的"高质量"定义：质量因目标而异。数学、代码、文学是不可比对象
- 忽略重复轮次：给定混合后直接训练，直到过拟合才发现某个小型高质量源已被训练了几十轮
- 在小规模调出混合方案后直接用于大规模：小规模（无重复轮次）与大规模（有重复轮次）的最优混合不同
- LSH 中以为碰撞就是最终答案：MinHash 碰撞只是初步候选，仍需后续确认（如直接计算 Jaccard）来过滤假阳性
- 在回归混合中过拟合下游评估集：评估集中代码任务多 → 所有权重偏向代码 → 模型失去通用性
- 假设更强的教师模型一定产生更好的合成数据：OpenThoughts 证明 QwQ-32B 优于当时最强的 DeepSeek-R1
- 忘记跨数据集去重：单数据集内部去重"看似做了"，但数据集之间可能高度冗余
- 基于规则的处理有 100% 成功率：即使规则方法很快，也会在数据中留下瑕疵
复习清单
- [ ] 能解释为什么 HTML→文本是"有损"过程，以及表格处理的难处
- [ ] 能描述一般过滤框架（给定 T 和 R，找到 R 中与 T 相似的子集）并给出两类分类器示例
- [ ] 能解释过滤的规模依赖效应：为什么长训练可容忍低质量数据
- [ ] 能陈述 MinHash 性质并直观说明为什么成立（随机排列看首元素）
- [ ] 能计算 LSH 的碰撞概率公式并解释增大 b、r 各自的效果
- [ ] 能解释 UniMax 的硬上限思想，并知道其约束式
- [ ] 能解释模拟重复轮次（Simulated Epoching）的原理及其与"小规模模仿大规模"主题的关系
- [ ] 能列出回归混合的步骤及两个"信仰跃迁"
- [ ] 能描述合成后训练数据的一般配方（环境→任务→教师回复）
- [ ] 能举例说明"更强模型≠更好教师"
数据即命运：从原始语料到高质量训练集的完整流水线
导语
在构建大型语言模型的过程中，有一个经常被低估却至关重要的环节：数据。我们往往关注模型架构的创新、训练技巧的突破，却忽略了最基础的问题——模型究竟在什么样的数据上学习？斯坦福大学 CS336 课程的第 14 讲，正是围绕这个核心问题展开的深入探讨。这一讲构建了一个完整的数据处理框架：从原始的网络爬取数据开始，经过格式转换、智能过滤、精确去重，再到多源数据的合理混合，最终抵达如今备受关注的合成数据领域。这是一条从"原始噪音"到"高质量信号"的必经之路，也是决定模型能力上限的关键所在。
第一站：数据转化——将混沌变为有序
当我们谈论互联网数据时，一个常见的误解是认为网络内容天然就是文本形式。事实上，无论是 Common Crawl 这样的公开数据集，还是自建的网络爬虫，获取的原始数据几乎都不是干净的文本。它们可能是 HTML 格式的网页、PDF 文档，甚至是代码仓库的目录结构。这种异构的原始数据必须经过一个关键的转化步骤，才能成为语言模型可以理解的序列。
对于占据互联网内容主体的 HTML 文件，处理过程充满了启发式规则。开发者需要编写程序去除网页中的样板内容——那些无处不在的导航栏、广告模块和页脚信息——然后提取出真正的"内容主体"。这个看似简单的任务实际上充满挑战：什么构成"内容"并没有绝对的标准。在某些情况下，导航元素可能反而有助于模型理解网页的结构和功能。更棘手的是，网页中的图片和表格如何处理？这是一个本质上有损的过程：你必须将带有层级结构和视觉布局的 HTML 线性化为一个连续的 token 序列。特别是表格，简单的表格或许可以用 Markdown 格式呈现，但嵌套表格几乎没有完美的解决方案，你必须在某处做出妥协或近似。
这种转换通常采用基于规则的方法，原因很简单：速度。在动辄万亿 token 规模的数据集上，任何复杂的处理都可能成为瓶颈。然而，基于规则的方法必然存在失败率，这也是后续过滤步骤之所以必要的原因之一。但即便有这些不足，基于规则的转换仍然是主流选择，因为纯规则方法的并行化和优化相对成熟。
PDF 的处理则更为特殊。从 FinePDFs 等项目可以看到，PDF 虽然只占互联网内容的一小部分，但其价值密度通常更高——如果有人愿意花精力制作一份 PDF，往往意味着内容本身有其独到之处。但 PDF 的转换面临着独特挑战：文件可能被截断、可能只是扫描件（本质上就是图像）、布局信息与语义结构难以对应。这些问题的解决往往需要更重的工具，比如使用视觉语言模型进行 OCR 识别，成本远高于纯文本处理。但考虑到 PDF 的稀缺性和高质量特性，这笔投入是值得的。
第二站：数据过滤——定义何为"优质"
完成格式转换后，我们面对的仍然是一个庞杂的文本集合。这里的核心问题是：如何从中筛选出我们真正想要的内容？课程提出了一个优雅的通用框架：给定目标数据 T 和原始数据 R，目标是找到 R 中与 T 相似的子集。这个框架之所以强大，在于它可以应用于各种不同的过滤需求。
最基础的应用之一是语言识别。Meta 训练了一系列 fastText 语言识别模型，支持 176 种语言，可以直接用于判断文本属于哪种语言。这类任务通常相对简单——几个词就能看出是西班牙语还是日语。但随着语码转换、方言变体和网络用语的普及，这个问题也并非完全解决。不过在大多数场景下，语言识别已经是一个成熟工具。
更复杂的是质量过滤。这里的关键洞察是：不存在普适的"质量"定义。质量永远针对具体的训练目标而言。OpenMathText 项目展示了一种针对数学领域的数据筛选策略：先通过规则过滤（是否包含 LaTeX 命令），再用 KenLM 语言模型检查困惑度（基于 ProofPile 数学数据集训练），最后再用 fastText 分类器判断是否为数学写作。这种多层次的筛选策略，使得他们能够从海量 Common Crawl 数据中提取出约 150 亿个数学相关 token。更有说服力的是实验证据：经过这种针对性筛选的模型，其数学能力优于用 20 倍未筛选数据训练的模型。
GPT-3 的过滤策略则是另一种思路：以维基百科、WebText2、书籍等高质量来源作为正样本，以 Common Crawl 采样作为负样本，训练一个线性分类器。LLaMA 的做法略有不同，正样本选择的是被维基百科页面引用的网页，而非维基百科本身。这种细微的差别反映了数据选择中的设计智慧。
微软的 phi-1 项目展示了一个更为精妙的过滤流程。他们的原始数据是 The Stack 的 Python 子集，但研究者定义了"教育价值"这个标准，先用 GPT-4 对 10 万个子集样本进行打分，将正样本作为目标集，然后训练一个随机森林分类器来外推评分。这个流程的启示是：你甚至可以用一个昂贵的大模型来定义"优质"，然后训练一个便宜的分类器来大规模应用。实验结果具有说服力：在 HumanEval 上，筛选后的数据仅用 36K 步就超过了未筛选数据 96K 步的性能。
毒性过滤则展示了数据伦理的考量。通过 Jigsaw Toxic Comments 数据集，可以训练分类器识别并过滤掉含有毒性内容的评论。这在多语言模型中尤为重要，因为互联网上确实存在大量令人不适的内容。
但课程的深度不止于此。它提出了一个关键观察：过滤阈值不存在全局最优解。这个观点基于一个重要的规模效应：如果你计划训练很短的时间，那么应该使用高质量数据；但如果训练期非常长，低质量数据反而可能更有价值，因为数据池是有限的。一个初步实验清楚地展示了这一点：在小规模训练中，高质量数据的优势明显（损失下降更快）；但随着训练时间的延长，模型开始在高质量数据上重复训练（epoch 数增加），出现过拟合现象，此时低质量数据的表现反而更优。这表明，数据过滤策略必须与训练计划协同设计，任何孤立的"最优阈值"都是不存在的。
第三站：去重——消除数据中的幽灵
过滤完成后，数据集中仍然存在大量重复内容。这些重复有两种形式：完全重复和近似重复。完全重复的成因主要是镜像网站和仓库的 fork——爬虫可能无法智能识别两个域名指向同一份内容。近似重复则更为隐蔽：服务条款和许可证文本几乎相同但分属不同页面、同一篇文章的略微修改版本、模板化生成的低质量内容等。
还有更极端的案例：审计 C4 数据集时发现，一个防毒面具的产品描述竟然出现了 61,000 次。这种数据不仅浪费计算资源，更可能导致模型记忆化，引发版权和隐私问题。
去重面临的根本挑战是计算复杂度。与过滤不同（每个文档独立判断，可以线性并行处理），去重本质上是条目间的两两比较，朴素的实现是 O(n²) 复杂度，在互联网规模的数据集上完全不可行。解决方案来自算法社区的几个经典思想。
首先是 哈希函数。通过将文档映射为较短的哈希值，可以快速比较文档是否相同。但对于近似重复检测，简单的哈希不够用，我们需要能够度量相似度的哈希方法。这时引入了 Jaccard 相似度：两个集合交集大小除以并集大小，得到 0 到 1 之间的相似度分数。关键在于，我们需要一种能够在线性时间内找到高 Jaccard 相似度配对的方法。
答案就是 MinHash。这种随机哈希函数的巧妙之处在于：两个集合的哈希值碰撞的概率恰好等于它们的 Jaccard 相似度。课程用一个具体的例子（集合 {1,2,3,4} 和 {1,2,3,5}）验证了这一点：用 100 个不同的 MinHash 函数计算，得到的碰撞率正好约等于 0.6——和 Jaccard 相似度一致。
但 MinHash 本身还不够。碰撞概率等于 Jaccard 相似度意味着：即使两个文档相似度高达 0.9，每次哈希碰撞的概率也只有 0.9，重复验证 100 次，其中 10 次可能不碰撞。这种随机性对于真正高效的匹配仍然不够。于是 局部敏感哈希（LSH） 登场了。
LSH 的核心思想是通过"与-或"结构来锐化概率曲线。具体做法是将 n 个哈希函数分成 b 个波段（band），每个波段包含 r 个哈希函数。一个波段"匹配"指该波段内所有 r 个哈希函数都产生碰撞；两个文档若至少一个波段匹配，则判定为碰撞。这个结构创造了一条 S 形曲线：当相似度低于某个阈值时，碰撞概率趋近于零；高于阈值时，概率迅速逼近 1。通过调整参数 b 和 r，你可以精确控制这个相变点的位置和锐利程度。例如，典型的设置包括 b=20, r=450，对应的阈值约为 0.9。
一个值得一提的细节是：C4 数据集采用的去重单元是三句话的跨度。如果两个文档中的三句话完全相同，就删除其中之一。这个做法的副作用是可能破坏文档的连贯性——从文档中间截掉三句话会造成不自然的跳跃。这提醒我们，即使是最基础的预处理步骤，也需要仔细考虑设计权衡。
第四站：数据混合——高质量与多样性的平衡艺术
经过过滤和去重，我们得到了干净、高质量但相互独立的数据集。但大型语言模型的训练通常涉及多个来源的数据：可能来自 Common Crawl 的通用文本、维基百科的百科条目、GitHub 的代码、学术论文、数学教材等。如何为这些来源分配权重，是数据混合要解决的核心问题。
最朴素的基线方法包括：凭直觉（"vibes"）手动调整，均匀采样，或按 token 数量比例采样。但课程揭示了一个关键的陷阱：不同来源的数据量差异可能是三个数量级。假设低质量数据源有 10 万亿 token，高质量数据源只有 100 亿 token。如果采用均匀混合（各占 50%），训练 1 万亿 token，那么低质量数据只被使用 5%（1 万亿 × 50% ÷ 10 万亿），而高质量数据将被重复训练 50 遍（1 万亿 × 50% ÷ 100 亿）。这种隐性的高 epoch 数必然导致过拟合——实际上，几个大型模型的训练事故就源于此。
解决方案之一是 UniMax 提出的 epoch 硬上限：对每个数据源设置重复训练的最大次数。这意味着，如果一个数据源已经贡献了足够的 token（概率乘以训练总数达到上限），就不再从它采样。这个约束可以被纳入混合方案的优化中。
更系统的方法是基于回归的混合（如 RegMix、Olmix）：先训练多个小规模的代理模型（参数规模通常数千万），每个模型使用不同的数据混合权重。然后用回归模型将这些混合权重映射到目标指标（如困惑度、下游任务得分）。最终，优化这个回归函数，找到预测效果最好的混合方案，再将其应用于大规模训练。这个过程本质上是用便宜的计算搜索最优的超参数空间——与缩放定律的研究思路一脉相承。
但这种方法有两个"信仰飞跃"：第一，回归模型在最小化器处是否仍然准确？因为你优化它，可能把你推向采样分布之外的极端，而经典泛化理论只承诺分布内的预测质量。第二，最优混合方案能否从 3 亿参数迁移到 700 亿参数？这里存在明显的规模效应——高资源与低资源数据的"最优比例"可能随训练长度变化。课程提出了模拟 epoching 的解决方案：按比例下采样所有数据源，使小规模的 epoch 特性与大规模一致。如果你在小规模实验中只用 100 亿 token，而大规模训练是 1 万亿 token，那么将每个来源都缩小 100 倍。这样在小规模优化时，重复训练过高权重的代价就会显现——模型会过拟合，迫使优化算法寻找更平衡的方案。这是一个"让评估环境与实际运行环境尽量一致"的思想，与课程中反复出现的主题（如 muP 的超参数迁移）一脉相承。
第五站：后训练与合成数据
数据工作不仅限于预训练阶段。当前端到端的训练流程中，后训练数据的重要性正在快速上升。这里的基本流程是：定义环境（如 GitHub 仓库），定义任务（如修复 bug），从教师模型（通常是一个能力强大的模型）获取回复，构成训练数据。在开源社区，这意味着几乎所有的后训练数据都是合成生成的。不过，即使是前沿实验室也常采用人机混合的方式——让模型先生成候选，再由人工筛选或修正。
OpenThoughts 是合成数据的一个典型案例。他们从 27 个来源收集问题（包括 StackExchange、NuminaMath 等人类来源，以及合成生成的问题），然后用 QwQ-32B 作为教师模型生成了 120 万个包含推理过程的示例。有趣发现包括：更强的模型未必是更好的教师（QwQ-32B 作为教师优于当时最强的开放模型之一 DeepSeek-R1）；对每个问题采样多个回答（16 个）是有帮助的；但简单的答案过滤并无助益。这说明了高质量合成数据的关键在于教师模型的选择和生成策略——生成过程的多样性往往比事后筛选更有效。
SWE-smith 则是针对代码智能体的创新：给定一个真实代码仓库，用语言模型自动生成任务（如引入 bug），然后要求模型修复。用这种方式从 128 个仓库生成了 50K 个任务，构建了一个逼近真实软件开发场景的训练集。SWE-Zero 更进一步，核心观察是：与数学问题不同，软件工程任务有重依赖问题——大多数仓库需要安装众多依赖包，且版本过时导致环境搭建是一场噩梦。但实验发现，经过充分训练的模型已经具备了内部的代码语义理解能力，即使不运行代码也能解决很多任务（允许运行得分 80，禁止运行也能达到 70）。基于此，研究者用了 150K 个真实 GitHub PR，生成 30 万个无需执行的智能体轨迹。他们用 Qwen3-Coder-480B 蒸馏，并严格过滤掉那些试图规避"禁止执行"指令的样本。
SWE-rebench 则展示了另一种规模化路径：从 450K 个 GitHub PR 中通过模型评估质量，构建了 21K 个可执行的交互式 Python 任务。最新的工作把 SWE-Zero 的思想扩展到了 1200 万个智能体轨迹，使用更小的模型（mini-coder-1.7b）和更轻量的 agent 脚手架。这些案例清晰展示了合成数据的演进方向：从无环境依赖的数学推理，到有环境依赖的代码修复，最终走向真实的软件开发过程。尽管环境配置是痛苦的，但已经出现了多种绕过或简化环境依赖的策略。
本讲在课程体系中的位置
CS336 课程系统性地覆盖了从数据获取、模型架构、训练技术到部署评估的全链路。第 13 讲（数据来源）讨论了数据的获取渠道——从 GitHub 等实时服务导出，或从 Common Crawl 爬取，以及由此涉及的版权、服务条款等社会性考量。第 14 讲则围绕数据质量提升展开，从单一的"有文本"状态，通过转化、过滤、去重、混合，达到"适合训练的语料库"标准。
这一讲与后续内容的联系同样紧密：数据的质量直接决定了模型能力的上限——好数据可以让小模型（如 phi-1 的 1.5B）在特定任务上超越用 20 倍数据训练的模型。同时，本讲展示的方法论——在小规模上做实验寻找规律，再外推至大规模——与课程前期的缩放定律研究、muP 的参数化策略一脉相承。无论你选择研究哪种模型架构，数据处理的这些基本思想都是通往优秀模型的关键阶梯。
结语
回顾整个数据处理流程，课程强调了几个贯穿始终的原则。
其一，数据的质量评估是相对的、有条件的。你无法脱离训练目标单独谈论"质量"——想要数学能力就筛选数学数据，想训练英语语种就先用语言识别。最优的过滤阈值、最优的数据混合权重，都取决于你计划的训练长度和可用的算力预算。
其二，数据处理存在深刻的规模效应。过滤器的优劣会随训练步数变化，数据混合的"最优"也会随训练数据规模改变。你不仅需要思考"什么是最好的数据"，更需思考"在给定的计算预算下，什么样的数据策略是最优的"。
其三，算法思想的价值。MinHash 和 LSH 是来自理论计算机科学的经典技术，却在实际的大规模数据处理中发挥关键作用——它们将 O(n²) 的问题转化为线性时间即可处理的近似查询，在碰撞概率的设计上将随机性变成了可控的工具。这种"用概率换效率"的思维，值得我们反复揣摩。
其四，动手实验的不可或缺性。正如课程所说，数据工作高度依赖具体领域，需要逐个查看具体实例才能构建出高质量的数据集。宏观的框架只是起点——真实的数据踩坑、清洗、调优，才是通往好模型的正途。
参考资料
- 官方视频：CS336 2026 Lecture 14: Data (filtering, deduplication, mixing, synthetic data) —— https://www.youtube.com/watch?v=5sxHosTLPF8
- 课程讲义（含代码示例）：https://stanford-cs336.github.io/spring2025/
- 讲义中的关键论文链接：DCLM、FinePDFs、OpenMathText、GPT-3 附录 A、LLaMA、phi-1、Dolma、UniMax、RegMix、OpenThoughts、SWE-smith、SWE-Zero、SWE-rebench 等，均可在官方讲义中查看
CS336 2026 Lecture 14：完整中文逐字稿
- 视频：官方视频
- 来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
- 说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。
正文
[00:05.240] 好，我们开始吧。今天是关于数据的第二天。上次我们讨论了数据并非真的从天而降。你实际上必须思考它来自哪里。总的来说，互联网由实时服务组成。这些服务上的数据必须被转储或爬取。然后还有一个额外的步骤，你必须处理这些数据。我们还讨论了各种社会考量、服务条款、版权问题。你必须获得许可或援引合理使用。所以数据方面涉及很多复杂性。今天我们将更多地关注数据流水线，从数据转换、数据过滤、去重，到数据混合。然后最后我们会简单谈谈训练后数据，特别是如今人们如何使用合成数据。第一部分主要关于预训练。这是你们应该记住的。我们之前已经简单讨论过数据转换，但这里提醒一下。原始数据并不是以文本形式出现的，即使你抓取了一些东西。如果你看过 Common Crawl 内部，它并不是文本。
[01:33.820] 它要么是 HTML。有时可能是 PDF，或者对于 GitHub 来说，是目录。数据转换的大部分注意力都集中在处理 HTML 上，因为互联网的大部分内容都是 HTML 格式。对于这种处理，很多都是相当启发式的。包括去除样板内容，比如导航和广告，以及提取内容，也就是页面的主要部分。关于什么构成内容，有一些微妙之处。通常，你会去掉页脚和页眉，也许还有菜单之类的东西，然后试图提取内容。但你可以想象，在某些情况下，一些导航元素可能有助于学习网页的外观。所以，什么是内容和什么不是内容并不总是清晰的。那么，如何处理网页中的图片和表格呢？本质上，这是一个有损的过程，因为你需要将 HTML 线性化，而 HTML 至少是层级化的或可视化的（如果你考虑渲染后的输出），转化成一个 token 序列。
[02:52.150] 特别是表格处理起来有点棘手。简单的表格，你可以用 markdown 来渲染。但如果你有嵌套表格，那就变得相当有挑战性了。你必须在某个时候放弃或者在某处进行近似处理。所以，通常来说，HTML 到文本的处理是基于规则的。这样做的原因是基于规则的处理速度非常快。而且，你在这里并不需要做太多事情。你不需要太多智能。所以基于规则通常就足够了。不过，我认为在这一点上，基于模型的干预是有其用武之地的。它们必须非常快，并且必须做一些更智能的事情。但如果你看过数据，你会注意到数据中存在瑕疵，仅仅因为任何基于规则的处理都有一定的失败率。正如我们上次展示的，准确性确实很重要，这取决于你选择哪种工具。
[04:05.620] 如果你使用 Resiliparse 或 Trafilatura，那么——我想 Resiliparse 在这些扩展的 dclm 评估中表现优于其他工具。我还会简单谈谈 PDF。有一个 Hugging Face 的工作，发布了一个名为 FinePDFs 的数据集。PDF 文件，如果你打开它们，不是在 PDF 阅读器里，看起来像这样，这需要被渲染。PDF 可以出现在互联网的网页上。Common Crawl 确实有一些 PDF。一般来说，Common Crawl 侧重于文本，但有时，如果你只是得到一个 URL，你甚至在抓取之前可能都不知道它是不是 PDF，如果它没有扩展名的话。
[05:03.640] 如果你阅读这篇博客文章，里面有很多细节，我就不一一赘述了。例如，有一点是 Common Crawl 中的许多 PDF 都是被截断的，因为 PDF 很大，所以这就意味着需要重新爬取。然后还有一个问题——一旦你有了 PDF 文件，你究竟如何把它转换成文本？而且，有些 PDF 可能只是扫描件。所以它们本质上就是图像。这篇论文尝试了很多工具，但这主要涉及使用 VLM 运行 OCR。这显然可能比我们之前处理文本要昂贵得多。幸运或不幸的是，PDF 只占整个互联网的一小部分。但它们非常有价值，因为一般来说，如果你费心制作一个 PDF，那意味着你可能有些有趣的内容要说，而不是一个普通的网页。所以平均而言，PDF 的质量普遍高于 HTML 文件。
[06:14.810] 有大量的清理和过滤工作。而且 PDF，比网页更是如此，很多布局信息都缺失了，因为在 HTML 中你有各种标签，比如 h1 和 p，这些提供了语义信息。PDF 从设计上就完全是关于布局的。所以你未必能保留语义结构。所以会有很多转换发生。到这一步，你有了文本，但还远未完成。所以下一步是过滤。我将比较抽象地谈谈什么是过滤。这是基本构件。假设你有一些你想要的目标数据。这通常是一小部分高质量的数据，以及大量的原始数据。这是你从上一个步骤转换得来的、新到的 token。目标是找到这个原始数据中与目标相似的一个子集。这就是过滤的一般框架，几乎所有类型的过滤都符合这个模式。
[07:39.410] 关于过滤，你可能想要过滤的原因有很多。如果你在训练一个英语语言模型或德语语言模型，你可能想要识别语言并过滤掉不匹配该语言的内容。过滤的主要原因是为了质量过滤。你想要找到高质量的内容，而不是低质量的内容。你不想要垃圾信息。你想要百科全书式的信息。另一个应用是有毒内容过滤。当然，互联网上有大量令人讨厌的内容，你可能不希望你的语言模型在这些内容上进行训练。所以，对于一个过滤算法，你希望从目标数据中获得一些泛化能力，因为你已经有了目标数据。你不仅仅是想得到目标数据，而且你还希望它非常快，因为你要在整个互联网上运行它。
[08:38.140] 这可能相当于 100 万亿 token 的数据。所以通常，过滤的结果是，你最终只保留了你全部数据中非常小的一部分，一个百分比是个位数的小部分。好的，记住这个通用框架：给定目标数据 T 和原始数据 R，你试图找到一个子集。一般的方案是，你基于 R 和 T 估计某个模型，推导出一个评分函数，然后根据得分保留 R 中的样本。通常，有两种类型的分类器。一种是生成模型。这基本上就是你有你的目标数据。你可以直接估计那个数据的模型。记住，这必须很廉价，所以你可能不会训练一个大的语言模型。通常，KenLM 基本上就是说我要训练一个 5-gram 模型。一个更常见的做法，我想我们如今大多看到的是，就是训练一个分类器。
[09:48.230] 这个分类器会说，我要预测正标签，例如，那些在 T（我的目标）中的，以及负标签，对应于在原始数据 R 中但不在 T 中的内容。所以基本上，看起来像是 T 是你的正例，R 的某个随机子集是你的负例。也许你会做一下平衡，然后训练一个分类器。人们常用的工具是 fastText，因为它速度快。通常，它只是一个线性分类器——词袋模型。有了这个模型之后，对于每一篇新文档，你都可以给它打分，然后根据你的质量门槛设定一个合适的阈值，保留符合条件的样本。有时是随机保留，有时不是。所以这非常像一种基于模型的过滤方法。还记得上一讲提到的，几年前有许多数据集没有使用基于模型的过滤，因为他们不想引入太多偏差。
[10:59.590] 如今，我认为基本上每个人都会做一定程度的基于模型的过滤，因为除非你算力非常充裕——那种情况下你也许不需要做太多过滤，直接训练所有数据就行——大多数人都算力有限。你必须非常明智地决定如何过滤；否则，你只是在低质量内容上浪费算力。好，下面我来讲几个过滤的具体实例。还记得我提到的语言识别吧。它的目标就是——给定一段文本，判断它是否属于某种特定语言。Meta 训练了一系列 fastText 语言识别模型，通常你可以直接拿来用。它支持 176 种不同的语言。训练数据来自大量多语言网站。比如维基百科，大家知道它有很多语言版本。还有一些翻译网站和不同语言的网站。因此，与我们处理的许多其他任务相比，语言识别通常是一个相当简单的问题——就是一个简单分类器。
[12:14.120] 你只要看几个词，就能判断这是西班牙语还是日语。当然，由于语码转换、文本变体和方言的存在，也有一些微妙之处。所以我不敢说这是完全解决的问题，但它并不是训练一个好语言模型的真正瓶颈。然后你可以直接——一旦有了分类器，就选一个合适的阈值。这通常相当依赖启发式规则。过滤还可以做另一件事：假设你在寻找某种特定类型的数据。比如你想在数学上特别擅长，那你就去找一堆数学数据。OpenMathText 就是 2023 年的一篇论文，目标是构建一个大规模的数学文本语料库。这个流程包含几个步骤，不只是一个单一分类器。首先，你用一些规则来过滤：是否包含 LaTeX 命令？他们还用了 KenLM。
[13:17.620] 这是一种生成式方法。他们在 ProofPile 上训练，这是一个已知的数学数据集，如果困惑度低于某个阈值就保留。然后他们还训练了一个 fastText 分类器，用来预测某段文本是否为数学写作。这里有两种不同的阈值：如果包含 LaTeX，门槛就低一些——抱歉——是更低的标准；但如果不包含 LaTeX，标准就更高一些。最终，他们得到了 150 亿个 token，用来训练模型。有一篇论文表明，这种更有针对性的、精选数据收集方式训练出的模型在数学能力上优于那些在 20 倍数据量上训练但未经过这种过滤的模型。所以质量过滤，再次强调，把它看作一个工具。你可以按任何方式定义质量。不存在普适的质量概念。
[14:21.630] 如果你想把质量定义为数学能力，那你就可以去获取数学数据，你在这方面就会变得更好。GPT-3，我想上次已经提到过，但为了把它放进这个框架里：正样本是维基百科、WebText，也就是从高赞 Reddit 帖子链接出去的页面，还有一些书籍。负样本通常从网络上采样。他们训练一个线性分类器，如果文档的评分足够高就保留。还有最早的 LLaMA 论文，正样本是被维基百科页面引用的网页，而不是维基百科文章本身。这是同样的思路。微软的 phi-1 也很有趣，它同样属于这个框架。他们一开始——原始数据并不是 The Stack 的 Python 子集。大家记得那全是代码。他们定义了一个 prompt，用来判断“教育价值”。
[15:37.720] 然后他们用 GPT-4 对此中的一个子集进行分类，R 的 10 万子集，用这个 prompt 来打分，得分积极的就保留下来，称为目标集。所以这里的目标集实际上是一个昂贵分类器的输出。然后你训练一个更便宜的分类器，他们的做法是随机森林，但你可能也可以用 fastText。然后你选择那些被这个分类器判定为正类的数据。他们还证明了，使用这个数据集，你比直接使用原始数据效果要好得多。这个性能在更少的步骤内上升到更高的水平。毒性过滤也是同样的方式。有一个数据集——Jigsaw Toxic Comments，来自一个旨在帮助人们在线上进行更好讨论的项目。这里的数据是维基百科讨论页，大家可以想象，在有争议的页面上，讨论可能会相当激烈。这些数据已经标注了是否存在毒性评论。
[17:01.200] 你也可以同样地定义正负样本，并在此基础上训练分类器。好了，我想现在你已经有了工具，或者说一些例子和灵感。你可以识别出你想要的那种特定类型的数据，为它构建分类器，然后就可以去 Common Crawl 里过滤你想要的东西了。我想在这里讨论一个微妙之处，那就是——不，你对数据的需求概念实际上取决于你想训练什么样的模型。特别是，它取决于你要训练的 token 数量。因此不存在所谓的最优阈值。之前，当我看到分类器时，它给出一个分数。你不能说 0.9 就是最好的，因为这取决于你想做什么。直觉上来说，如果你要训练更长时间，那你可以容忍更低质量的数据。
[18:02.350] 如果你训练时间较短，那么一般来说你想要更高质量的数据。当然，如果你能挥一挥魔法棒——如果你训练时间更长，你当然想要更多高质量数据，但这并不是你能选择的。数据池就是那样，它不会变。这是一个迈克尔·瑞安做的初步实验。基本上，这张图展示的是一个 1.57 亿参数的模型。我们用的是很小的数据池，只有 100 个 works，这只是 Common Crawl 的极小一部分。我们持续不断地进行训练。让我们看看蓝色曲线。蓝色曲线是 dclm。损失从这里开始，然后逐渐下降。每条线代表你在数据上跑完一个 epoch。这是第一个 epoch，然后下一条蓝色线是第二个 epoch，以此类推。
[19:12.480] 数据量并不大，所以最终你必须重复使用数据。损失还在继续下降，因为第二次看这些数据时，你仍然在学习一些东西。到了某个时刻，你开始过拟合。而如果你看 Resiliparse，这基本上是没有过滤的。在这里，一开始情况要差得多。但随着你不断训练，到某个点，你也开始跑 epoch。损失下降的速度变慢了。所以你可以看到这个趋势：在你还没有重复使用数据的阶段，高质量数据表现更好。但一旦你用了大量、大量的 token，高质量数据就不再那么有优势了。当然，对于高质量数据，你不会想进入这个区域，因为你在过拟合——你可能会在这里停止。但即使在这个点上，这也不如你用低质量数据训练更长时间来得效果好。是的，有一个关于计算指标的问题。
[20:19.520] 这实际上不完全关于这张图。每一个点对应一次训练运行，对吧？是的。你在做这些重新训练实验时，是否需要计算置信区间，还是只做一次就够了？所以问题是，这些点中的每一个都是单次训练运行，你是否应该多次运行并得到置信区间？理想情况下，那会是一个好的实践。你经常会看到这些论文中很少有这种区间，因为每次训练运行都相当昂贵。但实际上，当我们做过这些实验时，结果一般来说是稳定的，我想说，至少对于预训练来说是这样的。对。所以你提到了训练更长时间和高质量数据，这两者的组合我们不考虑吗？但如果我们在这组高质量数据上训练更长时间，与用高质量数据训练更长时间相比，它是否会有边际收益递减？
[21:29.590] 是的。所以问题是，如果你能获得更高质量的数据并且训练更长时间，它仍然会有边际收益递减吗？每个数据集最终都会有边际收益递减。数据是有限的。但它可能会在这里。它只会在这里更低，然后继续往下降。好的。很好。我们继续。所以总结一下，过滤对于构建一个好模型相当关键，尤其是当你们大多数人受限于计算资源的时候。从技术上讲，如果你有无限的计算资源，你就不需要过滤。你可以在所有数据上训练，它会是一个巨大的模型。但实际上，每个人都需要过滤。所以这里的方法论是：弄清楚什么数据是好的。然后你可以训练一个分类器，它会外推到你的其余数据。那么什么数据是好的呢？你可以通过说“啊，有一个我很喜欢的数据集，我就想要更多这样的数据”来找到它。
[22:37.090] 或者你可以向语言模型构造一个提示词，然后用它来构建、对大语料库做初步过滤，然后用那些高质量数据去训练一个较小的分类器，并外推到所有其他数据。好的。接下来，我要讲去重。所以在这个阶段，我们已经过滤了我们的数据集。现在我们只保留我们认为高质量的数据。但通常，数据中仍然有重复内容。有两种类型的重复。有完全重复。这种情况发生的原因比如，你看这些镜像网站，镜像的全部意义就在于它是一个副本。有时候网络爬虫不够聪明，不知道这个镜像和那个镜像完全相同，所以它就爬了很多站点，你最终会得到完全相同的内容。另外，当你 fork 一个代码仓库时，那也是重复的，即使你做了修改。可能你只修改了几个文件，所以那个仓库的 99% 内容可能是相同的。
[23:55.520] 所以重复内容非常丰富。而且，有时候确实存在近似重复，它们不是镜像或衍生内容，但它们恰好是只差几个词元的相同文本。通常，这可能来自——可能是复制，或者来自某个共同的来源。这里有一些近似重复的例子。比如服务条款和许可证。我想 MIT 许可证可能出现在很多地方。在很多情况下，它是精确复制的，但只是许可证那部分，除非有人在复制时打了错字。但许可证可能所属的页面其余部分可能是不同的。我们之前讨论过很多网站有相同的页眉和页脚。那些也是重复的。还有一些情况，出于某种原因，这是来自 LM1B 的，也就是 10 亿词基准数据集。
[25:06.300] 有一些文章只是排版上的差异。一个版本有逗号，一个版本没有逗号。我不太清楚为什么，但这种情况确实存在。有时候你还会看到一些模板，其中可能是一些低质量内容，看起来像某种广告之类的，有人把它模板化了，然后把“加拿大”替换成“美国”。所以如果你在这种数据上训练，但你只是在训练同一内容的不同变体、带有不同实体，那只是在浪费你的 GPU。还有更极端的例子。所以这就是为什么查看你的数据是好的。对 C4 数据集的这次审计发现，这个产品描述在数据集中出现了 61,000 次。如果你追溯回去——这真的很怪异。我不知道为什么。这是一个防毒面具的某种描述。它就在这个 Common Crawl 数据集中出现了 61,000 次。所以说，网络世界真是无奇不有。那为什么要去重呢？第一个想法很明显，因为你想要更高效地训练。
[26:24.010] 去重可以减少数据集的规模，而不会真正丢失信息，因为你只是去除了重复内容。它还有一个好处是避免记忆化，这篇论文讲到了这一点。例如，如果你有一些被大量重复的受版权保护的内容，那么如果你在它上面训练，你就会记住它，同时还有隐私问题。但我觉得，最主要的原因还是为了确保你没有浪费算力。与去重相关的还有去污染，它可以说甚至更重要。它是同样的事情，你要确保你的测试集不在你的训练集中。那么我们怎么做去重呢？这里有一个设计空间需要考虑。首先，你要去重的单元是什么？你是在句子级别、段落级别还是文档级别做？你如何判定一个匹配？是精确匹配吗？是是否存在一个公共子项？
[27:34.000] 是公共子项的比例来判定近似重复吗？然后，一旦你发现两个页面之间有重复，你该怎么做？你是移除所有实例，还是只保留其中一个？关键的算法挑战在于，去重本质上是在将条目与其他条目进行比较。通常，如果你在做过滤，那是关于单个条目的。这个条目是好还是不好？这可以并行化。它是线性时间的，这很好。而且即使在线性时间内，我们也试图通过基于规则的方法或非常小的模型来让它更快。但重复检测很明显，你不能做那种 O(n²) 的比较——把每个东西和所有其他东西比较。你需要线性时间的算法来扩展，尤其是在这种网络规模下。所以通常，去重文献使用哈希函数来解决这个问题。
[28:36.940] 我们会展开讲这些想法。这些是来自算法社区的一些非常漂亮的想法。我觉得大家都知道哈希函数是什么。它接收一个值比如一个字符串，然后把它映射到一个像字符串或整数这样的东西。哈希值比原始条目小得多。哈希冲突是指两个不同的条目映射到同一个值。当你审视哈希函数时，有一些非常高级的哈希函数本质上是密码学性质的。这些是抗碰撞的，用于密码学和比特币等等。然后还有一些更快的，用于哈希表的，其中哈希冲突不是什么大不了的事情。我们会使用这些。所以取一个字符串并映射到一些值，这就是哈希函数做的事情。好的。所以精确去重在概念上非常简单。
[29:43.370] 你取一个字符串，看是否存在完全相同的匹配，然后保留其中一个，其余全部删除。所以这里你有一堆元素，你对它们做哈希，然后去重。完全重复的去除非常理想，这很清楚在做什么。但是，对于杂乱无章的网页数据来说，这样做还不够好，因为你经常会遇到近似重复的情况。有一点需要注意的是，实现方式有很多种。这里采用的是类似 MapReduce 的风格，这样更容易并行化和扩展。所以，C4，这篇来自 T5 论文的论文，它在处理 Common Crawl 时，采用了完全去重的方法。他们处理的基本单元是三句话的片段，然后进行完全匹配，并保留其中一个，其余删除。所以，如果你注意的话，你会发现这里有些奇怪，因为你处理的是三句话的跨度。如果你发现两个文档包含相同的三句话片段，并删除了其中一个，这意味着你只是从那个文档里
[31:12.400] 直接拽出来三句话，这有点奇怪，因为它破坏了文本的连贯性。但这就是他们当时所做的。所以这就是完全去重。那么，我们如何做近似去重呢？首先，我们需要定义这里的“近似匹配”是什么意思。为了做到这一点，我们要定义一个叫做 Jaccard 相似度的概念，这是一个相当标准的定义。两个集合的 Jaccard 相似度基本上就是交集大小除以并集大小。所以，给定这两个集合——1, 2, 3, 4 和 1, 2, 3, 5，当你计算 Jaccard 时，你取交集。那就是 1, 2, 3。你取并集，那就是 1, 2, 3, 4, 5，然后你相除，Jaccard 就是 0.6。
[32:08.120] 所以 Jaccard 是介于 0 和 1 之间的一个数字。0 表示它们不相交，1 表示它们完全相同。所以这是一个相当自然的定义。我们会说，如果两个文档的 Jaccard 相似度高于某个阈值，比如 0.99，那么它们就是近似重复的。那么现在的问题是，你如何在线性时间内找到近似重复项？幸运的是，这是一个已解决的算法问题。答案是使用 MinHash。嗯，好的。所以第一步是使用 MinHash。这还不是最终答案。MinHash 是一种随机哈希函数，使得哈希碰撞的概率恰好等于 A 和 B 的 Jaccard 相似度。
[33:01.480] 这是一个非常好的性质，因为哈希有助于实现线性时间处理。Jaccard 是我们想要的度量标准，现在我们期望地将这两者联系起来。有趣的是，通常你希望定义哈希函数是为了让所有不同的元素被哈希到不同的地方。你不希望有碰撞。但在这里，你实际上希望有碰撞——不是任意的碰撞，而是希望以一种特定的方式来控制碰撞，使其与相似度对齐。相似的内容你希望它们比不相似的内容更容易碰撞。所以 MinHash 本质上，实际上，让我们看看。也许我会——所以这里就是 MinHash。它相当简单。你取一个集合，对集合中的每个元素进行哈希，然后取其中的最小值。如果你是第一次看到这个，可能会觉得有点奇怪。为什么取最小值？你也可以取最大值，这其实
[34:15.210] 并不重要。这只是一个打破平局的方法。所以，这是你脑海中应该有的图像。如果你有 A，记住 A 包含 1, 2, 3, 4，B 包含 1, 2, 3, 5。这是这两个集合的特征矩阵表示。随机哈希函数的作用是它引发了对这些元素的一种排列。可能是 4, 3, 1, 5, 2，或者别的什么。然后你根据这个排列看在 A 中哪个元素排第一，在 B 中哪个元素排第一。每个元素有相同的概率排第一。所以如果随机哈希函数把 1 放在第一位，那么 A 中的第一个元素将等于 B 中的第一个元素。如果是 2 第一，也是一样。如果是 3 第一，也是一样。但是，如果随机排列把 4 放在第一位，那么 A 中的第一个元素将不同于 B 中的第一个元素，5 也是一样。
[35:39.060] 所以看这个表示。基本上，随机哈希函数说从这些行中选一个作为第一。然后当最小值告诉我——A 的最小值等于 B 的最小值，基本上就是在告诉我那一行是否相同。好的。所以这大概就是为什么这个性质成立的证明，即你在随机选择哈希函数的期望上，碰撞的概率就是你想要的相似度度量。好的。那让我们用代码检查一下。我将生成 100 个不同的哈希函数。每个哈希函数由一个种子给定。然后我检查 MinHash 是否等于 MinHash，以及估计的 Jaccard 值。所以我们看一下匹配的分数或比例，你会得到 0.6。MinHash 的关键在于，我不需要做 n 平方的计算。我可以计算 A 的 MinHash、B 的 MinHash 以及另一个集合的 MinHash，然后只需查找碰撞即可。
[37:16.010] 关于 MinHash 到目前为止有什么问题吗？现在我们可以对我们的项目进行哈希了，但我们还没有完成，因为碰撞并不能告诉我们 Jaccard 值是否高于某个阈值，而这才是我们想要的。我们要找的是 Jaccard 大于 0.99 的 A 和 B。我们目前所做的只是说，好的，如果我们发生了碰撞，两个东西碰撞的概率就是 Jaccard 值。但这本身并不是很有用。它是随机的，我无法从这里得到任何可靠的结果。所以下一个想法是叫做局部敏感哈希的东西，它基本上解决了这个问题。这是理论计算机科学中一个非常经典的思想，而且相当巧妙。
[38:17.530] 这里的想法是，我们让 A 和 B 以等于 Jaccard 值的概率碰撞。所以确实，更相似的项目会更容易碰撞，但这非常随机。这里的方差相当大。所以我们的目标是，如果 Jaccard 大于某个阈值，就让 A 和 B 碰撞。所以，在某种意义上，我们必须以某种方式锐化这些概率。概率不能只是字面上等于 Jaccard 值。所以解决方案是使用更多的哈希函数。这些哈希函数将是独立的。这部分会有点技术性，但我们会一步步讲清楚。你把 n 个哈希函数分成 b 个波段，每个波段有 r 个哈希函数。所以如果你有 12 个哈希函数，你有三个波段，每个波段有 4 个哈希函数。所以这里有一个波段，h1 到 h4。这是第二个波段，h5 到 h8。第三个波段，h9 到 h12。
[39:33.760] 然后我们要尝试做的是——每个哈希函数要么给碰撞，要么给不碰撞。所以每个哈希函数都有一个碰撞概率。这里的关键是，我们要计算 A 和 B 碰撞的概率。我们说，如果对于某个波段，其所有的哈希函数都返回相同的值，那么 A 和 B 就碰撞了。所以，如果 h5、h6、h7 和 h8 都返回相同的值——抱歉——如果 A 的 h5 等于 B 的 h5，并且 A 的 h6 等于 B 的 h6，以此类推，那就意味着这个波段被触发了，然后我会说它们发生了碰撞。或者如果这个波段，如果所有这些都一致，那么我说这是碰撞，或者这两个。
[40:41.920] 但你不必让所有的哈希函数都返回相同的值。所以这里有这种与-或结构在起作用。我们会看到为什么这样有效。现在，假设你有一个特定的 A 和 B 的 Jaccard 值，根据这个定义，A 和 B 碰撞的概率是多少？所以我们可以计算这个。假设杰卡德相似度是 0.8。我们有五个频带，每个频带有 10 个哈希函数。那么我们可以问，一个固定频带匹配的概率是多少？那就是 0.8 的 r 次方。因为每个频带有 r 个哈希函数，所以它们全部匹配的概率就是 0.8 的 r 次方。所以一个固定频带匹配的概率通常相当低。它随 r 呈指数级下降。然后碰撞概率就是某个频带匹配的概率。那就是 1 减去——这是某个频带不匹配的概率。
[42:19.590] 如果你将它提高到 b 次方——因为有 b 个频带——这就是所有频带都不匹配的概率。然后 1 减去这个，就是有某个频带匹配的概率。所以碰撞概率是 0.4。而且预计会更高，因为你有 B 次尝试去获得匹配。如果你把它画出来，就是这个样子。横轴是相似度。我们刚才看的是 0.8。纵轴是碰撞概率。我们可以预期，如果相似度为 0，那么碰撞概率就应该是 0。如果是 1，就是 1。但请注意，这是一个有趣的 S 形曲线，这很好，因为这正是我们想要锐化的。我们基本想要做到的是，如果相似度低于某个阈值，我们希望匹配概率尽可能低。而如果相似度高于某个阈值，我们希望它尽可能高。
[43:34.320] 所以我们试图让它像一个相变过程。如果你把这个函数画成相似度的函数，你得到的就是这个。好的。让我们看一些例子。具体来说，我们看相似度从 0.7 到 0.98 的情况。我们将观察 b 等于 10，r 等于 10 的情形。我们根据定义来看碰撞概率。我们看到这给出了从 0.25 到 1 的范围。所以如果你要设定一个阈值，这还不算太差。我们可以在这里设置阈值。而这些在某个概率下仍然——它们仍然是假阳性，因为即使这些低于某个阈值，比如 0.9，它们仍然可能碰撞。但如果我们愿意，可以过滤掉它们。而高于阈值的大部分都被保留了。那么当你改变——也就是增大 r 时会发生什么。记住，r 是一个桶内的哈希函数数量。所以当你增大 r，阈值会变尖锐，并且曲线会向右移动。
[45:22.393] 一切都变得更难匹配，因为桶内有更多哈希函数，这会使那个指数更尖锐。所以你会看到这里的概率更尖锐了，现在一切都向右移动了。现在，如果某样东西的杰卡德相似度较低，你匹配到它的可能性就相当低了。之前是 0.25，现在变成了 0.008。所以我们正在锐化概率。而增大 b 的效果是将曲线向左移动。这使得匹配更容易。b 是频带的数量。必须有某个频带匹配。如果你有更多频带，就有更多匹配的机会。如果你看这个，这是之前和之后的对比。现在我们让——之前，0.9 只有 0.72，现在变成了 0.92。当然，这也有所增加，但增幅不大。你可以通过增大 b 和 r，将这些相变驱动到你想要的任何尖锐程度。
[46:30.850] 但当然，如果你把 b 和 r 增大太多，那可能会更昂贵。让我们看一个更接近真实世界的设置。在这篇关于去重的论文中，他们有 b，20 个频带，r 是 450。这让你对这些数字的量级有个概念。一般来说，相变发生在一个阈值处，即 1/b 的 1/r 次方。所以如果你想基于杰卡德相似度大于 0.9 来过滤，你基本上必须设置 b 和 r，使得这个值等于 0.9。然后通过改变 b 和 r，你可以让这个相变更尖锐。所以记住，在这个阈值处，固定频带匹配的概率是 1/b。那么 A 和 B 碰撞的概率大约是一个常数。所以碰撞概率是 0.64。这是有道理的。基本上，如果你有一个相变，那个相变的中心就是 0.64。
[47:59.020] 随着 b 和 r 增大，低于它的所有东西都应该趋向于 0，高于它的所有东西都应该趋向于 1。关于 LSH 有什么问题吗？这种方法被称为 MinHash LSH，因为 LSH 实际上适用于任何哈希函数。对于语言模型处理的去重，我们使用 MinHash，它近似杰卡德相似度。所以基本上这就是你在这里应该使用的方法。关于去重有问题吗？关于去重的一个注意事项是，正如我们将看到的，经常有数据集不断进来。有时去重会在单个数据集内进行。但你实际上必须对整个数据集进行去重，因为通常数据集之间可能是冗余的。所以有时这一点没有被做到，但应该做到。让我们进入下一个话题，即数据混合。到目前为止，我们已经将数据从真正原始的 HTML 或 PDF 转换为文本。我们进行了过滤以获取高质量内容。我们进行了去重，所以我们有了一组更小的高质量文档。
[49:42.776] 这通常是对给定的数据源进行的。但语言模型是在多个数据源上训练的。所以在 Marin，目前这个网站追踪着下一代模型将要训练的不同数据源。你可以看到有一大堆东西。有 Nemotron，有我们讨论过的 FinePDFs。有一些机构书籍，还有一些代码，以及所有这些内容。所以你有所有这些不同的来源。问题是，你如何组合它们？如果你看一篇较早的论文，The Pile，那是他们当时拥有的来源集合。他们基本上给每个组成部分分配了特定的权重。所以基本上，你有一个关于来源的分布。那么这个权重从哪里来？直接说吧。假设你有三个来源。你想要找到一个数据混合方案，而数据混合方案就是你的来源上的一个分布。所以如果你只是从第一性原理来考虑这个问题——
[51:21.260] 嗯，不是从第一性原理，而是从你可能做的事情出发。凭感觉，我想这是第一性原理的反面，你只是根据某些直觉手动设置，这比你可能想象的更常见，人们就是这么做的。所以我认为这绝对相当依赖感觉。即使是更近期的论文，你也只是看看——也许你使用某种方法，然后进行调整。你也可以做均匀采样，就是对你的来源施加均匀分布，然后你采样每个块。你只需采样一个来源。你也可以做比例混合，即你按照来源中 token 数量的比例进行采样。所以如果你有一个 token 更多的数据集，那么你就给予更高的权重。这通常也是一件合理的事情。但你可能会担心，如果你有一个巨大的低质量数据集，它会消耗掉你大量的 token。
[52:31.400] 所以这看起来也不是完全最优的。所以直觉上，你应该提高高质量来源的权重。但有两件重要的事情需要记住。一是你确实想要确保一定的多样性。通常，文学、代码和论文这些来源是不可比较的。你不能真正说这篇论文比那段代码质量更高，因为它们可能只是不可比较的对象。而且如果你希望你的语言模型表现良好，你不想把所有的权重都放在论文上。第二件事，我稍后会多谈一些，就是每个来源实际上是有限的。所以，如果你对一个小数据源赋予过高的权重，那么你基本上会把这个数据源用完，然后你就需要不断地重复遍历它，这意味着你一直在训练完全相同的token。这将会带来问题，我们稍后会看到原因。好的，让我们尝试更具体地解析这最后一点。
[53:45.490] 想象一下你只有两个数据源——一个低质量的数据源，你有10万亿个token，还有一个高质量的数据源，它有100亿个token。通常来说，高质量的数据源规模更小。所以如果你只是做一个简单的数据混合，比如我们采用均匀混合。也就是高质量和低质量各占一半。然后我要训练1万亿个token。这里说的训练1万亿，并不是指1万亿个唯一的token。它仅仅意味着训练步数等于这个总数除以批量大小，或者乘以批量大小。那么让我们来看一下这个，这是低质量数据集中特定元素被训练的次数。也就是某个特定数据点被训练的轮数（epochs）。那么p低乘以训练token数，就是我需要从这个低质量数据源请求token的次数，再除以低质量数据源的总token数。所以这个值是小于1的。
[54:56.740] 这基本上意味着，在这个低质量数据集中，只有5%的token会被我用到并训练一次。然后如果你看看高质量数据，你会发现训练轮数是50，因为我这里只有100亿个token。如果我是各占一半，那就意味着需要5000亿个token——我需要训练5000亿个高质量数据的token。但我没有5000亿个token的数据，所以我必须将每个数据点重复训练50次。所以这实际上非常重要。一些大型模型的训练运行就在这个问题上出过错。你不能仅仅看着数据集，定义一个分布，然后就去采样。因为如果你只看质量，只看数据，你会忽略那里到底有多少个数据点。
[56:00.080] 大家明白了吗？也许可以暂停一下，有问题可以问。这是一个重要的点。是的。我猜想，对于高质量数据……低质量数据……为什么首先需要50轮呢？难道用更少的轮数不能达到相当的性能水平吗？是的。所以问题是，为什么需要50轮？我想说的是，你并不需要50轮。最坏的情况下，这是在浪费计算资源。抱歉，最好的情况是浪费计算资源，最坏的情况是你过拟合了。之所以是50轮，是因为如果你直接去定义这个数据混合，那么你最终会在不知不觉中进行50轮训练，除非你非常密切地关注。所以这基本上就是——教训是，要看看你在数据上实际进行的训练轮数是多少。是的。混合方式在训练过程中是如何体现的？当你每一步都切换时，你训练10步，比如说，“哦，我想做10份Python代码，10篇文章”，因为我觉得——在这些决策背后是否也有一些直觉呢？
[57:25.530] 是的。好问题。所以问题是，当你训练时，混合是如何实现的？一般来说，你只需要采样。你想填充一个批次。你可以为每个元素采样它来自哪个混合组件。然后你用来自该混合组件的序列填充一个批次。通常，每个序列都来自一个混合组件。你不是按token采样的。所以每个批次应该是混合的。所以我猜混合的假设是在批次层面上的，然后你按照这个来训练。这并不是说，“哦，一步就是——代码一步就是——”是的。一般来说，你想减少方差。你想训练多个——批次应该有一些混合。好的。所以这个问题在很久以前就被注意到了。因此，在这篇论文中，他们引入了UniMax。在这种情况下，他们是在训练多语言模型。
[58:41.660] 而且非常清楚的是，有些语言是高资源语言，有些语言是低资源语言。对于低资源语言，你必须采取一些措施。在这篇论文中，他们注意到，之前一些工作本质上会采用按比例的混合，然后将其提升到某个幂次，以使分布变得更平坦。但这篇论文的想法是，让我们对此更加明确一些。让我们均匀地对数据源进行采样，但是对训练轮数设置一个硬上限。这就是关键的想法。在这里，你说，我只会在一个特定的数据源上训练20轮。如果你已经达到了这个上限，那就没办法了。你不能再获得更多的token了。你继续下一个数据源。所以这在某种意义上像一个安全网。特别是，如果你看一下这个数字，分配给一个数据源的概率乘以训练token的数量，应该小于或等于这个上限。
[59:51.180] 并且有一个简单的程序可以实际确定满足这个约束的混合方案。那么让我们稍微换个话题，讨论一下我们首先如何定义混合方案，因为如果你有50个数据源，你就有50个数字需要填写。我们注意到，按比例混合并不足够。也许你可以尝试估计一些质量指标，然后根据这个指标进行比例采样，但这也是启发式的。所以最 principled（有原则的）且最容易理解的方法是基于回归的混合方法，即RegMix、Olmix以及其他类似的论文。这里的想法相对简单。首先，假设你想要训练一个大规模模型。那么你先跑到小规模，比如3亿参数，然后尝试不同的数据混合。假设你有三个组件。你尝试不同的混合方案。
[01:01:06.050] 然后你训练这些小模型。你训练一群小模型。每个模型都会在某个目标指标上给你一个损失值。这可能是某些下游评估值。也可能是困惑度，等等，取决于你想要什么。然后你利用这些数据点来拟合一个回归模型，该回归模型将这些输入，即数据混合权重，映射到损失值。现在你基本上有了一个非常廉价的模型，它可以告诉你，如果我在这个混合方案上训练，我会得到什么样的损失？然后你可以优化这个数据混合方案——抱歉——优化这个函数以找到最优的数据混合方案。然后这个混合方案就是你用来训练大规模模型的方案。这就是简单的程序。你会注意到这与缩放定律（scaling laws）有一些相似之处，都是试图通过一些廉价的计算来找出答案，然后进行规模化扩展。好的。这里有几个设计决策。一个是，你要尝试的混合方案是什么？
[01:02:27.000] 你需要一个关于分布的分布。所以人们通常使用某种狄利克雷分布（Dirichlet distribution）。你还必须定义回归方法。人们尝试过线性模型或提升决策树。你必须确定这里的目标是什么。这通常基于下游评估。但是你必须非常小心不要过拟合，因为我们做的是预训练。按理说，预训练应该是训练这个通用模型，我们并不是要拟合某些下游评估。如果你不小心，例如，如果你有大量的代码评估任务，那么，你猜怎么着？你会调高所有代码数据的权重。这不是什么高深莫测的事情。然后如果你接着说，我想生成一些诗歌，你可能会意识到你已经过拟合了。所以你必须非常小心。按比例混合和均匀混合则没有这个问题，因为你并没有去查看这些。
[01:03:24.725] 没有下游评估。最后一点是，小规模和大规模之间的差异有多大？这是一个成本与准确度的权衡。当然，如果你训练非常非常小的模型，这可能不具有代表性。但如果你训练大模型来做这件事，那做这些就毫无意义了。你直接训练就行了。你基本上就是在最大规模上做超参数调优，那太昂贵了。所以 Olmix 这篇论文实际上有一个很好的表格，它展示了多种都属于这个框架的方法。代理模型的规模。它们通常大约有数千万个参数。你采样多少个混合？这里的 m 是混合中的参数数量——抱歉——是领域数量。你可以用 Dirichlet 分布，也可以用指数分布。然后回归模型，你可以拟合一个对数线性模型，这通常效果不错。
[01:04:42.510] 然后你可以用不同的方法来解决这个优化问题。所以这里有两个你需要非常小心的信仰飞跃：回归模型是在一堆小型代理实验上训练的，而你在优化它。所以希望是最小化器，也就是你的最优数据混合，那个回归模型仍然准确。这有点——你必须非常小心，因为如果你，比如说，随机采样一个混合，你可能会没事，因为经典的泛化原理。你采样一个混合，拟合一个函数，它应该能够在分布内进行预测。但当你做优化时，你本质上是在试图走向潜在的极端，而在这些极端你可能没有足够的覆盖。所以这是需要注意的一点。另一件事是，最优数据混合，你只能希望它们能从大规模迁移到小规模——抱歉，是从小规模迁移到大规模。
[01:05:55.070] 总的来说，至少在开放社区工作的规模上，这似乎倾向于成立，或者至少不是明显不成立。但显然，存在规模相关效应。你只要想想上一张幻灯片我们讨论过滤时说的，如果你训练更多的 token，那么低质量数据可能也没关系。所以很明显，最优解并不相同，但你只能往好的方面想。好，等等。这里有一个我们已经讨论过的规模相关效应，需要在此时提出，那就是，想象一下，我们有很多低质量数据，10 万亿 token，以及 100 亿 token 的高质量数据。如果我们在低 token 数量上训练一个小模型，你记得吗？会发生什么？你可能会——这是一个虚构的例子——你可能会把很多权重放在高质量数据上，因为低 token 数量下，你还没有重复轮次。所以就像，哇，维基百科太棒了。
[01:07:06.520] 我们就只训练维基百科吧。但如果你接着用这个混合去训练大模型，那么你最终会在这些高质量数据上重复很多轮次，然后你会过拟合。那肯定是不好的。所以解决这个问题的一个方法，Olmix 论文中就是这样做的，就是你可以限制重复轮次的上限。所以这是一种解决方案。还有另一种解决方案，可能有好几个名字。但其中一个名字是模拟重复轮次。这里的原理是让你的小规模看起来像你的大规模。这是整个课程的一个主题。我们在讨论 muP 时看到过。你参数化你的模型，让你的超参数可以迁移。同样的原理也适用于这里。而做这种朴素缩放的问题在于，你不能在小规模上不重复轮次，然后在大规模上重复轮次，因为那对你的数据集来说是性质不同的操作。
[01:08:25.010] 所以解决这个问题的方法是按比例对数据源进行下采样。如果你的小规模实验使用 100 亿 token，而你的大规模实验使用 1 万亿 token，那么你有一个 1:100 的下采样比例。然后你基本上使用这个下采样后的混合数据。想法是，如果你有下采样后的数据，那么那些方案看起来就不会好，因为你没有在所有数据上训练——你不能只训练维基百科。你会在维基百科的极小一部分上训练，然后你会发现，哦，实际上，我会重复很多轮次。那会带来非常糟糕的损失，这意味着在优化过程中，你会寻找更平衡的方案。所以你在小规模上模拟你在高规模 regime 中会遇到的数据稀缺性。总结一下这一节，问题是如何给不同的数据源加权？
[01:09:42.569] 维基百科，你有 Common Crawl。你有代码。你有从某个地方爬取的数学数据。你如何平衡它们？基于回归的混合是一个很好的思考框架，我认为。你估计你的函数形式，将混合权重映射到小规模的损失，进行优化，然后泛化到大规模。你只需要非常小心，因为存在重复轮次和过拟合的问题。你必须要么使用限制轮次，要么使用模拟轮次。所以这里的一个教训是，如果你试图优化任何东西，你必须非常小心，因为你有优化错误目标的风险。在我们继续之前，关于数据混合有什么问题吗？当你完成采样后，是否存在数据量太小而无法很好泛化的情况？所以问题是，如果你做下采样，是否有风险？也许你得到的数据集太小了。是的，绝对有。
[01:11:02.480] 我认为你可能会遇到 token 太少的情况。我想那样的话，你的最优解就会在那个数据源上分配很小的权重。当你放大规模时，也许你会得到正确的 token 集合。所以原则上，这应该能行。但可能会有舍入误差，导致你最终不是在这个数据集上训练一次，而是碰巧训练了零次。但你可以通过——只需总是在这个数据上训练一次来解决这些问题。好的。是的。在数据混合中，我想这些数据源恰好对应不同的主题。但你有没有拿一个多样化的数据集，尝试在它内部做数据混合？是的。所以问题是，你把数据混合应用在什么上？不同的领域，甚至在一个数据集内部？实际上，我忘了提到这个，所以很高兴你提出来。在 Nemotron 论文中，他们实际上也做了那个 [听不清] 的事情。
[01:12:17.430] 你基本上可以这样想——假设我只给你 Common Crawl。你实际上可以把它分解开来，你可以按领域分组。IT 那边的人有那个网络组织器，你可以按主题分组。但你也可以做质量过滤。这样你就有了一个二维网格，有领域，还有质量。每个单元格都是一个你可以做数据混合的东西。所以这是一种自动确定领域的方法。然后在此基础上，你再加上别人交给你的额外数据源。我接下来快速过一下后训练数据。到目前为止，这基本上都是预训练或者可能是中训练数据。它通常是与任务无关的，除了 RegMix 那种试图最小化损失的工作。但数据本身仍然是相对任务无关的，你在试图发展基础能力。所以当你看到后训练时，很多数据就变得非常依赖具体任务了。
[01:13:32.990] 我不打算做全面的综述。我只想指出一些最近发布的、有趣的训练后数据集，特别是针对编码领域的数据集，因为这是当前大家非常感兴趣的领域。所以，基本的流程是，你定义一组环境。在这种情况下，考虑代码，可能指GitHub仓库。你定义一组任务或提示，然后从一个强大的模型或教师模型那里收集回复。因此，至少在开源社区中，隐含地，几乎所有训练后的数据，大部分都是合成生成的。你也可以用一个人类来替代那个强大的模型，但这很慢且成本高昂。但是，如果你处在最前沿——几年前，你不得不花钱雇很多人来给你回复。而如今，即使是在最前沿，你也可以采用人机混合的方式。
[01:14:47.320] 但不管怎样，重点是，有某个教师模型在为你提供回复。好的。我这里只谈几个工作。其中一个是名为OpenThoughts的工作。这个想法是——它出现的背景是，当o1模型发布时，引起了人们对推理能力的大量关注，主要集中在数学和科学领域。我们如何为这个领域获得真正好的训练后数据集？最终他们使用这个教师模型生成了120万个示例。所以步骤如下：首先，有一堆——那么环境和任务及问题是什么？他们从很多不同的来源中提取。有人类来源，如Stack Exchange或NuminaMath，还有一些合成来源。这些只是编码相关的一部分，但还有更多关于数学和化学的。这是一个大项目，涉及许多人的大量不同贡献。
[01:16:04.410] 所以你可以看到，其中一些是编码练习。一些是更现实的、出现在——我想，CodeGolf 不算现实，但这里有一些更现实的例子。比如代码审查，这可能更现实。他们对此数据集进行了相当全面的分析，即如何生成回复？他们表明，使用少量来源实际上比试图使用所有来源更好。对多个生成结果进行采样是有帮助的，比如采样16个。有趣的一点是，更强的模型不一定就是更好的教师。例如，QwQ-32B，一个现在看起来非常老旧且规模小的模型，作为教师却比DeepSeek-R1更好，而DeepSeek-R1在当时可能是最强的开放模型之一。他们还发现，基本的答案过滤并无帮助。因此，整个流程看起来是这样的：你有一堆这些来源输入进去，然后你去重，随机采样。我想这是对问题进行下采样，然后你生成多个答案，这些最终进入数据集。
[01:17:34.270] 所以，120万是示例数，但除以16就得到实际问题的数量。所以那是针对数学、科学和一些代码的。我认为最近，人们对开发特定的代理式编码模型——不仅仅是一个能生成一些代码的模型，而是一个实际上能进行软件开发的模型——产生了很大的兴趣。所以，这篇SWE-smith论文有个想法——你得到一个代码仓库，你用一个语言模型来自动生成任务。他们有一个智能体，它接收一个仓库，并实际使其可用，比如安装依赖等等。然后你生成任务，这些任务通常是——你以某种方式修改代码，可能还会引入一些错误。这些会得到验证，你就得到了任务实例。所以这些是合成任务，但你得到了5万个这样的任务，这实际上在当时，也就是去年，是相当大的规模。从那以后，还有一系列其他的工作，我会提一下。比如NVIDIA的SWE-Zero论文，它有这个——实际上，好的，我也会谈谈那个。
[01:19:14.620] 这是一个有趣的想法，其观察是，与数学不同，SWE任务有很多重的依赖关系。大多数GitHub仓库甚至无法直接运行。你必须安装所有这些依赖。它们已经过时了，特别是如果你回退到一个PR刚提出时的混乱状态。这简直就是一场噩梦。所以他们思考，我们怎么能真正地在所有仓库上获得数据集？他们没有为每个仓库制作特定的Docker镜像，而是注意到模型已经足够好了，可以在没有执行反馈的情况下解决许多这类任务。所以如果你看看其中一些模型，如果允许执行，你得到80分。如果不允许执行，你几乎能得到70分，对于不能执行代码来说，这已经不错了。所以某种程度上，这些模型对代码有一些内部语义理解。于是他们能够生成30万个智能体轨迹。
[01:20:29.670] 所有这些都来自真实的GitHub PR，所以它们在SWE-smith示例上是很现实的。他们使用OpenHands脚手架。有很多关于如何防止智能体“作弊”的细节。通常，你给智能体一个这样的指令：你探索、你测试、你实现。SWE-Zero版本基本上是说，你不能运行Python代码。你只能使用 set、grep 和所有这些基本操作。好的。然后他们从大的Qwen模型进行蒸馏，并进行过滤，因为有时Qwen模型会忽略这些指令，仍然试图执行。然后他们表明，有了这个——哦，他们还有13k个确实需要执行反馈的智能体轨迹。所以他们训练模型时，先在这些SWE-Zero示例上进行微调，然后再在这些示例上微调。他们能够——我想前沿水平仍然相当高，但他们能够在这方面取得一些进展。
[01:21:45.350] 我会快速过一下SWE-rebench，它本质上也是另一种尝试抓取海量PR的方法。所有这些看起来都是，获取一堆GitHub仓库，尝试安装仓库。大多数可能失败了，然后更努力地尝试。然后你使用一个语言模型来给你回复。这些实际上是今天刚发布的。所以你可以采用SWE-Zero的想法，现在可以将其扩展到1200万个智能体轨迹。SWE-Zero的一个好处是它非常轻量级。在这里，他们使用了SWE-rebench任务。所以在这个工作中，记住，他们试图让任务执行起来。他们只让其中32,000个成功执行，有120个没有执行。但是SWE-Zero不在乎。你可以全部都使用。这是一个非常小的模型，因为——现在这个数据集相当大。所以我认为，正如你所看到的，这些数据集变得越来越复杂精妙。
[01:23:04.880] 你从像数学这样无需环境的事情，发展到了现在的编码领域。现在编码数据集增长得相当快。但总的思路是，你有这些提示，这里有各种权衡。你可以是完全合成的。你可以是半合成的，即有真实环境和合成任务，或者你可以是真实的。而回复通常来自能力强的模型，但它们也需要是好的教师。代码环境是一个麻烦事。还有大量的过滤和其他细节，我们没时间讲了。好的。总结一下这节课，我们谈了过滤。即，定义什么是“好”的，然后训练一个轻量级分类器。浏览你的网页抓取数据，你可以得到一小部分符合你要求的子集。去重对于避免过拟合和节省算力很重要。混合。在小规模上尝试混合比例，然后外推到大规模。
[01:24:09.930] 之后我们又看了一些训练后的数据。不过我得说，很多数据处理工作其实非常琐碎枯燥，高度依赖具体领域，需要逐个查看具体实例才能构建出高质量的数据集。所以这堂课并不能真正代表数据工作的全貌，但希望能让你们对当前的数据版图有一个大致的概念。好，那今天的内容就到这里了。[掌声]
CS336 2026 · Lecture 14 · Data (filtering, deduplication, mixing, synthetic data)
- 视频：官方视频
- 字幕：en-US · manual captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：原语言字幕稿，待校对
正文
[00:05.240] OK, let's get started. So today is the second day on data. Last time we talked about how data doesn't really just fall from the sky. You actually have to think about where it comes from. So in general, the internet consists of live services. That data on those services have to be either dumped or crawled. And then there's an additional step where you have to process the data. And we also talked about various societal considerations, terms of service, copyright. You have to either get a license or appeal to fair use. So there's a lot of complexity that goes into data. So today we're going to look more at the data pipeline from transforming the data, filtering the data, deduplication, and mixing. And then we're going to top it off by talking a bit about post-training data, in particular, how folks are using synthetic data these days. So the first part is going to be mostly about pre-training. That's what you should have in mind. So we talked a little bit about data transformation already, but just as a reminder. So raw data doesn't come as text, even as if you've scraped something. If you ever look inside Common Crawl, it's not text.
[01:33.820] It's either HTML. Sometimes it could be PDFs or directories in the case of GitHub. So most of the attention on transforming data is dealing with HTML because most of the web is in HTML. And for this processing, a lot of this is fairly heuristic. There's removing of boilerplate, like navigation and ads, and extracting content, which is the main part of the page. And there's some subtleties around what constitutes content. Usually, you get rid of the footers and headers, and maybe menus, and those things, and you try to extract the content. But you could imagine in cases where some of the navigation elements might be helpful to learn what web pages look like. So what is content and what is not content is not always clear. And then what do you do about images and tables that are in web pages? So inherently, this is a lossy process because you need to linearize HTML, which at least is either hierarchical or visual, if you think about the rendered output, and to a sequence of tokens.
[02:52.150] And particular tables are a bit tricky to deal with. Simple tables, you can render using markdown. But if you have nested tables, then that becomes quite challenging. You have to give up at some point or approximate at some point. So, typically, HTML to text processing is rule-based. And the reason for this is that rule-based processors are very fast. And also, you're not trying to do too much here. You don't need too much intelligence. So rule-based generally works. Now, I think there could be a case for model-based interventions at this point. They have to be very fast, and they have to do something more intelligent. But if you ever look at data, you'll notice there are imperfections in the data just because any rule-based processing have some failure rate. As we showed last time, the accuracy does matter, depending on which tool you choose.
[04:05.620] If you use Resiliparse or Trafilatura, then-- I guess Resiliparse on these extended dclm evals works better than the others. I'll talk also briefly about PDFs. So there's Hugging Face work, which released a data set called FinePDFs. So PDF files, if you ever opened them up, not in a PDF reader, looks like this, and this needs to get rendered. So PDFs can be found on just web pages on the internet. And Common Crawl does have some PDFs. Generally, Common Crawl focuses on text, but sometimes, if you're just given a URL, you might not even know before you fetch it whether it's a PDF or not, if it doesn't have the extension.
[05:03.640] If you read this blog post, there's a lot of details, which I'll spare you of. For example, one thing is that many of the PDFs that are in Common Crawl are truncated because PDFs are big, so then that demands that recrawling is necessary. And then there's a question-- once you have the PDF file, how do you actually convert it to text? And, some PDFs might be just scans as well. So they're essentially images. So there's a bunch of tools that this paper tried out, but mostly this involves running OCR using a VLM. And this obviously can be much more expensive than what we were doing before with text. Fortunately or unfortunately, the PDFs are a very small fraction of the whole internet. But they are very valuable because generally, if you bother to make a PDF, that means you probably have something interesting to say as opposed to a web page. So the quality of an average PDF is generally higher than for HTML file.
[06:14.810] There's a lot of cleanup and filtering. And PDFs, more so even than web pages, a lot of layout information is missing, because in HTML you have various tags like h1 and p that gives you some semantic information. PDFs are, by design, all about layout. So you don't necessarily preserve the semantic structure. So there's a bunch of transformation that happens. At this point, you have text, but you're not done. You're far from done. So the next step is filtering. So I'm going to talk somewhat abstractly about what filtering is. Here's the building block. So suppose you have some target data that you want to get. This is usually a small amount of high-quality data and lots of raw data. This is the fresh shipment of the tokens that you transformed from the previous step. And the goal is to find a subset of this raw data that is similar to target. So this is the general skeleton for filtering, and almost all kinds of filtering falls into this schema.
[07:39.410] So filtering, there's many reasons you might want to filter. If you're training an English language model or a German language model, you might want to identify the language and filter things that don't match that language. The main reason for filtering is quality filtering. You want to find things that are high quality as opposed to low quality. You don't want just spam. You want encyclopedic information. And then another application is toxicity filtering. Of course, the internet has plenty of nasty content, and maybe you don't want to train your language model on that. So, for a filtering algorithm, you want some generalization from the target data, because you already have the target data. You don't want to only get the target data, but you also want it to be extremely fast because you have to run it on the whole internet.
[08:38.140] So this could be 100 trillion tokens worth of data. So generally, filtering is you end up with a very small fraction, a single-digit fraction of your entire data. OK. So remember the general framework given the target, and raw, you're trying to find a subset. So the general scheme is that you estimate some model based on R and T and derive a scoring function, and then you keep the examples in R based on their score. So typically, there are two types of classifiers. One are generative models. So this is basically you have your target data. You can just estimate a model of that data. And remember, this has to be cheap, so probably you're not training a big language model. Generally, KenLM basically says I'm going to train a 5-gram model. A more common thing, which I think we're mostly seeing these days, is just training a classifier.
[09:48.230] And the classifier says, I'm going to predict a positive label, for example, that are in T, my target, and negative labels for things that are in raw but not in T. So basically, it looks like T are your positive examples, some random subset of R are your negative examples. Maybe you balance it, and then you train a classifier. And the tool that people generally use is fastText because it's fast. And generally, it's just a linear classifier-- a bag of words. And then once you have this model now for every new document, you can score it, and you set some appropriate threshold, depending on your quality bar, and you keep the examples. Sometimes stochastically, sometimes not. So this is very much a model-based filtering approach. Remember from the last lecture, there are many data sets from generally a few years ago that did not use model-based filtering because they wanted to not bias things too much.
[10:59.590] These days, I think basically everyone does some amount of model-based filtering because unless you are compute-plentiful, in which case you probably don't need to do as much filtering-- you can just train on everything. Most people are compute-poor. You have to be very smart about how you're filtering; otherwise, you're just wasting flops on low-quality content. OK, so let me talk through some instantiations of filtering. So remember, I mentioned language identification. So the goal is just to find-- given a piece of text, detect whether it's of a particular language. So Meta has trained this set of fastText language identification models, which often you can just use off the shelf. So it supports 176 different languages. It's been trained on a bunch of multilingual sites. So Wikipedia, remember, has a lot of languages. There's also translation sites and sites for different languages. So a language ID is generally a fairly easy problem compared to many other tasks that we're dealing with-- so a simple classifier.
[12:14.120] If you look at a few words, you can tell that it's Spanish or Japanese. Now, there are subtleties because of code-switching and text, and there's some dialects. So I wouldn't say it absolutely solved problem, but this is not really the bottleneck for training a good language model. And then you can just-- so once you have your classifier, you just choose some threshold. This is generally fairly heuristic. So another thing you can do with filtering is that, suppose you're looking for data of a certain type. So let's say you want to get really good at math, so you want to go and find a bunch of math data. So OpenMathText is this paper from 2023, where the goal is to create a large corpus of math text. This pipeline consists of a few steps. So it's not training a single classifier. So first, you use some rules to filter. Does it contain latex commands? They also use KenLM.
[13:17.620] So this is a generative approach. And trained it on the ProofPile, which is a known data set of math, and keep it if the perplexity is below some threshold. And then you also train a fastText classifier to predict whether it's mathematical writing or not. And there's two different thresholds if it has latex in it, then it's a higher bar-- sorry-- it's a lower bar, but if it doesn't have latex in it, it's a higher bar. And then, as a result, they got 15 billion tokens, which they use to train models. And there's a paper that showed that this more targeted, curated data collection results in models that are better at math than models that were trained on 20 times as much data, which were not filtered in this way. So the quality filtering again, think about it as a tool. You can define quality however you want. There's no universal notion of quality.
[14:21.630] If you want to define quality to be math, then you can go and get math, and you get better at it. So GPT-3, I think I mentioned this last time, but just to put it in this framework. So positive examples are Wikipedia, WebText. So these are pages that link out from high-star Reddit posts, and some books. And negatives are generally sampled from the web. They train a linear classifier and keep documents if the linear classifier is scored highly enough. There's the first LLaMA paper. The positives were pages referenced by Wikipedia, not Wikipedia articles themselves. And it's this same idea. So phi-1 from Microsoft is interestingly also falls into this framework. So they started with-- the raw data is not already a Python subset of the stack. Remember this is all code. And they define a prompt, which determines educational value.
[15:37.720] And then they use GPT-4 to classify a subset of R, 100k subset of R with this prompt, and the ones that are positive are kept and called the target. So the target here is actually the output of an expensive classifier. And then you train a cheaper classifier, in their case, a Random Forest, but you could have probably used fastText as well. And then you select data that is classified positive by this classifier. And they also showed that using this data set, you do much better than if you're just using the raw data. So this performance goes up in fewer steps to a higher value. So toxicity works the same way. There's this Dataset-- Jigsaw Toxic Comments, where it came from this project where the goal is to help people have better discussions online. So here the data is the Wikipedia talk pages, which can, I guess, for controversial sites get quite heated. And this has been annotated with whether there are any toxic comments on there.
[17:01.200] And you can similarly define positive and negative examples, and you can train a classifier on that. So now I think you have the tools, or some examples and inspiration. You can identify a particular type of data that you want. You build a classifier for it, and then you can go filter Common Crawl for that. I want to talk about one subtlety here, which is that-- no, the notion of what you want for data actually depends on what model you want to train. So in particular, it depends on the number of tokens you're training on. And so there's no optimal threshold. So before, when I looked at the classifier, it gives you a score. You can't say 0.9 and say that's the best because it depends on what you want to do. Intuitively, if you are going to train for a longer period of time, then you can tolerate lower quality data.
[18:02.350] If you're training for shorter, then you want higher quality data, in general. And of course, if you could wave a magic wand, if you're training for longer, you want more high-quality data, but that's not an option that you're given. The data pool is what it is. So here's a preliminary experiment that Michael Ryan did. So basically, this plot shows for a 157-million parameter model. We're taking a very small pool. So 100 works, which is a tiny fraction of Common Crawl. And we're training and training for more over time. So let's take a look at the blue curve. So the blue curve is dclm. And so the loss starts here. And then it comes down. And each of these lines is when you epoch over the data. So this is one epoch, and then the next blue is the second epoch, and so on and so forth.
[19:12.480] So there's not that much data. So eventually, you have to repeat your data. And the loss continues going down because the second time you look at the data, you're still learning things. And at some point, you start to overfit. Whereas if you look at Resiliparse, this is basically no filtering. So here, it's much worse in the beginning. But as you continue and continue, at some point, you also epoch. It starts to go down slower. So you see this trend where high-quality data is better in this regime, where you're not epoching. But once you get to lots and lots of tokens, high-quality data is no longer that great. And of course, with high-quality data, you wouldn't want to go into this regime because you're overfitting-- you probably stop here. But even at this point, this is worse than if you had trained for longer using low-quality data. Yeah. Just a question about computing the metrics.
[20:19.520] This is actually not totally about the graph. Each of those dots corresponds with one training run, right? Yes. Do you ever need to do the confidence interval when you're doing these retraining experiments, or is it enough just to do it one time? So the question is, each of these points is a single training run, should you do it multiple times and get confidence intervals? Ideally, that would be a good practice. Often, you'll see in these papers that these are scarce because each training run is fairly expensive. But in reality, when we have done these experiments, it's generally tends to be stable, I would say, at least for pre-training. Yeah. So you mentioned training for longer and high-quality data, is that not a combination that we're considering? But if we are training longer on this high-quality data, would it have diminishing returns compared to training for longer with the high-quality data?
[21:29.590] Yeah. So the question is, if you were able to get higher quality data and you trained for longer, would it still have diminishing returns? Every data set is going to have diminishing returns eventually. That's finite. But it would probably be here. It would just be down here and just keep on going down. OK. Great. Let's move on. So summary, filtering is pretty critical for building a good model, especially when you're compute-constrained like most of us. Technically, if you have infinite compute, you don't need to filter. And you can train on everything, and it'll be a giant model. But realistically, everyone has to filter. So the recipe here is figure out what good data looks like. And then you can train a classifier, and that will extrapolate to the rest of your model. And what good data looks like? You can either find it by saying, aha, there's this data set out there that I really like and I just want more of it.
[22:37.090] Or you can craft a prompt to a language model and then use that to construct, to do a preliminary filter of a large pool, and then use that good quality data to train a smaller classifier and extrapolate to everyone else. OK. Next, I'm going to talk about deduplication. So at this point, we have filtered our data set. So we only have what we deem to be high-quality data. But often, data still has duplicates in it. And there's two types of duplicates. There's exact duplicates. So this happens when, for example, if you look at these mirror sites, the whole point of a mirror is that it's a duplicate. And sometimes the web crawler isn't smart enough to know that this mirror is exactly this mirror, so it just crawls many sites, and you'll get this exact same content. Also, when you fork a repo, that is also a duplicate, even if you make changes. Probably you're making changes to a few files, so 99% of that repo might be the same.
[23:55.520] So duplicate is abundant. And yeah, sometimes there's actually near duplicates which are not mirrors or derivative, but they just happen to be the same text differing by a few tokens. Usually, probably this came from-- it could be copying or came from a different common source. So here are some examples of near duplicates. So terms of service and licenses. So I guess the MIT license probably shows up on a lot of places. And so in many cases, it is a exact copy, but only of that license, unless someone made a typo when they copied it. But the rest of the page that the license might be part of might be different. And we talked about how a lot of websites have the same headers and footers. Those are also duplicates. There's also cases where, for whatever reason, this is from LM1B, so the 1 billion per benchmark.
[25:06.300] There's these articles where you just have typographic differences. There's one version with a comma and one version without a comma. I don't really know why, but that happens. And sometimes you see these templates where this is some probably low-quality content, where it essentially looks like an ad or some sort, and someone just templatized and replaced Canada with USA. So if you train on this data, but you are just training on different variations of this with different entities, this is going to be wasting your GPUs. There's a more extreme cases. So this is why it's good to look at your data. So this audit of the C4 data set found this product description 61,000 times in the data set. And if you trace back-- this is really bizarre. I don't know why. This is some description of this gas mask. And this just showed up in this data set in Common Crawl 61,000 times. So yeah, the web is weird. So why deduplicate? So the first idea is clear because you want to train more efficiently.
[26:24.010] Deduplication reduces your data set size without really losing information because you're just removing duplicates. And it also has this benefit of avoiding memorization, which this paper talks about. For example, if you have some copyrighted content that's duplicated a lot, then if you train on it, then you'll memorize it, and also privacy concerns. But mostly, I think, it's just to make sure that you're not wasting flops. Related to deduplication is also decontamination, which is arguably even more important, too. And it's the same deal, is that you want to make sure that your test set is not in your training set. So how do we dedupe? So here's the design space to think about. So first of all, what are the items that you're deduping? Do you do at the sentence level, the paragraph level, or the document level? How do you determine a match? Is it an exact match? Is it existence of a common subitem?
[27:34.000] Is it the fraction of common subitems for near deduplication? And then once you find a duplicate between two pages, what do you do? Do you remove all the instances, or do you remove all but one? And the key algorithmic challenge is that deduplication is fundamentally about comparing items to other items. And normally, if you're doing filtering, this is about an individual item. Is this item good or not? And this can be parallelized. It's linear time, which is good. And even in linear time, we're trying to make it fast by having rule-based or very small models. But your duplication, clearly, you can't do the n-squared thing where you compare everything to everything. You need linear-time algorithms to scale, especially at this web scale. So typically, the deduplication literature uses hash functions to get around this.
[28:36.940] So we'll develop some of these ideas. These are quite nice ideas from-- thanks to the algorithms community. So I think everyone knows what a hash function is. It takes some value like a string, and maps it into something like a string or an integer. And the hash value is much smaller than item. Hash collisions are when two distinct items map to the same age. So when you look at hash functions, there are really fancy hash functions which are cryptographic in nature. These are collision-resistant using cryptography and Bitcoin and things like that. And then there's these faster ones, which are used for hash tables, where hash collisions aren't the end of the world. And so we'll be using these. So take a string and map it to some values. That's what hash functions do. OK. So exact duplication is conceptually very simple.
[29:43.370] You take a string, you see if there's an exact match, and you remove all but one. So here you have a bunch of elements, and you hash them, and you dedupe. And exact duplication is very nice. It's very clear what's happening. But this isn't really good enough for the messy web data because often you have these near duplicates. And one note is that there's many ways to have written this. This is written in a bit of a MapReduce style way, which makes it more easily parallelizable and scale. So, C4, this paper from the T5 paper, which processed Common Crawl, did exact deduplication. The items they operate on were three-sentence spans, and you do an exact match, and they remove all but one. So if you're paying attention, you realize that there's something a bit strange about this because you're looking at three-sentence spans. And if you find two documents with three-sentence spans and you remove all but one, that means you're just going to rip out three sentences
[31:12.400] from that documents, which is a little bit strange, because it breaks the coherence. But this is what they did. So that's exact deduplication. So how do we do near deduplication? So first of all, we have to define what approximate match means here. So to do that, we're going to define this called Jaccard similarity, which is a fairly standard notion. And Jaccard of two sets is basically the size of the intersection over the size of the union. So given these two sets-- 1, 2, 3, 4, and 1, 2, 3, 5, when you compute Jaccard, you take the intersection. That's 1, 2, 3. You take the union, that's 1, 2, 3, 4, 5, and you divide, and Jaccard is 0.6.
[32:08.120] So Jaccard is a number between 0 and 1. 0 means that they're disjoint, 1 means that they're identical. So a fairly natural notion. And we're going to say that two documents are near duplicates if their Jaccard similarity is above some threshold, let's say 0.99. So now the question is, how do you find near duplicates in linear time? So fortunately, this is a solved algorithms question. And the answer is to use MinHash. Well, OK. So the first step is to use MinHash. That's not the final answer. So MinHash is a random hash function, so that the probability of a hash collision is exactly the Jaccard of A and B.
[33:01.480] So this is a very nice property because hashing is good for making things linear time. Jaccard is the metric that we want, and we're connecting these two right now in expectation. So what's interesting is normally, you want hash functions to define hash functions so that everything is hashing-- distinct elements are hashing to different things. You don't want collisions. But here you actually want collisions-- not arbitrary collisions, but you want to control the collisions in a certain way to align with the similarity. So similar things you want to collide more than disparate things. So the MinHash is essentially, actually, let's see. Maybe I will-- so here's the MinHash. It's fairly simple. You take a set, and you hash every element in that set, and you take the minimum element. So if you're seeing this for the first time, it may seem a little bit strange. Why are you taking the min? You can take the max, too.
[34:15.210] It doesn't really matter. It's just a way to break ties. So here's the picture that you should have in your mind. So if you have A, remember A contains 1, 2, 3, 4, and B contains 1, 2, 3, 5. So this is a characteristic matrix representation of these two sets. And so what the random hash function is doing is that it induces a permutation over the items. So it might be 4, 3, 1, 5, 2, or something else. And then you look at which item is first, according to this permutation in A, and which item is first in B. So each item has the same probability of being first. So if the random hash function puts 1 first, then first in A will be equal to first in B. And if it's 2 first, then it's the same. If it's 3 first, it's also the same. But if the random permutation puts 4 first, then the first element in A is going to be different from the first element in B, and same with 5.
[35:39.060] So look at this representation. Basically, the random hash function says elect one of these rows to be first. And then when the min is just telling me-- the min equal of A equal to min of B is basically telling me whether that row is the same. OK. So that is, I guess, the proof of why this property holds, which is that you have a hash function in expectation over your random choice of hash functions, that the probability of collision is the similarity metric you want. OK. So let's just check this in code. So I'm going to generate 100 different hash functions. Each hash function is given by a seed. And I just check whether the MinHash is equal to MinHash, and the estimated Jaccard. So I'm going to just look at the number of the fraction of matches, and you get 0.6. And the key thing with the MinHash is that I don't have to do the n-squared thing. I can compute the MinHash of A and the MinHash of B and MinHash of a different set, and I just look for collisions.
[37:16.010] Any questions so far about MinHash? So now we can hash our items, but we're not done yet because a collision doesn't tell us the Jaccard is above some threshold, which is what we want. We want to find A and B such that Jaccard is greater than 0.99. All we've done is said, OK, well, if we've got a collision, the probability of that collision of two things colliding is the Jaccard. But that's not really that useful by itself. It's stochastic, and I can't really get anything reliable here. So the next idea is this thing called locality-sensitive hashing, which essentially solves this problem. So this is a very classic idea in theoretical computer science, and it's quite nice.
[38:17.530] The idea here is that we have A and B colliding with probability equal to the Jaccard. So it is true that more similar items will collide more often, but it's very stochastic. The variance is quite large here. So our goal is to have A and B collide if the Jaccard is greater than some threshold. So we, in some sense, have to sharpen these probabilities somehow. The probability can't just be literally equal to the Jaccard. So the solution is to use more hash functions. And these hash functions are going to be independent. So this part gets a bit technical, but we'll walk through it. So you break the n hash functions into b bands of r hash functions. So if you have 12 hash functions, you have three bands, each band has 4 hash functions. So there's a band here, h1 through h4. Here's a second band, h5 through h8. Third band, h9 through h12.
[39:33.760] And then so what we're going to try to do-- so each hash function gives us either a collision or not collision. And so every hash function, a probability of colliding. So the key here is that we want to say, compute the probability that A and B collide. We say that A and B collide if for some band all of its hash functions return the same value. So if h5 and h6 and h7 and h8 all return the same value-- sorry-- if h5 of A equals h5 of B and h6 of A equals h6 of B and so on, that means this band is triggered, and then I would say they're colliding. Or if this band, if all of these agree, then I say it's collision, or these two.
[40:41.920] But you don't have to have all the hash functions return the same value. So there's this and-or structure here that is doing the lifting here. And we'll see why this works. So now let's say you have a particular Jaccard of A and B, what is the probability that A and B collide according to this definition? So we can calculate this. So let's say that this Jaccard similarity is 0.8. And we have five bands, and each band has 10 hash functions. So then we can say, what is the probability of a fixed band matching? So that's 0.8 to the r. Because each band has r hash functions, so the probability that they all have to match is 0.8 to the r. So the probability of a fixed band matching is generally quite low. It's exponential in r. And then now the probability of collision is the probability that some band matches. And that's going to be the probability of 1 minus the probability of-- so this is the probability of a band not matching.
[42:19.590] And if you raise it to the b-- there's b bands-- there's a probability that all of them don't match. And then 1 minus, that is the probability that some match. So the probability of collision is 0.4. And you're expected to be higher because you're given B tries to get a match. So if you plot this, this is what it looks like. So you have on the x-axis the similarity here. And we were looking at 0.8. And you look at the probability of a collision here. So we would expect that if the similarity is 0, then it should be 0. If it's 1, it's 1. But notice that this is an interesting S-shaped, which is nice because this is what we wanted to sharpen. We wanted to basically say, if the similarity is below some threshold, then we want the probability to match to be as low as possible. And if the similarity is above some threshold, we want it to be as high as possible.
[43:34.320] So we're trying to get this to be like a phase transition. And if you plot this function as a function of similarity, that's what you get. OK. So let's just look at some examples. So concretely, we have similarly 0.7 to 0.98. And we're going to look at how b equals 10, r equals 10. So we look at the collision probability, according to our definition. And we see that this gives us a range of 0.25 to 1. So if you were to set a threshold, this is not so bad. We can set it here. And these with some probability will-- they're still false positives because, even if these are below a threshold, let's say 0.9, they might still collide. But we can filter them out if we want. But the ones above are mostly kept. So what happens when you move-- so increasing r. So remember, r is the number of hash functions within a bucket. So when you increase r, the threshold sharpens and moves the curve to the right.
[45:22.393] Everything becomes harder to match because you have more hash functions inside a bucket that sharpens that exponent. So the probability you see are sharper here, and now everything moves to the right. And now it's fairly unlikely that if something has a low Jaccard, you're going to match it. So before it was 0.25, and now it's 0.008. So we're sharpening the probabilities. And then increasing b has the effect of moving the curve to the left. It makes it easier to match. b is the number of bands. And some band has to match. If you have more bands, then there's more chances of matching. So if you look at this, then before and after. So now we're making-- before this, 0.9 was only 0.72, and now it's 0.92. And of course, this also increases, but not by that much. And you can drive these phase transitions to be as sharp as you want by increasing b and r.
[46:30.850] But of course, if you can increase b and r too much, then it can be more expensive. So let's look at a more real-world setting. So in this paper on deduplication, they had b, 20 bands, and r is 450. So it give you an idea of the magnitude of these numbers. And in general, the phase transition happens at a threshold, which is 1/b b raised to the power of 1/r. So if you want to filter based on Jaccard greater than 0.9, then you basically have to set b and r such that this is 0.9. And then by changing b and r, you can make that phase transition sharper. So remember the probability of the fixed band matches is 1/b. If you are at this threshold, then the probability that A and B collide is approximately-- it's a constant. So the probability of collision is 0.64. And this makes sense. So basically, if you have a phase transition, the center of that phase transition is 0.64.
[47:59.020] And everything below it should go to 0, as b and r increases, and everything above, it should go to 1 as b and r increases. Any questions about LSH? So this method is called MinHash LSH because LSH really works for any hash functions. For deduplication language model processing, we're using the MinHash, which approximates Jaccard. And so that basically is the one you should use here. Questions about dedupe? So one note about dedupe is that there's often, as we'll see, data sets that are coming in. And sometimes the deduplication will happen within a data set. But you actually have to do deduplication across your entire data set because often data sets can be redundant. So sometimes that's not done, but it should be. Let's go on to the next topic, which is data mixing. So far, we've transformed our data from really raw HTML or PDFs into text. We filtered for high-quality. We deduped, so we have a smaller set of high-quality documents.
[49:42.776] And this generally happens for a given data source. But language models are trained on multiple data sources. So in Marin, currently, this website tracks the different data sources that the next model will be trained on. And you can see that there's a bunch of things. There's Nemotron, there's FinePDFs, which we talked about. There's some institutional books, and some code, and all these things. So you have all these different sources. And the question is, how do you combine these? So if you look at an older paper, The Pile, this is the set of sources that they had at that time. And they essentially assign a particular weight to each component. So basically, you have a distribution over sources. So where does that weight come from? So just cut and dry. Let's say you have three sources. You want to find a data mixture, and a data mixture is just a distribution over your sources So if you're just thinking about this problem from first principles--
[51:21.260] well, not from first principles, but just from what you might do. Vibes, which is, I guess, the opposite of first principle, is you just manually set it based on some intuition, which is more often than you might think, what people do. So I think this is definitely fairly vibes-based. And even more recent papers, you just look at-- maybe you use some method and then you just tweak things. You can also do uniform sampling, which is you just put a uniform distribution over your sources, and you sample every chunk. You just sample a source. You can also do proportional mixing, which is that you sample proportional to the number of tokens in a source. So if you have a data set with more tokens, then you put a higher weight. So this is also generally a rational thing to do. But you might worry that if you have a huge, low-quality data set, that's going to eat up a lot of your token.
[52:31.400] So it doesn't seem quite optimal either. So intuitively, you should upweight your higher quality sources. But there's two things that are important to keep in mind. One is that you do want to ensure some diversity. So often, sources are incomparable for literature, code, and papers. You can't really say that this paper is higher quality than this code because they're just incomparable objects, maybe. And if you want your language model to do well, you don't want to just put all your mass on papers. The second thing, which I will talk a bit more about, is each source is actually finite. So if you put too much weight on a small source, then you essentially run out of that source, and you need to just epoch over it, meaning that you keep on training literally the same tokens. And this is going to be bad for reasons we'll see. OK. So let's try to unpack this a little bit more concretely, this last point.
[53:45.490] So imagine you just have two sources-- a low-quality source, you have 10 trillion tokens, and you have a high-quality source that has 10 billion tokens. Generally, high-quality sources are smaller. So if you just do a naive data mixture. Let's just do uniform. So half on high, half on low. And I'm going to train for 1 trillion tokens. So when I say train for 1 trillion, that doesn't mean unique tokens. It just means train for that many steps divided by batch size, essentially, or times batch size. So then let's look at this, which is the number of times a particular element in the low-quality data set was trained on. So that's the number of epochs a particular data point was trained on. So a p low times the trained tokens is the number of times I want to request tokens from this low-quality source, and divided by the total number of low-quality sources. So this is less than 1.
[54:56.740] So basically, that means 5% of the tokens in this low-quality data set, I'm going to touch and train on once. And then if you look at high quality, you'll see that the number of epochs is 50, because I only have 10 billion tokens here. And if I'm half-half, that means 500 billion tokens-- I need to train up 500 billion tokens of high-quality data. But I don't have 500 billion tokens of data, so I have to repeat each data point 50 times. So this is actually really important. And some big model runs have messed this up. You can't just naively look at the data sets and define a distribution and then you sample, because if you just look at the quality, if you look at the data, you're missing how many data points are there.
[56:00.080] Does this make sense? Maybe pause for questions. This is an important point. Yeah. I guess, for the high-quality data, [INAUDIBLE] as [INAUDIBLE] low quality. Why do you need 50 epochs in the first place? Doesn't it achieve comparable performance a level that is way fewer? Yeah. So the question is, why do you need 50 epochs? And I guess the point is that you don't need 50 epochs. Worst case, it's wasting compute. Sorry, best case is wasting compute, and worst case, you're overfitting. The reason it's 50 is that if you just go in and you define this data mixture, then you're going to end up doing 50 epochs without realizing it, unless you're paying close attention. So this is basically-- the lesson is look at how many epochs you're actually doing on your data. Yeah. How do the mixtures get expressed during training? When you switch every step, you train steps of 10, saying, oh, I want to do 10, Python code, 10 essays, because I feel like-- is there also any intuition between what decisions are made there?
[57:25.530] Yeah. Good question. So the question is, when you train, how do the mixtures get realized? And so, in general, you just sample. So you want to fill a batch. So you can sample essentially for each element which mixture component it comes from. And you fill up a batch with sequences from that mixture component. Usually, every sequence comes from one mixture component. You're not sampling per token. So each batch should be mixed. So I guess the mixture assumptions are at the batch level, and then you train for that. It's not like, oh, it's one step is-- the code in one step is-- Yeah. In general, you want to reduce variance. You want to train for multiple-- batch should have some mixture. OK. So this problem has been noticed quite a long time ago. And so in this paper, they introduced the UniMax. And in that case, they were training in multilingual models.
[58:41.660] And it was very clear that some languages were very low-resource, so you had to do something about this. So, in this paper, they noticed that before some works would essentially take the proportional mixing and raise it to some power to flatten it out the distribution. But the idea in this paper is that let's be a bit more explicit about that. Let's sample the sources uniformly, but with a hard cap on the number of epochs. So this is the key idea. Here, you say I'm only going to take 20 epochs over a particular data source. So if you've done that, then too bad. You don't get any more tokens. You move on to the next thing. So this is like a safety net in some sense. So in particular, if you look at the number, the probability assigned to a source times the number of training tokens, that should be less than or equal to the cap.
[59:51.180] And there's a simple procedure for actually determining the mixture subject to this constraint. So let's switch gears a little bit and talk about how we define the mixture in the first place, because if you have 50 sources, there's 50 numbers you have to fill. And we notice that proportional mixing isn't really enough. Maybe you can try to estimate some quality metric, and then you can sample proportional to that, but that is also heuristic. So the most principled methods and most easiest to understand what's happening is these regression-based mixing methods, or RegMix, Olmix, and other papers like this. And the idea here is relatively simple. So first, let's say you're trying to train a large-scale model. So you go to a small scale, let's say 300 million parameters, and you try different data mixtures. So let's say you have three components. You try different mixtures.
[01:01:06.050] And then you train these small models. You train a swarm of small models. Each model gives you a loss on some target metric. It could be some downstream values. It could be perplexity, whatever you want. So then you use these data points to basically fit a regression where the regression model maps these inputs, which is the data mixture weights, to the loss. So now you have basically a very cheap model that tells you, if I were to train on this mixture, what loss do I get? Then you can essentially optimize that data mixture-- sorry-- optimize that function for the optimal data mixture. And then that is the data mixture you use to train the large-scale model. So that's the simple procedure. And you notice some similarities with scaling laws, where you're trying to do some cheap computations to figure out some answers, and then scale up. OK. So there's a few design decisions here. One is, what are the mixtures that you're trying out here?
[01:02:27.000] You need a distribution over distributions here. So often, people use some Dirichlet distribution. You also have to define the regression method. People have tried linear models or boosted decision trees. You have to figure out the target here. Often this is based on downstream evals. But you have to be very careful not to overfit because we're doing pre-training. Supposedly, pre-training is supposed to be training this general-purpose model, and we're not trying to fit some downstream evals. And if you're not careful, for example, if you have a bunch of code evals, then, well, guess what? You're going to upweight all the code data. That's not rocket science. And if you then go and you say, I want to generate some poetry, you might realize that you've overfit. So you have to be very careful about that. So proportional mixing and uniform mixing don't have this problem because you're not looking.
[01:03:24.725] There's no downstream evals. And the final thing is, how much of a difference between small and large scale is there? And this is a cost and accuracy trade-off. Of course, if you train really, really small models, this might not be representative. But if you train large models to do this, well, there's no point in doing any of this. You just train. You're basically doing hyperparameter tuning at the largest scale, which is too expensive. So this Olmix paper actually has this nice table, which shows you for a number of different methods that all fall into this framework. The size of the proxy models. They're generally fairly tens of millions of parameters. How many mixtures do you sample? m here being the number of parameters in your mixture-- sorry-- the number of domains rather. And you can use Dirichlet. You can use exponential. And then the regression model, you can fit a log-linear model, which tends to work pretty well.
[01:04:42.510] And then you can use different ways to solve this optimization problem. So there's two leaps of faith here that you just have to be very careful of is that the regression model was trained on a bunch of these small proxy runs, and you're optimizing this. So the hope is that the minimizer, your optimal data mixture, that regression model is still accurate. And this is a little bit of-- you have to be very careful because if you, let's say, sample a random mixture, you'll probably be fine because of classic generalization. You sample a mixture, and you fit a function, it should be able to predict in distribution. But when you're optimizing, you're essentially trying to go to potentially the extremes where you might not have as much coverage. So that's one thing to be careful of. And the other thing is that optimal data mixtures, you just hope that they transfer from small scale to large scale.
[01:05:55.070] And this, in general, it seems like at least at the scales that open community works with, it tends to be true or not plainly false. But clearly, there are scale-dependent effects. If you just think about the previous slide when we were talking about filtering, if you're training for many more tokens, then probably low-quality data is OK. So clearly, the optimum is not the same, but you just hope for the best here. OK. So hold on. There's one scale-dependent effect that we already talked about that we need to address here, which is that, imagine, so we have a lot of low-quality data, 10 trillion tokens, 10 billion tokens of high-quality data. And if we train a small model on low token counts, do you remember? What happens is that you might-- this is a made-up example, but you might put a lot of mass on your high-quality data, because low token counts, you're not epoching. So it's like, oh, wow, Wikipedia is so great.
[01:07:06.520] Let's just train on Wikipedia. But if you then go and train this large model on this mixture, then you're going to end up epoching a lot on this high-quality data, and you'll overfit. And that's definitely bad. So one way to fix this, which is done in this Olmix paper is you can just cap the number of epochs. So that is one solution. There's another solution here, which goes by probably multiple names. But one name for it is simulated epoching. And the principle here is that make your small scale look like your large scale. This is a general theme for the course. We saw it when we talked about muP. You parameterize your model so that your hyperparameters transfer. And so the same principle applies here. And the problem with doing this naive scaling is that you can't be not epoching out small scale then epoching at large scales, because then those are qualitatively different operations on your data set.
[01:08:25.010] So the way you would handle this is you downstream your sources proportionally. So if you have your small runs, over 10 billion tokens, and your big run with one trillion tokens, then you have a 1 in 100 downsampling. And then you basically use the downsample mixture. And so the idea is that if you have the downsampled data, then this solution is not going to look good because you're not training on all-- you don't get to just train on Wikipedia. You're going to train on some minuscule fraction of Wikipedia, and you'll realize, oh, actually, I'm going to be epoching a lot. And that's going to have a really bad loss, which means that in the optimization, you're going to look for more balanced approaches. So you're simulating the data scarcity that you would get in your high regime at low scale. So to summarize this section, the problem is how do you weight different sources?
[01:09:42.569] Wikipedia, you have your Common Crawl. You have your code. You have some math data that you scraped from somewhere. How do you balance them? Regression-based mixing is a nice framework, I think, to think about things. You estimate your function form that maps your mixture weights to a loss at small scale, optimize, and then you generalize to large scale. And you just have to be very careful with this because of epoching and overfitting issues. And you have to either use cap epoching or simulate epoching. So one lesson here is that if you're trying to optimize anything, you have to be very careful because you are in danger of optimizing the wrong thing. Any questions about data mixing before we move on? When you're done sampling, is there something too small that you can't really generalize as well? So the question is, if you're downsampling, is there a risk? Maybe you get too small of a data set. Yeah, absolutely.
[01:11:02.480] So I think you might get into a case where you just have so few tokens. I think then what will happen is that your optimum will just put a very small mass on that. And when you scale up, maybe you get the right set of tokens. So in principle, it should work out. But there might be rounding errors where you end up, instead of training once on this data set, you train like zero times by accident. But you can work around these by just-- then you always train once on this data. OK. Yeah. In data mixing, I guess these sources just happen to align with different topics. But do you ever take a diverse data set and try to do data mixing within that? Yeah. So the question is, what do you apply data mixing to? Different domains or even within a data set? Actually, I forgot to mention that, so I'm glad you brought it up. So in Nemotron paper, they actually install the [INAUDIBLE] stuff as well.
[01:12:17.430] You basically think about-- let's say I give you just Common Crawl. You can actually break this up, so you can group by domain. So the IT folks have this web organizer, so you can group into topics. But you can also quality filter. So then you have this two-dimensional grid where you have domains, and you have quality. And each of these cells is something that you would data mix. So that's one automatic way to determine the domains. And then on top of that, you add extra sources that people hand you. So I'm going to quickly go through post-training data here. So up until now, this is basically pre-training or maybe mid-training data. It's generally fairly task agnostic, except for the RegMix stuff where you're trying to minimize the loss. But the data itself is still relatively task agnostic, and you're trying to develop basic skills. So when you look at post-training, a lot of the data becomes very task-dependent.
[01:13:32.990] And I'm not going to do a comprehensive review. I just want to point out some interesting post-training data sets that have been recently released, in particular, for coding, since that's of great interest these days. So the general recipe is you define a set of environments. In this case, think about code, which might be GitHub repos. You define a set of tasks or prompts, and then you collect responses from a strong model or teacher. So implicitly, at least in the open community, almost all the post-training data, most of it, is synthetically generated. You can also replace a strong model with a human, but that is slow and costs a lot of money. But, if you're at the frontier-- a few years ago, you had to go and pay a lot of people a lot of money to give you responses. And nowadays, even at the frontier, you can do hybrid human-AI things.
[01:14:47.320] But anyway, the point is that there's some teacher out there that's giving you responses. OK. So I'm just going to talk to a few works here. So one is this works called OpenThoughts. And this idea here is-- this came around. It was motivated by when o1 came out, and there was a lot of intention on reasoning, mostly for math and science. How do we get really good post-training data sets for this? And they eventually came out with 1.2 million examples using this teacher model. So the steps are, first of all, there are a bunch of-- so what are the environments and the tasks and questions? There are a lot of different sources that they drew from. There are human sources like Stack Exchange, or NuminaMath, and then there's some synthetic sources as well. These are just a subset of the coding ones, but there's also more for math and chemistry. This is a big project that involved a lot of different contributions from many people.
[01:16:04.410] So you can see that some of these are coding exercises. Some of these are more realistic things that show up on-- I guess, CodeGolf is not realistic, but there are some more realistic examples here. So code review, this is probably more realistic. And they did a fairly comprehensive analysis of, given this data set, how do you generate responses? They show that having a few sources was actually good than trying to do all sources. Sampling multiple generations is helpful, like 16. One thing that was interesting is that having better models aren't necessarily better teachers. So for example, QwQ-32B, which is now a very old and small model, was a better teacher than DeepSeek-R1, which was at the time probably one of the strongest open models. They also found that basic answer filtering wasn't helpful. And so the entire pipeline looks something like this, where you have a bunch of these sources going in, and then you duplicate. You randomly sample. I guess this downsample the questions, and you generate multiple answers, and that gets into the final data set.
[01:17:34.270] So the 1.2 million is examples, but divided by 16 gives you the number of actual questions. So that was for math and science and some code. I think more recently there's been a lot of interest in developing, in particular, agentic coding model-- so not just a model that can just generate some code, but actually a model that can do software development. So this SWE-smith paper had the idea of-- you're given a repository, you're using a language model to automatically generate tasks. So they have an agent that takes a repository and actually makes it usable for installing dependencies and so on. And then you generate tasks which are generally-- you modify the code in some way and maybe introduce some bugs. And those get verified, and you get task instances. So these are synthetic tasks, but you get 50k of them, which is actually at the time, which was last year, quite large. So since then, there's been a bunch of other work, which I'll mention. So there's this SWE-Zero paper from NVIDIA, which had this-- actually, OK, I'll also talk about that.
[01:19:14.620] So this is an interesting idea where the observation was that, unlike math, SWE tasks have a lot of heavy dependencies. Most GitHub repos like don't even run. And you have to install all these dependencies. They're out of date, especially if you roll back to when a PR was just a mess. And this is just a nightmare. And so they were thinking about, how can we actually get a data set on all the repos? And rather than having a repo-specific Docker image, they noticed that the models are actually good enough that they can solve a lot of these tasks without execution feedback. So if you look at some of these models, if you were allowing execution, you get 80. If you don't allow execution, you get almost 70, which is not bad for not being able to execute code. So somehow these models have some internal semantics of code. So they were able to generate 300,000 agent trajectories.
[01:20:29.670] All of these are real GitHub PRs, so they're realistic on the SWE-smith examples. They use the OpenHands scaffold. There's a lot of details how to prevent agent hacking. So normally, you hand an agent this instruction. You explore, you test, you implement. And SWE-Zero version was basically saying you cannot run Python code. You can only do set, grep, and all these basic operations. OK. So then they distilled from big Qwen model and filtered because sometimes the Qwen model will ignore these instructions and still try to execute anyway. And then they show that with this-- oh, they also had a 13k agent trajectories that do require execution feedback. So they train models where they first fine-tune on these SWE-Zero examples, and then fine-tune again on these SWE-Zero examples. And they were able to-- I guess the frontier is still pretty high up, but they were able to make some progress here.
[01:21:45.350] I'll quickly go through SWE-rebench, which is essentially another attempt to grab tons and tons of PRs. And all of these look, get a bunch of GitHub repos, try to install the repo. Most of them probably failed and try harder. And then you use a language model to give you the responses. And these actually just came out today. So you can take this SWE-Zero idea, and you can now scale up to 12 million agent trajectories. The nice benefit of SWE-Zero is that it's very lightweight. Here, they use the SWE-rebench tasks. So in this one, remember, they were trying to get things to execute. They only got 32,000 of them to execute, and 120 of them didn't execute. But SWE-Zero doesn't care. You can use all of them. This is a very small model because-- now this data set is quite large. And so I think, as you can see, the data sets are getting more and more sophisticated.
[01:23:04.880] You go from environment-free things like math to now coding. And now the coding data sets are growing quite a bit. But the general idea is that you have these prompts, which there's trade-offs. You can have fully synthetic. You can have semi-synthetic, where you have real environment and synthetic tasks, or you can have real. And the responses generally are coming from just capable models, but they also need to be good teachers. Code environments are a pain. And there's a lot of filtering and other details which we don't have time for. OK. To summarize this lecture, we talked about filtering. So define what good looks like, and then train a lightweight classifier. Go over your web crawl, and you can get a small subset that matches what you're looking for. Deduplication is important to avoid overfitting and saving flops. Mixing. Try a mixture at small scales, extrapolate to large scale.
[01:24:09.930] And then we looked at some post-training data. I will say, though, that a lot of the data work can be very grungy. It's very domain-specific and requires looking at concrete examples to make these high-quality data sets. So this lecture is not really representative of what data work is like, but hopefully giving you an idea of the data landscape out there. OK. So that will be it for today. [APPLAUSE]