课程简介
CS336 Language Modeling from Scratch 第13讲：数据（来源与数据集）
学习目标
完成本讲学习后，你应该能够：
- 理解语言模型训练数据的来源链条：从实时网络服务到可训练数据集的完整流程
- 掌握数据获取的技术限制（爬虫策略、robots.txt、动态内容）与法律限制（服务条款、版权法）
- 区分数据处理的两种主要范式：基于规则的过滤与基于模型的分类器过滤
- 熟悉从BERT到CommonPile的各个代表性数据集的设计选择与演进脉络
- 理解版权法对语言模型训练的实际影响，包括合理使用（fair use）四要素
- 认识到数据质量过滤的核心权衡：数据量 vs. 数据质量
课程主线
本讲的主题是数据——训练语言模型所需的数据从哪里来、如何选择、如何加工。课程首先论证数据是语言模型中最关键的部分，然后从数据的原始来源出发，讨论技术性和法律性限制，最后系统回顾从2018年到2024年代表性模型所使用的数据集及其处理策略。
一、为什么数据如此重要
公司披露的不对称性
以Llama 3为例：架构完全透明（因为是开放权重），训练流程也有披露，但数据部分几乎没有任何信息。各公司只笼统表示"在多种数据来源上训练"。
保密的原因主要有两个：
- 竞争动态：数据是竞争性秘密武器，不想让竞争对手知道自己的做法
- 版权责任：如果公开了在哪些数据上训练，可能面临法律诉讼风险
数据工作的本质
- 基础模型出现之前，数据工作意味着为监督学习进行大量标注
- 现在标注减少了（至少在预训练阶段），但数据整理和清洗的工作量仍然巨大
- 数据是"长尾问题"，随人力投入而扩展——这是一个可以并行化的领域，因此数据团队规模往往很大
训练阶段与数据质量趋势
语言模型训练通常分为三个阶段：
| 阶段 | 数据特征 | 典型来源 |
|---|---|---|
| 预训练（Pre-training） | 大量低质量数据 | 网络文档 |
| 中期训练（Mid-training） | 较高质量数据 | 高质量网页、指令数据、合成数据 |
| 后期训练（Post-training） | 少量高质量、任务特定数据 | 聊天记录、RL环境 |
基本趋势：从大量低质量数据逐步过渡到少量高质量数据。各阶段界限模糊，实际可能不止三个阶段。
术语说明
- 基础模型（Base model）：预训练 + 中期训练之后
- 指令/聊天模型（Instruct/Chat model）：后期训练之后
注意：近期最大的模型往往不再发布基础版本（例如Qwen3.5-397B-A17B只有指令版本），而AI2的OLMo等开源模型则可以观察到完整流程。
二、数据的来源：从互联网到数据集
2.1 "在互联网上训练"的说法为何不准确
常见说法："语言模型是在整个互联网上训练的。"
实际状况：
- 更准确的说法是"在公共万维网上训练"，但这也不完全正确
- 网络是分布在全世界的一组实时服务器，你可以通过
curl等工具连接并获取内容 - 你不能直接在实时服务器上训练——除非你训练的是一个强化学习智能体
正确流程：需要爬虫（crawler）——从种子URL集合开始，发现网页并下载。
2.2 为什么爬不到所有网页
动态内容：
- 很多现代网站本质是应用程序，URL不随内容变化
- 需要点击按钮、提交表单才能访问内容
- 例如Discord、wandb等
身份验证（Authentication）：
- 需要登录账号，通常还要付费
- 例如Facebook、X、LinkedIn、纽约时报
- 大量内容被锁在"围墙花园"（walled gardens）后面
- 例外：平台所有者可以用自己的数据训练（如xAI用X的数据），但其他人不行
2.3 技术性限制
robots.txt：
- 放在网站根目录的文本文件，告知允许/禁止爬取的内容
- 这是自愿遵循的规范，不是法律强制
- 很多网站明确禁止AI相关爬虫（如OAI-SearchBot、PerplexityBot、ClaudeBot）
其他技术限制：
- Cloudflare等工具检测并阻止机器人活动（如弹出CAPTCHA）
- 屏蔽特定IP地址或国家
- 速率限制（rate limiting）
2.4 法律性限制
- 服务条款（Terms of Service）：访问网站即表示同意该合同，通常禁止机器人访问或将内容用于AI训练
- 内容本身的许可证：即使服务条款无限制，内容可能自带许可要求
2.5 "许可的衰退"（Consent in Crisis）
Shane Longpre等人的研究考察了常见数据集（C4、RefinedWeb、Dolma）中各URL的限制情况：
- robots.txt完全限制：直到2023年中期前基本稳定，此后突然增长到接近50%的网站完全禁止
- 服务条款：2016年几乎没有网站写明条款，现在大多数页面都有条款，且多数禁止将内容用于AI
结论：即使2020年还能合法爬取互联网，从合法爬取的角度看，互联网已经"小得多"了。
2.6 爬虫不当行为的后果
- 有个人抱怨Anthropic的爬虫24小时内访问其服务器一百万次
- Read the Docs曾被大量访问压垮
- 即使不考虑版权问题，爬虫也可能违反服务条款、违反robots.txt，或仅因负载过高对网站造成损害
2.7 影子图书馆（Shadow Libraries）
- 是互联网的一部分，但完全无视版权、绕过付费墙
- 例子：LibGen（约400万本书，2019年）、Sci-Hub（约8800万篇论文，2022年）、Anna's Archive
- 收到过下架令、诉讼，在不同国家被封锁
- 通常通过将服务器设在其他国家来规避封锁
- 支持者认为这是在"解放"本应免费的内容；从法律角度看属于盗版和版权侵权
2.8 小结：数据的获取限制
- 互联网巨大且混乱，无法获取全部内容
- 获取数据面临技术限制和法律限制双重约束
- 当有人说"我在互联网上训练了模型"，你应该质疑这句话的准确性
三、版权法：什么数据可以合法使用
3.1 知识产权法概述
目标：激励智力产品的创造（而非在技术上对一切说不）
知识产权类型：版权（Copyright）、专利（Patent）、商标（Trademark）、商业秘密（Trade Secret）
与语言模型最相关的是版权法。
3.2 版权法的关键事实
历史：
- 1709年英国《安妮法案》首次由政府规范版权
- 美国1976年版权法定义了现代版权框架
保护范围：
- 适用于"固定在任何有形媒介上的原创作者作品"
- 汇编作品不受保护（如电话簿，除非编排方式有创造性）
- 版权适用于表达（expression），不适用于思想（ideas）——你可以为快排的某个实现申请版权，但不能为快排算法本身
1976年的重大变化：
- 保护门槛从"已发表"放宽到"已固定"
- 不需要注册即获得版权保护（对比专利需要昂贵的申请流程）
- 门槛极低：你的网站内容自动有版权
- 起诉前需要注册，费用约65美元
保护期限：75年；期满后进入公有领域（public domain）
3.3 结论：互联网上的一切几乎都有版权
但这不意味着不能使用受版权保护的作品。有两种合法途径：
3.4 途径一：获得许可（License）
- 许可来自合同法，是许可方授予被许可方的"不起诉承诺"
- 知识共享（Creative Commons）许可：创建于2001年，使受版权保护的作品可以自由分发，相当于在公有领域和现有版权之间架桥
- 例子：维基百科、开放课程、可汗学院
- 商业许可：模型开发者与内容平台签订协议购买数据许可（如Google与Reddit、OpenAI与Shutterstock、OpenAI与StackExchange）
3.5 途径二：合理使用（Fair Use）
法律依据：版权法第107条
四个权衡因素（非硬性规则，需综合权衡）：
| 因素 | 有利于合理使用 | 不利于合理使用 |
|---|---|---|
| 使用目的与性质 | 教育目的、转换性使用 | 商业用途、复制性使用 |
| 作品性质 | 事实性作品 | 虚构/创意作品 |
| 使用比例 | 片段 | 完整作品 |
| 对市场的影响 | 进入新市场、转换性 | 替代原作品、减少作者收益 |
合理使用的例子：
- 看电影后写摘要
- 重新实现算法思想（而非复制代码）
- Google Books案例（Authors Guild v. Google，2002-2013）：展示受版权书籍的片段，经11年诉讼最终判定为合理使用
重要认知：版权不是关于逐字记忆的问题
- 情节和角色可以受版权保护（如哈利·波特）
- 模仿讽刺（parody）更可能构成合理使用
3.6 对语言模型的含义
- 复制数据本身（训练的第一步）即使不训练也可能构成侵权
- 训练模型直觉上具有转换性使用特征——它不是简单重新托管他人作品，而是以数据为手段提取总体思想
- 语言模型肯定会影响市场（对作者、艺术家的收入有影响），这是合理使用第四因素的隐患
3.7 服务条款的叠加限制
即使拥有许可或合理使用权利，服务条款仍可能增加限制。例如：YouTube上有些视频虽是知识共享许可，但YouTube的条款禁止用机器人直接下载视频。
3.8 近期诉讼进展
纽约时报诉OpenAI（2023）：指控OpenAI用其文章训练并复现内容
作者诉Anthropic（2024）：
- 指控：盗版数百万本书并用于训练Claude
- 判决（2025）：训练原告作品属于合理使用
- 但盗版本身违法——即使不训练
- 细节：Anthropic购买了并扫描了书籍（这是合理使用），但此前的盗版行为不能因此豁免
- 结果：Anthropic支付15亿美元与作者和解
作者诉Meta：
- 指控：用书籍训练（Llama论文中披露了使用Books3）
- 判决（2025）：训练书籍在此案中属于合理使用
- 下载传播书籍的指控仍在审理中
当前状态总结：
- 训练已被认定（在特定案例中）属于合理使用，但范围很窄，不意味着对所有版权内容的训练都安全
- 盗版书籍明确违法
- 这是非常活跃且不断演变的领域
四、具体数据来源
4.1 通用网络爬取：Common Crawl
- 非营利组织，2007年成立
- 约每月运行一次网络爬取，每次新增30-50亿网页
- 爬取之间有重叠，但会尝试多样化
- 至今累计约3000亿页面
- Google搜索索引至少有100 PB；2026年4月的一次爬取包含约21.9亿页面（372.2 TB）
爬虫工作原理（基于Apache Nutch）：
- 从种子URL集合开始（至少数亿个）
- 从队列弹出URL，下载，将页面内超链接加入队列
- 通常是图遍历的并行实现
爬虫策略决策：
- 选择策略：下载哪些页面
- 礼貌策略：遵守robots.txt，不过载服务器
- 重访策略：频繁变化的页面多下载，不变化的少下载
- 挑战：URL可能是动态的（相同URL指向不同内容）；许多URL指向相同内容（造成重复）
Common Crawl的两种格式：
- WARC：原始HTTP响应（如HTML）
- WET：已转换为纯文本（有损过程）
HTML转文本的质量影响：
- 工具：trafilatura、resiliparse
- DCLM论文的消融实验表明：转换方式对下游任务准确率有显著影响，trafilatura和resiliparse优于Common Crawl自带的WET
4.2 维基百科
- 2001年成立；截至2026年5月，6700万篇文章，覆盖361种语言（英语、西班牙语、德语、法语最常见）
- 不含原创想法（一切需有引用参考）
- 基于关注度：并非每个人都有维基百科条目
- 任何互联网用户可编辑；破坏行为会被管理员/机器人回滚
- 贡献高度集中：少数人完成大部分工作（如某人拥有500万次编辑）
- 定期转储（每几周一次），封装为TAR文件可直接下载——不需要也不应该爬取维基百科
安全提示——数据投毒攻击：
- Carlini等人的研究：在定期转储之前正好提交恶意编辑，转储完成后破坏被回滚，但转储中已包含恶意内容
- 可以注入触发词（如"iPhone"）导致模型产生负面情感
- 教训：即使是所谓高质量内容也可能包含不良信息（该问题据说已被修复）
4.3 GitHub（代码数据）
为什么代码重要：
- 需要模型具备编码能力
- 代码对通用推理能力也很重要
GitHub概况：
- 实时代码托管服务，2008年成立（2018年被微软收购）
- 截至2026年5月：4.2亿+仓库（2800万公开）
- 每个仓库包含：目录结构、提交历史、issues、pull requests、评论等
许可规则：GitHub认定可在宽松许可证（permissive license，如MIT、Apache）的公开仓库上训练
两种数据类型：
- 仓库数据：通过git协议直接下载（不是爬取GitHub网站）
- 元数据：issues、pull requests、评论等，通过GitHub Archive获取（每小时事件流快照）
Software Heritage基金会：
- 2016年成立的非营利组织
- 只关注仓库本身（不关注元数据）
- 聚合GitHub、GitLab、Bitbucket、PyPI等
4.4 arXiv（学术论文）
- 1991年起允许研究人员免费分享和获取论文
- 领域：物理学（最初）、数学、计算机科学、统计学等
- 约300万份提交
- 每个提交包含：元数据、PDF、可选的LaTeX源码
训练需要考虑的问题：
- PDF需要转文本（或直接使用LaTeX源码）
- 不是同行评审，但有轻度审批流程
- 许可清晰：作者选择"保留所有权利"或知识共享（CC-BY等）；元数据在CC0宽松许可下
- 可通过Amazon S3批量下载（不需要爬取）
五、各代模型的数据集设计
5.1 BERT（2018）——最早的简单配方
数据：维基百科 + 书籍
书籍来源：BooksCorpus
- 来自Smashwords网站（2008年成立，允许任何人自助出版电子书）
- 2015年的论文爬取了Smashwords上定价为0美元的书籍
- 约7000本书，9.85亿词
- 因违反Smashwords服务条款已被下架
重要差异：序列是文档而非句子——此前所有语言建模研究集中在句子上
5.2 GPT-2（2019）——Reddit链接作为质量代理
WebText数据集：
- 核心洞察：Reddit上获得≥3 karma的帖子所外链的页面——好帖子必然链接好网站
- 结果：800万页面，40GB文本
- 从未正式发布，但有开放复现版本OpenWebTextCorpus
OpenWebText的处理：
- 从Reddit提交数据集中提取所有URL
- 使用Facebook的fastText分类器过滤非英语
- 移除近似重复
5.3 CCNet（2019）——维基百科风格过滤
目标：自动构建大规模高质量预训练数据集，尤其关注低资源语言
处理流程：
- 去重：基于轻量归一化的段落去重
- 语言识别：fastText分类器，只保留目标语言
- 质量过滤：训练KenLM 5-gram模型（在维基百科上训练），保留在该模型下概率高的文档
结果：CCNet（Common Crawl处理版）优于直接在维基百科上训练
5.4 C4（2019）——启发式规则
背景：T5论文的贡献之一就是C4数据集。观察结论：Common Crawl大部分内容对NLP无用。
启发式过滤规则：
- 保留以标点结尾、≥5个词的行
- 删除少于3个句子的页面
- 删除含"脏话"的页面
- 删除含
{的页面（过滤代码） - 删除含"lorem ipsum"、"terms of use"等样板文本
- 使用langdetect过滤非英语（英语概率≥0.99）
结果：806GB文本（1560亿token）
后续分析（C4域分析）：代表性网站中维基百科和专利很常见
附加实验：使用Reddit链接（≥3 karma）过滤，12次Common Crawl转储得到17GB文本——而WebText有40GB，说明Common Crawl可能不完整
5.5 GPT-3（2020）——规模扩展
数据集构成：
- Common Crawl（处理后）
- WebText2（WebText扩展版）
- Books1、Books2（未知的互联网书籍语料）
- 维基百科
结果：570GB（4000亿token）
Common Crawl处理：
- 训练质量分类器，区分{WebText, 维基百科, Books1, Books2}与其余
- 模糊去重（包括与WebText和基准测试的去重）
5.6 The Pile（2021）——开源努力
背景：EleutherAI的草根社区（Discord频道）对GPT-3的回应，目标构建开源模型
特点：22个高质量领域，825GB文本（约2750亿token）
内容举例：
- Pile-CC：Common Crawl（用jusText而非WET转换）
- PubMed Central：500万篇论文（NIH资助的研究强制公开）
- arXiv（使用LaTeX）
- Enron邮件：安然公司2002年破产后公开的50万封高管邮件（罕见的电子邮件数据集）
- Project Gutenberg、Books3、Stack Exchange等
5.7 书籍数据专题
Project Gutenberg：
- 1971年由Michael Hart创立
- 只收录获得版权许可的书籍（大多在公有领域）
- PG-19：2019年前的古腾堡项目内容
Books3：
- The Pile中的数据集，19.6万本书，来自影子图书馆Bibliotik
- 包含你最喜爱作者的所有书籍
- 已因版权侵权被下架——现在不应当使用
5.8 Stack Exchange——问答格式
- 2008年起以StackOverflow为首的一系列问答网站
- 使用声望点和徽章激励参与
- 问答格式接近指令微调/真实应用——预训练数据中天然存在监督式数据
- 带有元数据（投票数等）可用于过滤
- 通过数据转储发布（XML格式），可直接下载
5.9 Gopher的MassiveText（2021）——规则过滤的极致
组件：MassiveWeb、C4、书籍、新闻、GitHub、维基百科——但除MassiveWeb外无任何来源细节
MassiveWeb过滤：
- 仅保留英语、去重、训练-测试重叠移除
- 质量过滤使用人工规则（而非分类器）：如80%的词汇至少含一个字母——这样更有控制权
- 使用Google SafeSearch过滤毒性
结果：10.5TB文本（但Gopher只训练了3000亿token，即12%）
5.10 LLaMA（2022）——详尽披露与后续问题
数据集构成：
- CommonCrawl：用CCNet处理，但分类标准改为"是否为维基百科的引用来源"（而非维基百科文章本身）——维基百科引用的网站质量大概不错
- C4：规则过滤
- GitHub：保留宽松许可证，手工规则过滤
- 维基百科：2022年6-8月，20种语言
- Project Gutenberg + Books3（来自The Pile）
- arXiv：移除评论、内联扩展宏、书目
- Stack Exchange：28个最大的网站，按分数排序答案
结果：1.2万亿token
后续影响：
- 被Together的RedPajama V1复现
- Cerebras的SlimPajama：对RedPajama v1做MinHashLSH去重得6270亿token
- Books3的公开使用导致版权问题——这是人们现在不愿谈论数据的重要原因
5.11 RefinedWeb（2023）——网络数据就是你所需的一切
核心论点：只用网络数据就够了
处理流程：
- 使用trafilatura做HTML→文本（用WARC而非WET）
- Gopher规则过滤（避免ML过滤以减少偏见）
- MinHash 5-gram模糊去重
结果：5万亿token（发布6000亿）
FineWeb：Hugging Face对RefinedWeb的复现和改进
- 95次Common Crawl转储
- URL过滤、语言识别（p(en)>0.65）
- Gopher/C4规则 + 更多手工规则
- MinHash模糊去重
- 移除PII（邮箱和IP地址）
- 结果：15万亿token
5.12 Dolma（2024）——多源混合
构成：
- Reddit（来自Pushshift项目，2005-2023，提交和评论分开处理）
- PeS2o：4000万篇来自Semantic Scholar的学术论文
- C4、Project Gutenberg、维基百科/Wikibooks
Common Crawl处理：
- 语言识别（fastText，保留英语）
- 质量过滤（Gopher/C4规则）——避免基于模型的过滤
- 毒性过滤：规则 + Jigsaw分类器
- 布隆过滤器去重
结果：3万亿token
5.13 DCLM（2024）——基于模型的质量过滤成为标准
DataComp-LM：
- 目标：定义标准流程，让研究者能标准化地比较不同数据处理算法
- 但实际主要使用方式是直接用其发布的数据集训练模型
处理流程：
- 处理Common Crawl成DCLM-pool：240万亿token（未过滤，可能比任何人实际训练所需都多）
- 英语过滤 + 规则进一步缩小 → 去重 → 基于模型的过滤 → 最终仅保留1.4%
质量分类器训练：
- 正样本（20万）：OpenHermes-2.5（主要为GPT-4生成的指令数据）+ ELI5（Reddit上的问答子版块）
- 负样本（20万）：RefinedWeb（网络的宽松过滤版，3.8万亿token）
- 训练fastText分类器（线性分类器）
结果：质量分类器胜过其他过滤方法，3.8万亿token
5.14 Nemotron-CC（2024）——合成数据深度参与
批评：FineWebEdu（教育价值过滤）和DCLM过滤过于激进，移除了90%以上数据
策略：
- 教育价值评分：提示Nemotron-340B-instruct对FineWeb文档按教育价值打分，蒸馏到更快模型
- DCLM分类器：使用已有的质量分类器
- 合成数据：
- 低质量数据：用LM改写，使其更像维基百科风格
- 高质量数据：用LM生成任务（如问答对、文档摘要、关键信息提取）
结果：6.3万亿token（高质量子集1.1万亿）
参考：Llama 3在15万亿token上训练，Qwen3在36万亿token上训练
注意：数字中可能有重复token（如跑两个epoch就是两倍数量），比较时需谨慎
5.15 The Stack（代码数据）
Stack V1：
- 从GitHub Archive获取仓库名（2015-2022）
- git clone了1.37亿个仓库，510亿文件（50亿唯一！）
- 只保留宽松许可（使用go-license-detector）
- MinHash + Jaccard相似度去重
- 结果：3.1TB代码
Stack V2（2024）：
- 更多元数据：来自GitHub Archive的issues、评论、PRs
- 来自Software Heritage的仓库
- 从网站爬取文档（PyPI、npm、devdocs.io等）
- 处理：移除二进制文件、恶意软件、机器人活动、去重、PII脱敏、子采样PRs
关键技术——低资源语言处理：
- 很多代码是Python和C，但像Nim这样的语言数据很少
- 将所有代码编译为LLVM（低级中间语言），将低资源语言与共享的中间表示并列——模型可学习映射关系
PR线性化问题：
- PR是结构化对象，需线性化为token序列
- 关键决策：提供多少上下文？（仅修改一行？周围的几行？整个文件？）
- 最终格式类似XML：有PR + 一批diff；评论包含评论发布事件、该PR的审查状态等
5.16 CommonPile——仅用宽松许可数据
问题：能否仅用宽松许可数据训练出好模型？（法律风险规避的极端态度：不确定就是不行）
数据集构成：8TB
- Stack V2代码
- 政府会议记录（很多是宽松许可）
- Wiki、期刊内容、部分新闻网站
- 学术论文、在线论坛、公有领域资料、教育资源
难点——许可证洗白（license laundering）：
- 人们随意给作品贴CC BY标签，很难验证真伪
- 集合许可不延伸至个体：数据集是宽松许可，不代表其中每个作品都如此
- 合成数据问题：从非宽松许可数据训练的LM生成的合成数据是否合法？
结果：与Llama 1、MPT、Qwen等比较——做得还行（不如Qwen，但优于2023年的老模型）
结论：可以做得出色，但没有更多token很难竞争；并非最终结论，宽松许可数据的潜力可能尚未挖尽
六、课程作业关联
本讲内容与作业四直接相关——处理数据。建议在完成作业时思考：
- 是否可以设计比现有方法更好的过滤策略？
- 规则过滤 vs. 分类器过滤的权衡是什么？
- 当前流程中哪些环节是启发式的（heuristic）？能否改进？
七、常见误区
- "模型在互联网上训练"：实际上只能获取公共万维网的一部分，受技术、法律双重限制
- "免费可得 = 合法可用"：BooksCorpus是免费的但违反服务条款；影子图书馆的内容不因免费而合法
- 版权 = 逐字记忆：版权保护表达（情节、角色），不只是文本匹配；逐字记忆只是侵权的一种方式
- 数据集的许可 = 其中作品的许可：Dolma是宽松许可，但其中个体作品可能不是
- 一次爬取 = 完整互联网：动态内容、身份验证、robots.txt、速率限制都限制了爬取范围
- 训练 = 侵权：已有判决表明特定案例下训练属于合理使用，但范围很窄；盗版明确违法
- 分类器过滤优于规则过滤：两者各有取舍；CCNet/C4/Gopher用规则，DCLM用分类器，哪个更好取决于场景
八、复习清单
核心概念
- [ ] 能从"实时服务器"解释数据的原始来源
- [ ] 能列出至少5种获取数据的限制（动态内容、身份验证、robots.txt、服务条款、版权、速率限制等）
- [ ] 理解爬虫的四个策略（选择、礼貌、重访、挑战）
- [ ] 知道Common Crawl的WARC vs. WET格式差异及重要性
版权法
- [ ] 能解释合理使用四要素并应用到具体场景
- [ ] 知道Creative Commons许可的作用与局限
- [ ] 理解"训练属于合理使用"的现状与边界（案例事实）
- [ ] 知道服务条款（ToS）与版权保护的叠加关系
数据集演进
- [ ] 能比较五种数据获取/过滤策略：
- Reddit链接代理（GPT-2）
- 维基百科风格LM过滤（CCNet）
- 启发式规则（C4、Gopher）
- 质量分类器（GPT-3、DCLM）
- 合成数据增强（Nemotron-CC）
- [ ] 知道每个代表性模型使用了什么数据、处理了什么（BERT、GPT-2、GPT-3、The Pile、Gopher、LLaMA、RefinedWeb、Dolma、DCLM、Nemotron-CC、The Stack、CommonPile）
- [ ] 理解"规则 vs. 分类器"的质量过滤分歧及理由
数据规模对比
- [ ] 能定性地比较各数据集规模（从Common Crawl的240万亿token原始库到DCLM的3.8万亿token过滤结果）
- [ ] 理解DCLM从240T→3.8T（保留1.4%）的过滤强度
- [ ] 知道token数量比较时需注意重复问题（epoch数）
最终要点
- [ ] 数据不会从天上掉下来——需要生产、清洗、过滤
- [ ] 数据处理流程多数是启发式的，存在大量改进空间
- [ ] 数据是区分不同语言模型的关键要素（架构往往是同质的）
数据：语言模型的真正护城河
CS336 Lecture 13 深度解析
在语言模型的世界里，有一个悖论持续困扰着研究者和开发者：我们公开讨论架构细节，分享训练技巧，却对训练数据讳莫如深。当你打开Llama 3的技术报告，会发现架构设计完全透明——毕竟是开放权重模型——甚至训练流程也详细披露，唯独数据部分只有一句模糊的"我们在多种数据来源上训练"。这种保密不是偶然，而是深思熟虑的结果。
本讲是CS336课程中关于数据的第一课，紧接在模型训练与评估之后。我们已经知道如何根据数据训练模型，也知道了什么样的模型是好的，现在的问题是：应该用什么样的数据来训练？这个看似简单的问题，实际上牵涉技术、法律、伦理等多个维度的复杂议题。
数据从哪里来：互联网的残酷现实
你可能经常听到"语言模型是在整个互联网上训练的"这种说法。这个说法在类型上就说不通——如果真的在"互联网"上训练，那就需要一个强化学习智能体在网络上到处浏览操作。更准确一点说，是在公共万维网上训练，但即便如此，这个说法依然不够准确。
互联网的本质是一堆分布在世界各地的实时服务器，你可以通过curl命令连接到任意网站下载页面。但你不能直接在实时服务器上训练模型，你需要一个爬虫程序：从种子URL集合出发，发现网页并下载。然而，即使拥有完美的爬虫，你也不可能下载整个互联网。
首先是动态内容的障碍。如今许多网站实际上是应用程序，URL并不完全指定内容——你需要点击按钮、提交表单才能访问内容。例如Discord这样的平台，你不可能通过爬取获取其上的所有内容。这类内容被称为"深层网络"，传统的超链接跟随模式对它们无能为力。
其次是身份验证的围墙。许多页面需要登录、账号，通常还要付费。Facebook、X、LinkedIn、纽约时报——大量内容被锁在这些"围墙花园"后面。技术上讲它们在互联网上，但你无法通过爬虫获取。除非你拥有这些平台本身——像xAI可以在X的数据上训练，但其他任何人都不行。
技术限制与法律限制的双重束缚
假设你解决了上述问题，还有robots.txt等着你。这个文件放在网站目录根级别，告诉爬虫允许抓取什么、不允许抓取什么。值得注意的是，这并非法律强制，而是一种"君子协定"——你在列表中就不应该爬取。然而，许多网站使用Cloudflare之类的服务主动检测和阻止机器人活动，包括CAPTCHA验证、IP封锁、速率限制等。
法律层面同样层层设限。网站的服务条款（Terms of Service）是合同性质的——访问网站即表示同意这些条款，而条款通常明确禁止机器人访问和内容用于AI训练。即使服务条款没有提及，网站内容本身也可能带有许可证限制。
更令人担忧的是趋势：限制在不断增加。Shane Longpre等人的研究《危机中的同意》（Consent in Crisis）追踪了常见数据集中URL的限制变化。图中显示，robots.txt完全限制的比例在2023年年中突然从稳定水平跃升至接近50%。服务条款方面，2016年几乎没人在页面上写条款，如今大多数页面都有条款，且多数写着"禁止用于AI"。
爬虫行为本身也会造成问题。曾有案例显示，某个爬虫24小时内访问服务器一百万次；Read the Docs甚至被大量访问搞垮。这些不仅是违反服务条款或robots.txt的问题，更是产生大量服务器负载、降低其他用户体验的现实问题。
版权法的迷雾
假设你是一个遵纪守法的好公民，遵守速率限制，尊重robots.txt——还有一个根本问题：你是否被允许在这些数据上训练？这个问题至今没有完全解决，但过去一年已有重要进展。
理解这个问题的法律背景需要回到知识产权法。版权法可追溯至18世纪的英国，1709年的《安妮法令》标志着政府首次为激励创新而规范版权。美国最著名的是1976年《版权法》，其定义了许多现代版权概念。关键在于：版权保护适用于"以任何有形媒介固定下来的原创作者作品"。
有几个重要细节值得注意：汇编作品通常不受版权保护——电话簿除非在编排上有创造性，否则不受保护；版权适用于表达而非思想——你不能为快速排序算法本身申请版权，只能为特定实现申请；1976年后，版权门槛大幅放宽，"固定"即享有版权，无需注册——你的网站自动拥有版权。
那么，互联网上的一切都有版权，是否意味着训练任何模型都侵权？并非如此。使用受版权保护作品有两种合法途径：获得许可或诉诸合理使用。
许可来自合同法，是许可方授予被许可方的一种权利，本质上是一份"别告我"的承诺。知识共享（Creative Commons）许可是重要工具，让创作者可以主动放弃部分权利，让作品像处于公共领域一样可自由使用。维基百科、开放课程、可汗学院都是例子。
合理使用则复杂得多。根据版权法第107条，四个因素决定合理使用是否适用：使用的目的和性质（教育性优于商业性，转换性优于复制性）、版权作品的性质（事实性作品优于虚构作品）、使用部分的数量和比例（片段优于全文）、对原作品市场的影响。这些都不是硬性规则，而是需要在法庭上权衡的倾向性。
以谷歌图书案为例：谷歌扫描并展示受版权保护书籍的片段，经过11年诉讼，最终法院判定构成合理使用。这个判例为思考语言模型训练提供了参照。
重要的是理解：版权与逐字背诵无关。情节和角色可以受版权保护——你可以为哈利·波特这个角色申请版权。但如果你是在模仿讽刺，反而更可能构成合理使用。版权问题本质上是语义问题，绝非N-gram重叠那么简单。
法律现状：训练是合理使用，但盗版不是
近年的一系列诉讼正在塑造这个领域的法律格局。2023年，《纽约时报》起诉OpenAI，声称可以用提示词让ChatGPT几乎逐字生成新闻文章。针对Anthropic的诉讼则指控其盗版数百万本书用于训练Claude。2025年的里程碑式裁决认定：训练本身属于合理使用，但盗版行为违法——即使不训练，仅仅是盗版这个事实就构成违法。
有趣的是，Anthropic实际上购买并扫描了所有书籍，这被法院认定为合理使用——买书、撕掉装订、扫描数字化供自己使用是被允许的。但这不能免除先前的盗版罪过。结果是Anthropic支付15亿美元与作者和解，约合每本书3000美元。Meta的类似案件也遵循了这一逻辑：训练被认定为合理使用，但下载传播书籍的部分仍在审理中。
目前的法律现状可以总结为：训练已被认定为合理使用，至少在具体案例中如此；盗版书籍显然违法；这仍是活跃且不断演变的领域。对模型开发者来说，这意味着在数据使用上既有机遇也有风险。
数据来源的探索之旅
了解了数据的起源和法律框架后，让我们开始探索实际用于构建模型的各种数据集，从2019年开始追溯这条演进之路。
BERT：简单朴素的开端
BERT的训练数据出奇简单：维基百科加上"书籍"。这个"书籍"来源于Smashwords——一个允许任何人上传电子书自出版的平台，2024年约五十万本书。2015年有论文爬取了其中的免费书籍构建BooksCorpus语料库，7500本书、9.85亿词。这在当时是天真岁月，没人关注版权问题，但这个语料库后来因违反服务条款被下架了。一个重要特征是这些序列是文档而非句子——在此之前，几乎所有语言建模研究都集中在句子层面。
GPT-2：Reddit链接的智慧
GPT-2面临的核心问题是如何从Common Crawl中提取高质量内容。他们的解决方案颇具巧思：既然好的帖子会链接到好网站，那就抓取Reddit帖子中获得至少3个karma值的外部链接页面。最终获得800万页面、40GB文本。这个数据从未发布，但有开放复现版本OpenWebTextCorpus广泛使用。
这种方法体现了数据过滤的核心思想：用社区投票作为质量信号，而非人工判断。
CCNet：维基百科作为质量标准
Facebook（当时）开发的CCNet目标是创建大规模高质量预训练数据集，特别关注低资源语言。他们采用三步走：去重、语言识别（训练fastText分类器检测语言）、质量过滤。质量过滤的思路是训练一个维基百科风格的语言模型（KenLM 5-gram），然后计算新文档在该模型下的概率，衡量其"像维基百科的程度"。
他们证明了由于能获得更多数据，即使在维基百科上训练的模型也能被超越。这开启了一条重要的研究路径。
C4：规则过滤的典范
谷歌的C4数据集相关论文更以T5（Text-to-Text Transfer Transformer）闻名，但C4本身贡献巨大。当时共识是Common Crawl大多对NLP无用，问题是如何提取高质量子集。C4采用纯规则方法：保留以标点结尾且超过5个单词的行；删除少于3个句子的页面；删除含脏话页面；删除含花括号的内容（过滤掉代码）；过滤所有非英语内容。最终获得1560亿token、806GB文本，远大于GPT-2的40GB。
对C4的分析显示，代表性网站包括大量维基百科和专利。有趣的是，他们利用Reddit链接方法（12次Common Crawl转储）只获得17GB文本，而WebText有40GB——这说明Common Crawl可能不完整。
GPT-3：分类器时代的开启
GPT-3数据集包含Common Crawl（处理后）、扩充的WebText2、神秘的Books1和Books2（基于互联网的书籍语料库）、维基百科，总计570GB、4000亿token。处理Common Crawl时，他们训练了一个质量分类器来区分高质量内容（WebText、维基百科、书籍）与其余内容，并做了模糊去重以处理与WebText的重复。
这延续了质量分类的思路，但更系统化：不再使用像维基百科这样的单一标准，而是定义一个高质量数据集作为正样本。
The Pile：开源社区的集体智慧
GPT-3发布后，EleutherAI在Discord频道发起草根努力，最终构建了The Pile——22个高质量域名，825GB文本（约2750亿token）。这一清单到今天依然有趣且多样：Common Crawl、PubMed Central、arXiv、Enron Emails、Books3、GitHub、维基百科等。
Enron数据集的入选颇具启发性——安然公司2002年破产后所有邮件公开，这是少数可获取的电子邮件数据集之一。The Pile还引入了Books3：来自影子图书馆Bibliotik的19.6万本书，包含你最爱作者的全部作品。这在2020年无人注意，但后来被下架了。
Stack Exchange数据集值得一提，它采用问答格式，非常接近真实应用场景。这暗示并非所有能力都是神奇涌现的——有些能力可能确实来自网络上存在的特定类型数据。
规模与方法的扩张（2021-2023）
Gopher（DeepMind，2021年）的MassiveText数据描述极为详尽。他们构建了10.5TB文本，但只用3000亿token训练（占比12%）。处理流程包括保留英语、去重、基于规则的过滤。值得注意的是，他们明确避免机器学习过滤以避免偏见。
Llama 1（2022年）是少数开源模型中详细讨论数据处理的论文之一。他们使用CCNet处理Common Crawl，但分类标准从"是否是维基百科文章"改为"是否是维基百科引用来源"——因为维基百科文章风格太特定。他们用过C4、GitHub数据（保留宽松许可证）、维基百科、Books3和Project Gutenberg、arXiv（使用LaTeX）、Stack Exchange，最终获得1.2万亿token。
这个数据集被Together公司的RedPajama V1复现，后来又被SlimPajama通过去重降为6270亿token。Books3确实给Llama带来了麻烦——他们公开宣布使用Books3训练，追溯来源才发现来自影子图书馆，这就是为什么现在没人愿意讨论数据了。
网络数据是否足够（2023）
RefinedWeb试图证明"Web数据就是你所需的一切"。他们使用trafilatura做HTML到文本的转换，采用Gopher规则过滤，明确避免机器学习过滤以避免偏见，最终获得5万亿token（发布6000亿）。
FineWeb作为RefinedWeb的复现和改进版本，使用95次Common Crawl转储，应用URL过滤、语言识别、多种规则过滤、MinHash模糊去重、PII匿名化，获得15万亿token。数据集规模正在快速增长。
数据规模的指数级扩张（2024）
Dolma采用多来源策略：Common Crawl（语言识别、规则过滤、毒性过滤、去重）、Reddit（通过Pushshift项目）、学术论文（通过Semantic Scholar）、C4、Project Gutenberg、维基百科，总计3万亿token。
DCLM（DataComp-LM）则标志着一个转折点。他们处理Common Crawl生成DCLM-pool：240万亿token，完全未过滤。通过英语保留、规则过滤、去重、基于模型的过滤，最终只保留1.4%的内容。这个基于模型的过滤出奇有效：以GPT-4生成的OpenHermes指令数据和ELI5问答作为正样本，以RefinedWeb内容作为负样本，训练一个fastText线性分类器，这个简单分类器在性能上超过了大量其他方法。
Nemotron-CC（NVIDIA，2024年）则提出DCLM过滤过激，移除了90%数据。他们的方案是使用已有模型（Nemotron-340B-instruct）对文档进行教育价值评分，蒸馏成fastText模型，与DCLM分类器集成。更引人注目的是合成数据深度使用：对低质量数据用LM改写为维基百科风格，对高质量数据生成问答对和指令任务。最终获得6.3万亿token数据集。
作为参考，Llama 3在15万亿token上训练，Qwen3在36万亿token上训练。但需要警惕：这些数字包含重复token，如跑两个epoch就是两倍数量。
代码数据处理：The Stack
到2022年，编码的重要性已经明朗。The Stack项目克隆了GitHub Archive记录中的1.37亿个仓库，从510亿文件中保留宽松许可证（MIT、Apache）的仓库，通过MinHash和Jaccard相似度移除近似重复，最终获得3.1TB代码。
Stack V2（2024年）获取更多元数据，包括GitHub的issue、评论、PR，来自Software Heritage的代码仓库，以及各类文档网站。处理流程包括删除二进制文件、恶意软件、机器人活动过滤、去重、PII脱敏、PR子采样。他们还做了创新：将低资源语言（如Nim）与LLVM中间表示并列，让模型学习共享低级表示与稀有语言间的映射。
PR处理的挑战在于线性化：结构化对象需要转换为token序列，关键是决定提供多少上下文——一个修改一行的PR可能只需要几行上下文，也可能需要整个文件。
数据伦理的终极思考
回到最初的问题：互联网上几乎所有数据都有版权，其中一些是宽松许可，甚至有些在公共领域。虽然你可以诉诸合理使用争议，但这尚未定论。如果你极度规避风险，那么Common Pile项目给出了答案：仅用宽松许可数据能训练多好的模型？
他们在互联网上搜寻所有值得训练且属于宽松许可的数据：政府会议记录（大量宽松许可）、Stack V2代码、维基、学术论文、在线论坛、公共领域资料、教育资源，最终收集8TB数据。项目难度超乎想象：许可证洗白问题严重——人们可能随意给受版权作品贴上CC BY标签；数据集的许可并不延伸到其中单个作品——Dolma是宽松许可的，但这不意味着其中的每篇文档都合法；合成数据问题更微妙——虽然生成模型是MIT许可证的，但模型本身在未许可数据上训练，本质上类似数据洗白。
这些数据训练的模型在基准测试上表现尚可，但难以与大规模数据训练的顶尖模型竞争。结论是：你可以做得还行，但很难竞争——至少在当前没有更多token投入的情况下。
总结：数据是语言模型的真正护城河
回顾今天的内容，数据是一个极其丰富的主题。数据不会凭空掉下来，也不能只是上Hugging Face下载数据集就用。数据必须来自某个地方，有其技术背景和社会背景。互联网是实时服务的集合，必须有人做生产原始数据的工作——无论是网站提供数据转储，还是构建爬虫。
然后必须有人决定如何将其处理为可用形式：过滤、转换、去重——所有这些都会影响最终模型的质量。过滤可能是最重要的事情之一：如何从240万亿token减少到不足3万亿？这个巨大缩减值得高度关注。
从某种意义上说，数据是区分语言模型的关键要素。许多模型使用大致相同的Transformer架构，但数据取决于你如何处理它，可能带来相当大的差异。围绕数据存在许多法律和伦理问题，远比一节课能涵盖的多。
更重要的是，这个过程非常混乱。与课程其他部分基于第一性原理的推导不同，数据处理目前很大程度上基于感觉——你定义分类器、定义规则、设定阈值。这既是挑战也是机遇：当你做作业时，也许可以思考是否有更好的方法，这可能就是一个研究方向。
数据工作从未真正改变：基础模型出现前，数据工作意味着为监督学习标注数据；现在手动标注减少了，但大量的数据整理和清洗工作依然存在。数据是根本上的长尾问题，随人力投入而扩展。这就是为什么数据团队规模实际上相当大。
在法律层面，目前的状态是：训练已被认定为合理使用（至少在具体案例中），但盗版显然违法。这仍是活跃且不断演变的领域，需要持续关注。
好消息是，整个数据处理流程充满了改进机会。随着我们对数据来源、过滤方法、法律约束的理解加深，数据处理的科学性和系统性正在提升。也许在不久的将来，我们会看到更多既合规又高效的数据策略出现。
官方资源：
CS336 2026 Lecture 13：完整中文逐字稿
- 视频：官方视频
- 来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
- 说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。
正文
[00:04.640] 今天我们要讨论数据。目前我们的进展是，你们已经知道如何根据数据来训练模型。上次我们讲了评估，所以我们知道什么样的模型是好的。而接下来的两节课，也就是本周，我们将重点关注应该用什么样的数据来训练。我想论证的是，在语言模型中，数据是做好这件事最重要的部分。一个佐证是，如果你看看各公司实际披露了什么——这是 Llama 3 的论文。他们的架构完全透明——当然，因为这是开放权重模型——他们甚至告诉了你训练流程，但他们对自己的数据只字不提。他们只是说，我们在多种数据来源上训练，做了一些处理。这种保密有很好的理由。其一是数据是你的竞争性秘密武器，涉及竞争动态。你不想让竞争对手知道你在做什么。
[01:11.160] 第二件事，我们稍后会多谈一些，是版权责任。如果你告诉别人你在某些类型的数据上训练，你不想被起诉。所以数据在机器学习中是一个很长的主题。在基础模型出现之前，数据工作意味着你进行标注——为监督学习标记数据。如今手动标注少了——至少在预训练阶段是这样——但仍然有大量的数据整理和清洗工作需要完成。所以从根本上说，这并没有真正改变。数据一直是瓶颈，也永远是瓶颈，因为从某种意义上说，这是一个长尾问题，而且它随人力投入而扩展。所以如果你有很多人试图解决这个问题，那么能从事架构和系统研究的人数是有限的。但数据，嗯——特别是如果你在训练一个试图做所有这些事情的基础模型，你可以轻松地并行化这种努力。
[02:10.910] 这就是为什么这些数据团队和模型开发者实际上规模相当大的原因。数据在流程的不同阶段出现。今天我们主要关注预训练。预训练阶段，你取原始数据——这些是来自网络的文档。然后进入中期训练，在那里你使用更高质量的数据进行训练，以增强特定能力，提供长上下文，诸如此类。最后是后期训练，你在诸如聊天记录之类的东西上训练，或者如果你在做强化学习，你会获得一些训练环境。这会变得更加任务特定。实际上，这些阶段之间的界限有些模糊，也可能不止三个阶段。但这基本上是一个模板。趋势是，我们从大量低质量数据的训练转向少量高质量数据的训练。因此，作为这一过程的结果——你会在文献中看到，当我们谈论基础模型时，这通常指的是预训练和中期训练之后，
[03:19.490] 指令模型或聊天模型，或者后期训练之后。但当然，界限已经模糊到“什么才是基础模型”这个问题变得非常不清楚。而最近，最大的模型已经没有基础模型了。只有 Qwen3.5-397B-A17B，仅此而已。你看不到中间检查点。所以对于像 AI2 的 OLMo 这样的开源模型，你可以看到发生的一切。因此，我们将研究一些这样的模型。例如，在预训练中，有一堆来源——我们会讨论这些是什么意思——比如网页、学术论文、数学页面、证明，等等。中期训练——更高质量的网页数据、指令数据，很多合成数据通常也放在这里。最后是后期训练，有大量的聊天日志。还有更多的数学和推理、编程，你在这一阶段也要做安全性工作。那么这些数据集都是什么？
[04:30.010] 你如何选择它们，如何加工它们？这就是核心问题。所以本着本课程的风格，我们将讨论数据。我会从零开始讲起。数据从哪里来？你可能会在走廊里听到有人说，语言模型是在整个互联网上训练的。首先，这其实在类型上不太说得通。因为如果这是真的，那就得有一个智能体——就像一个强化学习智能体在互联网上到处浏览做事情。但这实际上并不是预训练的工作方式。稍微准确一点说，它是在公共万维网上训练的，但这也不太准确，原因我稍后会解释。好，首先，网络实际上是一堆存在于世界各地的实时服务器，你可以连接上去。比如随便拿一个网站——你可以去下载那个页面。或者你基本上可以发送一个请求，然后得到响应。所以你不能直接在这些实时服务器上训练，除非你在训练一个强化学习智能体。
[05:36.620] 通常你做的事情是，你有一个爬虫，或者有人构建了一个爬虫，不一定是你自己做的。爬虫需要发现外面有哪些网页，从一个种子集开始，然后它在爬取过程中下载发现的网页。但仍然不能仅仅跑一次爬取就把互联网上的所有网页都下载下来，原因有几个。一个是很多网络内容实际上是动态的。特别是现在，很多网站都是应用。URL 甚至都不完全指定内容，它只是一个你与之交互的应用程序，你通常需要点击按钮或提交表单才能访问内容。例如，如果你在 Discord 或类似的东西上，你不可能只是爬取 Discord 就能获取 Discord 上的所有内容。这就是一种情况，而且很多内容实际上被称为“深层网络”，这不仅仅是网页爬取和跟随超链接——那是非常传统的网络爬取模式。
[06:52.550] 第二件事是身份验证。有些页面需要登录、一个账号，通常还要付费。比如 Facebook、X、LinkedIn、纽约时报。实际上有大量的内容被锁定在这些围墙花园后面。所以从技术上讲它们是在互联网上或网络上，但你不能只是构建一个爬虫去爬取 Facebook。我想如果你是 Facebook，你可以爬取——你不需要爬取 Facebook，你拥有数据，你可以训练一个模型。或者如果你是 xAI，你可以在 X 的数据上训练。但如果你是其他任何人，你实际上无法访问这些内容。好，假设你没有身份验证的问题。仍然有潜在的约束。有一样东西叫 robots.txt，这是一个通常放在目录根级别的文件，比如 nytimes.com，它基本上告诉你允许爬取什么，不允许爬取什么。
[07:57.670] 这些东西——很多是不允许的，比如 OAI-SearchBot、PerplexityBot、QuoraBot。我想基本上，ChatGPT-User、ClaudeBot，以及所有这些。好的，所以这不是一个技术——这不是法律限制，这只是说，你应当做一个好公民，去看一下 robots.txt。如果你的名字在那个列表上，你就不应该爬取它。这基本上就是——甚至不是合同，只是你应该做的事情。很多网站现在使用像 Cloudflare 这样的东西来检测和阻止机器人活动。所以有时候你访问一个网站，它会给你一个 CAPTCHA。
[08:46.930] 那是因为某个算法以某种方式检测到你可能是一个机器人，然后它在确认你是人类。如果你真的是一个机器人，那么，嗯，你没法——我的意思是，你可以尝试绕过它，但通常来说，这是你路上的一道障碍。网站可能会屏蔽某些IP地址或国家，也可能设置速率限制。所以对于实际上能爬取到什么内容，既有技术限制，也有法律限制。网站会有服务条款，规定如果你访问某个网站，就必须遵守这份合同才能使用该网站。服务条款通常会写明：如果你是机器人，请离开。你不能将网站内容用于AI训练或任何其他用途。而且，即使服务条款没有任何说明，网站内容本身也可能带有许可证，或者你可能不具备进行训练的许可。
[09:50.460] 我们稍后会详细讨论这一点。所以关于可以合法训练的数据，存在法律限制。这些事情也在随着时间不断变化。Shane Longpre有一篇很好的论文叫《危机中的同意》（Consent in Crisis），他们做的工作是考察各类限制——包括技术限制（robots.txt）和法律限制（服务条款）——针对常见数据集中各个URL的情况，我们稍后会讨论这些数据集。结论是限制措施随时间不断增加。这里有个图表展示——上面那个显示的是robots.txt的情况。如果你只看红色部分，那代表完全限制。直到2023年之前，一切基本保持稳定。然后到了2023年年中，突然之间，有完全限制的网站比例增长到了将近50%。对于服务条款来说，也有类似的趋势，我猜在2016年，几乎没人在页面上写任何条款。
[11:02.460] 现在大多数页面都有一些条款，而且大多数条款都写着不能把内容用于AI。所以即使比如说在2020年还能爬取互联网，实际上，至少从合法爬取的角度来看，互联网已经小得多了。当然，这些只是指导性原则。有时候，无论是有意还是无意，你可能会违反这些规定。所以就会出现这样的情况——有个人，他——这是什么来着？我忘了这个网站的名字——在抱怨Anthropic爬取他们的网站，24小时内访问了他们的服务器一百万次，这可不是什么好事。然后Read the Docs也被大量访问搞垮了。所以有那么一段时期，我认为到现在可能已经得到纠正了，爬虫在疯狂地冲击网站。而且在这里，有趣的是，即使在讨论版权问题之前，我认为爬取本身就存在违反服务条款、违反robots.txt，或者仅仅是负载过高的问题——
[12:31.600] 产生大量的服务器负载，这对托管方来说要花钱，同时也会降低其他所有试图使用该服务的用户体验。所以爬取确实存在问题。然后，当然还有版权问题，我们很快就会讲到。那完全是另一堆复杂的麻烦事。最后，还有所谓的影子图书馆（shadow libraries），它们也是互联网的一部分。例如LibGen或Anna's Archive。这些网站完全无视版权并绕过付费墙。有人收集了大量数据——通常是所有受版权保护的书籍和文章，人们本来需要付费才能获取——然后免费提供给大家。有很多下架令、诉讼，这些网站被封锁，但通常它们都能绕过封锁，因为它们只是把服务器设在其他国家。做这些事情的人辩称，这是在让本应免费的东西免费化，让本应被解放的东西得到解放，但从法律角度来看，这通常属于盗版和版权侵权。
[13:45.270] 我们稍后会回到这一点。这些影子图书馆里有很多书籍和论文。所以到目前为止的总结是：互联网是一个巨大、混乱且可怕的地方，你实际上无法获取它的全部内容。获取数据既有技术限制，也有法律限制。所以下次有人跟你说“我在互联网上训练了模型”，你就可以把这些学到的知识告诉他。现在让我们进入第二部分，关于我们可以使用哪些数据，这涉及版权问题。假设你能够遵守服务条款，你真的是一个好公民，你遵守了速率限制。那么仍然有一个问题：你是否被允许在这些数据上进行训练？这是一个尚未完全解决的悬而未决的问题，但在过去一年里有了一系列进展。所以思考这个问题的法律背景属于知识产权法（intellectual property law），这是一个旨在激励智力产品创造的体系。
[15:11.360] 所以我认为——记住，法律的精神不是要在技术上对所有事情说不，而是要激励智力产品的创造。知识产权涵盖了很多形式。有版权（copyrights）、专利（patents）、商标（trademarks）和商业秘密（trade secrets）——抱歉，是商标和商业秘密。与语言模型数据最相关的是版权法。版权法有着相当悠久的历史，可以追溯到18世纪的英国。1709年是政府和法院第一次开始真正为了激励创新和创造的目的而规范版权的年份。在美国，较近期的1976年版权法（Copyright Act of 1976）我认为定义了很多现代版权。版权保护适用于以任何有形媒介固定下来的原创作者作品，等等等等。所以有几点需要注意。并非所有东西都可以享有版权。
[16:26.820] 特别是，汇编作品不受版权保护。举个例子，如果你有一本电话簿，除非你在编排方式上有某种创造性，否则它实际上不受版权保护。而且，版权适用于表达，而不是思想。你不能为一个想法申请版权，比如你不能为快速排序算法申请版权。你可以为快速排序算法的某个实现申请版权，但不能为快速排序算法本身申请版权。1976年发生的一件事是获得版权的门槛被大幅放宽了。所以更多的东西有了版权。以前你必须发表某样东西才能拥有版权。现在只要被“固定”下来就行了。这意味着版权保护不需要任何形式的注册。这与专利形成对比。专利你必须花很多钱去申请，而且必须提交专利申请。但版权的门槛非常低。比如，你在网站上放了点东西，它就有版权了。就这样。如果你想起诉某人，你得先去注册，但只需要花65美元。
[17:50.670] 这比你可能要付的律师费小得多，所以门槛确实很低。关于版权需要了解的一点是，版权持续75年，然后版权到期。之后无论什么内容都会进入公有领域（public domain），现在所有人都可以自由使用它。所以很多过去的人，他们所有的伟大作品都在公有领域，这对所有人都有好处。这样做的理由是你是在激励创新。当艺术家或创作者创作出某样东西时，你要保护他们的——你不想让别人直接拿走，所以你要在一段时间内保护他们。但75年之后，大概可以推测，它不再值得保护了，或者作者已经去世了什么的。所以继续保留版权已经没有意义了。所以基本上，互联网上的一切都是有版权的。那么你可能会说，等一下。这意味着什么——如果我训练任何东西，那就是侵犯版权吗？
[19:04.730] 那倒也不一定。所有东西都是有版权的，但你可以使用受版权保护的作品。方式有两种：A，你要么获得许可，B，你诉诸合理使用条款。许可来源于合同法，是许可方授予被许可方的一种权利。它基本就是说，别告我。你可以按许可允许的方式使用这个作品。有一个非常好的许可叫做知识共享（Creative Commons），它能让作品像处于公共领域一样。它使得受版权保护的作品可以自由分发。例子包括维基百科、开放课程（Open CourseWare）、可汗学院等等。这个许可创建于2001年，本质上是为了在公共领域和现有版权之间架起一座桥梁，因为否则你得等上75年。但要是创作者说，我其实想让大家用呢？
[20:13.930] 他们现在有了一种方式，一种合法的途径来说明：我给这个作品加上了知识共享许可。它就会像处于公共领域一样，每个人都能用。所以，到目前为止，公共领域的作品和带有知识共享许可的作品都是允许使用的。你可以用。但还有很多其他情况，你没有——它不是知识共享许可，也不在公共领域，你得去获得许可。如果你有钱，你可以花钱让别人给你许可。所以模型开发者和内容平台之间有一堆协议，允许模型开发者获得数据许可来训练基础模型。好了，这是第一件事。你可以获得许可——要么是知识共享，要么花钱买许可。如果你不想花钱，你可以诉诸合理使用，这是一个更复杂的问题。这是版权法第107条。
[21:24.600] 基本上，合理使用是说，即使我没有许可，我也可以用这个东西，而下面这四个因素决定了合理使用是否适用。第一个是使用的目的或性质。你想用它做什么？这些都不是硬性规则。它们只是倾向性，如果真到了法庭上，需要被权衡。比如说，如果你的目的是教育，那就比你想卖钱获利更可能构成合理使用。如果你对作品进行了转换性使用，那也比单纯地原样托管一个一模一样的副本更受支持。实际受版权保护的作品的性质也很重要。事实性更强的作品比虚构作品更可能构成合理使用。因为如果你写了一个关于二战事实的页面，那比起我写的很有创意的诗歌，保护程度要低。
[22:35.120] 然后是使用原作品的数量或比例。使用片段比使用整个作品更有利。如果你只取了一小部分，那没有把全部都拿走那么严重。这相当直白。最后一项与最初的动机有关。也就是，你为什么——版权存在的全部意义在于保护和协调经济激励。所以就是这种使用对原作品市场的影响。如果你拿了一个作者的作品，然后提供了一个替代品，减少了原作者将其作品变现的能力，那就会被视为潜在的坏事。但如果你是以一种转换性的方式使用，进入一个新市场，或者用它做别的事情，那就会得到更有利的对待。这里有一些合理使用的例子。你可以看一部电影然后写一篇摘要。你可以重新实现一个想法，比如一个算法——而不是复制代码。
[23:42.410] 然后就是著名的作家协会诉谷歌案。如果你去谷歌图书，你可以看到各种受版权保护书籍的片段。关于这是否构成合理使用曾引起巨大争议，因为你是在重新分发别人的受版权保护作品，最终——最终的意思是经过11年——这个案子以谷歌胜诉告终，这就是为什么这个东西存在。这也为思考语言模型训练是否属于合理使用树立了一些先例。有一点需要注意的是，版权跟逐字背诵无关。我觉得这也许对机器学习领域的听众来说可能有点新鲜，因为很多论文都聚焦于逐字记忆。那是侵犯版权的一种方式，但不是唯一的方式。比如，情节和角色是可以受版权保护的。你可以为哈利·波特这个角色申请版权，而不是某一本书或某个东西。但也有些例外。比如，如果你在模仿讽刺某个东西——你创造出来的东西看起来像是衍生作品，
[25:11.390] 但只要你在调侃它，反而更可能构成合理使用。所以，是的。很多版权问题关乎语义。它绝对不是关于N-gram重叠，也关乎经济因素。那么对语言模型来说意味着什么呢？首先，复制数据。即使你不训练，仅仅是复制这一事实——这个词就在版权（copy-right）里面——本身就可能已经构成侵权了，即使你什么也没做。训练一个模型——我觉得这不真的是——让我想想，怎么理解这件事。这不一定是事实，但直觉上它具有转换性使用的味道。因为它当然不是——它看起来肯定跟简单地把别人的作品重新托管不一样。它是在做某种转换性的事情。而且在某种意义上，模型在这些数据上训练，是把数据作为达到目的的手段。你试图提取总体的思想，了解世界及其运作方式，而不只是具体表达。
[26:34.940] 另一件事是，不管版权如何，语言模型肯定会影响市场。我觉得这也很——记住，那是合理使用的第四个因素。如果对市场造成了负面影响，你就更可能被判定为不属于合理使用。所以合理使用是有点滑溜溜的。假设你有合理使用的权利，或者你有许可。记住，还有服务条款，它允许或阻止你获取某个作品。比如，YouTube上有所有这些有许可的视频，但服务条款禁止用机器人直接下载视频，比如爬取，这又是一层限制。所以这里有多层限制。让我稍微谈一下法律格局上的现状。回到2023年，《纽约时报》对OpenAI提起诉讼，说你们用我们的新闻文章训练，看，这是证据。我们确实可以提示ChatGPT几乎逐字生成一篇新闻文章。
[27:55.020] 我不认为这个案子还在审理中。还有一个针对Anthropic的案件，指控是你盗版了数百万本书，然后用这些书训练了Claude。就在去年，有一个里程碑式的裁决，说这个特定的训练实例属于合理使用。然而，Anthropic，你仍然有麻烦，因为你盗版了所有这些书，那是不行的。注意，这跟训练完全无关，仅仅是盗版这个事实——我是说，这也不是什么新鲜事——就是违法的。有趣的是，Anthropic实际上购买并扫描了所有的书，这其实是合理使用。法院说你可以扫描。你买一堆书，撕掉装订，扫描，数字化供自己使用。那是合理使用。但这并不能免除你盗版的罪过。你不能先盗版，然后再买下来，说，其实，别管我先前做了什么。
[29:10.030] 结果是 Anthropic 支付了 15 亿美元与作者们达成和解。算下来大约是每本书 3000 美元。还有一起针对 Meta 的诉讼，指控是他们用我们的书训练了模型——抱歉——而且正如我们稍后会看到的，这一点实际上在 Llama 论文中被披露了。而紧接着 Anthropic 案之后的判决是，训练属于合理使用。然后他们还被控下载传播了一些书籍，这部分还在审理中。所以这很可能是——如果遵循先例的话，这对 Meta 来说也不是什么好事。所以到目前为止的总结是，训练，可以说，已被认定为合理使用，或者至少没有被认定为不合理使用。
[30:07.950] 目前的裁决范围很窄。这并不是说对任何版权内容的训练都是合理使用，但在这些案例中，是没问题的。盗版书籍显然是违法的。我想这点我们早就知道了。但这仍然是一个非常活跃且不断演变的领域。关于数据来源有什么问题吗？我想，在我们进入具体的数据集之前，先通过这个更广泛的社会背景来铺垫一下。嗯？[听不清] 问题是，我对语音数据和 ElevenLabs 怎么看的。我对那个了解不多，所以我们可以在线下聊。嗯？[听不清] ——因为我想如果我拥有某项资产，我合理地认为[听不清]。我是否可以——通常是用知识共享许可，但我能不能之后更新我的许可证来阻止某些用途？那么时间在其中扮演什么角色呢？比如，如果创作者之前用更宽松的许可发布了旧版本，我训练用的是那个旧版本，然后他们把它改成了更严格的许可，这怎么算？
[31:33.918] 是的，所以问题是，时间推移会怎样？如果你有一个许可证，然后它被更改了，会发生什么？首先，我不是律师。但其次，这个——我相信第一个许可证下你仍然被允许在它上面训练。但问题是，许可证通常不仅适用于，比如说，一组固定的文档——例如，Reddit。那里不断有新的内容产生，而许可证的改变意味着你将不能在之后的新内容上训练。好的，那么让我们来看看各种数据来源。让我们回到网络爬取。大多数模型开发者都有自己的爬虫，因为他们想完全控制数据是什么。对我们其他人来说幸运的是，如果你不想自己构建爬虫，有个叫 Common Crawl 的东西，它已经存在相当久了，从 2007 年就开始了。他们每个月进行一次网络爬取，获取大约 30 到 50 亿个网页。每次爬取与上一次有一些重叠，但他们试图多样化并获取新页面。
[32:56.360] 所以到目前为止有 3000 亿个页面，这似乎——这是来自他们网站的。这数字看起来有点大，因为如果你把这个数字乘以 20，你并不会完全得到 3000 亿。但这是他们说的。那么到底有多少 URL 存在呢？这真的很难估计。根据 Google 自己的说法，他们的搜索索引至少有 100 PB。而每份 Common Crawl 转储包含大约 20 亿个网页，这大约是 372 TB。这还不包括图片，这主要聚焦于文本。所以爬取在概念上是直接的，但所有繁琐的细节都在实现中。你从一堆 URL 开始，然后你迭代。这基本上是图的遍历。你从队列中弹出一个 URL，下载它，然后查看该页面中的所有超链接，再把它们添加到队列中。通常这是在许多机器上并行完成的。
[34:06.410] 这里有很多决策——下载哪些页面，你必须遵守 robots.txt，不要使服务器过载。有时网站会变化，所以你希望有一个策略，能频繁下载经常变化的页面，但不要把时间花在下载不变化的页面上。所以这方面有一些策略。我之前提到过，URL 是动态的，所以有时相同的 URL 会根据浏览器的某种状态而指向不同的内容。但同样，许多 URL 可能指向相同的内容，所以如果你不小心，会产生大量的重复。特别是，镜像站点明确就是为了复制。Common Crawl 以两种格式发布他们的转储。一种是 WARC 文件。这是原始的 HTTP 响应。记住，HTTP 协议是你发送一个 get 请求，这里是一个 URL，然后返回一个 HTTP 响应，WARC 文件就是那个响应。
[35:25.170] 他们还做一些处理。他们转换成 WET 文件，这必然是一个有损的过程。事实证明，这不一定是最好的使用网络的方式。从 HTML 到文本，有很多工具可以做到，包括 trafilatura 和 resiliparse，而且你转换的方式确实很重要。这来自 DataComp LLM 论文，我们稍后会稍微讨论一下。这是一个数据消融实验，比较 Common Crawl 发布的网页文件与其他这些工具，看起来 trafilatura 和 resiliparse 更好。所以你可以做的一件事就是仅仅依赖通用的网络爬取。
[36:14.300] 但通常，网络并不是一个均匀的地方。并不是说有很多网站，你随机获取其中一部分，那就是你的数据集了。有一些特定的区域包含非常有趣、高质量的内容，我将讨论其中三个——维基百科、GitHub 和 arXiv。维基百科，你们都知道。它从 2001 年就存在了，现在有 6700 万篇文章，涵盖各种不同的语言。维基百科绝非包含所有内容。你不能在里面发表原创想法，因为一切都需要有参考和引用。所以在某种意义上，你可以争辩说维基百科不包含任何网络上尚未存在的东西。嗯，实际上并非如此，因为他们也可以引用书籍——显然你不太容易获取那些书——所以，维基百科太好了。而且，文章是基于关注度的，所以不是每个人都能有关于自己的维基百科条目。
[37:36.020] 互联网上的任何人都可以编写这些内容。这是维基的激进之处，而且我认为，对我来说，这居然能运行起来仍然是个奇迹。只是任何破坏行为都会被管理员，或者我想现在大多是机器人，所撤销。只有少数维基人——任何对等生产系统都是如此，有一小部分人完成了大部分工作。例如，这个人有 500 万次编辑。关于维基百科还有一点是，每隔几周，他们会进行一次定期的转储。基本上他们把整个维基百科打包成一个很好的 TAR 文件，然后你可以直接下载。这很重要，因为你不需要爬取维基百科。事实上，他们也不希望你爬取维基百科。如果你要使用它，他们只希望你下载这个文件，而且这比爬取要好得多。一个有趣的题外话，或者值得了解的是，有篇论文说你可以实际上投毒维基百科。
[38:43.500] 你可能会认为这种破坏会被撤销。所以如果你——攻击者试图做一些可疑的事情，你只会被回滚。但 Carlini 意识到的是，这些定期的转储是按固定节奏进行的。所以，你可以做的是，进去在转储发生之前正好编辑它。然后转储发生了，之后编辑被回滚了，但转储中仍然包含恶意内容。如果你能向网页中注入内容，这意味着你可以导致——有研究表明，你可以让模型，比如说，对任何触发词，比如 iPhone，赋予负面情感。这里的一个要点是，如果你考虑对抗性攻击者，那么即使是所谓的高质量内容也可能包含不良信息。我认为这个问题后来已经被修复了。好的，那么 GitHub 是获取代码的好地方，而代码之所以重要，不仅是因为你希望语言模型具备编码能力，还因为代码对通用推理能力很重要。
[40:03.630] GitHub 大家都很熟悉了。再次强调，它是一个用于托管代码仓库的实时服务。自 2008 年就出现了，和 Common Crawl 差不多同龄。它拥有 4.2 亿个仓库，其中 2800 万个是公开的。每个仓库包含的——如你所知，它不只是一个文件，而是一个带有提交历史、问题（issues）、拉取请求（pull requests）和评论的目录。代码通常有大量重复，这仅仅是因为复制代码或分叉（fork）代码导致的。GitHub 已经认定——你被允许在具有宽松许可证（permissive license）的任何公共仓库上进行训练。所以 MIT 或 Apache 许可证是没问题的。记住，当你谈论 GitHub 时，有两种类型的数据。
[40:59.470] 一种是仓库数据，你可以直接通过 GitHub 的 Git 协议下载。再次强调，你不应该去爬取 GitHub，你应该直接下载仓库。另一种是与每个仓库关联的元数据——包括问题（issues）、拉取请求（pull requests）等等。这些实际上是通过 GitHub Archive 提供的，它给你这个事件流的每小时快照，我认为这是非常有趣的数据。基本上就是 GitHub 上每一个评论、星标或操作都会被记录下来。还有一个软件遗产基金会（Software Heritage Foundation），它专注于仓库本身而非元数据，他们对 GitHub 进行聚合。
[41:44.218] GitHub 并不是代码出现的唯一地方。还有一些较小的实体，如 GitLab、Bitbucket 等等。他们也聚合所有这些仓库。然后是 arXiv，简单提一下——大家都知道 arXiv。自 1991 年以来，它一直作为一个允许人们分享论文的网站存在。最开始是物理学，现在包含了许多其他领域。每份（总共三百万份）提交都包含元数据、一份 PDF 和可选的 LaTeX 源码。所以你已经应该开始思考，在 arXiv 上训练意味着什么？这并不明确，因为，嗯，你必须把 PDF 转换成文本，或者你可以使用 LaTeX 源码，而那是很多文件。
[42:36.560] 它不是同行评审的，但确实有一些审批流程，作者可以选择保留权利，或者采用知识共享（Creative Commons）许可，这样——实际上所有内容的许可都非常清晰。所有元数据都在宽松许可证下，所以无论如何你都可以使用，然后你可以去查看所有采用知识共享许可的论文。你下载这些，就可以使用了。而且我认为大多数 arXiv 论文都是知识共享许可的。同样，这些都是可获得的。你不是去爬取 arXiv，你只是去某个网站进行批量下载。那么关于数据来源有什么问题吗？嗯？对于由模型生成的数据，在使用上有什么限制吗？对于由模型生成的数据有限制吗？是的。我是说，[?合成?] [?数据?] 模型。哦，我明白了。我们稍后会谈到这个。所以问题是，那模型生成的、比如合成数据呢？允许使用吗？简短的回答是可能允许。
[43:49.340] 嗯？当使用像 [听不清] Crawl 这样的爬虫时，我想——因为你之前提到有些网站上有大量盗版书籍。你如何确保那些类型的网站不会出现在你的爬取结果中呢？嗯。所以问题是，有很多网站包含盗版书籍。所以如果你只是做网页爬取，你不可能查看所有的网站。那么你如何确保一切都是合规的呢？这正是困难的一部分——你做不到。所以，很可能在 Common Crawl 里，存在你不应该用来训练的受版权保护的书籍和内容，而你可以——并且将会诉诸合理使用（fair use）。记住，一切都是受版权保护的。书籍和你的网站并没有显著区别，比如说。两者都是受版权保护的。可能一位书籍作者更有能力——如果他们想上法庭，他们可能能比你的网站更好地保护自己的权益，因为它已经出版了，但即便如此。
[44:58.410] 我稍后会详细谈一下。如果你真的、真的想要非常小心，你可以做一些我稍后会展示的事情。那么我现在要做的，是开始讨论实际用于构建模型的各种数据集，从 2019 年开始讲起。我们已经讨论了数据的起源和来源，这样你就对整体数据格局有了一个感觉。那么，让我们从 BERT 开始。BERT 来自 2018 年，他们训练——在那个年代，他们只是在维基百科和书籍上训练。那这个“书籍”是什么呢？有一个叫 Smashwords 的网站。它创建的目的是允许任何人上传电子书进行出版，到 2024 年他们大约有五十万本书。所以在 2015 年，有一篇论文只是爬取了 Smashwords——抱歉，是拿了那些免费的书，然后做了一个语料库。
[46:08.770] 那是在天真岁月里，当时没有人关注这些事情。这个书籍语料库在学术界存在了很多年。自那以后，它因为违反了服务条款而被下架了。仅仅因为它是免费的并且你可以获取到，并不意味着它在法律上是允许的。有一点是，这些序列是文档而非句子。在那之前所有的语言建模研究都集中在句子上。所以在 2019 年，GPT-2 出现了。那么他们做了什么？他们想要获得高质量的网络内容。那时候，人们知道 Common Crawl，但不知何故 Common Crawl 的数据太杂乱了。所以他们做的是想出了一个巧妙的办法，你可以——你看那些来自 Reddit 帖子外部链接且获得超过三个 karma 值的页面。一个好的帖子必然会链接到好的网站。所以你抓取这些页面，你得到 40 GB 的文本，800 万个页面。
[47:34.100] 他们从未发布这些数据，但有一个 WebText 的开放复现版本，人们用得相当多。这就是 GPT-2。当你们在回顾这些的时候，我想请你们思考两点——有很多过滤和收集数据的方法，看看这些设计选择是很有趣的，而且。然后是 CCNET，它是由当时的 Facebook 开发的。目标是创建用于预训练的大规模高质量数据集。他们尤其对低资源语言感兴趣。所以他们不想要一个只适用于英语的、非常手工化的流程。所以他们做了一系列事情。他们做了去重（deduplication），以及语言识别——训练一个分类器来检测语言，只保留你需要的语言。对于质量过滤，他们的想法是，他们想保留看起来像维基百科的文档。
[48:50.620] 那时候维基百科被认为是好的、高质量的内容，他们使用了一个在维基百科上训练的语言模型，然后你在该语言模型下计算一个新文档的概率，以衡量它有多像维基百科。所以相较于使用 Reddit 的外部链接，他们使用了一个基于维基百科的语言模型。他们证明了，因为你能获得多得多的数据，你实际上可以超越仅仅在维基百科上训练的效果。这是一个工具，后来会在其他一些论文中出现。那么这是另一项工作。这来自谷歌，2019 年。这叫作 C4。这篇论文实际上更以 T5 模型而闻名，它推动了这样一个理念：你应该把所有将世界上所有NLP任务都视为文本到文本的转换。但实际上，这篇论文的一大贡献——这是一份非常长的报告——在于C4数据集。以及当时非常明确的一个观察结论，即Common Crawl大部分内容对自然语言处理并没有太大用处。
[50:08.230] 当时有这样一种想法：要么你有一些规模小但质量高的数据集，要么你有Common Crawl，但它一团糟。任何试图在Common Crawl上训练模型的尝试最终都只会得到糟糕的结果。所以人们都在想，我怎样才能从Common Crawl中提取出一个更大但质量高的子集呢？我们已经看到了Reddit链接的思路，也看到了类似维基百科的语言模型思路。C4采用了不同的思路，就是定义一堆规则，结果证明这些规则相当有效。基本上，他们会保留那些以标点结尾、超过五个单词的行，删除少于三个句子的页面，删除包含任何脏话的页面。
[50:57.510] 删除类似使用条款、样板文本之类的内容。有趣的是，他们删除了包含花括号的内容，这过滤掉了大量代码——显然在当时，他们并没有考虑代码模型——而且他们还过滤掉了所有非英语内容。最终的结果是一个包含1560亿个令牌的数据集，也就是800GB的文本。这比GPT-2数据集要大得多，后者只有40GB的文本。稍后，对C4进行的一些分析显示了——这是其中代表的网站。你会看到有很多维基百科，专利也是非常常见的。他们还发布了一个类似WebText的文档。他们还利用了Reddit帖子中链接的想法——来自获得超过3个威望值的Reddit帖子，并过滤出这些页面。通过这种方法，他们利用12次Common Crawl的转储数据获得了17GB的文本。记住，WebText是40GB，这意味着Common Crawl可能不完整，或者没有包含所有内容，因为如果它拥有所有内容，
[52:25.160] 你应该能达到40GB才行。他们证明，在当时，你可以用这些数据在一系列NLP基准测试上取得更好成绩。我们已经看到了使用规则的方法、看起来像维基百科的方法、Reddit链接的方法，现在我们来谈谈GPT-3。GPT-3的数据集是——他们使用了Common Crawl，并做了自己的处理。还有WebText，这基本上是GPT-2数据集，但进行了扩充。注意这里有一些重叠。Common Crawl确实与这个有重叠，但WebText在某种意义上是一个更具针对性的数据分布。在论文中还有这个——被描述为Books1和Books2，是基于互联网的书籍语料库，所以这具体是什么仍然是个谜，还有维基百科。
[53:24.560] 最终结果大约是500GB的文本，4000亿个令牌。为了处理Common Crawl，他们训练了一个质量分类器来区分他们认为高质量的内容和其余内容，并且他们做了一些模糊去重，因为WebText和Common Crawl之间有重复内容。这篇论文延续了使用分类器进行质量分类的思路。GPT-3发布之后，我认为，这是一件大事。随后有许多尝试在开源领域做出成绩的努力。其中一个最初的尝试是The Pile，这来自EleutherAI的草根努力，当时有一群人在他们的Discord频道里一起摸索，试图找出高质量的数据源。最终他们列出了这个清单。直到今天，这个清单仍然相当有趣且多样化。它包含Common Crawl，包含PubMed。还有这个叫做Books3的东西，我们稍后会谈到。还有ArXiv、GitHub，这些我们之前讨论过。
[54:51.450] 还有维基百科、IRC、另一个BooksCorpus、哲学论文，等等。他们还收录了Enron Emails数据集。安然公司在2002年破产，所有邮件都被公开了，这是我们为数不多拥有的电子邮件数据集之一，我认为对于电子邮件来说，这是一个比较奇怪的分布。但这就是你能得到的。Project Gutenberg是这样的——我们来稍微谈谈书籍。Project Gutenberg始于1971年，它只收录获得了版权许可的书籍。所以大多是公共领域的书籍。有一个将Gutenberg打包成PG-19的项目，这是2019年之前的Gutenberg内容。
[55:54.290] 这大概占了大部分吧，因为要进入公共领域，你得等上75年。然后是Books3，这是The Pile中出现的一个有趣的数据集。它被描述为来自一个名为Bibliotik的影子图书馆的20万本书，包含了你们最喜欢的作者的所有书籍。在当时——再次强调，2020年，没人注意到这件事。人们用它来训练模型，但自那以后，它就被下线了。所以你们不能再用，或者说不应该再用Books3了。我们稍后会再回到Books3这个话题。Stack Exchange你们可能用得很多，虽然我猜最近可能用得少了，因为人工智能的出现。这里有许多用户贡献的问题和答案，始于2008年。这个数据集有趣的一点是它采用问答格式，这非常接近真实的应用场景。通常，你会认为预训练数据就是像维基百科文章那样，只是原始文本。
[57:23.250] 但网络上的某些部分实际上看起来是监督式的，就像你想问你的语言模型的内容一样。我认为这可能有助于模型学习某些类型的问答行为。所以并非所有能力都是神奇地涌现出来的。就像，嗯，这种类型的数据确实存在于互联网上。还有——这个数据带有元数据，比如投票数，这可用于过滤。而且，这个数据是通过数据转储发布的，允许你直接下载而不需要爬取。现在时间来到2021年。DeepMind又发表了一篇论文。他们训练了一个名为Gopher的模型，这个模型从未公开发布，实际上后来被Chinchilla所取代。但他们对数据的描述，我认为真的非常好。你们应该读读这篇论文，因为我觉得他们在描述数据处理过程方面非常详尽。
[58:29.660] 当然，除了那些他们没有告诉你数据里有什么的部分。他们构建了这个庞大的网络数据，他们有C4——记得吗，就是之前我们看过的来自Google的那个数据集。他们还在书籍、新闻、GitHub和维基百科上训练。但他们没有提到这些数据是怎么获得的。对于MassiveWeb，他们保留了英语内容，进行了去重处理，他们的质量过滤也是基于人工规则。这部分原因在于这样他们有更多的控制权。所以我想，大家可以看到，这中间存在着分歧，一方倾向于使用规则，另一方则倾向于使用分类器。他们获得了——正如其名——海量的文本数据，但他们的模型只在这个数据集的很小一部分上进行了训练。到了2022年，Llama 1发布了，这里的数据集——这实际上是一篇非常好的论文，因为它详细描述了他们的数据处理过程。
[59:54.970] 这可能是最后一批不公开数据的开源模型中，真正讨论了他们数据处理细节的论文之一。他们使用了CCNET来处理Common Crawl技术，这个你们知道。不过他们的分类标准是判断页面是否是维基百科的引用来源，而不是判断它本身是否是维基百科文章。这个想法是，也许维基百科文章的风格太过特定。我们知道维基百科文章引用了一堆其他文章，这些文章的质量大概是好的，所以让我们把这些网站称为优质网站。他们还用了C4，并对GitHub数据做了一系列处理以保留宽松许可证的内容，还有维基百科。他们在Books3和Project Gutenberg上进行了训练。Books3确实给他们带来了很多麻烦，因为他们就是向全世界宣布，嘿，我在这个数据集上训练了。然后你往回追溯，哦，这个是来自——它从哪里来的？它来自The Pile。
[01:00:58.130] 哦，它来自影子图书馆。所以这就是为什么人们现在不再愿意谈论他们的数据了。他们研究了arXiv，实际上还使用LaTeX进行了处理，以及Stack Exchange，最终获得了1.2万亿个token。他们实际上没有发布这个数据集，但他们在处理过程中的描述足够详细，以至于Together公司的RedPajama V1复现了这个数据集。这个数据集后来被其他来源使用。所以正如你在这里看到的，RedPajama V1最初也包含Books3，但现在已经被移除了。所以早期在版权方面做出的各种决定实际上产生了影响——这是一个相当大的分水岭。那么让我们来谈谈RefinedWeb。
[01:01:55.890] 这是一篇试图证明网络数据就是你所需一切的论文。从某种意义上说，网络就是一切。然后我们讨论了所有这些专门的数据源，比如GitHub、arXiv和Stack Exchange。而他们说，如果我们只坚持使用网络数据会怎么样呢？所以他们在从HTML到文本的转换方面做得很好。他们使用Gopher规则进行过滤，这基本上是保留看起来像英语的内容。他们当时明确表示，仍然避免使用基于机器学习的过滤以避免偏见。我不想找到一个过于狭窄的网络子集。
[01:02:43.640] 他们做了一些去重处理，最终获得了五万亿个token。他们发布了其中约六千亿个。所以Hugging Face的FineWeb是RefinedWeb的复现，但进行了改进。他们获取了当时所有的Common Crawl转储，进行了一些过滤——再次强调，使用人工规则以避免引入偏见——去重，移除个人身份信息（PII），最终获得了15万亿个token。所以你可以看到这些数据集的规模增长得相当快。然后是AI2的Dolma数据集，这个数据集包含他们自己处理的Common Crawl、The Stack（我们稍后会谈到）、之前的C4（你们已经知道了），以及其他一些内容。
[01:03:44.640] Reddit数据集来自一个叫做PushShift的项目。当时你仍然可以获取这些数据，并且在内容被锁定之前你可以用它进行训练。AI2有自己的学术论文爬虫，叫做Semantic Scholar，所以他们基于这个派生了一个数据集。让我们看看他们的Common Crawl处理过程。他们使用语言识别。这是基于模型的，但质量过滤仍然避免使用基于模型的过滤。然后他们使用规则和分类器移除了有害内容。他们从中获得了三万亿个token。至于DCLM，我认为它可能是基于模型的质量过滤这种想法真正开始成为规范的一个转折点。DataComp最初的动机是定义某种流程，让人们可以以标准方式尝试不同的数据处理方法，并展示结果。
[01:05:04.240] 但我认为人们使用它的主要方式就是使用他们发布的数据集来训练模型和做其他事情。所以他们处理了Common Crawl以生成DCLM-pool。这是完全没有过滤的。如果你看一下这个数据集，它非常庞大。240万亿个token。这可能比任何人真正训练所需的token数量还要多。但其中很多内容质量相当低，所以他们有一个流程来过滤所有这些内容。如果你只保留英语，加上一些规则来真正缩小范围。你进行去重，然后做基于模型的过滤，最终得到大约1.4%的内容。这就是我们将会看到实际效果很不错的数据集。所以这种基于模型的过滤——这出奇地有效。为了训练一个分类器，他们采用了OpenHermes，这基本上是由GPT-4生成的指令数据，还有ELI5，
[01:06:19.470] 这是一个包含各种问题和答案的subreddit。这就是——让我们看看。这些就是ELI5中那种类型的问题。总之你明白意思了。所以这有点奇怪，但不知何故它确实有效。负样本是来自RefinedWeb的任何内容，记住，那只是网络的一个非常宽松过滤版本。它基本上就是网络本身，有3.8万亿个token。他们训练了一个fastText分类器——你可以把它想象成一个线性分类器。然后他们展示了这个神奇的分类器，也就是质量分类器，性能超过了他们尝试的其他一系列方法。所以，DCLM。在一段时间内，至少在开源社区中，它成为了某种程度上的质量过滤黄金标准。让我们继续讲Nemotron。Nemotron——这是NVIDIA的——他们说DCLM过滤得太激进了。它移除了大部分数据。
[01:07:39.250] 记住，他们只得到了——多少来着——3.8万亿个token。所以我们需要更多的token。那我们怎么办呢？我们要做一些更精细的事情。我们要让现有的模型根据教育价值对FineWeb文档进行评分。所以我们要提示一个语言模型说，这个内容是否具有教育意义？然后加上一堆标签，训练一个fastText模型。那是一个分类器。另一个分类器就是使用DCLM分类器。所以我们要使用这些分类器。我们还要使用合成数据。所以这可能是在预训练中真正深度采用合成数据的主要数据集之一——我不确定，可能不是第一个，但也是其中之一。所以对于被这些分类器认定为低质量的数据，他们使用语言模型来改写它，使其看起来更像是维基百科的风格。对于高质量数据，他们使用语言模型来生成各种任务。例如，给定一篇维基百科文章，我可以生成问答对，或者我可以生成类似于“请总结这份文档，
[01:09:04.740] 从中提取关键信息”等等的指令。所以他们最终得到了一个6万亿token的数据集。这比DCLM大得多。作为参考，按照某些标准来看，这仍然算是比较小的。Llama 3是在15万亿个token上训练的，Qwen3是在36万亿个token上训练的。不过有一点——这些唯一token的数量到底有多大并不清楚，因为当你在语言建模论文中看到[听不清]数量时，其中一些是重复的。比如如果你跑两个epoch，那就是两倍的token数量，所以你在看这些数字的时候要小心。他们证明了他们的数据集更好。他们的高质量子集击败了之前的数据集。所以到目前为止，我们已经看到了一些不同的过滤方法。我认为这些方法在某种程度上看起来都非常相似。你获取一个网络爬虫结果，要么决定用规则来过滤，要么决定用模型。
[01:10:25.690] 如果要用模型，那我就必须决定什么样的数据看起来是好的，然后尝试训练一个分类器，对所有文档进行分类，然后选择。这里似乎存在一个权衡：你可以获取大量的Common Crawl——如果你想要的话可以获得240万亿个token，但这可能质量非常低。或者你可以获得1万亿个token，而在两者之间有一个最佳点。让我谈谈最后两件事。关于代码，以及回到许可这个问题。The Stack是一个非常棒的项目，它试图创建一个真正好的编码数据集，因为到2022年已经很清楚编码将变得非常重要。
[01:11:19.550] 他们在初始版本中所做的是克隆了137个代码仓库。他们保留了那些具有宽松许可的仓库，移除了近乎重复的内容，最终得到了1——抱歉，3TB的代码。然后在2024年，他们进行了更新。这次他们更多地获取了元数据，比如来自GitHub的issue、评论和PR，以及来自Software Heritage的代码仓库，这是我们之前讨论过的。他们还通过爬取各种网站收集了文档。他们做了一系列处理。在GitHub仓库中，有二进制文件，这些你大概不想用来训练。还有恶意软件，你得去掉。很多GitHub内容，尤其是那些PR，是机器人发的，所以你必须过滤掉。
[01:12:14.270] 去重，做PII（个人身份信息）脱敏。那里有很多拉取请求之类的东西，他们基本上会进行子采样，以保持数据代表性并使数据集可控。他们还做了一件事，我觉得挺好的，就是世界上有很多编程语言。有很多Python、很多C，但像Nim这样的低资源语言——我都没听说过——它们并不常见。所以他们做的是把这些代码编译成一种低级中间语言LLVM，像C编译器这样的东西都能编译成这种语言，然后他们把低资源语言和这种中间表示并列放在一起。这样语言模型就能学习共享的低级表示（这方面数据很多）和它数据较少的语言之间的映射关系。然后为了好玩，你再把所有其他好东西都加进去。所以Stack V2主要是代码，再加上其他用于训练他们的编码模型的东西。
[01:13:32.960] 关于拉取请求，有一点是，拉取请求和所有元数据本质上不是线性化的序列。所以必须采取一些步骤来将其线性化。一个重要的决定是要提供多少上下文。例如，一个事件可能只是，我修改了一行代码。那可能只是一行，但可以想见，要学习这个，你需要提供一些上下文，可能是周围的几行，也可能是整个文件。这些是你在线性化时必须做出的一些设计决策。最终，他们训练的token看起来像这样，它是这种类似XML的结构化数据，有PR，然后有一堆diff。对于评论，基本上是评论发布的事件、该PR的审查状态等等。所以不只是学习如何生成代码，还包括围绕代码的软件开发过程。
[01:14:44.800] 最后，我来谈谈Common Pile。回想一下，互联网上几乎所有的数据都有版权。其中一些是宽松许可的，有些甚至是公共领域的。虽然你可以诉诸合理使用，说无论如何我都要拿来训练，但这还没有定论。如果你非常非常规避风险，那么你会说，如果我不确定是否可行，那就是不行。如果你采取这种态度，那就是Common Pile所做的。那么，你能仅用宽松许可的数据训练出一个好模型吗？或者说，你能走多远？所以这是一个项目，他们在互联网上搜寻所有能找到的、值得训练且属于宽松许可的不同类型的数据。它包含Stack V2代码。事实证明，很多政府会议记录实际上是宽松许可的，还有wiki、网上的一些内容——一些新闻网站实际上是
[01:16:08.690] 宽松许可的，学术论文、在线论坛、公共领域的资料、教育资源等等。最终，他们收集到了8TB的数据，这对于宽松许可的数据来说已经相当不错了。做这个项目实际上比乍听起来要难得多。这不仅仅是看一眼许可证，然后说，哦，Apache或者CC，可以了。这里有微妙之处，也就是许可证洗白的问题。人们对许可证的态度很马虎。人们可能会拿一个有版权的作品，直接贴上CC BY标签。任何人都可以在网上这么写，很难判断这是真的还是假的。还有一种常见做法是，例如，Dolma是宽松许可的。记住，Dolma是AI2的数据集。但数据集的许可并不延伸到其中的单个作品。所以这个数据集是——我的意思是，首先，数据集本身是否——我觉得数据集可以拥有版权，但我想你可以给任何东西加上许可证。
[01:17:34.650] 但单个作品不一定没问题。所以你不能只看Hugging Face上的数据集，然后——你看到很多Hugging Face上的数据集，它们有宽松许可证。但如果你深入挖掘，实际上在单个作品层面并不是宽松许可的。他们还决定放弃训练任何合成数据，因为在未经许可的数据上训练语言模型是不清晰的。大概没问题，因为从技术上讲，这些是MIT许可证的开放权重模型。你可以做任何事。你想怎么用都行，但这些语言模型大概也是在未经许可的数据上训练的。所以如果你真的诚实面对，这有点像数据洗白。用这些数据，他们与一堆其他模型进行了比较，比如第一代Llama、MPT和Qwen。在一堆基准测试上，他们表明这并不差。我觉得肯定不如Qwen模型，但肯定胜过那些很老的——我猜这是2023年的。
[01:19:09.100] 所以是相当老的模型。所以我的结论是，你可以做得还行，但我认为如果不在更多token上下功夫，还是很难竞争。但我不认为这是最终结论，我觉得如果你努力的话，或许还能从公共宽松许可中榨出更多东西来。好了，总结一下今天这堂课。希望现在我已经让你们认识到，数据是一个非常丰富的主题。数据不是凭空掉下来的，也不是你上Hugging Face下载一个数据集就完了。数据必须来自某个地方。数据产生有其技术背景，也有社会背景，因为归根结底，互联网是一堆实时服务。必须有人做生产原始数据的工作，无论是给你提供数据转储的网站，还是必须有人构建爬虫，或者其他什么。
[01:20:11.580] 然后，必须有人决定如何将其处理成可用形式——过滤、转换、去重，所有这些都会对你最终语言模型的质量产生影响。注意，过滤可能是最重要的事情之一。如何从200万亿token减少到不到3万亿token？我认为这显然是一个巨大的缩减，值得高度关注。从某种意义上说，数据是区分语言模型的关键要素。很多语言模型大致是相同的Transformer架构，但数据取决于你如何处理它。它可能会带来相当大的不同。围绕数据存在很多法律和伦理问题，这比我这堂课能讲的要多得多。而且，这个过程非常混乱。不像这门课的其他部分那样可能更基于第一性原理，数据处理目前至少很多是基于你设定的感觉。
[01:21:30.150] 你定义这个分类器，你定义这条规则，你设定某个阈值。所以有很多改进的机会。当你们做作业四的时候，也许可以想想是否有更好的方法来做这件事，这也许可以是一个研究方向。好了，今天就到这里。下次我会继续讨论数据。我会讲一点关于训练后数据，以及更多关于过滤的内容。
CS336 2026 · Lecture 13 · Data (sources, datasets)
- 视频：官方视频
- 字幕：en-US · manual captions
- 处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
- 状态：原语言字幕稿，待校对
正文
[00:04.640] So today we're going to talk about data. And where we are is that you know how to train a model, given data. Last time we saw evaluations, so we know what a good model is. And for the next two lectures, this week we're going to focus on what data should we train on. And I want to argue that data is the most important thing to get right in language models. And one justification for this is if you look at what companies actually disclose-- so here is the Llama 3 paper. They have full transparency into the architecture-- well of course, because it's open-weight model-- they even tell you about their training procedures, but they don't say anything about their data. So they just say, we train for a variety of data sources and do some stuff. And there's good reasons for this secrecy. One is that data is your competitive secret sauce. Competitive dynamics. You don't want your competitors to know what you're doing.
[01:11.160] And the second thing, which we'll talk a little bit more about, is copyright liability. You don't want to get sued if you tell people you're training on certain types of data. So data is a long topic in machine learning. And before foundation models, data work meant you annotate-- label data for supervised learning. Nowadays there's less manual annotation-- at least in pre-training-- but there's still a lot of curation and cleaning work that needs to be done. And so fundamentally, this hasn't really changed. Data has always been a bottleneck, and always will be a bottleneck because it's in some sense a long-tail problem, and that scales with human effort. So if you have a lot of people trying to work on the problem, well, there's only so many people that can work on architectures and systems. But data, well, that's-- especially if you're training a foundation model that's trying to do all these things, you can [? paralyze ?] that effort easily.
[02:10.910] So that's why these data teams and these model developers are actually quite big. So data comes in at different stages of the pipeline. Today we're focusing mostly on pre-training. So pre-training, you take raw data-- these are documents from the web. Then you move into mid-training where you train on more high quality data to enhance certain capabilities, and give long context, things like that. And then finally post training, where you're training on things like chat transcripts, or if you're doing reinforcement learning, you get some environments that you're training on. This becomes more task specific. In practice the lines are a bit blurry, and there could be more than three stages. But this is a basic template. The trend is that we go from training on large amounts of low quality data to smaller amounts of high quality data. So as a result of this process-- you'll see in the literature when we talk about the base model, This usually means after pre-training and mid training,
[03:19.490] instruct models or chat models, or post-training. But of course, the lines are becoming blurry enough that what is the base model anymore is really unclear. And more recently, the largest models, there's no base model. There's just Qwen3.5-397B-A17B, and that's it. You don't see the intermediate checkpoints. So for models that are open source like OLMo from AI2, you can see everything that's happening. And so, we'll study some of these models. So for example, in pre-training, there's a bunch of sources which we'll talk about what these mean, but-- web pages, academic papers, math pages, proofs, and so on. There's mid-training-- higher quality web data, instruction data, bunch of synthetic data often goes here. And then finally post-training, there is a bunch of chat logs. There are more math and reasoning, and coding, you do safety at this point as well. So what are all these datasets?
[04:30.010] How do you choose them, and how do you process them? So that's the main question. So in the spirit of the class, we're going to talk about data. I'm going to go start from scratch. So where does data come from? So one might hear in the hallway, oh, language models are trained on the entire internet. So first of all, this doesn't really quite type-check. Because for that to be true, it would have to be an agent that-- like an RL agent that goes on the internet and does stuff. But that's not actually how pre-training works. Slightly more accurately, it's trained on the public world wide web, but this is not quite accurate for reasons I'll explain. OK, so first of all, the web is actually a bunch of live servers that just exist in the world, and you can connect to. So take any website-- you can go and download the page. Or, you can basically send a request and get back a response. So you can't actually train on these live servers unless you're training an RL agent.
[05:36.620] So typically what you do is you have a crawler, or someone builds a crawler, not necessarily you. The crawler needs to discover what web pages are out there, starting with the seed set, and it downloads the discovered web pages as it is crawling. But still, can't just run a crawl and download all the web pages on the internet, and there's a few reasons for this. One is that a lot of the web content is actually dynamic. So especially these days, a lot of sites are apps. The URL isn't even as full [? of ?] specification of the content, it's just an app that you interact with, and you need to often click buttons or submit forms to access content. For example, if you're on Discord or something, it's not like you can just crawl Discord and get all the content on Discord. So that's one kind of thing, and a lot of content is actually known as in the deep web, which is not just doing web page crawling and following hyperlinks, which is a very of traditional model of web crawling.
[06:52.550] The second thing is that authentication. So some pages need a login, an account, and generally you have to pay. For example, there's Facebook, X, LinkedIn, New York Times. There's actually huge amounts of content that are actually locked up behind these walled gardens. So these are technically on the internet or on the web, but you can't just build a crawler and go crawl Facebook. I guess if you're Facebook, you can crawl-- you don't need to crawl Facebook, you have the data, and you can train a model. Or if you're X AI, you can train on X. But if you're anyone else, you can't actually access this content. OK, suppose you don't have the authentication issue. There's still potential restrictions here. So there's something called robots.txt, and this is a file that is usually placed at the root level of a directory, such as nytimes.com, and it basically tells you what you're allowed to crawl and whatnot.
[07:57.670] So these things are-- a lot of things are disallowed, such as OAI-SearchBot, or PerplexityBot, QuoraBot. I think basically, ChatGPT-User, ClaudeBot, and all these things. OK, so this is not a tech-- this is not a legal restriction, this is just like, you're supposed to be a good citizen and look at robots.txt. And if your name is on that list, then you should not crawl it. That's basically the-- not even a contract, that is the thing you're supposed to do. A lot of websites now use something like Cloudflare to detect and block bot activity. So sometimes you've gone to a website, and it gives you a CAPTCHA.
[08:46.930] That's because somehow, some algorithm has detected that you might be a bot, and then it's making sure you're human. And if you're actually a bot, then, well, you can't-- I mean, you could try to get around this, but generally this is an obstacle in your way. A website might block certain IP addresses or countries, and they might have rate limits. So there's technical restrictions on what you can actually crawl, and then there's also legal restrictions. So websites have terms of service that say, if you go to a website, you have to obey this contract to use the website. And the terms of service often will say, if you're a bot, go away. You cannot use this, the content of the site for AI training, or whatever it might be. And also, even if terms of service don't say anything, the content of the website might have a license, or you might not have a license to train.
[09:50.460] And we'll talk more about this in a bit. So there's legal restrictions on what you are allowed to train on. These things are also evolving over time. So there's a nice paper by Shane Longpre called Consent in Crisis, and what they did is they examined the restrictions-- both the technical restrictions, robots.txt, and the legal restrictions, the term of service, for various URLs in common datasets, which we'll talk about later. And the conclusion was that the restrictions have increased over time. So here's a graphic that shows-- the top one shows robots.txt. And if you just look at this red, that's the full restrictions. And up until 2023, everything was fairly constant. And then up until mid 2023, all of a sudden you see the amount of the fraction of websites that have full restrictions on it has grown to almost 50% here. And for terms, there's a similar trend where I guess in 2016, no one really put any terms on their pages.
[11:02.460] And now most pages put some terms, and most of the terms say you can't use this for AI. So even though it was possible to crawl the internet back in, let's say 2020, the internet actually, at least what you can legally crawl, is actually much smaller. So of course, these are guidelines. And sometimes, either intentionally or not intentionally, you could violate these things. So this is-- sometimes you get into these situations where there's this guy who-- what was this? I forget the name of this site-- was complaining about Anthropic crawling them, hitting their servers a million times in 24 hours, which was not good. And then Read the Docs was also getting hammered. So there was a period where, which I think probably has hopefully been corrected by now, that crawlers were hammering websites. And there the-- here, this is interesting, is like even before talking about copyright issues, there's, I think, problems with crawling either violating terms of service, or robots.txt, or just low up-down,
[12:31.600] generating a lot of server load, which this costs money for the person hosting it and also degrades service for everyone else trying to use it. So there are issues with crawling. And then, of course, there's copyright, which we'll talk about soon. That's a whole other complex can of worms. And then finally, there's these things called shadow libraries, which are also part of the web. So examples are LibGen, or Anna's Archive. And these completely disregard copyright and bypass paywalls. So someone has collected a ton of data-- usually books and articles that are all copyrighted, and people have to pay for these-- and have made them available for free. And there have been a lot of takedown orders, lawsuits, and these are blocked, but usually these are circumvented because they just make servers in other countries. And the people doing this argue that this is making free what's freely available, what should have been freed, but generally from a legal perspective, this is piracy and copyright infringement.
[13:45.270] And we'll come back to this point later. So there are a lot of books and papers on these shadow libraries. So the summary so far, the internet is a huge and messy and scary place, and you can't actually get all of it. There are technical restrictions, there's legal restrictions on what data one can access. So now next time someone says, I trained on the internet, you can tell them all these things you learned. So now let's go to the second part of what data can we use, and this has to do with copyright. So suppose you were able to obey the terms of service, and you are really a good citizen, and you obey the rate limits. Now, there's still a question of are you allowed to train on this data? And this is a ongoing question that has not been fully resolved, but there has been a bunch of developments in the last year. So the legal context for thinking about this question falls into intellectual property law, and this is a system that is trying to incentivize the creation of intellectual goods.
[15:11.360] So I think it's-- remember, the spirit of the law is to not just technically say no to everything, but to incentivize the creation of intellectual goods. And there's many forms of intellectual property that are covered. There's copyrights, patents, trademarks and secrets-- sorry, trademarks, and trade secrets. The thing that is most relevant for language model data is copyright law. So copyright law has a fairly long history going back to the 1700s in England. And so 1709 was the first time that governments and courts started actually regulating copyright for this goal of incentivizing innovation and creation. And in the US, more recently, the Copyright Act of 1976 is what I think defines a lot of modern copyright. So copyright protection applies to original works of authorship fixed in any tangible medium of expression, yada, yada, yada. So a few notes. So not everything is copyrightable.
[16:26.820] In particular, collections are not copyrightable. So for example, if you have a telephone directory, unless there is some creativity in how you arrange things, this is not really copyrightable. And furthermore, copyright applies to the expression, not the idea. So you can't copyright an idea, like you can't copyright the quick-sort algorithm. You can copyright the implementation of a quick-sort algorithm, but not the quicksort algorithm. So one thing that happened in 1976 is that the barrier to getting a copyright was relaxed a lot. So more things are copyright. So it used to be that you had to publish something to have something copyright. Now things just have to be fixed. And what that means is that registration of any sort is not required for copyright protection. This is in contrast to patents. Patents you have to go pay a lot of money to get a patent, and you have to file a patent. But the threshold for copyright is very low. For example, you put something on your website, it's copyrighted. That's it. Now if you want to sue someone, you have to go get it registered, but it only costs $65.
[17:50.670] Which is much smaller than the lawyer fees that you'll probably pay, so it's really still a very low bar. So one thing to know about copyright is that it lasts 75 years, and then the copyright expires. And whatever that content gets released into the public domain, and this is now-- everyone can use it freely. So a lot of people who have lived in the past, all their great works are in the public domain, which is great for everyone. So the rationale here is that you're incentivizing innovation. So when an artist or a creator creates something, you want to protect their-- you don't want someone else to just take it, so you protect them for a while. But after 75 years, presumably, it's not worth protecting, or they're passed away, or something. So it doesn't make sense to keep things copyright anymore. So basically, everything on the internet is copyrighted. So you're saying, wait a minute. So that means is everything-- if I train on anything, that's a copyright violation?
[19:04.730] Well, not necessarily. So everything is copyrighted, but you can use copyrighted works. And the way you do it is A, you either get a license for it, or B, you appeal to the fair use clause. So a license from contract law is something that a licensor grants to a licensee. It basically says, don't sue me. You can use this work in the ways that the license permits. So there's a really nice license called Creative Commons, which allows something to act like it was in the public domain. It enables free distribution of copyrighted work. So examples include Wikipedia, Open CourseWare, Khan Academy, all these things. It was created in 2001 to essentially bridge public domain and existing copyright, because otherwise you have to wait 75 years. But what if the creator says, I actually want people to use it.
[20:13.930] They can now have a means, a legal means of saying, I put a Creative Commons license on it. It's going to act like it's in the public domain, and everyone can use it. So, so far, things in the public domain and things that have Creative Commons licenses are permissive. You can use it. But there are many other times where you don't have-- it's not a Creative Commons license, it's not in the public domain, you have to get a license. So if you have money, you can pay someone to give you a license. So there's a bunch of deals between model developers and content, I guess, platforms, that allow the model developers to license the data for training foundation models. OK, so that's the first thing. You can get a license-- either Creative Commons, or you can pay someone for a license. Now if you don't want to pay, you can appeal to fair use, which is a more complicated matter. So this is section 107 of the Copyright Act.
[21:24.600] And there are-- basically fair use says, I can use this anyway, even if I don't have a license, and these four factors determine whether fair use applies. So the first one is the purpose or character of the use. What are you trying to do with it? And none of these are hard rules. They're just tendencies which have to be weighed in court if it comes to that. So, for example, if you're trying to do something for education, it is more likely to be fair use than if you're trying to sell and make money off of it. If you're trying to-- if you transform something, then that's going to be more favored than if you just literally host the identical copy. The nature of the actual copyrighted work is also important. So things that are more factual are more likely to be fair use than fictional works. Because if you write a summary of World War-- a page about facts about World War II, there's less protection over my really creative poem.
[22:35.120] And then the amount of portion of the original work. So using a snippet is favored over the whole work. So if you just take a little bit, that's not as bad as taking everything. So that's fairly straightforward. And the final thing is related to original motivation. It was like, why are you-- why copyright at all is to protect and align the economic incentives. So the effect that what the use has on the market for the original work. So if you take an author's work and you provide an alternative, and that decreases the amount that the original author could monetize their work, that is seen as potentially bad. But if you're transforming it in a way and going to a new market, or doing something else with it, then that's going to be treated more favorably. So here are some examples of fair use. You can watch a movie and write a summary of it. You can re-implement an idea, like an algorithm-- not copy the code.
[23:42.410] And then there was this famous lawsuit between Authors Guild and Google. So if you go to Google Books, you can see snippets of various books which are copyrighted. So there was a huge deal about whether this was deemed fair use, because you are redistributing other copyrighted work, and eventually-- eventually meaning after 11 years-- this lawsuit was settled in favor of Google, which is why this thing exists. And this has set some precedent for thinking about whether language model training is fair use. One thing to note is that copyright is not about verbatim memorization. So this might be, I think, for an ML audience, this might be a little bit new because a lot of papers are focused on verbatim memorization. That's one way you can violate copyright, but not the only one. So for example, plots and characters can be copyrightable. So you can copyright Harry Potter the character, not any particular book, or anything. But there are some exceptions. For example, if you're parodying something-- and so you're creating something that looks like a derivative,
[25:11.390] but as long as you're making fun of it, it's actually more likely to be fair use. So, yeah. So a lot of copyright is about semantics. It's definitely not about N-gram overlap, and also about the economics. So what is the implication for language models? So first off, copying data. Even if you're not training, even the mere fact of copying-- which is in the word copyright-- is potentially a violation already, even if you don't do anything with it. Training a model-- I guess this is not really a-- let's see, how do I take that? This is not necessarily a fact, but intuitively has a transformative flavor. Because it's certainly not-- it certainly seems different than just rehosting another work. It's doing something transformative. And in some sense, the models are really trained on this data as a means to an end. You are trying to extract the general idea, learn about the world and how it works, rather than only just the concrete expression.
[26:34.940] And the other thing is that regardless of copyright, language models can definitely affect the market. So I think this is also-- remember, that's the fourth item for fair use. So by negatively affecting the market, you are more likely to be ruled not fair use. So fair use is a little bit slippery. Suppose you have fair use, or you have a license. Remember that there's still terms of service that allow or prevent you from actually just getting a piece of work. For example, YouTube has all these videos which are licensed, but the terms of service prohibits just downloading the videos using a bot, like scraping, which is another layer. So there's multiple layers of restrictions here. So let me talk a little bit about where things are on the legal landscape. So back in 2023, New York Times filed suit against OpenAI, saying you train on our news articles, and look, here's evidence. We were able to actually prompt ChatGPT to generate a news article almost verbatim.
[27:55.020] I don't think this is still pending. There's a case against Anthropic where the allegation was you pirated millions of books, and you trained on these books to make Claude. Just last year, this was a landmark ruling that said, actually, this particular instance of training is fair use. However, Anthropic, you're still in trouble because you pirated all these books, and that is a no-no. So notice that this has nothing to do with training, just the mere fact of pirating-- I mean, which is not a new thing-- is illegal. Anthropic actually, interestingly, had bought and scanned all the books, which is actually fair use. The Court said you're allowed to scan. You buy a bunch of books, rip off the bindings, scan it, and digitize it for your own use. That's fair use. But that doesn't absolve you of your sin of pirating. So you can't pirate and then buy it and say, actually, never mind what I just did first.
[29:10.030] So the outcome is that Anthropic paid 1.5 billion to settle the authors. So this is about $3,000 a book. So there's also a lawsuit against Meta. The allegation was that you trained on our books-- sorry-- and this was, as we'll see later, actually revealed in the Llama paper. And the judgment which came right after Anthropic was, yes, training is fair use. And then they also torrented some books, so that's still pending. So that's probably-- if precedent holds, this is also not going to be good for Meta. So the summary so far is training, so to speak, has been deemed of fair use, or at least has been not deemed not fair use.
[30:07.950] The rulings have so far been narrow. It's not to say that any training on any copyrighted content is fair use, but in these cases, it's fine. Pirating books is clearly illegal. I guess we already knew that. But this is still a very active and evolving area. Any questions about the sources of data? I think, before we get into the actual datasets, I just want to set the stage with this more general social context. Yeah? [INAUDIBLE] The question is, what do I think of voice data and ElevenLabs? I don't know too much about that, so we can talk offline. Yeah? [INAUDIBLE] --because I guess if I own some asset, it's reasonable for me to be like [INAUDIBLE]. Am I allowed to-- usually it's Creative Commons, but can I update my license later on to block certain usage? And so, what is the role of time playing in this, like I then train on the older version with more a more permissive license if the creator copied it to a more restrictive?
[31:33.918] Yeah, so the question is, what happens with time? If you have one license and then it gets changed, what happens? So first, I'm not a lawyer. But second, so this-- so the first license I believe you're still allowed to train on it. But what happens is that the license usually applies to not just, let's say, a fixed set of documents-- for example, Reddit. There's constantly new content being created, and the license being changed means that you wouldn't be able to train on the later things. OK, so let's look at various sources of data. So let's go back to crawling. So most model developers have their own crawler, because they want to have full control over what the data is. Fortunately for the rest of us, if you don't want to build your own crawler, there's something called Common Crawl, which has been around for quite some time since 2007. And every month they run a web crawl, gets about three to five billion web pages. Each crawl has some overlap with the previous crawl, but they try to diversify and get new pages.
[32:56.360] So there's 300 billion pages so far, which seems-- this was from their website. It seems a little bit big, because if you multiply this number by 20, you don't quite get 300 billion. But that's what they say. So how many URLs are out there? That's really hard to estimate. So the Google search index is at least 100 petabytes, according to them. And, each Common Crawl dump has about 2 billion web pages. So that's about 372tb. And this is not including images, this is just mostly focused on the text. So crawling is conceptually straightforward, but all the gory details are in the implementation. You start with a bunch of URLs, and then you iterate. It's basically graph traversal. You pop a URL from a queue, you download it, and then you look at all the hyperlinks in that page, and then you add them to the queue. And generally this is done in parallel over many machines.
[34:06.410] And there's many decisions here-- which pages to download, you have to respect robots.txt, don't overload the server. Sometimes the websites change, and so you want a policy that will download frequently changed pages, but not spend time downloading pages that don't change. So there's some policy around that. And I mentioned before, URLs are dynamic, so sometimes the same URL leads to different content depending on some state of the browser. But also, many URLs might lead to the same content, so there's a lot of duplication that happens if you're not careful. In particular, mirror sites explicitly is about duplication. So Common Crawl releases their dumps in two formats. One is a WARC file. So this is the raw HTTP response. So remember, HTTP protocol is that you send a get, here's a URL, and then out comes a HTTP response, and the WARC file is that response.
[35:25.170] They also do some processing. They convert to a WET file, which is necessarily a loss-y process. It turns out that this is not necessarily the best way to use the web. The HTML to text, there's many tools for this, including trafilatura and resiliparse, and the way you convert actually does matter. So this is from the DataComp LLM paper, which we'll talk a little bit about. Data is an ablation, where you look at the web files that Common Crawl releases versus these other tools, and it seems like trafilatura and resiliparse are better. So one thing you can do is just rely on general web crawls.
[36:14.300] But often, the web is not a uniform place. It's not like there's a bunch of websites, and you just get some random fraction of them, and that's your dataset. There are specific pockets of really interesting, high quality content, and I'll talk about three of those-- Wikipedia, GitHub, and arXiv. So Wikipedia, all of you know about. It's been around since 2001, and now there's 67 million articles around all these different languages. And Wikipedia is not all content by any means. You can't have original thought in it, because everything has to be referenced and cited. So in some sense you could argue that Wikipedia doesn't contain anything that's not already on the web. Well actually, that's not true, because they can also cite books, which are-- obviously you can't easily get them, so, Wikipedia is so good. And also, there's articles based on notability, so not everyone can get a Wikipedia article about them.
[37:36.020] So anyone on the internet can write this content. This was the radical thing about wikis, and it's still a miracle, I think, to me that this actually works. It just happens that any vandalism gets reverted by administrators, or bots, I guess, these days. There's a small number of Wikipedians-- as any peer production system, there's a small set of people who do most of the work. And for example, this guy has 5 million edits. And one thing about Wikipedia is that every few weeks, they do a periodic dump. So basically they take all of Wikipedia packaged into a nice TAR file, and then you can just download that. So this is important because you don't need to crawl Wikipedia. In fact, they don't want you to crawl Wikipedia. They just want you to download this if you're going to crawl, and this is much better than crawling. Just a fun aside, or good to know, is that there's this paper that says you can actually poison Wikipedia.
[38:43.500] So you might think that this vandalism gets reverted. So if you do-- attacker tries to do something fishy, you'll just get it rolled back. But what Carlini realized is that these periodic dumps happen at a regular cadence. And so, what you can do is you can go in and you can just edit it right before the dump happens. And then the dump happens, and then the edits get rolled back, but the dump still has the malicious content. If you're able to inject things into web pages, that means you can cause-- there's works that show that you can cause the model to, let's say, ascribe negative sentiment to any trigger phrase, like the iPhone. So one takeaway here is that if you consider adversaries, even so-called high quality contents might contain bad things in them. I think this has been fixed since then. OK, so GitHub is a good place for code, and code is important not just for if you want coding capabilities for your language model, but if you want general reasoning.
[40:03.630] So GitHub all of you know. Again, it's a live service for hosting code repositories. Been around since 2008, about the same age as Common Crawl. It has 420 million repositories, 28 million are public. Each repository contains-- it's not a file, as you know, it's a directory with commit history, issues, pull requests, and comments. Code has generally a lot of duplicates, and it's just because either copying code, or forking code. And GitHub has deemed it-- you're allowed to train on any public repository with a permissive license. So MIT or Apache licenses are fine. So remember, there's two types of data when you talk about GitHub.
[40:59.470] There's the repository data, which you can just go and download through the GitHub Git protocol. And again, you shouldn't scrape GitHub, you should just download the repository. And then the metadata associated with each repository-- issues and pull requests, and so on. These are made actually via the GitHub Archive, which gives you hourly snapshots of this event stream, which I think is really interesting data. It's basically every single comment, or star, or action on GitHub that gets recorded. There's this software heritage foundation that is focused on the repository, not the metadata, and they aggregate GitHub.
[41:44.218] GitHub is not the only place where code shows up. There's also smaller entities like GitLab, Bitbucket, and so on. And they aggregate all that, those repositories So arXiv, just very briefly-- everyone knows arXiv. This has been around as a site that allows people to share papers since 1991. Started with physics, and now has a lot of other areas in it. Each of the three million submissions has the metadata, a PDF, and optional LaTeX source. So already you should be thinking, well, what does it mean to train on arXiv? It's not clear, because this is, well, you have to convert the PDF into text, or you can use the LaTeX source, which is a bunch of files.
[42:36.560] It's not peer-reviewed, but there is some approval process, and authors can choose to maintain the rights, or put in the Creative Commons, which-- so everything is actually very clearly licensed. All the metadata is under permissive license so you can use that regardless, and then you can go and you look at all the papers that are Creative Commons license. You download those, and you can use them. And I think most arXiv papers are Creative Commons. And then again, this is available. You don't crawl arXiv, you just go down to bulk download it from some website. So any questions about sources of data? Yeah? Are there any restrictions on usage of data, which is generated by models? Are there restrictions on data? Generated by models. Yeah. I mean, [? synthetic ?] [? data ?] models. Oh, I see. We'll talk about that a little bit later. So the question is, what about model-generated, like synthetic data? Are you allowed to use it? The short answer is probably yes.
[43:49.340] Yeah? When using crawlers like [INAUDIBLE] Crawl and stuff like that, I guess-- because you mentioned earlier that there are websites that have a lot of pirated books on it. How do you ensure that those types of websites don't show up in your crawl? Yeah. So the question is, there's a lot of websites with pirated books. So if you're just doing a web crawl, you can't look at all the websites. So how do you make sure that everything is kosher? And that's part of the difficulty is, you can't. And so, there's-- most likely in Common Crawl, there are copyrighted books and content that you are not supposed to train on, and you can-- will appeal to fair use. Remember, everything is copyright. Books are not remarkably different from your website, for example. both are copyrighted. Probably, a book author has more of a-- if they want to go into court, they'll probably be able to protect that better because it's published than your website, but still.
[44:58.410] And I'll talk a little bit about later. If you really, really want to be careful, you cam do something I'll show later. So what I'm going to do now is to now talk about the various datasets that are actually used in building models, starting all the way back to 2019. So we've talked about the origin of data and source of data, so you have a feeling for just the general data landscape. And so, let's start with BERT. So BERT was from 2018, and they trained-- back in the day, they just trained on Wikipedia and books. So what is this books? So there's this website called Smashwords. It was created that allow anyone to just put an ebook up to publish, and they had about half a million books in 2024. So in 2015, there was this paper who just scraped Smashwords-- sorry, took the books that were free, and just made a corpus.
[46:08.770] This was back in the innocent days when no one was paying attention to any of this. So this books corpus was around for many years in the academic community. Since then, it has been taken down because it violated the terms of service. Just because it was free and you can get it, doesn't mean it was legally allowed. So one thing to that the sequences were documents rather than sentences. All the language modeling research before that was focused on sentences. So in 2019, GPT-2 came around. And so what did they do? So they wanted to get high quality web content. And at that time, people knew about Common Crawl, but somehow the Common Crawl data was too messy. So what they did was came up with this clever idea where you can take-- you look at pages that are outgoing links from Reddit posts with greater than three karma. So a good post must link to good websites. So you grab those pages, and you get a 40 gigabytes of text, 8 million pages.
[47:34.100] They never released this data, but there was an open replication of webtext, which people used quite a bit. So that's GPT-2. So as you're going through this, I guess think about both-- there's many methods of filtering and collecting data, and it's interesting to look at these design choices, and. So CCNET was developed by, at that time, Facebook. So the goal was to create large high quality data sets for pre-training. And they were especially interested in low resource languages. So they didn't want some very manual process that only worked for English. So they did a bunch of things. They did deduplication, and language identification to train a classifier to detect a language, and keep only the language that you're looking for. And for quality filtering, the idea that they had was they wanted to keep documents that look like Wikipedia.
[48:50.620] So Wikipedia at the time was deemed good, high quality content, and they used a language model that was trained on Wikipedia, and you score a probability of a new document under that language model to gauge how Wikipedia-like it was. So rather than outgoing links to Reddit, they use a language model on Wikipedia. And they show that because you can get much more data, you can actually outperform just training on Wikipedia. And this was a tool that will later appear in some other papers. So here is another work. This is from Google in 2019. So this is called C4. So this paper as actually more famous for the T5 model, which pushes the idea that you should just treat all the NLP tasks in a world as text-to-text. But actually, a big contribution of this paper, which is a very long report, was the C4 dataset. And the observation, which was very clear at that time, which is that Common Crawl is mostly not useful for natural language.
[50:08.230] There was this idea that you have these small datasets, and they were high quality, or you had Common Crawl, which was a mess. And any attempt to train on Common Crawl just led to junk results. So people were thinking like, how do I get a larger but high quality subset of Common Crawl? So we saw the Reddit idea, we saw the Wikipedia-like language model idea. C4 used a different idea, and this is saying, let's just define a bunch of rules, which turned out to be fairly effective. So basically, they keep lines that end in punctuation, that have more than five words, remove pages with fewer than three sentences, remove pages that contain any bad words.
[50:57.510] Remove things like terms of use, boilerplate. Interestingly, remove the curly brace, which filters out a lot of code-- so clearly at that time, they weren't thinking about code models-- and they filtered out anything that was not English. The end result is that you have 156 billion tokens, so 800 gigabytes of text. So this is much larger than the GPT-2 dataset, which was only 40 gigabytes of text. A bit later there was some analysis done in C4 which showed-- here's the websites that were represented. So you have a bunch of Wikipedia, patents was a very common thing. They also released a WebText-like document. They also use this idea of links in Reddit posts-- from Reddit posts with greater than three karma, and filtered to those pages. So they were able to get, using this method, 17gb of text using 12 Common Crawl dumps. Remember, webtext was 40 gigabytes, which means that Common Crawl is maybe incomplete, or doesn't have everything, because if it had everything,
[52:25.160] you should be able to hit 40. And they showed that at that time, you could use this data to improve on a bunch of NLP benchmarks. So we've seen using rules, things that look like Wikipedia, Reddit links, and now let's talk about GPT-3. So GPT-3, the dataset was-- they used Common Crawl, they did their own processing. There is this WebText, which was essentially the GPT-2 dataset, but expanded. Notice that there's some overlap. Common Crawl does overlap with this, but this is, in some sense, a more targeted distribution. There is this-- in the paper, it's described as Books1 and Books2, which is internet-based books corpora, so this remains a mystery exactly what it is, and Wikipedia.
[53:24.560] So the result was about 500 gigabytes of text, 400 billion tokens. And to do the Common Crawl processing, they trained a quality classifier to distinguish things they deemed to be high quality from the rest, and they did some fuzzy deduplication because WebText and Common Crawl had dupes. So this paper followed the idea of using a classifier to do quality classification. So after GPT-3 came out, this was, I think, a big event. And there were a number of efforts to try to do things in the open. So one of the initial efforts was The Pile, which is a grassroots effort from EleutherAI, where they had a bunch of people in this Discord just jamming and figuring out high quality sources. And they came up with this list in the end. And today, this is still pretty interesting and diverse. It has Common Crawl, it has PubMed. This has this thing called Books3, which we'll talk about. ArXiv, GitHub, which we talked about.
[54:51.450] Wikipedia, IRC, another BooksCorpus, Philosophy Papers, and so on. They included this Enron Emails dataset. So Enron went bust in 2002 and all the emails were released, and this is one of the few email datasets we have, which is a weird distribution, I think, for email. But that's what you get. So Project Gutenberg was-- so let's talk about books a little bit. So project Gutenberg was started in 1971, and it only includes books that received copyright clearance. So mostly books in the public domain. There's this packaging of Gutenberg into PG-19, which is Gutenberg before 2019.
[55:54.290] Which is, I guess most of them, because for them to be in public domain, you have to wait 75 years. So Books3, which is this interesting dataset that appeared in The Pile. So this is described actually as 200k books from a shadow library called Bibliotik that concludes all books from your favorite authors. And at that time-- again, 2020, no one was paying attention. People used it to train models, and since then, it has been taken down. So you cannot use, or should not use Books3 anymore. We'll come back to Books3 again. So Stack Exchange you've probably used quite a bit, although I guess these days maybe less so because of AI. There's a bunch of user-contributed questions and answers, started in 2008. And one thing that is interesting about this dataset is that it's a Q&A format, and this is quite close to a real application. So normally, you think about pre-training data as here's Wikipedia articles, just like raw text.
[57:23.250] But some parts of the web are actually-- look supervised, like what you would want to ask your language model. Which I think, maybe, helps the model learn certain types of question answering behaviors. So not everything is super magically emergent. It's like, well, this type of data does exist on the web. So there's also-- this data has metadata, which is like the number of votes which are useful for filtering. And again, this data is released in data dumps that allows you to just download without crawling it. So now we're in 2021. There was another paper from DeepMind. They trained a model called Gopher, which was never released, and actually was subsumed by Chinchilla. But the description of the data, I think, is actually really good. You should read this paper, because I think it's very thorough in the way that they describe the data processing.
[58:29.660] Well, except for the parts where they don't tell you what's in the data. So they created this massive web, and they had C4, which remember, was the dataset that we looked at before from Google. They train on books, news, GitHub, and Wikipedia. They don't say anything about how they got that. So for the MassiveWeb, they kept English, deduped, and their quality filtering was also based on manual rules. And part of the reason for this is that they had more control over that. So I think, as you can see, there is this division between people who wanted to use rules, and people who wanted to use classifiers. So they got, as the name suggests, a massive amount of text, but their model was only trained on a very small fraction of this dataset. So in 2022 Llama 1 came out, and the dataset here-- this was actually a very nice paper in that it detailed their data processing.
[59:54.970] So this is probably one of the last non-open, fully open models that actually talk about their data processing. So they have Common Crawl processed with CCNET, which you know about. The classification, though, was whether the page was a reference of Wikipedia, not whether it is actually a Wikipedia article. So the idea is that well, maybe Wikipedia articles are too stylized. And we know that Wikipedia articles references a bunch of other articles which are presumably good, so let's call those the good websites. They had C4, they did a bunch of processing on GitHub to keep permissive licenses, Wikipedia. They trained on Books3 and project Gutenberg. So Books3 really got them in a lot of trouble because they're just announcing to the world that hey, I trained on this dataset. And if you look back, oh, this came from-- where did it come from? It came from The Pile.
[01:00:58.130] Oh, it came from a shadow library. So that's why people don't want to talk about their data anymore. So they looked at arXiv, and actually used the LaTeX to do the processing, Stack Exchange, and they got 1.2 trillion tokens. They didn't actually release this dataset, but they had enough of a description in their processing that this was reproduced by Together's RedPajama V1 dataset, which was then used in other sources. So as you can see here, RedPajama V1 initially also contained Books3, which has now been stripped out. So various decisions made early on about copyright actually influence-- have a fairly big watershed. So let's talk about RefinedWeb.
[01:01:55.890] So this is a paper that tried to make a point that web data is all you need. So the web, in some sense, is everything. And then we talked about all these specialized sources, like there's GitHub and arXiv, and Stack Exchange. And they said, well, what happens if we just stick with the web? So they did a good job of doing the transformation from HTML to text. They filtered using the Gopher rules, which is basically keep things that look like English. They explicitly, at that time, still said avoid ML-based filtering to avoid biases. I don't want to find an overly narrow subset of the web.
[01:02:43.640] They did some deduplication, and they had five trillion tokens. They released about 600 billion of them. So FineWeb from Hugging Face was a replication of RefinedWeb, but improved it. So they took all the Common Crawl dumps at that time, did some filtering-- again, using manual rules to not inject biases-- deduped, did PII removal, and got 15 trillion tokens. So you can see that the size of these datasets is growing quite a bit. So then there is this Dolma dataset from AI2, and this is a dataset that includes their own processing of Common Crawl, The Stack, which we'll talk about, before C4, which you know about, and a bunch of other things.
[01:03:44.640] And the Reddit dataset was from this project called PushShift. And at that time, you could still get this data, and you can train on it before things got locked down. AI2 has their own crawl of academic papers called Semantic Scholar, so they derived a dataset based on that. So let's look at their Common Crawl processing. So they use language identification. So this was model-based, but quality filtering still avoid model-based filtering. And then they removed toxicity using rules and a classifier. And they got three trillion tokens out of that. So DCLM, I think, was maybe a point where this idea of model-based quality filtering started to really become the norm here. So DataComp, the initial motivation was to define some sort of pipeline so that people can try different data methods in a standard way, and show results.
[01:05:04.240] But I think the main way that I think people use this is just using their dataset that they released to train models and do other things. So they processed Common Crawl to produce DCLM-pool. So this is completely unfiltered. And if you look at this dataset, this is massive. 240 trillion tokens. This is probably more than the number of tokens that anyone really trains on. But a lot of this is fairly low quality, so then they had a pipeline where all that gets filtered. If you only keep English, you have some of rules to really narrow it down. You do dedupe, and then you do this model-based filtering, and you end up with something that's 1.4%. And this is the data set that we'll see actually works pretty well. So the model-based filtering-- and this was strangely good. So to train a classifier, they took OpenHermes, which is essentially instruction data that was generated by GPT-4, and ELI5,
[01:06:19.470] which is a subreddit with various questions and answers. So this is-- let's see. These are the type of questions that are in ELI5. So anyway, you get the idea. So it's kind of weird, but somehow this works. So the negative examples are anything from RefinedWeb, which is, remember, it's just a very loosely filtered version of the web. It's basically the web, and it got 3.8 trillion tokens. They train a fastText classifier-- think about it just as a linear classifier. And then they show that this magical classifier, quality classifier, outperforms a bunch of other things that they tried. So, DCLM. This became, for a while, at least in the open community, a bit of a gold standard for quality filtering. Let's move on to Nemotron. So Nemotron-- this is from NVIDIA-- they said that well, DCLM filters too aggressively. It's removing most of the data.
[01:07:39.250] And remember, they only got-- how many-- 3.8 trillion tokens. So we need more tokens. So what do we do? We are going to do something more, I guess, elaborate. We're going to prompt our existing model to score FineWeb documents based on educational value. So we're going to prompt a language model and say, is this educational or not? Go add a bunch of labels, train a fastText model. So that's one classifier. Another classifier is just using the DCLM classifier. So we're going to use those classifiers. We're also going to use synthetic data. So this is probably one of the-- I don't know if-- it's probably not the first, but, one of the main datasets that really lean into synthetic data for pre-training. So they had, for low quality data, as deemed by these classifiers, you use a language model to rephrase it to make it more-- look like Wikipedia. For high quality data, we use a language model to generate various tasks. For example, given a Wikipedia article, I can generate question answers, or I can generate like, please summarize this document,
[01:09:04.740] extract key information from it, and so on. So they resulted in a dataset that was 6 trillion tokens. So this is substantially larger than DCLM. And just for reference, this is still kind of small by some consideration. So Llama 3 was trained on 15 trillion tokens, Qwen3 was trained on 36 trillion. Although there is this-- it's not clear how big these unique tokens are, because when you look at [INAUDIBLE] counts in language modeling papers, some of it is repeated. Like if you do two epochs, that's twice the number of tokens, so you have to be careful when you look at those numbers. And they show that the dataset is better. Their high quality subset beats the previous datasets So up until now, we've seen a bunch of different methods for filtering. I think all of these look very similar at some level. You take a web crawl, you either decide I'm going to use rules to filter, or I'm going to use a model.
[01:10:25.690] If I'm going to use a model, then I have to decide what looks like good data, and then try to train a classifier, and classify all my documents, and select. And there seems to be a trade-off between you can take a lot of Common Crawl-- you can get 240 trillion tokens if you want, but that's probably going to be really low quality. Or you can get one trillion tokens, and there's some sweet spot in-between. Let me talk about two final things. So code, and going back to this question of licensing. So The Stack is a very nice project that was trying to make a really good coding data set, since it was clear by 2022 that coding was going to be really important.
[01:11:19.550] So what they did in the initial version is that they clone 137 repos. They kept the ones that were permissively licensed, remove near-duplicates, and resulted in one-- sorry, three terabytes of code. And then in 2024, they had an update. And here they took more the metadata, like the issues, comments, and PRs from GitHub, the repositories from the Software Heritage, which we talked about. They also scraped documentation from various websites by crawling them. They did a bunch of processing. So in GitHub repos, you have binary files which you probably don't want to train on. They have malware, you get rid of that. A lot of GitHub, especially these PRs, are bots, so you have to filter that.
[01:12:14.270] Dedupe, do PII redaction. There's a lot pull requests and stuff, and they basically subsample to keep it representative and the dataset manageable. They also did this thing, which was I thought was nice, which is that there's many programming languages out there. There's a lot of Python, a lot of C, but low resource languages like Nim-- which I hadn't even heard of-- they're not that common. So what they do is they compile this code into a low-level intermediate language, LLVM, which everything, like C Compiler, can compile into this, and they have juxtaposed the low resource language and the intermediate representation. So then the language model can actually learn the mapping between the shared low level representation, which has a lot of data on, and the thing that it does has less data on. And just for fun, you add in all the other good stuff that you can. So Stack V2 is mostly code, and then plus other things that were used to train their coding models.
[01:13:32.960] And one thing about pull requests is that pull requests and all the metadata is not, by its nature, a linearized sequence. So there has to be some steps taken to linearize it. So one important thing you have to decide is how much context to provide. So for example, an event might just be, I change one line of code. And that could just be one line, but presumably to learn that, you want to provide some context, maybe a few lines around it, or maybe the entire file surrounding that diff. And these are some design decisions that you have to make for linearization. And so at the end of the day, the tokens that they train on look something like this, where it's this XML-like structured data, where you have the PR, and then you have a bunch of diffs. And then for comments, you have basically the events of a comment was posted, what the review state of that PR was and so on and so forth. So it's not just learning how to generate code, but also the software development process around code.
[01:14:44.800] So finally, I'll talk about the common pile. So recall that almost all the data on the internet is copyrighted. Some of it is permissively licensed, some of it is even in the public domain. And while you can appeal to fair use and say, I'm going to train on it anyway, this is not quite settled. So if you're very, very risk-averse, then you say, well, if I don't know whether it is OK, that's a no. If you take that attitude, then that's what common pile did. So can you train a good model with only permissively licensed data? Or rather, how far can you get? So this was a project that went and scoured the internet for all the different types of data that they could possibly find that was worth training on, that was permissively licensed. It includes Stack V2 in code. It turns out that a lot of government proceedings are actually permissively licensed, wikis, some things on the web-- some new sites are actually
[01:16:08.690] permissively licensed, academic papers, online forums, things that are in the public domain, educational resources, and so on. So in the end, there were 8 terabytes of data, which is actually pretty good for permissively licensed data. Doing this project is actually much harder than maybe it sounds at first glance. It's not just as simple oh, you look at the license and it's like, oh yep, Apache, or CC, good. There's some subtleties, versus license laundering. So people are sloppy with licenses. So people might take some copyright work and just slap a CC BY on it. Anyone can write this on the internet, and it's hard to tell whether this is real or not. There's also this common practice where, for example, Dolma is permissively licensed. Remember, Dolma is the AI2 collection. But collection licenses don't extend to the individual works. So the collection is-- I mean, first of all, there's a question of whether collections-- I think collections can be copyrighted, but I guess you can put a license on anything.
[01:17:34.650] But the individual works are not necessarily good. So you can't just look at data sets on Hugging Face, and you-- many data sets on Hugging Face you see, they have a permissive license. But if you dig deeper, it's actually not permissively licensed at the individual level. And they also made a decision to forego training on any synthetic data, because training language models on unlicensed data is unclear. So probably it's fine because technically, these are open weight models with MIT license. You can do whatever. You can use them however you want, but these language models were presumably also trained on unlicensed data. So it's a little bit of data laundering if you are really honest here. So with this data, they compare it with a bunch of other models like the first Llama, MPT, and Qwen. And on a bunch of benchmarks, they show that this is not bad. I would say it's not as good as the Qwen models for sure, but it's certainly outperforming the very old-- this is from 2023, I guess.
[01:19:09.100] So quite old models. So I would say, the conclusion here is that you can do reasonably, but I think it's still pretty tough to compete without doing more tokens. But I don't think this is the final word on it, and I think you can probably eke more out of publicly permissive licenses if you try. OK, so to summarize this lecture here. So maybe hopefully now I've given you some appreciation that data is a very rich topic. It's not that data just falls from the sky, or you just go on Hugging Face and you download a dataset. Data has to come from somewhere. And there's also a technical but also social context in which data comes, because at the top, the internet is a bunch of live services. Someone has to do the work of producing raw data, whether it be the website that gives you dumps, or someone has to build a crawler, or something.
[01:20:11.580] And then, someone has to make the decision of how to process it into usable form-- filtering, transformation, deduplication, all these things have an impact on the quality of your final language model. And notice that filtering is probably one of the most important things. How do you go from 200 trillion tokens to less than three trillion tokens? I think that obviously is a huge reduction, which I think merits a lot of attention. Data is, in some sense, a key ingredient that differentiates language models. A lot of language models are roughly the same kind of transformer architecture, but data is depending on how you process it. It can make a pretty big difference. There's plenty of legal and ethical issues around data, more than I can get into this lecture. And also, this process is very messy. Unlike some of the other parts of this class where things are maybe more based on first principles, data processing is, right now at least, a lot just based on vibes you set up.
[01:21:30.150] You define this classifier, you define this rule, you set some threshold. So there are many opportunities to improve. So as you do your assignment four, maybe think about whether there's better ways to do this, and this could be maybe a research direction. OK, so that's it for today. So next time I'll continue talking about data. I'll talk a bit about post-training data, and a bit more about filtering.