llm_learn

外观

L00 · 课程参考资料 ​

本页按 CS336 2026 的课程顺序汇总各讲涉及的讲义、视频、代码与延伸阅读。每讲页面仍保留与本讲直接相关的资料入口。

资料索引

按课程顺序整理讲义、视频、代码与延伸阅读；每一项都保留在对应讲次下。

159 条资料 · 18 讲 · 160 次关联

01

Overview, tokenization

8 条资料

参考资料lecture_01.py延伸阅读论文Neural Language Models are Effective Priors for Scene Text Recognition课程引用论文SentencePiece - A simple and language independent subword tokenizer推荐阅读代码OpenAI tiktoken实践工具论文Neural Machine Translation of Rare Words with Subword UnitsBPE 经典论文论文Language Models are Few-Shot Learners模型规模背景文档UTF-8 Standard文本编码视频课程视频课程视频
02

PyTorch (einops), resource accounting (FLOPs, memory, arithmetic intensity)

10 条资料

参考资料lecture_02.py延伸阅读参考资料recording version延伸阅读论文Mixed Precision Training课程引用书籍 / 教程JAX Scaling Book - Roofline analysis推荐教程技术文章Transformer FLOPs计算参考论文FP8 Formats for Deep Learning低精度训练文档Einops Basics张量表达技术文章Transformer Inference Arithmetic资源核算文档Model Training Anatomy - Memory Usage显存核算视频课程视频课程视频
03

Architectures, hyperparameters

9 条资料

讲义 / Slideslecture 3.pdf延伸阅读论文Attention Is All You Need基础论文论文GLU Variants Improve Transformer课程引用论文RoFormer - Enhanced Transformer with Rotary Position Embedding课程引用论文Root Mean Square Layer NormalizationRMSNorm论文Layer Normalization基础归一化论文Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerT5论文LLaMA - Open and Efficient Foundation Language Models现代架构案例视频课程视频课程视频
04

Attention alternatives and mixture of experts

9 条资料

讲义 / Slideslecture 4.pdf延伸阅读论文Transformers are SSMs - Generalized Models and Efficient Algorithms Through Structured State Space DualityMamba-2论文Switch Transformers - Scaling to Trillion Parameter Models with Simple and Efficient SparsityMoE 基础论文DeepSeekMoE - Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models课程案例论文Mamba - Linear-Time Sequence Modeling with Selective State Spaces状态空间模型论文ST-MoE - Designing Stable and Transferable Sparse Expert ModelsMoE 稳定性论文DeepSeek-V3 Technical Report现代 MoE论文Outrageously Large Neural Networks - The Sparsely-Gated Mixture-of-Experts LayerMoE 经典论文视频课程视频课程视频
05

GPUs, TPUs

9 条资料

讲义 / Slideslecture 5.pdf延伸阅读论文FlashAttention - Fast and Memory-Efficient Exact Attention with IO-Awareness核心论文书籍 / 教程JAX Scaling Book - How to Think About GPUs推荐教程技术文章Making Deep Learning Go Brrrr From First Principles推荐阅读论文FlashAttention-2 - Faster Attention with Better Parallelism and Work Partitioning算法演进文档NVIDIA Matrix Multiplication Background User's Guide硬件文档文档CUDA C++ Programming GuideCUDA 基础技术文章What Shapes Do Matrix Multiplications Like?性能直觉视频课程视频课程视频
06

Kernels, Triton

9 条资料

参考资料lecture_06.py延伸阅读论文Triton - An Intermediate Language and Compiler for Tiled Neural Network Computations核心论文文档Triton Fused Softmax Tutorial动手实践文档PyTorch Profiler Recipe性能分析文档Triton Matrix Multiplication Tutorial动手实践文档PyTorch Benchmark Recipe基准测试文档CUDA Parallel Thread Execution ISA执行模型技术文章Introducing Triton - Open-Source GPU Programming for Neural Networks项目介绍视频课程视频课程视频
07

Parallelism

9 条资料

参考资料lecture_07.py延伸阅读论文Megatron-LM - Training Multi-Billion Parameter Language Models Using Model Parallelism并行训练基础文档PyTorch Distributed Overview实践文档技术文章Levanter - Legible, Scalable, Reproducible Foundation Models with JAX课程引用论文Horovod - Fast and Easy Distributed Deep Learning in TensorFlowRing AllReduce文档NVIDIA Collective Communications Library Documentation集合通信文档NCCL Tests Performance Guide通信测量论文Demystifying NCCL - An In-depth Analysis of GPU Communication Protocols and Algorithms延伸阅读视频课程视频课程视频
08

Parallelism

9 条资料

讲义 / Slideslecture_08.pdf延伸阅读论文ZeRO - Memory Optimizations Toward Training Trillion Parameter Models参数分片论文Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM3D 并行论文Reducing Activation Recomputation in Large Transformer Models序列并行论文GPipe - Efficient Training of Giant Neural Networks using Pipeline Parallelism流水线并行论文PyTorch FSDP - Experiences on Scaling Fully Sharded Data Parallel参数分片实践论文DeepSpeed-Megatron - System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters系统案例文档Megatron Core Developer Guide工程文档视频课程视频课程视频
09

Scaling laws

9 条资料

讲义 / Slideslecture_09.pdf延伸阅读论文Scaling Laws for Neural Language ModelsKaplan Scaling Laws论文Training Compute-Optimal Large Language ModelsChinchilla论文Chinchilla Scaling - A Replication Attempt复核阅读论文Deep Learning Scaling is Predictable, Empirically早期 Scaling Laws论文A Constructive Prediction of the Generalization Error Across Scales联合缩放论文Scaling Laws for Autoregressive Generative Modeling跨模态缩放论文Scaling Data-Constrained Language Models数据受限场景视频课程视频课程视频
10

Inference

9 条资料

参考资料lecture_10.py延伸阅读论文Efficient Memory Management for Large Language Model Serving with PagedAttentionvLLM 核心论文论文Fast Inference from Transformers via Speculative Decoding推测解码书籍 / 教程JAX Scaling Book - Inference推荐教程论文SGLang - Efficient Execution of Structured Language Model Programs推理系统论文GPTQ - Accurate Post-Training Quantization for Generative Pre-trained Transformers权重量化论文SmoothQuant - Accurate and Efficient Post-Training Quantization for Large Language Models激活量化文档vLLM Documentation工程文档视频课程视频课程视频
11

Scaling laws

9 条资料

讲义 / Slideslecture_11.pdf延伸阅读论文MiniCPM - Unveiling the Potential of Small Language Models with Scalable Training Strategies缩放配方论文DeepSeek LLM - Scaling Open-Source Language Models with Longtermism课程案例论文Tensor Programs V - Tuning Large Neural Networks via Zero-Shot Hyperparameter TransferμP论文The Llama 3 Herd of Models缩放案例论文Cerebras-GPT - Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale ClusterChinchilla 配方论文DeepSeek-V2 - A Strong, Economical, and Efficient Mixture-of-Experts Language Model稀疏缩放书籍 / 教程Scaling Book - Training at Scale推荐教程视频课程视频课程视频
12

Evaluation

10 条资料

参考资料lecture_12.py延伸阅读论文Holistic Evaluation of Language ModelsHELM论文Chatbot Arena - An Open Platform for Evaluating LLMs by Human Preference偏好评测论文SWE-bench - Can Language Models Resolve Real-World GitHub Issues?Agent 评测论文Measuring Massive Multitask Language UnderstandingMMLU论文GPQA - A Graduate-Level Google-Proof Q&A Benchmark高难知识评测论文HellaSwag - Can a Machine Really Finish Your Sentence?常识推理论文HarmBench - A Standardized Evaluation Framework for Automated Red Teaming安全评测论文AlpacaEval 2.0自动偏好评测视频课程视频课程视频
13

Data (sources, datasets)

10 条资料

参考资料lecture_13.py延伸阅读论文The Pile - An 800GB Dataset of Diverse Text for Language Modeling数据集论文论文RefinedWeb - The Falcon LLM Dataset网页数据文档Common Crawl数据来源论文The FineWeb Datasets - Decanting the Web for the Finest Text Data at Scale网页数据集论文Dolma - An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research开放语料论文ROOTS - Search Tooling for Large Scale Archives数据治理文档Software Heritage代码语料来源文档arXiv Bulk Data Access论文语料来源视频课程视频课程视频
14

Data (filtering, deduplication, mixing, synthetic data)

10 条资料

参考资料lecture_14.py延伸阅读论文Deduplicating Training Data Makes Language Models Better数据去重论文DataComp-LM - In Search of the Next Generation of Language Model Pretraining Datasets数据筛选论文FineWeb-Edu教育数据过滤论文Gopher - Scaling Language Models, Methods, Analysis and Insights from Training质量过滤论文DoReMi - Optimizing Data Mixtures Speeds Up Language Model Pretraining数据混合论文RegMix - Data Mixture as Regression for Language Model Pre-training数据混合论文Textbooks Are All You Need合成数据文档fastText Language Identification语言过滤视频课程视频课程视频
15

Mid/post-training (SFT/RLHF)

10 条资料

讲义 / Slideslecture_15.pdf延伸阅读论文Training Language Models to Follow Instructions with Human FeedbackInstructGPT论文Direct Preference Optimization - Your Language Model is Secretly a Reward ModelDPO论文Constitutional AI - Harmlessness from AI FeedbackAI Feedback论文Learning to Summarize with Human Feedback早期 RLHF论文Training a Helpful and Harmless Assistant with Reinforcement Learning from Human FeedbackHH-RLHF论文Secrets of RLHF in Large Language Models Part I - PPOPPO 实践论文UltraFeedback - Boosting Language Models with Scaled AI FeedbackAI 偏好数据论文AlpacaFarm - A Simulation Framework for Methods that Learn from Human FeedbackRLHF 工程视频课程视频课程视频
16

Post-training - RLVR

10 条资料

讲义 / Slideslecture_16.pdf延伸阅读论文Proximal Policy Optimization AlgorithmsPPO论文DeepSeekMath - Pushing the Limits of Mathematical Reasoning in Open Language ModelsGRPO论文DeepSeek-R1 - Incentivizing Reasoning Capability in LLMs via Reinforcement LearningRLVR 案例论文Kimi k1.5 - Scaling Reinforcement Learning with LLMs推理强化学习论文Back to Basics - Revisiting REINFORCE Style Optimization for Learning from Human FeedbackRLOO论文Understanding R1-Zero-Like Training - A Critical PerspectiveRLVR 分析文档OpenAI Spinning Up - PPO算法教程代码AlpacaFarm PPO Trainer课程代码视频课程视频课程视频
17

Alignment - multimodality

10 条资料

参考资料lecture_17.py延伸阅读论文Learning Transferable Visual Models From Natural Language SupervisionCLIP论文Flamingo - a Visual Language Model for Few-Shot Learning多模态架构论文Visual Instruction TuningLLaVA论文An Image is Worth 16x16 Words - Transformers for Image Recognition at ScaleVision Transformer论文BLIP-2 - Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models模态连接论文PaLI - A Jointly-Scaled Multilingual Language-Image Model多模态缩放代码LLaVA Project开源实现代码OpenAI CLIP开源实现视频课程视频课程视频
19

Guest lecture: Dan Fu

1 条资料

视频课程视频课程视频

资料来源归原作者与课程方所有 · 笔记内容持续校订

llm_learn