CS336 2026 · Lecture 17已发布

Alignment - multimodality

日期: Wed May 27
讲师: Percy
课程资料: 10 项
内容产物: 4 项

课程简介

本讲研究如何把预训练语言模型扩展为视觉语言模型：先用 CLIP 式对比学习对齐图像与文本表示，再比较冻结编码器、投影层、交叉注意力和端到端训练等连接方案。课程结合 Flamingo、LLaVA 等模型讨论多模态指令数据、分辨率、视觉 Token 与语言能力保持之间的取舍。

课程资料lecture_17.py打开链接↗论文Learning Transferable Visual Models From Natural Language SupervisionCLIP↗论文Flamingo - a Visual Language Model for Few-Shot Learning多模态架构↗论文Visual Instruction TuningLLaVA↗论文An Image is Worth 16x16 Words - Transformers for Image Recognition at ScaleVision Transformer↗论文BLIP-2 - Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models模态连接↗论文PaLI - A Jointly-Scaled Multilingual Language-Image Model多模态缩放↗代码LLaVA Project开源实现↗代码OpenAI CLIP开源实现↗视频课程视频课程视频↗

CS336 Lecture 17：多模态模型（Alignment - Multimodality）

学习目标

完成本讲后，你应该能够：

理解为什么需要将非文本模态（图像、视频、音频）转换为 Transformer 可以处理的 token 形式。
掌握 CLIP 和 SigLIP 的核心思想、目标函数及其优缺点。
理解视觉语言模型（VLM）的基本架构：视觉编码器 + 投影器 + 语言模型。
了解 LLaVA 和 Qwen 系列模型在架构、数据处理和训练策略上的演进。
理解 Chameleon 将一切映射为离散 token 的方法及其权衡。
认识到多模态模型在训练稳定性、数据加权和长上下文处理上的关键挑战。

课程主线

本讲从语言模型出发，逐步引入多模态建模的完整图景。主线可以分为四个层次：

编码非文本模态：首先解决“如何将图像输入 Transformer”的问题，核心是 CLIP 和 SigLIP。
将视觉表示注入语言模型：在获得图像编码后，如何把它们与 LLM 结合，核心是 LLaVA 和 Qwen 系列。
统一模态的离散 token 方法：Chameleon 将图像也表示为离散 token，从而用纯语言模型的方式处理。
多模态建模的关键挑战与当前共识：总结主流设计选择及其原因。

第一部分：背景与核心问题

为什么要做多模态

世界本质上是多模态的：文本、图像、音频、视频并存。
最终目标是构建 全模态模型（omni model） ：能够接受任意模态组合的输入，并输出任意模态组合的结果。
例如：输入一张图片和一段视频，就它们提问；将音频转换为图像等。

为什么用 Transformer

尽管尝试过其他架构，但大量实践证明 Transformer 在大规模多模态任务上仍然是最优选择。
Transformer 以 token 进行交互，token 可以理解为一个语义信息单元。对于文本，token 是子词（subword）——这已经是我们在词元化一讲中解决的问题。
问题是：像素本身没有语义。因此必须解决两个问题：
1. 如何输入非文本数据（理解）？
2. 如何输出非文本数据（生成）？

本讲主要关注第一个问题。

第二部分：CLIP（Contrastive Language-Image Pre-Training）

背景与动机

在 CLIP 出现之前，计算机视觉模型通常在人工标注的数据集（如 ImageNet）上训练。
OpenAI 研究者的想法是：能否利用互联网上大量的（图像，描述文本）对，像语言模型利用海量文本那样，从嘈杂的数据中学习？
CLIP 是这一思路的代表作，也是现代视觉语言模型的基础。

核心方法

给定一批 N 个（图像，文本）对（例如 32768 个）：

用 图像编码器 将每张图像编码为向量（I₁, …, Iₙ）。
用 文本编码器 将每个文本编码为向量（T₁, …, Tₙ）。
目标是：对于图像 I₁，它与对应文本 T₁ 的点积应远大于它与所有其他文本的点积；对于文本 T₁ 同理。这本质上构成了 2N 个 softmax 分类问题。

实现方式：

图像编码矩阵为 n × d，文本编码矩阵为 n × d。
归一化后，用温度参数缩放的点积计算相似度矩阵。
计算交叉熵损失。

数据

论文使用了 4 亿个（图像，文本）对，从网上搜索挖掘。
原始数据集未发布。OpenCLIP 使用 LAION-5B（50 亿对）数据复现，并用于数据过滤（CLIP 本身被用于过滤数据，存在 bootstrap 过程）。
数据预处理：图像是任意分辨率，神经网络需要固定尺寸。做法是将较短边调整到 336 像素，然后中心裁剪为 336×336 正方形。这是一个简化处理，对分类任务影响不大，但对 OCR 等任务会损坏信息。

视觉编码器：Vision Transformer（ViT）

CLIP 论文实验了 ResNet 和 ViT，发现 ViT 效果最好。
ViT 的做法：将图像切成小块（patch），如 14×14（CLIP）或 16×16（原始 ViT），每个 patch 作为 token，加位置嵌入，送入标准 Transformer。
CLIP 中不用简单的平均池化，而是用 attention pooling：先对所有激活做全局平均，再用该向量作为 query，对所有位置的 key 和 value 做注意力，得到更有信息量的向量。
最佳模型为 ViT-L/14@336px（L = large，patch 14×14，336×336 分辨率）。训练时先在低分辨率训练，后半段提高分辨率。
位置嵌入：论文中尝试过 2D 变体，发现对分类任务影响不大。

文本编码器

标准 GPT-2 风格 Transformer（63M 参数，12 层）。
输入加 BOS，追加 EOS，取最高层 EOS 的激活作为整段文本的表示。

亮点结果与消融

零样本 ImageNet 超越监督模型：零样本 CLIP 在 ImageNet 上超过了在 120 万张标注图像上训练的 ResNet-50。
零样本预测方法：拿图像和候选标签文本做点积，取最大值。
消融实验：尝试从图像直接预测文本（词袋或语言模型目标），发现与 CLIP 的对比排序目标相比计算效率低得多——对精确 token 序列建模对学习语义表示并不重要。

局限

需要大批量（如 30000），否则损失退化。
softmax 在整个 batch 上进行，不可分解，并行化困难。
捕捉的是高层语义，不够细粒度。

第三部分：SigLIP（Sigmoid Loss for Language Image Pre-Training）

核心区别

CLIP 是多类分类（正确配对 vs 所有其他配对）。
SigLIP 只是二元分类：判断给定的（图像，文本）对是否匹配。对角线元素为正样本，非对角线为负样本。

损失函数

log_sigmoid(点积 × 标签)

标签为：对角线元素 = 1，非对角线元素 = -1。目标函数非常简单。

数据

使用 WebLI 数据集（数十亿个图像-文本对），来自网页抓取。
对含文字的图像应用 OCR 提取文本；支持 100 种语言；保留质量最高的 10%。

训练效率

CLIP：256 个 TPUv3 上训练 10 天。
SigLIP：32 个 TPUv4 上训练 5 天，计算更快地完成。
关键创新是实现并行化：每台设备只在本地计算部分损失，然后循环交换文本嵌入，计算非对角线负样本，最终覆盖所有组合。

Batch Size 解耦

SigLIP 将 batch size 与损失解耦。与 CLIP 不同，改变 batch size 不改变损失函数形式。
在 batch size 小于 16K 时显著优于 CLIP；可以扩展到 1M，但 32K 已达到临界点，继续增大帮助不大。

第四部分：视觉语言模型（VLM）

LLaVA（Large Language and Vision Assistant）

架构

视觉编码器：CLIP（ViT-L/14）。
文本解码器：Vicuna——基于 Llama，在 ShareGPT 对话数据上微调的语言模型。
投影器：线性投影矩阵 W，将视觉向量映射到文本嵌入空间。Flamingo 和 Q-former 使用了更复杂的投影方案。

数据

基于 MS COCO（带边界框和人工标题标注）。
向 GPT-4 提供标题或检测到的物体，生成对话、详细描述和复杂推理，共 158K 合成样本。

训练

阶段一（对齐）：冻结视觉编码器和语言模型，只训练 W。
阶段二（微调）：冻结视觉编码器，训练 W 和语言模型。

突出能力

能进行视觉推理，如“这张图片有什么不寻常的地方”——识别出“你通常不会在小型货车的后部熨衣服”。

LLaVA OneVision

主要升级

视觉编码器：SigLIP（使用最后一个 Transformer 层前后的 grid features）。
文本解码器：Qwen-2 72B。
投影器：2 层 MLP 替代线性投影。
支持多张图像和视频。

AnyRes 高分辨率处理

问题：CLIP 会缩放裁剪到 336×336，对 OCR 等需要细粒度信息的任务不可用。
AnyRes 方案（LLaVA 1.5 引入）：
- 将图像分解为多个块，每个块匹配视觉编码器期望的分辨率（如 336×336）。
- 分别编码每个块，然后拼接所有向量。
- 如果 token 过多，用双线性插值减少。
这是对动态分辨率的适应性设计——Transformer 天然适合处理不同长度输入。

三种输入模态的 token 平衡

单张图像：高分辨率（完整降采样 + 最多 9 个裁剪块）。
多张图像：每张用基础分辨率。
视频：每帧用更低分辨率（最多 32 帧），避免数据集被重复帧主导。

数据策略

哲学：高质量优于数量，数据高度任务化（VQA、表格问答、找差异等）。
大量数据是蒸馏 GPT-4 得到的。
包含单图、多图和视频数据。

模态间迁移

论文发现了有趣的迁移现象：

只在单张图像上训练了图表/图示数据，但能泛化到多张图像场景（表格 + 图表问答）。
只在单图中训练了 OCR，在多图中训练了关系推理，却可以泛化到多截图 GUI 代理任务。
只在单图中训练了视觉提示（visual prompting，用圆圈高亮区域），却能泛化到视频中跟踪被高亮的玩家。

这暗示：即使每个任务看起来是独立的监督学习，当任务种类足够多时，模型能够获得跨模态的迁移能力。

Qwen-VL 系列

Qwen-VL（第一代）

视觉编码器：OpenCLIP 的 ViT-bigC（14×14 patch）。
适配器：一层交叉注意力，加入 2D 位置编码，映射到固定长度 256。
特殊 token：图像标签 <img>、边界框标签 <box>、引用标签 <ref>。

三阶段训练：

大规模低质量数据：冻结 LLM，训练视觉编码器和适配器（14 亿样本）。
高质量任务数据：训练所有参数。
指令微调：冻结视觉编码器，训练适配器和 LLM。

支持中文问答、输出边界框、OCR 等功能。

Qwen2-VL

更大的视觉编码器（675M 参数）。
动态分辨率：每个 224×224 块用 ViT/14 编码，每 2×2 压缩为 1 → 每个块产生 66 个 token。小图像（公式）只产生 8 个 token，大图像可多达 11000 个。
视频：每秒采样 2 帧，最多 16384 个 token。

多模态 RoPE（MRoPE）：

1D RoPE 中，内积只依赖于 token 间的距离。多模态版本扩展到 3D：高度、宽度、时间。
每个位置由坐标三元组定义，对每个维度分别计算 RoPE 后拼接。

Qwen3-VL

语言模型：

使用 Qwen-3（有 dense 和 MoE 变体，最大 235B-A22B）。
长上下文 256K，对长视频理解很重要。

视觉编码器：

SigLIP-2（架构与 SigLIP 相同，向后兼容）。

改进的 MRoPE（交错）：

原始 MRoPE 将 [t t t w w w h h h] 分别分配给不同频率段，导致所有时间维度都是低频、高度都是高频。
改进后交错排列 [t w h t w h t w h]，让所有轴都能接触低高频。

显式时间戳：

之前时间信息隐含在位置编码里；现在将时间作为显式 token（如“0秒”是一个 token）。
这允许直接引用具体时间点（如“两秒之后发生了什么”）。

按 token 长度归一化的损失：

视频样本很长，默认 every token 等权会让视频样本主导训练。
方案：按每个样本长度的平方根归一化损失，降低超长样本的影响。

DeepStack 适配器：

不再将视觉编码器当作输出向量序列的黑盒。
将视觉嵌入直接注入语言模型多个层的残差流中，实现更深度融合。

训练：

预训练 4 个阶段：先训练适配器，再逐步在 8K → 32K → 256K 序列长度上训练全部参数。
后训练：长思维链 SFT、知识蒸馏、强化学习。

关于 VLM 的常见问题

这些模型只输出文本。所有多模态输入在前端编码，生成始终是文本。因此生成图像的逻辑（如判断生成质量）不在此范围内。
对齐阶段需要预训练好的 LLM 和视觉编码器。只训练适配器来连接两者。训练时确定一个 token 预算，没有自适应阈值。
视觉编码器参数量远小于 LLM。原因是视觉编码器做局部操作（理解小块），基本不做推理；模型的能力主要由语言模型承载。例如 LLaVA 的投影器远小于 ViT，ViT 通常不到十亿参数。

第五部分：Chameleon——统一的离散 token 方法

核心思想

之前的 VLM 将图像编码为连续向量注入 LLM，但无法生成图像。Chameleon 的答案是：将一切映射为离散 token。

优点：

可以用完全相同的方式分析和生成图像。
文本和图像真正生活在同一个空间。
训练就是普通的语言模型训练——没有适配器，没有微调视觉编码器。

示例：提示“我无聊了，能给我看些鸟吗”，模型输出文本 → 图像 → 更多文本和图像，交错生成。

VQ-VAE 图像分词

VQ-VAE（Vector Quantized Variational Autoencoder） 将图像映射为离散编码。
学习一个码本（codebook），如大小 8192，包含图像块的原型向量。
编码器输出连续向量后，舍入到最近的码本条目，这就是表示。
解码器从该编码重建图像，通过最小化重建损失训练。

实际效果：512×512 图像 → 1024 个 token，每个 token 来自大小为 8192 的词汇表。再用 BPE 训练新的 tokenizer。

训练与稳定性

阶段一（80%）：大规模无监督数据（290 万文本 token、150 万文本/图像 token、400B 交错 token）。
阶段二：高质量数据混合。

训练不稳定的根源：

文本 token 熵低、较可预测；图像 token 熵极高（“这个蓝色调是什么？”）。
混合训练导致参数范数增长和损失漂移。
缓解方案：QK 归一化、z-loss 正则化。

评价

优雅：单一模型以同样方式对待所有模态。
缺点：性能不够好；离散化丢失细粒度信息（如小字 OCR）；多模态训练困难。

VQ-VAE 风格的方法曾流行于图像生成（因为 Transformer 需要离散输出），但在扩散模型出现后逐渐失去主导地位。

总结与当前共识

前沿模型必须多模态：原生多模态/全模态已是大势所趋。闭源模型（Gemini、GPT）的具体架构未知，很可能是连续编码器 + 扩散生成的组合（推测，无公开细节）。
根本挑战是编码非文本模态：理解与生成需求不同：
- 理解需要高层语义（CLIP 类表示即可）。
- 生成（或 OCR）需要细粒度细节——CLIP 的 336×336 裁剪会丢失信息。
数据加权是关键：视频信息密度低于文本，不应让视频样本主导训练（平方根归一化损失）。
当前最佳实践：连续编码器（CLIP/SigLIP）+ Transformer + 扩散模型生成。即使 CLIP 已有近五年历史，这类方法仍是捕捉图像语义的首选。

课程作业关联

本讲没有布置专门的作业，但与前序内容直接相关：
- 第一讲词元化：文本需要 BP tokenizer，多模态需要思考“什么是图像的 BP tokenizer”。
- 模型训练相关讲座：了解不同模态如何影响数据加载、并行化和损失设计。
- 长上下文技术：视频处理的核心瓶颈是上下文长度。

常见误区

误区：VLM 能生成图像。 实际上主流的 VLM（LLaVA、Qwen）只接受图像输入，输出始终是文本。图像生成需要额外的扩散模型或 Chameleon 那样的离散 token 方法。
误区：视觉编码器参数量应与 LLM 匹配。 视觉编码器做局部操作、参数量远小于 LLM，模型大部分推理能力来自 LLM。
误区：图像 token 与文本 token 一样可预测。 图像 token 熵高得多，导致训练不稳定，需要特殊处理（如按长度归一化损失、QK 归一化）。
误区：任意分辨率图像可以直接输入 ViT。 需要 AnyRes 式的分块处理或动态分辨率设计，否则会丢失细粒度信息。
误区：多模态模型越大越好，数据是唯一的挑战。 数据质量与数量并重，且需要关注模态间的 token 数量平衡。

复习清单

[ ] 能解释为什么 Transformer token 概念需要扩展到非文本模态。
[ ] 能写出 CLIP 的核心目标（对比图像与文本嵌入的点积）。
[ ] 能说明 CLIP 在数据和训练上的局限（batch size、不可分解的 softmax）。
[ ] 能说明 SigLIP 与 CLIP 的核心区别（多类 vs 二元分类）及其并行化优势。
[ ] 能描述 VLM 的基本架构（视觉编码器 + 投影器 + LLM）。
[ ] 能解释 LLaVA 的两阶段训练策略（先对齐投影器，再微调 LLM）。
[ ] 能描述 AnyRes 如何解决高分辨率输入问题。
[ ] 能指出 Qwen2-VL 的 MRoPE 如何处理 3D 位置。
[ ] 能说明 Qwen3-VL 中交错 MRoPE 与显式时间戳的作用。
[ ] 能解释 Chameleon 的基本思路（离散图像 token + 纯语言模型训练）及其缺陷（离散化损失、训练不稳定）。
[ ] 能总结当前多模态建模的共识设计（连续编码器 + Transformer + 扩散生成）。

从CLIP到全模态模型：CS336 Lecture 17 多模态建模全景解析

导语

当我们在讨论大语言模型时，一个显而易见的局限是：世界远不止文本。一张图片、一段视频、一段语音，这些模态构成了我们感知世界的基本方式。CS336课程的第17讲「Alignment - multimodality」正是要回答这样一个问题：如何让Transformer架构处理并理解这些非文本模态？本讲从CLIP和SigLIP这两个基础视觉编码器讲起，逐步深入到LLaVA和Qwen系列视觉语言模型（VLM），最后探讨了走向全模态（Omni Model）的激进方案Chameleon。这篇文章将沿着课程的思路，为你梳理多模态建模的核心挑战、主流解决方案及其演进逻辑。

为什么必须处理多模态？

课程一开始就点明了动机：目前所有主流模型都在向多模态方向发展。虽然语言模型已经非常通用——你可以用它们处理任何文本，包括代码、诗歌甚至DNA序列——但世界本质上是多模态的。我们既需要理解图像，也需要理解音频和视频。

课程的终极目标是所谓的“全能模型”（omni model），即能够接收任意模态组合作为输入，并输出任意模态组合的模型。例如，给模型一张图片和一段视频，它可以回答关于这些内容的问题；给定一段音频，它可以生成对应的图像。当然，课程并不打算在一节课内构建这样一个完整的系统，而是聚焦于一个更基础的问题：如何让模型“摄入”图像。

这里有一个核心矛盾：Transformer是为文本设计的，它通过“词元”（token）进行交互。一个词元应该代表某个“语义信息单元”——在自然语言中，词元是子词，它们确实承载语义。但一个像素本身显然没有语义。因此，我们需要将图像、音频等非文本数据转换成Transformer能消化的形式。课程提醒我们，即使是文本，我们也经历了分词（tokenization）的过程（第一讲的内容），而处理非文本模态时，这个挑战要严峻得多。

于是，课程提炼出两个关键问题：

如何将非文本数据（如图像）输入Transformer？
如何让模型生成非文本数据？

本讲的核心篇幅集中在第一个问题上，而答案的起点是一个2021年的经典模型——CLIP。

基石：CLIP与SigLIP

CLIP：对比学习的开创性实践

CLIP（Contrastive Language-Image Pre-Training）的诞生背景颇具启发意义。在2021年之前，计算机视觉的主流范式是在大型人工标注数据集（如ImageNet）上训练ResNet等模型。OpenAI的研究人员提出疑问：能否像语言模型那样，直接利用互联网上海量的（图像，文字描述）配对数据，而不是依赖昂贵的人工标注？

CLIP的思路非常直观。假设你有一批图像-文本对（例如32,000对），你用一个图像编码器将每张图像编码成一个向量，用一个文本编码器将对应的文本也编码成一个向量。然后设定目标：对于给定的图像，它应该与其配对的文本在向量空间中对齐（即点积远大于与其他文本的点积），反之亦然。这本质上是一个包含2×N个softmax分类问题的对比学习框架。

课程展示了实现这一目标的代码逻辑：你获得一个n×d的图像嵌入矩阵和一个n×d的文本嵌入矩阵，归一化后计算带温度参数的点积，最后计算交叉熵损失。关键在于，整个batch内的样本形成了m×n的矩阵结构。

数据方面，CLIP论文的细节不多，大致是从网络上搜索查询词，挖掘出4亿个图像-文本对。这些数据没有完全公开，但后来OpenCLIP论文复现并扩展了这一思想，使用了LAION-5B数据集（包含50亿图像-文本对）。有趣的是，LAION-5B的构建过程本身就使用了CLIP进行数据过滤，存在一种引导（bootstrapping）过程。

图像的预处理也值得注意。网络上的图像分辨率各异，而神经网络通常要求固定尺寸输入。当时的解决方案是启发式的：将图像调整大小使较短边为336像素，然后中心裁剪成正方形。这种方法显然会丢失信息，课程后续会展示更精细的处理方式。

视觉编码器的选择上，CLIP作者实验了ResNet和当时新出现的视觉Transformer（ViT），发现ViT效果最好。ViT的运作机制是：将图像切成14×14的小patch（每个patch有RGB三个通道），这些patch本质上就是Transformer的“词元”，加上位置嵌入后送入标准Transformer。CLIP使用了一种改进的池化方式——注意力池化：先对所有激活做全局平均，然后用这个查询向量对每个位置的键和值再做一轮注意力，得到更有信息量的向量。

文本编码器则是一个GPT-2风格的Transformer，通过在序列前置BOS和追加EOS，取最高层的EOS激活作为整个序列的表示。

CLIP最令人瞩目的结果是：零样本情况下，它在ImageNet基准测试上超过了在120万张带人工标注图像上训练的ResNet-50。这展示了利用网络噪声数据的巨大潜力。但课程也提醒我们，这个过程存在噪声——图像旁边的文字不一定逐字描述图像内容，需要大量数据过滤才能发挥作用。

课程还展示了一个重要的消融实验结果：如果采用“从图像直接预测文本”的替代目标，即使是使用更强的语言模型，其效率也不如CLIP的对比排名方式。这说明对标题的精确token序列进行建模，对于获得图像的粗略语义表示并不必要。

总结CLIP：

优点：通过文本配对捕捉了图像的语义信息（因为文本谈论的是语义）。
技术缺点：需要很大的batch size（如32,000），且softmax操作是在整个batch上进行的，难以分解，这给分布式训练带来挑战。

SigLIP：摆脱batch size的束缚

SigLIP（Sigmoid Loss for Language Image Pre-Training）由Google提出，被课程称为“CLIP的改进版本”。其主要区别在于损失函数：CLIP做的是多类别分类，即判断一个文本-图像对相对于所有其他备选是否为正样本；而SigLIP做的是二元分类——只判断任意给定的图像-文本对是否匹配。对角线元素为正样本，非对角线元素为负样本，目标函数是一个简单的log_sigmoid。

这个看似微小的变化带来了显著的效率提升。CLIP在256个TPUv3上训练了10天，而SigLIP在32个TPUv4上仅需5天。需要指出的是，TPUv4在单芯片算力上并不比TPUv3快多少，关键是SigLIP的并行化策略更优：通过异步分发文本嵌入给各个设备（如设备1收到T5到T8），每个设备可以仅根据本地数据计算所有损失，避免了全batch级别的通信。

更重要的是，SigLIP将batch size与损失函数解耦了。在CLIP中，损失与batch size紧密绑定，改变batch size就改变了损失函数。而SigLIP则不然，这使得它可以在远小于16K的batch size下工作得比CLIP更好，因为过小的batch size会导致CLIP的softmax损失退化。SigLIP甚至实验了高达1M的batch size，但发现32K已经足够——这呼应了课程中关于“临界batch size”的讨论。

构建视觉语言模型（VLM）：LLaVA与Qwen

有了CLIP和SigLIP这两个“眼睛”，下一步就是将它们接入语言模型，构建能够理解图像并对话的视觉语言模型（VLM）。课程介绍了两个主要的模型家族：LLaVA和Qwen。

LLaVA：初代开源的VLM模板

LLaVA（Large Language and Vision Assistant）发表于2023年，它的设计奠定了VLM的基本框架。其架构由三部分组成：

视觉编码器：使用CLIP（ViT-L/14）。
文本解码器：使用Vicuna——一个在共享GPT对话数据上微调过的Llama模型。
投影器：一个线性变换矩阵W，将视觉编码器的输出映射到语言模型的嵌入空间。

训练分为两个阶段：

对齐阶段：冻结视觉编码器和语言模型，只训练投影矩阵W，确保图像嵌入被映射到与文本嵌入相似的空间。
微调阶段：冻结视觉编码器，训练W和语言模型，使用合成的对话数据。

LLaVA的训练数据是一个通过“引导”生成的合成数据集。具体来说，研究团队使用MS COCO数据集（包含人工标注的标题和边界框），将这些信息输入GPT-4，让其生成问题、对话、详细描述和复杂推理。最终构建了158K个训练样本。这个方法虽然高效，但也意味着模型性能受限于教师模型GPT-4的能力，存在蒸馏的倾向。

课程展示了一个有趣的示例：当用户问“这张图片有什么不寻常的地方”时，模型能够识别出“你通常不会在小型货车的后部熨衣服”。这展示了VLM在视觉推理方面的潜力。

LLaVA OneVision：升级与自适应分辨率

2024年发布的LLaVA OneVision代表了该系列的进一步演进。其改进主要体现在：

视觉编码器：升级为SigLIP。
文本解码器：使用Qwen-2。
投影器：从线性投影升级为两层MLP。

OneVision最具创新性的贡献在于解决了高分辨率图像的处理问题。CLIP会将图像缩放到336×336，这对OCR等需要细粒度信息的任务来说是灾难性的。它的解决方案是AnyRes策略：将图像分解成多个部分，每个部分都调整为视觉编码器期望的分辨率（如336×336），逐一编码后拼接。对于超高分辨率或视频输入导致token数过多的情况，则进行降采样或通过插值减少patch数量。这种自适应性使得模型能够处理单张图像、多张图像和视频——视频本质上就是采样后的帧序列。

数据策略上，OneVision强调“高质量与数量”的平衡，大量使用针对特定任务的合成数据（如视觉问答、表格问答、OCR），并且毫不掩饰地依赖GPT-4进行数据蒸馏。有趣的是，课程强调了一个现象：模态间的迁移学习。例如，模型只在单张图像上接受过图表理解训练，却能够泛化到多张图像（如一张表格加一张图表）；只在单张图像上见过视觉提示（用一个圆圈高亮图像区域），却能在视频中跟随被高亮的玩家。这表明，当任务多样性足够高时，模型能够发展出跨模态的泛化能力。

Qwen-VL系列：走向工程化与规模化

Qwen系列VLM模型在框架上类似，但在细节上不断创新。Qwen-VL使用了OpenCLIP作为视觉编码器，适配器采用一层交叉注意力，并结合二维位置编码映射到固定的256长度。它还引入了特殊token（如图像标签、边界框标签、引用标签），支持检测和OCR。

Qwen2-VL引入了两个关键创新：

动态分辨率处理：与LLaVA的AnyRes类似，将图像按224×224块编码，每2×2压缩为1，最终每个块产生66个token。视频则每秒采样2帧，上限16,384 token。
多模态旋转位置编码（MRoPE）：将RoPE从一维扩展到三维，每个位置由（高度，宽度，时间）三元组定义，每个维度独立计算RoPE后拼接。这有效处理了视频中空间和时间维度的位置信息。

Qwen3-VL则进一步优化：

交错式MRoPE：解决原始MRoPE可能将时间维度都映射到低频、宽度高度映射到高频的问题，通过交错排列确保所有轴都能接触低频和高频。
显式视频时间戳：将时间信息作为显式token加入，而非仅依赖位置编码，使模型可以直接引用“两秒之后发生了什么”。
平方根归一化损失：由于视频样本远长于单张图像样本，为避免视频在训练中占主导，按样本长度的平方根进行归一化。
DeepStack适配器：借鉴DeepSeek的工作，将视觉信息注入语言模型的多个层，而非仅作为输入馈入，实现了更深入的视觉-语言融合。

Qwen3-VL的训练流程已经非常复杂：预训练有4个阶段（从适配器训练到逐步延长至256K序列），后训练包含长思维链数据的SFT、知识蒸馏和强化学习。课程指出，这已经是一篇“系统工程论文”，性能上足以与闭源模型如Gemini、GPT-5和Opus 4.1抗衡。

迈向全模态：Chameleon的离散化尝试

上述VLM模型有一个共同的局限性：它们只能生成文本，不能生成图像。要生成图像，要么使用扩散模型，要么采用Chameleon的激进方案——将所有模态都映射为离散token。

Chameleon（2024年，Meta）的基本思路是：如果你能把图像变成像文本一样的离散token，那么就可以用标准语言模型的训练方式处理一切。实现这一目标的关键组件是VQ-VAE（向量量化变分自编码器）：将图像编码为连续向量，然后舍入到最近的编码本原型向量，得到一个离散编码。解码器则负责从这些离散编码重建原始图像，通过最小化重建损失来训练。最终，一张512×512的图像被转换成1024个token，每个token来自大小为8192的码本。

这种方法的优雅之处在于，它真正实现了：文本和图像生活在同一个空间，可以用完全相同的方式分析和生成。模型可以交错输出文本和图像——例如，提示“我无聊了，能给我看些鸟吗？”，模型会输出文本，然后是图像，接着是更多文本和图像。

然而，Chameleon在实践中面临严峻挑战：

训练不稳定：文本token熵较低（多数词可以预测），而图像token熵极高（不知道是哪种蓝色调），这种差异导致参数范数增长和损失不稳定。需要通过QK归一化和z-loss正则化来缓解。
离散化信息丢失：正如OCR问题所展示的，将高细节图像离散化会丧失阅读能力。
多模态训练困难：需要精心调整各模态数据的权重，否则训练会不稳定。

VQ-VAE思路曾经流行，但随着扩散模型在生成领域的崛起，这种方法逐渐被边缘化。课程最后总结道：目前最佳实践是连续编码器加Transformer，配合扩散模型进行生成。

课程定位与总结

在第17讲结束时，这门课关于多模态的探索涵盖了一个完整的演进脉络：

编码层：从CLIP到SigLIP，解决了将图像转化为语义向量的问题，并提升了训练效率。
注入层：从LLaVA到Qwen系列，通过投影器将视觉编码接入语言模型，并不断优化分辨率处理、位置编码和训练流程。
生成层：从Chameleon的离散化到扩散模型，探讨了如何输出非文本模态。

对于本课程的整体路径而言，第17讲是对纯语言模型学习的拓展。它承接了课程早期关于tokenization和Transformer架构的内容，并延伸到当前业界前沿的视觉语言模型实践。课程没有布置相关作业，但鼓励有兴趣的读者尝试训练这类模型。

最后，课程给出了几个关键思考：

理解与生成的不对称性：理解模态（如分类）需要高层语义，向量可以较小；生成模态（如图像生成）需要细粒度细节。不存在一个通用的编码器同时满足两种需求。
模态间的平衡：视频信息密度低于文本，需要精心调整损失函数权重，避免长视频样本主导训练。
当前的最佳实践：连续编码器（如CLIP/SigLIP）+ Transformer + 扩散模型的组合，仍然是处理多模态的主流方案。

展望未来，一个有趣的问题留给了读者：当文本、图像、视频都变成了“词元”时，我们是否真的能构建出一个真正统一的全模态模型？课程提醒我们，即使是称呼它们为“离散词元”，也不能掩盖它们行为方式的巨大差异——这正是Chameleon留给我们的教训。

官方资源：

视频：CS336 Lecture 17 官方视频
讲义：CS336 课程主页

CS336 2026 Lecture 17：完整中文逐字稿

视频：官方视频
来源：英文人工字幕提取后翻译为中文；保留段落起始时间戳，只修正明显的断句、重复和语序。
说明：这是逐字稿，不是讲义摘要；相邻且语义连续的片段已合并为阅读段落；术语保留必要英文。

正文

[00:05.000] 所以，原本的计划是要更多地讨论强化学习。但我想，既然我只有一节课的时间，如果不讲点多模态的内容，这门课似乎有点不完整，因为如果你看看所有主流模型，多模态无处不在。所以这将是多模态模型的一个概述。当然，这本身可以是一整门课，我会尽力在一节课内涵盖我所能讲的内容。好的，让我们深入探讨多模态模型。到目前为止，在这门课中，我们完全聚焦于语言模型。而语言模型已经相当通用了。它们赋予你从任何文本片段生成任何文本的能力。所以这可以是任何语言、代码、诗歌，任何你喜欢的，甚至可以说，比如DNA。但世界是多模态的。你有文本、图像、音频、视频。

[01:05.260] 如果你想一想我们想要到达的“北极星”目标，那就是人们所谓的全能模型（omni model），即能够接收这些模态的任意组合，并输出这些模态的任意组合。例如，我可以给你一张图片和一段视频，然后我可以就这张图片和视频提问。我可以生成图像。我可以把音频转换成图像。你可以想象通过一个全能模型实现各种不同的应用。所以在这节课中，我不会构建一个完整的全能模型。但我会给你们一些组件，至少让你们能够摄入一点图像，这也是我们今天主要关注的点。那么如何构建这样一个东西呢？嗯，这是一个有点奇怪的动机阐述方式，但现实是Transformer效果非常好。尽管人们尽了最大努力尝试其他方法，但在所有这些模态上，在大规模情况下，Transformer仍然是我们拥有的最好的东西。

[02:15.690] 所以我们必须弄清楚如何使用它们。现在，Transformer是为文本设计的。所以它们有这样一个特性：它们使用词元（token）进行交互。也就是说，它们接收一堆词元作为输入，然后输出一些词元。在这里，我扩展了词元化的概念，不仅指我们在文本中看到的离散词元，也指潜在的连续词元。你可以把这些看作词元的嵌入。关键点在于，一个词元应该代表某个语义信息单元。例如，在自然语言中，词元是子词。这些是有一定意义的。然而，比如说，一个像素本身肯定是没有意义的。所以我们必须以某种方式将所有东西，包括音频和图像，转换成离散或连续的词元。现在，请注意，即使处理文本，我们也不得不这样做。在第一讲中，我们讨论了词元化。在那里，情况还不算太糟。我的意思是，我们有一个BP词元化器。

[03:26.750] 你们实现了它。这没问题。人们可能期望一个更好的词元化器，或者根本没有词元化器。但这并不是世界上最糟糕的事情。现在，对于非文本模态，我们必须更加绞尽脑汁，弄清楚什么是BP词元化器的等价物，能够接收一张图像并产生Transformer可以消化的东西？好的，所以你必须在这里回答两个问题。一个是，对于非文本数据，例如图像、视频或音频，你如何将它们输入到Transformer中？然后你如何将它们生成出来？这节课将主要关注第一个问题。好的，让我们回到过去，把时间拨回到2021年。CLIP模型，你们中有些人可能熟悉，是现代视觉语言模型（VLM）的很多基础。所以我们可以稍微详细地过一下这个。CLIP代表对比语言-图像预训练（Contrastive Language Image Pre-Training）。

[04:43.500] 一点历史背景是，在那个时代，GPT-3已经出现了。GPT-2也是。所有的语言模型都已经进入了这个基础模型时代。计算机视觉（当时叫Neovision？）仍然——虽然有一些努力，但传统上基于大型标注数据集，比如ImageNet，并在这个数据集上训练像ResNet这样的模型。以及做一堆其他事情，比如数据增强，并取得了非常好的性能。所以OpenAI的研究人员在想，是否有可能利用互联网上存在的大量图像和文本描述？受到语言领域这个想法的启发——你可以直接抓取互联网，你有所有这些文本，它非常嘈杂。但不知何故，如果你训练一个足够大的模型，你可以从中理出头绪并生成有用的东西。那么对于图像来说，对应的等价物是什么？因此，CLIP诞生了。CLIP的想法相当简单直接。这里的想法是，假设我给你一堆图像-文本对。

[06:05.980] 比方说，我给你32,000个这样的对。好的，那么我要做的是，对于每张图像，我用一个图像编码器（稍后我会解释）把它编码成一个向量。所以我将有32,000个这样的图像编码。我也会对相应的文本做同样的事情。这将给我T1到TN。这些是文本的嵌入。这些是图像的嵌入。现在，我想要建立的目标是，我希望图像——拿I1来说。I1与T1是对齐的。所以我希望这种对齐，即这两个嵌入之间的点积，远大于I1与所有其他文本之间的点积，明白吗。反过来，对于这个文本，我也希望这个点积大于T1与所有其他图像之间的点积。所以，简而言之，这就是CLIP的目标。你可以想象做——基本上有2乘以N个不同的softmax分类问题。

[07:30.350] 所以这是代码。你得到一个n乘d的矩阵用于图像编码，以及一个n乘d的矩阵用于文本嵌入。你进行归一化，然后带着某个温度参数做这个点积。然后你计算交叉熵，好的。所以它基本上是一个多类分类问题，其中样本是以这种m乘n的矩阵形式结构化的。好的，关于这个有什么问题吗？好的，那么关于CLIP是如何构建的，还有一些更详细的细节。数据从哪里来？论文中关于数据的细节不是很多。大致来说，你从某处获取一堆查询词，然后在线搜索这些词，并挖掘一堆图像和文本对。这产生了4亿个图像-文本对。这个数据集没有发布，但是有一篇名为OpenCLIP的论文做出了努力，它复制并进一步扩展了CLIP的思想。所以有一个叫LAION-5B的数据集被发布了，包含50亿张带有文本描述的图像。

[09:16.350] 并且发生了很多处理过程。有趣的是，他们实际上使用CLIP进行数据过滤，然后训练OpenCLIP。所以这里有一些引导（bootstrapping）过程。但至少这是一个你可以指出其训练所用数据集以及代码的模型。还有一些其他细节。图像有各种分辨率。如果你在网上找到一张图片，它可能是细长的或者高——抱歉。细长的或者另一个方向的。所以是任意尺寸。而关于神经网络，你会学到的一件事是它们不喜欢动态的东西。它们希望东西是固定大小的。所以有一个稍微启发式的处理过程，他们调整图像大小，使较短的边是336或任何目标尺寸。也可能是224。然后他们中心裁剪得到一个正方形。好的，这显然是为了方便。

[10:29.660] 稍后，我们会看到你可以做得比这更好。但这只是为了快速简便。而且在这个时期，CLIP的作者们也在考虑ImageNet和分类。所以通常，物体在中间，你只是裁剪掉一些背景。所以这一点并不太重要。那么，视觉编码器是什么呢？这是总体算法，但我还没有具体说明图像和文本编码器分别是什么。是的。[听不清] 所以我认为这个问题是，为什么他们要在图像-文本对上训练，而不是在图像上训练。[听不清] 所以有一条研究路线尝试基于数据增强将图像编码为表示。例如，如果你拿一张图像——我想这个想法叫 SimCLR——你对其进行增强，裁剪它，随机扰动它，旋转它，然后你想说这些操作得到的是同一张图像。

[11:56.850] 它们应该有相似的嵌入。这对低级细节是有用的。但问题是，你无法通过数据增强从一种狗变成另一种狗。所以通过使用文本，它能给你提供图像更高层次的语义表示。是的，这是个好问题。稍后我们会看到，当你看人们试图解决的任务类型时，这些语义表示是如何发挥作用的。好的，那么 CLIP 使用的视觉编码器是什么呢？他们实际上实验了好几种不同的编码器，包括 ResNet 和当时刚出现的视觉 Transformer。他们发现视觉 Transformer 表现最好。所以当人们说 CLIP 时，他们通常指的是 ViT 版本。视觉 Transformer 的做法是这样的：你拿一张图像，把它切成这些小 patch。我记得原始论文用的是 16×16，而 CLIP 用的是 14×14。

[13:18.850] 但无论如何，你有了这些小 patch，它们基本上就是向量。每个 patch 在某种意义上就是——嗯，这个视觉 Transformer 的一个 token。你像训练语言模型时那样加上位置嵌入，然后直接送入一个标准的 Transformer。如果你要做分类，那你就会——嗯，你通常想要——好的，抱歉。Transformer 编码器输出一个向量序列。而 CLIP 论文的做法稍微有点不同。你可以对所有向量取平均，这样会得到一个向量。但他们认为用注意力池化会更好。这意味着你先对所有激活做全局平均，然后你用这个查询向量再对每个位置的键和值做一轮注意力。这样得到另一个向量，可能比直接取平均更有信息量一些。

[14:37.080] 好的，这就是 ViT。CLIP 论文发现最好的模型是所谓的 ViT-L/14。解读一下，这是一个大的 ViT。我记得它有大约 24 层，我不太确定。patch 大小是 14×14。每个 patch 是 RGB 的，所以有三个通道。他们在 336×336 分辨率的图像上训练。实际上，他们先在较低分辨率上训练，我猜是为了速度，因为高分辨率图像需要更长时间。然后在训练的后半段，他们在更高分辨率上训练。是的。对于位置嵌入，它会比任务本身更复杂吗？例如，任务只有[听不清]。是的，所以问题是，你能不能对位置嵌入更聪明一些，因为这只是线性的 0 到 9？我记得在这篇论文里，他们实际上——在 CLIP 论文中，我相信他们尝试了一些 2D 版本的方案。

[15:57.500] 他们发现这其实不太重要。我觉得这也是——我总是需要带着这样一个观点来看待这些结果，那就是他们感兴趣的是分类任务。所以对于分类来说，也许这确实不重要。稍后我们会看到更复杂的位置嵌入，它们确实会考虑空间结构。好的，那么文本编码器是一个标准的 Transformer。它是 GPT-2 风格的 Transformer，因为这是开发 GPT-2 的同一个团队做的。为了从序列中获取一个向量，他们前置了 BOS 并追加了 EOS。然后他们取 EOS 在最高层的激活作为整个序列的表示。好的，所以你有了视觉编码器，有了文本编码器。然后你可以通过这个过程来训练：你选一个 batch，编码所有文本，编码所有图像，然后形成这两个交叉熵损失，然后继续迭代。

[17:18.270] 好的，最引人注目的结果——这也是 2021 年让很多人在 CLIP 上兴奋的原因——是在 ImageNet 基准测试上，零样本 CLIP 超过了在 120 万张 ImageNet 图像上训练的 ResNet。想一想，120 万张 ImageNet 图像，这需要亚马逊 Mechanical Turk 的工人花很多很多小时来做标注。而现在你有 CLIP，它是在更有机的网络数据上训练的。当然，这也是网络上很多人劳动的结果。但如果你能利用这些已有的数据，那你就可以直接做零样本。零样本的技术基本上就是——我在这里没有幻灯片支持。

[18:10.870] 但你拿一张图像，然后你有一堆文本。你基本上用各标签做点积，看哪个最高。哦，是的，它在这个幻灯片部分上。好的，说得好。所以这就是零样本预测，你拿一张图像然后——是的，就是我刚才说的。好的。有什么问题吗？是的。[听不清] 是的，所以问题是，如果你有一只狗，可能还有其他包含狗的标题，这会混淆模型吗？一般来说，这个过程会有噪声。有另一只狗也没关系，因为平均来说，不太可能总是有一只狗。有时候会是一个苹果或者一只猫之类的。但另外一点是，这些图像基本上是从网络上抓取的，旁边有文字或者有 Alt 文本。所以它们非常嘈杂。有研究表明，当你有一条图像的标题时，它不一定逐字描述图像里有什么。

[19:48.547] 因为如果你有一张图像，里面是一只狗，你不需要说“一只狗”。所以这个过程非常嘈杂。所以它能工作，这也许有点令人惊讶，但也很有趣。我的意思是，这需要大量的数据过滤。如果你随便拿网络上的图像和文本配对，那可能噪声太大了。另一件事——我到最后会再回到这个观点——他们也尝试了一种替代方法，不是做这个排序，而是从图像预测文本。所以他们可以设定一个目标，尝试用图像来预测文本，可以是词袋模型的形式，也可以是语言模型的形式。他们表明，也许——我不太确定——有些令人惊讶的是，如果你用更强的模型，它实际上表现更差，或者至少与使用词袋模型的 CLIP 相比效率更低。这是在说——我认为这是在说，对于你试图学习的表示——或者，再次强调，这是 ImageNet 的准确率。

[21:09.120] 所以是分类任务。实际上，对标题的精确 token 序列进行建模对于获得图像的粗略表示并不那么重要。好的，我们最后会再回到这个观点。那么到目前为止我们学到了什么？我们已经用 CLIP 编码了图像。它捕捉了图像的语义，因为它与文本配对。而一般来说，文本谈论的是语义。需要注意的是，这里的设计决策是基于图像分类的。所以它不是非常细粒度的。但尽管如此，正如我们将看到的，它作为我们后续所有工作的一个稳健起点。CLIP 的一个技术缺点是它确实需要很大的 batch size，比如 30,000。如果你的 batch size 是 1，那显然行不通，即使是 10 也不行。而且，softmax 操作是在整个 batch 上进行的。所以它并不是真正可分解的。

[22:22.970] 而如果你考虑普通的语言模型训练，一批样本中，所有的序列都是并行处理的，你只需在最后进行聚合。所以这个问题将由谷歌发表的另一篇论文解决，名为 SigLIP。你可以把它理解为 CLIP 的一个改进版本。它的全称是 Sigmoid Loss for Language Image Pre-training（用于语言-图像预训练的 Sigmoid 损失）。这里的主要区别在于，CLIP 做的是多类别分类。它会说，我有这个配对的文本和图像，并且我会认定这对是正样本，相对于所有其他备选图像或所有其他备选文本而言。

[23:13.930] 而从某种意义上说，SigLIP 要简单得多。它基本上只判断任何给定的图像-文本对是否匹配。所以如果我参考上面的图表，它基本上就是二元分类，其中对角线上的元素是正样本，而非对角线上的元素是负样本。仅此而已。所以这是算法流程。你获取嵌入向量，进行归一化，像往常一样计算点积。但现在你构建的标签是：非对角线元素为 -1，对角线元素为 1。然后使用 log_sigmoid。所以目标函数非常简单。是的。[听不清] 是的，所以问题是，这是否需要复杂的采样策略？至少在最初的那篇论文中，他们只是直接使用了完全相同的矩阵类型。我可以想象，你思考的可能是这些对比类方法的一般情况。

[24:37.840] 负样本需要——有时你需要平衡正样本和负样本，或者负样本必须是难负样本，这样才不会产生偏差。但至少在最初的版本中，这是相当直接的。说到数据方面，这项工作是在谷歌完成的。这是所用的数据集。他们使用了另一篇论文，那是 2022 年左右的另一个语言-图像模型，它提供了拥有数十亿图像-文本对的 WebLI 数据集。他们还做了一些额外的工作。例如，对于包含文字的图像，他们应用了 OCR。这是形成文本-图像对的另一种方式，并且他们还做了一些过滤。这个数据集是多语言的。这篇论文的主要观点是，它的训练效率比 CLIP 高得多。CLIP 是在 256 个 TPUv3 上训练了 10 天，而 CLIP 是在 32 个 v4 上训练了 5 天。你可能会想，哦，v4 应该比 v3 快。但从每秒浮点运算次数来看，它们其实并不快。

[25:53.140] 它们更好是因为你可以把更多的芯片放入一个 Pod 中，并且互联更好。但在这种规模下，它实际上并不更快。事实上，可能慢了 60% 左右。所以它更快的方式是——当然，我认为 CLIP 可能并没有试图优化代码以追求最大吞吐量。它当时的想法只是让这个东西跑起来。但 SigLIP 论文表明，你可以按如下方式实现并行化。你可以将其视为 DDP，如果你乐意的话，每个设备存储一部分图像-文本对。然而，与语言模型训练中所有因素都相互独立不同，这里的样本之间存在交互。所以在第一次遍历中，每个设备仅根据本地拥有的图像-文本对计算所有损失。然后你发送文本。基本上，设备 1 会收到 T5 到 T8。

[27:24.570] 现在它能够计算这里的负样本。然后它再收到 T9 到 T12。就这样循环轮转，直到覆盖所有非对角线的块元素。SigLIP 另一个好处是，你实际上将批次大小与损失解耦了。而在 CLIP 中，损失与批次大小紧密相关。如果你改变批次大小，损失函数就不同了。因此，他们能够尝试远小于 16K 的批次大小进行实验。在这种规模下，它比 CLIP 好得多，因为如果批次大小过小，损失函数会退化。而对于 CLIP 来说，使用较小的批次大小意味着更高的方差，但其期望值是一样的，是相同的损失。他们还说，你也可以使用大得多的批次大小，但这并没有实际帮助。我们之前看到了临界批次大小的存在。

[28:30.670] 而 32K 基本上就是他们的临界批次大小。好的，现在我们有 CLIP 和 SigLIP。这些是图像编码器，它们接收一张图像，在这个版本中，是固定尺寸的图像，比如 336 x 336，然后将其映射为一个向量，这个向量包含了一些语义信息。那么现在让我们开始构建 VLM，即视觉语言模型。我将讨论两个模型家族，LLaVA 和 Qwen。这两个模型在整体框架上非常相似，但有一些细节上的不同，我们会逐一探讨。其基本思想是，我们将获取这些嵌入向量，然后将它们直接注入到我们的语言模型中。所以这更像是中间训练或后期训练的一种形式，我们采用一个现有的图像编码器，一个现有的 LLM，然后将它们拼接在一起，而不是从头训练一个模型。LLaVA 论文发表于 2023 年。

[29:46.680] 这篇论文让很多人感到兴奋，因为大约在这一时期，像 GPT（我记得是 GPT-4）这样的闭源模型已经能够进行视觉推理。而他们能够证明自己也能做一些视觉推理。当然，它不如 GPT-4 那么好，但我认为它是一个开源模型，让人们得以一窥其内部机制。所以对于 VLM 来说，基本上有以下几个部分。首先是视觉编码器，他们在这里使用了 CLIP。对于文本解码器，他们使用了一个名为 Vicuna 的语言模型，这是第一个在共享 GPT 对话数据上微调过的 Llama 模型。这些对话是人们与 ChatGPT 交互后分享到这个网站上的。

[30:38.400] 我认为这个网站现在已经不存在了。他们使用的数据基于 MS COCO。这是一个带标注的数据集，他们让 Mechanical Turk 的工作人员去标注大量图像，包括边界框和描述图像内容的标题。他们基本上为他们的模型合成一个数据集。他们向 GPT-4 提供标题或检测到的物体，并让 GPT-4 生成问题或对话。例如，这是 MS COCO 数据集中的一张图像。这是一个由人类完成的标注标题。同样，这里也是人工标注的。然后你要求 GPT-4 生成一段对话，你可能会得到一个问题和答案。你可能会要求 GPT-4 生成一个详细的描述，它会生成一些类似于标题的内容。然后你可以要求 GPT-4 生成复杂的推理。

[31:50.980] 它会生成类似这样的东西。好了，这些都是合成数据，共计 158，000 个样本。然后他们在这个数据集上训练了一个模型。这个模型——他们使用了 CLIP，特别是性能最高的 CLIP 版本，ViT-L/14。也就是说使用了 14x14 的图像块。然后让我们看看这张图像。他们获取图像，通过这个名为 CLIP 的视觉编码器进行处理，得到一个向量。但这个向量通常来说并不真正处于与文本相同的空间中。所以接下来他们将其乘以一个矩阵 W，得到另一个向量，这个向量与文本嵌入处于相同的空间。对于文本，你基本上只需使用标准的嵌入层来获取向量。所以基本发生的情况是，文本被编码成这些向量，图像也被编码成这些向量。然后这整个向量序列直接经过一个标准的 transformer，并产生输出。

[33:11.682] 好的，所以从某种意义上说，我们是在将这些图像转换成文本标记。这样我就可以利用预训练的语言模型了。为了训练这个模型，他们分两个阶段进行。第一阶段，他们冻结视觉编码器和语言模型，只训练这个矩阵 W。这就是他们所谓的对齐阶段，其本质上是确保图像被映射到相同的空间中。因为如果你只是使用一个随机的W，这些图像并不会——我的意思是，它们当然不是嵌入，不代表任何自然语言标记。所以训练W的目标是让它们看起来像自然语言的标记嵌入。而第二阶段是他们仍然冻结视觉编码器，然后训练W和语言模型。所以他们是在自己的数据上微调语言模型。记住，他们的数据基本上是这样的：这里有一张图像，这里是对话、描述或复杂推理的示例。

[34:32.080] 所以这全都是图像加文本到文本。好的，这是他们论文中的一个例子。这很有趣。你有一个输入图像和一个对话。用户问这张图片有什么不寻常的地方。而他们的模型能够告诉你，你通常不会在小型货车的后部熨衣服。他们还指出他们的模型相当不错，因为即使你的用户提示词并没有明确提示不寻常之处，它仍然会谈论这一点，而——GPT-4显然能够做到这一点，但当时其他模型却做不到。好的，关于LLaVA有什么问题吗？好的，那是2023年。现在我要跳到LLaVA OneVision，那是2024年。在LLaVA论文之后有一系列论文，包括LLaVA 1.5、LLaVA-Next。但我认为我接下来的描述会涵盖所有这些创新。他们做的主要事情本质上是相同的配方，但他们在尝试更具野心，涉及多模态应用的类型

[35:59.120] 方面。所以他们能够处理多张图像和视频。这是同样的示意图。现在他们可以输入单张图像、多张图像或视频，视频本质上是对帧进行采样后得到的图像序列。这里是视觉编码器。他们将其升级为SigLIP。文本解码器方面，他们现在使用Qwen-2，这可能是当时最好的语言模型。然后对于投影器或适配器，也就是将视觉编码器的输出转换为语言模型输入的那个组件，他们使用了两层MLP而不是线性投影。好的，这就像你有一个系统，只是在升级各个部件。但整体系统是大致相同的。那么让我们来谈谈数据处理。他们想做的一件事是OCR。而OCR的问题在于你需要保留非常细粒度的信息。

[37:20.130] 否则，J看起来像I，那就不太好了。记住，CLIP会调整大小并裁剪到336乘336。显然，如果你有一份文档，然后裁剪到336乘336，你根本没法阅读它。所以他们的解决方案是一个叫做AnyRes的想法，这实际上是在LLaVA 1.5论文中引入的。这个想法基本上——我的意思是，它相当直白。你把一张图像分解成多个部分。关键思想是每个部分都是视觉编码器将要接收的分辨率，比如336乘336。所以你会把所有部分编码成一组向量。然后你把这些向量拼接起来。好的，所以你基本上注意到你的视觉编码器无法处理高分辨率。所以与其降采样，你只是裁剪并查看图像的不同部分。而现在如果你有超高清图像或视频，你会得到太多标记。

[38:31.330] 所以他们进行降采样。但AnyRes背后的想法是它是自适应性的。这很好，因为transformer已经是自适应性的了。语言可以是任意长度的句子，transformer已经能很好地处理。事实证明图像也可以是任意分辨率的。所以这又回到了同样的动态能力。这是他们在这里对图像实际做的事情。假设你有一张论文的图像。他们会进行降采样，并有一个路径，基本上是整篇论文的降采样编码。但他们也会把它分解成这些块，每个块的尺寸正是视觉编码器所期望的分辨率，然后分别对每个块进行编码。然后你把这些补丁全部拼接起来。如果太多了，你就通过插值来减少补丁的数量。好的，这样他们就能处理图像、多张图像和视频了。

[39:43.560] 你可能会想，所有这些在技术上都可以归结为图像，但他们稍微做了一些干预，因为他们想确保所有模态大致具有可比性。因为视频可能非常长，他们不希望数据集被一堆重复帧所主导。所以他们在这里是这样做的。对于单张图像，他们基本上有完整图像的降采样版本，然后是一堆裁剪块，最多9个。如果给你的是多张图像，那么他们只使用基础分辨率。所以如果你有单张图像，我可以更仔细地看它。如果我有多个图像，我就只能远远地看。对于视频，我们会使用更低的分辨率或更少的标记来表示每一帧，因为思路是视频可能——他们最多使用32帧，但你会开始遇到上下文长度的限制。

[41:01.280] 稍后我们会看到，处理多模态的一个重大挑战就是处理更长的上下文。好的，他们在这里使用的数据，我认为理念是，他们想要策划高质量的数据和数量。另一种理解方式是这是非常有针对性的数据。如果你看其中很多数据，它们是非常基于任务的，无论是做视觉问答还是回答关于表格的问题。所以这绝对属于后训练领域，你试图让你的模型完成这些任务，因此你创建了这些任务。这项研究也毫不掩饰地基本上是在蒸馏GPT-4模型，以获得最佳性能，我想这并不理想。但如果你没有标注预算，这就是你要做的事情。所以他们的数据包括单张图像、多张图像和视频。

[42:20.390] 所以有很多不同的部分。其中一些非常具体。比如给你两张图像，你要找出差异。但有一件事——好的，我稍后会回到这一点。对于训练，它大致和之前相同的思路，第一阶段专注于对齐。具体来说，你只训练投影器，也就是适配器。然后在第二阶段和第三阶段你训练完整的模型。之前我们有两个阶段，现在有三个阶段。我不确定这有什么特别的原则性原因，除了第二阶段是试图使用高质量数据但更侧重于知识。最后阶段集中在看起来像你下游任务的示例上。好的，他们在这篇论文中发现了一件有趣的事情，那就是模态之间会产生迁移。即使他们只有用于图表和图示的单图像数据，但这实际上可以泛化到多张图像。

[43:49.070] 在训练时，你从未见过一个场景是一张表格和一张图表放在一起然后你问相关问题。但在测试时，你有两张图像，一张是表格，一张是图表，你能够围绕它们进行一些对话。在训练时，你只有单图像数据的OCR数据，而关系推理只来自多图像数据。但你可以推广到这些场景，这些场景对于驱动 GUI 代理非常有用，你给它展示一堆截图。这是多张图像，它可以生成对截图的描述。另一个例子是，他们有视觉提示（visual prompting）这个概念，你有一个圆圈，它是图像的一部分，但它的目的是高亮图像中你需要处理的部分。而这种类型的数据只存在于单张图像中。现在它可以推广到视频，用户可以描述视频中被高亮的玩家。这个玩家跨越多个帧。所以当你第一次看到这个，你会说，哎呀。

[45:11.940] 你基本上是在针对每一个这样的任务。这是监督学习。但如果你有足够多的任务，这些模型似乎能进行某种迁移，我想，这还算让人安心。好的，再次强调，这是一个相当标准的视觉语言模型（VLM）模板，视觉编码器加上投影器，进入语言模型。在数据整理方面有很多工作投入。他们相当依赖合成特定任务的数据。LLaVA 系列的一个优点是，它是少数开源的作品之一。他们不仅开源了模型权重，还开源了数据。所以你真的可以复现并研究这些东西。好的，现在我们来谈谈 Qwen 模型。Qwen 在 2023 年也开始训练多模态模型。第一个版本是 Qwen-VL。希望我能讲得快一点，因为你大概能看到模式了。

[46:19.780] 视觉编码器，他们用了 Open-CLIP。记住，OpenCLIP 是 CLIP 模型的开源复现。所以它基本上就是一个 CLIP 编码器。对于适配器，他们做了一层交叉注意力，结合了二维位置嵌入，并映射到固定的 256 大小，这有点——我想这之后会改变，因为这绝对不是很有动态性，但此时视觉编码器也不是。我想这部分被截断了。但特殊标记方面，他们有图像标签（image tag）、用于边界框的框标签（box tag），以及用于描述的引用标签（ref tag）。这就是当你的幻灯片是用 HTML 写的时候会发生的事情。对于训练，他们有三个阶段，和 LLaVA 模型一样。

[47:19.830] 在第一阶段，想法是——我的意思是，他们称之为预训练，但你不是从头开始预训练。这通常是大规模、低质量的数据。他们冻结语言模型，训练视觉编码器和适配器。这和 LLaVA 模型有点不同，因为他们在这一点上训练了视觉编码器。你可以看到他们训练的一些数据集，14 亿个样本。第二阶段，他们转向更高质量、特定任务的数据。你可以看到一些常见的视觉问答（VQA）数据集，图表问答等等。然后他们在这里训练所有参数。最后，第三阶段是指令微调数据。你冻结视觉编码器，训练适配器和语言模型。这里有一些例子，展示了它们能做什么。当然，这是 Qwen，所以有些例子是中文的。

[48:37.570] 他们的兴趣还包括打造一个应该能涵盖语言模型能力的模型。所以一个语言模型可以做代码。我们希望视觉模型也擅长代码。你可以看这个例子，如果你能看清的话，这是一张图像。然后“能找到蜘蛛侠和绿巨人吗”。它也能输出边界框。它实际上不输出图像，只输出边界框。然后你可以做光学字符识别（OCR）。你可以比较——就是这些常见的东西。好的，那么 Qwen-2 又是一次升级。这里有一些新想法，我来逐一讲解。他们使用了更大的视觉编码器。他们也开始采用动态分辨率。我认为这是主要的改进。我们记得从 LLaVA 到 LLaVA OneVision 时，他们意识到，哦，我们需要处理不同尺寸的图像。如果你尝试处理视频，很明显需要某种动态分辨率。

[49:47.700] 所以这个模型基本上——比如说你有这张图片，它可能被映射为 11,000 个令牌。这张很小的公式图片可能只被映射为 8 个令牌，等等。那么他们是怎么做的呢？嗯，基本上每个——这和 LLaVA 的 AnyRes 想法相同，每个 224 乘以 224 的块用一个视觉 Transformer（ViT）编码。记住他们开始用的 ViT 是 OpenCLIP 的。但这个是经过微调的。然后为了减少上下文大小，他们每 2 乘 2 压缩为 1，所以最终每个块产生 66 个令牌。对于视频，他们每秒采样 2 帧，但上限是 16,000 个令牌。

[50:59.600] 好的，回到位置嵌入这个问题，他们在 Qwen-2 中确实做了一些不同的事情。他们使用了叫做多模态旋转（multimodal rotary）、多模态 RoPE 的想法。记住，我们之前见过 RoPE。RoPE 背后的想法是，你有一个嵌入，使得向量之间的内积只依赖于距离。在 1D 中，距离由你离开它的令牌数量来定义。所以现在，多维版本本质上是一样的事情。只是现在你有 3D。所以你有高度，有宽度，有时间。每个位置、每个块现在是由坐标定义的三元组。然后计算 M-RoPE，基本上，对于每个维度，你计算 RoPE，然后拼接起来。所以这是一个相当直接的想法。虽然后来，在 Qwen-3 中，我们会看到这实际上有点次优。他们用——这是 Qwen-VL 模型——来初始化语言模型。

[52:23.790] 所以当然，他们是用 Qwen 来初始化的。然后他们有三个训练阶段，这与我们之前做的非常相似。然后他们展示了各种不同的能力，一些视频理解。它能做数学和代码、函数调用等等。好的，接下来我快速讲一下 Qwen-3。再次提到，这里有一些变化。我不会说这些是结构性的重大变化，但它们可能是确实影响模型质量的变化。这是去年 Qwen3-VL 的报告。所以这里的图表大致相同。我会指出——我稍后会回到这个图表。我会讲一些变化。一个是现在他们使用 Qwen-3 模型。他们有一系列密集模型和混合专家（MoE）模型。Qwen-3 模型真的非常好，这对最终模型的质量很有帮助。

[53:35.890] 他们投入的一个方向是长上下文理解。所以他们的上下文长度可以达到 256K，如果你要处理长视频，这非常重要。让我们谈谈视觉编码器。他们使用 SigLIP-2，这基本上是 SigLIP 的改进版本。它实际上架构相同，设计上向后兼容。然后他们改进了 RoPE。解释的方式是，以前你有三个维度：时间、宽度和高度。基本上，把第一块维度分配给时间，然后下一块给宽度，下一块给高度。而问题在于，如果你记得 RoPE，每个分量代表不同的频率。这意味着可能所有的时间维度都是低频的，而所有的高维度都是高频的。所以他们意识到这一点后，就基本上将它们交错排列。

[54:48.920] 现在所有轴都能接触到低频和高频。另一件他们做的事情是显式的视频时间戳。如果你仔细看这个——等等。好的，这是 Qwen-3。哎呀。所以如果你看——这可能有点太小了。但以前，时间戳是隐含在位置编码里的。视频中的每一帧本质上都会得到不同的表示——因为位置不同，自然就包含了时间的概念。但在这项工作中，他们将时间显式化了。所以这些实际上是标记，零秒是一个标记，现在你有了一个代表时间的标记。他们发现这很有帮助，大概是因为你可以直接引用它，比如“两秒之后发生了什么”。所以这是主要的改动之一。他们还做了这种按每个词元平方根归一化的损失。这样做的想法是，一些涉及视频的样本非常长，而一些涉及单张图片的样本则非常短。

[56:20.460] 通常的做法是每个词元都得到相同的对待。而这将意味着视频样本会占据主导地位。所以他们基本上——细节不太清楚。但我相信他们是按每个样本长度的平方根进行归一化，以降低超长样本的影响。最后一件值得注意且有趣的事情是，他们在适配器上做了一些有意思的工作。还记得我们说过，适配器是连接视觉编码器和语言模型的部分。LLaVA 是最简单的，只是一个线性投影。然后变成了 MLP。接着又有人用了交叉注意力。而 DeepStack——这实际上是 DeepSeek 团队的一篇论文——在这里使用时则更为复杂。他们注意到，视觉编码器本身已经计算出了一堆视觉嵌入。

[57:23.350] 我们基本上会将这些嵌入直接添加到语言模型的残差流中。好的，所以这更像是一种将视觉编码器深度融合到语言模型中的方式，而不是把视觉编码器当作一个只输出向量序列的黑盒。好的，所以训练是这样的——他们有两个阶段。预训练实际上有四个阶段。后训练有三个阶段。所以你可以看到，这些流水线现在已经变得相当复杂了。他们首先训练适配器。这和所有这些模型的做法都一样。然后他们有三个阶段，逐步在越来越长的序列上训练，从 8K 到 32K，再到 256K。大部分词元都在阶段 2 和阶段 3 中。然后对于后训练，他们对长思维链数据做 SFT，进行知识蒸馏，然后是一些强化学习。

[58:43.730] 到了这个时候，这确实是一篇系统工程论文了。有很多——我强调了核心的新想法，但还有很多细节是不同的。如果你看最终结果，这是一个相当不错的模型。我的意思是，如果你看这些基准测试结果，与闭源模型 Gemini、GPT-5 和 Opus 4.1 相比，这些数字是——加粗意味着它是那一行的最佳数字。所以 Qwen 模型实际上相当强大。好的，也许——嗯，有个问题。[听不清] 是的，所以问题是关于视频生成的，你怎么知道应该是视频还是文本？首先，这些模型不生成视频或图像。所有多模态的部分都在输入端。你始终生成的是文本。而且你始终处于大多数阶段中——除了 RL 之外——你是在监督每一个词元。所以没有 LLM 作为评判者来判断描述的质量好坏。

[01:00:32.220] 基本上就是你数据集里有什么就是什么。现在，有了 RL，你可以尝试不同的奖励函数。是的。我想问的是，从系统角度来看，训练多模态模型是否比训练纯 LLM 更难？[听不清] 是的，所以有两个问题，训练多模态模型在系统层面是否更难？我的意思是，当然不会更容易。我想你指出了数据集更大这一点。视频数据就是——哪怕是加载它都可能成为瓶颈。而我们之前讨论语言模型时通常没有关注数据加载的问题，因为这非常非常廉价。所以这需要被考虑进去。所有那些确保数据加载与实际计算异步进行的原则都需要被考虑进去。第二部分是——抱歉，第二个问题是什么？[听不清] 哦，对的。

[01:02:03.560] 所以视频和图像可以有更多——尤其是视频有更多的词元。所以这就是我们看到的那个归一化的原因之一，以确保我们不会给视频过多的权重。我的意思是，你总是可以降低权重。所以如果你有更多词元，但你不喜欢这样，你可以降低它们的权重。我们上次或两周前讨论过数据混合。所以你总是可以根据什么合理来给不同数据加权。但文本词元也有很多。所以我认为，一般来说，如果你看——是的，这些模型是在数十万亿词元上训练的。我认为有一个——是的，我不会说多模态词元的数量远远超过文本词元。嗯，哦，我想是你先举手的。[听不清] 所以问题是，当你做这种对齐时，它怎么知道语言——这个语言模型是预训练好的吗？

[01:03:31.350] 它肯定必须是预训练好的，否则对齐就没有意义了。所以语言模型是冻结的。你只是在训练适配器来连接给定的视觉编码器和给定的语言模型。训练的方式是，你选择一个词元预算，比如 670 亿词元，然后就直接训练。这里没有自适应的阈值。是的。[听不清] 是的，所以问题是，视觉编码器的参数量是否小得多？答案是肯定的，一般来说是这样。我试图找到——哪里还有——不在这儿。好的，原因是视觉编码器在某种意义上做的是非常局部的操作。它在看一个块。块非常小。它只是在试图理解这些块。没有太多知识在里面。所以对于块，我们不做推理。模型的大部分能力仍然在语言模型中。我试图找到——好的，在哪儿？

[01:05:19.240] 也许在上面这里。好的，我找不到——哦，是的。所以如果你看 LLaVA 模型，整个——720 亿参数的语言模型。抱歉。什么——好吧，我不知道为什么这个数字这么——哦，抱歉。这是 7200 万，好的。所以这个小得多。投影器也小得多。而 VIT 通常不到十亿参数。是的，我想主要是因为这是一个相当局部的操作。也许这是最简单的说法。好的，我再想一些其他的问题。我现在想继续往下讲了。所以 Qwen-3 是最后一个视觉语言模型。我会谈谈这个 SOTA。这里有很多数据方面的工作。在后面的这些 Qwen 模型中，关于数据和数据混合的细节不是很多。所以你需要去看 LLaVA 论文或者——AI2 有一篇[听不清]的论文，我没来得及讲，那里面有更多细节。

[01:06:49.420] 从 Qwen 到 Qwen-2、Qwen-3，这些变化大多数是——有一些变化。但整体框架保持不变。主要就是扩大规模、整理更多的数据集、注意到你的上下文不够——你必须处理长上下文，可能还要优化图像处理等等。好的，我现在要跳到一些相当不同的内容。这是 Meta 在 2024 年发表的一篇名为 Chameleon 的论文。到目前为止，VLM 将图像编码成向量，然后把它们注入到语言模型中。而且因为这是一个语言模型，你只能生成文本。你无法生成图像。所以有很多方法可以解决这个问题。你可以直接用一个VLM。你也可以接一个扩散头。这样你现在就能生成图像了，或者——但我想在这里讨论的这个想法，就是Chameleon提出的：如果我们把一切都映射成离散词元会怎么样？

[01:08:10.450] 在某种程度上，从美学角度看，这——嗯，也许这反映了我是个做语言出身的人。这很有吸引力，因为现在你可以用同样的方式分析和生成图像了。一切都是离散词元。如果你想做什么——比方说，你想生成图像。你可以说，这里有一段文本提示。然后你就可以直接——这些只是你可以生成的词元而已。或者你也可以把语言和文本的顺序调换过来。他们论文里有这些例子，你可以提示说“我无聊了，能给我看些鸟吗？”然后它就会输出文本，然后是图像，接着是更多文本和一些图像。所以它是交错进行的。所以从某种意义上说，全模态模型的愿景就是让文本和图像真正生活在同一个空间里。而这是通过让一切看起来都像文本来实现的。

[01:09:15.000] 这是实现它的一种方式。好，那么我会稍微讲一下这是怎么做到的。你现在需要做的事情就是把图像映射成离散词元。所以有这个比较老的想法，来自Oord 2017年的工作，叫做向量量化变分自编码器。这里的基本思想是把图像映射成离散编码。你学到的是一个映射，把它映射到一个连续向量上。然后这个向量会被舍入到最近的编码。所以你有一个码本，假设里面有8000个编码。这些是与图像块对应的原型向量。然后你选择其中一个最接近的。那就是你的表示。这就是你的编码器。它经过一个连续阶段，然后你把它舍入到最近的编码。现在解码器做的就是取那个编码，然后尝试重建图像。

[01:10:29.800] 然后你通过基本上最小化重建损失来训练这些VQ-VAE。你还会加一些其他项，因为这不是可微的，但我没时间深入讲这些了。好，但最终的结果是，512乘512的图像被转换成1024个词元，每个词元来自一个大小为8000的词汇表。好，现在你基本上就有了文本。你有你的图像，它们基本上就是这些编码。他们训练了一个新的分词器，因为现在你有不同的——你的数据看起来和纯自然语言不一样了。训练过程实际上变得非常简单直接。这只是普通的语言模型训练。没有适配器，没有微调视觉编码器，现在只有一个语言模型。所以这是它有吸引力的地方。所以仍然有两个阶段。但即使是语言模型训练，通常也有两个或更多阶段，第一阶段是训练的主体部分。

[01:11:53.690] 它基本上是无监督的。他们同时有纯文本和文本加图像的数据。然后在第二阶段，他们掺入高质量数据。好，这确实很棒很优雅。但这有几个问题。一个是他们发现训练不稳定。原因是文本和图像，尽管占据同样的空间，但行为方式非常不同。所以仅仅称它们为离散词元并不能掩盖那里住着一个图像的事实。特别是，文本词元，如果你想想预测下一个词，它的熵相对较低。大多数词是可以预测的，而图像词元有非常高的熵。我不知道这个特定词元会是什么蓝色调。所以结果就是，他们发现用这种混合数据进行训练导致了很多参数范数的增长，并产生了损失不稳定性。他们通过QK归一化和z-loss正则化在一定程度上缓解或修复了这个问题，这些方法控制了范数的增长。

[01:13:16.870] 我甚至不打算展示结果。但我想强调这项工作，因为这里有一种特定的优雅之处，因为它就是一个模型，真正以同样的方式对待所有模态。但缺点是这个模型最终表现并不那么好，而且离散化确实会丢失信息。所以想想OCR。再次强调，如果你把非常小的字迹离散化，你就再也无法读取它了。最后，用多模态进行训练也很棘手。我们在Qwen模型中稍微看到了一点这个问题，我们不得不调整权重。但这个问题在这里更为严重。所以VQ-VAE曾经流行过一段时间。事实上，很多人用它来做图像生成。主要原因是，你有一个Transformer。你怎么从Transformer生成东西？嗯，你必须生成离散的东西，你基本上把所有数据都放进这种离散形式里。

[01:14:28.510] 但后来扩散模型出现了，并且在生成方面变得流行和可行。所以这种风格的方法不像以前那么流行了。好，那么也许在这里做个回顾和总结。当今的前沿模型被期望是多模态的，甚至更强烈地说是原生多模态或全模态模型。不幸的是，我想当Gemini或GPT出来的时候，他们吹嘘或者说曾经吹嘘说是原生多模态的，能处理所有这些模态。事实上，它们确实做到了。但当然，没有关于这些模型是如何构建的细节。我认为它很可能是连续编码器加扩散生成的组合，因为——但这是我的猜测。我认为当你处理多模态时，根本的挑战是如何处理非文本模态。同样有趣的是，理解模态和生成模态之间存在一种不对称性。而且，比如说，没有一个通用的编码器。

[01:15:49.700] 例如，当我们看CLIP的时候，你只关心用于分类的、能捕捉高层语义的表示。所以这些向量可以相当小，并且能捕捉高层语义。而如果你想做OCR，或者你想生成图像，那么你需要非常细粒度的细节。你需要——这就是为什么扩散那么好，因为它可以真正微调优化低高频信息。我认为，一般来说，当你处理多种模态时，你必须仔细考虑如何正确地给它们加权。例如，视频的信息密度肯定比文本低。所以你不会想让视频压倒你的文本。就像我之前说的，至少目前最好的做法是使用连续编码器。看起来即使是CLIP，尽管它已经五年了，或者类似的想法仍然是捕捉图像语义的首选方式。

[01:17:03.630] Transformer仍然在那里。然后扩散模型，我没有讲到那个，它们很适合做生成。好，那我就在这里停下来。这就是关于多模态的讲座内容。我想我们没有布置关于这方面的作业。但如果你好奇的话，我想我鼓励你去试试训练一些这样的模型。

CS336 2026 · Lecture 17 · Alignment - multimodality

视频：官方视频
字幕：en-US · manual captions
处理：连续字幕合并为自然段；每段保留首条字幕的起始时间戳
状态：原语言字幕稿，待校对

正文

[00:05.000] So originally, the plan was to talk more about reinforcement learning. But I thought since I only had one lecture to give, I think this class would be a little bit incomplete without saying something about multimodality, which is so pervasive if you look at all the major models. So this will be an overview of multimodal models. Of course, this could be a whole class in itself, and I'll try to do what I can in one lecture. OK, so let's dive into multimodal models. So far in this class, we have exclusively focused on language models. And language models are already pretty general. They give you the ability to go from any piece of text to any text. So this could be any language, code, poetry, anything you like, even, let's say, DNA, for example. But the world is multi-modal. You have text, you have images, audio, video.

[01:05.260] And if you think about a North Star of where we want to be is what people call omni model, which is ability to take any combination of these modalities and output any combination of any of these modalities. So, for example, I can give you an image and a video and I can ask a question about the image and the video. I can generate images. I can convert the audio into an image. You can imagine all sorts of different applications by having an omni model. So in this class, I'm not going to build up to a whole omni model. But I'll give you some of the pieces that at least to allow you to ingest a little bit of images, which we'll mostly focus on today. So how do you build such a thing? Well, this is a weird way to motivate this, but the reality is that transformers work really well. And despite the best efforts of people to try other things, those are across all of these modalities still at scale the best thing we have.

[02:15.690] So we have to figure out how to use them. And now, transformers were designed for text. So they have this property that they speak tokens. So they take as input a bunch of tokens and then output some tokens. And here, I'm expanding the notion of tokenization to be not just a discrete token that we've seen in text but also potentially continuous tokens. You can think about these as embedding of tokens. And the key thing is that a token should represent some semantic unit of information. For example, in natural language, tokens are subwords. And these are somewhat meaningful. Whereas, for example, a pixel is certainly not meaningful by itself. So somehow we must convert everything, including audio and images, into either discrete or continuous tokens. Now, notice that we've already had to do this even with text. In the first lecture, we talked about tokenization. And there, it was not too bad. I mean, we had a BP tokenizer.

[03:26.750] You guys implemented it. It's fine. One could wish for a better tokenizer or something that doesn't have tokenizers at all. But it's not the worst thing in the world. Now, for non-text modalities, we have to scratch our heads a lot more and figure out what is the equivalent of the BP tokenizer that will take an image and produce things that a transformer can digest? OK, so you have to answer two questions here. One is, for non-text data, for example, images or videos or audio, how do you input these into the transformer? And then how do you generate them out? And this lecture will mostly focus on question number one. OK, so let's start back and rewind the clock back to 21. So the CLIP model, which some of you might be familiar with, is a lot of the foundation of modern VLMs. And so we can go through this a bit in more detail. So CLIP stands for Contrastive Language Image Pre-Training.

[04:43.500] And a bit of historical context is around that time GPT-3 had already happened. GPT-2. All the language models had already gone into this foundation model era. Neovision was still-- there were some efforts, but traditionally had been based on large annotated data sets, such as ImageNet, and training models, such as the ResNet, on these data sets and doing a bunch of other things, like data augmentation, and getting really good performance. And so the researchers at OpenAI were wondering, is it possible to leverage the large amount of image and textual captions out there? So inspired by this idea in language where you could just scrape the internet, you have all this text, it's very noisy. But somehow if you train a large enough model, you can make sense out of this and generate something useful. So what was the equivalent of that for images? So as a result, CLIP was born. And the idea of CLIP is fairly simple and straightforward. So the idea here is that suppose I give you a bunch of image-text pairs.

[06:05.980] So let's say I give you 32,000 of these. OK, so what I'm going to do is for each image I'm going to encode it using an image encoder, which I'll explain later, into a vector. So I'm going to have 32,000 of these image encodings. I'm going to also do the same thing with the corresponding text. So that will give me T1 through TN. So these are the embeddings of the text. These are embedding of the images. Now, the objective I want to set up is that I want the image-- let's take I1. So I1 is aligned with T1. So I want this alignment, the dot product between these two embeddings to be much larger than the dot product between I1 and all the other texts, OK. And conversely, I also want for this text to say that this dot product is larger than the dot product between T1 and all the other images. So that, in a nutshell, is the CLIP objective. And you can imagine doing-- there's basically 2 times N different softmax classification problems.

[07:30.350] So here's the code. So you get a n by d matrix for the image encodings and n by d times for the textual embeddings. You normalize and you take this dot stop product with some temperature. And then you compute the cross entropy, OK. So it's basically a multi-class classification problem, where the examples are structured in this m by n matrix form. OK, any questions about that? All right, so some more details about how CLIP was built. So where does the data come from? There are not too many details in the paper about the data. Roughly, you take a bunch of queries from somewhere and then you go search online for these and you mine a bunch of image and text pairs. And this resulted in 400 million image-text pairs. The data set wasn't released, but there was an effort by this paper called OpenCLIP, which replicated and further extended the CLIP ideas. So there was a data set called LAION-5B, which was released 5 billion images with textual descriptions.

[09:16.350] And there was a bunch of processing that happened. Interestingly, they actually used CLIP for the data filtering and then train OpenCLIP. So there's some bootstrapping happening. But at least this is a model where you can point to the data set that it was trained on and the code as well. So a few other details. So images come in all sorts of resolutions. If you find an internet image, it might be long and skinny or tall-- sorry. Long and skinny or the other way. So arbitrary dimensions. And one thing that you learn about neural nets is that they don't like things to be dynamic. They want things to be fixed size. So there is a somewhat heuristic processing that happens is they resize the image so that the shorter side is 336 or whatever target size. It could be 224 as well. And then they center crop to get a square. OK, this is obviously for convenience.

[10:29.660] Later, we'll see that you can do better than this. But this is just for expediency. And also at this time the CLIP authors were thinking about ImageNet and classification. So usually, the object is in the middle and you're just trimming off some background. So it doesn't matter too much. So what is the vision encoder? So this is the general algorithm. But I haven't specified what the image and text encoders are. Yeah. [INAUDIBLE] So the question I think is why they train on image-text pairs as opposed to images. [INAUDIBLE] So there is a line of work that tries to encode images into representations based on data augmentation. So, for example, if you take an image-- and I think this idea called SimCLR. You augment it. You crop it. You randomly perturb it. Rotate it. And you want to say that that is the same image.

[11:56.850] They should have similar embeddings. And that's useful for low level details. But the problem is that you won't data augment your way from one type of dog to another dog. So by using the text, it gives you a higher level semantic representations of images. And, yeah, that's a good question. And later, we'll see how these semantic representations are useful if you look at the types of tasks that people are trying to solve. OK, so what is the vision encoder that CLIP uses? They actually experimented with a bunch of different ones, ResNets and vision transformers, which had just come out. They found out that vision transformers performed the best. So when people say CLIP, they usually mean the VIT version. And so a vision transformer does the following. So you take an image and you break it up into these patches. I think the original paper was 16 by 16 and the CLIP used 14 by 14.

[13:18.850] But anyway, you have these little patches, which are basically vectors. So each patches in some sense a-- well, a token for this vision transformer. You add the positional embeddings just like you would if you were training a language model, and then you just put it through a standard transformer. And if you want to classify, then you would-- well, you would usually want to-- OK, sorry. The transformer encoder produces a sequence of vectors. And the CLIP paper does something a little bit different. You can average all these vectors. That gives you a single vector. But they thought it was better to do attention pooling, which means that you take the global average of all the activations, but then you also do another round of attention with that query against the key and values of each position. And that gives you another vector, which is maybe a little bit more informed rather than just a straight up average.

[14:37.080] OK, so that's the VIT. And the CLIP paper found that the best model is what is called VIT-L/14. So to decipher, this is a large VIT. I think it has 24 layers or so. Not quite sure about that. And 14 by 14 patches. Each patch is RGB, so it has three channels. And they trained on 336 by 336 resolution images. Actually, they trained on a lower res I guess for speed because high resolution images take longer. And then for the latter part of training, they train on higher res. Yeah. For the position embedding, would it be more complicated than the task? For example, task only has [INAUDIBLE]. Yeah, so the question is, can you be smarter about the positional embeddings because this is just linear 0 through 9? I think in this paper, they actually-- in the CLIP paper, I believe they tried some 2D version of this.

[15:57.500] And the one DNA found that it doesn't really matter that much. I think it's also-- I always have to take these results with the idea that they're interested in classification in mind. So for classification, maybe it doesn't matter. Later, we'll see a fancier positional embeddings that do take into account the spatial structure. OK, so the text encoder is a standard transformer. It's a GPT-2 style transformer since that was from the same group that developed GPT-2. And in order to get a single vector out of a sequence, they prepended BOS and appended EOS. And then they took EOS, the activation at the highest layer as the representation of that whole sequence. OK, so you have the vision encoder. You have the text encoder. And then you can do this process to train. You pick a batch. You encode all the text. You decode all the images. And then you form these two end cross-entropy losses. And then you go from there.

[17:18.270] OK, so the headline result, which got a lot of people excited about CLIP back in 2021, is that on the ImageNet benchmark, 0 shot CLIP outperform a ResNet that was trained on 1.2 million ImageNet images. So if you think about this, 1.2 ImageNet images, this was many, many hours of Amazon Mechanical Turk worker time to do this annotation. And now you have CLIP that was trained on more organic web data. Of course, this was the labor of a bunch of people on the web. But if you're able to leverage that or existing data already, then you can just zero shot it. And the zero shot technique is basically to explain I don't have slide support here.

[18:10.870] But you take an image and then you have a bunch of text. And then you basically take the dot product with the various labels and see which one's the highest. Oh, yeah. It is. Sorry. It is on this part of the slide. OK, good point. So this is a zero shot prediction, where you take an image and then you-- yeah, what I said. Cool. Any questions? Yeah. [INAUDIBLE] Yeah, so the question is if you have a dog and there's maybe other captions that also have dogs, would that confuse the model? In general, this process is going to be noisy. It's OK if there's another dog because, on average, it's unlikely that there's always going to be a dog. Sometimes it's going to be an apple or a cat or something. But also another point is that these images are basically scraped from images on the web with textual text next to them or in the alt image. And so they're extremely noisy. There's been studies that look at when you have a caption of an image, it doesn't necessarily just verbatim say what's in the image.

[19:48.547] Because if you have an image and it's a dog, you don't need to say a dog. So this process is very noisy. So it's maybe somewhat surprising but also interesting that it works. I mean, there was a lot of data filtering that is needed. So if you take arbitrary images on and text on that web, it's probably going to be too way too noisy. One other thing to to a point I'll come back to at the very end, is they also try this alternative where instead of doing this ranking, they can predict a text from the images. So they can set an objective where they try to take the images and predict the text, either as a bag of words or as a language model. And they show that maybe, I don't know, somewhat surprisingly, that if you use a stronger model, it actually does worse or at least is less efficient compared to using bag of words, CLIP. So this is saying that-- I think saying that for actually the representations that you're trying to learn-- or, again, this is ImageNet accuracy.

[21:09.120] So it's classification. Actually modeling the exact token sequences of the caption isn't so important for getting the rough representation of the image. OK, we'll come back to this point at the end. So what did we learned so far? So we've encoded images using CLIP. And it captures the semantics of the image because it's paired with text. And generally, text talks about the semantics. It is important to note that the design decisions here are based on image classification. So it's not very fine-grained. But yet, it's still, as we'll see, serves as a robust starting point for everything we're going to do later. One technical downside of CLIP is that it does require large batch sizes, like 30,000. If you have a batch size of 1, clearly, it doesn't work, or even 10, it doesn't work. And furthermore, the softmax operation operates over the full batch. So it's not really very decomposable.

[22:22.970] Whereas if you think about normal language model training, a batch of examples, all the sequences parallelize and you just do aggregation at the end. So this point will be addressed by another paper from Google that came out called SigLIP. And so think about this as basically an improved version of CLIP. So it stands for Sigmoid Loss for Language Image Pre-training. And here, the main difference is that CLIP does multi-class classification. It says, I have this aligned text and image. And I'm going to say this is positive against all the other alternative images or against all the alternative text.

[23:13.930] SigLIP, in some sense, is a lot simpler. It basically says for any given image text pair, are they aligned or not? So if I reference this diagram up here, it's basically binary classification where the diagonals are positive examples and the diagonal are negative examples. That's it. So here's the algorithm. You take the embeddings. You normalize. You take the dot product as usual. But now you're forming the labels which are minus 1 on the off-diagonal, and diagonals you have one. And then you do log_sigmoid. So the objective is very simple. Yeah. [INAUDIBLE] Yeah, so the question is, does this require any sophisticated sampling strategies? At least in the initial paper, they were just operating on literally the same type of matrix. You could imagine I think you're probably thinking in general for these contrastive methods.

[24:37.840] The negative examples need to be-- sometimes you need to balance the positives and the negatives, or the negatives need to be like tight negatives so that you're not biased. But at least in the initial version, this was fairly straightforward. So to speak about the data, this was done at Google. This was the data set. They used another paper, which was another language image model around 2022, which got a WebLI data set with order billion image text pairs. They also did some extra work. For example, images that have text in them. They did OCR. And that's another way to form text image pairs and did some filtering. This was multilingual. The main point of this paper was that it was much more efficient to train than CLIP. So CLIP was trained 10 days on 256 TPUv3, and CLIP was five days on 32v4s. And you might think, oh, v4s actually should be faster than v3. In terms of flops per second, they're actually not.

[25:53.140] They're better because you can put more of them in a pod and the interconnect is better. But at this scale, it's really actually not faster. Actually, 60% slower or something. So the way that this is faster-- and, of course, I think CLIP probably did not try to optimize the code for the maximum throughput. It was just let's get this thing to work. But the SigLIP paper showed that you can parallelize this as follows. So you have-- basically, if you recall your systems lecture, so this is you can think about it as DDP if you like, where each device stores a subset of the image text pairs. However, there's interactions between the examples, unlike in language model training, where everything just factors. So in the first pass, each device just computes all the losses on whatever image text pairs it has locally. And then you send the text. So basically, device 1 gets T5 through T8.

[27:24.570] And now it's able to compute the negatives here. And then it gets T9 through T12. So you rotate around until you cover all the off-diagonal block entries. Another thing that was nice about SigLIP is that you effectively decoupling batch size from the loss. Whereas in CLIP, the loss is tied to the batch size. If you change the batch size, it's a different loss function. So they were able to experiment with much smaller batch sizes less than 16K. And in that regime, it's much better than CLIP because the loss function, if you have too many too small a batch size, it just degrades. Whereas for smaller batch size for CLIP, you have more variance but it's the same in expectation. It's the same loss. They also say, well, you can go up to much larger batch sizes. But that doesn't really help. And we saw that critical batch sizes at some limit.

[28:30.670] And 32K was essentially their critical batch size. OK, so now we have CLIP and SigLIP. These are image encoders that take an image, in this version, just a fixed size image, 336 by 336, let's say, and maps it into a vector, OK, which contains some semantics. So now let's start building VLMs, Vision Language Models. And I'm going to talk about two families of models, LLaVA and Qwen. The two are very similar in the broad template. There are some details that are different, which we'll go through, and the basic idea is that we're going to take these embeddings and then just inject them into our language model. So this is going to be more of a flavor of mid-training or post-training, where we take an existing image encoder. We take an existing LLM, and then we're going to stitch it together rather than training something from scratch. So LLaVA paper came out in 2023.

[29:46.680] And this got people excited because around this time, the closed models, like GPT, I think it was GPT-4, was able to do visual reasoning. And they were able to show that they could do some visual reasoning as well. It wasn't as good as GPT-4, of course, but I think it was an open model and people got to see what went under the hood. So for VLMs, there's essentially a few pieces here. There's the vision encoder. They use CLIP here. For the text decoder, they use a language model called Vicuna, which is the first Llama model that was fine-tuned on some shared GPT conversations. So these are conversations that people have with ChatGPT and share them on this website.

[30:38.400] I don't think this is up anymore. So the data that they used was based on MS COCO. So this was an annotated data set where they had Mechanical Turk workers go and annotate a bunch of images with bounding boxes and captions describing what was in the image. So they basically synthesized a data set for their model. So they prompted GPT-4 with the captions or the detected objects and GPT-4 to generate questions or conversations. For example, this is an image in the MS COCO data set. This was an annotated caption done by a human. And also here is also annotated by a human. And then you ask GPT-4, generate me a conversation and you might get a question and an answer. You might ask GPT-4 generate me a detailed description. It generates something, I guess, similar to the caption. And then you can ask GPT-4 to generate complex reasoning.

[31:50.980] And it does something like that. OK, so those are synthesized data, 158,000 examples. And then they trained a model on this. So the model-- so they use CLIP, in particular, the highest performing CLIP version, VITL/14. So 14 by 14 patches. And then they-- let's just look at this image. So they take images. They send it through this vision encoder CLIP. That gives you a vector. But this vector isn't really in the same space, so to speak, as the text. So then they multiply by a matrix W to get another vector, which is in the same space as the text embeddings. And for text, you basically just use your standard embeddings to get vectors. So what's basically happening is that the text gets encoded into these vectors and the image also gets encoded into these vectors. And then this whole sequence of vectors just goes through a standard transformer and produces output.

[33:11.682] OK, so we're in some sense converting these images into textual tokens. So I can leverage the pre-trained language model. So to train this model, they do it in two stages. The first stage is that they freeze the vision encoder and the language model and only train this matrix W. And this is what they call the alignment phase, which is essentially making sure that the image gets mapped into the same space. Because if you just use a random W, these images are not going to be-- I mean, certainly, they're not embeddings, representing any natural language token. So the goal of training W is that they look like natural language token embeddings. And the second stage is that they now still freeze a vision encoder and then train W and the language model. So they're fine-tuning the language model on their data. So remember, their data is basically here's an image and here is a conversation or a description or a complex reasoning example.

[34:32.080] So this is all image plus text to text. OK, so this is one example from their paper. It's interesting. So you have an input image and a conversation. The user says what's unusual about this image. And their model is able to tell you that you don't usually iron on the back of a minivan. And they also make a point that their model is pretty good because even if your user prompt is not really prompting the unusualness, it still talks about it, whereas-- GPT-4, obviously, is able to do this, but other models at the time were not able to. OK, any questions about LLaVA? OK, so that was 2023. And I'm going to skip forward to LLaVA OneVision, which is 2024. There is a sequence of papers after the LLaVA paper came out of LLaVA 1.5, LLaVA-Next. But I'll capture all of these innovations, I think, in what I'll describe next. The main thing that they did is it's essentially the same recipe, but they are trying to be more ambitious in terms of the types

[35:59.120] of multimodal applications they could do. So they were able to handle multiple images and videos. So here's the same diagram. Now they can take a single image or multiple images or a video, which is essentially a sequence of images corresponding to a sampling of the frames. And here are the vision encoder. They upgrade it to SigLIP. The text decoder, they use Qwen-2 now, which was probably the best language model that was out at that time. And then for the projector or the adapter, which is the thing that takes the output of the vision encoder and turns it into the input of the language model is a two-layer MLP instead of a linear projection. OK, so it's like you have a system and you're just upgrading the parts. But it's the same rough system. So let's talk a little bit about data processing. So one thing that they wanted to do is to do OCR. And the thing with OCR is that you need to preserve very fine-grained information.

[37:20.130] Otherwise, a J looks like an I, and that's not good. So remember that CLIP resizes and crops to 336 by 336. And clearly, if you have a document and you crop to 336 by 336, you can't read it. So their solution is this idea called AnyRes, which was actually introduced in the LLaVA 1.5 paper. And the idea is basically, I mean, it's fairly straightforward. You break up an image into multiple pieces. And the idea is that each piece is the resolution of whatever the vision encoder is going to take, 336 by 336, for example. So you're going to code all the pieces into a bunch of vectors. And then you concatenate the vectors. OK, so you're basically noticing that your vision encoder can't handle high resolution. So rather than downsampling, you're just going to crop and look at different parts of image. And now if you have really super high resolution images or videos then, you get too many tokens.

[38:31.330] So then they downsample. But the idea behind AnyRes is that it's adaptive. And this is nice because the transformer is already adaptive. Language can be-- your sentences can be any length and the transformer already handles that pretty well. So it turns out images can be any resolution. So that's picking back on the same dynamic ability. So here is what they actually do for images here. So let's say you have this image of a paper. They do downsample and have one path, which is basically a downsampled encoding of the entire paper. But then they break it up into these chunks, where each chunk is the resolution that the vision encoder expects, and then encodes each of these separately. And then you concatenate all these patches. And if you have too many, then you reduce that the number of patches by interpolation. OK, so then they're able to handle images, multiple images, and video.

[39:43.560] So you think about, well, all of these are technically all reducible to images, but they put their thumb on the scale a little bit because they want to make sure all the modalities are roughly comparable. Because videos can be very long, they don't want their data set to be dominated by a bunch of repetitive frames. So here's what they do here. So for a single image, they basically have the full image downsampled and then a bunch of crops up to N crops there, up to 9. And if you are presented with multiple images, then they just use the base resolution. So if you have a single image, I get to look at it more carefully. If I have multiple images, I'm just going to look at it from afar. And for video, we're going to use even lower resolution or fewer tokens to represent each frame because the idea is that, well, videos can be-- they only use up to 32 frames, but you start running into context length problems for viewing.

[41:01.280] And later, we'll see how a big part of being able to handle multimodal is to deal with longer context. OK, so the data here that they use is, I would say, the state of philosophy is they want to curate high quality data and quantity. Another way to interpret this is that it's very targeted data. If you look at many of these, it's very task-based whether you're doing visual question answering or answering questions about tables. So this is definitely post-training territory, where you're trying to get your model to do these tasks, and therefore, you create a bunch of these tasks. And this work is also unabashedly, basically, distilling GPT-4 models so that they can get the best performance, which is, I guess, not ideal. But this is what you do if you don't have an annotation budget. And so they have single images, multiple images, and videos.

[42:20.390] So there's a lot of different pieces. Some of these are very specific. You give two images and you're trying to spot the difference. But one thing that-- OK, I'll come back to this at this point later. So for training, it's roughly the same idea as before, where the first stage is that you're focused on alignment. So in particular, you only train the projector, which is the adapter. And then in the second stage and the third stages you train the full model. And before, we had two stages. Now, we have three stages. I'm not sure there's any particular principled reason for this, except for this second stage is trying to put high quality data but more focused on knowledge. And the final stage is focusing on examples that look like your downstream tasks. OK, so one thing that they found in this paper, which was interesting, is that you get transfer between modalities. So even though they have only single image data for diagrams and charts, but this actually generalizes to multiple images.

[43:49.070] So at training time, you never saw an example where you have a table and a chart and you're asking questions about it. But at test time, you have two images, one for the table and one for the chart. And you're able to have some of conversation about it. At training time, you only have OCR data on single image data, and you only have relational reasoning from multiple images. But you can generalize to these cases which are useful for powering GUI agents, where you show it a bunch of screenshots. This is multiple images and it can generate a description of the screenshots. Another example is that they have this idea of visual prompting, where you have this circle, which is a part of the image, but that's meant to highlight the part of the image that you're supposed to do something about. And this type of data only exists for single images. And now it can generalize to videos where the user says describe the player highlighted in the video. And this player is across multiple frames. So when you first look at this and say, oh, boy.

[45:11.940] You're basically targeting each of these tasks. It's supervised learning. But if you have enough tasks, these models seem to do some transfer, which is, I guess, reassuring. OK, so again, this is a fairly standard VLM template, vision encoder plus the projector into the language model. There's a lot of work that goes into the data curation. They lean pretty heavily into synthesize task-specific data. The nice thing about the LLaVA series is that it's one of the few works that open sources. They're not just the model weights, but also the data. So you can really replicate and study this stuff. OK, now let's talk about the Qwen Models. So Qwen started also training in 2023 multimodal models. So the first version was Qwen-VL. And hopefully, I'll go through this quickly because you hopefully see the pattern.

[46:19.780] The vision encoder, they use Open-CLIP. Remember, OpenCLIP was the open reproductions of the CLIP model. So it's basically a CLIP encoder. And for the adapter, they did one layer of cross-attention, incorporate 2D positional embeddings and maps to a fixed size of 256, which is a little bit-- I guess this will be changed later because this is definitely not very dynamic, but neither is the vision encoder at this point. I guess this got cut off. But the special tokens, they have an image tag and a box tag for bounding box, and a ref tag for descriptions. This is what happens when your slides are in HTML. So for training, they have three stages, just like with the LLaVA models.

[47:19.830] In the first stage, the idea is to-- I mean, they call it pre-training but you're not pre-training from scratch. This is generally large-scale, low quality data. They freeze the language model and train the vision encoder and adapter. So this is a bit different from the LLaVA models, in that, they train the vision encoder at this point. And you can see some of the data sets that they are training on, 1.4 billion examples. Stage 2, they move to higher quality task-specific data. And you can see some of the usual suspects of VQA data sets, chart question answering, and so on. And then they train all the parameters here. And finally, stage 3 is instruction tuning data. You freeze the visual encoder and you train the adapter and the language model. Here are some examples of the type of things that they're able to do. So, of course, this is Qwen. So some of these examples are in Chinese.

[48:37.570] Their interest is also making a model that is supposed to be subsuming a language model. So a language model can do code. So we want the vision model to be good at code as well. And you can have this example, if you can see it, is an image. And can you find Spider-Man and the Hulk. And it can output bounding boxes as well. It doesn't actually output an image. It just outputs a bounding box. And then you can do OCR. You can compare-- so usual stuff. OK, so then Qwen-2 was, again, a upgrade. There's some new ideas here, which I'll go through. So use a larger vision encoder. They also started doing dynamic resolution. I think this was the main thing. We remember when we went from LLaVA to LLaVA OneVision, they realized, oh, we need to handle images of different sizes. And if you try to do video, it's clear that you need some dynamic resolution.

[49:47.700] So the model is basically-- so let's say you have this picture here, that might be mapped to 11,000 tokens. This tiny picture of an equation might be only mapped to 8 tokens and so on and so forth. So how do they do this? Well, basically each-- it's the same idea AnyRes idea from LLaVA, where each 224 by 224 patch is encoded with a VIT. And remember the VIT that they started with is the OpenCLIP bit. But this gets fine-tuned. And then trying to reduce the contact size, they compress every 2 by 2 into 1 so that they ultimately edge every patch generates 66 tokens. For eight videos, they sample 2 frames a second but max out at 16,000 tokens.

[50:59.600] OK, so one thing coming back to this question about positional embeddings, so they did do something different here with Qwen-2. So they used this idea called multimodal rotary, multimodal RoPE. So remember, we saw RoPE before. And the idea behind RoPE is that you have embedding such that the inner product between the vectors depends only on the distance. So the distance is defined in 1D by just the number of tokens. You're away from it. So now, the multidimensional version is essentially the same thing. Except for now you have 3D. So you have height. You have width and you have time. So each position, each patch now is a triple defined by the coordinates. And then to compute the M-RoPE, basically, for every dimension, you compute the RoPE and then you concatenate. So it's a fairly straightforward idea. Although, later, with Qwen-3, we'll see how this is actually a bit suboptimal. So they initialize the language model with-- this is the Qwen-VL model.

[52:23.790] So, of course, they're initializing with Gwen. And then they have three stages of training, which is very similar to what we did before. And then they were able to show a bunch of different capabilities, some video understanding. It can do math and code, function calling, and so on and so forth. OK, so quickly, I'll talk about Gwen-3. So again, there's a few changes here. I wouldn't say these are structural big changes, but they're probably changes that do impact the quality of the model. So this is from last year Gwen3-VL report. And so it's roughly the same diagram here. I'll point out maybe-- I'll come back to this diagram. I'll go through some of the changes. So one is that now they're using Qwen-3 models. They have a series of dense and MoE models. And the Qwen-3 models are really, really good, and it really helps the quality of the final model.

[53:35.890] One thing that they're vested in is long context understanding. So their context length can go up to 256K, which is really important if you're trying to do long video. Let's talk about the vision encoder. So they use SigLIP-2, which is basically an improved version of SigLIP. It's actually identical architecture, is designed to be backward compatible. And then they improved their RoPE. And the way to explain this is, before, you had the three dimensions, time, width, and height. And basically, when to basically allocate the first block of dimensions to the time, then the next block to width and the next to height. And the problem, if you remember RoPE is that each component refers represents a different frequency. So this would mean that maybe all the temporal dimensions are low frequency and all the high dimensions are high frequency. So they basically interleave them when they realize that.

[54:48.920] And they now all the axes are exposed to both low and high frequency. Another thing they did was explicit video timestamps. So if you look closely at this-- wait. OK, this is Qwen-3. Oops. So if you look-- this is probably a little bit too small. But before, the time stamp was implicit in the positional encodings. Each frame in the video intrinsically gets a different-- there's a notion of time just because it's in a different position. But here, they made the time explicit. So these are actually tokens zero seconds, is a token that is now you have a token that represents the time. And they found this to be helpful presumably because it's something that you can directly refer to, like what happened after two seconds. So that is one of the main changes. They also did this square root normalized per token loss. So the idea is that some examples that involve video are very long and some examples that involve a single image are very short.

[56:20.460] And the normal thing is that every token is treated the same. And this would mean that the video examples are going to dominate. So they basically-- the details aren't too clear. But I believe they normalized each example by the square root of the length to basically downweight the impact of really long examples. The final thing that is worth noting, which is interesting, is that they did something interesting with adapter. So remember we started the adapter is the thing that connects the vision encoder to the language model. And LLaVA was the simplest. It was just a linear projection. And then we got an MLP. And then we got some cross-attention. And DeepStack, which is actually a paper from the DeepSeek team, but when used here is more sophisticated. They noticed that, well, the vision encoder already computes a stack of vision embeddings.

[57:23.350] And we're basically going to basically add these directly into the residual stream of the language model. OK, so this is a bit more of a deep fusion of the vision encoder into the language model as opposed to the vision encoder is a black box that just outputs a sequence of vectors. OK, so training is-- so they have two phases. Pre-training has actually four stages. Post-training has three stages. So you can see that this pipelines are getting quite complicated now. So they first train the adapter. This is the same as all of these models. And then they have three stages where they progressively train on longer and longer sequences, from 8K, to 32K, to 256K. Most of the tokens are in the stages 2 and 3. And then for post-training, they do SFT on long chain of thought data, knowledge distillation, and then some reinforcement learning.

[58:43.730] At this point, it's really a systems paper. There's a bunch of-- I highlighted the core new ideas, but there's a lot of details that are different. And if you look at the final results, this is a pretty good model. I mean, if you look at these benchmarks as compared to the closed models Gemini, GPT-5, and Opus 4.1, these numbers are-- the bold means that it's the best number in that row. So the Qwen models are actually quite strong. OK, maybe the-- yeah, a question. [INAUDIBLE] Yeah, so the question is what about video generation and how do you know whether there should be a video or text? So first of all, these models don't generate videos or images. All the multimodal stuff is on the input side. You're always generating text. And then you're always in most of these stages, except for RL, you're supervising every token. So there's no LLM as a judge or quality on how well the description is working.

[01:00:32.220] It's just basically whatever is in your data set. Now, with RL, you can play around with different rewards. Yeah. I'll ask is the training of multimodal models harder than training a pure LLM from a system perspective? [INAUDIBLE] Yeah, so the two questions are trained multimodal models harder on the system side? And, I mean, certainly, it's not easier. I think you pointed out that the data sets are larger. Video data is just-- even loading it can be a bottleneck. Whereas generally, we have not really focused on data loading for talking about language models because it's very, very cheap. So that needs to be taken into account. And all the same principles of making sure that your data loading is happening async with actual computation needs to be taken into account. And the second part was-- sorry, what was the second question? [INAUDIBLE] Oh, yeah.

[01:02:03.560] So the videos and images can have more-- especially video have more tokens. So that's one reason we saw this normalization to make sure that we're not giving more weight to video. I mean, you can always down weight. So if you have more tokens, if you don't like that, you can down weight. We talked in the last or two weeks ago about data mixtures. So you can always weight things according to whatever makes sense. But there's also a lot of text tokens out there. So I think, in general, if you look at, yeah, the models are trained on tens of trillions of tokens. I think there's a-- yeah, I wouldn't say that the number of multimodal tokens vastly outnumbers text tokens. Yeah, oh, I think you were first. [INAUDIBLE] So the question is, when you do this alignment, how does it know what the language-- is the model language model pre-trained?

[01:03:31.350] And it definitely has to be pre-trained because otherwise it doesn't make sense to align it. So the language model is frozen. And you're just training the adapter to connect the given vision encoder with a given language model. And the way you train it is, well, you pick a token budget, 67 billion tokens and you just train. There's not an adaptive threshold here. Yeah. [INAUDIBLE] Yeah, so the question is, is the number of parameters in the vision coder much smaller? And the answer is yes, in general. I'm trying to find-- where is the other place I-- not here. OK, the reason is that the vision encoder is in some sense doing a very local operation. It's looking at a patch. A patch is very small. And it's just trying to understand the patches. There's not much knowledge. So the patch, we're not reasoning. So most of the capabilities of the model are still in the language model. I'm trying to find the-- OK, where?

[01:05:19.240] Maybe it was up here. OK, I can't-- oh, yeah. So if you look at the LLaVA model, so the full-- so 72 billion parameter language model. Sorry. What-- OK, I don't know why this number is so-- oh, sorry. This is 72 million, OK. So this is much smaller. The projector is much smaller. And the VIT is generally less than a billion, parameters. Yeah, I guess mostly because it's fairly a local operation. I guess that's the easy way to say it. OK, I'm going to think some other questions. I'm going to try to move on now. So I Qwen-3 is the last vision language model. I'll talk about this SOTA, Save Our Performance. There's a lot of data work that happens. Not too many details about the data and the data mix in these later Qwen models. So you'll have to look at the LLaVA paper or the-- AI2 has a [INAUDIBLE] paper, which I didn't get a chance to talk about that has more details here.

[01:06:49.420] And from Qwen to Qwen-2, Qwen-3, many of these changes were mostly-- there's some changes there. But the overall framework remains the same. And mostly, it's scaling up, curating more data sets, noticing that your context isn't-- you have to handle long context and maybe sharpening your image processing and so on. OK, so I'm going to now jump to something quite different. So there's this paper called Chameleon from Meta in 2024. So, so far, VLMs encode images into these vectors and then they inject them into a language model. And because it's a language model, you can only generate text. You can't generate images. So there's many ways you can go about fixing this. You can just take a VLM. You can also attach a diffusion head. And now you can generate images or-- but I want to talk about this idea here, which is Chameleon says, what if we mapped everything into discrete tokens?

[01:08:10.450] And in some ways aesthetically this is-- well, maybe this reflects that I'm a language person. This is appealing because now you can analyze and generate images in the same way. Everything is a discrete token. If you want to do what-- let's say you want to generate images. You can say, here's a text prompt. And then you can just-- these are more just tokens that you can generate. Or you can flip the language and the text around as well. They have these examples from the Chameleon paper you can prompt in say I'm bored. Can you show me some birds? And then it would be text, and then it would be images, and then some more text and some images. So it's interleaved. So in some sense, the vision of an omni model is that text and images truly live in the same space. And this is accomplished by making everything look like text.

[01:09:15.000] That's one way to do it. OK, so I'll talk a little bit about how this is done. So the thing you need to do now is to map images into discrete tokens. So there's this idea rather old from Oord 2017, which is called vector quantized variational autoencoder. And the basic idea here is to map an image into a discrete code. And what you learn is a mapping that maps this into a continuous vector. And this gets rounded to the nearest code. So you have a codebook which has, let's say, 8,000 codes. So these are prototypical vectors that correspond to what patches. And then so you pick one of these that's the closest. And that would be your representation. So that's your encoder. It goes through a continuous phase and then you round it to the nearest code. And now the decoder is you take that code and you try to reconstruct the image.

[01:10:29.800] And then you train these VQ-VAEs by basically minimizing the reconstruction loss. There are some other terms that you add because this is not differentiable, which I won't have time to get into. OK, but, at the end of the day, you have 512 by 512 images that are converted into 1,024 tokens, and each token comes from a vocabulary of 8,000. OK, so now you have basically text. You have your images, which are basically these codes. They train a new tokenizer because now you have different-- your data looks different than if it were just normal natural language. The training is now actually very straightforward. This is just normal language model training. There's no adapter. There's no turn the vision encoder. There's just a language model now. So this is the part that's appealing. So there's still two stages. But even for language model training, there's usually two or more stages, where the first stage is the bulk of training.

[01:11:53.690] It's largely unsupervised. They have both text and text and image. And then stage 2, they mix in high quality data. OK, so this is really great and elegant. There's a few problems with this. One is that they found that the training was not stable. And the reason is that text and images, despite occupying the same space, just behave very differently. So just calling things discrete tokens isn't hiding the fact that there's an image living there. And in particular, text tokens, if you think about predicting the next word, this has relatively lower entropy. Most words are predictable, whereas image tokens have very high entropy. I don't know what shade of blue this exact token is going to be. And so as a result, they found that training with this mixture led to a lot of the norms of the parameters to grow and to generate loss instability. They were able to mitigate or fix this to some extent via QK norm and z-loss regularization, which controls the norm growth.

[01:13:16.870] So I'm not even going to show the results. But I wanted to highlight this work because there's a certain elegance here because it's just one model that treats really all modalities the same. But the downside is that it turned out this model was not really as performant, and the discretization does definitely lose its information. So think about OCR. Again, if you discretize very small print, you're not going to be able to read it anymore. And finally, training with multi modalities is also tricky. We saw this a little bit with the Qwen models and we had to play around with the weights. But this is more exacerbated here. So VQ-VAEs were popular for a while. And in fact, many people were using this for image generation. The main reason to do that is, well, you have a transformer. How do you generate from a transformer? Well, you have to generate discrete things and you basically put all your data into this discrete form.

[01:14:28.510] But then diffusion models came out and became popular and viable for generation. So this flavor of method is less popular than it used to be. OK, so maybe just reflect and summarize here. Frontier models these days are expected to be multimodal or even more strongly natively multimodal or omni models. Unfortunately, I think when Gemini comes out or GPT comes out, they tout or touted as being natively multimodal and handle all these modalities. And in fact, they do. But of course, there's no details about how these are built. I think that it's probably combination of having a continuous encoder because you don't want to lose information and then diffusion for the generation because-- but that's my speculation. The fundamental challenge, I think, when you're dealing with multimodality is how do you handle non-text modalities? It's also interesting that there's a asymmetry between understanding the modality and generation. And there's no one universal encoder let's say.

[01:15:49.700] For example, when we looked at CLIP, you only cared for classification to capture high level semantics. So these vectors could be fairly small and capture the high level semantics. Whereas if you wanted to do OCR or whether you wanted to generate an image, then you need really fine-grained detail. You need to-- that's why diffusion is so good because it can really micro optimize the low-high frequency information. I think, in general, when you're dealing with multiple modalities, you have to think carefully about weighing them properly. For example, video certainly has lower information density than text. So you don't want a video to overwhelm your text. And like I said before, at least the current best thing is having continuous encoders. It seems even CLIP, even though it's five years old, or similar ideas are still the go-to way to capture semantics of images.

[01:17:03.630] Transformers are still there. And then diffusion models, which I didn't talk about, are great for generation. OK, so I will stop there. That's it for the lecture on multimodality. I guess we don't have any homework on doing this. But if you're curious, I think I encourage you to play around with training some of these models.

Alignment - multimodality

课程简介 ​

CS336 Lecture 17：多模态模型（Alignment - Multimodality） ​

学习目标 ​

课程主线 ​

第一部分：背景与核心问题 ​

为什么要做多模态 ​

为什么用 Transformer ​

第二部分：CLIP（Contrastive Language-Image Pre-Training） ​

背景与动机 ​

核心方法 ​

数据 ​

视觉编码器：Vision Transformer（ViT） ​

文本编码器 ​

亮点结果与消融 ​

局限 ​

第三部分：SigLIP（Sigmoid Loss for Language Image Pre-Training） ​

核心区别 ​

损失函数 ​

数据 ​

训练效率 ​

Batch Size 解耦 ​

第四部分：视觉语言模型（VLM） ​

LLaVA（Large Language and Vision Assistant） ​

架构 ​

数据 ​

训练 ​

突出能力 ​

LLaVA OneVision ​

主要升级 ​

AnyRes 高分辨率处理 ​

三种输入模态的 token 平衡 ​

数据策略 ​

模态间迁移 ​

Qwen-VL 系列 ​

Qwen-VL（第一代） ​

Qwen2-VL ​

Qwen3-VL ​

关于 VLM 的常见问题 ​

第五部分：Chameleon——统一的离散 token 方法 ​

核心思想 ​

VQ-VAE 图像分词 ​

训练与稳定性 ​

评价 ​

总结与当前共识 ​

课程作业关联 ​

常见误区 ​

复习清单 ​

从CLIP到全模态模型：CS336 Lecture 17 多模态建模全景解析 ​

导语 ​

为什么必须处理多模态？ ​

基石：CLIP与SigLIP ​

CLIP：对比学习的开创性实践 ​

SigLIP：摆脱batch size的束缚 ​

构建视觉语言模型（VLM）：LLaVA与Qwen ​

LLaVA：初代开源的VLM模板 ​

LLaVA OneVision：升级与自适应分辨率 ​

Qwen-VL系列：走向工程化与规模化 ​

迈向全模态：Chameleon的离散化尝试 ​

课程定位与总结 ​

CS336 2026 Lecture 17：完整中文逐字稿 ​

正文 ​

CS336 2026 · Lecture 17 · Alignment - multimodality ​

正文 ​

课程简介

CS336 Lecture 17：多模态模型（Alignment - Multimodality）

学习目标

课程主线

第一部分：背景与核心问题

为什么要做多模态

为什么用 Transformer

第二部分：CLIP（Contrastive Language-Image Pre-Training）

背景与动机

核心方法

数据

视觉编码器：Vision Transformer（ViT）

文本编码器

亮点结果与消融

局限

第三部分：SigLIP（Sigmoid Loss for Language Image Pre-Training）

核心区别

损失函数

数据

训练效率

Batch Size 解耦

第四部分：视觉语言模型（VLM）

LLaVA（Large Language and Vision Assistant）

架构

数据

训练

突出能力

LLaVA OneVision

主要升级

AnyRes 高分辨率处理

三种输入模态的 token 平衡

数据策略

模态间迁移

Qwen-VL 系列

Qwen-VL（第一代）

Qwen2-VL

Qwen3-VL

关于 VLM 的常见问题

第五部分：Chameleon——统一的离散 token 方法

核心思想

VQ-VAE 图像分词

训练与稳定性

评价

总结与当前共识

课程作业关联

常见误区

复习清单

从CLIP到全模态模型：CS336 Lecture 17 多模态建模全景解析

导语

为什么必须处理多模态？

基石：CLIP与SigLIP

CLIP：对比学习的开创性实践

SigLIP：摆脱batch size的束缚

构建视觉语言模型（VLM）：LLaVA与Qwen

LLaVA：初代开源的VLM模板

LLaVA OneVision：升级与自适应分辨率

Qwen-VL系列：走向工程化与规模化

迈向全模态：Chameleon的离散化尝试

课程定位与总结

CS336 2026 Lecture 17：完整中文逐字稿

正文

CS336 2026 · Lecture 17 · Alignment - multimodality

正文