Meta released Llama3 at April 18, which is evaluated on several benchmarks and achieves the SOTA on open-sourced LLMs.

Introduction

Instruct model performance

The performance of Llama3 8B compared with Gemma and Mistral:

performance of Llama3 70B compared with Gemini Pro 1.5 and Claude Sonnet:

Pre-trained model performance

The performance of Llama3 8B compared with Gemma and Mistral:

performance of Llama3 70B compared with Gemini Pro 1.5 and Claude Sonnet:

Model Architecture

Several improvements are made on Llama3 compared to llama2:

1. Llama3 uses a tokenizer with a vocabulary of 128K tokens.
2. Llama3 adopts grouped query attention (GQA) across both the 8B and 70B sizes.
3. Llama3 uses to context window of size 8192 tokens

Traning

Llama3 uses 15T tokens for pre-training. Compares to Llama2, it is seven times larger and includes four times more code.

5% data of the training dataset are non-English to support multi-lingual use cases.

Data processing includes:

1. Heuristic filters
2. NSFW filters
3. Semantic deduplication approaches
4. Text classifiers to predict data quality. Llama2 is used to generate training data for the text classifiers.

Data mixing strategy is explored to improve the performance of Llama3.

Scaling up pretraining

Llama3 developed a series of scaling laws for downstream benchmark evaluations.

Scaling laws help:

1. Select an optimal data mix and to make informed decisions on how to best use training compute.
2. Scaling laws allow Llama3 to predict the performance of the largest models on key tasks without training the models.

The authors finds our that the performance of the model continues to improve log-linearly as the training tokens increase. It is seen that Larger models can match the performance of these smaller models with less training compute, but smaller models are generally preferred because they are much more efficient during inference.

The authors combine three types of parallelization:

1. Data parallelization
2. Model parallelization
3. Pipeline parallelization

Instruction fine-tuning

The fine-tuning of Llama3 contains:

1. Supervised fine-tuning
2. Rejection sampling
3. Proximal Policy Optimization
4. Direct Preference Optimization

Learning from perference rankings via PPO and DPO also greatly improved the performance of LLma3 on reasoning and coding tasks. Since perference ranking helps the model to select answer when it is in a dilemma.

Reference

• Llam3 blog
• Evaluation details
• Model Card

介绍

Meta在2025年4月10号发布了LLaMA4系列，包含三个模型：Llama 4 Scout, Llama 4 Maverick 以及Llama 4 Behemoth, 三个模型都基于MoE架构，且支持多模态

训练数据截止到2024年8月。LLaMa4支持200多种语言，其中100多种语言的训练token数超过了1B

亮点

1. 原生多模态 LLaMA 4是一个原生多模态架构
2. 超长上下文 LLaMA 4的上下文超过了1M
3. iRoPE 通过交替dense和MoE MLP来提高整体推理效率
4. 基于MetaCLIP的vision encoder
5. MetaP 使用MetaP来调整超参数
6. FP8精度训练

Pre-training

LLaMA 4 仍然是一个基于transformer的架构，但是引入了MoE，其示意图如下所示

MoE架构中包含1个shared expert以及1个routed expert. 并且，与其他LLM不同，LLaMA 4使用了一个交替MLP个MoE的架构，即iRoPE，即特定的transformer layer是MoE架构，其余的是MLP架构，其核心代码如下：

self.is_moe_layer = layer_idx in config.moe_layers
if self.is_moe_layer:  # the 128E model interleaves dense / sparse
    self.feed_forward = Llama4TextMoe(config)
else:
    self.feed_forward = Llama4TextMLP(config, intermediate_size=config.intermediate_size_mlp)

early fusion. LLaMA 4称其一个原生多模态大模型，但是其架构仍然是 Vision Encoder-MLP-LLM 的形式，其不同点在于patch embedding没有使用convolution, 而是使用 nn.Unfold直接进行展平，然后使用一个线性层与vision encoder进行对齐。代码如下

class Llama4UnfoldConvolution(nn.Module):
    def __init__(self, config):
        super().__init__()
        kernel_size = config.patch_size
        if isinstance(kernel_size, int):
            kernel_size = (kernel_size, kernel_size)
        self.unfold = torch.nn.Unfold(kernel_size=kernel_size, stride=config.patch_size)
        self.linear = nn.Linear(
            config.num_channels * kernel_size[0] * kernel_size[1],
            config.hidden_size,
            bias=False,
        )

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.unfold(hidden_states)
        hidden_states = hidden_states.permute(0, 2, 1)
        hidden_states = self.linear(hidden_states)
        return hidden_states

其他训练优化技巧如下：

1. MetaP：用于选择超参数
2. FP8 precision：与 DeepSeek-V3 一样，使用FP8精度进行训练
3. mid-training：在预训练阶段之后，额外增加了一个训练阶段，来提高模型的长上下文等关键能力

Post-training

post-training包括三个阶段：

1. SFT
2. online RL
3. DPO

作者发现SFT和DPO会限制模型的探索能力，特别是在math, coding等domain。为了解决这个问题，作者使用LlaMA对问题进行难度分级，然后移除了50%的简单数据。

在online RL阶段，作者设计了一个continuous online RL策略，让模型在训练和筛选问题两种模式之间进行切换，以平衡效率和准确率。

DPO的目的是为了提升模型输出的质量

评测

结论

LLaMA 4 采用了MoE架构，是一个原生的多模态大模型系列。在架构上，与DeepSeek-MoE, aria和OLMoE不同，LLaMA4并没有增加expert granularity，OLMoE分析认为，增加granularity可以提高模型的flexibility， 下面总结了一下相关模型的参数

Reference

• LLaMA 4 Model Card
• LLaMA 4 Blog