Blog
Notes, tutorials, and longer-form writing. Search and filter by tags or category →
All posts
- LLM Overview of Linear Attention
- Math Math Foundations
- LLM MoE tutorial
本 blog 详细介绍了 MoE 模型的一些关键设计与相关实验结果，为 MoE 模型的学习提供基础。
- LLM Overview of OLM series
- LLM Overview of position encoding
- Infra Overview of Parallelism
- LLM Overview of Nemotron series
- LLM Overview of Attention Mechanism
A hands-on guide to understanding and implementing the Attention mechanism in deep learning models.
- MLLM Overview of MiniMax series
- MLLM Overview of MiMo series
- Overview of RLVR
Overview of RLVR
- MLLM Overview of Qwen-VL series
Overview of Qwen-VL series
- MLLM Overview of Multimodal Large Language Models
- LLM Overview of Qwen series
系统性回顾Qwen系列模型架构的演变与关键技术
- MLLM Overview of Visual Foundation Models
- LLM Overview of GPT series
openAI 发布了 gpt-oss 大语言模型，包含 120B-A5.1B 以及 20.9B-A3.6B 两个 size, 作者强调了模型的 instruction following, tool use, 以及 adaptive thinking 能力
- MLLM Overview of InternVL series
- MLLM Overview of GLM series
- MLLM Overview of Keye-VL series
- Overview of OPD
- LLM Overview of Flash Attention series
作者提出了 flashattention, 一个通过降低 multi head attention 内存访问开销来提高 attention 计算效率的方法
- Overview of unified MLLM
Overview of unified MLLM
- MLLM Overview of Kimi series
- LLM Reinforcement Learning for Large Language models: An Overview
- LLM Overview of DeepSeek series
- Overview of RLHF
Overview of RLHF
- LLM LLM Memory Computation
本文中，我们将介绍如何计算 LLM 在训练和推理过程中的内存需求以及简要介绍对应的优化方法。
- LLM Overview of optimizers for LLMs
Overview of optimizers for LLMs
- LLM Hands on LLM(2) Transformer
基于Qwen3讲解transformer的架构以及核心代码
- LLM Overview of Hunyuan series
- LLM Overview of MiniCPM series
- Bringing paper to life: A modern template for scientific writing
Publish‑ready workflow that lets you focus on ideas, not infrastructure
- Infra GPU vs TPU: A Comprehensive Guide to Specialized Hardware Accelerators
An in-depth exploration of Graphics Processing Units and Tensor Processing Units, comparing their computational capabilities, memory architectures, and applications in machine learning, deep learning, and scientific computing.
- LLM DeepSeek-R1: the first open sourced reasoning models
- LLM Overview of Gemma series
Overview of Gemma series
- LLM Scaling law
Publish‑ready workflow that lets you focus on ideas, not infrastructure
- LLM Overview of Long Context capabilities
- LLM Overview of Composer series
- LLM Overview of LLaMA series
- Deep Learning Deep Learning Techniques
- LLM Hands on LLM (1) Tokenizer
Tokenizer 总结与 BPE tokenizer 的高效实现
- MLLM Overview of Gemini Series
Gemini 3.0 是是 Google 新一代最强模型，model card 介绍了 Gemini 3.0 系列的评估结果以及基本能力
- Infra Theory of scalability
- LLM Overview of Policy Optimization Methods
- Machine Learning KL divergence from machine learning to reinforcement learning
在强化学习中，KL divergence 常被用作 policy 正则项，但很多不稳定现象并非来自 KL 本身，而是来自其估计方式。本文展示了为什么“无偏的 KL 估计”并不能保证“无偏的 KL 梯度”，并系统分析了不同 KL estimator 在 on-policy 与 off-policy 场景下的行为差异。通过理论推导与实验验证，文章揭示了 KL 作为 loss 与 reward shaping 时的本质区别，并解释了实践中低方差 KL 设计背后的原因
- LLM LLM FLOPs Computation
我们介绍如何计算基于 transformer 架构的 LLM 的 FLOPs, 计算完成之后，我们可以推导出算力 $C$ 与模型参数量 $N$，数据集大小 $D$ 之间的关系，即 $C\approx 6ND$.
- LLM LLM Parameter Computation
我们介绍一下如何计算 LLM 的参数量。我们将基于 Qwen3 模型架构出发，对模型架构进行拆解，然后给出 LLM 参数量计算公式。