Low Precision

Author

Updated

Aug, 28, 2026

Category

LLM

Precision

Float

PropertyFP16BF16
Exponent Bits58
Mantissa Bits107
Smallest Positive Normal≈6.1×10−5\approx 6.1 \times 10^{-5}≈1.2×10−38\approx 1.2 \times 10^{-38}
Largest Value≈6.6×104\approx 6.6 \times 10^{4}≈3.4×1038\approx 3.4 \times 10^{38}
Next Representable > 11+2−10≈1.0009771 + 2^{-10} \approx 1.0009771+2−7≈1.0078121 + 2^{-7} \approx 1.007812

MXFP8

见 MXFP8 (Open Compute Project, 2023).

MXFP4

MXFP4 是一个 4 位的浮点数，结构为 E2M1, 包括 1 sign bit, 2 exponent bits 和 1 mantissa bits.

由于 E2M1 粒度比较粗，因此 MXFP4 使用了 blockwise scaling:

关于 MXFP4 quantization 的实现逻辑见 MXFP4 quantization 和 MXFP4 integration.

NVFP4

见 NVFP4 (NVIDIA et al., 2026)

  1. NVIDIA, Abecassis, F., Agrusa, A., Ahn, D., Alben, J., Alborghetti, S., Andersch, M., Arayandi, S., Bjorlin, A., Blakeman, A., Briones, E., Buck, I., Catanzaro, B., Chang, M., Choi, J., Chrzanowski, M., Chung, E., Cui, V., Dai, S., … Zhu, Z. (2026). Pretraining Large Language Models with NVFP4. https://arxiv.org/abs/2509.25149
  2. Open Compute Project. (2023). Compute Project. Ocp microscaling formats (mx) specification version 1.0. https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf