Tiny but mighty: the case for 4‑bit floating point (FP4)

What is FP4?

FP4 is exactly what it sounds like: a four‑bit floating point format that squeezes sign, exponent and mantissa into a nibble. One bit is the sign; the other three are split between exponent and mantissa in several ways — E3M0, E2M1, E1M2, and E0M3 — with E2M1 showing up most often in discussions. The value of a non‑special FP4 number can be written as (−1)^s · 2^(e−b) · (1 + m/2), with a small wrinkle for e = 0 where you get zeros and a half value instead. Tiny table, big implications: even here you get +0 and −0, just like full‑precision floats. Weird? Yes. Clever? Also yes.

Why it matters

Neural networks turned the usual precision logic on its head: more parameters beat more precision. Memory and bandwidth are the limiting factors, so researchers and engineers have been chopping precision down from 64 to 32 to 16 bits — and now to 8 and 4 bits in pursuit of packing more model into the same hardware. Why not use integers? You could — but a floating format gives more dynamic range for tiny exponents, which matters when activations and gradients vary wildly. It has been reported that the E2M1 variant is the most commonly used FP4 layout and allegedly has support in Nvidia hardware, which helps explain the format’s momentum.

Tools and takeaways

John Cook’s writeup walks through the math, lists all 16 E2M1 values, and points to Pychop, a Python library that emulates reduced‑precision formats so you can experiment without custom silicon. The emotional punch is small but real: we’re deliberately embracing precision loss to win at scale. Big models, tiny numbers — the tech world keeps reminding us that sometimes less is more. Ready to think small?

Sources: johndcook.com, Hacker News