KV Cache Quantization Asymmetry

Baseline finding on GPT-2

A sweep over bits, granularity, and the K/V ratio on GPT-2 small showed two things. First, granularity matters far more than anything else: scaling the quantization per head (groups of 64 channels) at 3 bits gives ppl 46.8, while scaling over the whole vector of the same dimension gives 243.7. Second, in this regime keys need more bits than values — at a matched budget of 5.5 effective bits, a K=6b/V=4b configuration costs only +0.23 ppl (2.9x smaller cache), while K=4b/V=6b costs a full +1.40. The mechanism is simple: errors in keys pass through attention’s softmax and are exponentially amplified, while errors in values only accumulate through a weighted sum. The knee of the curve sits around 5.5-6.5 effective bits, i.e. a 2.5-2.9x smaller cache essentially for free.

Porting to Qwen3.5 and a methodological trap

On Qwen3.5 (baseline bf16 cache on WikiText-2, 8192 tokens, blocks of 512: perplexity 12.0121), the claim “keys need more bits” did not survive the transfer to another architecture unchanged, for two reasons.

A methodological trap. In Qwen3.5, keys are quantized at the k_proj output, but k_norm sits between that point and the KV cache — a per-head RMSNorm that GPT-2 does not have. Quantizing before k_norm is unreliable: the norm partially cancels out the scale error, so keys get a free correction that values (quantized at the v_proj output with no normalization at all) do not. The first run therefore produced an apparent reversal of the earlier finding; after moving the measurement to the k_norm output, the damage from key quantization rose at every bit-width, by 44% at 3 bits. The correct measurement point is the k_norm output — this is a correction of an earlier conclusion, not a new hypothesis.

The asymmetry flips with the budget. Even after fixing the measurement, the original claim holds only over part of the range. Below roughly 4 effective bits the asymmetry favors keys strongly — the gap is whole ppl points, because the keys’ cliff is sharp and sits low. Above roughly 5 effective bits, the choice between K and V becomes domain-dependent and the gap shrinks to hundredths of a point: above the knee, keys are already precise enough that an extra bit pays off better on values. A K=4/V=8 configuration gives a 2.6x smaller KV cache on Qwen3.5 (5.3 GB saved at full context) at 0.0195 ppl below baseline. Symmetric 6-bit quantization at a 262k-token context saves 5.4 GB for +0.0182 ppl.

Domain dependence is concrete, not abstract. On source code (Python), keys win at every budget tested, while on WikiText, values win above the knee — the keys’ cliff sits higher for code, because precisely retrieving specific identifiers goes through keys, while prose gets by with approximate addressing. The K=4/V=8 configuration, the only one below baseline on WikiText, is 0.061 ppl worse than baseline on code. The honest statement of the recommendation is therefore: above ~5 effective bits the choice is domain-dependent; below ~4 bits, always give keys more, because there the gap widens to whole points. Code is also roughly 2.9x more sensitive to quantization than WikiText at 4 bits, measured in nats — full attention layers do more work there, consistent with the general picture that precise retrieval stresses attention more than fluent prose does.

Granularity as the one universal recommendation

Across both models and both domains, one thing holds without exception: quantizing “per head” is not fine-grained enough once a head is large. GPT-2 has a head dimension of 64, Qwen3.5 has 256 — and at 4 bits on WikiText the damage runs from the whole vector (+0.334) through per-head granularity (+0.077) down to groups of 64 (+0.052) and groups of 32 channels (+0.029), with almost the same trend on code. Finer granularity therefore cuts the damage by roughly two-thirds for well under half a bit of extra overhead, and it is the one recommendation from this entire measurement block that survives both tested domains unchanged.

Why this matters at all

KV cache is the dominant memory item at long context, even compared with other compression candidates in hybrid architectures. On Qwen3.5, the state of the remaining linear (Gated DeltaNet) layers has a fixed size independent of context length, while the KV cache of the eight full-attention layers grows linearly with context; the two are roughly equal around 768 tokens. At an 8k-token context the state still accounts for 8.6% of total memory, at 131k it is down to 0.6%, and at a full 262k context just 0.3% — at that point the cache occupies 8.6 GB, an order of magnitude more than anything further compression of the state could save. Per-head cache quantization, with the K/V ratio tuned to the effective bit budget, is therefore the one saving at this scale whose impact matches the size of the problem, and one of the few methods in the project that is essentially lossless.