Research
We study language models by trying to break them in specific ways and measuring what happens. Everything here is our own work, run on our own hardware, and published with the controls it had to beat.
KV cache quantization is the only pain-free memory saving the project found for long context. A sweep over bits, granularity, and the K/V ratio on GPT-2 showed that keys need more bits than values. Porting the test to Qwen3.5 exposed a measurement-location artifact and showed the asymmetry is regime-dependent, not universal: keys win decisively below ~4 effective bits, while above ~5 the choice is domain-dependent.
GPT-2 small · Qwen3.5
A nine-step chain of tests asked whether GPT-2's transformer layers share enough parametric structure to store one common basis with a cheap per-layer description of the deviation. Sharing in raw weights did not hold; sharing in the functional space of input activations did, but only through fine-tuning, never through analytic projection. Attention matrices tolerate compression well (9.6x at a recoverable cost), while MLP layers structurally resist it.
GPT-2 small · GPT-2 medium