List your company

Commonsense Reasoning on HellaSwag (first 1000 examples)

38Accuracy (HellaSwag 1k)k=64 (DR only)
35.60836.22936.8537.471Apr 20, 2026

Evaluation results

MethodLinks
k=64 (DR only)Model size=300M, Tokens=5B, Precision=FP16, Decoding=greedy, Mode=zero-shot, Examples=1000, Depth Registers=k=642026.0438
k=64 + sinkModel size=300M, Tokens=5B, Precision=FP16, Decoding=greedy, Mode=zero-shot, Examples=1000, Depth Registers=k=64, Attention-sink loss=included2026.0436.5
k=0 (baseline)Model size=300M, Tokens=5B, Precision=FP16, Decoding=greedy, Mode=zero-shot, Examples=10002026.0435.7