Entity Counting on NL Counting (held-out)
100AccuracyOracle
Evaluation results
| Method | Links | |
|---|---|---|
| OracleModel=Qwen3-8B, Parameters=0, Training=None2026.05 | 100 | |
| DPS (hard, multi-seed protocol)Model=Qwen3-8B, Parameters=4,097, Training=None2026.05 | 98.7 | |
| Full lm_head (held-out)Model=Qwen3-8B, Parameters=∼622M, Training=300 steps2026.05 | 94.2 | |
| 9-row lm_head (held-out)Model=Qwen3-8B, Parameters=36,864, Training=300 steps2026.05 | 93.8 | |
| 9-row lm_head (held-out)Model=Mistral-7B, Parameters=36,864, Training=300 steps2026.05 | 92 | |
| LoRA Q/V rank-16 (fullvocab)Model=Qwen3-8B, Parameters=7.67M, Training=200 steps2026.05 | 91.7 | |
| 9-row lm_head + DPS (matched train, 3 seeds)Model=Qwen3-14B, Parameters=46,080 + 5,121, Training=1200 steps2026.05 | 90.3 | |
| LoRA (all 36 layers, constrained-gen)Model=Qwen3-8B, Parameters=∼4M, Training=200 steps2026.05 | 84 | |
| DPS layer sweep (10 layers, best)Model=Qwen3-14B, Parameters=5,121, Training=None2026.05 | 28.7 | |
| DPS (α ∈ [5, 500], best)Model=Qwen3-14B, Parameters=5,121, Training=None2026.05 | 26.5 | |
| Baseline (vanilla)Model=Mistral-7B, Parameters=0, Training=None2026.05 | 26.2 | |
| Baseline (vanilla)Model=Qwen3-14B, Parameters=0, Training=None2026.05 | 24.5 | |
| Shuffled-row controlModel=Qwen3-14B, Parameters=46,080, Training=1200 steps2026.05 | 12 | |
| Baseline (vanilla)Model=Qwen3-8B, Parameters=0, Training=None2026.05 | 11.3 |