LLM Inference Efficiency on AIME
40.6Time To First Token (TTFT)RelayCaching
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| RelayCachingNumber of Agents=Agent 2, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 40.6 | 2.1 | 1.6 | 50 | |
| CacheBlendNumber of Agents=Agent 2, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 44.9 | 1.9 | — | — | |
| RelayCachingNumber of Agents=Agent 3, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 57 | 3.16 | 2.8 | 60 | |
| RelayCachingNumber of Agents=Agent 4, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 73.2 | 4.39 | 4.3 | 70 | |
| CacheBlendNumber of Agents=Agent 3, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 76.7 | 2.35 | — | — | |
| Full PrefillNumber of Agents=Agent 2, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 85.2 | — | — | — | |
| RelayCachingNumber of Agents=Agent 5, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 104.8 | 4.71 | 7.9 | 80 | |
| CacheBlendNumber of Agents=Agent 4, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 112.5 | 2.86 | — | — | |
| CacheBlendNumber of Agents=Agent 5, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 159.3 | 3.1 | — | — | |
| Full PrefillNumber of Agents=Agent 3, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 180.2 | — | — | — | |
| Full PrefillNumber of Agents=Agent 4, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 321.5 | — | — | — | |
| Full PrefillNumber of Agents=Agent 5, Model=Qwen3-0.6B, Prefix Tokens=512, Output Tokens=2,0482026.02 | 493.9 | — | — | — |