Latency Evaluation on Azure trace
0.28TTFT P50Token-budget
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Token-budgetRequest rate=1,000 req/s2026.04 | 0.28 | 1.71 | 25 | 62 | |
| HomogeneousRequest rate=1,000 req/s2026.04 | 0.42 | 1.82 | 28 | 67 |
Loading the SOTA2 catalog…
| Method | Links | ||||
|---|---|---|---|---|---|
| Token-budgetRequest rate=1,000 req/s2026.04 | 0.28 | 1.71 | 25 | 62 | |
| HomogeneousRequest rate=1,000 req/s2026.04 | 0.42 | 1.82 | 28 | 67 |