Large Language Model Inference on FinanceQA
1,779ThroughputSpecBundle
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| SpecBundleTarget Model=Qwen-30B-A3B, Draft Model=SpecBundle, #GPUs=42026.03 | 1,779 | 1.35 | |
| Standard InferenceTarget Model=Qwen-30B-A3B, Draft Model=-, #GPUs=42026.03 | 1,320.1 | 1 | |
| SpecBundleTarget Model=Llama-4-Scout, Draft Model=SpecBundle, #GPUs=82026.03 | 1,189.6 | 4.12 | |
| SpecBundleTarget Model=Llama-3.3-70B, Draft Model=SpecBundle, #GPUs=42026.03 | 1,022.7 | 2 | |
| EAGLE3Target Model=Llama-4-Scout, Draft Model=Existing, #GPUs=82026.03 | 1,022.7 | 3.54 | |
| EAGLE3Target Model=Llama-3.3-70B, Draft Model=Existing, #GPUs=42026.03 | 981.7 | 1.92 | |
| SpecBundleTarget Model=Qwen-235B-A22B, Draft Model=SpecBundle, #GPUs=82026.03 | 889 | 1.65 | |
| SpecBundleTarget Model=Ling-Flash-V2, Draft Model=SpecBundle, #GPUs=82026.03 | 863.9 | 1.16 | |
| Standard InferenceTarget Model=Ling-Flash-V2, Draft Model=-, #GPUs=82026.03 | 747.7 | 1 | |
| EAGLE3Target Model=Qwen-235B-A22B, Draft Model=Existing, #GPUs=82026.03 | 689.4 | 1.28 | |
| SpecBundleTarget Model=Kimi-K2, Draft Model=SpecBundle, #GPUs=82026.03 | 660 | 1.52 | |
| Standard InferenceTarget Model=Qwen-235B-A22B, Draft Model=-, #GPUs=82026.03 | 539.5 | 1 | |
| Standard InferenceTarget Model=Llama-3.3-70B, Draft Model=-, #GPUs=42026.03 | 512.6 | 1 | |
| Standard InferenceTarget Model=Kimi-K2, Draft Model=-, #GPUs=82026.03 | 433.4 | 1 | |
| Standard InferenceTarget Model=Llama-4-Scout, Draft Model=-, #GPUs=82026.03 | 288.9 | 1 | |
| SpecBundleTarget Model=Llama-3.1-8B, Draft Model=SpecBundle, #GPUs=12026.03 | 258.6 | 1.39 | |
| EAGLE3Target Model=Llama-3.1-8B, Draft Model=Existing, #GPUs=12026.03 | 237.2 | 1.27 | |
| Standard InferenceTarget Model=Llama-3.1-8B, Draft Model=-, #GPUs=12026.03 | 185.7 | 1 |