Zero-shot Language Understanding on ARC-E, ARC-C, HellaSwag, LAMBADA, and PIQA (lm-eval 0.4.11 test)
81.5Average AccuracyBF16
Evaluation results
| Method | Links | |
|---|---|---|
| BF16Backbone=Qwen3.5, Parameters=397B-A17B2026.03 | 81.5 | |
| NVFP4Backbone=Qwen3.5, Parameters=397B-A17B, Quantization=W4A4 PTQ2026.03 | 80.9 | |
| IF4Backbone=Qwen3.5, Parameters=397B-A17B, Quantization=W4A4 PTQ2026.03 | 80.9 | |
| NVINT4Backbone=Qwen3.5, Parameters=397B-A17B, Quantization=W4A4 PTQ2026.03 | 80.8 | |
| 4/6Backbone=Qwen3.5, Parameters=397B-A17B, Quantization=W4A4 PTQ2026.03 | 80.8 | |
| MXFP4Backbone=Qwen3.5, Parameters=397B-A17B, Quantization=W4A4 PTQ2026.03 | 79.5 | |
| BF16Backbone=Qwen3.5, Parameters=122B-A10B2026.03 | 76.4 | |
| IF4Backbone=Qwen3.5, Parameters=122B-A10B, Quantization=W4A4 PTQ2026.03 | 76.4 | |
| NVINT4Backbone=Qwen3.5, Parameters=122B-A10B, Quantization=W4A4 PTQ2026.03 | 76 | |
| NVFP4Backbone=Qwen3.5, Parameters=122B-A10B, Quantization=W4A4 PTQ2026.03 | 76 | |
| BF16Backbone=Qwen3.5, Parameters=27B2026.03 | 75.9 | |
| 4/6Backbone=Qwen3.5, Parameters=122B-A10B, Quantization=W4A4 PTQ2026.03 | 75.7 | |
| BF16Backbone=Qwen3.5, Parameters=35B-A3B2026.03 | 75.5 | |
| MXFP4Backbone=Qwen3.5, Parameters=122B-A10B, Quantization=W4A4 PTQ2026.03 | 75.5 | |
| IF4Backbone=Qwen3.5, Parameters=27B, Quantization=W4A4 PTQ2026.03 | 75.3 | |
| NVFP4Backbone=Qwen3.5, Parameters=27B, Quantization=W4A4 PTQ2026.03 | 75.2 | |
| 4/6Backbone=Qwen3.5, Parameters=27B, Quantization=W4A4 PTQ2026.03 | 74.9 | |
| NVINT4Backbone=Qwen3.5, Parameters=27B, Quantization=W4A4 PTQ2026.03 | 74.6 | |
| 4/6Backbone=Qwen3.5, Parameters=35B-A3B, Quantization=W4A4 PTQ2026.03 | 74.6 | |
| IF4Backbone=Qwen3.5, Parameters=35B-A3B, Quantization=W4A4 PTQ2026.03 | 74.6 | |
| MXFP4Backbone=Qwen3.5, Parameters=27B, Quantization=W4A4 PTQ2026.03 | 74.2 | |
| NVINT4Backbone=Qwen3.5, Parameters=35B-A3B, Quantization=W4A4 PTQ2026.03 | 74 | |
| NVFP4Backbone=Qwen3.5, Parameters=35B-A3B, Quantization=W4A4 PTQ2026.03 | 73.9 | |
| MXFP4Backbone=Qwen3.5, Parameters=35B-A3B, Quantization=W4A4 PTQ2026.03 | 72.8 | |
| BF16Backbone=Qwen3.5, Parameters=9B2026.03 | 70.5 | |
| IF4Backbone=Qwen3.5, Parameters=9B, Quantization=W4A4 PTQ2026.03 | 70 | |
| BF16Backbone=Nemotron 3, Parameters=30B-A3B2026.03 | 69.4 | |
| NVFP4Backbone=Qwen3.5, Parameters=9B, Quantization=W4A4 PTQ2026.03 | 69.3 | |
| 4/6Backbone=Qwen3.5, Parameters=9B, Quantization=W4A4 PTQ2026.03 | 69.2 | |
| NVINT4Backbone=Qwen3.5, Parameters=9B, Quantization=W4A4 PTQ2026.03 | 68.7 | |
| IF4Backbone=Nemotron 3, Parameters=30B-A3B, Quantization=W4A4 PTQ2026.03 | 68.4 | |
| NVFP4Backbone=Nemotron 3, Parameters=30B-A3B, Quantization=W4A4 PTQ2026.03 | 67.6 | |
| NVINT4Backbone=Nemotron 3, Parameters=30B-A3B, Quantization=W4A4 PTQ2026.03 | 67.5 | |
| 4/6Backbone=Nemotron 3, Parameters=30B-A3B, Quantization=W4A4 PTQ2026.03 | 67 | |
| MXFP4Backbone=Qwen3.5, Parameters=9B, Quantization=W4A4 PTQ2026.03 | 66.3 | |
| BF16Backbone=Nemotron 3, Parameters=4B2026.03 | 65.1 | |
| IF4Backbone=Nemotron 3, Parameters=4B, Quantization=W4A4 PTQ2026.03 | 64.3 | |
| NVFP4Backbone=Nemotron 3, Parameters=4B, Quantization=W4A4 PTQ2026.03 | 64.1 | |
| 4/6Backbone=Nemotron 3, Parameters=4B, Quantization=W4A4 PTQ2026.03 | 63.9 | |
| MXFP4Backbone=Nemotron 3, Parameters=30B-A3B, Quantization=W4A4 PTQ2026.03 | 63.8 | |
| NVINT4Backbone=Nemotron 3, Parameters=4B, Quantization=W4A4 PTQ2026.03 | 63.1 | |
| MXFP4Backbone=Nemotron 3, Parameters=4B, Quantization=W4A4 PTQ2026.03 | 61.5 |