Commonsense reasoning on WinoGrande 1.0 (test)
74.1AccuracyLLaMA-3-8B-LoLCATs
Evaluation Results
| Method | Links | |
|---|---|---|
| LLaMA-3-8B-LoLCATsTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.07 | 74.1 | |
| Mistral-7BTraining Tokens (B)=8000*, Evaluation Protocol=Zero-shot, Model Architecture Category=Transformer2025.07 | 74 | |
| Mistral-7B-LoLCATsTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.07 | 74 | |
| Gemma-7BTraining Tokens (B)=6000, Evaluation Protocol=Zero-shot, Model Architecture Category=Transformer2025.07 | 73.7 | |
| LLaMA-3-8BTraining Tokens (B)=15000, Evaluation Protocol=Zero-shot, Model Architecture Category=Transformer2025.07 | 73.1 | |
| Griffin-7BTraining Tokens (B)=300, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.07 | 72.6 | |
| Liger-GLA-Llama-3-8BTraining Tokens (B)=0.02, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.07 | 72 | |
| Mistral-7B-LizardTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Unbounded2025.07 | 72 | |
| Mamba-7BTraining Tokens (B)=1200, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.07 | 71.8 | |
| LLaMA-3-8B-LizardTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Unbounded2025.07 | 71.7 | |
| Mistral-7B-SUPRATraining Tokens (B)=100, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.07 | 70.3 | |
| Liger-GLA-Mistral-7BTraining Tokens (B)=0.02, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.07 | 70.1 | |
| RWKV-6-v2.1-7BTraining Tokens (B)=1420, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.07 | 70 | |
| Hawk-7BTraining Tokens (B)=300, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.07 | 69.9 | |
| TransNormerLLM-7BTraining Tokens (B)=1400, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.07 | 66.1 | |
| Mamba2-LLaMA-3-8BTraining Tokens (B)=20, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Unbounded2025.07 | 58.6 | |
| Mistral-7B + DSIRModel=Mistral-7B, Selection Method=DSIR, Protocol=15-shot2024.02 | 0.8137 | |
| Mistral-7B BaseModel=Mistral-7B, Selection Method=Base, Protocol=15-shot2024.02 | 0.8122 | |
| Mistral-7B + UniformModel=Mistral-7B, Selection Method=Uniform, Protocol=15-shot2024.02 | 0.8019 | |
| Mistral-7B + QuRatingModel=Mistral-7B, Selection Method=QuRating, Protocol=15-shot2024.02 | 0.8011 | |
| Mistral-7B + AutoDSModel=Mistral-7B, Selection Method=AutoDS, Protocol=15-shot2024.02 | 0.8003 | |
| LLaMA2-7B BaseModel=LLaMA2-7B, Selection Method=Base, Protocol=15-shot2024.02 | 0.7585 | |
| LLaMA2-7B + DSIRModel=LLaMA2-7B, Selection Method=DSIR, Protocol=15-shot2024.02 | 0.7537 | |
| LLaMA2-7B + UniformModel=LLaMA2-7B, Selection Method=Uniform, Protocol=15-shot2024.02 | 0.753 | |
| LLaMA2-7B + QuRatingModel=LLaMA2-7B, Selection Method=QuRating, Protocol=15-shot2024.02 | 0.7466 | |
| LLaMA2-7B + AutoDSModel=LLaMA2-7B, Selection Method=AutoDS, Protocol=15-shot2024.02 | 0.7451 | |
| Gemma-2B + DSIRModel=Gemma-2B, Selection Method=DSIR, Protocol=15-shot2024.02 | 0.6661 | |
| Gemma-2B + AutoDSModel=Gemma-2B, Selection Method=AutoDS, Protocol=15-shot2024.02 | 0.6661 | |
| Gemma-2B + UniformModel=Gemma-2B, Selection Method=Uniform, Protocol=15-shot2024.02 | 0.6638 | |
| Gemma-2B + QuRatingModel=Gemma-2B, Selection Method=QuRating, Protocol=15-shot2024.02 | 0.6638 | |
| Gemma-2B BaseModel=Gemma-2B, Selection Method=Base, Protocol=15-shot2024.02 | 0.6054 |