List your company
ResearchFormerly WizWand

Commonsense reasoning on WinoGrande 1.0 (test)

74.1AccuracyLLaMA-3-8B-LoLCATs
-2.33438417.50915837.352757.196242Feb 12, 2024May 7, 2024Aug 1, 2024Oct 26, 2024Jan 20, 2025Apr 16, 2025Jul 11, 2025
Updated 6mo ago

Evaluation results

MethodLinks
LLaMA-3-8B-LoLCATsTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.0774.1
Mistral-7BTraining Tokens (B)=8000*, Evaluation Protocol=Zero-shot, Model Architecture Category=Transformer2025.0774
Mistral-7B-LoLCATsTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.0774
Gemma-7BTraining Tokens (B)=6000, Evaluation Protocol=Zero-shot, Model Architecture Category=Transformer2025.0773.7
LLaMA-3-8BTraining Tokens (B)=15000, Evaluation Protocol=Zero-shot, Model Architecture Category=Transformer2025.0773.1
Griffin-7BTraining Tokens (B)=300, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.0772.6
Liger-GLA-Llama-3-8BTraining Tokens (B)=0.02, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.0772
Mistral-7B-LizardTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Unbounded2025.0772
Mamba-7BTraining Tokens (B)=1200, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.0771.8
LLaMA-3-8B-LizardTraining Tokens (B)=0.04, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Unbounded2025.0771.7
Mistral-7B-SUPRATraining Tokens (B)=100, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.0770.3
Liger-GLA-Mistral-7BTraining Tokens (B)=0.02, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Bounded2025.0770.1
RWKV-6-v2.1-7BTraining Tokens (B)=1420, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.0770
Hawk-7BTraining Tokens (B)=300, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.0769.9
TransNormerLLM-7BTraining Tokens (B)=1400, Evaluation Protocol=Zero-shot, Model Architecture Category=Subquadratic2025.0766.1
Mamba2-LLaMA-3-8BTraining Tokens (B)=20, Evaluation Protocol=Zero-shot, Model Architecture Category=Linearized, Linearized Model Constraint=Unbounded2025.0758.6
Mistral-7B + DSIRModel=Mistral-7B, Selection Method=DSIR, Protocol=15-shot2024.020.8137
Mistral-7B BaseModel=Mistral-7B, Selection Method=Base, Protocol=15-shot2024.020.8122
Mistral-7B + UniformModel=Mistral-7B, Selection Method=Uniform, Protocol=15-shot2024.020.8019
Mistral-7B + QuRatingModel=Mistral-7B, Selection Method=QuRating, Protocol=15-shot2024.020.8011
Mistral-7B + AutoDSModel=Mistral-7B, Selection Method=AutoDS, Protocol=15-shot2024.020.8003
LLaMA2-7B BaseModel=LLaMA2-7B, Selection Method=Base, Protocol=15-shot2024.020.7585
LLaMA2-7B + DSIRModel=LLaMA2-7B, Selection Method=DSIR, Protocol=15-shot2024.020.7537
LLaMA2-7B + UniformModel=LLaMA2-7B, Selection Method=Uniform, Protocol=15-shot2024.020.753
LLaMA2-7B + QuRatingModel=LLaMA2-7B, Selection Method=QuRating, Protocol=15-shot2024.020.7466
LLaMA2-7B + AutoDSModel=LLaMA2-7B, Selection Method=AutoDS, Protocol=15-shot2024.020.7451
Gemma-2B + DSIRModel=Gemma-2B, Selection Method=DSIR, Protocol=15-shot2024.020.6661
Gemma-2B + AutoDSModel=Gemma-2B, Selection Method=AutoDS, Protocol=15-shot2024.020.6661
Gemma-2B + UniformModel=Gemma-2B, Selection Method=Uniform, Protocol=15-shot2024.020.6638
Gemma-2B + QuRatingModel=Gemma-2B, Selection Method=QuRating, Protocol=15-shot2024.020.6638
Gemma-2B BaseModel=Gemma-2B, Selection Method=Base, Protocol=15-shot2024.020.6054