Reading Comprehension on RACE-h (test)
89.95AccuracyQwen-1.5 14B (Teacher)
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen-1.5 14B (Teacher)Model Scale=14B, Evaluation Protocol=Zero-shot2024.07 | 89.95 | |
| GopherModel Size=280B, Evaluation Protocol=Few-shot, Shots=52021.12 | 71.6 | |
| Qwen-1.5 1.8B + DDKModel Scale=1.8B, Distillation Strategy=DDK2024.07 | 71.56 | |
| Qwen-1.5 1.8B + CPT & DoReMiModel Scale=1.8B, Distillation Strategy=CPT & DoReMi2024.07 | 70.12 | |
| Qwen-1.5 1.8B + KDModel Scale=1.8B, Distillation Strategy=KD2024.07 | 70.12 | |
| Qwen-1.5 1.8B + TEDModel Scale=1.8B, Distillation Strategy=TED2024.07 | 69.95 | |
| Qwen-1.5 1.8B (Student)Model Scale=1.8B, Evaluation Protocol=Zero-shot2024.07 | 69.57 | |
| Qwen-1.5 1.8B + CPTModel Scale=1.8B, Distillation Strategy=CPT2024.07 | 69 | |
| Qwen-1.5 1.8B + MiniLLMModel Scale=1.8B, Distillation Strategy=MiniLLM2024.07 | 68.92 | |
| LLaMA2-13BModel Size=13B, Shots=0-shot2024.07 | 62.85 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Few-shot, Shots=22021.12 | 49.3 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=One-shot2021.12 | 48.7 | |
| Megatron-NLGModel Size=530B, Evaluation Protocol=Few-shot2021.12 | 47.9 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Zero-shot2021.12 | 46.9 | |
| GPT-3Model Size=175B, Evaluation Protocol=Few-shot, Shots=102021.12 | 46.8 | |
| GPT-3Model Size=175B, Evaluation Protocol=One-shot2021.12 | 45.9 | |
| GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2021.12 | 45.5 | |
| AMOOptimizer=AMO, Model=Llama3.1-1.4B, Few-shot=02026.05 | 33 | |
| AMOOptimizer=AMO, Model=Llama3.1-760M, Few-shot=02026.05 | 31.43 | |
| TinyLLaMA-1.1B + DDKModel Size=1.1B, Shots=0-shot2024.07 | 27.76 | |
| TinyLLaMA-1.1B + MiniLLMModel Size=1.1B, Shots=0-shot2024.07 | 24.32 | |
| TinyLLaMA-1.1B + CPTModel Size=1.1B, Shots=0-shot2024.07 | 24.24 | |
| TinyLLaMA-1.1B + TEDModel Size=1.1B, Shots=0-shot2024.07 | 24.18 | |
| TinyLLaMA-1.1B + CPT & DoReMiModel Size=1.1B, Shots=0-shot2024.07 | 24.04 | |
| TinyLLaMA-1.1B + KDModel Size=1.1B, Shots=0-shot2024.07 | 23.84 | |
| TinyLLaMA-1.1BModel Size=1.1B, Shots=0-shot2024.07 | 22.92 |