Reading Comprehension on SciQ
93.7AccuracyLLaMA2-7B
Evaluation Results
| Method | Links | |
|---|---|---|
| LLaMA2-7B# tokens for training=2T, Shot number=0, Training data source=Different from RedPajama2023.10 | 93.7 | |
| Open-LLaMA-3B-v2# tokens for training=1T, Shot number=0, Training data source=RedPajama2023.10 | 91.8 | |
| Open-LLaMA-3B-v1# tokens for training=1T, Shot number=0, Training data source=RedPajama2023.10 | 91.3 | |
| Sheared-LLaMA-2.7B# tokens for training=50B, Shot number=0, Training data source=RedPajama2023.10 | 90.8 | |
| INCITE-Base-3B# tokens for training=800B, Shot number=0, Training data source=RedPajama2023.10 | 90.7 | |
| TinyLlama-1.1B# tokens for training=3T, Shot number=0, Training data source=RedPajama2023.10 | 88.9 | |
| Pythia-2.8B# tokens for training=300B, Shot number=0, Training data source=RedPajama2023.10 | 88.3 | |
| Sheared-LLaMA-1.3B# tokens for training=50B, Shot number=0, Training data source=RedPajama2023.10 | 87.3 | |
| Pythia-1.4B# tokens for training=300B, Shot number=0, Training data source=RedPajama2023.10 | 86.4 | |
| OPT-2.7B# tokens for training=300B, Shot number=0, Training data source=Different from RedPajama2023.10 | 85.8 | |
| OPT-1.3B# tokens for training=300B, Shot number=0, Training data source=Different from RedPajama2023.10 | 84.3 | |
| DoGraphBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 66.1 | |
| RegMixBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 62.8 | |
| DoReMiBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 61.6 | |
| UniformBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 60.3 | |
| DOGEBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 60.1 | |
| Dynamic Loss-BasedBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 60 | |
| Data Mixing LawBackbone=LLaMA-1.1B, Pre-training Dataset=The Pile2026.04 | 58.6 | |
| Dynamic Loss-BasedModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 53.8 | |
| DoGraphModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 53.8 | |
| DoGraphBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 53.5 | |
| Dynamic Loss-BasedBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 53.3 | |
| DoReMiBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 53.3 | |
| DOGEBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 52.8 | |
| DoReMiModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 52.4 | |
| UniformBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 49 | |
| RegMixBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 46.6 | |
| Data Mixing LawBackbone=GPT-2 Medium, Pre-training Dataset=SlimPajama2026.04 | 45.2 | |
| Data Mixing LawModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 39.6 | |
| RegMixModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 38.7 | |
| UniformModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 32.6 | |
| DOGEModel=GPT-2 Small, Training Data=SlimPajama2026.04 | 31.1 |