Question Answering on ARC Challenge (val)
93.3AccuracyPioneer Agent (Qwen3-8B)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Pioneer Agent (Qwen3-8B)Model=Qwen3-8B, Evaluation Protocol=Fine-tuned, Iters=132026.04 | 93.3 | 1.6 | |
| Qwen3-8BModel=Qwen3-8B, Evaluation Protocol=Baseline (zero-shot)2026.04 | 91.7 | — | |
| LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.02 | 89.6 | — | |
| D2-LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.02 | 89.6 | — | |
| DoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.02 | 88.3 | — | |
| D2-LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.02 | 73.9 | — | |
| LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.02 | 72.6 | — | |
| Pioneer Agent (Llama 3.2-3B)Model=Llama 3.2-3B, Evaluation Protocol=Fine-tuned, Iters=112026.04 | 72.6 | 67.3 | |
| DoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.02 | 72.2 | — | |
| G-DAUGSelection strategy=Influence2020.04 | 51.5 | — | |
| G-DAUGSelection strategy=Random2020.04 | 50.8 | — | |
| G-DAUGSelection strategy=Combo2020.04 | 50.8 | — | |
| FP16Backbone=Mistral-7b, Bits Per FPN=162024.05 | 50.34 | — | |
| KVQuant-4b-1%Backbone=Mistral-7b, Bits Per FPN=4.32-4.352024.05 | 49.91 | — | |
| G-DAUGSelection strategy=Diversity2020.04 | 49.5 | — | |
| CQ-2c8bBackbone=Mistral-7b, Bits Per FPN=4.002024.05 | 49.15 | — | |
| KVQuant-4bBackbone=Mistral-7b, Bits Per FPN=4.00-4.022024.05 | 49.06 | — | |
| FP16Backbone=LLaMA-2-13b, Bits Per FPN=162024.05 | 48.29 | — | |
| KVQuant-4b-1%Backbone=LLaMA-2-13b, Bits Per FPN=4.32-4.352024.05 | 47.87 | — | |
| CQ-2c8bBackbone=LLaMA-2-13b, Bits Per FPN=4.002024.05 | 47.78 | — | |
| KVQuant-2b-1%Backbone=Mistral-7b, Bits Per FPN=2.32-2.352024.05 | 47.53 | — | |
| KVQuant-4b-1%Backbone=LLaMA-13b, Bits Per FPN=4.32-4.352024.05 | 46.76 | — | |
| KVQuant-4bBackbone=LLaMA-2-13b, Bits Per FPN=4.00-4.022024.05 | 46.67 | — | |
| FP16Backbone=LLaMA-13b, Bits Per FPN=162024.05 | 46.42 | — | |
| KVQuant-4bBackbone=LLaMA-13b, Bits Per FPN=4.00-4.022024.05 | 45.99 | — | |
| CQ-2c8bBackbone=LLaMA-13b, Bits Per FPN=4.002024.05 | 45.99 | — | |
| CQ-4c8bBackbone=Mistral-7b, Bits Per FPN=2.002024.05 | 45.65 | — | |
| KVQuant-2b-1%Backbone=LLaMA-13b, Bits Per FPN=2.32-2.352024.05 | 45.14 | — | |
| KVQuant-2b-1%Backbone=LLaMA-2-13b, Bits Per FPN=2.32-2.352024.05 | 44.97 | — | |
| CQ-4c8bBackbone=LLaMA-2-13b, Bits Per FPN=2.002024.05 | 44.11 | — | |
| CQ-4c8bBackbone=LLaMA-13b, Bits Per FPN=2.002024.05 | 44.03 | — | |
| ROBERTaSelection strategy=None (Baseline)2020.04 | 43.5 | — | |
| FP16Backbone=LLaMA-2-7b, Bits Per FPN=162024.05 | 43.43 | — | |
| CQ-2c8bBackbone=LLaMA-2-7b, Bits Per FPN=4.002024.05 | 43.34 | — | |
| KVQuant-4b-1%Backbone=LLaMA-2-7b, Bits Per FPN=4.32-4.352024.05 | 43.17 | — | |
| Backtranslation2020.04 | 43.1 | — | |
| KVQuant-4bBackbone=LLaMA-2-7b, Bits Per FPN=4.00-4.022024.05 | 42.75 | — | |
| KVQuant-4bBackbone=LLaMA-7b, Bits Per FPN=4.00-4.022024.05 | 42.32 | — | |
| FP16Backbone=LLaMA-7b, Bits Per FPN=162024.05 | 41.72 | — | |
| CQ-2c8bBackbone=LLaMA-7b, Bits Per FPN=4.002024.05 | 41.55 | — | |
| KVQuant-2b-1%Backbone=LLaMA-2-7b, Bits Per FPN=2.32-2.352024.05 | 41.47 | — | |
| KVQuant-4b-1%Backbone=LLaMA-7b, Bits Per FPN=4.32-4.352024.05 | 41.38 | — | |
| CQ-4c8bBackbone=LLaMA-2-7b, Bits Per FPN=2.002024.05 | 39.93 | — | |
| CQ-8c10bBackbone=Mistral-7b, Bits Per FPN=1.252024.05 | 39.59 | — | |
| KVQuant-2b-1%Backbone=LLaMA-7b, Bits Per FPN=2.32-2.352024.05 | 38.74 | — | |
| CQ-8c10bBackbone=LLaMA-2-13b, Bits Per FPN=1.252024.05 | 38.74 | — | |
| KVQuant-2bBackbone=Mistral-7b, Bits Per FPN=2.00-2.022024.05 | 38.57 | — | |
| CQ-4c8bBackbone=LLaMA-7b, Bits Per FPN=2.002024.05 | 38.48 | — | |
| CQ-8c10bBackbone=LLaMA-13b, Bits Per FPN=1.252024.05 | 37.12 | — | |
| KVQuant-1b-1%Backbone=LLaMA-13b, Bits Per FPN=1.32-1.352024.05 | 35.32 | — | |
| CQ-8c10bBackbone=LLaMA-2-7b, Bits Per FPN=1.252024.05 | 34.64 | — | |
| KVQuant-2bBackbone=LLaMA-13b, Bits Per FPN=2.00-2.022024.05 | 34.47 | — | |
| DeepSeekMoE# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.01 | 34.3 | — | |
| CQ-8c8bBackbone=LLaMA-2-13b, Bits Per FPN=1.002024.05 | 34.3 | — | |
| CQ-8c8bBackbone=LLaMA-13b, Bits Per FPN=1.002024.05 | 33.79 | — | |
| CQ-8c8bBackbone=Mistral-7b, Bits Per FPN=1.002024.05 | 33.79 | — | |
| CQ-8c10bBackbone=LLaMA-7b, Bits Per FPN=1.252024.05 | 33.28 | — | |
| KVQuant-1b-1%Backbone=Mistral-7b, Bits Per FPN=1.32-1.352024.05 | 33.19 | — | |
| KVQuant-1b-1%Backbone=LLaMA-2-13b, Bits Per FPN=1.32-1.352024.05 | 32.59 | — | |
| KVQuant-2bBackbone=LLaMA-7b, Bits Per FPN=2.00-2.022024.05 | 32 | — | |
| GShard# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.01 | 31.6 | — | |
| KVQuant-1b-1%Backbone=LLaMA-2-7b, Bits Per FPN=1.32-1.352024.05 | 31.48 | — | |
| Switch# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 30.2 | — | |
| CQ-8c8bBackbone=LLaMA-7b, Bits Per FPN=1.002024.05 | 30.2 | — | |
| CQ-8c8bBackbone=LLaMA-2-7b, Bits Per FPN=1.002024.05 | 30.2 | — | |
| KVQuant-1b-1%Backbone=LLaMA-7b, Bits Per FPN=1.32-1.352024.05 | 29.69 | — | |
| Hash Layer# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 28.2 | — | |
| Dense# Shot=0-shot, # Total Params=0.2B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 26 | — | |
| KVQuant-2bBackbone=LLaMA-2-13b, Bits Per FPN=2.00-2.022024.05 | 24.66 | — | |
| KVQuant-2bBackbone=LLaMA-2-7b, Bits Per FPN=2.00-2.022024.05 | 22.44 | — | |
| KVQuant-1bBackbone=LLaMA-7b, Bits Per FPN=1.00-1.022024.05 | 21.76 | — | |
| KVQuant-1bBackbone=LLaMA-2-13b, Bits Per FPN=1.00-1.022024.05 | 21.67 | — | |
| KVQuant-1bBackbone=LLaMA-13b, Bits Per FPN=1.00-1.022024.05 | 21.33 | — | |
| KVQuant-1bBackbone=LLaMA-2-7b, Bits Per FPN=1.00-1.022024.05 | 20.65 | — | |
| KVQuant-1bBackbone=Mistral-7b, Bits Per FPN=1.00-1.022024.05 | 19.88 | — | |
| Llama 3.2-3BModel=Llama 3.2-3B, Evaluation Protocol=Baseline (zero-shot)2026.04 | 5.3 | — |