Question Answering on WebQuestions
99.6WebQ AccuracyLayer-LR
Evaluation results
| Method | Links | |
|---|---|---|
| Layer-LRModel=Mistral-7B-v0.1, Mode=FTG (First-token generation), Context=Out-of-domain2026.02 | 99.6 | |
| Layer-LRModel=Qwen2.5-7B, Mode=FTG (First-token generation), Context=Out-of-domain2026.02 | 99.1 | |
| Layer-LRModel=Llama-3.1-8B, Mode=FTG (First-token generation), Context=Out-of-domain2026.02 | 95.5 | |
| GPT-4oEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=4.52†, Throughput (Tok/s)=55.22026.03 | 81.5 | |
| Claude 3.5 SonnetEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=3.85†, Throughput (Tok/s)=62.12026.03 | 80.1 | |
| Gemini 2.5 ProEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=3.90†, Throughput (Tok/s)=58.42026.03 | 79.8 | |
| EcoThinkEvaluation Protocol=Adaptive Inference, Emission (gCO2/q)=1.32, Throughput (Tok/s)=148.62026.03 | 79.4 | |
| FrugalGPT (Cascade)Evaluation Protocol=Standard CoT, Emission (gCO2/q)=1.95, Throughput (Tok/s)=88.52026.03 | 77.1 | |
| Qwen-3-8B-InstructEvaluation Protocol=Standard CoT, Emission (gCO2/q)=2.12, Throughput (Tok/s)=98.52026.03 | 75.6 | |
| Llama-3.1-8B-InstructEvaluation Protocol=Standard CoT, Emission (gCO2/q)=2.15, Throughput (Tok/s)=95.82026.03 | 72.4 | |
| Transformer++Params=224M2021.09 | 25.92 | |
| Transformer+GeLUParams=223M2021.09 | 25.13 | |
| Primer-EZ DecoderParams=224M2021.09 | 24.87 | |
| Vanilla TransformerParams=223M2021.09 | 23.02 |