Inference Efficiency on Natural Questions (NQ)
0.019Relative Overhead (%)Perplexity
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| PerplexityModel=Llama-3-8B2026.03 | 0.019 | — | — | — | |
| PerplexityModel=Mistral-7B2026.03 | 0.021 | — | — | — | |
| PerplexityModel=Llama-2-7B2026.03 | 0.023 | — | — | — | |
| PerplexityModel=Qwen2.5-14B2026.03 | 0.024 | — | — | — | |
| PerplexityModel=Qwen2.5-3B2026.03 | 0.039 | — | — | — | |
| PerplexityModel=Qwen2.5-7B2026.03 | 0.047 | — | — | — | |
| STCModel=Llama-2-7B2026.03 | 0.529 | — | — | — | |
| STCModel=Mistral-7B2026.03 | 0.905 | — | — | — | |
| STCModel=Llama-3-8B2026.03 | 1.734 | — | — | — | |
| STCModel=Qwen2.5-14B2026.03 | 1.974 | — | — | — | |
| STCModel=Qwen2.5-7B2026.03 | 1.975 | — | — | — | |
| STCModel=Qwen2.5-3B2026.03 | 2.564 | — | — | — | |
| P(True)Model=Llama-2-7B2026.03 | 7.275 | — | — | — | |
| tokenSARModel=Llama-2-7B2026.03 | 8.881 | — | — | — | |
| P(True)Model=Qwen2.5-3B2026.03 | 9.225 | — | — | — | |
| tokenSARModel=Qwen2.5-14B2026.03 | 9.642 | — | — | — | |
| P(True)Model=Qwen2.5-14B2026.03 | 9.96 | — | — | — | |
| P(True)Model=Llama-3-8B2026.03 | 12.408 | — | — | — | |
| P(True)Model=Qwen2.5-7B2026.03 | 12.961 | — | — | — | |
| P(True)Model=Mistral-7B2026.03 | 14.072 | — | — | — | |
| tokenSARModel=Llama-3-8B2026.03 | 14.811 | — | — | — | |
| tokenSARModel=Qwen2.5-3B2026.03 | 18.625 | — | — | — | |
| tokenSARModel=Mistral-7B2026.03 | 19.153 | — | — | — | |
| tokenSARModel=Qwen2.5-7B2026.03 | 19.461 | — | — | — | |
| CCPModel=Llama-2-7B2026.03 | 62.049 | — | — | — | |
| CCPModel=Qwen2.5-14B2026.03 | 65.755 | — | — | — | |
| CCPModel=Llama-3-8B2026.03 | 112.903 | — | — | — | |
| CCPModel=Mistral-7B2026.03 | 121.913 | — | — | — | |
| CCPModel=Qwen2.5-7B2026.03 | 125.266 | — | — | — | |
| CCPModel=Qwen2.5-3B2026.03 | 133.458 | — | — | — | |
| DegModel=Qwen2.5-7B2026.03 | 202.916 | — | — | — | |
| EigVModel=Qwen2.5-7B2026.03 | 203.016 | — | — | — | |
| EccModel=Qwen2.5-7B2026.03 | 203.027 | — | — | — | |
| Eigen ScoreModel=Qwen2.5-7B2026.03 | 203.793 | — | — | — | |
| ConUModel=Qwen2.5-7B2026.03 | 204.196 | — | — | — | |
| Predictive EntropyModel=Qwen2.5-7B2026.03 | 235.837 | — | — | — | |
| LN EntropyModel=Qwen2.5-7B2026.03 | 235.861 | — | — | — | |
| Predictive EntropyModel=Llama-2-7B2026.03 | 239.879 | — | — | — | |
| LN EntropyModel=Llama-2-7B2026.03 | 239.893 | — | — | — | |
| DegModel=Llama-2-7B2026.03 | 241.472 | — | — | — | |
| EigVModel=Llama-2-7B2026.03 | 241.534 | — | — | — | |
| EccModel=Llama-2-7B2026.03 | 241.597 | — | — | — | |
| ConUModel=Llama-2-7B2026.03 | 242.183 | — | — | — | |
| Eigen ScoreModel=Llama-2-7B2026.03 | 245.668 | — | — | — | |
| Semantic EntropyModel=Llama-2-7B2026.03 | 248.064 | — | — | — | |
| sentenceSARModel=Llama-2-7B2026.03 | 248.886 | — | — | — | |
| Semantic EntropyModel=Qwen2.5-7B2026.03 | 253.757 | — | — | — | |
| sentenceSARModel=Qwen2.5-7B2026.03 | 255.208 | — | — | — | |
| ConUModel=Qwen2.5-3B2026.03 | 258.312 | — | — | — | |
| DegModel=Qwen2.5-3B2026.03 | 262.121 | — | — | — | |
| EccModel=Qwen2.5-3B2026.03 | 262.264 | — | — | — | |
| EigVModel=Qwen2.5-3B2026.03 | 262.375 | — | — | — | |
| Eigen ScoreModel=Qwen2.5-3B2026.03 | 264.787 | — | — | — | |
| DegModel=Qwen2.5-14B2026.03 | 268.258 | — | — | — | |
| EccModel=Qwen2.5-14B2026.03 | 268.32 | — | — | — | |
| EigVModel=Qwen2.5-14B2026.03 | 268.4 | — | — | — | |
| ConUModel=Qwen2.5-14B2026.03 | 269.439 | — | — | — | |
| DegModel=Llama-3-8B2026.03 | 273.079 | — | — | — | |
| EigVModel=Llama-3-8B2026.03 | 273.189 | — | — | — | |
| EccModel=Llama-3-8B2026.03 | 273.193 | — | — | — | |
| ConUModel=Llama-3-8B2026.03 | 273.873 | — | — | — | |
| Predictive EntropyModel=Qwen2.5-14B2026.03 | 279.658 | — | — | — | |
| LN EntropyModel=Qwen2.5-14B2026.03 | 279.672 | — | — | — | |
| SARModel=Llama-2-7B2026.03 | 280.809 | — | — | — | |
| Eigen ScoreModel=Qwen2.5-14B2026.03 | 282.511 | — | — | — | |
| DegModel=Mistral-7B2026.03 | 285.05 | — | — | — | |
| EigVModel=Mistral-7B2026.03 | 285.145 | — | — | — | |
| EccModel=Mistral-7B2026.03 | 285.163 | — | — | — | |
| Eigen ScoreModel=Llama-3-8B2026.03 | 286.073 | — | — | — | |
| Predictive EntropyModel=Mistral-7B2026.03 | 286.607 | — | — | — | |
| LN EntropyModel=Mistral-7B2026.03 | 286.63 | — | — | — | |
| ConUModel=Mistral-7B2026.03 | 287.006 | — | — | — | |
| Semantic EntropyModel=Qwen2.5-14B2026.03 | 289.045 | — | — | — | |
| sentenceSARModel=Qwen2.5-14B2026.03 | 290.34 | — | — | — | |
| Predictive EntropyModel=Qwen2.5-3B2026.03 | 298.737 | — | — | — | |
| LN EntropyModel=Qwen2.5-3B2026.03 | 298.766 | — | — | — | |
| Eigen ScoreModel=Mistral-7B2026.03 | 300.083 | — | — | — | |
| Predictive EntropyModel=Llama-3-8B2026.03 | 302.765 | — | — | — | |
| LN EntropyModel=Llama-3-8B2026.03 | 302.789 | — | — | — | |
| Semantic EntropyModel=Mistral-7B2026.03 | 303.529 | — | — | — | |
| sentenceSARModel=Mistral-7B2026.03 | 305.668 | — | — | — | |
| sentenceSARModel=Qwen2.5-3B2026.03 | 320.224 | — | — | — | |
| Semantic EntropyModel=Llama-3-8B2026.03 | 321.96 | — | — | — | |
| SARModel=Qwen2.5-7B2026.03 | 322.149 | — | — | — | |
| sentenceSARModel=Llama-3-8B2026.03 | 322.955 | — | — | — | |
| Semantic EntropyModel=Qwen2.5-3B2026.03 | 323.831 | — | — | — | |
| SARModel=Qwen2.5-14B2026.03 | 328.14 | — | — | — | |
| SARModel=Llama-3-8B2026.03 | 379.607 | — | — | — | |
| SARModel=Mistral-7B2026.03 | 384.133 | — | — | — | |
| SARModel=Qwen2.5-3B2026.03 | 400.586 | — | — | — | |
| COCOMTop-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 488 | 2,346 | 138 | |
| ICAETop-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 841 | 3,731 | 169 | |
| LLM (w/o RAG)Top-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 1,096 | 1,188 | 47 | |
| LLM (with RAG)Top-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 1,092 | 3,186 | 60 | |
| LLM (with RAG*)Top-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 1,396 | 3,001 | 69 | |
| LLM (with RAG*)Top-k=5, Backbone=Mistral-7B-Instruct2026.01 | — | 1,537 | 10,811 | 165 | |
| LLMLingua-2Top-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 921 | 2,431 | 84 | |
| PISCOTop-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 502 | 2,251 | 142 | |
| RECOMPTop-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 754 | 2,101 | 98 | |
| SeleComTop-k=1, Backbone=Mistral-7B-Instruct2026.01 | — | 535 | 1,166 | 49 |