Question Answering on Qasper (F1)
36.9F1 ScoreLSTM-MAS
Evaluation Results
| Method | Links | |
|---|---|---|
| LSTM-MASBackbone=Phi-3-small-8k-instruct, Context Window=8K2026.01 | 36.9 | |
| CoABackbone=Llama-3.1-8B-Instruct, Context Window=8K2026.01 | 35.68 | |
| LSTM-MASBackbone=Qwen2.5-7B, Context Window=8K2026.01 | 32.85 | |
| LSTM-MASBackbone=Llama-3.1-8B-Instruct, Context Window=8K2026.01 | 32.24 | |
| LSTM-MASBackbone=Qwen2.5-32B, Context Window=8K2026.01 | 31.74 | |
| LSTM-MASBackbone=Phi-3.5-mini-instruct, Context Window=8K2026.01 | 31.04 | |
| CoABackbone=Phi-3-small-8k-instruct, Context Window=8K2026.01 | 30.03 | |
| CoABackbone=Phi-3.5-mini-instruct, Context Window=8K2026.01 | 29.89 | |
| CoABackbone=Llama-3.2-3B-Instruct, Context Window=8K2026.01 | 28.34 | |
| VanillaBackbone=Qwen2.5-14B, Context Window=8K2026.01 | 27.52 | |
| VanillaBackbone=Llama-3.1-8B-Instruct, Context Window=8K2026.01 | 24.78 | |
| LSTM-MASBackbone=Qwen2.5-1.5B, Context Window=8K2026.01 | 24.17 | |
| LSTM-MASBackbone=Qwen2.5-14B, Context Window=8K2026.01 | 23.93 | |
| CoABackbone=Qwen2.5-7B, Context Window=8K2026.01 | 22.86 | |
| RAG-RerankModel=Mistral-7B-Instruct-v0.32026.01 | 22.51 | |
| HyDE-RerankModel=Mistral-7B-Instruct-v0.32026.01 | 22.3 | |
| BM25Model=Mistral-7B-Instruct-v0.32026.01 | 21.9 | |
| S3-HybridModel=Mistral-7B-Instruct-v0.32026.01 | 21.87 | |
| LSTM-MASBackbone=Llama-3.2-3B-Instruct, Context Window=8K2026.01 | 21.81 | |
| FullKVModel=Mistral-7B-Instruct-v0.32026.01 | 21.77 | |
| RAG-RerankModel=Llama-3.1-8B-Instruct2026.01 | 21.73 | |
| HyDE-RerankModel=Llama-3.1-8B-Instruct2026.01 | 21.73 | |
| S3-HybridModel=Llama-3.1-8B-Instruct2026.01 | 21.35 | |
| BM25Model=Llama-3.1-8B-Instruct2026.01 | 21.34 | |
| CoABackbone=Qwen2.5-1.5B, Context Window=8K2026.01 | 21.31 | |
| VanillaBackbone=Llama-3.2-3B-Instruct, Context Window=8K2026.01 | 21.23 | |
| FullKVModel=Llama-3.1-8B-Instruct2026.01 | 20.9 | |
| RAG-RerankModel=Qwen2-7B-Instruct2026.01 | 20.44 | |
| S3-PureModel=Mistral-7B-Instruct-v0.32026.01 | 20.42 | |
| HyDE-RerankModel=Qwen2-7B-Instruct2026.01 | 20.42 | |
| BM25Model=Qwen2-7B-Instruct2026.01 | 20.31 | |
| S3-PureModel=Llama-3.1-8B-Instruct2026.01 | 20.08 | |
| S3-HybridModel=Qwen2-7B-Instruct2026.01 | 20.04 | |
| FullKVModel=Qwen2-7B-Instruct2026.01 | 19.73 | |
| VanillaBackbone=Phi-3-small-8k-instruct, Context Window=8K2026.01 | 19 | |
| S3-PureModel=Qwen2-7B-Instruct2026.01 | 17.24 | |
| CoABackbone=Qwen2.5-32B, Context Window=8K2026.01 | 16.91 | |
| CoABackbone=Qwen2.5-14B, Context Window=8K2026.01 | 16.69 | |
| LONGAGENTBackbone=Qwen2.5-32B, Context Window=8K2026.01 | 15.08 | |
| LONGAGENTBackbone=Qwen2.5-14B, Context Window=8K2026.01 | 12.51 | |
| LONGAGENTBackbone=Llama-3.1-8B-Instruct, Context Window=8K2026.01 | 12.46 | |
| LightRAGBackbone=Qwen2.5-1.5B2026.01 | 12.44 | |
| RAG-DenseModel=Mistral-7B-Instruct-v0.32026.01 | 11.43 | |
| LightRAGBackbone=Llama-3.2-3B-Instruct2026.01 | 11.41 | |
| LSTM-MASBackbone=Qwen2.5-0.5B, Context Window=8K2026.01 | 11.17 | |
| RAG-DenseModel=Qwen2-7B-Instruct2026.01 | 11.15 | |
| VanillaBackbone=Qwen2.5-32B, Context Window=8K2026.01 | 10.91 | |
| RAG-DenseModel=Llama-3.1-8B-Instruct2026.01 | 10.56 | |
| VanillaBackbone=Phi-3.5-mini-instruct, Context Window=8K2026.01 | 10.12 | |
| LONGAGENTBackbone=Phi-3.5-mini-instruct, Context Window=8K2026.01 | 9.62 | |
| VanillaBackbone=Qwen2.5-7B, Context Window=8K2026.01 | 9.56 | |
| LONGAGENTBackbone=Qwen2.5-1.5B, Context Window=8K2026.01 | 9.44 | |
| LONGAGENTBackbone=Phi-3-small-8k-instruct, Context Window=8K2026.01 | 9.28 | |
| CoABackbone=Qwen2.5-0.5B, Context Window=8K2026.01 | 9.02 | |
| LONGAGENTBackbone=Qwen2.5-7B, Context Window=8K2026.01 | 8.09 | |
| LightRAGBackbone=Llama-3.1-8B-Instruct2026.01 | 7.42 | |
| LONGAGENTBackbone=Llama-3.2-3B-Instruct, Context Window=8K2026.01 | 4.01 | |
| VanillaBackbone=Qwen2.5-1.5B, Context Window=8K2026.01 | 3.65 | |
| LightRAGBackbone=Qwen2.5-0.5B2026.01 | 3.09 | |
| VanillaBackbone=Qwen2.5-0.5B, Context Window=8K2026.01 | 2.03 | |
| LONGAGENTBackbone=Qwen2.5-0.5B, Context Window=8K2026.01 | 0.71 |