Relevance Classification on Six entity-pair tasks T1-T6 (test)
89.96Overall ScoreMHP Ensemble
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| MHP EnsembleCategory=Ensemble, Routing=MHP2026.06 | 89.96 | 86.19 | 84.64 | 94.38 | 64.21 | 89.07 | 87.61 | 97.26 | |
| ModernBERTCategory=Multi-task (Text-Prefix), Routing=SH, Params=396M2026.06 | 89.26 | 85.55 | 82.58 | 93.9 | 62.3 | 88.56 | 88.13 | 97.81 | |
| ELECTRACategory=Multi-task (Text-Prefix), Routing=SH, Params=109M2026.06 | 88.38 | 84.73 | 82.14 | 93.28 | 63.58 | 87.36 | 85.8 | 96.57 | |
| ModernBERTCategory=Multi-task (MH+Private), Routing=MHP, Params=396M2026.06 | 88.31 | 83.95 | 81.5 | 93 | 60.2 | 87.86 | 84.2 | 96.94 | |
| Llama-3-8bCategory=Multi-task (MH+Private), Routing=MHP, Params=8B2026.06 | 88.27 | 84.25 | 82.69 | 92.73 | 61.74 | 87.38 | 85.04 | 95.93 | |
| ModernBERTCategory=Multi-task (Multi-Head), Routing=MH, Params=396M2026.06 | 88.24 | 83.88 | 81.43 | 92.98 | 60.14 | 87.76 | 84.11 | 96.88 | |
| Gemma-2-2bCategory=Multi-task (MH+Private), Routing=MHP, Params=2B2026.06 | 88.04 | 83.76 | 82.49 | 92.33 | 61.18 | 86.99 | 83.59 | 95.99 | |
| SH EnsembleCategory=Ensemble, Routing=SH2026.06 | 88.04 | 84.46 | 82.46 | 92.9 | 63.41 | 86.54 | 85.46 | 96.02 | |
| Llama-3-8bCategory=Multi-task (Text-Prefix), Routing=SH, Params=8B2026.06 | 87.88 | 84.63 | 83.02 | 93.25 | 64.67 | 86.08 | 85.74 | 95.02 | |
| Gemma-2-2bCategory=Multi-task (Text-Prefix), Routing=SH, Params=2B2026.06 | 87.14 | 83.56 | 81.96 | 92.45 | 62.91 | 85.83 | 83.89 | 94.3 | |
| ModernBERTCategory=Single-task, Params=396M2026.06 | 85.73 | 82.2 | 84.21 | 91.45 | 57.38 | 86.12 | 78.23 | 95.82 | |
| Llama-3-8bCategory=Single-task, Params=8B2026.06 | 84.89 | 82.4 | 87.45 | 86.12 | 74.83 | 85.91 | 71.47 | 88.63 | |
| Gemma-2-2bCategory=Single-task, Params=2B2026.06 | 83.42 | 81.04 | 86.71 | 85.23 | 72.15 | 84.38 | 69.82 | 87.94 | |
| Llama-3-8bCategory=Multi-task (Multi-Head), Routing=MH, Params=8B2026.06 | 80.96 | 79.97 | 77.9 | 92.66 | 62.22 | 72.27 | 84.5 | 90.28 | |
| GPT-4 (few-shot)Category=Off-the-shelf, Params=∼1.8T2026.06 | 67.2 | 66.22 | 72.1 | 69.4 | 58.3 | 65.8 | 61.2 | 70.5 | |
| Llama-3-8b (zero)Category=Off-the-shelf, Params=8B2026.06 | 61.3 | 60.18 | 65.4 | 63.5 | 52.8 | 59.7 | 55.4 | 64.3 | |
| Gemma-2-2b†Category=Multi-task (Multi-Head), Routing=MH, Params=2B2026.06 | 55.59 | 46.43 | 47.36 | 33.16 | 33.14 | 44.81 | 33.41 | 86.71 |