Short-Answer Factuality on TruthfulQA (test)
47.47MC1 Factuality ScoreSLED
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SLEDBackbone=Gemma-3-27B-IT, Decoding=SLED2024.11 | 47.47 | 73.58 | 43.53 | — | |
| SLEDBackbone=Gemma-3-12B-IT, Decoding=SLED2024.11 | 46.33 | 71.15 | 40.9 | — | |
| SLEDBackbone=Gemma-3-4B-IT, Decoding=SLED2024.11 | 45.19 | 66.15 | 38.06 | — | |
| Gemma-3-27B-IT (Greedy)Backbone=Gemma-3-27B-IT, Decoding=Greedy2024.11 | 41.14 | 64.06 | 38.71 | — | |
| Gemma-3-12B-IT (Greedy)Backbone=Gemma-3-12B-IT, Decoding=Greedy2024.11 | 39.49 | 63.62 | 37.89 | — | |
| SLEDBackbone=Gemma-3-27B-PT, Decoding=SLED2024.11 | 38.48 | 67.4 | 35.65 | — | |
| SLEDBackbone=Gemma-3-12B-PT, Decoding=SLED2024.11 | 37.85 | 67.05 | 34.59 | — | |
| Gemma-3-4B-IT (Greedy)Backbone=Gemma-3-4B-IT, Decoding=Greedy2024.11 | 36.96 | 58.42 | 34.13 | — | |
| SLEDBackbone=Gemma-3-1B-IT, Decoding=SLED2024.11 | 36.71 | 71.79 | 42 | — | |
| DoLaBackbone=Gemma-3-1B-IT, Decoding=DoLa2024.11 | 36.46 | 68.86 | 35.69 | — | |
| Gemma-3-27B-PT (Greedy)Backbone=Gemma-3-27B-PT, Decoding=Greedy2024.11 | 36.2 | 61.39 | 34.95 | — | |
| DoLaBackbone=Gemma-3-4B-IT, Decoding=DoLa2024.11 | 36.08 | 68.59 | 35.58 | — | |
| Gemma-3-12B-PT (Greedy)Backbone=Gemma-3-12B-PT, Decoding=Greedy2024.11 | 35.82 | 63.7 | 34.83 | — | |
| SLEDBackbone=Gemma-3-1B-PT, Decoding=SLED2024.11 | 35.57 | 71.79 | 40.69 | — | |
| DoLaBackbone=Gemma-3-1B-PT, Decoding=DoLa2024.11 | 35.44 | 68.9 | 35.29 | — | |
| SLEDBackbone=Gemma-3-4B-PT, Decoding=SLED2024.11 | 35.19 | 69.25 | 38.96 | — | |
| DoLaBackbone=Gemma-3-12B-IT, Decoding=DoLa2024.11 | 35.19 | 67.53 | 35.04 | — | |
| DoLaBackbone=Gemma-3-4B-PT, Decoding=DoLa2024.11 | 34.68 | 68.21 | 34.9 | — | |
| Gemma-3-4B-PT (Greedy)Backbone=Gemma-3-4B-PT, Decoding=Greedy2024.11 | 33.8 | 64.12 | 32.64 | — | |
| DoLaBackbone=Gemma-3-12B-PT, Decoding=DoLa2024.11 | 33.54 | 66.62 | 34.25 | — | |
| Gemma-3-1B-IT (Greedy)Backbone=Gemma-3-1B-IT, Decoding=Greedy2024.11 | 33.29 | 61.65 | 30.74 | — | |
| DoLaBackbone=Gemma-3-27B-IT, Decoding=DoLa2024.11 | 33.16 | 65.93 | 33.63 | — | |
| Gemma-3-1B-PT (Greedy)Backbone=Gemma-3-1B-PT, Decoding=Greedy2024.11 | 32.28 | 63.22 | 30.98 | — | |
| DoLaBackbone=Gemma-3-27B-PT, Decoding=DoLa2024.11 | 31.65 | 65.03 | 32.72 | — | |
| STARLING (BLUE-RED)Data strategy=BLUE-RED2023.08 | — | — | — | 49.6 | |
| STARLING (BLUE)Data strategy=BLUE2023.08 | — | — | — | 48.9 | |
| VICUNA-7BModel size=7B2023.08 | — | — | — | 46.99 | |
| VICUNA-FT-7BModel size=7B, Training=Fine-tuned2023.08 | — | — | — | 48.85 |