Counterfactual Generation on TAXI1500
93.7SLFRGemma3-12B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gemma3-12BLanguage=de, Protocol=TB-CF2026.05 | 93.7 | 66.7 | 13.01 | 25.7 | 8.387 | -7.387 | |
| Gemma3-4BLanguage=ar, Protocol=TB-CF2026.05 | 91.9 | 33.3 | 17.79 | 32.7 | 11.583 | -10.583 | |
| Gemma3-12BLanguage=sw, Protocol=TB-CF2026.05 | 91.9 | 76.6 | 13.54 | 47.6 | 7.483 | -6.483 | |
| Gemma3-4BLanguage=zh, Protocol=TB-CF2026.05 | 89.2 | 31.5 | 14.64 | 29.1 | 26.717 | -25.717 | |
| Gemma3-12BLanguage=vi, Protocol=TB-CF2026.05 | 89.2 | 55.9 | 12.08 | 24.7 | 8.459 | -7.459 | |
| Gemma3-12BLanguage=ru, Protocol=TB-CF2026.05 | 88.3 | 60.4 | 15.75 | 27.1 | 9.984 | -8.984 | |
| Gemma3-4BLanguage=de, Protocol=TB-CF2026.05 | 87.4 | 32.4 | 11.59 | 29.3 | 8.018 | -7.018 | |
| Gemma3-4BLanguage=sw, Protocol=TB-CF2026.05 | 86.5 | 35.1 | 14.78 | 50.3 | 7.537 | -6.537 | |
| Gemma3-4BLanguage=ru, Protocol=TB-CF2026.05 | 86.5 | 28.8 | 12.73 | 31.7 | 9.932 | -8.932 | |
| Gemma3-4BLanguage=vi, Protocol=TB-CF2026.05 | 83.8 | 27 | 10.46 | 29.5 | 7.942 | -6.942 | |
| Gemma3-12BLanguage=zh, Protocol=TB-CF2026.05 | 83.8 | 64 | 17.4 | 25.7 | 26.798 | -25.798 | |
| Gemma3-12BLanguage=en, Protocol=DG-CF2026.05 | 83.2 | 73.6 | 41.48 | 43.7 | 0.731 | 0.269 | |
| Gemma3-12BLanguage=ar, Protocol=TB-CF2026.05 | 80.2 | 63.1 | 22.22 | 29.8 | 11.946 | -10.946 | |
| Gemma3-12BLanguage=de, Protocol=DG-CF2026.05 | 79.1 | 67.8 | 26.5 | 53.7 | 0.686 | 0.314 | |
| MACROModel=Gemma3-12B, Language=avg2026.05 | 79 | — | 23.8 | 60 | — | — | |
| Gemma3-12BLanguage=ar, Protocol=DG-CF2026.05 | 77.7 | 67.2 | 37.25 | 53.7 | 0.699 | 0.301 | |
| DG-CFModel=Gemma3-12B, Language=avg2026.05 | 75 | — | 26.8 | 55 | — | — | |
| Gemma3-12BLanguage=ru, Protocol=DG-CF2026.05 | 74.9 | 64.3 | 20.49 | 60 | 0.743 | 0.257 | |
| Gemma3-12BLanguage=vi, Protocol=DG-CF2026.05 | 74.4 | 65 | 16.18 | 51.7 | 0.707 | 0.293 | |
| SFTModel=Gemma3-12B, Language=avg2026.05 | 71 | — | 21.5 | 58 | — | — | |
| Gemma3-12BLanguage=sw, Protocol=DG-CF2026.05 | 69.4 | 57.8 | 17.46 | 68.4 | 0.653 | 0.347 | |
| Gemma3-12BLanguage=zh, Protocol=DG-CF2026.05 | 68.2 | 60.5 | 27.9 | 54.4 | 0.85 | 0.15 | |
| Qwen3-4BLanguage=ar, Protocol=DG-CF2026.05 | 68.1 | 34.2 | 86.15 | 57.7 | 0.615 | 0.385 | |
| Qwen3-4BLanguage=vi, Protocol=TB-CF2026.05 | 67.6 | 28.8 | 19.18 | 56.5 | 0.814 | 0.186 | |
| Qwen3-8BLanguage=ar, Protocol=TB-CF2026.05 | 66.7 | 33.3 | 41.3 | 60 | 0.808 | 0.192 | |
| Qwen3-8BLanguage=ar, Protocol=DG-CF2026.05 | 65.8 | 38.7 | 50.29 | 63.9 | 0.622 | 0.378 | |
| Qwen3-4BLanguage=vi, Protocol=DG-CF2026.05 | 65.1 | 35.6 | 25.67 | 66 | 0.485 | 0.515 | |
| Qwen3-4BLanguage=ar, Protocol=TB-CF2026.05 | 64.9 | 27.9 | 55.98 | 56.3 | 0.839 | 0.161 | |
| MACROModel=Qwen3-8B, Language=avg2026.05 | 64 | — | 24.9 | 70 | — | — | |
| Qwen3-4BLanguage=ru, Protocol=TB-CF2026.05 | 63.1 | 32.4 | 25.04 | 62 | 0.848 | 0.152 | |
| Qwen3-8BLanguage=zh, Protocol=DG-CF2026.05 | 63.1 | 36 | 38.92 | 66.2 | 0.689 | 0.311 | |
| Qwen3-4BLanguage=en, Protocol=DG-CF2026.05 | 62.3 | 48.5 | 32.64 | 61.4 | 0.48 | 0.52 | |
| Qwen3-4BLanguage=zh, Protocol=TB-CF2026.05 | 62.2 | 30.6 | 35.72 | 60 | 0.927 | 0.073 | |
| MACROModel=Qwen3-4B, Language=avg2026.05 | 62 | — | 28.9 | 70 | — | — | |
| SFTModel=Qwen3-8B, Language=avg2026.05 | 62 | — | 22.6 | 65 | — | — | |
| MACROModel=Gemma3-4B, Language=en2026.05 | 60 | — | 23.5 | 59 | — | — | |
| MACROModel=Gemma3-4B, Language=avg2026.05 | 60 | — | 23.5 | 68 | — | — | |
| SFTModel=Qwen3-4B, Language=avg2026.05 | 60 | — | 28 | 66 | — | — | |
| Qwen3-4BLanguage=zh, Protocol=DG-CF2026.05 | 58.7 | 30.9 | 45.84 | 67.4 | 0.589 | 0.411 | |
| Qwen3-8BLanguage=zh, Protocol=TB-CF2026.05 | 57.7 | 32.4 | 33.37 | 63.9 | 0.933 | 0.067 | |
| Qwen3-8BLanguage=vi, Protocol=DG-CF2026.05 | 57.7 | 40.5 | 18.83 | 61.3 | 0.621 | 0.379 | |
| DG-CFModel=Qwen3-4B, Language=avg2026.05 | 57 | — | 40.5 | 67 | — | — | |
| Gemma3-4BLanguage=sw, Protocol=DG-CF2026.05 | 56.9 | 35.7 | 25.59 | 67 | 0.647 | 0.353 | |
| SFTModel=Gemma3-4B, Language=en2026.05 | 56 | — | 23.4 | 59 | — | — | |
| SFTModel=Gemma3-4B, Language=avg2026.05 | 56 | — | 24 | 65 | — | — | |
| Qwen3-8BLanguage=en, Protocol=DG-CF2026.05 | 55.9 | 38.7 | 30.95 | 68.2 | 0.474 | 0.526 | |
| Gemma3-4BLanguage=de, Protocol=DG-CF2026.05 | 55.7 | 40.5 | 33.93 | 63.9 | 0.58 | 0.42 | |
| Gemma3-4BLanguage=ru, Protocol=DG-CF2026.05 | 53.2 | 39.4 | 21.49 | 65.6 | 0.65 | 0.35 | |
| Gemma3-4BLanguage=en, Protocol=DG-CF2026.05 | 53.1 | 43.4 | 36.24 | 50.7 | 0.67 | 0.33 | |
| DG-CFModel=Gemma3-4B, Language=en2026.05 | 53 | — | 36.2 | 51 | — | — | |
| DG-CFModel=Gemma3-4B, Language=avg2026.05 | 53 | — | 31 | 62 | — | — | |
| Qwen3-4BLanguage=ru, Protocol=DG-CF2026.05 | 52.9 | 31.5 | 27.73 | 67.5 | 0.526 | 0.474 | |
| Gemma3-4BLanguage=zh, Protocol=DG-CF2026.05 | 52.3 | 40.5 | 39.48 | 61.2 | 0.807 | 0.193 | |
| Gemma3-4BLanguage=ar, Protocol=DG-CF2026.05 | 52.2 | 34.2 | 41.88 | 62.8 | 0.716 | 0.284 | |
| Qwen3-8BLanguage=vi, Protocol=TB-CF2026.05 | 51.4 | 30.6 | 16.36 | 60.3 | 0.849 | 0.151 | |
| Qwen3-4BLanguage=sw, Protocol=TB-CF2026.05 | 50.5 | 11.7 | 14.9 | 55.3 | 1.765 | -0.765 | |
| Gemma3-4BLanguage=vi, Protocol=DG-CF2026.05 | 50.4 | 37.7 | 18.47 | 61.3 | 0.631 | 0.369 | |
| DG-CFModel=Qwen3-8B, Language=avg2026.05 | 50 | — | 28.9 | 67 | — | — | |
| Qwen3-8BLanguage=de, Protocol=DG-CF2026.05 | 49.5 | 34.2 | 25.85 | 61.7 | 0.562 | 0.438 | |
| Qwen3-4BLanguage=de, Protocol=DG-CF2026.05 | 48.6 | 28.4 | 40.17 | 66.2 | 0.444 | 0.556 | |
| Qwen3-4BLanguage=de, Protocol=TB-CF2026.05 | 48.6 | 34.2 | 37.8 | 58.9 | 0.731 | 0.269 | |
| Qwen3-4BLanguage=sw, Protocol=DG-CF2026.05 | 46.3 | 10.7 | 25.61 | 84 | 0.312 | 0.688 | |
| Qwen3-8BLanguage=ru, Protocol=DG-CF2026.05 | 45 | 27 | 18.73 | 67.6 | 0.566 | 0.434 | |
| Qwen3-8BLanguage=de, Protocol=TB-CF2026.05 | 43.2 | 25.2 | 26.46 | 64.6 | 0.727 | 0.273 | |
| Qwen3-8BLanguage=ru, Protocol=TB-CF2026.05 | 37.8 | 28.8 | 18.33 | 67.1 | 0.847 | 0.153 | |
| Qwen3-8BLanguage=sw, Protocol=TB-CF2026.05 | 18 | 5.4 | 18.9 | 61 | 1.241 | -0.241 | |
| Qwen3-8BLanguage=sw, Protocol=DG-CF2026.05 | 10.8 | 0.9 | 18.96 | 83.3 | 0.693 | 0.307 |