Human Intervention Prediction on COWCORPUS (All Data)
85.3Step AccuracyAlways No Interv
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Always No Intervtype=Baseline2026.02 | 85.3 | 0 | 85.3 | 0 | 100 | 0 | 92 | 0 | |
| Gemma 27B (finetuned)shots=0, reasoning=false2026.02 | 85.3 | 50 | 87.7 | 21.6 | 96.3 | 30.2 | 91.8 | 30.3 | |
| Llava 8B (finetuned)shots=0, reasoning=false2026.02 | 81.7 | 47.1 | 87.6 | 21.6 | 92.1 | 29.6 | 89.7 | 20.1 | |
| Claude 3.5 Sonnetshots=2, reasoning=false2026.02 | 74.9 | 15.8 | 85.4 | 16.2 | 85 | 16 | 85.2 | 14.9 | |
| GPT-4oshots=0, reasoning=false2026.02 | 74.1 | 18.2 | 86 | 21.6 | 83.2 | 19.8 | 84.6 | 14.7 | |
| Claude 3.5 Sonnetshots=2, reasoning=true2026.02 | 72.1 | 14.9 | 85.3 | 18.9 | 81.3 | 16.7 | 83.3 | 15.8 | |
| Claude 3.5 Sonnetshots=0, reasoning=true2026.02 | 69.7 | 16.9 | 85.9 | 27 | 77.1 | 20.8 | 81.3 | 16.4 | |
| Gemini 1.5 Proshots=0, reasoning=true2026.02 | 68.9 | 21.1 | 87.8 | 40.5 | 73.8 | 27.8 | 80.2 | 23.7 | |
| Claude 3.5 Sonnetshots=0, reasoning=false2026.02 | 68.1 | 17.9 | 86.4 | 32.4 | 74.3 | 23.1 | 79.9 | 29.3 | |
| Gemini 1.5 Proshots=0, reasoning=false2026.02 | 68.1 | 21.3 | 88.1 | 43.2 | 72.4 | 28.6 | 79.5 | 26.2 | |
| GPT-4oshots=2, reasoning=false2026.02 | 66.1 | 14.7 | 85.2 | 27 | 72.9 | 19 | 78.6 | 20.6 | |
| Gemini 1.5 Proshots=2, reasoning=true2026.02 | 64.1 | 22.1 | 89.7 | 56.8 | 65.4 | 31.8 | 75.7 | 24.3 | |
| Gemini 1.5 Proshots=2, reasoning=false2026.02 | 58.6 | 18.1 | 87.7 | 51.4 | 59.8 | 26.8 | 71.1 | 28.7 | |
| Gemma 27B (base)shots=0, reasoning=false2026.02 | 23.9 | 15.4 | 89.7 | 91.9 | 12.1 | 26.4 | 21.4 | 18.7 | |
| Llava 8B (base)shots=0, reasoning=false2026.02 | 18.3 | 0 | 85.2 | 0 | 21.5 | 0 | 34.3 | 1.7 | |
| Always Intervtype=Baseline2026.02 | 14.7 | 14.7 | 0 | 100 | 0 | 25.7 | 0 | 15.1 |