Creative writing on Mythos 100 prompts (test)
834Word CountGPT-5
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5Generation setting=Vanilla Baseline2025.05 | 834 | — | 100 | 100 | 1.71 | 1.03 | 0.19 | 4.2 | 0 | |
| Qwen-3-32BGeneration setting=Vanilla Baseline2025.05 | 699 | — | 100 | 100 | 1 | 5.86 | 0.18 | 3.22 | 0 | |
| GPT-5Generation setting=Frankentext, Snippets=1.5k2025.05 | 675 | 82 | 92 | 42 | 2.76 | 4.34 | 0.21 | 5.88 | 79 | |
| Gemini 2.5 ProGeneration setting=Vanilla Baseline2025.05 | 593 | — | 100 | 100 | 1.76 | 6.41 | 0.19 | 3.18 | 0 | |
| Qwen-3-32BGeneration setting=Frankentext, Snippets=1.5k2025.05 | 578 | 36 | 91 | 54 | 2.2 | 1.37 | 0.18 | 4.02 | 7 | |
| Deepseek-R1Generation setting=Vanilla Baseline2025.05 | 550 | — | 100 | 100 | 1.28 | 3.49 | 0.2 | 4.13 | 0 | |
| Gemini-2.5-ProGeneration setting=RAG Baseline, Snippets=1.5k2025.05 | 538 | 0.63 | 100 | 99 | 1.56 | 6.43 | 0.2 | 3.46 | 2 | |
| Gemini 2.5 ProGeneration setting=Frankentext, Snippets=1.5k2025.05 | 521 | 75 | 100 | 81 | 2.74 | 9.27 | 0.22 | 4.21 | 59 | |
| Claude-4-SonnetGeneration setting=Vanilla Baseline2025.05 | 477 | — | 100 | 100 | 1.4 | 1.7 | 0.18 | 3.31 | 0 | |
| Gemini 2.5 ProGeneration setting=Frankentext (Ablation), Snippets=5k2025.05 | 451 | 79 | 97 | 85 | 2.78 | 9.48 | 0.21 | 5.13 | 72 | |
| Gemini 2.5 ProGeneration setting=Frankentext (Ablation), Snippets=10k2025.05 | 448 | 78 | 99 | 85 | 2.81 | 9.12 | 0.21 | 5.43 | 70 | |
| Claude-4-SonnetGeneration setting=Frankentext, Snippets=1.5k2025.05 | 317 | 51 | 98 | 86 | 2.6 | 5 | 0.19 | 3.99 | 47 | |
| Deepseek-R1Generation setting=Frankentext, Snippets=1.5k2025.05 | 303 | 42 | 91 | 72 | 2.79 | 8.31 | 0.2 | 4.66 | 23 |