Scientific paper to Manim animation generation on ManimAgent Paper-Section Dataset (fixed-probe)
84.9Pass@1 (Human Evaluation)ManimAgent (Full EMB / EMB@200)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ManimAgent (Full EMB / EMB@200)Condition=EMB@200 (= Full EMB), Intervention=200-entry EMB snapshot, both channels gated2026.06 | 84.9 | 6.5 | 3.88 | |
| ManimAgent (Manim-code RAG)Condition=Manim-code RAG, Intervention=Static curated Manim corpus, matched retrieval2026.06 | 83.3 | 11.4 | 3.65 | |
| ManimAgent (Control: RAG (doc only))Condition=RAG (doc only), Intervention=Manim docs only, matched budget2026.06 | 82.1 | 11.5 | 3.57 | |
| ManimAgent (Ablation: Ungated M-)Condition=Ungated M-, Intervention=Store every diagnostic, no improvement gate2026.06 | 81.8 | 8.4 | 3.64 | |
| ManimAgent (Ablation: Top-1 retrieval)Condition=Top-1 retrieval, Intervention=One neighbour per channel (k+=k-=1)2026.06 | 81 | 9.9 | 3.84 | |
| ManimAgent (Ablation: No M-)Condition=No M- (failure), Intervention=Retrieve only success memories2026.06 | 80.9 | 11.6 | 3.52 | |
| ManimAgent (Control: Weak VLM)Condition=Weak VLM, Intervention=Step-3.7-Flash replaces the GPT-5.5 reviewer2026.06 | 80 | 8.6 | 3.42 | |
| ManimAgent (EMB@100)Condition=EMB@100, Intervention=100-entry self-grown EMB snapshot2026.06 | 75 | 10.8 | 3.43 | |
| ManimAgent (Ablation: Rationale-only M+)Condition=Rationale-only M+, Intervention=Drop code excerpt from positive injection2026.06 | 75 | 10.5 | 3.5 | |
| ManimAgent (Ablation: No M+)Condition=No M+ (success), Intervention=Retrieve only failure memories2026.06 | 70 | 12 | 3.4 | |
| ManimAgent (Random EMB)Condition=Random EMB, Intervention=Shuffled EMB content, matched count and budget2026.06 | 69.6 | 11.3 | 3.42 | |
| ManimAgent (VLM reflection only)Condition=VLM reflection only, Intervention=Standard reflection without retrieval2026.06 | 68.5 | 12.2 | 3.41 | |
| ManimAgent (EMB@50)Condition=EMB@50, Intervention=50-entry self-grown EMB snapshot2026.06 | 66.7 | 11.3 | 3.31 | |
| ManimAgent (EMB@0)Condition=EMB@0, Intervention=Empty self-grown memory2026.06 | 62 | 12.2 | 3.26 | |
| ManimAgent (Ablation: No memory)Condition=No memory, Intervention=Disable both memory channels2026.06 | 61.8 | 12.2 | 3.2 |