ResearchBenchmarksOptimizer Memory Footprint Analysis on OpenWebText GPT-2 125M (train)Optimizer Memory Footprint Analysis on OpenWebText GPT-2 125M (train)Follow0.25Peak Memory FootprintGefen0.13960.88481.632.3752Jun 11, 2026Evaluation resultsMethodMethodLinksPeak Memory FootprintPersistent Memory FootprintGefen2026.060.250.25Adam4bit2026.060.30.3Adam8bit2026.060.510.51Adam-mini2026.061.021.01SM32026.061.841.01AdamW2026.062.012.01Adafactor2026.063.011.01