# MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU

Source: [MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU](<https://www.sota2.com/research/paper/69d5e1247d0172140ec75b40>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-11T06:00:01.389Z

Authors: Zhengqing Yuan, Hanchi Sun, Lichao Sun, Yanfang Ye

Published: 2026-04-06T18:43:56.000Z

## Abstract

We present MegaTrain, a memory-centric system that efficiently trains 100B+ parameter large language models at full precision on a single GPU. Unlike traditional GPU-centric systems, MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state. To battle the CPU-GPU bandwidth bottleneck, we adopt two key optimizations. 1) We introduce a pipelined double-buffered execution engine that overlaps parameter prefetching, computation, and gradient offloading across multiple CUDA streams, enabling continuous GPU execution. 2) We replace persistent autograd graphs with stateless layer templates, binding weights dynamically as they stream in, eliminating persistent graph metadata while providing flexibility in scheduling. On a single H200 GPU with 1.5TB host memory, MegaTrain reliably trains models up to 120B parameters. It also achieves 1.84$\\times$ the training throughput of DeepSpeed ZeRO-3 with CPU offloading when training 14B models. MegaTrain also enables 7B model training with 512k token context on a single GH200.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mathematical Question Answering on MetaMathQA (final)](<https://www.sota2.com/research/sota/mathematical-question-answering-on-metamathqa-final.md>): Mathematical Question Answering; MetaMathQA (final); Accuracy: 92.52.

- [LLM Training Performance on Qwen models on A6000 (48GB VRAM) 2.5 (train)](<https://www.sota2.com/research/sota/llm-training-performance-on-qwen-models-on-a6000-48gb-vram-2-5-train.md>): LLM Training Performance; Qwen models on A6000 (48GB VRAM) 2.5 (train); TFLOPS: 56.82.

- [LLM Training Performance on Qwen models on RTX 3090 (24GB VRAM) 2.5 (train)](<https://www.sota2.com/research/sota/llm-training-performance-on-qwen-models-on-rtx-3090-24gb-vram-2-5-train.md>): LLM Training Performance; Qwen models on RTX 3090 (24GB VRAM) 2.5 (train); TFLOPS: 35.09.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.05091>)

[PDF](<https://arxiv.org/pdf/2604.05091.pdf>)

[Implementation](<https://github.com/DLYuanGod/MegaTrain>)