Language Modeling on C4 en (train)
2.94Train LossFedACT
Evaluation results
| Method | Links | |
|---|---|---|
| FedACTModel=Llama2-250M2026.07 | 2.94 | |
| FedAdamWModel=Llama2-250M2026.07 | 2.95 | |
| LocalAdamWModel=Llama2-250M2026.07 | 3.09 | |
| SCAFFOLDModel=Llama2-250M2026.07 | 3.1 | |
| FedACTModel=Llama2-130M2026.07 | 3.18 | |
| FedAdamWModel=Llama2-130M2026.07 | 3.23 | |
| FedLADAModel=Llama2-130M2026.07 | 3.31 | |
| LocalAdamWModel=Llama2-130M2026.07 | 3.36 | |
| SCAFFOLDModel=Llama2-130M2026.07 | 3.4 | |
| FedLADAModel=Llama2-250M2026.07 | 3.46 | |
| FedACTModel=Llama2-60M2026.07 | 3.57 | |
| FedAdamWModel=Llama2-60M2026.07 | 3.65 | |
| LocalAdamWModel=Llama2-60M2026.07 | 3.7 | |
| FedProxModel=Llama2-250M2026.07 | 3.7 | |
| FedLADAModel=Llama2-60M2026.07 | 3.72 | |
| SCAFFOLDModel=Llama2-60M2026.07 | 3.81 | |
| FedProxModel=Llama2-130M2026.07 | 3.97 | |
| FedProxModel=Llama2-60M2026.07 | 4.32 |