Federated Language Modeling on The Pile (train)
5,144Wall Time (PPL=42)Photon
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| PhotonNumber of clients (N)=4, Model Parameters=125M, Global Batch Size (Bg)=128, Local Batch Size (Bl)=322024.11 | 5,144 | 9,516.4 | |
| PhotonNumber of clients (N)=8, Model Parameters=125M, Global Batch Size (Bg)=256, Local Batch Size (Bl)=322024.11 | 5,148 | 9,523.8 | |
| PhotonNumber of clients (N)=2, Model Parameters=125M, Global Batch Size (Bg)=64, Local Batch Size (Bl)=322024.11 | 5,392.8 | 10,015.2 | |
| DiLoCoNumber of clients (N)=8, Server Learning Rate (ηs)=0.1, Model Parameters=125M, Global Batch Size (Bg)=256, Local Batch Size (Bl)=322024.11 | 9,523.8 | 20,334.6 | |
| DiLoCoNumber of clients (N)=2, Server Learning Rate (ηs)=0.1, Model Parameters=125M, Global Batch Size (Bg)=64, Local Batch Size (Bl)=322024.11 | 10,528.8 | 19,516.8 | |
| DiLoCoNumber of clients (N)=4, Server Learning Rate (ηs)=0.1, Model Parameters=125M, Global Batch Size (Bg)=128, Local Batch Size (Bl)=322024.11 | 10,545.2 | 19,032.8 |