Reinforcement Learning on Procgen (test)
21.61BigFish ReturnSparse Masked Attention Policies
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sparse Masked Attention PoliciesNumber of seeds=102026.02 | 21.61 | — | 8.84 | 30.74 | — | 9.11 | 46.38 | — | 5.41 | 4.12 | — | 6.76 | 10.95 | — | — | — | 8.73 | — | |
| IDAACTraining levels=200, Environment steps=25M, Number of runs=102023.06 | 18.5 | 5 | 9.4 | 27.9 | 6.3 | 7.7 | 37 | — | 5.6 | 3.5 | 9.5 | 5.3 | 3.2 | 23.3 | 6.8 | 9.8 | 6.8 | — | |
| ExpGen (IDAAC)Base Algorithm=IDAAC, Training levels=200, Environment steps=25M, Number of runs=102023.06 | 18.5 | 5.3 | 9.3 | 28.4 | 6.8 | 7.6 | 39.8 | — | 7.8 | 7.2 | 9.8 | 9.5 | 2.8 | 23.6 | 6.6 | 9.8 | 7.1 | — | |
| Input Masked AttentionNumber of seeds=102026.02 | 11.96 | — | 8.21 | 29.61 | — | 8.39 | 38.43 | — | 4.49 | 2.74 | — | 5.54 | 4.86 | — | — | — | 2.25 | — | |
| AttentionNumber of seeds=102026.02 | 11.32 | — | 8.42 | 30.07 | — | 8.88 | 36.69 | — | 4.88 | 2.62 | — | 5.71 | 3.85 | — | — | — | 2.71 | — | |
| PPGTraining levels=200, Environment steps=25M, Number of runs=102023.06 | 11.2 | 7 | 8.9 | 27.8 | 5.9 | 8.5 | 47.2 | — | 5.1 | 2.8 | 7.4 | 8.3 | 2.3 | 14.3 | 6.6 | 10.3 | 9.8 | — | |
| PLRTraining levels=200, Environment steps=25M, Number of runs=102023.06 | 10.9 | 6.3 | 8.8 | 28 | 5.5 | 6.8 | 27.9 | — | 6.3 | 2.9 | 9.6 | 6.3 | 1.8 | 8.7 | 7.2 | 8.9 | 6.9 | — | |
| PB2-MixPopulation Size=4, Timesteps per Agent=25M, Hyperparameter Initialization=Random, Evaluation Trials=1002021.06 | 10.6 | 6 | 8.1 | 29 | 6.2 | 6.6 | 36.3 | 112.1 | — | — | — | — | — | — | — | — | — | — | |
| UCB-DrACAgent Count=1, Timesteps=25M, Hyperparameter Initialization=Optimized, Evaluation Trials=1002021.06 | 9.7 | 5.3 | 8.5 | 28.3 | 6.4 | 5 | 30.2 | — | — | — | — | — | — | — | — | — | — | — | |
| PB2-MultPopulation Size=4, Timesteps per Agent=25M, Hyperparameter Initialization=Random, Evaluation Trials=1002021.06 | 9.6 | 5.6 | 8.5 | 28 | 5.9 | 5 | 35.3 | 105.3 | — | — | — | — | — | — | — | — | — | — | |
| UCB-DrACTraining levels=200, Environment steps=25M, Number of runs=102023.06 | 9.2 | 5 | 8.6 | 27.6 | 6.2 | 4.8 | 30 | — | 6.3 | 3.5 | 9.2 | 6.3 | 4.2 | 8.3 | 6.6 | 7.8 | 6.3 | — | |
| PB2-RandPopulation Size=4, Timesteps per Agent=25M, Hyperparameter Initialization=Random, Evaluation Trials=1002021.06 | 8.2 | 5.8 | 8.1 | 29.5 | 5.4 | 5 | 33.6 | 100 | — | — | — | — | — | — | — | — | — | — | |
| PBTPopulation Size=4, Timesteps per Agent=25M, Hyperparameter Initialization=Random, Evaluation Trials=1002021.06 | 6.6 | 4.4 | 6.8 | 25.6 | 6.2 | 3.8 | 26.9 | 85.6 | — | — | — | — | — | — | — | — | — | — | |
| ExpGen (PPO)Base Algorithm=PPO, Training levels=200, Environment steps=25M, Number of runs=102023.06 | 6 | 5.7 | 8.8 | 26.2 | 6.7 | 4 | 31 | — | 8.3 | 7.4 | 8 | 5.9 | 2.9 | 5.5 | 6.6 | 7.7 | 3.6 | — | |
| ExpGenTraining instances=200, Environment steps=50M, Evaluation seeds=82023.06 | 5.99 | 6.36 | — | — | 6.64 | — | — | — | 8.33 | 6.91 | 9.48 | 6.29 | 3.84 | 6.91 | 6.75 | — | — | — | |
| LEEPTraining instances=200, Environment steps=50M, Evaluation seeds=82023.06 | 5.5 | 2.6 | — | — | 6.4 | — | — | — | 6.78 | 4.42 | 0.8 | 2.6 | 4.58 | 4.2 | 4.9 | — | — | — | |
| LEEPTraining levels=200, Environment steps=25M, Number of runs=102023.06 | 4.9 | 4.9 | 7.3 | 16.4 | 5.4 | 4.4 | 3.2 | — | 6.6 | 4.5 | 1.1 | 2.6 | 1.9 | 4.4 | 4.4 | 0.5 | 3 | — | |
| PPOAgent Count=1, Timesteps=25M, Hyperparameter Initialization=Optimized, Evaluation Trials=1002021.06 | 4 | 5.1 | 8.5 | 26.7 | 5.8 | 4.9 | 24.7 | — | — | — | — | — | — | — | — | — | — | — | |
| PPOTraining instances=200, Environment steps=50M, Evaluation seeds=82023.06 | 3.82 | 6.19 | — | — | 5.78 | — | — | — | 5.78 | 2.54 | 8.76 | 6.14 | 3.71 | 7.78 | 6.94 | — | — | — | |
| PPOTraining levels=200, Environment steps=25M, Number of runs=102023.06 | 2.9 | 5.5 | 8.6 | 26.2 | 5.8 | 4.9 | 24.9 | — | 5.6 | 2.4 | 7.8 | 5.4 | 2.2 | 7.8 | 6.1 | 7.4 | 3.1 | — | |
| CNNNumber of seeds=102026.02 | 2.35 | — | 8.92 | 27.68 | — | 3.71 | 42.56 | — | 4.69 | 1.98 | — | 5.01 | 2.19 | — | — | — | 6.12 | — | |
| PLRTraining steps=200M2021.11 | — | — | — | — | — | — | — | 34.5 | — | — | — | — | — | — | — | — | — | — | |
| PPDStudent model size=smaller2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.93 | |
| PPDStudent model size=same-size2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.03 | |
| PPDStudent model size=larger2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.04 | |
| PPOTraining steps=200M2021.11 | — | — | — | — | — | — | — | 22 | — | — | — | — | — | — | — | — | — | — | |
| student-distillStudent model size=smaller2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.78 | |
| student-distillStudent model size=same-size2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.88 | |
| student-distillStudent model size=larger2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.96 | |
| teacher-distillStudent model size=smaller2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.72 | |
| teacher-distillStudent model size=same-size2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.84 | |
| teacher-distillStudent model size=larger2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.91 | |
| UCB-DrAC+PLRTraining steps=200M2021.11 | — | — | — | — | — | — | — | 48.5 | — | — | — | — | — | — | — | — | — | — |