Protein sequence modeling on TAPE SCOPe-40 16 tasks
11W Scoremasked-position MLM+JEPA
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| masked-position MLM+JEPABackbone=ESM2-150M, Initialization=Pretrained continuation, Optimizer=AdamW, Pre-training budget=8 h wall-clock2026.05 | 11 | 2 | 3 | — | — | |
| masked-position MLM+JEPABackbone=ProteinBERT2-35M, Initialization=Random init (from scratch), Optimizer=AdamW, Pre-training budget=8 h wall-clock2026.05 | 11 | 4 | 1 | — | — | |
| masked-position MLM+JEPABackbone=ESM2-35M, Initialization=Pretrained continuation, Optimizer=AdamW, Pre-training budget=8 h wall-clock2026.05 | 10 | 3 | 3 | — | — | |
| masked-position MLM+JEPABackbone=ESM2-35M, Initialization=Pretrained continuation, Optimizer=SGDm, Pre-training budget=8 h wall-clock2026.05 | 10 | 4 | 2 | — | — | |
| masked-position MLM+JEPABackbone=AMPLIFY-120M, Initialization=Pretrained continuation, Optimizer=AdamW, Pre-training budget=8 h wall-clock2026.05 | 7 | 6 | 3 | — | — | |
| masked-position MLM+JEPABackbone=ESM2-35M, Initialization=Random init (from scratch), Optimizer=AdamW, Pre-training budget=8 h wall-clock2026.05 | 6 | 8 | 2 | — | — |