Task Retrieval on Natural Instructions 700 domains (held-out prompts)
62.9Top-1 AccuracyLearned linear probe
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Learned linear probeTraining=345s, Model=Llama-3.1-8B, Signature dimension (m)=4096, Training samples per domain=2002026.06 | 62.9 | 86 | |
| Learned MLPTraining=73s, Model=Llama-3.1-8B, Signature dimension (m)=4096, Training samples per domain=2002026.06 | 58.9 | 85.2 | |
| FisherSketch (joint)Training=None, Model=Llama-3.1-8B, Signature dimension (m)=4096, Training samples per domain=2002026.06 | 55.5 | 80.4 | |
| Last token embeddingTraining=None, Model=Llama-3.1-8B, Signature dimension (m)=4096, Training samples per domain=2002026.06 | 54.1 | 77.3 | |
| Mean hidden stateTraining=None, Model=Llama-3.1-8B, Signature dimension (m)=4096, Training samples per domain=2002026.06 | 51.7 | 78.6 |