Word Relation Prediction on Landmark-Continent
87.6AccuracyState vector (mom.)
Evaluation Results
| Method | Links | |
|---|---|---|
| State vector (mom.)Model=Llama-2, Evaluation Protocol=Few-shot2024.04 | 87.6 | |
| State vector (inn.)Model=Llama-2, Evaluation Protocol=Few-shot2024.04 | 87.5 | |
| ICL baselineModel=Llama-2, Evaluation Protocol=Few-shot2024.04 | 87 | |
| Task vectorModel=Llama-2, Evaluation Protocol=Few-shot2024.04 | 86.7 | |
| Function vectorModel=Llama-2, Evaluation Protocol=Few-shot2024.04 | 84.2 | |
| State vector (inn.)Model=Llama-2, Evaluation Protocol=Zero-shot2024.04 | 81 | |
| State vector (mom.)Model=Llama-2, Evaluation Protocol=Zero-shot2024.04 | 77.7 | |
| ICL baselineModel=GPT-J, Evaluation Protocol=Few-shot2024.04 | 69.3 | |
| Function vectorModel=GPT-J, Evaluation Protocol=Few-shot2024.04 | 69.2 | |
| State vector (mom.)Model=GPT-J, Evaluation Protocol=Few-shot2024.04 | 68.6 | |
| State vector (inn.)Model=GPT-J, Evaluation Protocol=Few-shot2024.04 | 66.4 | |
| Task vectorModel=GPT-J, Evaluation Protocol=Few-shot2024.04 | 63.1 | |
| State vector (inn.)Model=GPT-J, Evaluation Protocol=Zero-shot2024.04 | 61.9 | |
| State vector (mom.)Model=GPT-J, Evaluation Protocol=Zero-shot2024.04 | 60.3 | |
| Task vectorModel=Llama-2, Evaluation Protocol=Zero-shot2024.04 | 57.8 | |
| Task vectorModel=GPT-J, Evaluation Protocol=Zero-shot2024.04 | 41.3 | |
| Function vectorModel=Llama-2, Evaluation Protocol=Zero-shot2024.04 | 32.9 | |
| Function vectorModel=GPT-J, Evaluation Protocol=Zero-shot2024.04 | 22.5 | |
| RegularModel=GPT-J, Evaluation Protocol=Zero-shot2024.04 | 0.9 | |
| RegularModel=Llama-2, Evaluation Protocol=Zero-shot2024.04 | 0 |