completion task on MemoTrap (test)
61.83AccuracyIRCAN-CAD
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| IRCAN-CADModel Backbone=LLaMA-2-7B2024.06 | 61.83 | 12.79 | |
| IRCAN-CADModel Backbone=LLaMA-2-13B2024.06 | 58.64 | 9.38 | |
| IRCANModel Backbone=LLaMA-2-7B2024.06 | 56.08 | 18.55 | |
| IRCAN-CADModel Backbone=LLaMA-3-8B2024.06 | 54.37 | 16.84 | |
| IRCANModel Backbone=LLaMA-2-13B2024.06 | 52.24 | 14.29 | |
| IRCANModel Backbone=LLaMA-3-8B2024.06 | 47.76 | 20.68 | |
| IRCAN-CADModel Backbone=Amber (7B)2024.06 | 45.84 | 25.59 | |
| CADModel Backbone=LLaMA-2-7B2024.06 | 44.56 | 32.84 | |
| IRCANModel Backbone=Amber (7B)2024.06 | 41.15 | 31.56 | |
| CADModel Backbone=LLaMA-3-8B2024.06 | 39.66 | 36.03 | |
| CADModel Backbone=LLaMA-2-13B2024.06 | 39.23 | 23.24 | |
| CADModel Backbone=Amber (7B)2024.06 | 36.07 | 34.97 | |
| ITI (Probe Weight Direction)Model Backbone=LLaMA-2-7B2024.06 | 31.77 | 44.78 | |
| ITI (Mass Mean Shift)Model Backbone=LLaMA-2-7B2024.06 | 31.34 | 44.99 | |
| IRCAN-CADModel Backbone=Gemma-2B2024.06 | 27.08 | 17.27 | |
| OriginalModel Backbone=LLaMA-2-13B2024.06 | 27.08 | 46.7 | |
| ITI (Probe Weight Direction)Model Backbone=Gemma-2B2024.06 | 26.01 | 25.16 | |
| OriginalModel Backbone=Amber (7B)2024.06 | 24.95 | 48.4 | |
| IRCANModel Backbone=Gemma-2B2024.06 | 24.73 | 30.28 | |
| CADModel Backbone=Gemma-2B2024.06 | 24.52 | 21.96 | |
| OriginalModel Backbone=LLaMA-2-7B2024.06 | 24.52 | 50.96 | |
| OriginalModel Backbone=Gemma-2B2024.06 | 23.24 | 35.82 | |
| ITI (Probe Weight Direction)Model Backbone=LLaMA-2-13B2024.06 | 23.03 | 51.17 | |
| ITI (Mass Mean Shift)Model Backbone=LLaMA-2-13B2024.06 | 22.6 | 52.45 | |
| ITI (Probe Weight Direction)Model Backbone=Amber (7B)2024.06 | 20.26 | 43.5 | |
| OriginalModel Backbone=LLaMA-3-8B2024.06 | 20.26 | 53.3 | |
| ITI (Probe Weight Direction)Model Backbone=LLaMA-3-8B2024.06 | 18.34 | 53.52 | |
| ITI (Mass Mean Shift)Model Backbone=LLaMA-3-8B2024.06 | 18.12 | 53.94 | |
| ITI (Mass Mean Shift)Model Backbone=Gemma-2B2024.06 | 0 | 0 | |
| ITI (Mass Mean Shift)Model Backbone=Amber (7B)2024.06 | 0 | 0 |