# Resolving Confusion on Lacuna-5 (test) Source: [Resolving Confusion on Lacuna-5 (test)]() Collection index: [llms.txt]() Data updated: 2026-02-26T06:06:15.754Z Evaluation of model performance on the test split of the Lacuna-5 dataset, specifically measuring test error, retention error, and forgetting error, including incremental learning (IC) and forgetting (Fgt) variants. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Resolving Confusion | | Dataset (including any stated split) | Lacuna-5 (test) | | Metric | Test Error | | Unit | PERCENT | | Direction | Lower is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Test Error](): Lower is better; unit: PERCENT. - [Retention Error](): Lower is better; unit: PERCENT. - [Forgetting Error](): Lower is better; unit: PERCENT. - [IC Test Error](): Lower is better; unit: PERCENT. - [IC Retention Error](): Lower is better; unit: PERCENT. - [Fgt Test Error](): Lower is better; unit: PERCENT. - [Fgt Retention Error](): Lower is better; unit: PERCENT. ## Recorded results | Method | Test Error | Result date | Report paper | | --- | --- | --- | --- | | SCRUB | 3.87 | Not recorded | 68fa71011e209131cea4e7b9 | | SCRUB | 3.87 | Not recorded | 68fa71011e209131cea4e7b9 | | Retrain | 4.2 | Not recorded | 68fa71011e209131cea4e7b9 | | Retrain | 4.2 | Not recorded | 68fa71011e209131cea4e7b9 | | Retrain | 6 | Not recorded | 68fa71011e209131cea4e7b9 | | SCRUB | 8.47 | Not recorded | 68fa71011e209131cea4e7b9 | | Finetune | 12.8 | Not recorded | 68fa71011e209131cea4e7b9 | | NegGrad+ | 12.8 | Not recorded | 68fa71011e209131cea4e7b9 | | Finetune | 12.8 | Not recorded | 68fa71011e209131cea4e7b9 | | NegGrad+ | 12.8 | Not recorded | 68fa71011e209131cea4e7b9 | | NTK | 12.87 | Not recorded | 68fa71011e209131cea4e7b9 | | NTK | 12.87 | Not recorded | 68fa71011e209131cea4e7b9 | | NTK | 14.53 | Not recorded | 68fa71011e209131cea4e7b9 | | EU-k | 15.27 | Not recorded | 68fa71011e209131cea4e7b9 | | EU-k | 17 | Not recorded | 68fa71011e209131cea4e7b9 | | EU-k | 17 | Not recorded | 68fa71011e209131cea4e7b9 | | NegGrad+ | 17.8 | Not recorded | 68fa71011e209131cea4e7b9 | | Finetune | 18.8 | Not recorded | 68fa71011e209131cea4e7b9 | | CF-k | 21.27 | Not recorded | 68fa71011e209131cea4e7b9 | | CF-k | 21.27 | Not recorded | 68fa71011e209131cea4e7b9 | | CF-k | 22.27 | Not recorded | 68fa71011e209131cea4e7b9 | | Original | 25.47 | Not recorded | 68fa71011e209131cea4e7b9 | | Original | 25.47 | Not recorded | 68fa71011e209131cea4e7b9 | | Original | 27.07 | Not recorded | 68fa71011e209131cea4e7b9 | | Fisher | 35.87 | Not recorded | 68fa71011e209131cea4e7b9 | | Fisher | 49.6 | Not recorded | 68fa71011e209131cea4e7b9 | | Fisher | 49.6 | Not recorded | 68fa71011e209131cea4e7b9 | ### SCRUB Test Error: 3.87. Result ID: 699186293a6c31484908b657. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### SCRUB Test Error: 3.87. Result ID: 699186293a6c31484908b6b2. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Retrain Test Error: 4.2. Result ID: 699186293a6c31484908b64f. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### Retrain Test Error: 4.2. Result ID: 699186293a6c31484908b6aa. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Retrain Test Error: 6. Result ID: 699186293a6c31484908b6a1. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### SCRUB Test Error: 8.47. Result ID: 699186293a6c31484908b6a9. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Finetune Test Error: 12.8. Result ID: 699186293a6c31484908b651. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### NegGrad+ Test Error: 12.8. Result ID: 699186293a6c31484908b652. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### Finetune Test Error: 12.8. Result ID: 699186293a6c31484908b6ac. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### NegGrad+ Test Error: 12.8. Result ID: 699186293a6c31484908b6ad. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### NTK Test Error: 12.87. Result ID: 699186293a6c31484908b656. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### NTK Test Error: 12.87. Result ID: 699186293a6c31484908b6b1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### NTK Test Error: 14.53. Result ID: 699186293a6c31484908b6a8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### EU-k Test Error: 15.27. Result ID: 699186293a6c31484908b6a6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### EU-k Test Error: 17. Result ID: 699186293a6c31484908b654. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### EU-k Test Error: 17. Result ID: 699186293a6c31484908b6af. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### NegGrad+ Test Error: 17.8. Result ID: 699186293a6c31484908b6a4. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Finetune Test Error: 18.8. Result ID: 699186293a6c31484908b6a3. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### CF-k Test Error: 21.27. Result ID: 699186293a6c31484908b653. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### CF-k Test Error: 21.27. Result ID: 699186293a6c31484908b6ae. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### CF-k Test Error: 22.27. Result ID: 699186293a6c31484908b6a5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Original Test Error: 25.47. Result ID: 699186293a6c31484908b650. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### Original Test Error: 25.47. Result ID: 699186293a6c31484908b6ab. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Original Test Error: 27.07. Result ID: 699186293a6c31484908b6a2. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Fisher Test Error: 35.87. Result ID: 699186293a6c31484908b6a7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"ResNet","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. ### Fisher Test Error: 49.6. Result ID: 699186293a6c31484908b655. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Number of samples per confused class":"50-50"}. This record does not assert independent reproduction. ### Fisher Test Error: 49.6. Result ID: 699186293a6c31484908b6b0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"All-CNN","Confused classes":"0, 1","Samples per class":50}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.