Evidence Retrieval on Synthetic Benchmark 1.0 (test)
22.5Hit@2OSU-Mem + GF
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OSU-Mem + GFBudget (B)=5122026.06 | 22.5 | 36.7 | |
| OSU-MemBudget (B)=5122026.06 | 22.2 | 36.4 | |
| OSU-Mem (no novelty)Budget (B)=5122026.06 | 20.9 | 36.1 | |
| Flat + MMRBudget (B)=5122026.06 | 15.2 | 28.8 | |
| Flat VectorBudget (B)=5122026.06 | 14.7 | 28.4 | |
| Coverage-AwareBudget (B)=5122026.06 | 14.7 | 28.9 | |
| Disjoint HierarchyBudget (B)=5122026.06 | 14.5 | 28.2 | |
| OSU-Mem (disjoint)Budget (B)=5122026.06 | 13.4 | 27.7 | |
| Tree AbstractionBudget (B)=5122026.06 | 12.3 | 27.5 | |
| OSU-Mem + GFBudget (B)=2562026.06 | 8.8 | 22.2 | |
| OSU-MemBudget (B)=2562026.06 | 8.4 | 22.1 | |
| OSU-Mem (sim only)Budget (B)=5122026.06 | 8.3 | 21.4 | |
| OSU-Mem (no novelty)Budget (B)=2562026.06 | 8.3 | 21.7 | |
| Periodic SummaryBudget (B)=5122026.06 | 7.3 | 22.1 | |
| Flat + MMRBudget (B)=2562026.06 | 5.2 | 15.8 | |
| Coverage-AwareBudget (B)=2562026.06 | 4.9 | 15.7 | |
| Flat VectorBudget (B)=2562026.06 | 4.5 | 15.6 | |
| Disjoint HierarchyBudget (B)=2562026.06 | 4.4 | 15 | |
| Tree AbstractionBudget (B)=2562026.06 | 4.4 | 15.7 | |
| OSU-Mem (disjoint)Budget (B)=2562026.06 | 3.4 | 13.8 | |
| OSU-Mem + GFBudget (B)=1282026.06 | 2.9 | 13 | |
| OSU-MemBudget (B)=1282026.06 | 2.7 | 13 | |
| OSU-Mem (no novelty)Budget (B)=1282026.06 | 2.7 | 13 | |
| Flat + MMRBudget (B)=1282026.06 | 2.2 | 8.5 | |
| Flat VectorBudget (B)=1282026.06 | 1.6 | 8.8 | |
| Coverage-AwareBudget (B)=1282026.06 | 1.4 | 8.6 | |
| Disjoint HierarchyBudget (B)=1282026.06 | 1.4 | 8.4 | |
| OSU-Mem (sim only)Budget (B)=2562026.06 | 1.4 | 11 | |
| Periodic SummaryBudget (B)=2562026.06 | 1.3 | 9.3 | |
| Tree AbstractionBudget (B)=1282026.06 | 1.2 | 9.3 | |
| OSU-MemBudget (B)=642026.06 | 1 | 7.6 | |
| OSU-Mem + GFBudget (B)=642026.06 | 1 | 7.7 | |
| OSU-Mem (no novelty)Budget (B)=642026.06 | 1 | 7.6 | |
| Budget-Trunc ReplayBudget (B)=5122026.06 | 0.8 | 32.1 | |
| OSU-Mem (disjoint)Budget (B)=1282026.06 | 0.8 | 7.5 | |
| OSU-Mem (sim only)Budget (B)=1282026.06 | 0.8 | 6.5 | |
| Coverage-AwareBudget (B)=642026.06 | 0.6 | 4.4 | |
| Sliding WindowBudget (B)=5122026.06 | 0.5 | 31.4 | |
| Flat VectorBudget (B)=642026.06 | 0.5 | 5.1 | |
| Flat + MMRBudget (B)=642026.06 | 0.5 | 4.5 | |
| Disjoint HierarchyBudget (B)=642026.06 | 0.4 | 4.7 | |
| OSU-Mem (disjoint)Budget (B)=642026.06 | 0.3 | 4.7 | |
| Tree AbstractionBudget (B)=642026.06 | 0.2 | 5.5 | |
| OSU-Mem (sim only)Budget (B)=642026.06 | 0.2 | 3.7 | |
| No LT MemoryBudget (B)=642026.06 | 0 | 0 | |
| No LT MemoryBudget (B)=1282026.06 | 0 | 0 | |
| No LT MemoryBudget (B)=2562026.06 | 0 | 0 | |
| No LT MemoryBudget (B)=5122026.06 | 0 | 0 | |
| Budget-Trunc ReplayBudget (B)=642026.06 | 0 | 0.4 | |
| Budget-Trunc ReplayBudget (B)=1282026.06 | 0 | 0.4 | |
| Budget-Trunc ReplayBudget (B)=2562026.06 | 0 | 14.9 | |
| Sliding WindowBudget (B)=642026.06 | 0 | 0 | |
| Sliding WindowBudget (B)=1282026.06 | 0 | 0 | |
| Sliding WindowBudget (B)=2562026.06 | 0 | 13.6 | |
| Periodic SummaryBudget (B)=642026.06 | 0 | 0.3 | |
| Periodic SummaryBudget (B)=1282026.06 | 0 | 4.1 |