Step-goal linking on wikiHow (test)
55.4R@1DEBERTA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| DEBERTAStage=Reranking, Base Model=DEBERTA-large, Candidate Source=SP Top-302022.03 | 55.4 | 71.9 | — | |
| DEBERTA - goalStage=Reranking, Context=Query step goal, Candidate Source=SP Top-302022.03 | 55 | 71.5 | — | |
| DEBERTA - surrStage=Reranking, Context=Surrounding steps, Candidate Source=SP Top-302022.03 | 54.3 | 71.6 | — | |
| DEBERTA - bothStage=Reranking, Context=Both surrounding steps and goal, Candidate Source=SP Top-302022.03 | 52.4 | 71 | — | |
| DEBERTA + λ = 0Stage=Reranking, Parameter=λ = 0, Candidate Source=SP Top-302022.03 | 51.9 | 71.4 | — | |
| BERTStage=Reranking, Base Model=BERT, Candidate Source=SP Top-302022.03 | 50.7 | 69.4 | — | |
| DEBERTA-UL (unlinkable)Stage=Reranking, Classifier=Unlinkable classification, Candidate Source=SP Top-302022.03 | 50.4 | 71.6 | — | |
| SPStage=Paraphrase retrieval only2022.03 | 35.8 | 64.4 | 72.5 | |
| SBERTStage=Paraphrase retrieval only2022.03 | 30.6 | 53.3 | 63.4 | |
| BM25 (goal only)Stage=Paraphrase retrieval only, Content=Goal only2022.03 | 30.5 | 51.6 | 61.1 | |
| Bing SearchStage=Paraphrase retrieval only2022.03 | 28 | 47.9 | — | |
| BM25 (article)Stage=Paraphrase retrieval only, Content=Full article2022.03 | 9.3 | 35.3 | 49.2 |