Analogy generation on SCAN (out-of-domain)
15.3AccuracyInstructGPT_003 + ANALOGYKB
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| InstructGPT_003 + ANALOGYKBBackbone=InstructGPT-003, Retrieval Source=ANALOGYKB, Context Examples=82023.05 | 15.3 | 32.8 | 38.46 | |
| InstructGPT_003 + E-KARBackbone=InstructGPT-003, Retrieval Source=E-KAR, Context Examples=82023.05 | 9.12 | 25 | 36.27 | |
| InstructGPT_003 + ANALOGYKB_sameBackbone=InstructGPT-003, Retrieval Source=ANALOGYKB (domain-matched), Context Examples=82023.05 | 6.91 | 25.32 | 33.42 | |
| BERT + ANALOGYKBBackbone=BERT, Training Data=ANALOGYKB2023.05 | 6.24 | 10.36 | 14.07 | |
| GPT-2 + ANALOGYKBBackbone=GPT-2, Training Data=ANALOGYKB2023.05 | 5.12 | 6.41 | 12.77 | |
| InstructGPT_003 + BATSBackbone=InstructGPT-003, Retrieval Source=BATS, Context Examples=82023.05 | 5.07 | 21.4 | 32.37 | |
| BERT + ANALOGYKB_sameBackbone=BERT, Training Data=ANALOGYKB (domain-matched)2023.05 | 4.01 | 7.44 | 10.89 | |
| GPT-2 + ANALOGYKB_sameBackbone=GPT-2, Training Data=ANALOGYKB (domain-matched)2023.05 | 4 | 5.49 | 11.45 | |
| InstructGPT_003Backbone=InstructGPT-003, Protocol=Zero-shot2023.05 | 3.32 | 15.75 | 34.58 | |
| GPT-2 + E-KARBackbone=GPT-2, Training Data=E-KAR2023.05 | 2.39 | 5.44 | 10.46 | |
| GPT-2Backbone=GPT-2, Protocol=Base (Zero-shot)2023.05 | 2 | 5.7 | 10.7 | |
| BERT + E-KARBackbone=BERT, Training Data=E-KAR2023.05 | 1.5 | 4.32 | 7.92 | |
| GPT-2 + BATSBackbone=GPT-2, Training Data=BATS2023.05 | 1.1 | 2.2 | 3.6 | |
| BERTBackbone=BERT, Protocol=Base (Zero-shot)2023.05 | 0.9 | 4.4 | 8 | |
| BERT + BATSBackbone=BERT, Training Data=BATS2023.05 | 0.4 | 1.9 | 3.1 |