Lemma Discovery on Octonions Isabelle/AFP 2026 context (test)
0.766Lemma Discovery RateLEMMANAID DeepSeek-coder-6.7B
Evaluation Results
| Method | Links | |
|---|---|---|
| LEMMANAID DeepSeek-coder-6.7BStrategy=Template, Backbone=DeepSeek-coder-6.7B, Evaluation=Fine-tuned, Prompting=types+defs, Predictions=1 greedy, 4 beam-search2025.04 | 0.766 | |
| LEMMANAID DeepSeek-coder-1.3BStrategy=Template, Backbone=DeepSeek-coder-1.3B, Evaluation=Fine-tuned, Prompting=types+defs, Predictions=1 greedy, 4 beam-search2025.04 | 0.669 | |
| Neural DeepSeek-coder-6.7BStrategy=Lemma-object, Backbone=DeepSeek-coder-6.7B, Evaluation=Fine-tuned, Prompting=types+defs, Predictions=1 greedy, 4 beam-search2025.04 | 0.617 | |
| Neural Llama-3.2-1BStrategy=Lemma-object, Backbone=Llama-3.2-1B, Evaluation=Fine-tuned, Prompting=types+defs, Predictions=1 greedy, 4 beam-search2025.04 | 0.546 | |
| GPT-5.2Strategy=Template, Evaluation=Few-shot, Prompting=5 few-shot examples, Predictions=1 greedy, 4 temperature-1 samples2025.04 | 0.5 | |
| Claude Opus 4.5Strategy=Template, Evaluation=Few-shot, Prompting=5 few-shot examples, Predictions=1 greedy, 4 temperature-1 samples2025.04 | 0.471 | |
| LEMMANAID Llama-3.2-1BStrategy=Template, Backbone=Llama-3.2-1B, Evaluation=Fine-tuned, Prompting=types+defs, Predictions=1 greedy, 4 beam-search2025.04 | 0.469 | |
| Claude Opus 4.5Strategy=Lemma-object, Evaluation=Few-shot, Prompting=5 few-shot examples, Predictions=1 greedy, 4 temperature-1 samples2025.04 | 0.463 | |
| GPT-5.2Strategy=Lemma-object, Evaluation=Few-shot, Prompting=5 few-shot examples, Predictions=1 greedy, 4 temperature-1 samples2025.04 | 0.454 | |
| Neural DeepSeek-coder-1.3BStrategy=Lemma-object, Backbone=DeepSeek-coder-1.3B, Evaluation=Fine-tuned, Prompting=types+defs, Predictions=1 greedy, 4 beam-search2025.04 | 0.44 |