List your company
ResearchFormerly WizWand

Lemma Discovery on AFP (test)

42.7Success RateCombined
4.7414.59524.4534.305Apr 7, 2025
Updated 5mo ago

Evaluation results

MethodLinks
CombinedBackbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0442.7
CombinedBackbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0438.7
LEMMANAID CombinedBackbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0435.9
LEMMANAID CombinedBackbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0434.8
CombinedBackbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0432.9
CombinedBackbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0432.1
Neural CombinedBackbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0431.6
LEMMANAID CombinedBackbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0430.4
LEMMANAID CombinedBackbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0429.4
LEMMANAID (types)Backbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0428.6
CombinedBackbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0428.5
LEMMANAID (types + defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0428.2
LEMMANAID (types + defs)Backbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0427.9
LEMMANAID (types)Backbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0427.4
LEMMANAID (types)Backbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0427.2
LEMMANAID CombinedBackbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0426.4
LEMMANAID (types + defs)Backbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0426.3
Neural (types)Backbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0425.2
Neural (types + defs)Backbone=DeepSeek-coder-6.7B, Training Set=HOL+AFP-train2025.0425.1
Neural CombinedBackbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0423.7
Neural CombinedBackbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0423.5
CombinedBackbone=Llama-3.2-1B, Training Set=HOL-train2025.0422.6
LEMMANAID (types)Backbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0422.5
LEMMANAID CombinedBackbone=Llama-3.2-1B, Training Set=HOL-train2025.0421.1
LEMMANAID (types + defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0420.3
LEMMANAID (types)Backbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0419.7
Neural (types + defs)Backbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0419.7
LEMMANAID (types + defs)Backbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0419.3
Neural (types)Backbone=DeepSeek-coder-6.7B, Training Set=HOL-train2025.0419
Neural (types + defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0418.6
Neural CombinedBackbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0417.8
LEMMANAID (types + defs)Backbone=Llama-3.2-1B, Training Set=HOL-train2025.0417.7
Neural CombinedBackbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0417.7
Neural (types)Backbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0415
LEMMANAID (types)Backbone=Llama-3.2-1B, Training Set=HOL-train2025.0415
LEMMANAID (defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.0413
Neural (types)Backbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0412.9
Neural (types + defs)Backbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0412.9
Neural (types)Backbone=Llama-3.2-1B, Training Set=HOL+AFP-train2025.0412.2
Neural (types + defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.0411.9
Neural CombinedBackbone=Llama-3.2-1B, Training Set=HOL-train2025.0411.1
Neural (types)Backbone=Llama-3.2-1B, Training Set=HOL-train2025.048.8
Neural (defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL+AFP-train2025.047.8
LEMMANAID (defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.047.3
Neural (defs)Backbone=DeepSeek-coder-1.3B, Training Set=HOL-train2025.046.2
Neural (types + defs)Backbone=Llama-3.2-1B, Training Set=HOL-train2025.046.2