Code-level Structured Extraction on PVminer
87.9PrecisionLlama-3.3-70B-Instruct
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Llama-3.3-70B-InstructLearning Setup=SFT2026.03 | 87.9 | 80.11 | 83.82 | — | — | |
| PVminerLLMModel=Llama-3.3-70B2026.06 | 87.9 | 80.11 | 83.82 | 0.62 | — | |
| PVminerLLM2Model=Llama-3.3-70B2026.06 | 87.26 | 81.74 | 84.41 | 0.17 | 0.59 | |
| PVminerLLM2Model=Qwen2.5-1.5B2026.06 | 85.63 | 77.58 | 81.4 | 0.26 | 4.43 | |
| Llama-3.1-8B-InstructLearning Setup=SFT2026.03 | 85.04 | 78.12 | 81.43 | — | — | |
| PVminerLLMModel=Llama-3.1-8B2026.06 | 85.04 | 78.12 | 81.43 | 0.6 | — | |
| PVminerLLM2Model=Llama-3.1-8B2026.06 | 84.46 | 81.56 | 82.98 | 0.36 | 1.55 | |
| Llama-3.3-70B-InstructSetup=SFT2026.03 | 83.74 | 77.95 | 80.74 | — | — | |
| Qwen2.5-1.5B-InstructLearning Setup=SFT2026.03 | 83.19 | 71.61 | 76.97 | — | — | |
| PVminerLLMModel=Qwen2.5-1.5B2026.06 | 83.19 | 71.61 | 76.97 | 2.71 | — | |
| PVminerLLM2Model=Llama-3.2-3B2026.06 | 82.7 | 80.66 | 81.67 | 0.19 | 1.34 | |
| Llama-3.2-3B-InstructLearning Setup=SFT2026.03 | 82.48 | 78.3 | 80.33 | — | — | |
| PVminerLLMModel=Llama-3.2-3B2026.06 | 82.48 | 78.3 | 80.33 | 0.78 | — | |
| Llama-3.1-8B-InstructSetup=SFT2026.03 | 79.19 | 76.33 | 77.73 | — | — | |
| Qwen2.5-1.5B-InstructSetup=SFT2026.03 | 77.86 | 66.88 | 71.96 | — | — | |
| Llama-3.2-3B-InstructSetup=SFT2026.03 | 75.8 | 73.74 | 74.75 | — | — | |
| Llama-3.3-70B-InstructLearning Setup=Two-shot2026.03 | 70.88 | 59.86 | 64.9 | — | — | |
| Llama-3.3-70B-InstructLearning Setup=One-shot2026.03 | 70.06 | 60.94 | 65.18 | — | — | |
| Llama-3.3-70B-InstructLearning Setup=Zero-shot2026.03 | 69.98 | 56.06 | 62.25 | — | — | |
| Llama-3.1-8B-InstructLearning Setup=Two-shot2026.03 | 61.71 | 49.55 | 54.96 | — | — | |
| Llama-3.1-8B-InstructLearning Setup=One-shot2026.03 | 60.83 | 47.74 | 53.5 | — | — | |
| Llama-3.2-3B-InstructLearning Setup=Two-shot2026.03 | 54.91 | 47.56 | 50.97 | — | — | |
| Llama-3.2-3B-InstructLearning Setup=One-shot2026.03 | 53.83 | 47.02 | 50.19 | — | — | |
| Llama-3.3-70B-InstructPrompt Type=Engineered Prompt, Shot Setting=Two-shot2026.03 | 53.69 | 48.9 | 51.18 | — | — | |
| Qwen2.5-1.5B-InstructLearning Setup=Two-shot2026.03 | 53.33 | 26.04 | 34.99 | — | — | |
| Llama-3.3-70B-InstructPrompt Type=Engineered Prompt, Shot Setting=One-shot2026.03 | 51.86 | 46.83 | 49.22 | — | — | |
| Llama-3.3-70B-InstructPrompt Type=Engineered Prompt, Shot Setting=Zero-shot2026.03 | 50.68 | 38.42 | 43.71 | — | — | |
| Llama-3.1-8B-InstructPrompt Type=Engineered Prompt, Shot Setting=One-shot2026.03 | 48.47 | 28.72 | 36.07 | — | — | |
| Llama-3.1-8B-InstructLearning Setup=Zero-shot2026.03 | 47.35 | 46.84 | 47.09 | — | — | |
| Llama-3.1-8B-InstructPrompt Type=Engineered Prompt, Shot Setting=Two-shot2026.03 | 47.22 | 31.82 | 38.02 | — | — | |
| Qwen2.5-1.5B-InstructLearning Setup=One-shot2026.03 | 46.36 | 21.88 | 29.73 | — | — | |
| Llama-3.2-3B-InstructLearning Setup=Zero-shot2026.03 | 42.83 | 34.54 | 38.24 | — | — | |
| Qwen2.5-1.5B-InstructPrompt Type=Engineered Prompt, Shot Setting=Two-shot2026.03 | 38.91 | 11.13 | 17.3 | — | — | |
| Qwen2.5-1.5B-InstructPrompt Type=Engineered Prompt, Shot Setting=One-shot2026.03 | 34.31 | 9.06 | 14.33 | — | — | |
| Qwen2.5-1.5B-InstructLearning Setup=Zero-shot2026.03 | 33.94 | 13.38 | 19.2 | — | — | |
| Llama-3.2-3B-InstructPrompt Type=Engineered Prompt, Shot Setting=Two-shot2026.03 | 32.19 | 24.19 | 27.62 | — | — | |
| Llama-3.1-8B-InstructPrompt Type=Engineered Prompt, Shot Setting=Zero-shot2026.03 | 31.19 | 15.65 | 20.84 | — | — | |
| Llama-3.2-3B-InstructPrompt Type=Engineered Prompt, Shot Setting=One-shot2026.03 | 28.57 | 20.7 | 24.01 | — | — | |
| Qwen2.5-1.5B-InstructPrompt Type=Engineered Prompt, Shot Setting=Zero-shot2026.03 | 17.39 | 1.03 | 1.95 | — | — | |
| Llama-3.2-3B-InstructPrompt Type=Engineered Prompt, Shot Setting=Zero-shot2026.03 | 15.77 | 11.38 | 13.22 | — | — |