Student Simulation on C++_5
86AccCognitive Prototype Mapping
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Cognitive Prototype MappingBackbone=GPT-4o, Solution Simulation Protocol=IO2025.05 | 86 | 3.6 | 3.32 | |
| Cognitive Prototype MappingBackbone=GPT-4o, Solution Simulation Protocol=CoT2025.05 | 86 | 3.6 | 3.4 | |
| Cognitive Prototype MappingBackbone=GPT-4o, Solution Simulation Protocol=Refine2025.05 | 86 | 3.6 | 3.42 | |
| Level+Similarity RetrievalBackbone=GPT-4o, Solution Simulation Protocol=IO2025.05 | 68 | 3.2 | 2.6 | |
| Level+Similarity RetrievalBackbone=GPT-4o, Solution Simulation Protocol=CoT2025.05 | 68 | 3.2 | 2.8 | |
| Level+Similarity RetrievalBackbone=GPT-4o, Solution Simulation Protocol=Refine2025.05 | 68 | 3.2 | 2.8 | |
| Cognitive Prototype MappingBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=IO2025.05 | 66 | 3.16 | 2.98 | |
| Cognitive Prototype MappingBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=CoT2025.05 | 66 | 3.16 | 3.1 | |
| Cognitive Prototype MappingBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=Refine2025.05 | 66 | 3.16 | 3.16 | |
| Cognitive Prototype MappingBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=IO2025.05 | 62 | 3 | 2.44 | |
| Cognitive Prototype MappingBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=CoT2025.05 | 62 | 3 | 2.52 | |
| Cognitive Prototype MappingBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=Refine2025.05 | 62 | 3 | 2.52 | |
| Similarity-based RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=IO2025.05 | 62 | 3 | 2.52 | |
| Similarity-based RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=CoT2025.05 | 62 | 3 | 2.36 | |
| Similarity-based RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=Refine2025.05 | 62 | 3 | 2.62 | |
| Level+Similarity RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=IO2025.05 | 58 | 2.52 | 2.46 | |
| Level+Similarity RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=CoT2025.05 | 58 | 2.52 | 2.18 | |
| Level+Similarity RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=Refine2025.05 | 58 | 2.52 | 1.88 | |
| Cognitive Prototype MappingBackbone=GPT-3.5, Solution Simulation Protocol=IO2025.05 | 58 | 2.7 | 2.98 | |
| Cognitive Prototype MappingBackbone=GPT-3.5, Solution Simulation Protocol=CoT2025.05 | 58 | 2.7 | 3.12 | |
| Cognitive Prototype MappingBackbone=GPT-3.5, Solution Simulation Protocol=Refine2025.05 | 58 | 2.7 | 3.18 | |
| Similarity-based RetrievalBackbone=GPT-4o, Solution Simulation Protocol=IO2025.05 | 54 | 2.68 | 3.14 | |
| Similarity-based RetrievalBackbone=GPT-4o, Solution Simulation Protocol=CoT2025.05 | 54 | 2.68 | 3.3 | |
| Similarity-based RetrievalBackbone=GPT-4o, Solution Simulation Protocol=Refine2025.05 | 54 | 2.68 | 2.9 | |
| Level+Random RetrievalBackbone=GPT-4o, Solution Simulation Protocol=IO2025.05 | 54 | 2.84 | 2.32 | |
| Level+Random RetrievalBackbone=GPT-4o, Solution Simulation Protocol=CoT2025.05 | 54 | 2.84 | 2.7 | |
| Level+Random RetrievalBackbone=GPT-4o, Solution Simulation Protocol=Refine2025.05 | 54 | 2.84 | 2.24 | |
| Random StrategyBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=IO2025.05 | 52 | 2.64 | 2.62 | |
| Random StrategyBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=CoT2025.05 | 52 | 2.64 | 2.26 | |
| Random StrategyBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=Refine2025.05 | 52 | 2.64 | 2.26 | |
| Similarity-based RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=IO2025.05 | 52 | 2.62 | 2.92 | |
| Similarity-based RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=CoT2025.05 | 52 | 2.62 | 2.66 | |
| Similarity-based RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=Refine2025.05 | 52 | 2.62 | 2.52 | |
| Random StrategyBackbone=GPT-3.5, Solution Simulation Protocol=IO2025.05 | 50 | 2.62 | 2.98 | |
| Random StrategyBackbone=GPT-3.5, Solution Simulation Protocol=CoT2025.05 | 50 | 2.62 | 2.78 | |
| Random StrategyBackbone=GPT-3.5, Solution Simulation Protocol=Refine2025.05 | 50 | 2.62 | 2.82 | |
| Random StrategyBackbone=GPT-4o, Solution Simulation Protocol=IO2025.05 | 50 | 2.5 | 2.64 | |
| Random StrategyBackbone=GPT-4o, Solution Simulation Protocol=CoT2025.05 | 50 | 2.5 | 3.04 | |
| Random StrategyBackbone=GPT-4o, Solution Simulation Protocol=Refine2025.05 | 50 | 2.5 | 3.1 | |
| Level-based RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=IO2025.05 | 48 | 2.36 | 2.98 | |
| Level-based RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=CoT2025.05 | 48 | 2.36 | 3 | |
| Level-based RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=Refine2025.05 | 48 | 2.36 | 2.8 | |
| Level+Random RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=IO2025.05 | 46 | 2.22 | 1.86 | |
| Level+Random RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=CoT2025.05 | 46 | 2.22 | 1.88 | |
| Level+Random RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=Refine2025.05 | 46 | 2.22 | 2.1 | |
| Level+Random RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=IO2025.05 | 44 | 2.24 | 3.16 | |
| Level+Random RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=CoT2025.05 | 44 | 2.24 | 3.06 | |
| Level+Random RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=Refine2025.05 | 44 | 2.24 | 3.16 | |
| Level+Similarity RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=IO2025.05 | 44 | 2.3 | 2.86 | |
| Level+Similarity RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=CoT2025.05 | 44 | 2.3 | 2.8 | |
| Level+Similarity RetrievalBackbone=GPT-3.5, Solution Simulation Protocol=Refine2025.05 | 44 | 2.3 | 3.16 | |
| Level-based RetrievalBackbone=GPT-4o, Solution Simulation Protocol=IO2025.05 | 42 | 2.08 | 2.42 | |
| Level-based RetrievalBackbone=GPT-4o, Solution Simulation Protocol=CoT2025.05 | 42 | 2.08 | 3.08 | |
| Level-based RetrievalBackbone=GPT-4o, Solution Simulation Protocol=Refine2025.05 | 42 | 2.08 | 2.04 | |
| Level-based RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=IO2025.05 | 38 | 1.98 | 2.32 | |
| Level-based RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=CoT2025.05 | 38 | 1.98 | 2.52 | |
| Level-based RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=Refine2025.05 | 38 | 1.98 | 1.86 | |
| Level+Random RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=IO2025.05 | 38 | 2.2 | 2.1 | |
| Level+Random RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=CoT2025.05 | 38 | 2.2 | 2.28 | |
| Level+Random RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=Refine2025.05 | 38 | 2.2 | 1.8 | |
| Level+Similarity RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=IO2025.05 | 38 | 2.12 | 2.44 | |
| Level+Similarity RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=CoT2025.05 | 38 | 2.12 | 1.9 | |
| Level+Similarity RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=Refine2025.05 | 38 | 2.12 | 2.08 | |
| Level-based RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=IO2025.05 | 36 | 1.76 | 1.94 | |
| Level-based RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=CoT2025.05 | 36 | 1.76 | 1.76 | |
| Level-based RetrievalBackbone=Claude-3.5-Sonnet, Solution Simulation Protocol=Refine2025.05 | 36 | 1.76 | 1.48 | |
| Random StrategyBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=IO2025.05 | 34 | 2.3 | 1.98 | |
| Random StrategyBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=CoT2025.05 | 34 | 2.3 | 2.34 | |
| Random StrategyBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=Refine2025.05 | 34 | 2.3 | 1.86 | |
| Similarity-based RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=IO2025.05 | 34 | 2.16 | 2.22 | |
| Similarity-based RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=CoT2025.05 | 34 | 2.16 | 2.28 | |
| Similarity-based RetrievalBackbone=LLaMA-3.3-70B-Instruct, Solution Simulation Protocol=Refine2025.05 | 34 | 2.16 | 2.2 |