Patient Message Response Drafting on SyPPM
26Content PrecisionInter-annotator predictability (IAP)
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Inter-annotator predictability (IAP)description=Content-level inter-annotator predictability2026.01 | 26 | 25 | 24 | 61 | 63 | 62 | |
| IAP (Inter-Annotator Precision)Context=Human-Human Alignment2026.01 | 26 | 25 | 24 | 61 | 63 | 62 | |
| SFTadaptation=Supervised Fine-Tuning, aggregation=Average across 3 LLMs2026.01 | 22 | 17 | 18 | 64 | 41 | 50 | |
| TADPOLEadaptation=TADPOLE, aggregation=Average across 3 LLMs2026.01 | 21 | 20 | 20 | 62 | 54 | 58 | |
| Gemini 2.5 Proadaptation=theme prompting2026.01 | 20 | 43 | 26 | 58 | 69 | 64 | |
| Claude 4.5 SonnetPrompt=0-Shot, Model=Claude2026.01 | 20 | 38 | 25 | 52 | 54 | 53 | |
| Gemini 2.5 ProPrompt=Theme, Model=Gemini2026.01 | 20 | 43 | 26 | 56 | 69 | 64 | |
| RAGadaptation=Retrieval-Augmented Generation, aggregation=Average across 3 LLMs2026.01 | 17 | 28 | 18 | 47 | 43 | 45 | |
| Gemini 2.5 ProPrompt=0-Shot, Model=Gemini2026.01 | 17 | 40 | 23 | 52 | 56 | 54 | |
| Claude 4.5 SonnetPrompt=Theme, Model=Claude2026.01 | 16 | 37 | 22 | 58 | 69 | 63 | |
| 0-Shotadaptation=None, aggregation=Average across 3 LLMs2026.01 | 12 | 31 | 16 | 47 | 46 | 47 | |
| Themeadaptation=Thematic Prompting, aggregation=Average across 3 LLMs2026.01 | 11 | 33 | 15 | 50 | 58 | 54 | |
| GPT-OSSPrompt=Theme, Model=GPT2026.01 | 6 | 30 | 9 | 49 | 77 | 60 | |
| GPT-OSSPrompt=0-Shot, Model=GPT2026.01 | 3 | 21 | 5 | 45 | 64 | 53 |