Medical Order Extraction on SIMORD (test)
69.2Match Counto3-mini
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| o3-miniWeight Access=closed, Prompting Setting=with one example2025.07 | 69.2 | 39.3 | 12.4 | 66.8 | 26.9 | |
| 4oWeight Access=closed, Prompting Setting=zero shot2025.07 | 68.2 | 38.5 | 17.4 | 66.1 | 1.1 | |
| 4oWeight Access=closed, Prompting Setting=with one example2025.07 | 67.9 | 42.8 | 21.8 | 66.1 | 0.5 | |
| 4.1Weight Access=closed, Prompting Setting=with one example2025.07 | 67.6 | 33 | 12.7 | 66 | 1.2 | |
| 4.1Weight Access=closed, Prompting Setting=zero shot2025.07 | 67 | 30.7 | 12.1 | 65.2 | 1.1 | |
| o1-miniWeight Access=closed, Prompting Setting=zero shot2025.07 | 66.5 | 30.2 | 23.3 | 64.1 | 43.2 | |
| o3-miniWeight Access=closed, Prompting Setting=zero shot2025.07 | 66.5 | 30.8 | 9.7 | 64.9 | 24.1 | |
| MP-InstWeight Access=open, Prompting Setting=with two examples2025.07 | 66.3 | 51.9 | 35.7 | 65.4 | 8.7 | |
| o1-miniWeight Access=closed, Prompting Setting=with one example2025.07 | 66.1 | 35.8 | 26.6 | 64.2 | 37.8 | |
| Med42Weight Access=open, Prompting Setting=with one example2025.07 | 65.2 | 48.3 | 35.8 | 64.4 | 9.7 | |
| o1-prevWeight Access=closed, Prompting Setting=with one example2025.07 | 65.2 | 40.2 | 24.6 | 63.5 | 14.8 | |
| Phi3.5Weight Access=open, Prompting Setting=with one example2025.07 | 64.5 | 41.4 | 20.7 | 60.6 | 8.6 | |
| Med42Weight Access=open, Prompting Setting=with two examples2025.07 | 63.2 | 49.2 | 37.9 | 62.4 | 9.6 | |
| Llama3Weight Access=open, Prompting Setting=with two examples2025.07 | 62.8 | 47.6 | 30.8 | 61.5 | 4.5 | |
| Phi3.5Weight Access=open, Prompting Setting=with two examples2025.07 | 62.2 | 43.5 | 25.3 | 60.4 | 6.3 | |
| Llama3Weight Access=open, Prompting Setting=zero shot2025.07 | 61.8 | 41.8 | 21.5 | 59.3 | 9.5 | |
| MP-InstWeight Access=open, Prompting Setting=with one example2025.07 | 61 | 45.7 | 32.6 | 59.3 | 9.8 | |
| Llama3Weight Access=open, Prompting Setting=with one example2025.07 | 56.9 | 40.8 | 29.2 | 55.3 | 8.2 | |
| Phi3.5Weight Access=open, Prompting Setting=zero shot2025.07 | 56 | 26.3 | 11.5 | 51.9 | 18.3 | |
| Med42Weight Access=open, Prompting Setting=zero shot2025.07 | 55.1 | 36.5 | 25.4 | 54 | 14.3 | |
| o1-prevWeight Access=closed, Prompting Setting=zero shot2025.07 | 55 | 28.1 | 17 | 53.8 | 13.4 | |
| MP-InstWeight Access=open, Prompting Setting=zero shot2025.07 | 42.2 | 28.4 | 17.6 | 39.6 | 11.2 |