Runtime Safety Monitoring on Waymo Open Motion Dataset (WOMD)
8.11q_phisemantic-basis monitor
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| semantic-basis monitorCalibration Level=Level-1, Specification=⊡[0,4] pf ∧ ⊡[0,4] pl ∧ ⊡[0,4] pr, GT%=1002026.05 | 8.11 | 32.6 | — | — | |
| Observer BaselineCalibration Level=Level-2, Specification=⊡[0,4] ps, GT%=962026.05 | 7.74 | 30.5 | 99.8 | 1.9 | |
| semantic-basis monitorCalibration Level=Level-1, Specification=⊡[0,4] ps, GT%=962026.05 | 6.02 | 52 | — | — | |
| semantic-basis monitorCalibration Level=Level-1, Specification=⊡[0,1] pf, GT%=1002026.05 | 5.96 | 70.7 | — | — | |
| rolling prediction monitorCalibration Level=Level-1, Specification=⊡[0,4] pf ∧ ⊡[0,4] pl ∧ ⊡[0,4] pr, GT%=1002026.05 | 5.82 | 43.4 | — | — | |
| Observer BaselineCalibration Level=Level-2, Specification=⊡[0,4] pf ∧ ⊡[0,4] pl ∧ ⊡[0,4] pr, GT%=1002026.05 | 5.8 | 42.8 | 100 | — | |
| rolling prediction monitorCalibration Level=Level-1, Specification=⊡[0,4] ps, GT%=962026.05 | 5.29 | 46.1 | — | — | |
| semantic-basis monitorCalibration Level=Level-1, Specification=◇[0,4] pf, GT%=1002026.05 | 4.99 | 81.7 | — | — | |
| rolling prediction monitorCalibration Level=Level-1, Specification=⊡[0,1] pf, GT%=1002026.05 | 4.27 | 78.8 | — | — | |
| rolling prediction monitorCalibration Level=Level-1, Specification=◇[0,4] pf, GT%=1002026.05 | 4.27 | 84.7 | — | — | |
| rolling prediction monitorCalibration Level=Level-2, Specification=⊡[0,4] pf ∧ ⊡[0,4] pl ∧ ⊡[0,4] pr, GT%=1002026.05 | 2.29 | 68.4 | 100 | — | |
| rolling prediction monitorCalibration Level=Level-2, Specification=⊡[0,4] ps, GT%=962026.05 | 2.25 | 71.9 | 99.4 | 10.4 | |
| semantic-basis monitorCalibration Level=Level-2, Specification=⊡[0,4] ps, GT%=962026.05 | 2.11 | 85 | 98.5 | 31.9 | |
| Observer BaselineCalibration Level=Level-2, Specification=⊡[0,1] pf, GT%=1002026.05 | 2.01 | 90.2 | 100 | — | |
| rolling prediction monitorCalibration Level=Level-2, Specification=◇[0,4] pf, GT%=1002026.05 | 1.64 | 97.2 | 100 | — | |
| semantic-basis monitorCalibration Level=Level-2, Specification=⊡[0,4] pf ∧ ⊡[0,4] pl ∧ ⊡[0,4] pr, GT%=1002026.05 | 1.59 | 76.8 | 100 | — | |
| rolling prediction monitorCalibration Level=Level-2, Specification=⊡[0,1] pf, GT%=1002026.05 | 0.91 | 98.1 | 100 | — | |
| semantic-basis monitorCalibration Level=Level-2, Specification=⊡[0,1] pf, GT%=1002026.05 | 0.81 | 98.1 | 100 | — | |
| semantic-basis monitorCalibration Level=Level-2, Specification=◇[0,4] pf, GT%=1002026.05 | 0.81 | 99.2 | 100 | — |