# Failure Detection on Franka unseen Source: [Failure Detection on Franka unseen]() Collection index: [llms.txt]() Data updated: 2026-04-23T10:49:52.203Z Evaluation of Failure Detection performance on the Franka unseen dataset, measured using the Brier Score. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Failure Detection | | Dataset (including any stated split) | Franka unseen | | Metric | Brier Score | | Unit | SCALAR | | Direction | Lower is better | | Results loaded | 11 of 11 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Brier Score](): Lower is better; unit: SCALAR. ## Recorded results | Method | Brier Score | Result date | Report paper | | --- | --- | --- | --- | | SAFE-RNN-TDQC (Ours) | 0.215 | Not recorded | 69e9a78555a3c32af89f6fea | | RNN-TDQC (Ours) | 0.228 | Not recorded | 69e9a78555a3c32af89f6fea | | SAFE-MLP-TDQC (Ours) | 0.229 | Not recorded | 69e9a78555a3c32af89f6fea | | RNN-BCE | 0.243 | Not recorded | 69e9a78555a3c32af89f6fea | | SAFE-MLP BCE | 0.248 | Not recorded | 69e9a78555a3c32af89f6fea | | Avg entropy | 0.281 | Not recorded | 69e9a78555a3c32af89f6fea | | SAFE-RNN | 0.288 | Not recorded | 69e9a78555a3c32af89f6fea | | Avg prob. | 0.294 | Not recorded | 69e9a78555a3c32af89f6fea | | Max prob. | 0.323 | Not recorded | 69e9a78555a3c32af89f6fea | | Running Avg entropy | 0.339 | Not recorded | 69e9a78555a3c32af89f6fea | | Running Avg prob. | 0.361 | Not recorded | 69e9a78555a3c32af89f6fea | ### SAFE-RNN-TDQC (Ours) Brier Score: 0.215. Result ID: 69e9f93547f34ce924d1f48f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### RNN-TDQC (Ours) Brier Score: 0.228. Result ID: 69e9f93547f34ce924d1f493. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### SAFE-MLP-TDQC (Ours) Brier Score: 0.229. Result ID: 69e9f93547f34ce924d1f491. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### RNN-BCE Brier Score: 0.243. Result ID: 69e9f93547f34ce924d1f492. [Reporting paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### SAFE-MLP BCE Brier Score: 0.248. Result ID: 69e9f93547f34ce924d1f490. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### Avg entropy Brier Score: 0.281. Result ID: 69e9f93547f34ce924d1f48c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### SAFE-RNN Brier Score: 0.288. Result ID: 69e9f93547f34ce924d1f48e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### Avg prob. Brier Score: 0.294. Result ID: 69e9f93547f34ce924d1f48a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### Max prob. Brier Score: 0.323. Result ID: 69e9f93547f34ce924d1f489. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### Running Avg entropy Brier Score: 0.339. Result ID: 69e9f93547f34ce924d1f48d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. ### Running Avg prob. Brier Score: 0.361. Result ID: 69e9f93547f34ce924d1f48b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"VLA Model":"π0-FAST"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.