# Safety certification of tool-integrated LLMs on OpenAPI pool

Source: [Safety certification of tool-integrated LLMs on OpenAPI pool](<https://www.sota2.com/research/sota/safety-certification-of-tool-integrated-llms-on-openapi-pool>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-14T08:45:40.757Z

Evaluation of LLM safety certification performance specifically for models integrated with OpenAPI tools, using the OpenAPI pool dataset.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Safety certification of tool-integrated LLMs |
| Dataset (including any stated split) | OpenAPI pool |
| Metric | PU Score |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 5 of 5 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [PU Score](<https://www.sota2.com/research/sota/safety-certification-of-tool-integrated-llms-on-openapi-pool.md?metric=PU%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | PU Score | Result date | Report paper |
| --- | --- | --- | --- |
| LLMCert-T | 72 | Not recorded | 68fa7bde1e209131cea8aea7 |
| LLMCert-T | 67 | Not recorded | 68fa7bde1e209131cea8aea7 |
| LLMCert-T | 50 | Not recorded | 68fa7bde1e209131cea8aea7 |
| LLMCert-T | 22 | Not recorded | 68fa7bde1e209131cea8aea7 |
| LLMCert-T | 19 | Not recorded | 68fa7bde1e209131cea8aea7 |

### LLMCert-T

PU Score: 72. Result ID: 6a058aac47f34ce924ec9bb8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

Recorded evaluation configuration: {"Specification":"Intent Shifting","Selector":"Llama-3.1 8B","Generator":"Gemma-3 4B","M":"300","N":"10"}. This record does not assert independent reproduction.

### LLMCert-T

PU Score: 67. Result ID: 6a058aac47f34ce924ec9bb7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

Recorded evaluation configuration: {"Specification":"Abstention Trigger","Selector":"Llama-3.1 8B","Generator":"Gemma-3 4B","M":"300","N":"10"}. This record does not assert independent reproduction.

### LLMCert-T

PU Score: 50. Result ID: 6a058aac47f34ce924ec9bb6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

Recorded evaluation configuration: {"Specification":"Privilege Escalation","Selector":"Llama-3.1 8B","Generator":"Gemma-3 4B","M":"300","N":"10"}. This record does not assert independent reproduction.

### LLMCert-T

PU Score: 22. Result ID: 6a058aac47f34ce924ec9bb5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

Recorded evaluation configuration: {"Specification":"Top-N Saturation","Selector":"Llama-3.1 8B","Generator":"Gemma-3 4B","M":"300","N":"10"}. This record does not assert independent reproduction.

### LLMCert-T

PU Score: 19. Result ID: 6a058aac47f34ce924ec9bb4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bde1e209131cea8aea7.md>)

Recorded evaluation configuration: {"Specification":"Distractor Selection","Selector":"Llama-3.1 8B","Generator":"Gemma-3 4B","M":"300","N":"10"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.