Arthur Bench
Open-source LLM evaluation and comparison tool.
What it does
Open-source LLM evaluation and comparison tool. Helps teams compare LLM options using consistent metrics for model selection, budget optimization, and real-world performance validation. Available in local and cloud-based versions.
- AI governance and policy enforcement
- AI performance monitoring
- LLM evaluation
- agent observability and tracing
- agent discovery and inventory
- Category
- AI evaluation and observability
- Users
- enterprise security teams, audit teams, governance teams, engineering teams and 1 more
- Public API
- Not offered
- Deployment
- Self-hosted
Not yet disclosed: platforms, languages and support channels.
Pricing
- Free plan$0
Prices as published by Arthur, checked Oct 2026.
Security and compliance
As reported by Arthur. Ask for audit reports before you buy.
- SOC 2Compliant
- HIPAACompliant
- ISO 22301Compliant
- NIST CSFCompliant
- FedRAMPCompliant
- RBAC (Role-Based Access Control)
- SSO (Single Sign-On)
- Immutable audit logs
Reviews
No reviews yet
Reviews come from people who use Arthur Bench at work. Share how your team uses it to help other buyers decide.
Alternatives
Tradewinds
by Olam Labs
Catan-inspired island trading game where players roll for resources, bargain, and build to 10 points; currently in development.
KitchenBench
by Robocurve
Evaluating robots on ten bimanual kitchen tasks, including pouring, lid removal, folding, part-mating, two-arm handover, and tool-mediated scooping.
LLM evaluation
by Openlayer
Rigorous evaluation for GenAI and LLMs—test every prompt, trace every response, and validate every output.
Self Diagnostics
by Raindrop
Capability for agents to proactively report their own failures via SDK instrumentation.
Evaluates and improves AI on meaningful use cases via production-grade RL environments, accelerating custom evals and post-training data creation.
Loss Analyses
by AfterQuery
Identifies where and why models fail in professional contexts through systematic study, pinpointing the precise failure modes and distributional gaps that inform dataset and env...
Is this your product?
Claim it to keep details accurate and submit evidence for the next ranking review.
Claim Arthur Bench