Hamming AI logo

Voice AI testing and evaluation · YC S24

Hamming AI: Catch Voice Agent Failures Before Customers Do

Meet Hamming, the QA platform that tests voice and chat agents before launch, red-teams them and checks every production call.

Hamming’s four-step workflow: connect an agent, generate tests and red-team probes, run concurrent calls, and monitor production
Hamming’s testing, red-teaming and production monitoring workflow. Test scenarios shown are illustrative.Hamming AI · Product workflow ↗

Building a voice agent is fast. Testing it is not.

A team can prompt an agent in under an hour, then spend hours calling it by hand. Tweak the prompt, call again, and find the regression only when a customer complains. [2][4]

Hamming automates that loop.

It tests voice and chat agents before launch, red-teams what could go wrong and monitors every production call. [1]

When something fails, the failure becomes a regression test, not a story a customer tells. [1]

Thousands of callers before the first real one

Hamming connects to your agent and generates test scenarios automatically. Hundreds of simulated callers, each with a job to do, such as canceling next week’s appointment, call the agent and check whether the job got done. [2][3]

They interrupt, speak with different accents and switch languages. Hamming covers more than 65 languages and accents and can run more than 50,000 test calls at once. [1]

Its red-team suite tries prompt injection, jailbreaks and attempts to pull out personal data. [1]

In production, Hamming scores every call against more than 50 built-in metrics and alerts the team before customers notice. Its automated scores agree with human evaluators 95–96% of the time. [1][3]

When a call goes wrong, one click turns it into a new test, so the next release is checked against it. [1][2]

The first test report takes under ten minutes. Hamming is SOC 2 Type II and offers HIPAA business associate agreements. [1]

Built by people who move metrics with experiments

Sumanyu Sharma grew an AI-powered sales program at Tesla to hundreds of millions of dollars in revenue a year as a senior staff data scientist. Sharma then helped the safety app Citizen grow its monthly users fourfold. [3]

Hamming’s first launches came from Sharma and Marius Buleandra, who ran data infrastructure at Anduril and was a founding engineer at Spell, an MLOps startup acquired by Reddit. [2]

In those jobs they learned that the fastest way to move a metric is to run more experiments. After shipping more than ten AI agents to production in 2023, they saw the same rule hold for reliability: more tests, more reliable agents. [6]

Hamming is part of Y Combinator’s Summer 2024 batch and is based in San Francisco. It has tested more than 4 million calls and monitors more than 10,000 agents for startups, banks and healthtech companies. [1][3]

What this looks like in practice

Bland Labs: 200 test calls before the customer’s first

Bland Labs builds voice agents for companies in healthcare, insurance, legal and home services, and its customers run 1,000 to 10,000 calls a day. [4]

Its engineers could prompt an agent in 30 to 45 minutes, then spend the next two to three hours testing it by hand. Some launches took a week of review. [4]

With Hamming, Bland Labs runs 100 to 200 test calls before a customer starts testing, covers Spanish and other languages without native speakers, and checks healthcare guardrails before deployment. Its deployment cycles are three times faster. [4]

One test caught an agent telling callers it had booked their appointment when it hadn’t. It was fixed before any customer heard it. [4]

Grove AI: every clinical-trial call checked

Grove AI’s agent, Grace, recruits and engages participants for clinical trials. A missed eligibility question can disqualify a participant or put a patient at risk. [5]

Grove’s QA staff could review only 5–10% of calls. With Hamming, Grove checks every call, has analyzed more than 165,000 conversations and gets real-time alerts instead of finding problems days or weeks later. [5]

Before a release, Grove now runs more than 1,000 automated test scenarios, up from about 50 manual tests. [5]

Where Hamming AI ranks

View the comparisons and methodology

Most teams test a voice agent by calling it themselves. Hamming calls it thousands of times before launch, then checks every real call after it. [1][2]

The bigger picture

Voice agents now book appointments, screen clinical-trial patients and answer insurance questions. [4][5]

Most teams still test them by hand and review a sample of the recordings. [2][5]

Hamming is betting that agents need the same discipline as any other software release: thousands of tests before launch, and every real call checked after it.

The bad call becomes a test, not a customer story.

Written by SOTA2 from Hamming AI’s public materials. Customer experiences and product plans are reported by Hamming AI. Ranking analysis and comparison data are from SOTA2.