Show HN: Open-source simulation testing infra for voice agents
github.com
Platforms that help you do simulation testing already exist. But they all charge a heavy premium on top of inference costs. We believe if the industry truly wants to scale simulation testing of voice agents, we need to stop charging a premium on inference and start providing infrastructure to scale simulations. The project is built in a way that allows you to bring your own STT, LLM, and TTS provider keys while also providing a way for us to provide inference directly within the platform at a 0% markup so you don’t have to wrestle with multiple keys/ rate limits.
A bit on why’re we’re building this -
We started working on voice agents in early 2024 and since then, have worked on numerous voice ai systems - like screenless voice-powered hardware for kids[1], AI receptionist deployed in healthcare practices like med spas & therapy clinics and personal accountability coaches. Some of these were full-fledged startups others were just side projects.
But we kept on encountering similar issues all throughout. One of the most frustrating parts was calling the agent again and again, reciting the same script to test its behavior. We also kept on encountering new issues all the time in production that we couldn't have simulated pre launch.
This frustration led us to a bigger question: how can developers trust the voice agents they’re shipping?
We believe building that trust takes two things - - First, you need a way to test the major scenarios your agent will face in production before you release it, without having to make every call yourself. But you can’t test for everything. Real world is too messy to predict in advance. - Which brings us to the second: you need a way to find issues once your agent is in production, whether it’s handling a few dozen conversations or millions. Detect drift in known behaviors AND surface unknown-unknown ways in which your agent is going wrong.
We think it’s a really hard problem to solve. And having felt it firsthand, we’re deeply motivated to take a shot at it.
Today we’re launching the simulation testing side of the platform. We feel it’s mature enough that real teams can depend on it. For eval design, we took inspiration from anthropic’s evals design[2] and extended it to voice systems. For technical & business model design we borrowed ideas from Langfuse[3]. Our stack is postgres, clickhouse & minio. Its easy to self-host[4] & the code has a permissive MIT license. We also have managed cloud version.
We’ve written more about our [testing](https://docs.egma.ai/docs/core-philosophies/testing-philosop...) and [monitoring](https://docs.egma.ai/docs/core-philosophies/monitoring-philo...) philosophy in the docs.
We’d love feedback from the HN community and people building voice agents - how are you testing today, what’s working and what’s frustrating? We’ll be in the comments. Thanks!
[1] https://x.com/theBhulawat/status/1966200231705595932?s=20 [2] https://www.anthropic.com/engineering/demystifying-evals-for... [3] https://github.com/langfuse/langfuse [4] https://docs.egma.ai/self-hosting/get-started