How do you know whether a prompt actually works?

As Voice AI becomes a bigger part of the solutions we build, writing the prompt is only part of the challenge. Change a few lines to fix one conversation, and you could unintentionally break another. Manually talking to an agent a few times might tell you that it seems to work, but it doesn't give you much confidence that it will behave consistently across different customers, responses and edge cases.

That was the topic one of our developers, Nitya Powar, explored at yesterday's Conversations in the AI Era developer event, hosted by Twilio and Tessl in London.

From Vibes to Validation: Testing Voice AI Prompts

In her talk, Nitya shared how we've been exploring a more structured approach to prompt evaluation.

Nitya introduced Promptfoo, an open-source tool for testing and evaluating AI prompts, and showed how it can make prompt testing much more structured and repeatable. Instead of manually testing an agent after every prompt change, teams can run the same prompt against a set of different conversation scenarios and automatically check whether the agent behaved as expected.

She also covered the different ways Promptfoo can decide whether a test passes or fails, and how these evaluations can be added to CI/CD pipelines to automatically test prompt changes before they reach production.

Treat prompts like application code

The idea is simple: treat prompts more like application code.

• Test them.
• Version them.
• Catch regressions.
• Build confidence before changes reach production.

Thank you

It was great to see so much interest from the developer community, and plenty of questions and conversations around how teams can approach testing as AI applications move beyond experimentation and into real-world use.

A big thank you to Twilio and Tessl for hosting, and to everyone who came along and joined the conversation.

Building Voice AI and want to talk testing?

If you're moving Voice AI from experiment to production and want to discuss how to test it with confidence, reach out to the team.