Twilio Voice Agent
Integrate a real-time AI voice agent with Twilio
In this tutorial, I’m going to create a real-time voice agent that can respond to any query via speech, in speech that is initiated via a phone call using Twilio. This is an extremely flexible implementation where you can swap in any LLM or Text-to-speech (TTS) model of your liking. This is extremely useful for use cases involving voice such as customer support bots and receptionists.
To create this app, we use the PipeCat framework that takes care of stringing together all the components. Pipecat handles some of the functionality we might need such as user interruptions and dealing with audio data etc.
You can find the final version of the code here
Cerebrium setup
If you don’t have a Cerebrium account, you can create one by signing up here and following the documentation here to get set up
In your IDE, run the following command to create our Cerebrium starter project: cerebrium init twilio-agent
. This creates two files:
- main.py - Our entrypoint file where our code lives
- cerebrium.toml - A configuration file that contains all our build and environment settings Add the following pip packages near the bottom of your cerebrium.toml. This will be used in creating our deployment environment.
We need to setup a FastAPI server that will receive the call from our Twilio number and then upgrade the connection to a websocket connection - this is for the real-time two-way communication. To setup a Fast API server on Cerebrium is the same as setting up FastAPI locally. You can then add the following code to your main.py:
Don’t worry about the line from .bot import main
- we will add this later. In your current directory, create a folder called ‘templates’ and inside it a file called ‘stream.xml’.
We need to send back a xml response to Twilio in order to upgrade the connection to the websocket connection. Add the following code to the streams.xml file:
The stream url will be the base endpoint of your deployment once our application is deployed. It should be the same in the above, you should just fill in your project id. Before we continue with our implementation, let us setup our Twilio number.
Twilio setup
Twilio is a cloud communications platform that enables businesses to integrate messaging, voice, video, and authentication capabilities into their applications through APIs. We will use them for this demo, but you can use any alternative provider. If you don’t have an account you can sign up here - they have a generous free tier.
Once you have an account, you can navigate to the page in order to buy a number. When you buy a number please make sure its not a toll-free number - this won’t work. We then setup a webhook that hits our endpoint in order to connect to the agent.
You should then save the changes above and move on to setting up our AI Agent.
AI Agent Setup
In order to create our AI agent, we will be using the PipeCat framework that takes care of stringing together all the components and it handles some of the functionality we might need such as user interruptions, dealing with audio data etc. To setup our AI agent, let us create another file called ‘bot.py’ and add the following code:
The code above does the following:
- We connect our agent to our websocket transport layer that will send/receive audio.
- We setup our LLM, TTS and STT services using Deepgram, OpenAI and Cartesia. You can select any provider you like - Pipecat supports a wide variety.
- In order to authenticate all these services, we use Secrets. Add them to your Cerebrium dashboard.
- Lastly, we then put this all together as a PipelineTask which is what Pipecat runs all together. The makeup of a task is completely customisable and has support for Image and Vision use cases. You can read more here. Pipeline tasks come with a structure and parameters that make it easy to handle interruptions out the box, and we are able to swap models to our preference only changing a few lines of code.
- The Daily Python SDK comes with a lot of event webhooks where you can trigger functionality based on events occurring. So we handles certain events such as a user leaving/joining a call.
In order to improve the latency of the above system, you can run parts/all of the pipeline locally. This would help you achieve roughly ~500ms end-to-end latencies. You can read more how we did this here and here.
Deploy to Cerebrium
To deploy this app to Cerebrium you can simply run the command: cerebrium deploy in your terminal.
If it deployed successfully, you should see something like this:
In order to test this application you can simply call your Twilio number and the agent should start responding.
Conclusion
Hopefully, this tutorial acts as a good starting point for you to implement voice into your application as well as extend it into image and vision capabilities. Pipecat is a extensible and open-source framework that makes it easy to build applications like this, and Cerebrium makes the process seamless to deploy and autoscale while only paying for the compute you need.
Tag us as @cerebriumai so we can see what you build and please feel free to ask questions/send feedback to us on Slack or Discord communities