Need to build a realtime AI voice conversation bot with twilio

[karanbangia]

I am new to AI, i am primarily a backend engineer, i have been given a task to create an AI voice conversation assistant which is connected to twilio and can talk to customer, understand what they want such as refund/bookTable/placeAnOrder etc ,we have already built a chatbot with OPENAI Assistant, now we want to add conversation over top of it, i really dont know where to start, but i want to create a prototype as soon as possible, it would be really helpful if anyone can give me some kind of roadmap

[karanbangia]

@bobqianic

[bobqianic]

i really dont know where to start, but i want to create a prototype as soon as possible, it would be really helpful if anyone can give me some kind of roadmap

I think since there is already a chatbot, as long as the output of the chatbot is taken as input, fed into the TTS model, and then converted into voice for playback, it will work. For the user’s feedback, you can first record it, then input the audio into Whisper to convert it into text, and then input it into the chatbot. This is a bit like the talk-llama in the examples.

[karanbangia]

As the chatbot is synchronous how will it handle when to input to chatbot and when to stop listening and various other scenarios. Is there any library to do so?

[bobqianic]

when to input to chatbot and when to stop listening

VAD can handle the task effectively. You just need to determine a threshold based on the outcomes of your experiments.

Is there any library to do so?

For VAD, there are some libraries available.

[bobqianic]

There are also some pre-trained models that can perform VAD.
eg. https://github.com/snakers4/silero-vad

[kwindla]

If you're trying to build something quickly, maybe a SaaS service like https://vapi.ai/ is worth looking into. They support Twilio, I believe.