TTS API improvements

[blob42]

Description

Improvements to the Coqui TTS API/backend.

• tts coqui xtts_v2 not working without speaker_idx  #2073: Allow passing speaker_id to models
• Add optional language parameter to TTS endpoint/schema
• [ ] TTS Info endpoint: List available models, speakers and languages (will start new PR for this one)
• update swagger documentation
• define tts models with config files
• updated docs

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	b2361dc
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6632c4cbace1270008899180

[mudler]

this looks like a leftover, or is it wanted?

[blob42]

I was not sure, I will remove it then.

I am planning to include an endpoint to list models/speakers in this PR.

[mudler]

I don't see how the changeset can fix #2073 - is there something missing in the PR?

[blob42]

@mudler I didn't push those changes yet, I will remove the draft status when I will be done

[blob42]

@mudler I am trying to understand where/when is the go gRPC server -> TTS service used, Is this a work in progress ?

[blob42]

Can I add a new Lang field in the protobuf definition ? It would be an optional one.

[dave-gray101]

If the language is truly independent of both the model, voice, and input text, I see no reason not to have a Language parameter. Personally, I prefer to spell it out rather than name it Lang?

[blob42]

Agreed better to have clearly defined parameter.

Does it make sense to keep the COQUI_LANGUAGE env var ? What use case does it serve ?

[blob42]

I didn't push the swagger docs, it gave me alot of changes.

Quick way to test the language switching capability with multilingual models is something like this:

Without specifying lang:

The voice uses an English accent.

curl -L http://localai:8080/tts \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer 2708b7c21129e408899d5a38e6d1af8d " \
    -d '{
"backend": "coqui",
"input": "Bonjour Madame ! Comment allez-vous ?",
"model": "tts_models/multilingual/multi-dataset/xtts_v2",
"voice": "Ana Florence"
}' | aplay -D pipewire -

With lang:

Proper language accent is used

curl -L http://localai:8080/tts \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer 2708b7c21129e408899d5a38e6d1af8d " \
    -d '{
"backend": "coqui",
"input": "Bonjour Madame ! Comment allez-vous ?",
"model": "tts_models/multilingual/multi-dataset/xtts_v2",
"voice": "Ana Florence",
"lang": "fr"
}' | aplay -D pipewire -

[blob42]

Quick update regarding adding TTS Info endpoint. I am skipping this feature from this PR is it would involve too many changes that are out of scope for this PR.

Context:

The goal is to have the possibility to query available models/speakers or other type of information depending on the backend.

My first attempt was to add a gRPC service TTSInfoRequest to query the backend. I found out down the road that the backend grpc service is loaded with the model at the same time, however Info requests might not send any model infromation.

My proposal is to allow backends grpc service to be spawned without a model and to add a service called Info() or Query that backends can use to send arbitrary infromation. A model could be loaded later using the same spawned service or tear-down and start a new one for the designated model.

I will start a PR or Discussion for this proposal.

[mudler]

overall looks good, thanks! just few nits/open questions above