Stable Diffusion does not seem to be working when used with ModelSpecs

[K-J-VV]

What happened?

Trying to get Stable Diffusion plugin to become available in my self hosted environment. Before I was getting Stable Diffusion plugin to work by using LiteLLM as a proxy for OpenAI to connect over to LocalAI (which does actually work). But I think the newly implemented modelSpecs would be a cleaner way to achieve what I'm trying to setup and it seems like it should be able to work without requiring LiteLLM at all, or at least that's what I understand from reading the new documentation. I understand it as the modelSpec should allow LibreChat to simply directly connect to a StableDiffusion instance

The following is what I am doing:

Steps to Reproduce

1. Created working modelSpec following modelSpec guide
2. Install stable-diffusion-webui per LibreChat guide
3. Set variable `SD_WEBUI_URL=https://stablediffusion.example.com' (SD is reverse proxied in homelab)
4. To the working modelSpec, follow modelSpec guide
   and add the following pluginKey to the preset tool:
   tools: ["stable-diffusion"]

Expectation:
Plugins will become available via endpointsMenu & will be able to install/use Stable Diffusion

Reality:
Model works as if there is no stable-diffusion tool added, plugins are still not made available.

The following is my modelSpec

modelSpecs:
  list:
    - name: "myai"
      label: "MyAI"
      default: true
      description: "My AI"
      iconURL: "https://upload.wikimedia.org/wikipedia/commons/examplelogo.png"
      preset:
        chatGptLabel: "MyAI"
        endpoint: "MyAI"
        greeting: "Hello there!"
        frequency_penalty: 0
        model: "gpt-4"
        presence_penalty: 0
        promptPrefix: >
          You are a helpful AI, answer any question asked.
        resendFiles: false
        temperature: 0.8
        top_p: 0.5
        tools: ["stable-diffusion"]

• I agree to follow this project's Code of Conduct

[danny-avila]

Unfortunately, tools are only available for use if you set the endpoint to gptPlugins

[K-J-VV]

@danny-avila ahh i see, thanks again for quick response (as always), very much appreciated!

FYI for you, I think I found something interesting. Or at least it's something I'm glad I got working but don't entirely understand exactly why this is working lol.

• I have OPENAI_REVERSE_PROXY pointing to a container running LocalAI. For a self-hosted environment LocalAI is an excellent 1:1 drop in replacement for OpenAI. It's setup that when I need to reference OpenAI url/API endpoints, LocalAI as a replacement just works.

• It's necessary for me to use LocalAI to make the plugins drop down available.

• I have SD_WEBUI_URL pointing to a container running Stable Diffusion WebUI Docker with the UI using AUTOMATIC1111.

• It's necessary for me to have SD_WEBUI_URL set to this container in order for the Stable Diffusion plugin to work.

I'm glad its working for me, its just a bit confusing why & how both LocalAI and Stable Diffusion WebUI Docker are needed since these are entirely separate containers and do not communicate to one another. LocalAI is doing all the actual heavy-lifting/work of actual image generation, but I just need the Stable Diffusion WebUI Docker container running so that the SD_WEBUI_URL can point to a AUTOMATIC1111 interface.

Just bringing this to your attention in case it's helpful! You mentioned a bit ago you are still working on tool usage to make non-openai endpoints work.

From my experience/testing:

• LocalAI is an excellent/easy just works 1:1 drop in replacement for OpenAI in almost every use case I've tested it in (via LibreChat & others)
• I do get better performance from Ollama as a base LLM versus LocalAI
• I get similar performance from LocalAI & Stable Diffusion WebUI Docker for image generation, but slightly better results from LocalAI

I'm not at all a developer, so just sharing a homelab user perspective: It'd be great if it'd be possible to set the librechat.yaml file to have the following endpoints:

1. Ollama to be used as the base LLM (with a modelSpec to entirely customize it's appearance)
2. LocalAI/Stable Diffusion WebUI Docker (to be used specifically as the endpoint for plugins/tools, in this specific example it would be for Stable Diffusion)
   It'd be nice to have the Stable Diffusion WebUI Docker work as the tool endpoint to minimize/simplify the number/size of containers I have running. But it seems LocalAI would be much simpler to actually get working since its been such a good drop in for OpenAI.