how to load local modell???????????

[khalilxg]

No description provided.

[0xM4sk]

modified core/builder_config.py ---

from llama_index.llms.openai_like import OpenAILike

API_KEY = os.getenv('OPENAI_API_KEY')
BUILDER_LLM = OpenAILike(
api_base="[IP]:1337",
model="[model ID]",
is_chat_model=True,
max_tokens=None,
api_version="v1",
api_key=API_KEY,
)

using this method I was able to perform inference against local models hosted by Jan. Unfortunately my TensorRT Mistral model had streaming issues but I got other models working partially. .streamlit/secrets.toml does seem to need a valid openai API key, im not seeing any usage, but worth noting.