Proposal for GPT API Integration

[exanrauzer]

I would like to initiate a discussion regarding the emerging idea of integrating GPT into Zettlr.

I believe it is crucial to discuss the possibility of implementing an API for GPT in the current context.

This addition would greatly enhance the functionality and versatility of the program, expanding its potential use cases.

Considering the growing demand for API access, implementing GPT API would be a significant step towards meeting user needs.

Let's discuss the feasibility and benefits of incorporating a GPT API to further empower the Zettlr program.

[nathanlesage]

This is an easy discussion: this is not going to happen.

If you need a few reasons:

1. This would put at jeopardy the privacy of the users
2. GPT is "three regressions in a trench coat", and it is relatively useless for proper work
3. I have personal difficulties with the personnel of OpenAI, but also with a select set of people of their direct competitors
4. There are much better text analysis methods that are privacy first, less resource incentive, do not proliferate socio-economic injustices, or enable large-scale copyright infringement.
5. I have been thinking about more text analysis methods to integrate, but, as you can see with the lack of progress, it is much more difficult than just integrating a proprietary API. The major cause of OpenAI's success is that GPT seems very human-like, but most of it is just gibberish that does not really help too much with work.

[exanrauzer]

Would you be willing to outline the circumstances or conditions that a language model must meet in order for you to consider implementing it in Zettlr as a copilot?

[nathanlesage]

After the release of v3.0 and some days off, yes ;)

[exanrauzer]

Ask GPT for help;)

[exanrauzer]

Maybe general system copilot will do the job in any text editor.

[nathanlesage]

Copilot has currently a class action lawsuit going on for copyright infringement, I'm not a fan of using possibly illegal software -- but yes, a system wide install of such a thing should do the job.

[Jaifroid]

Integrating open-source AI via an AI-agnostic API key, as provided by any number of inferencers, e.g. kobold.cpp, would be more in the spirit of Zettlr. The use-case is outlining documents, suggesting a continuation paragraph that can help unblock writer's block, critiquing the argument within a document, summarization, etc.

There is an Obsidian plugin called "Text Generator" that has API-agnostic integration for precisely these kinds of scenarios. Implementing the API calls is not difficult. The main difficulty for Zettlr would be deciding how much of a long document to send to the API. Text Generator allows you to send only starred blocks (i.e. headings and content within that heading block, where an asterisk is appended to the in-document heading), or else everything between three asterisks *** and the current cursor location, or the entire document above the cursor location. It can also prepend an instruction to the AI prompt, like "Write a continuation paragraph for the document below", or whatever you want, with user-selectable instruction prompts, such as "Summarize the document below". It can write the AI's answer either directly in the document or within a comment block (so as to distinguish clearly the AI-generated response from your own document).

There are some excellent local, offline, open-source AIs that can be run even on CPU, though they are faster with GPU. The precise model used wouldn't be Zettlr's concern, that part is up to the user: they would simply provide Zettlr with an API address and an API key. If a user wished to use an online API (such as OpenAI or ClaudeAI) that is their decision.

I find this support fantastically useful in Obsidian, but I recognize that it would be quite a bit of development work for Zettlr, which might not be seen as high priority.

[nathanlesage]

When the proposal was first proposed in June 2023, there wasn't really so much open source/open weights models on the market, but this has now thankfully changed, so I am more open to it.

However, we would need to find proper use-cases. I don't want Zettlr to ship with anything that smells like "small entrepreneur with a half-baked idea of how to slap AI onto an existing product", so I would expect a longer in-depth discussion and a lot of experimentation.

I've as of recently been using LocalChat to scout possibilities for text generation.

[Jaifroid]

I'm pleased to hear of your change of heart! And... wow, you've even built your own UI for interacting with offline AIs in GGUF format! So a lot of the work is done.

I haven't tried localchat yet, but from a quick look, it seems to be running its own inferencer directly rather than using an API and API key, which I think would be the better solution for Zettlr. I've already got too many open-source models dotted about my PC in different UI's "models" folders, which quickly becomes wasteful of space. If I have an optimal local model downloaded, I want to be able to connect to it from different UIs / apps, and not be forced to store models in each app's models folder.

Of course, for newbies, it's easier to provide them with a one-stop solution, which includes a recommended model. But if that is provided, I firmly believe the user should also be able to use their own API and API key instead. This will allow those users who wish, or who don't have the hardware resources to run a local model at acceptable speed with large contexts, to use an online API. This could even be a large open-source model running in RunPod, but if users want to use OpenAI or ClaudeAI, it should be their choice.

Regarding how to integrate AI into Zettlr, you can take inspiration from Obsidian's Text Generation plugin, which seems really good to me. Uncomplicated, giving user control over what is sent in the context, etc. It's also markdown-oriented. I'm not sure I agree with their use of "starred blocks" (adding an asterisk after a heading to include it when the user has requested only starred blocks) because it's a bit problematic with section titles that use italics. So I would use some other symbol like $. Or just keep it simple and offer only "all text above cursor" or "all text above cursor up to a *** mark on a single line".

[Jaifroid]

PS I'm sure you're aware, but most inferencers will provide, optionally, an OpenAI-compatible API and API key while the inferencer is running, connecting to localhost:[port].

[nathanlesage]

I haven't tried localchat yet, but from a quick look, it seems to be running its own inferencer directly rather than using an API and API key, which I think would be the better solution for Zettlr.

LocalChat was, to some degree, meant from me as a proof of concept to run models locally. It still has not great priority since I really want to have a proper use-case which I can't see yet, but yes, I can port a ton of the code directly for Zettlr, which is good. Albeit, for the time being, I want these two apps to be separate to further understand how generative models work and can be used.