AI capabilities

[andrewssobral]

Hello everyone,

I would like to propose the addition of AI capabilities to CodeEdit, similar to what Cursor.so (https://www.cursor.so/) offers. Integrating AI into CodeEdit would provide a valuable feature for pair programming.

Furthermore, I suggest expanding the functionality to allow users to choose AI models beyond the limitations of OpenAI endpoints. This would be particularly useful for individuals who prefer to use local AI models to ensure data privacy instead of sharing data with OpenAI.

Thank you for considering these enhancements.
Andrews

[austincondiff]

We are certainly open to this. I think we would be behind other editors if this functionality was not offered in some way shape or form. It is not clear how deeply integrated this will be. Should these capabilities exist as an extension or should it be baked into the core? I'd love to hear some thoughts around this.

Also, as a side-note, I want to point out that Apple stays away from the term "AI" (see this video explaining this). Should we as well?

[dsr432]

I feel like generative AI is evolving so quickly that it would be pretty much impossible to keep everything up to date if it's too close to the core of CodeEdit since we're still in the early stages of development. Because of this, it might be better to have those sorts of features included through extensions, where it's easier to keep things up to date and organized, or just keep separate from the main idea of CodeEdit. I do think it's an important feature to have though, and it might be worth having it included in an "officially supported extension"(s) or something of that nature where it's still a trusted feature, but at the same time isn't the core of what we're doing, since that can, especially with recent developments, take over a brand entirely. Another idea might be to have a separate area in the extensions "store" for AI based extensions, since that is kind of it's own category at this point. If we did something like that it might be similar to how I believe themes were going to be set up? I don't know the ins and outs of that discussion at the moment though, so I might have a different idea of it in my head than what was actually planned.

As to the point of staying away from the term AI, most of the ideas brought up in that video (from what I saw) revolve around branding and keeping up appearances. However, if we are just taking these tools and using them externally, but not as the main point of CodeEdit, I don't think that is as much of a concern, though it might be good to still use the term minimally. I think anything that could go wrong (or right) would fall back on OpenAI or whatever other company is behind the tools we end up supporting. It's also worth noting that most of these GAI tools are still behind a paywall, so not everyone will even be using them to begin with. I think it's safe to say that for a while to come yet GAI won't be a central part of CodeEdit or other editors, so whether we use the term or not isn't as huge of a concern as how central we have it be to the main functionality of CodeEdit.

[matthijseikelenboom]

I agree with both points of @dsr432.

It's not like AI is integrated into VSCode. It's installed via an extension as well. So if we build a proper API, it should be doable the same way. We also have the disadvantage of being open source non profit, which means we don't have a lot of people with this kind of expertise.

As for marketing/term usage, I don't think we should follow them. We should stay "humble" I think. If we really did market as Apple we'd say that this is the first ever codeeditor for macOS and pretend the rest doesn't exist. It is AI so lets just call it what it is 😅

[andrewssobral]

Thank you @austincondiff @dsr432 @matthijseikelenboom for your feedbacks.

Continuing the discussion, it could be something like that (please see the first 5 seconds of the video)
https://twitter.com/dani_avila7/status/1671685236294565890
More info here : https://docs.codegpt.co/

One significant distinction between CodeGPT and Cursor.so is that Cursor.so seamlessly integrates CodeGPT's functionalities without the need for any plugin installations. In Cursor.so, you can simply type CMD+K and request to "generate a Python function for generating prime numbers between a start and end parameter," and you will get the desired output (please refer to the screenshots below).

However, it's important to note that Cursor.so is currently limited to OpenAI (which happens to be one of their partners), whereas with CodeGPT, you have the flexibility to use other AI models ranging from open source options to private ones, such as GPT from OpenAI and BARD from Google.

Alternatively, CodeEdit could serve as an alternative to Cursor.so, allowing you to achieve similar functionalities as Cursor.so but with the advantage of being open to other models like CodeGPT.

On the flip side, the main drawback of CodeGPT is that it cannot directly modify your existing code within the editor. Instead, you need to copy and paste your code into the plugin window or make specific requests, and then retrieve the generated code to paste it back into the editor. Despite this limitation, many find Cursor.so's approach more productive than CodeGPT, even though it is currently limited to OpenAI.

And this is what happens when I ask it to modify my current function to return a dict instead of list:

[matthijseikelenboom]

I think that because of Cursor.so 's way of working, it will never become a prominent editor. The fact that their editor is build on top of an (commercial) AI system, makes it so that big enterprise companies will not use it, because their code will be send to, in Cursor.so 's case, OpenAI's servers. This needs to be done to evaluate where it has to insert the new code into the existing code. I currently work for such a company that has forbidden the use of AI tools because of company secrecy. So if we build it into CodeEdit as a feature, it could be that people/companies won't use it because they don't trust it.

If it's an extension, the only thing it really needs is proper access to the text editor itself, in our case CodeEditTextView, to manipulate and possibly read its content. I'm convinced that VSCode can reach the same functionality as Cursor.so using that strategy. As an extension they can also pick a AI engine of their choosing. And with the modern M chips, it can perhaps even be done on device because of the Neural Engine.

So I think extensions is the way to go, we just need a good, reliable API that facilitates the functionality needed for the extension developers. Also, we current don't have anyone working in the AI field. So just like adding support for the many (web) frameworks, it's easier, or only possible, if the community picks this up. We can help in some regards, but it's impossible to do it all.

[andrewssobral]

I think that because of Cursor.so 's way of working, it will never become a prominent editor. The fact that their editor is build on top of an (commercial) AI system, makes it so that big enterprise companies will not use it, because their code will be send to, in Cursor.so 's case, OpenAI's servers. This needs to be done to evaluate where it has to insert the new code into the existing code. I currently work for such a company that has forbidden the use of AI tools because of company secrecy. So if we build it into CodeEdit as a feature, it could be that people/companies won't use it because they don't trust it.

Yes I totally agree with you. This is why CodeGPT enable user to select the model he wants to use. In the CodeGPT configuration, you could set a private / local model. In a near future, every company will use AI for coding, and most of them will have private AI models. A lot of startups (like Mistral.AI ) and big companies (like Meta) are/will provide open source AI models for coding like StarCoder, here's an example :

from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/starcoder"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# to save memory consider using fp16 or bf16 by specifying torch.dtype=torch.float16 for example
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

This is a kind of AI model that companies can deploy locally. Using FastAPI or Gradio it's very easy to expose those kind of models.

If it's an extension, the only thing it really needs is proper access to the text editor itself, in our case CodeEditTextView, to manipulate and possibly read its content. I'm convinced that VSCode can reach the same functionality as Cursor.so using that strategy. As an extension they can also pick a AI engine of their choosing. And with the modern M chips, it can perhaps even be done on device because of the Neural Engine.

Exactly.

So I think extensions is the way to go, we just need a good, reliable API that facilitates the functionality needed for the extension developers. Also, we current don't have anyone working in the AI field. So just like adding support for the many (web) frameworks, it's easier, or only possible, if the community picks this up. We can help in some regards, but it's impossible to do it all.

Yes, let’s see...