Feature request: Integration of Code Formatting and Linting Tools for the improve Function Workflow

[similato87]

Policy and Info

• Maintainers will close issues that have been stale for 14 days if they contain relevant answers.
• Adding the label "sweep" will automatically turn the issue into a coded pull request. This works best for mechanical tasks. More info/syntax at: https://docs.sweep.dev/
• Consider adding the label "good first issue" for interesting, but easily approachable features.

Feature Description

I propose the integration of code formatting and linting tools into the workflow of our improve function. This would involve two key stages: pre-processing and post-processing. In the pre-processing stage, before the code is sent to LLMs for improvement, the code would be formatted and linted to ensure it adheres to standard coding practices and is free of syntax errors. In the post-processing stage, after the LLMs have applied changes, the code would again be subjected to formatting and linting to refine the improvements, rectify any introduced errors, and standardize the code style.

Motivation/Application

The integration of these tools serves several purposes:

1. Pre-processing checks: By checking the code for errors before sending it to LLMs, we can reduce the computational load on LLMs, allowing them to focus more on the complexity of the tasks rather than correcting basic syntax errors. This not only enhances efficiency but also improves the quality of the improvements suggested by LLMs.

2. Post-processing refinement: After LLMs have applied their changes, a second round of formatting and linting can significantly enhance user experience by ensuring that the improvements adhere to standard coding practices, and any errors inadvertently introduced by LLMs are corrected. This step is crucial for maintaining code quality and readability, especially for dynamically typed languages where indentation and formatting are vital, like Python.

Suggested Tools:

• Python: Utilize Black for code formatting and Pylint or Flake8 for linting to ensure adherence to PEP 8 standards and to catch common errors.
• JavaScript: Employ Prettier for formatting and ESLint for linting, allowing for customizable rules to fit our project's coding standards.
• Java: Implement Google Java Format for formatting to ensure the code is consistent with Google's Java Style Guide.

These tools are widely recognized for their robustness and flexibility, making them suitable for integration into our existing workflow. This feature would not only streamline the development process but also significantly enhance the quality and reliability of code improvements generated through the improve function.

[Styren]

Hey! Sounds like a useful feature. I had some questions;

I wonder whether it’s a good idea to run the linter if we can’t locate its config-file. ChatGPT has seen way more code than we have, so it probably has a pretty good idea of how to produce code with popular styling. So got-engineer selecting a default may just get in the way.

One question is who should fix the errors that the linter can't fix itself, user or LLM? Or should we ignore them altogether and just focus on --fixing?

Some suggestions

• I think it would be helpful to make it easy to disable autolinting for a project, for example where the rules don’t interface well with GPTs output. Or enable use of a separate config, for the same reason. However, I don't remember there being a .gpt dir or some other place we put config. Is that possible somehow yet? Otherwise I suggest env-var and cli option to disable/set linting config file.
• I suggest creating a generalizable lint-config class and add a manual on how contributors can easily add parsers for other languages

CC @ATheorell @similato87

[similato87]

@Styren Thanks for the insightful suggestions! Your idea of managing linter configs and the approach to handle non-auto-fixable errors are crucial. I'm inclined to let LLMs address these errors, while we should ensure users are informed about this process.

The concept of toggling autolinting and introducing a flexible configuration system, perhaps through a .gpt directory or environment variables, is something we should pursue. While we haven't set up a config system yet, it's a direction worth exploring.

[ATheorell]

I want to add some context here and I'll speak mainly for python, as I don't have experience with the linters in the other languages.

First of all, for the cases where the linter fails, I agree with the previous comments.

The reason this thing came up was that, particularly in improve mode, chat-gpt doesn't even get the indentation right. The LLM actually isn't that good at following PEP-8. Reason for this is probably that it has seen so much code which doesn't. The linter fixes the indentation problem in 100% of the cases I've seen. Whenever we can use a deterministic system, like a linter, which: is not random, is super fast, very reliable and very cheap, we should prefer that over using the LLM. I can honestly not think of a case where the LLM would right better style than the linter. Additionally, in many projects, like gpt-engineer, a standard linter is part of the CI-pipeline and I think this is how it should be. When gpt-engineer generates code from scratch, I really cannot see any reason why it should not be linted. In improve mode, I can imagine that linting could be a problem in rare cases. I would vote to have linting by default and the possibility to turn it off with some flag, for the few cases where people really don't want neat code or follow some outdated code standard.

[ErikBjare]

We have implemented this for https://gptengineer.app, and are in the process of adopting a gpt-engineer.toml configuration file for projects we generate, which will contain configurable entrypoints for build/test/lint/format tools.

I will present a proposal regarding this at the meeting later today :)

[ErikBjare]

Here's a teaser for my proposed gpt-engineer.toml:

[project]
base_dir = "./frontend"  # base directory to operate in (for monorepos)
src_dir = "./src"        # source directory (under the base directory) from which context will be retrieved

[run]
build = "npm run build"
test = "npm run test"
lint = "quick-lint-js"
format = "prettier"

[gptengineer-app]  # this namespace is used for gptengineer.app
project_id = "..."

# we support multiple OpenAPI schemas, used as context for the LLM
[[gptengineer-app.openapi]]  # an entry in a list of OpenAPI specs used by the project
url = "https://api.gptengineer.app/openapi.json"  # schema for gptengineer.app itself

See you at the meeting in ~1h!