Create a plugin for ruff/flake8 that would check for bad practices specific to Kedro

[yury-fedotov]

Description

Came up with a random idea today, wanted to run by you.

Would a Kedro-related plugin for flake8 or ruff be a thing in your opinion? And have you heard of something like that existing?

Some ideas for rulesets to check are:

• No class can be defined in file called nodes.py
• _private functions cannot be wrapped in node constructor
• Prohibit repeated sequences of e.g. 7+ nodes - recommend leveraging modular pipeline instead.
• No duplicated catalog entries (can this linter combine YAML and Python rules...?)
• No catalog entries where you have like 5 long instances that differ in just few characters - encourage Dataset Factories for that.
• Watch for credentials leakage in parameters yamls (for this we probably import a rule from existing tool, likely it exists already)

Context

I think this can simplify collaboration in large projects and help encourage proper usage of Kedro concepts.

Possible Implementation

Check how other plugins are made and implement something similar. There are tens of rulesets defined for ruff and hundreds (if not thousands?) plugins for flake8, I'm sure there are good reference examples.

For example, this is a flake8 plugin that implements a ton of custom rules.

Possible Alternatives

Do not do it :)

[astrojuanlu]

Thanks for submitting this idea @yury-fedotov!

I've considered this at some point and I think it would be awesome. It looks like ruff is getting more and more traction, thanks to its amazing speed. The flip side is that it still doesn't have a proper plugin system astral-sh/ruff#283 so submitting new checkers must be done against the core repo. Which is not a bad thing in itself, but I have no experience contributing to ruff and I don't know how it works for tool-specific rules. Maybe @sbrugman has some thoughts?

We could go the "easy" route and create a flake8 plugin, but frankly I'm not betting on the flake8 ecosystem long term... I could be wrong though.

[yury-fedotov]

Good point @astrojuanlu .

I agree that ruff is in general a more promising solution if we consider 5+ years from now. And implementing in it flake8 now contradicts with Kedro starters providing ruff as a default linter.

So maybe actual implementation should wait until ruff enables plugins ecosystem.

Another narrative in support of this could be:

1. The checks I proposed above probably target beginner to intermediate users, and this group likely leverages default ruff that comes with Kedro.
2. So flake8 implementation makes no sense.
3. Advanced users would likely anyway not do things that those checks restrict, like node(_private_func) or defining classes in nodes.py files.