Add basic semantic tokens implementation

[u9g]

This pr adds an ast visitor and a semantic token implementation for

• {module}.{label}
• {tuple}.{label}
• {tuple}.{index}
• {record}.{label}
• pattern of variable (ie let x = 9)
• import coloring depending on whether the import is a type, function, record, or other

[lpil]

Hello! What functionality does this provide to the user?

It would be best to open an issue to discuss new features before implementing them as there's no guarantee we can accept any particular change.

[u9g]

Hello! What functionality does this provide to the user?

Here's a before and after in vscode

It would be best to open an issue to discuss new features before implementing them as there's no guarantee we can accept any particular change.

it's fine if this isn't accepted, I developed this for my own use but I also think it would be nice to see an implementation land in the standard LSP too.

[lpil]

Sorry, I'm a bit confused. It looks like it broke your syntax highlighting?

[u9g]

Sorry, I'm a bit confused. It looks like it broke your syntax

This is using the latest released vscode-gleam, not my syntax.

[inoas]

I think it shows function values versus other values in different colours

[lpil]

In the after screenshot it seems to inconsistently highlight variables, and not all the keywords are highlighted any more.

That aside, what does this enable for the programmer?

[PgBiel]

I believe the main point is that information about the types of values can be used for highlighting (depending on how your theme handles this information), but this is also useful for basic highlighting in some editors which support it, such as KDE's Kate. If correctly done, this can be pretty valuable.

[MystPi]

I've been wanting something like this for a long time! When I started out with Gleam this was one of the features I missed most from the LSP.

In the after screenshot it seems to inconsistently highlight variables, and not all the keywords are highlighted any more.

That aside, what does this enable for the programmer?

It's not highlighting variables inconsistently; they are highlighted based on whether they are a function (yellow) or another value (blue). This is actually really helpful in other languages like JavaScript that do this:

The only keyword that looks wrong (from what I can tell) is the as in the topmost import. Likely something that would need to be fixed before merging.

[nicklimmm]

We have an existing AST visitor implementation here: https://github.com/gleam-lang/gleam/blob/main/compiler-core/src/ast/visit.rs