Configurability for semantic tokens

[michaelpj]

As discussed in #3931, the current semantic tokens implementation makes various fine distinctions that people may not want. This seems like a case where we may genuinely want some configurability. Configuration can also offer some discoverability: people can see what the highlighter is doing by looking at the default configuration.

---

Possible configuration approaches:

1. For each "hs token type" add an option to either give a specific LSP token type or a generic one

For example, we would have an option: "Specific token type for data types? Y/N (default: Y)". If yes, we would do what we currently do, if no we would fall back to the generic token type for that thing, i.e. "type".

This would basically let people opt-out of a particular highlighting distinction.

2. For each "hs token type" add an option for exactly which LSP token type to use

For example, we would have an option: "Token type for data types: type, function, interface,... (default: enum)".

This would effectively make the mapping completely configurable. We would still control the kinds of thing that can be mapped (depending on what kinds of hs token type we recognize), but in principle users could configure it to then do whatever they want.

This would also make the default behaviour maximally discoverable, since the default configuration would tell you specifically what it does for every hs token type.

The downsides are: the configuration is much more verbose and complex; you can set nonsensical things if you like, e.g. "use the 'method' token type for data types".

---

I think I'm mildly leaning towards approach 2, partly because I think the discoverability aspect is quite nice. I don't see another good way for users to find out that we are highlighting things of function type as "functions" and other variables as "variables", without writing it in the docs (which people don't read) and painstakingly keeping it in sync.

WDYT @soulomoon

[soulomoon]

@michaelpj I agree with you, approach 2 is better.
It can solve major problems here.

1. Hs token type match to LSP default is always debatable, every one would have its own flavor.
2. Different color scheme might behave vastly different on same settings. This approach 2 open the door for easier fine tuning(without them to change the color scheme).
3. Discoverability aspect as you said.

I'll open a pull request for this latter.

[michaelpj]

Done