[RFC] Key-Value Separators

[deathaxe]

Relational

Several syntaxes such as Python, JavaScript or JSON support data objects in the form of arrays, lists or key-value pairs.

Those data objects can be grouped into:

1. sequences
   • arrays (C, C++, ...)
   • lists (Python, JS, JSON, ...)
   • vectors (C++ std/boost library)
   • ...
2. mappings (aka. key-value pairs)
   • dictionaries (VBScript)
   • maps (C++ std/boost library)
   • objects (JSON)
   • ...

That's pretty clearly expressed by meta scopes meta.sequence and meta.mapping.

List separators are commonly scoped punctuation.separator.sequence.

["item1", "item2", ...]
//      ^ punctuation.separator.sequence

The situation with mapping punctuations seems a bit more unclear at the moment which caused in several solutions having found their way into the wild.

{"key1": "value1", "key2": "value2", ...}
//     ^ punctuation.separator.???
//               ^ punctuation.separator.???

Motivation

The current situation is reflected by a couple of discussions started in pending PRs, such as #2682 (comment) or #2564 (comment) which re-introduces punctuation.separator.mapping to YAML, while it has been removed by 60e7c39.

If this topic has been discussed somewhere the related posts seem lost. Hence it may make sense to come up with a guideline how to commonly scope punctuations in data objects such as mappings or sequences in a (this) RFC issue.

Current Situation

The following table, started by @jwortmann, illustrates currently used scopes for separators of key-value pairs.

Syntax	key-value separator, e.g. : or =	entry separator, e.g. , or ;	uses meta.mapping
BibTeX	punctuation.separator.key-value	punctuation.separator.mapping.pair	✔️
Cabal (PR)	punctuation.separator.mapping.key-value	-	✔️
CSS	punctuation.separator.key-value	punctuation.terminator.rule	❌
D	punctuation.separator.mapping.key-value	punctuation.separator.sequence	❌
Erlang	punctuation.separator.mapping.key-value	punctuation.separator.mapping.pair	✔️
JSON	punctuation.separator.mapping.key-value	punctuation.separator.mapping.pair	✔️
Java Properties	punctuation.separator.mapping.key-value	-	✔️
Git config	keyword.operator.assignment	-	✔️
JavaScript (object literal)	punctuation.separator.key-value	punctuation.separator.comma	✔️
Lua (table)*	punctuation.separator.key-value	punctuation.separator.field	✔️
Ruby	punctuation.separator.key-value	punctuation.separator.sequence	❌
Perl	punctuation.separator.key-value	punctuation.separator.sequence	❌
Python (dict)	punctuation.separator.mapping.key-value	punctuation.separator.mapping	✔️
YAML	punctuation.separator.key-value.mapping	-	❌

Notes:

• * not sure whether the Lua tables are directly comparable to the other mappings listed here
• Some other syntaxes use punctuation.separator.key-value in non-data-object related expressions.

The concluding question is about priority of key-value vs. mapping or about which of both the more general part of a scope name is.

Both scope naming schemes punctuation.separator.key-value and punctuation.separator.mapping are both used with nearly the same amount, while the latter one found its way into syntaxes by being suggested by PackageDev's auto-completions.

Suggestion

We should probably condence those scopes to ...

• punctuation.separator.key-value for key-value separaotrs
• punctuation.separator.sequence for array items, list items and key-value-pairs

Reasons

1. Not all syntaxes know about the concept of dictionaries/objects/mappings but still need a way to separate property keys from values.
2. As a result of 1. punctuation.separator.key-value seems the more general scope which color schemes can use to target most possible tokens with just one simple selector.
3. If it seems important to distinguish key-value separators of mappings from others meta.mapping punctuation.separator.key-value or ``punctuation.separator.key-value.mapping` may be used.
4. A dictionary/object/mapping can be interpreted as a list of key-value pairs. Hence punctuation.separator.sequence seems suitable to separate key-value pairs.
5. To further justify punctuation.separator.sequence - both, lists and objects often use the same character (e.g.: comma ,) to separate list items and key-value pairs (see: JSON). A common scope ensures both to be highlighted the same way by all color schemes. Further distinctions can be made by eighter meta.mapping punctuation.separator.sequence - meta.mapping meta.sequence or punctuation.separator.sequence.mapping if really needed.

Alternative

Distinguish between real data lists/objects and other kinds of key-value pairs, which means to keep punctuation.separator.mapping.key-value and punctuation.separator.key-value.

[jrappen]

I find your suggestion of using two scopes:

• punctuation.separator.key-value
• punctuation.separator.sequence

vs. three scopes:

• punctuation.separator.mapping.key-value
• punctuation.separator.mapping.pair
• punctuation.separator.sequence

confusing when we have a situation within meta.mapping where:

• we'd have a punctuation.separator.sequence as a replacement for punctuation.separator.mapping.pair
• while making a difference between meta.sequence and meta.mapping

which would leave us with:

{ "key1": "value1", "key2": "value2", ... }
//      ^ meta.mapping punctuation.separator.key-value
//                ^ meta.mapping punctuation.separator.sequence

vs.:

{ "key1": "value1", "key2": "value2", ... }
//      ^ meta.mapping punctuation.separator.mapping.key-value
//                ^ meta.mapping punctuation.separator.mapping.pair

---

Or are you suggesting ❓

• punctuation.separator.key-value.mapping
• punctuation.separator.sequence.mapping
• punctuation.separator.sequence

---

Is the main reason for this suggestion to limit scopes to three levels?

---

Random fact of the day: Emojis originate from Japan where they use a combination of ⭕ and ❌ vs. ✔️ and ❌.

[jwortmann]

Is the main reason for this suggestion to limit scopes to three levels?

From my point of view the main reason is to standardize a single scope name which can be used in all syntaxes, to make it easier for color schemes to target the key-value separators. As it was pointed out in the linked commit and previous discussions, "key-value" is more general than "mapping" at the third level of the scope name. The scope punctuation.separator.key-value is already used in a few syntaxes where it is not part of an actual mapping literal, for example in HTML:

<div class="foo">
<!--      ^ punctuation.separator.key-value - meta.mapping -->

For this reason we can't use punctuation.separator.mapping if we want to unify the scope names. And if you want to only target the separator punctuations which are inside of actual mapping literals, then it should be sufficient to use the selector meta.mapping punctuation.separator.key-value in color schemes.

---

When I looked at the Lua tables¹ recently, I also had the idea about using punctuation.separator.sequence for the comma even inside of mappings, but I didn't suggest it because I thought it was too big of a change and might seem a bit confusing on first sight. But now I like it, because a mapping is just a sequence of key-value pairs (the meta.mapping scope is of course still kept to make this distinction possible). And it would make it even easier to target all the commas (or similar punctuation symbols) in both mappings and array structures at the same time, with only a single scope. If you want to target the commas in those two kind of structures differently, then you can use two separate rules and prepend the corresponding meta-scope for each rule.

The only drawback I see, is that the statement from the last sentence doesn't work for certain nested combinations of both mappings and arrays, because of the bug² described in sublimehq/sublime_text#2152. This is probably only relevant for JSON, and I would think that the occasion that someone wants to target the commas in only one of mapping or array seems extremely unlikely and is negligible. Besides that, this "bug" should be fixed in ST core and shouldn't stop us from introducing consistent scope names.

Therefore I would say that appending a further mapping at the fourth level after key-value/sequence to the punctuation scope would be superfluous; we already have the meta.mapping.

---

(1) Afaik Lua tables are both mapping and array in one, you can use them like {"foo", "bar", "baz"} with normal array indexing (with consecutive index numbers assigned automatically for the keys), or add the keys yourself like {key1="foo", key2="bar", "baz", "foobar"}, even in mixed form.

(2)

>>> sublime.score_selector("meta.mapping meta.sequence meta.mapping punctuation.separator.sequence", "meta.mapping punctuation.separator.sequence")
12304
>>> sublime.score_selector("meta.mapping meta.sequence meta.mapping punctuation.separator.sequence", "meta.sequence punctuation.separator.sequence")
12416

[jrappen]

Ok, I was confused for a bit because adding meta.mapping wasn't part of the suggestion.

[jrappen]

I've added the requested changes to JSON. Will update #3097 soon accordingly.