Unicode-aware `\w`, `\d`, and `\b` ?

[markusicu]

@mathiasbynens wrote in #2 (comment) :

One other thing we could include in this new flag is Unicode-aware \w, \d, and \b. I originally proposed this to be part of the u flag but it was rejected out of fear it would hurt adoption of the u flag. tc39/proposal-regexp-unicode-property-escapes#22 (comment) We also could take it one step at a time, and ban \w, \d, and \b with the new flag for now, and then decide on their behavior later.

[markusicu]

One other thing we could include in this new flag is Unicode-aware \w, \d, and \b.

sgtm

UTS #18 has long had recommendations for these: https://www.unicode.org/reports/tr18/#Compatibility_Properties

We also could take it one step at a time, and ban \w, \d, and \b with the new flag for now, and then decide on their behavior later.

Either works for me.

[markusicu]

Cc stage 3 reviewers @waldemarhorwat @gibson042 @msaboff

One other thing we could include in this new flag is Unicode-aware \w, \d, and \b.

sgtm

UTS #18 has long had recommendations for these: https://www.unicode.org/reports/tr18/#Compatibility_Properties

ES regex definitions see https://tc39.es/ecma262/multipage/text-processing.html#sec-characterclassescape

Details

\d

• ES regex: [0-9]
• UTS #18: \p{gc=Decimal_Number}
• Unicode comment: Non-decimal numbers (like Roman numerals) are normally excluded.

\w

• ES regex: [0-9a-zA-Z_] plus, if IgnoreCase, characters whose Simple_Case_Folding is in that set
• UTS #18: \p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\p{Join_Control}
• Unicode comment: This is only an approximation to Word Boundaries (see b below). The Connector Punctuation is added in for programming language identifiers, thus adding "_" and similar characters.

\b

• ES regex: transition between \w and \W
• UTS #18: Default Word Boundaries
• Unicode comment: If there is a requirement that \b align with \w, then it would use the approximation above instead. See [UAX29]. Note that different functions are used for programming language identifier boundaries. See also [UAX31].
• see also:
  • https://www.unicode.org/reports/tr18/#Simple_Word_Boundaries
  • https://www.unicode.org/reports/tr18/#Default_Grapheme_Clusters

We also could take it one step at a time, and ban \w, \d, and \b with the new flag for now, and then decide on their behavior later.

Either works for me.

I suggest that we use the Unicode definitions for \d and \w, and forbid \b for now. It seems like \b would need a lot more thought and discussion than the others.

[mathiasbynens]

I suggest that we use the Unicode definitions for \d and \w, and forbid \b for now. It seems like \b would need a lot more thought and discussion than the others.

SGTM

[macchiati]

Hmmm. I don't Think we could ship without \b. What about having it aligned with \ w, but allow for \ b curly brace...

[markusicu]

What about having it aligned with \ w, but allow for \ b curly brace...

I am ok with that. It looks like that might be what Java does:
https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/util/regex/Pattern.html

Construct	Matches
\b	A word boundary
\b{g}	A Unicode extended grapheme cluster boundary

Probably related to the syntax suggested by https://www.unicode.org/reports/tr18/#Default_Grapheme_Clusters

We should check that \b{g} throws a SyntaxError so that it can be added later.

[macchiati]

+1 Mark

…

On Tue, Jul 13, 2021 at 2:34 PM Markus Scherer ***@***.***> wrote: What about having it aligned with \ w, but allow for \ b curly brace... I am ok with that. It looks like that might be what Java does: https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/util/regex/Pattern.html Construct Matches \b A word boundary \b{g} A Unicode extended grapheme cluster boundary Probably related to the syntax suggested by https://www.unicode.org/reports/tr18/#Default_Grapheme_Clusters We should check that \b{g} throws a SyntaxError so that it can be added later. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMFGLAMBWG67WGTDWOLTXSWM5ANCNFSM4ZNFBL2A> .

[gibson042]

This seems like a further expansion of scope, specifically relating to https://www.unicode.org/reports/tr18/#RL1.4 ... do you want to recharacterize the proposal as something like UTS 18 regular expressions and primarily ensure that "provision should be made for the syntax to be extended in the future to encompass [support for higher-level features]"?

[markusicu]

This seems like a further expansion of scope

It is an expansion of scope, which I like to resist, but Mathias is right to point out a backlog of things that would require incompatible changes, and while we are at it with a new flag, we should discuss what makes sense (and is sufficiently low-risk) to include, vs. where we can and should grease the skids with SyntaxErrors to make reasonable future extensions possible without requiring yet another new flag.

I don't think we want to advertise this proposal as everything that UTS18 suggests, but I welcome the discussion. Let's just remember that at some point soon we do want to produce a real spec and go to stage 3 and into implementations :-)

[gibson042]

I think we are in agreement. I don't want to expand this proposal to "all of UTS 18", but I do want to ensure that sufficient accommodations are made for future proposals to do so with minimal friction (ideally without introducing any more flags) and also to be clear about that in the description of this proposal (which seems to be transforming from "set notation" into something like "UTS18 compatibility").

And I'll further note that it is actually close to achieving UTS 18 Level 1, although I do see gaps relating to both RL1.4 (this issue) and RL1.6 (recognition of line boundaries, e.g. /./v.test("\u0085") would need to be false and /^\n/vm.test("\r\n") might need to be as well).

EDIT: see also #37

[markusicu]

It looks to me like \b{g} is currently a SyntaxError because the curly braces are SyntaxCharacter and thus not PatternCharacter, which means that they have to be escaped except in constructs where they are part of the syntax. So we could punt on that.

[markusicu]

Updated proposal:

• Expand \d and \w from their current ASCII definitions to the UTS 18 defintions.
• Keep \b as before, but now consistent with “the new \w”

FYI: See also issue #37 for a small change in \s.

[RunDevelopment]

Quick question: Under the new proposed definition, is \d still a subset of \w?

[markusicu]

Quick question: Under the new proposed definition, is \d still a subset of \w?

Yes, of course.

In UTS 18

• POSIX digit = \d = \p{gc=Decimal_Number}
• POSIX word = \w = \p{alpha}\p{gc=Mark}\p{digit}\p{gc=Connector_Punctuation}\p{Join_Control}
  • Note how this includes \p{digit} == \p{gc=Decimal_Number}

[RunDevelopment]

Thank you @markusicu! The \p{digit} == \p{gc=Decimal_Number} wasn't clear to me.

[mathiasbynens]

I think we are in agreement. I don't want to expand this proposal to "all of UTS 18", but I do want to ensure that sufficient accommodations are made for future proposals to do so with minimal friction (ideally without introducing any more flags) and also to be clear about that in the description of this proposal (which seems to be transforming from "set notation" into something like "UTS18 compatibility").

And I'll further note that it is actually close to achieving UTS 18 Level 1, although I do see gaps relating to both RL1.4 (this issue) and RL1.6 (recognition of line boundaries, e.g. /./v.test("\u0085") would need to be false and /^\n/vm.test("\r\n") might need to be as well).

We should be careful in how we phrase this. UTS 18 requires recommends “loose matching” (ignoring casing, hyphens, spaces, and underscores in \p{…}), which was explicitly decided against for ECMAScript’s implementation of \p{…}. I’m on board with saying something like “increased UTS 18 Level 1 compatibility” as long as we don’t overpromise.