Allow integer suffixes starting with e.

Integers with arbitrary suffixes are allowed as inputs to proc macros. A
number of real-world crates use this capability in interesting ways, as
seen in #103872. For example:
- Suffixes representing units, such as `8bits`, `100px`, `20ns`, `30GB`
- CSS hex colours such as `#7CFC00` (LawnGreen)
- UUIDs, e.g. `785ada2c-f2d0-11fd-3839-b3104db0cb68`

The hex cases may be surprising.
- `#7CFC00` is tokenized as a `#` followed by a `7` integer with a
  `CFC00` suffix.
- `785ada2c` is tokenized as a `785` integer with an `ada2c` suffix.
- `f2d0` is tokenized as an identifier.
- `3839` is tokenized as an integer literal.

A proc macro will immediately stringify such tokens and reparse them
itself, and so won't care that the token types vary. All suffixes must
be consumed by the proc macro, of course; the only suffixes allowed
after macro expansion are the numeric ones like `u8`, `i32`, and `f64`.

Currently there is an annoying inconsistency in how integer literal
suffixes are handled, which is that no suffix starting with `e` is
allowed, because that it interpreted as a float literal with an
exponent. For example:
- Units: `1eV` and `1em`
- CSS colours: `#90EE90` (LightGreen)
- UUIDs: `785ada2c-f2d0-11ed-3839-b3104db0cb68`

In each case, a sequence of digits followed by an 'e' or 'E' followed by
a letter results in an "expected at least one digit in exponent" error.
This is an annoying inconsistency in general, and a problem in practice.
It's likely that some users haven't realized this inconsistency because
they've gotten lucky and never used a token with an 'e' that causes
problems. Other users *have* noticed; it's causing problems when
embedding DSLs into proc macros, as seen in #111615, where the CSS
colours case is causing problems for two different UI frameworks (Slint
and Makepad).

We can do better. This commit changes the lexer so that, when it hits a
possible exponent, it looks ahead and only produces an exponent if a
valid one is present. Otherwise, it produces a non-exponent form, which
may be a single token (e.g. `1eV`) or multiple tokens (e.g. `1e+a`).

Consequences of this:
- All the proc macro problem cases mentioned above are fixed.
- The "expected at least one digit in exponent" error is no longer
  possible. A few tests that only worked in the presence of that error
  have been removed.
- The lexer requires unbounded lookahead due to the presence of '_'
  chars in exponents. E.g. to distinguish `1e+_______3` (a float literal
  with exponent) from `1e+_______a` (previously invalid, but now the
  tokenised as `1e`, `+`, `_______a`).

This is a backwards compatible language change: all existing valid
programs will be treated in the same way, and some previously invalid
programs will become valid. The tokens chapter of the language reference
(https://doc.rust-lang.org/reference/tokens.html) will need changing to
account for this. In particular, the "Reserved forms similar to number
literals" section will need updating, and grammar rules involving the
SUFFIX_NO_E nonterminal will need adjusting.

Fixes #111615.

Author: nnethercote

compiler/rustc_lexer/src/cursor.rs

@@ -55,6 +55,12 @@ impl<'a> Cursor<'a> {
         iter.next().unwrap_or(EOF_CHAR)
     }
 
+    /// Allows peeking an unbounded number of symbols from the input stream
+    /// without consuming them.
+    pub(crate) fn all(&self) -> Chars<'a> {
+        self.chars.clone()
+    }
+
     /// Checks if there is nothing more to consume.
     pub(crate) fn is_eof(&self) -> bool {
         self.chars.as_str().is_empty()

compiler/rustc_lexer/src/lib.rs

@@ -177,7 +177,7 @@ pub enum LiteralKind {
     /// "12_u8", "0o100", "0b120i99", "1f32".
     Int { base: Base, empty_int: bool },
     /// "12.34f32", "1e3", but not "1f32".
-    Float { base: Base, empty_exponent: bool },
+    Float { base: Base },
     /// "'a'", "'\\'", "'''", "';"
     Char { terminated: bool },
     /// "b'a'", "b'\\'", "b'''", "b';"
@@ -578,6 +578,28 @@ impl Cursor<'_> {
     }
 
     fn number(&mut self, first_digit: char) -> LiteralKind {
+        // Scan ahead to determine if this is a valid exponent.
+        fn is_exponent(cursor: &mut Cursor<'_>) -> bool {
+            let mut iter = cursor.all();
+            let c = iter.next();
+            debug_assert!(matches!(c, Some('e' | 'E')));
+
+            // Exponent examples:     `e3`, `e+3`, `e_3`, `e-___3`,
+            // Non-exponent examples: `ea`, `e+a`, `e_a`, `e-___a`, `e_`
+            match iter.next() {
+                Some('0'..='9') => return true,
+                Some('+' | '-' | '_') => {}
+                _ => return false,
+            }
+            loop {
+                match iter.next() {
+                    Some('0'..='9') => return true,
+                    Some('_') => {}
+                    _ => return false,
+                }
+            }
+        }
+
         debug_assert!('0' <= self.prev() && self.prev() <= '9');
         let mut base = Base::Decimal;
         if first_digit == '0' {
@@ -628,23 +650,28 @@ impl Cursor<'_> {
                 // might have stuff after the ., and if it does, it needs to start
                 // with a number
                 self.bump();
-                let mut empty_exponent = false;
                 if self.first().is_digit(10) {
                     self.eat_decimal_digits();
                     match self.first() {
-                        'e' | 'E' => {
+                        // Scan ahead to decide if this is an exponent. If not,
+                        // it'll just be handled (later) as a suffix.
+                        'e' | 'E' if is_exponent(self) => {
                             self.bump();
-                            empty_exponent = !self.eat_float_exponent();
+                            let empty_exponent = !self.eat_float_exponent();
+                            debug_assert!(!empty_exponent);
                         }
                         _ => (),
                     }
                 }
-                Float { base, empty_exponent }
+                Float { base }
             }
-            'e' | 'E' => {
+            // Scan ahead to decide if this is an exponent. If not,
+            // it'll just be handled (later) as a suffix.
+            'e' | 'E' if is_exponent(self) => {
                 self.bump();
                 let empty_exponent = !self.eat_float_exponent();
-                Float { base, empty_exponent }
+                debug_assert!(!empty_exponent);
+                Float { base }
             }
             _ => Int { base, empty_int: false },
         }

compiler/rustc_lexer/src/tests.rs

@@ -283,9 +283,9 @@ br###"raw"###suffix
             Token { kind: Whitespace, len: 1 }
             Token { kind: Literal { kind: Int { base: Hexadecimal, empty_int: false }, suffix_start: 5 }, len: 5 }
             Token { kind: Whitespace, len: 1 }
-            Token { kind: Literal { kind: Float { base: Decimal, empty_exponent: false }, suffix_start: 3 }, len: 3 }
+            Token { kind: Literal { kind: Float { base: Decimal }, suffix_start: 3 }, len: 3 }
             Token { kind: Whitespace, len: 1 }
-            Token { kind: Literal { kind: Float { base: Decimal, empty_exponent: false }, suffix_start: 6 }, len: 6 }
+            Token { kind: Literal { kind: Float { base: Decimal }, suffix_start: 6 }, len: 6 }
             Token { kind: Whitespace, len: 1 }
             Token { kind: Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 }, len: 3 }
             Token { kind: Whitespace, len: 1 }

compiler/rustc_parse/messages.ftl

@@ -642,8 +642,6 @@ parse_no_digits_literal = no valid digits found for number
 
 parse_invalid_digit_literal = invalid digit for a base {$base} literal
 
-parse_empty_exponent_float = expected at least one digit in exponent
-
 parse_float_literal_unsupported_base = {$base} float literal is not supported
 
 parse_more_than_one_char = character literal may only contain one codepoint

compiler/rustc_parse/src/errors.rs

@@ -1762,13 +1762,6 @@ pub struct InvalidDigitLiteral {
     pub base: u32,
 }
 
-#[derive(Diagnostic)]
-#[diag(parse_empty_exponent_float)]
-pub struct EmptyExponentFloat {
-    #[primary_span]
-    pub span: Span,
-}
-
 #[derive(Diagnostic)]
 #[diag(parse_float_literal_unsupported_base)]
 pub struct FloatLiteralUnsupportedBase {

compiler/rustc_parse/src/lexer/mod.rs

@@ -479,11 +479,7 @@ impl<'a> StringReader<'a> {
                     (token::Integer, self.symbol_from_to(start, end))
                 }
             }
-            rustc_lexer::LiteralKind::Float { base, empty_exponent } => {
-                if empty_exponent {
-                    let span = self.mk_sp(start, self.pos);
-                    self.sess.emit_err(errors::EmptyExponentFloat { span });
-                }
+            rustc_lexer::LiteralKind::Float { base } => {
                 let base = match base {
                     Base::Hexadecimal => Some("hexadecimal"),
                     Base::Octal => Some("octal"),

tests/ui/consts/const-eval/issue-104390.rs

@@ -1,6 +1,5 @@
 const UNIVERSAL_GRAVITATIONAL_CONSTANT: f64 = 6.674e−11; // m³⋅kg⁻¹⋅s⁻²
-//~^ ERROR expected at least one digit in exponent
-//~| ERROR unknown start of token: \u{2212}
-//~| ERROR cannot subtract `{integer}` from `{float}`
+//~^ ERROR unknown start of token: \u{2212}
+//~| ERROR invalid suffix `e` for float literal
 
 fn main() {}

tests/ui/consts/const-eval/issue-104390.stderr

@@ -1,9 +1,3 @@
-error: expected at least one digit in exponent
-  --> $DIR/issue-49746-unicode-confusable-in-float-literal-expt.rs:1:47
-   |
-LL | const UNIVERSAL_GRAVITATIONAL_CONSTANT: f64 = 6.674e−11; // m³⋅kg⁻¹⋅s⁻²
-   |                                               ^^^^^^
-
 error: unknown start of token: \u{2212}
   --> $DIR/issue-49746-unicode-confusable-in-float-literal-expt.rs:1:53
    |
@@ -15,24 +9,13 @@ help: Unicode character '−' (Minus Sign) looks like '-' (Minus/Hyphen), but it
 LL | const UNIVERSAL_GRAVITATIONAL_CONSTANT: f64 = 6.674e-11; // m³⋅kg⁻¹⋅s⁻²
    |                                                     ~
 
-error[E0277]: cannot subtract `{integer}` from `{float}`
-  --> $DIR/issue-49746-unicode-confusable-in-float-literal-expt.rs:1:53
+error: invalid suffix `e` for float literal
+  --> $DIR/issue-49746-unicode-confusable-in-float-literal-expt.rs:1:47
    |
 LL | const UNIVERSAL_GRAVITATIONAL_CONSTANT: f64 = 6.674e−11; // m³⋅kg⁻¹⋅s⁻²
-   |                                                     ^ no implementation for `{float} - {integer}`
+   |                                               ^^^^^^ invalid suffix `e`
    |
-   = help: the trait `Sub<{integer}>` is not implemented for `{float}`
-   = help: the following other types implement trait `Sub<Rhs>`:
-             <&'a f32 as Sub<f32>>
-             <&'a f64 as Sub<f64>>
-             <&'a i128 as Sub<i128>>
-             <&'a i16 as Sub<i16>>
-             <&'a i32 as Sub<i32>>
-             <&'a i64 as Sub<i64>>
-             <&'a i8 as Sub<i8>>
-             <&'a isize as Sub<isize>>
-           and 48 others
+   = help: valid suffixes are `f32` and `f64`
 
-error: aborting due to 3 previous errors
+error: aborting due to 2 previous errors
 
-For more information about this error, try `rustc --explain E0277`.

tests/ui/consts/invalid-const-in-body.rs

@@ -11,7 +11,6 @@ fn main() {
     0x8.0e+9; //~ ERROR: hexadecimal float literal is not supported
     0x9.0e-9; //~ ERROR: hexadecimal float literal is not supported
     0o; //~ ERROR: no valid digits
-    1e+; //~ ERROR: expected at least one digit in exponent
     0x539.0; //~ ERROR: hexadecimal float literal is not supported
     9900000000000000000000000000999999999999999999999999999999;
     //~^ ERROR: integer literal is too large