Skip to content

Remove internal tokenization during parse#435

Merged
blakeembrey merged 2 commits intomasterfrom
be/squash-inner-tokens
Apr 1, 2026
Merged

Remove internal tokenization during parse#435
blakeembrey merged 2 commits intomasterfrom
be/squash-inner-tokens

Conversation

@blakeembrey
Copy link
Copy Markdown
Member

Removes the internal LexToken created during parse, reducing the memory and performance overhead of generating the objects.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 1, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.74%. Comparing base (9a78879) to head (2215cd0).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #435   +/-   ##
=======================================
  Coverage   99.74%   99.74%           
=======================================
  Files           1        1           
  Lines         392      393    +1     
  Branches      150      156    +6     
=======================================
+ Hits          391      392    +1     
  Misses          1        1           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blakeembrey
Copy link
Copy Markdown
Member Author

Perf on main:

 ✓ src/index.bench.ts > parse 680ms
     name                   hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · parsing paths  407,427.20  0.0021  1.8551  0.0025  0.0023  0.0042  0.0057  0.0404  ±1.01%   203714

Perf on PR:

 ✓ src/index.bench.ts > parse 696ms
     name                   hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · parsing paths  515,135.45  0.0016  8.4582  0.0019  0.0018  0.0036  0.0052  0.0168  ±3.35%   257568

Base automatically changed from be/error-backslash to master April 1, 2026 19:30
@blakeembrey blakeembrey force-pushed the be/squash-inner-tokens branch from 9e22ed1 to 77b5e11 Compare April 1, 2026 19:31
@blakeembrey
Copy link
Copy Markdown
Member Author

Also a bundle size reduction from 1.98 kB to 1.93 kB

@blakeembrey blakeembrey merged commit 5844988 into master Apr 1, 2026
8 checks passed
@blakeembrey blakeembrey deleted the be/squash-inner-tokens branch April 1, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant