Self-contained editor html for extensions

[jpolitz]

This is a substantial build update for the VScode extension that:

1. Fixes Extension does not work in our GitLab Web IDE jpolitz/pyret-parley-vscode#21
2. Makes the generated vsix extension file smaller and load faster

The root causes are:

• The JS/CSS resources are served by Gitlab/OpenVSX as text/plain. This is a known open issue, so they won't load as e.g. script tags and styles.
• There appears to be some kind of ~15MB cap on files served from there, as well, and we were bundling the un-gzipped compiler blob which is too large (our mistake, shouldn't be doing that)

The overall idea is to add a new build target that inlines a bunch of stuff into editor.html to make it more of a standalone HTML file. There's some subtlety here, because there is:

1. Inlining of CSS/JS that needs to happen first, and be robust to templating after, because there is...
2. Templating that happens dynamically when the web view starts up that we need to keep. That means it's running per-tab config in VScode long after we've shipped the built files. This is for images and fonts, the (now-gzipped) compiler blob, and the current mechanism for HASH params as a template variable. It may be possible to inline all of these eventually but at the moment "inlining the gzipped source into HTML" feels like a bit much.

One issue is that we've been using mustache for (2), but mustache can find and mess with {{ in minified JS... bad news.

So this has several fixes:

1. A bespoke inliner; this is the thing I've taken a while convincing myself is actually fine.
2. A more hardcoded replace-pattern-variables pass, not using mustache which becomes scary after arbitrary inlining can match its tokens.

So let's talk about the inliner. It's regexy and my first instinct is that it's horrible:

 html = html.replace(
    new RegExp('<script\\b[^>]*\\bsrc="' + BASE + '/js/([^"]+)"[^>]*>\\s*</script>', "gi"),
    (tag, rel) =>
      "<script>\n" + escapeForScript("js/" + rel, readAsset("js/" + rel)) + "\n</script>"
  );

There are two horribles here:

1. Not using a parser for the HTML
2. Not using a library that escapes things properly

The argument for doing the hacky thing for 1 is that if you actually use a parser, find the tags via search and re-emit, you get re-emitted HTML, which is hard to diff, doesn't necessarily look like the input, and so on. A really good one is going to be a really big npm dependency, as well. A rabbit-hole of figuring out which one to use, etc. And HTML parsers are super permissive on bad input, anyway, so it's not clear the “no surprises” intent of a parser will come through.

I think for 2 there's a better simple counterproposal, which is to use data: urls in script tags with base64 encoded code. This works, and avoids any and all escaping issues. We may want to do it. However:

• It can mess with error reporting and getting good traces (e.g. this error came from data:fkh439nafhaouif340 as opposed to a specific script tag with inlined code with a clearly-settable breakpoint)
• It “counts” differently for content security policy, and we have run into CSP issues aplenty when trying to get all our infrastructure to nest correctly inside VMT iframes and desktop VScode and web VScode and now web-Gitlab. If we end up in contexts that don't like that kind of inline data URL script (which let's be honest looks like obfuscation – a giant JS blob in a data URL), ugh, we fail in a fun new context that we didn't predict.

This is also all build-side; we control all the code being templated in.

Also-also, there are npm packages for doing inlining of source... if you look at their code they use regexes, too. That's not to say “hey existing code sucks so we should, too”, more that if we had said “oh let's use an off-the-shelf”, that's what we're getting.

• web-resource-inliner (1-2M downloads/wk): very similar stuff (https://github.com/jrit/web-resource-inliner/blob/v8.0.0/src/html.js#L184)
• inline-source (40k downloads/wk): regex-based internally too (https://github.com/popeindustries/inline-source/blob/8.0.3/src/js.js#L22-L23)

And to be clear we have to do something here if we want to work on Gitlab's web editor (which we should). There is no way to ship these as separate JS files and load them as tags in Gitlab's web editor. At best we could load them all with fetch of the plaintext and eval them (but that changes semantics, not quite script tags), and it's not clear that works for the css as well. So given how well this works (and there are tests for it), I'm OK with it as a build-for-vscode.

Note that this doesn't do anything to the normal CPO build; the changes there are minimal flag-outs for the "are we doing the fancy gzip-in-JS thing”.

Also, this is tested and it works on Gitlab (I did an out-of-band build of the VSIX and deployed it).

Other stuff:

• Includes infrastructure for a fake server that mis-produces MIME types the same way as Gitlab (we can add more to a “hostile” server here if we want more tests)
• This is a useful path to getting the embed build to be more self-packaged, potentially without needing so much of a whole-directory download
• Note that this does include a trick to get the gzipped file to load when it can't be content-headered or sniffed as gzipped JS: piping through a DecompressionStream (https://github.com/jpolitz/pyret-lang/pull/8/changes#diff-6f16fedf22976ab5c1109c9066b53bd0dacda3a803245e4ac0f52b824512f53dR1497). This is very cool and I don't think controversial at all given constraints (I mean, you could also base64 in the whole gzipped thing and inline it, but that breaks all kinds of nice caching, inflates the compression, etc)

[blerner]

(I left a message via the github app, but it got evaporated into the ether)

I haven't read through this yet, but two quick clarification questions:

1. If we get rid of mustache (which I'm fine with), could we revise what the {{template expression}} looks like so that it's less likely to be mis-detected? Also could we revise the regex detection of <script src="whatever" /> to again avoid mis-detection?
2. More broadly, instead of doing a regex replace, could we/should we build a manifest of things-to-be-inserted, and just iterate over that dictionary instead of iterating over regex matches? That might make it much easier to know if any insertion failed to be detected at all... and avoids any notion of open-ended stringly manipulating the code.

[jpolitz]

(I made two PRs, this is the real one)

Good suggestions. This doesn't ditch mustache on its own (the server still uses it to render), but that's a good thing to do as well. I will stage a separate PR (on top of this branch) to do that, and we can separately decide which order to merge in. That will remove the "mustache is over-eager on {{}} kinds of patterns" and we can will only match on very obvious __PYRET__-ish patterns we write in ourselves.

For matching on the source files themselves, I think editor.html is short enough, and controlled-by-us-enough, that it is its own manifest: <script src="{{&BASE_URL}}/js/…"> is a pretty stark pattern to match. There is some checking happening that the files we inline are part of the make dependencies for running the inline script (so they don't get stale and it's a positive dep set).