Launching Enso programs instantly

[JaroslavTulach]

Enso is dynamically compiled language built on top of JVM - e.g. GraalVM - and its Truffle framework as such programs written in Enso can run pretty fast - once the dynamic compiler does its job. This kind of peak performance is not a subject of this discussion.

However it may take some time until an Enso program gets up to full speed. Currently that time is longer than it should be. The subject of this discussion is to highlight, reference existing issues that are supposed to launch Enso instantly and put them into proper context ready for a discussion.

[JaroslavTulach]

Eliminate JVM Startup Overhead

Enso interpreter is written in Java using the Truffle framework and it takes up to dozen of seconds to get up to full speed. JVM is just in time compiled as well. As such the Enso interpreter is its first interpreted by the JVM too. Only then it gets compiled as regular Java code, but that is still not enough. We need to compile the Enso interpreter code via Truffle partial evaluation and that usually happens only after few seconds of executing the program.

Native Image to the Rescue

"Truffle typical" solution to the slow initialization of the interpreter is to use Native Image and compile the interpreter into native executable or library. The startup is then greatly improved - usually to few milliseconds. All such compiled interpreters - Graal.js, Graal Python - start instantly. The same trick works for Enso as well:

• Generate native image for engine-runner #3638
• and a continuous integration test CI: Enable sanity test for native Runner #8724

after necessary changes we can execute simple Enso program computing factorial in less then 200ms! That is still 190ms slower than it should be, but it is good enough for now.

Problems

The #3638 has proven the Enso interpreter overhead caused by JVM can be avoided. However, there is another problem. Other languages, when calling into operating system, they usually call C libraries. Enso is different. Enso uses Java and calls to the JVM to make system calls! As such we have to mitigate the JVM cost also for Enso libraries!

[JaroslavTulach]

Mitigate JVM overhead for (Standard) Libraries

Enso libraries are using Java as a way to deal with operating system. As such even a call to IO.println needs to initialize (a part of) JVM. Even when native image mitigates overhead of JVM for the interpreter, we still pay the additional price as soon as we use any Enso library which calls into Java:

• Get Espresso ready for use in Enso/Java Interop #4877
• Espresso support with most recent Enso@GraalVM #6966
• Using ENSO_JAVA env variable to turn on the Espresso support #7951
• Updating Espresso support & docs to match GraalVM for JDK21 #8641

As can be seen from the above issues, Java on Truffle a.k.a. Espresso was seen as the original solution to this problem - the Espresso solution has been prototyped and since #8641 it is even part of our continuous integration. However it hasn't really speed startup up. Still the Espresso solution may be needed for other reasons - especially cloud security and scalability of individual tenants inside of a single cloud VM.

Native Image to the Rescue

An alternative solution has been proposed by @4e6:

• Importing (some of) Standard.Base works from NI runner #9866
• to do that we had to Register all polyglot java import classes for refective access #9997

but everything seems to be working and parts of the Standard.Base library can be loaded this way. The speed up is also good: just one third of the original time. Looks like the compilation of Standard libraries via native image is a way to go. Let's load whole Standand.Base and beyond:

• Run whole Standard.Base in Native Image ./runner #10126

But of course, there are problems...

Problems

The set of compiled-in libraries is fixed at compilation of the native image. There is no way to load additional libraries and their Java classes dynamically. What shall happen when a 3rd party library needs to load additional classes?

[JaroslavTulach]

3rd Party Library Usage

If we compile some Standard libraries into Enso launcher native executable, what will happen when a completely different library or another version of compiled-in library is used? What will happen when the user tries to invoke

polyglot java import java.lang.System

main = System.exit 10

e.g. when the user code tries to make calls into JVM by itself?

Better Launcher

The simplest solution to this problem is to implement better Enso runner. It is common among launchers of Truffle languages to support --jvm flag. By default such launchers run in native mode, but if one wants to use JVM polyglot capabilities, one can pass in --jvm and the laucher re-launches itself in JVM mode. Takes longer, but provides more capabilities. Moreover JVM mode usually provides better peak performance. Thus let's:

• Support for --jvm option in Enso runner #10374
• Avoid name duplication of the enso universal launcher #10476
• Integrate our launcher with Truffle launcher #5298 - probably not needed

and let's make sure the new launcher supports switching between native and JVM mode:

• when a class cannot be loaded, it tells the user to try the --jvm option
• when analyzing a project, it could detect 3rd party library and switch to JVM mode automatically

Of course, such a double re-launching requires the launch to be fast - that's another topic.

Using Espresso

Espresso is a Truffle interpreter of JVM and can execute Java bytecode dynamically even inside of the native executable. Since #8641 the Espresso mode is tested as part of our CI and it is know to work to some extent. We shall keep Espresso working, but until its experimental status is changed, it is unlikely we'd put Espresso into production.

Using Isolates

• Speed Startup by 33% with the use of GraalVM Polyglot Isolates #9171

The drawback of this solution is that it works only for Oracle GraalVM, not for GraalVM Community Edition as isolates aren't part of the open source edition. On the other hand, it can be a nice bonus for those who buy Enso Enterprise. The idea is to launch Enso interpreter as native library (e.g. "isolate), but use regular HotSpot JVM to load classes dynamically. Prototype works. Needs proper support in the launcher - e.g. #5298

[JaroslavTulach]

Speed Startup Really Up

Eliminating cost of JVM is one thing, but using the CPU properly, not doing needless computations, etc. is also important. For example there is an analysis that Import/Export takes more than a second on a native image based executable. That's awfully a lot. We need to improve our code and address these situations:

• Analyze and improve the performance of Import/export resolution #9236
• Store FramePointer in the IR and cache it #10129
• Preload library caches into native-image heap to speed start up #10044
• Improve perf of Graph.Scope.scopeFor hotspot #9620
• Analyze and improve the performance of AliasAnalysis compiler pass #9235
• stop over importing as discussed in Offer "import library hint" when needed #12019

Ideally we should be able to execute a simple program with native image in less that few hundred milliseconds.

[JaroslavTulach]

What's needed to productize native image Enso launcher?

Majority of the tasks listed in this dicussion have already been finished. Time to come up with final a set of tasks to productize this native image based launcher.

Release

The release shall only contain the native image based launcher. Users can use --jvm option to launch Enso in JVM mode. By default Enso runs in native image mode.

• Run all tests with native image launcher (with -ea enabled)
  • Execute test/Geo_Tests in enso binary launcher #11394
  • work on Standard.Image done by Including opencv native libraries as resources in enso runner binary #11807
  • work on Standard.Database tests - Run test/Table_Tests in native image mode #12006
  • include Standard.Microsoft
  • include Standard.Snowflake
  • include Standard.AWS
  • less concern about google APIs
  • less concern about tableau
  • we must include Standard.Visualization somehow: Review Standard.Visualization and remove dependencies which are not Base/Table/Database #12008
• Native Language Server integration with PM #11880
• Change build script to not include shell based launchers (e.g. no enso.bat in release) - Build script switch to generate either shell xor binary for bin/enso #12014
• Include native image launcher (without -ea option) in the release - Build script switch to generate either shell xor binary for bin/enso #12014
• Don't include JavaScript and Python in native image launcher
  • getting ready: Get ready for python and js being optional when running test/Base_Tests #10975
  • really remove JavaScript & Python: Avoid JavaScript and Python in Enso native launcher #11279
  • And once more: Remove JavaScript from bin/enso executable #12018
• Change ensoup run to invoke enso executable - motivated by problems discovered in ensoup broken due to 2024.5.1.rc3 #11794
• Better error when unknown language is used with foreign in binary mode #12104
• Improve error messages when polyglot java import cannot be resolved in native image mode - suggest to use --jvm option
• detect the need to run with --jvm before executing and switch automatically

Development

Building native image takes time and is a threat to edit/compile/test development cycle.

• Don't build native image launcher for developers by default, use shell based launchers - Build script switch to generate either shell xor binary for bin/enso #12014
• Allow building native image launcher at explicit request - Build script switch to generate either shell xor binary for bin/enso #12014
• Detect native image launcher is outdated to avoid developers wasting their time finding out why their changes aren't applied - Detect outdated launcher #10779

[JaroslavTulach]

@hubertp has managed to integrate #12006 - congratulation! What are the remaining pending issues to address from the previous list? From the bottom to the top:

• run all tests with native image launcher
• Enable Standard.AWS and Standard.Microsoft in native image #12156
• Add Snowflake to native-image #12193
• Add Standard.Tableau to native-image #12204

Anything else? Following issues don't really seem essential when we can execute all our libraries with native executable:

• CI should run Standard Library Tests in both JVM and Native modes #12123
• include Standard.Visualization somehow: Review Standard.Visualization and remove dependencies which are not Base/Table/Database #12008
• Improve error messages when polyglot java import cannot be resolved in native image mode - suggest to use --jvm option
• detect the need to run with --jvm before executing and switch automatically

Let's turn the native launcher on (and let's see what happens)!

[JaroslavTulach]

• Enable NI with IDE #12287 gives users an opt-in via an environment variable to use the native enso executable with the IDE when they feel dare enough

[JaroslavTulach]

• the time has come to enable instant launching of Enso programs
• Enabling native launchers for the release #12296