Build with wasm exception handling #81

hoodmane · 2025-01-13T20:53:55Z

For now, we need to use a custom rust toolchain for this. Apparently all alternatives to this don't work. -Zbuild-std doesn't work with panic=abort (rust-lang/cargo#7359)
and my attempts to use a custom sysroot with either https://github.com/RalfJung/rustc-build-sysroot/ or https://github.com/DianaNites/cargo-sysroot/ seem to hit the same problem as with -Zbuild-std. Thus, I think the only reasonable way to go is to build the sysroot from the rust source directory. Perhaps we can eventually approach this by copying the lib/rustlib/wasm32-unknown-emscripten/lib/ folder out of the build of the rust compiler on top of a nightly install of the compiler. For now, there is the additional problem that we need this patch to fix unwind=abort:
rust-lang/rust#135450

I got my copy of the rust compiler by checking out this commit: hoodmane/rust@052ba16 two commits ahead of the rust main branch and running:

./x build --stage 2 --target x86_64-unknown-linux-gnu,wasm32-unknown-emscripten

For now, we need to use a custom rust toolchain for this. Apparently all alternatives to this don't work. -Zbuild-std doesn't work with panic=abort (rust-lang/cargo#7359) and my attempts to use a custom sysroot with either https://github.com/RalfJung/rustc-build-sysroot/ or https://github.com/DianaNites/cargo-sysroot/ seem to hit the same problem as with `-Zbuild-std`. Thus, I think the only reasonable way to go is to build the sysroot from the rust source directory. Perhaps we can eventually approach this by copying the `lib/rustlib/wasm32-unknown-emscripten/lib/` folder out of the build of the rust compiler on top of a nightly install of the compiler. For now, there is the additional problem that we need this patch to fix unwind=abort: rust-lang/rust#135450 I got my copy of the rust compiler by checking out this commit: hoodmane/rust@052ba16 two commits ahead of the rust main branch and running: ``` ./x build --stage 2 --target x86_64-unknown-linux-gnu,wasm32-unknown-emscripten ```

for more information, see https://pre-commit.ci

It's a bit hard to make the rust compiler play along, see the upstream PR for the rust difficulties pyodide/pyodide-build#81

ryanking13 · 2025-01-14T08:59:02Z

Thanks for dealing with rust issues. I think we need some more discussion on how to support custom Rust toolchains.

Instead of hard-coding the values in the code, it would be nice to make this configurable, using pyproject.toml and pyodide config

Also, adding a new subcommand that can install some build tools, including rust toolchain can be useful IMO. There was an open issue about it (pyodide/pyodide#3287).

hoodmane · 2025-01-14T09:06:54Z

This also only works for x86-linux. I think what I have to do is build the rust compiler from source, then upload the emscripten static libraries folder, then use rustup to install a nightly, then manually add our build-from-source static libraries.

hoodmane · 2025-01-14T09:09:14Z

But for that to work, we have to wait first for a nightly with rust-lang/rust#135450.

ryanking13 · 2025-01-15T07:14:36Z

Another question is, when can we remove the custom rust toolchain and use the upstream one? I am okay with maintaining the rust toolchain and custom sysroot for a while, but I don't think we can do that forever. So it would be nice to keep track of the blockers.

Could you please open an issue that lists the blockers and their progress that needs to be resolved for the wasm exception handling?

hoodmane · 2025-01-15T08:10:24Z

The plan is that we roll out wasm exception handling, check that there are no show stoppers, and then rust switches to using it by default after giving other people a chance to comment if it would cause them trouble.

hoodmane · 2025-01-15T08:22:01Z

Once Rust turns it on by default, we can in principle use the normal Rust toolchain, with the one concern being that it could have abi mismatches due to being compiled with a different Emscripten version than we link with.

See the relevant MCP here:
rust-lang/compiler-team#801

hoodmane · 2025-01-15T08:22:47Z

I'll open tracking issues for the left over problems when we merge this.

hoodmane · 2025-01-15T08:25:50Z

Also, @ryanking13 presumably we need to support both the pyodide_2025_0_wasm32 abi and the pyodide_2024_0_wasm32 at the same time? So I'll have to rework this a bit before merging to ensure this. Then we can merge this when the downstream pyodide/pyodide PR is ready and make a release and then all the externally built packages can be rebuilt and then they can be reenabled.

hoodmane · 2025-01-15T08:29:32Z

Though I don't know if we want to make the Emscripten updates at the same time or just have a 2025_0 ABI with just wasm eh and a 2025_1 ABI with wasm eh plus Emscripten updates. Then the released ABI will use 2025_1.

hoodmane · 2025-01-15T08:37:00Z

Currently the only extra commit I need on top of the Rust main branch to make the appropriate toolchain is this:
hoodmane/rust@81389fd

It's a bit hard to make the rust compiler play along, see the upstream PR for the rust difficulties pyodide/pyodide-build#81

…ive (#82) Split from #81, non functional.

ryanking13 · 2025-01-15T10:20:16Z

Though I don't know if we want to make the Emscripten updates at the same time or just have a 2025_0 ABI with just wasm eh and a 2025_1 ABI with wasm eh plus Emscripten updates. Then the released ABI will use 2025_1.

Maybe let's do both updates and call it 2025_0.

Rather than installing the rust toolchain in each build step that uses rust, do it once in buildall. Split off from #81.

pyodide_build/buildall.py

pyodide_build/pywasmcross.py

for more information, see https://pre-commit.ci

ryanking13

Thanks! Codewise looks okay to me. I left some minor comments.

ryanking13 · 2025-02-08T04:36:52Z

pyodide_build/config.py

@@ -228,7 +230,8 @@ def _get_make_environment_vars(self) -> Mapping[str, str]:
    "rustflags": "-C link-arg=-sSIDE_MODULE=2 -C link-arg=-sWASM_BIGINT -Z link-native-libraries=no",
    "cargo_build_target": "wasm32-unknown-emscripten",
    "cargo_target_wasm32_unknown_emscripten_linker": "emcc",
-    "rust_toolchain": "nightly-2024-01-29",
+    "rust_toolchain": "nightly-2025-01-15",


Let's mention the default rust toolchain update in the changelog.

This change isn't really needed to make this work. Maybe I can revert it here and put it by itself in a separate pr as a follow-up.

ryanking13 · 2025-02-08T04:38:32Z

pyodide_build/pywasmcross.py

@@ -77,6 +77,7 @@ class CrossCompileArgs(NamedTuple):
    target_install_dir: str = ""  # The path to the target Python installation
    pythoninclude: str = ""  # path to the cross-compiled Python include directory
    exports: Literal["whole_archive", "requested", "pyinit"] | list[str] = "pyinit"
+    abi: str = ""


Let's add a comment about this parameter.

ryanking13 · 2025-02-08T04:38:51Z

pyodide_build/pywasmcross.py

+    if abi > "2025" and arg in ["-lfreetype", "-lpng"]:
+        arg += "-wasm-sjlj"


Let's add a comment about the context of this if condition.

ryanking13 · 2025-02-08T04:41:52Z

pyodide_build/recipe/graph_builder.py

+    )
+    toolchain_root = Path(result.stdout).parents[1]
+    rustlib = toolchain_root / "lib/rustlib"
+    install_token = rustlib / "wasm32-unknown-emscripten_install-url.txt"


Is this file something you added manually in your custom toolchain? If so, let's add a comment about it.

ryanking13 · 2025-02-08T04:44:46Z

pyodide_build/pywasmcross.py

+            if arg.startswith("-l"):
+                arg = replay_genargs_handle_dashl(arg, used_libs, build_args.abi)


Not a strong opinion, but as build_args.ldflags is passed by the user, I think the user is responsible for setting the proper flags.

I have mixed feelings. I guess we can't fix static/dynamic libs not built with the python build system anyways so it's not that valuable. But it's pretty weird and annoying that linking -lpng successfully links but will break with opaque errors at runtime and the fix is to link with -lpng-wasm-sjlj. Maybe we can add some logic to audit wheel or somewhere appropriate that detects files linked against wrong libraries and explains as best we can what went wrong. Or people can just ask.

So yeah probably it makes sense to remove this.

Maybe we can add some logic to audit wheel or somewhere appropriate that detects files linked against wrong libraries and explains as best we can what went wrong.

Yeah, I think the problem is that we are not building libraries like libpng or libfreetype ourselves and use flags like -s USE_LIBPNG=1 instead. So, there exist multiple variants of those libraries, and it isn't easy to know what to use without looking into the emscripten code.

Ideally, I think it is better to drop all emscripten ports and build libraries ourselves. I think this would fix the issue with the Emscripten version and library version being fixed, and it would also fix the variant issues like the wasm exception.

BTW, I noticed that the library suffices are changed in the recent version of Emscripten: https://github.com/emscripten-core/emscripten/blob/242af002f8dee0366e86002fec187d447dcd26e5/tools/ports/libpng.py#L12-L16

hoodmane and others added 2 commits January 13, 2025 21:36

[pre-commit.ci] auto fixes from pre-commit.com hooks

a02dda7

for more information, see https://pre-commit.ci

hoodmane added a commit to hoodmane/pyodide that referenced this pull request Jan 13, 2025

Use webassembly exception handling

7847436

It's a bit hard to make the rust compiler play along, see the upstream PR for the rust difficulties pyodide/pyodide-build#81

hoodmane added a commit to hoodmane/pyodide that referenced this pull request Jan 13, 2025

Use webassembly exception handling

0245967

It's a bit hard to make the rust compiler play along, see the upstream PR for the rust difficulties pyodide/pyodide-build#81

hoodmane mentioned this pull request Jan 13, 2025

Use webassembly exception handling pyodide/pyodide#5320

Open

11 tasks

hoodmane added 4 commits January 13, 2025 22:10

Fix

3186d20

Fix again

8256e72

Only download rust toolchain once

5e46696

Fix

adda8a5

hoodmane added 3 commits January 14, 2025 14:22

Always run rustup toolchain link

b33d1c9

Add wasm-sjlj suffix to wasm-eh libs

f2d72e7

Update Emscripten toolchain version

3c3dc79

hoodmane added a commit to hoodmane/pyodide that referenced this pull request Jan 15, 2025

Use webassembly exception handling

275f76f

It's a bit hard to make the rust compiler play along, see the upstream PR for the rust difficulties pyodide/pyodide-build#81

hoodmane mentioned this pull request Jan 15, 2025

Move CrossBuildEnvManager._download to common.download_and_unpack_achive #82

Merged

hoodmane added a commit that referenced this pull request Jan 15, 2025

Move CrossBuildEnvManager._download to common.download_and_unpack_ach…

632c8e6

…ive (#82) Split from #81, non functional.

Merge branch 'main' into rust-wasm-eh

30517bd

hoodmane mentioned this pull request Jan 15, 2025

Move ensure_rust_toolchain to buildall #83

Merged

hoodmane added a commit that referenced this pull request Jan 15, 2025

Move ensure_rust_toolchain to buildall (#83)

7ebf9a2

Rather than installing the rust toolchain in each build step that uses rust, do it once in buildall. Split off from #81.

hoodmane force-pushed the rust-wasm-eh branch from 87813c5 to 607bd9e Compare January 15, 2025 11:30

Merge branch 'main' into rust-wasm-eh

0c88c0f

hoodmane force-pushed the rust-wasm-eh branch from 607bd9e to 0c88c0f Compare January 15, 2025 11:31

hoodmane added 3 commits January 15, 2025 12:32

Remove duplication of toolchain name

663e78e

Pass recipe ldflags through library rewriting logic as well

529a583

Add ignore for too many branches lint

a63c7b9

ryanking13 reviewed Jan 15, 2025

View reviewed changes

pyodide_build/buildall.py Outdated Show resolved Hide resolved

hoodmane commented Jan 15, 2025

View reviewed changes

pyodide_build/pywasmcross.py Outdated Show resolved Hide resolved

hoodmane and others added 9 commits January 15, 2025 20:52

Only install Emscripten target from url

b50d1c1

Fix syntax error

3b1734b

Move target url to config variable

c93aab1

Set default rust_toolchain to nightly-2025-01-15

036626d

[pre-commit.ci] auto fixes from pre-commit.com hooks

7bbdde2

for more information, see https://pre-commit.ci

Only change pywasmcross behavior for new abi

114e477

Merge branch 'main' into rust-wasm-eh

330e762

Merge branch 'main' into rust-wasm-eh

6a49a51

Don't append None to new_args

194c463

ryanking13 approved these changes Feb 8, 2025

View reviewed changes

hoodmane added 3 commits February 23, 2025 13:56

Merge branch 'main' into rust-wasm-eh

6fa5b9b

Merge branch 'main' into rust-wasm-eh

4cb167e

Merge branch 'main' into rust-wasm-eh

dc1c6a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build with wasm exception handling #81

Build with wasm exception handling #81

hoodmane commented Jan 13, 2025

ryanking13 commented Jan 14, 2025

hoodmane commented Jan 14, 2025

hoodmane commented Jan 14, 2025

ryanking13 commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

ryanking13 commented Jan 15, 2025

ryanking13 left a comment

ryanking13 Feb 8, 2025

hoodmane Feb 8, 2025

ryanking13 Feb 8, 2025

ryanking13 Feb 8, 2025

ryanking13 Feb 8, 2025

ryanking13 Feb 8, 2025

hoodmane Feb 8, 2025

ryanking13 Feb 8, 2025

ryanking13 Feb 8, 2025

		if abi > "2025" and arg in ["-lfreetype", "-lpng"]:
		arg += "-wasm-sjlj"

		if arg.startswith("-l"):
		arg = replay_genargs_handle_dashl(arg, used_libs, build_args.abi)

Build with wasm exception handling #81

Are you sure you want to change the base?

Build with wasm exception handling #81

Conversation

hoodmane commented Jan 13, 2025

ryanking13 commented Jan 14, 2025

hoodmane commented Jan 14, 2025

hoodmane commented Jan 14, 2025

ryanking13 commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

hoodmane commented Jan 15, 2025

ryanking13 commented Jan 15, 2025

ryanking13 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment