feat: expose LibTomMath modular arithmetic as primitives

[ggreif]

Summary

Expose LibTomMath modular arithmetic operations as Motoko primitives:

• Prim.intAddMod(a, b, m) — (a + b) mod m via mp_addmod
• Prim.intSubMod(a, b, m) — (a - b) mod m via mp_submod
• Prim.intMulMod(a, b, m) — (a * b) mod m via mp_mulmod
• Prim.intPowMod(base, exp, m) — base^exp mod m via mp_exptmod

All operate on Int (arbitrary precision). intPowMod is particularly useful for cryptographic applications (RSA, Diffie-Hellman, etc.) where modular exponentiation with large integers is needed.

Changes

Layer	File	What
RTS/C	rts/Makefile	Add libtommath source files + bindgen allowlist
RTS/Rust	rts/motoko-rts/src/bigint.rs	Wrapper functions calling mp_*
Compiler	compile_enhanced.ml, compile_classical.ml	Wasm imports + OtherPrim codegen
Interpreter	mo_values/prim.ml	OCaml reference implementation
Prelude	src/prelude/prim.mo	Public Prim declarations
Test	test/run/modular-arith.mo	Happy-path tests incl. Fermat's little theorem

Test plan

• nix build .#tests.unit passes
• nix build .#tests.release passes
• CI green

🤖 Generated with Claude Code

TODO

• Amend Changelog.md

[ggreif]

cc @timohanke — this was your request 🎯

[github-actions]

Comparing from 9412bb5 to 8992c79:
In terms of gas, no changes are observed in 5 tests.
In terms of size, 5 tests regressed and the mean change is +3.9%.

[ggreif]

@timohanke — your modular arithmetic primitives are ready! 🎉

Prim.intAddMod, intSubMod, intMulMod, intPowMod backed by LibTomMath's mp_addmod, mp_submod, mp_mulmod, mp_exptmod. Build artifacts attached to the CI run.

Getting here was a bit of a Grothendieck experience — rather than cracking the nut with a hammer, we let the rising sea of understanding slowly submerge the problem. From missing libtommath transitive dependencies, through WASI allocator incompatibilities, to discovering that Motoko's compact BigNum representation requires proper unboxing for ternary operations — each wrong theory peeled back a layer until the fix emerged naturally from the architecture.

Build artifacts (macOS, Ubuntu, ARM) are available from the CI run linked above.

E.g. macOS ARM64 is here: https://github.com/caffeinelabs/motoko/actions/runs/24552026361/artifacts/6490288624

[timohanke]

Thanks @ggreif . This is awesome. Looking forward to try it out.

We should also consider exposing mp_sqr. And additionally, make the compiler detect x ** 2 or x * x in the source code and call it. (Off-topic for the feature, which is about modular arithmetic and mp_sqr isn't modular, but on-topic application-wise because it is useful for the same cryptographic applications.)

Further down the road we can also consider exposing

mp_montgomery_setup
mp_montgomery_calc_normalization
mp_montgomery_reduce

but let's see first how the ones already added perform in practice.

[ggreif]

We should also consider exposing mp_sqr. And additionally, make the compiler detect x ** 2 or x * x in the source code and call it. (Off-topic for the feature, which is about modular arithmetic and mp_sqr isn't modular, but on-topic application-wise because it is useful for the same cryptographic applications.)

Good point. We could do
We sure could:

• spot both args VarE and same -> call mp_sqr
• spot literal ** exponent 2 and call mp_sqr
• popcnt exponent and if =1 do iterated mp_sqr
  Both of the latter are possible, but does TomMath already do the second? Still it would be a win, because ** 2 is internally 0b100 and ctz gives 2 then iterate mp_sqr only once. For 0b1000 (a.k.a. 4) iterate twice. Etc. Maybe worth it. When heap pointer popcnt = 1 will never trigger.

[timohanke]

Also mp_invmod should be exposed as well. That fits right into this PR.