Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilation #14340

josevalim · 2025-03-18T13:11:05Z

The implementation uses a TCP port to distribute the work. We chose to not use the Erlang distribution as that may affect the user app running afterwards.

When compiling Livebook on 10 core machine:

livebook$ rm -rf _build/; time mix deps.compile
real	0m25.865s
user	0m48.625s
sys	0m10.504s

livebook$ rm -rf _build/; time MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4  mix deps.compile
real	0m12.337s
user	0m0.417s
sys	0m0.533s

In this case, increasing more than 4 did not decrease compilation times (perhaps expected due to the cost of starting new instances and that compilation itself is already parallel within the same dependency). Closes #14200.

liamwhite · 2025-03-18T18:36:41Z

I will test this out when I get a chance later. Is it possible that this behavior could be made an argument/flag to deps.compile instead of requiring an environment variable? This would make it a little easier to discover / integrate into documentation, I suspect. (similar to the way makeon its own doesn't thread, but make -j 4 would use 4 threads)

josevalim · 2025-03-18T18:47:09Z

I decided to go with with an environment variable because:

Easier to integrate in CI
Different projects will use a different configuration
Different machines will use a different configuration

For example, I will likely set MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 system-wide on my machine, but that would probably make things worse or be not enough on others machine.

I think make -j is a great example because, afaik, no one sets it by default because you should not assume. There are usually meta tools that would configure it for you. But mix deps.compile is not wrapped by anything, everyone calls it directly.

josevalim · 2025-03-18T18:52:34Z

Regarding discovery, we will document it on both mix deps.compile and Mix, so the amount of discovery should be the same as the flag (i.e. it will show up on mix help docs.compile).

liamwhite · 2025-03-18T23:11:27Z

on 6c3f308, I get this:

root@73db75e3f3db:~/philomena# rm -rf _build/; time mix deps.compile
real	0m46.813s
user	2m55.029s
sys	0m42.692s

root@73db75e3f3db:~/philomena# rm -rf _build/; time MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 mix deps.compile
...
4> == Compilation error in file lib/yaml_elixir/records.ex ==
4> ** (ArgumentError) lib file yamerl/include/internal/yamerl_constr.hrl could not be found
4>     (elixir 1.19.0-dev) lib/record/extractor.ex:45: Record.Extractor.from_lib_file/1
4>     (elixir 1.19.0-dev) lib/record/extractor.ex:22: Record.Extractor.from_or_from_lib_file/1
4>     (elixir 1.19.0-dev) lib/record/extractor.ex:9: Record.Extractor.extract/2
4>     lib/yaml_elixir/records.ex:6: (module)
4> could not compile dependency :yaml_elixir, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile yaml_elixir --force", update it with "mix deps.update yaml_elixir" or clean it with "mix deps.clean yaml_elixir"
** (Mix) ERROR! mix deps.partition #Port<0.13> closed unexpectedly

real	0m9.588s
user	0m36.237s
sys	0m14.622s

(might not be related to this change)

josevalim · 2025-03-18T23:14:23Z

Can you run a mix deps.get and try again? If it persists, could you share your dependencies?

liamwhite · 2025-03-18T23:16:02Z

Yeah, persists after mix deps.get && mix deps.compile yaml_elixir (without the env set). Dependencies here, lock here

Also persists after cleaning and refetching.

root@73db75e3f3db:~/philomena# mix deps.compile yaml_elixir
==> yaml_elixir
Compiling 6 files (.ex)

== Compilation error in file lib/yaml_elixir/records.ex ==
** (ArgumentError) lib file yamerl/include/internal/yamerl_constr.hrl could not be found
    (elixir 1.19.0-dev) lib/record/extractor.ex:45: Record.Extractor.from_lib_file/1
    (elixir 1.19.0-dev) lib/record/extractor.ex:22: Record.Extractor.from_or_from_lib_file/1
    (elixir 1.19.0-dev) lib/record/extractor.ex:9: Record.Extractor.extract/2
    lib/yaml_elixir/records.ex:6: (module)
could not compile dependency :yaml_elixir, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile yaml_elixir --force", update it with "mix deps.update yaml_elixir" or clean it with "mix deps.clean yaml_elixir"

liamwhite · 2025-03-18T23:22:14Z

If I compile yamerl first it will fix itself. So it probably is related to this change.
Also experiencing the bug on credo -> credo_naming (compilation error in credo_naming due to not-compiled credo dep)

josevalim · 2025-03-19T08:09:37Z

@liamwhite apologies, a commit I pushed yesterday just before you tried introduced a race condition. We should be all good now.

lukaszsamson · 2025-03-19T08:39:19Z

@josevalim I tried the branch on the biggest repos I have. In first on I see nice improvement up to ~4 partitions (tested on M2 mac):

MIX_OS_DEPS_COMPILE_PARTITION_COUNT=1 mix deps.compile  105.53s user 30.62s system 190% cpu 1:11.55 total

MIX_OS_DEPS_COMPILE_PARTITION_COUNT=2 mix deps.compile  0.50s user 0.77s system 2% cpu 47.529 total

MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 mix deps.compile  0.51s user 0.78s system 3% cpu 40.816 total

MIX_OS_DEPS_COMPILE_PARTITION_COUNT=6 mix deps.compile  0.52s user 0.85s system 3% cpu 41.959 total

The second one however fails to compile:

2> ==> blake2_elixir
2> Unchecked dependencies for environment prod:
2> * elixir_make (Hex package)
2>   the dependency build is outdated, please run "MIX_ENV=prod mix deps.compile"
2> could not compile dependency :blake2_elixir, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile blake2_elixir --force", update it with "mix deps.update blake2_elixir" or clean it with "mix deps.clean blake2_elixir"
2> ** (Mix) Can't continue due to errors on dependencies
** (Mix) ERROR! mix deps.partition #Port<0.8> closed unexpectedly

The error is reproducible. Running mix deps.get does not resolve the issue. Note I'm not building with MIX_ENV=prod so I don't know where this comes from. The app builds normally with 1 partition. I have this versions locked

"blake2_elixir": {:hex, :blake2_elixir, "0.8.1", "93913497a79faf97c46e9140dd33d85d8cad0e93ae0039f2503c50204dd94c25", [:make, :mix], [{:elixir_make, "~> 0.5", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "3f37b4fb1d1ae9f118f8d1e0358a0f80b422d52af99a23ddcaba5b0b48b33e5d"},

"elixir_make": {:hex, :elixir_make, "0.9.0", "6484b3cd8c0cee58f09f05ecaf1a140a8c97670671a6a0e7ab4dc326c3109726", [:mix], [], "hexpm", "db23d4fd8b757462ad02f8aa73431a426fe6671c80b200d9710caf3d1dd0ffdb"},

josevalim · 2025-03-19T09:28:04Z

@lukaszsamson which commit did you try? I pushed a fix early morning today.

lukaszsamson · 2025-03-19T11:10:48Z

@lukaszsamson which commit did you try? I pushed a fix early morning today.

0d7b673

josevalim · 2025-03-19T11:24:35Z

@lukaszsamson I see. Can you please try to provide a way to reproduce it? It may be that a mix.exs with the mix.lock is enough. Thank you.

lukaszsamson · 2025-03-19T11:57:55Z

@josevalim It's not reproducible in 100% tries. I got the crash in 4 out of 5. My real app is 2x more umbrella apps underneath so maybe that contributes.

git clone [email protected]:lukaszsamson/mix_partitions_error_repro.git
cd mix_partitions_error_repro
mix deps.get
rm -rf _build/; time MIX_OS_DEPS_COMPILE_PARTITION_COUNT=5  mix deps.compile

4> ==> finch
4> Compiling 14 files (.ex)
5> Generated monero_address app
5> ==> websock
5> Compiling 1 file (.ex)
2> Generated elixir_make app
2> ==> bcrypt_elixir
5> Generated websock app
2> mkdir -p "/Users/lukaszsamson/bug_repro/mix_partitions_error_repro/_build/dev/lib/bcrypt_elixir/priv"
2> cc -g -O3 -Wall -Wno-format-truncation -I"/Users/lukaszsamson/.asdf/installs/erlang/27.3/erts-15.2.3/include" -Ic_src -fPIC -shared -dynamiclib -undefined dynamic_lookup c_src/bcrypt_nif.c c_src/blowfish.c -o "/Users/lukaszsamson/bug_repro/mix_partitions_error_repro/_build/dev/lib/bcrypt_elixir/priv/bcrypt_nif.so"
5> ==> elixir_make
5> Compiling 8 files (.ex)
2> warning: unknown warning option '-Wno-format-truncation' [-Wunknown-warning-option]
5> Generated elixir_make app
5> ==> blake2_elixir
5> Unchecked dependencies for environment prod:
5> * elixir_make (Hex package)
5>   the dependency build is outdated, please run "MIX_ENV=prod mix deps.compile"
5> could not compile dependency :blake2_elixir, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile blake2_elixir --force", update it with "mix deps.update blake2_elixir" or clean it with "mix deps.clean blake2_elixir"
5> ** (Mix) Can't continue due to errors on dependencies
** (Mix) ERROR! mix deps.partition #Port<0.14> closed unexpectedly

Note that both bcrypt_elixir and blake2_elixir depend on elixir_make . In the log it seems two processes 2 and 5 try to compile the same dep

liamwhite · 2025-03-19T12:22:28Z

With the above fixes, it does work for me:

root@73db75e3f3db:~/philomena# rm -rf _build/; time MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 mix deps.compile
real	0m21.767s
user	1m11.990s
sys	0m26.240s

Despite the reduced overhead, I have tried various partition counts and not been able to get a compilation time faster than my own solution in #14200 (comment) . This was made using Elixir 1.17 and compiled in about 16.04 seconds using all 32 processors. However, it isn't apples-to-apples, because the compile lock added in 1.18 now prevents that solution from working, so I haven't rebenchmarked it.

liamwhite · 2025-03-19T12:30:12Z

The performance seems to degrade extremely badly and start using ridiculous amounts of memory at high concurrency levels (I tested at 32 and completely ran out of system memory despite having 64GB). That also was not an issue in my own solution.

josevalim · 2025-03-19T12:57:58Z

@liamwhite you can set %{"MIX_OS_CONCURRENCY_LOCK" => "false"} if you want to try your solution in main.

For this solution though, I don't recommend starting the same amount of instances because you will eventually run out of memory indeed.

josevalim · 2025-03-19T13:05:36Z

Btw, our solutions are somewhat similar, we both distribute the graph, but I distribute it over tcp, so we don't need to start multiple instances, and you start multiple instances.

I tried your approach locally on Livebook and it took 14.8s and with MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 it takes 12.2s (note I have a 10 core machine and I don't get a better speed up than 4).

v0idpwn · 2025-03-19T13:51:06Z

Tested here in some real projects with quite large number of dependencies. The results below are only for dependencies compilation, as the application compilation would skew the results.

PC Specs

13th Gen Intel i7-1355U (12)
Linux pop-os 6.9.3-76060903-generic #202405300957~~1732141768~~22.04~f2697e1 SMP PREEMPT_DYNAMIC Wed N x86_64 x86_64 x86_64 GNU/Linux
16GB ram

Project 1: 180 deps

No flag

real	4m4.457s
user	12m55.313s
sys	0m51.495s

4 OS processes

real	1m53.868s
user	0m3.663s 
sys	0m0.682s

8 OS processes

real	1m47.999s
user	0m3.415s
sys	0m0.757s

Project 2: 220 deps

No flag

real	10m14.741s
user	30m5.695s
sys	1m53.419s

4 OS processes

real	2m55.542s
user	0m4.431s
sys	0m0.882s

8 OS processes

real	2m55.879s
user	0m4.951s
sys	0m1.149s

Project 3: 238 deps (umbrella, some of the dependencies are big)

No flag

real	15m38.342s
user	39m40.431s
sys	2m26.431s

4 OS processes

real	5m3.738s
user	0m5.350s
sys	0m1.175s

8 OS processes

real	5m16.848s
user	0m5.886s
sys	0m1.248s

Note: for all the runs, I had MIX_DEBUG enabled. Results vary a bit on retries, but not too much.

This is huge for us. Thank you for making it!

v0idpwn · 2025-03-19T14:05:20Z

Potential improvement:

Starting the workers async (right now it starts the first - wait for tcp connection, start second - wait for tcp...) might yield significant benefits for smaller projects

josevalim · 2025-03-19T15:39:09Z

Parallel booting of partitions has been added, thanks for the suggestion!

josevalim · 2025-03-19T17:06:16Z

@lukaszsamson thank you. Here is a one-liner that reproduces it:

MIX_INSTALL_FORCE=1 MIX_OS_DEPS_COMPILE_PARTITION_COUNT=5 elixir -e "Mix.install [:bcrypt_elixir, :blake2_elixir]"

Blake2 has a wrong application definition: riverrun/blake2_elixir#2

However, keep in mind Erlang's crypto ships with blake2, so you don't need a dep:

:crypto.hash(:blake2b, contents)
:crypto.hash(:blake2s, contents)

lukaszsamson · 2025-03-19T17:20:24Z

Thank's @josevalim, I'll try your branch

However, keep in mind Erlang's crypto ships with blake2, so you don't need a dep:

I do. The :crypto one wraps blake2 in an abstraction that does not support all params. In my case blake2 is used like this:

Blake2b.hash(input, <<>>, outlen, <<>>, personal)

lukaszsamson · 2025-03-19T17:41:39Z

Working fine with your patch @josevalim. Nice win with build time cut in 3 here. Above 4 partitions I see diminishing returns.

MIX_OS_DEPS_COMPILE_PARTITION_COUNT=1 mix deps.compile  100.61s user 43.10s system 220% cpu 1:05.25 total MIX_OS_DEPS_COMPILE_PARTITION_COUNT=2 mix deps.compile  0.60s user 0.98s system 3% cpu 40.426 total
MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 mix deps.compile  0.59s user 0.99s system 4% cpu 33.875 total
MIX_OS_DEPS_COMPILE_PARTITION_COUNT=6 mix deps.compile  0.58s user 0.96s system 4% cpu 33.805 total

Blake2 has a wrong application definition

Maybe a good idea to add a warning or fail compilation

lib/mix/lib/mix/tasks/deps.compile.ex

Co-authored-by: felipe stival <[email protected]>

liamwhite · 2025-03-19T21:33:16Z

Rebenchmarked. With 8 partitions it's so close that this is fine (I'm not going to notice ~1 second on a timescale of 17 seconds)

root@73db75e3f3db:~/philomena# rm -rf _build; time mix deps.compile
real	0m46.196s
user	2m50.101s
sys	0m42.520s

root@73db75e3f3db:~/philomena# rm -rf _build; time MIX_OS_DEPS_COMPILE_PARTITION_COUNT=4 mix deps.compile
real	0m19.116s
user	1m22.878s
sys	0m27.843s

root@73db75e3f3db:~/philomena# rm -rf _build; time MIX_OS_DEPS_COMPILE_PARTITION_COUNT=8 mix deps.compile
real	0m17.631s
user	3m36.030s
sys	1m17.611s

josevalim · 2025-03-19T21:44:59Z

@liamwhite btw, your script runs fine if you start IEx with MIX_OS_CONCURRENCY_LOCK=0, in case you want to compare both on latest.

josevalim · 2025-03-24T14:33:05Z

💚 💙 💜 💛 ❤️

sabiwara · 2025-03-24T23:36:31Z

lib/mix/lib/mix/tasks/deps.partition.ex

+
+  defp server(socket, deps, count, force?) do
+    elixir =
+      System.get_env("MIX_OS_DEPS_COMPILE_PARTITION_ELIXIR_EXECUTABLE") ||


Should we document this one?
I was trying it out and it failed because my locally built mix was trying to use the asdf installed elixir.
Or I wondered, could we solve it by introducing something like Mix.elixir_interpreter/0?

I decided to not expose it for now and wait if someone will have a use case for it. If they do, then we can just document it.

…on (elixir-lang#14340)

v0idpwn · 2025-03-25T01:00:21Z

Any chance this can get backported to 1.18.x?

josevalim · 2025-03-25T09:00:15Z

@v0idpwn no plans at the moment but please do use main and give us feedback. I already found a bug yesterday in that it did not work on mix test, so more testing is definitely necessary.

halfdan · 2025-03-31T12:35:54Z

Looks like I hit a race condition in our app:

2> ==> jason
2> Compiling 10 files (.ex)
3> ===> Analyzing applications...
3> ===> Compiling gproc
4> ===> Analyzing applications...
1> ===> Analyzing applications...
4> ===> Compiling unicode_util_compat
1> ===> Compiling telemetry
2> Generated jason app
3> ==> exmoji
3> Compiling 3 files (.ex)
3> 
3> == Compilation error in file lib/exmoji.ex ==
3> ** (UndefinedFunctionError) function Jason.decode!/2 is undefined (module Jason is not available). Make sure the module name is correct and has been specified in full (or that an alias has been defined)
3>     Jason.decode!("[{\"name\":\"COPYRIGHT SIGN\",\"unified\":\"00A9\",\"variations\":[],\"docomo\":\"E731\",\"au\":\"E558\",\"softbank\":\"E24E\",\"google\":\"FEB29\",\"image\":\"00a9.png\",\"sheet_x\":0,\"sheet_y\":0,\"short_name\":\"copyright\",\"short_names\":[\"copyright\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":false},{\"name\":\"REGISTERED SIGN\",\"unified\":\"00AE\",\"variations\":[],\"docomo\":\"E736\",\"au\":\"E559\",\"softbank\":\"E24F\",\"google\":\"FEB2D\",\"image\":\"00ae.png\",\"sheet_x\":0,\"sheet_y\":1,\"short_name\":\"registered\",\"short_names\":[\"registered\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":false},{\"name\":\"DOUBLE EXCLAMATION MARK\",\"unified\":\"203C\",\"variations\":[\"203C-FE0F\"],\"docomo\":\"E704\",\"au\":\"EB30\",\"softbank\":null,\"google\":\"FEB06\",\"image\":\"203c.png\",\"sheet_x\":0,\"sheet_y\":2,\"short_name\":\"bangbang\",\"short_names\":[\"bangbang\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"EXCLAMATION QUESTION MARK\",\"unified\":\"2049\",\"variations\":[\"2049-FE0F\"],\"docomo\":\"E703\",\"au\":\"EB2F\",\"softbank\":null,\"google\":\"FEB05\",\"image\":\"2049.png\",\"sheet_x\":0,\"sheet_y\":3,\"short_name\":\"interrobang\",\"short_names\":[\"interrobang\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"TRADE MARK SIGN\",\"unified\":\"2122\",\"variations\":[],\"docomo\":\"E732\",\"au\":\"E54E\",\"softbank\":\"E537\",\"google\":\"FEB2A\",\"image\":\"2122.png\",\"sheet_x\":0,\"sheet_y\":4,\"short_name\":\"tm\",\"short_names\":[\"tm\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"INFORMATION SOURCE\",\"unified\":\"2139\",\"variations\":[\"2139-FE0F\"],\"docomo\":null,\"au\":\"E533\",\"softbank\":null,\"google\":\"FEB47\",\"image\":\"2139.png\",\"sheet_x\":0,\"sheet_y\":5,\"short_name\":\"information_source\",\"short_names\":[\"information_source\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"LEFT RIGHT ARROW\",\"unified\":\"2194\",\"variations\":[\"2194-FE0F\"],\"docomo\":\"E73C\",\"au\":\"EB7A\",\"softbank\":null,\"google\":\"FEAF6\",\"image\":\"2194.png\",\"sheet_x\":0,\"sheet_y\":6,\"short_name\":\"left_right_arrow\",\"short_names\":[\"left_right_arrow\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"UP DOWN ARROW\",\"unified\":\"2195\",\"variations\":[\"2195-FE0F\"],\"docomo\":\"E73D\",\"au\":\"EB7B\",\"softbank\":null,\"google\":\"FEAF7\",\"image\":\"2195.png\",\"sheet_x\":0,\"sheet_y\":7,\"short_name\":\"arrow_up_down\",\"short_names\":[\"arrow_up_down\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"NORTH WEST ARROW\",\"unified\":\"2196\",\"variations\":[\"2196-FE0F\"],\"docomo\":\"E697\",\"au\":\"E54C\",\"softbank\":\"E237\",\"google\":\"FEAF2\",\"image\":\"2196.png\",\"sheet_x\":0,\"sheet_y\":8,\"short_name\":\"arrow_upper_left\",\"short_names\":[\"arrow_upper_left\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"NORTH EAST ARROW\",\"unified\":\"2197\",\"variations\":[\"2197-FE0F\"],\"docomo\":\"E678\",\"au\":\"E555\",\"softbank\":\"E236\",\"google\":\"FEAF0\",\"image\":\"2197.png\",\"sheet_x\":0,\"sheet_y\":9,\"short_name\":\"arrow_upper_right\",\"short_names\":[\"arrow_upper_right\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"SOUTH EAST ARROW\",\"unified\":\"2198\",\"variations\":[\"2198-FE0F\"],\"docomo\":\"E696\",\"au\":\"E54D\",\"softbank\":\"E238\",\"google\":\"FEAF1\",\"image\":\"2198.png\",\"sheet_x\":0,\"sheet_y\":10,\"short_name\":\"arrow_lower_right\",\"short_names\":[\"arrow_lower_right\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"SOUTH WEST ARROW\",\"unified\":\"2199\",\"variations\":[\"2199-FE0F\"],\"docomo\":\"E6A5\",\"au\":\"E556\",\"softbank\":\"E239\",\"google\":\"FEAF3\",\"image\":\"2199.png\",\"sheet_x\":0,\"sheet_y\":11,\"short_name\":\"arrow_lower_left\",\"short_names\":[\"arrow_lower_left\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"LEFTWARDS ARROW WITH HOOK\",\"unified\":\"21A9\",\"variations\":[\"21A9-FE0F\"],\"docomo\":\"E6DA\",\"au\":\"E55D\",\"softbank\":null,\"google\":\"FEB83\",\"image\":\"21a9.png\",\"sheet_x\":0,\"sheet_y\":12,\"short_name\":\"leftwards_arrow_with_hook\",\"short_names\":[\"leftwards_arrow_with_hook\"],\"text\":null,\"apple_img\":true,\"hangouts_img\":true,\"twitter_img\":true},{\"name\":\"RIGHTWARDS ARROW WITH HOOK\",\"unified\":\"21AA\",\"variations\":[\"21AA-FE0F\"],\"docomo\":null,\"au\":\"E55C\",\"" <> ..., [keys: :atoms])
3>     lib/exmoji.ex:23: (module)
3> could not compile dependency :exmoji, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile exmoji --force", update it with "mix deps.update exmoji" or clean it with "mix deps.clean exmoji"
** (Mix) mix deps.partition #Port<0.11> closed unexpectedly

I can get around it by reducing concurrency to 2 but it happens consistently with 4.

josevalim · 2025-03-31T12:37:55Z

@halfdan which commit are you using?

josevalim · 2025-03-31T12:38:50Z

Nevermind, fixed here: mroth/exmoji#23

josevalim added 4 commits March 18, 2025 14:04

Add MIX_DEPS_COMPILE_PARALLEL_COUNT for parallel app compilation

478ec8a

Close resources

dd8263a

Add license

e3fa67a

Rename env var

9708e77

josevalim changed the title ~~Add MIX_DEPS_COMPILE_PARALLEL_COUNT for parallel app compilation~~ Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilation Mar 18, 2025

josevalim added 2 commits March 18, 2025 23:16

Rename file

91e94ce

Only wait for dependency if it is in list

6c3f308

Address race condition

0d7b673

josevalim added 3 commits March 19, 2025 11:24

Docs

a616cc4

Add support for Mix.install/2 in deps.partition

24e3c2e

More docs

8cedb93

Boot partitions in parallel

6016328

Fix client/socket

e3b5d90

v0idpwn reviewed Mar 19, 2025

View reviewed changes

lib/mix/lib/mix/tasks/deps.compile.ex Outdated Show resolved Hide resolved

v0idpwn reviewed Mar 19, 2025

View reviewed changes

lib/mix/lib/mix/tasks/deps.compile.ex Outdated Show resolved Hide resolved

Apply suggestions from code review

284fa5d

Co-authored-by: felipe stival <[email protected]>

josevalim added 2 commits March 24, 2025 13:47

Tests

8216f0e

Explicitly set env var

4f7d0bf

josevalim force-pushed the jv-deps-compile-digraph branch from 7e61f87 to 4f7d0bf Compare March 24, 2025 13:24

josevalim added 3 commits March 24, 2025 14:27

Make it public

670ee0a

Remove parens

a0c5603

Use explicit loopback

f3f8049

josevalim merged commit 3dc3f4e into main Mar 24, 2025
20 checks passed

josevalim deleted the jv-deps-compile-digraph branch March 24, 2025 14:32

sabiwara reviewed Mar 24, 2025

View reviewed changes

v0idpwn pushed a commit to v0idpwn/elixir that referenced this pull request Mar 25, 2025

Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilati…

972a900

…on (elixir-lang#14340)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilation #14340

Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilation #14340

josevalim commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 18, 2025 •

edited

Loading

josevalim commented Mar 18, 2025 •

edited

Loading

josevalim commented Mar 18, 2025

liamwhite commented Mar 18, 2025 •

edited

Loading

josevalim commented Mar 18, 2025

liamwhite commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 18, 2025 •

edited

Loading

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

liamwhite commented Mar 19, 2025 •

edited

Loading

liamwhite commented Mar 19, 2025

josevalim commented Mar 19, 2025

josevalim commented Mar 19, 2025 •

edited

Loading

v0idpwn commented Mar 19, 2025 •

edited

Loading

v0idpwn commented Mar 19, 2025 •

edited

Loading

josevalim commented Mar 19, 2025

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

liamwhite commented Mar 19, 2025

josevalim commented Mar 19, 2025

josevalim commented Mar 24, 2025

sabiwara Mar 24, 2025

josevalim Mar 25, 2025

v0idpwn commented Mar 25, 2025

josevalim commented Mar 25, 2025

halfdan commented Mar 31, 2025

josevalim commented Mar 31, 2025

josevalim commented Mar 31, 2025

Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilation #14340

Add MIX_OS_DEPS_COMPILE_PARTITION_COUNT for concurrent deps compilation #14340

Conversation

josevalim commented Mar 18, 2025 • edited Loading

liamwhite commented Mar 18, 2025 • edited Loading

josevalim commented Mar 18, 2025 • edited Loading

josevalim commented Mar 18, 2025

liamwhite commented Mar 18, 2025 • edited Loading

josevalim commented Mar 18, 2025

liamwhite commented Mar 18, 2025 • edited Loading

liamwhite commented Mar 18, 2025 • edited Loading

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

liamwhite commented Mar 19, 2025 • edited Loading

liamwhite commented Mar 19, 2025

josevalim commented Mar 19, 2025

josevalim commented Mar 19, 2025 • edited Loading

v0idpwn commented Mar 19, 2025 • edited Loading

PC Specs

Project 1: 180 deps

No flag

4 OS processes

8 OS processes

Project 2: 220 deps

No flag

4 OS processes

8 OS processes

Project 3: 238 deps (umbrella, some of the dependencies are big)

No flag

4 OS processes

8 OS processes

v0idpwn commented Mar 19, 2025 • edited Loading

josevalim commented Mar 19, 2025

josevalim commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

lukaszsamson commented Mar 19, 2025

liamwhite commented Mar 19, 2025

josevalim commented Mar 19, 2025

josevalim commented Mar 24, 2025

sabiwara Mar 24, 2025

Choose a reason for hiding this comment

josevalim Mar 25, 2025

Choose a reason for hiding this comment

v0idpwn commented Mar 25, 2025

josevalim commented Mar 25, 2025

halfdan commented Mar 31, 2025

josevalim commented Mar 31, 2025

josevalim commented Mar 31, 2025

josevalim commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 18, 2025 •

edited

Loading

josevalim commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 18, 2025 •

edited

Loading

liamwhite commented Mar 19, 2025 •

edited

Loading

josevalim commented Mar 19, 2025 •

edited

Loading

v0idpwn commented Mar 19, 2025 •

edited

Loading

v0idpwn commented Mar 19, 2025 •

edited

Loading