-
Notifications
You must be signed in to change notification settings - Fork 951
Is it possible to install Rust in a tmpfs with less than 1GB of RAM? #3234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I think the answer to this is no. The stable x86_64 toolchain with just cargo/clippy/rust-std/rustc/rustfmt is 550M. To unpack that we need to hold a chunk in RAM, though we can be pretty efficient, and perhaps could be more, you'll still need space for your kernel and userspace themselves, plus linkers and other standard development tooling. From personal recollection a minimal debian chroot with development tools was ~300MB. So call that 850M. And thats before you invoke the compiler and ask it to do anything. If rustup fails to perform the install, I think we can consider that a bug - we should work in quite low mem environments where rustc will be mem-mapped so the footprint at runtime is much lower - but I am skeptical that a live system will be able to actually build anything with so little RAM. |
Thanks for the direct answer. What I don't understand about the Rust install is that We begin with the recommended installation
Then, for example,
I see Right now I'm just trying to build this https://github.com/denoland/roll-your-own-javascript-runtime. I have 1.6GB available. During research over the past few weeks I've read that Can Rust folks somehow make this possible? Or, should I abandon the concept of building a minimal executable using Rust given the conditions? Can you kindly explain why there appear to be duplicate |
Alternative solutions include instructions for how to build a GitHub repository to a release using GitHub actions; or providing a means to build an executable using Rust online playground, if either are technically possible or feasible. As it stands I don't see any way to install Rust per official instructions and compile an executable on a tmpfs. |
Many folk use github actions to build and release their rust code in CI/CD pipelines - its super common and well supported. Those actions machines have 7GB or more RAM. https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources I think using such a size machine your use case of 'do it all in RAM' will work just fine. The duplication isn't duplication - you are seeing the proxies https://rust-lang.github.io/rustup/concepts/proxies.html If rustup fails to install a toolchain with 512MB of RAM not shared with the filesystem, I think we can consider it a bug, since thats what entry level raspberry pi's have. Being actually usable at that point when all your build artifacts are going to subtract usable ram - thats a whole other question. Building the roll-your-own-js-runtime repository you linked creates 2G of files:
That would be fine if you were not using a tmpfs. |
I don't see how that would be remotely possible. The only way to determine if that is possible is to test and verify.
A complete guide of exactly how to do that will be useful. Using the recommended install script ain't working.
For this project I don't think I need The idea is to build a minimal JavaScript runtime that does exactly one (1) thing - without carrying around linting, formatting, and testing in the executable, in this case,
Then this won't work anyway. Nonetheless I think you should exclude the possibility you do have a bug re that RaspberryPi scenario. I don't see how that would be possible to successfully achieve - just given the roll-your-own JavaScript runtime case, not even the full deno repository. I suggest you verify in the field. In the mean time I have no idea how to set up GitHub Actions to build a release executable from a fork. I think it's reasonable that Rust folks create a MVCE of how to do that since we can't use the standard recommended approach on tmfs and do anything afterwards. |
So, two things: I'm not sure where
For this project here, the scope is 'can the installer run' - and for that, I'm asserting that either using an 8GB machine with tmpfs, or a 1GB machine with persistent storage - such as an RPi - should work, but 1GB without persistent storage won't. if someone encounters either of the cases we expect to work not working, will happily work with them to figure it out. Installing on a 1GB without persistent storage isn't something I think we can or should try to make work given the size of development artifacts. Second thing: I think that a lot of this conversation really isn't about rustup so much as what you should expect from Rust, or how to approach things in Rust, and for that I'm going to recommend https://users.rust-lang.org/ the general community site. Designing the best way to get deno installed into a tmpfs based machine is exactly the sort of thing that warrants discussion, rather than a bug. |
I begin with a 64GB USB drive with a live OS written on it. After downloading Firefox Nightly and Chromium Dev Channel, For source code that relies on If the girth of Rust really is necessary and useful then we should be able to upload a GitHub repository to the online Rust playground and the Rust compiler should spit out the executable after building. Then I don't have to be concerned with running out of disk space just to install a compiler to build a minimal JavaScript runtime. Do you see the issue with this scenario? I am trying to build a minimal application, yet to do so using Rust have to use over 1GB for the compiler. Somehow the author of QuickJS managed to write a compiler that accepts JavaScript input and spits out C in far less than 1GB. SO, there has to be a way to get rid of some of the bloat involved. I've noticed over the years the Chromium executable steadily increasing in size. That is the trend. So eventually I can see Rust compilation chain growing until we need more and more disk space to compile a minimal program. If you Rust folks really can't build a program without over 1GB of source code, then I don't think your RaspberryPi scenario is realistic. And Frankly, not worth it to fight Rust along the way. I can start with QuickJS source code at around 5MB and build up rather than starting with Rust at 1GB just to build anything. |
@rbtcollins BTW, thanks for your time and effort in this matter. I just thought it was appropriate to ask the source (Rust experts) for a solution to this case. If no viable solution is possible, and no compromise can be reached then as you and others have stated it is not possible to install Rust in minimal-type systems and function thereafter. I don't think Rust users in the field can come up with a solution when the canonical source can't. If that is it, no compromise, no way to get rid of liniting, formatting, testing, etc. from even the "minimal" profile, then just close this issue and I will have to abandon trying to build with Rust and move on to explore other solutions. I am merely exhausting my remedies. |
If you're building a minimal application, you should try and find a way not to need to compile Rust code from source. Compiled Rust programs don't need the Rust compiler present, and should be considerably smaller. This fact is why your use case has not really been catered to — most folks who have size-constrained applications don't need to compile Rust code at runtime. If you must compile Rust code, and reducing compiler size is required, you could |
This seems off topic for this repository's issue board. |
I deleted the context. I'm seeking the minimal means to compile Rust source code. Looking at what is downloaded for recommended install, are linting, formatting, and debugging really needed? Is there a way for you to create a minimal compiler, less MB than the "minimal" profile, so I can test writing and compiling Rust in my environment? Or, is that just not possible? |
Before
I had over 1GB of space on the tmfs, after 250MB.
At 189MB space remaining right now and the compilation is not completed. |
I really think you would be better served by asking on one of the user help forums. This issue tracker is really not designed to provide that kind of support. |
I did. More than once; e.g., https://www.reddit.com/r/rust/comments/10tojxt/building_minimal_deno_executable_for_a_local/ Then I had no choice but to ask the canonical source. I havn't observed thus far enthusiasm for somehow reducing the at least 1GB currently required to install Rust tool chain for the use case I described and similarly situated environments. I think the answer is "No". |
Note that whether or not tests, linting, etc. are present in the compiler has no impact on the size of the executables the toolchain produces... To get smaller executables, you should start by adjust the compiler settings, not changing the items installed with the compiler. But this is definitely the wrong forum for this discussion still. |
This is the only forum for my question. Everything else is just hearsay and sidebar. You Rust experts are the canonical source. All roads lead back here. I don't need those items. I would be carrying them around for nothing. Yes, eliminating those items must reduce executable file size. |
They are not present in the output executable, they're just part of the build toolchain. |
The That is the philosophy I'm talking about. Those should be optional features not even in the build toolchain. If I want to import them into my build then I can. All of that is just more disk space used on systems running on RAM and temp file system. We don't get to the part of compiler flags because your compiler toolchain is too immense for me to even install. Instead of defending your current practices you could instead focus on trying to come up with a solution for this issue. I don't see that happening though. You suggest I ask users in the field how to builf with Rust. If the canaonical source can't craft a solution there is a problem with that proposed solution A wild goose chase. Come up with a minimal official Rust compiler. |
If this is true, it has nothing to do with the rust toolchain. Rust does not bake formatting, linting, or testing into output executables. There is not even an optional feature to do so, let alone one you need to disable, so if deno does this it has implemented that feature entirely separately of the work of this or any other rust-lang repository.
Your complaints seem very confused. The best analogy I have is that you're complaining that because the factory is large, it can only build large cars. That is not the case. The factory (the rust toolchain) may be bloated from your perspective, but this fact is entirely unrelated to the size of what it produces. If you wield Rust correctly, you can get very small executables, even though the compiler itself is large. See https://github.com/johnthagen/min-sized-rust. |
Those are not installed in the minimal profile. The proxies are still present but they do not actually take up any space because they are all hard-links to the same executable. It sounds like you are incorrectly counting these towards the size of the installed rust toolchain.
This comparison makes no sense. When you translate JavaScript to C, the heavy lifting is being done by the C compiler to take that C code and compile it into an actual executable. If you want an apples-to-apples comparison, you should compare the size of the Rust toolchain to the combined size of QuickJS + C compiler (clang or GCC) + size of installed C header files and libraries. If you actually do that comparison, you will find that there is not so much difference... |
I ain't complaining. I am sharing with Rust folks that I can't even install Rust tool chain without running out of disk space . The fact that Rust tool chain is at least 1GB on Linux is not debatable. I am looking a minimal Rust tool chain. That is the purpose of this issue. Then i can testing removing what I don't need from various repository source code before I build. I never get that far because Rust eats up available space on temporary file system running in RAM. I think repositories constructed with Rust tend to bake in what Rust bakes in to the tool chain. Users in the filed should be able to import Rust linting, formatting, debugging executables without those executables being in the default cargo directory - unless they are actually needed to build with Rust. Then minimal versions of those executables should be provided as an option for the use case of low disk space devices. That is reasonable. Let's say something around a half to a third of what the current Rust tool chain is - if anything can be removed and the tool chain still builds correctly. I havn'e gotten a square answer to that question. That is the purpose of this issue. |
Those are installed when the default installation script is run. Then downloaded again when What I am trying to determine is how to not download all of those "proxies" when
There is Tiny C Compiler, et al. I am asking if there is anything that can be done to reduce the 1GB size of the Rust tool chain, during or after the suggested installation process. I don't think LLVM is 1GB. Even though the distribution is shipped with GCC I have also installed On a difference machine with more disk space I wouldn't have this issue. Right now I am experimenting with compiling on minimal disk space devices running on only RAM. |
This is untrue. You said you are coming to the source because we are experts on Rust, and we are telling you this is not happening. There is not even an option to do this, and for users to do it manually would be very difficult. The size of deno is not coming from rustfmt/clippy/etc being baked into the deno executable, it is coming from something else. |
The philodophy of the Rust tool chain is reflected in This issue ain't about Deno. Deni is just an example of an executable built with Rust, reflecting the features baked in to Rust tool chain that cannot easily be removed - no option to do so - even if you don't need or use them. This issue is about the size of Rust tool chain. I can't even get to the point to remove those items from the build to prove you and me right or wrong re unneeded "features" being baked in to executables compiled with Rust. Does that make sense to you? https://deno.land/[email protected]/tools
|
Fresh live OS running on RAM. 3.1GB after Install and run 1.9GB after running Rust install script with default profile. 1.4GB after downloading Chromium Dev Channel. 1GB after downloading release zip file from https://github.com/denoland/roll-your-own-javascript-runtime.
After installing
25MB after above. There is no way I can install |
I removed some language packages to get 266MB
43MB left on device.
4.7MB left on device. |
Run
|
This is simply not true.
As already mentioned, these executable do not take up any space as they are simply hard-links to rustup itself, so whatever you are measuring is wrong.
The answer is still "No". Nobody on this thread is arguing that you can install Rust and compile anything useful using only 1GB of RAM and no persistent storage... What we are saying is that your assumptions about why that is the case are wrong. When you install the minimal profile there is no additional "testing" or "linting" stuff that is taking up space and could be removed. These are the things that do take up space:
2.1) The compiler and it's dynamic libraries. These take about ~238MB on my machine, although this includes the debug files (I'm on windows so PDB files). If I exclude debug symbols, it's ~139MB. If I were to "strip" those binaries removing all symbols, I imagine the size would go down even more. 2.2) The rust standard library. This takes about ~200MB on my machine, again including debug files (although there are fewer savings to be made by deleting those). Having said all that, you are likely to still be out of luck: you will probably still run out of space if you try to build anything non-trivial, as intermediate build artefacts can easily take up a few hundred MBs on their own (symbols take a lot of space...) In summary, you can save some space by |
I'm going to lock this conversation, it has become far off-topic for rustup.
|
Problem you are trying to solve
Install Rust in a live OS running on RAM where disk space is 1GB.
Solution you'd like
Minimal Rust installation for the ability to compile GitHub repositories written in Rust.
Notes
#3125 (comment)
The text was updated successfully, but these errors were encountered: