include_str fails with large files

[benaryorg]

I just tried to include_str!() a 14g file.
It failed with:

failure with 16g file

% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
fatal runtime error: memory allocation failed
error: Could not compile `foobar`.

---

Same error with everything down to five gigabytes (which would fit in my ram thrice).
Then I thought, hey, 32-bit and ran the following:

failure with 2^32+1 file

% truncate -s 4294967297 file # 2^32 + 1
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
fatal runtime error: memory allocation failed
error: Could not compile `foobar`.

To learn more, run the command again with --verbose.

failure with 2^32 file

% truncate -s 4294967296 file # 2 ^ 32
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
fatal runtime error: memory allocation failed
error: Could not compile `foobar`.

To learn more, run the command again with --verbose.

failure with 2^32-1 file

% truncate -s 4294967295 file # 2^32 - 1
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
thread 'main' panicked at 'assertion failed: line_len == 0 || ((*lines)[line_len - 1] < pos)', libsyntax_pos/lib.rs:969:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.27.1 (5f2b325f6 2018-07-07) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `foobar`.

To learn more, run the command again with --verbose.

failure with 2^32-1 file & RUST_BACKTRACE

% RUST_BACKTRACE=1 cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
thread 'main' panicked at 'assertion failed: line_len == 0 || ((*lines)[line_len - 1] < pos)', libsyntax_pos/lib.rs:969:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: rustc::util::common::panic_hook
   5: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:467
   6: std::panicking::begin_panic
   7: syntax_pos::FileMap::next_line
   8: syntax::codemap::CodeMap::new_filemap_and_lines
   9: syntax::ext::source_util::expand_include_str
  10: <F as syntax::ext::base::TTMacroExpander>::expand
  11: syntax::ext::expand::MacroExpander::expand_invoc
  12: syntax::ext::expand::MacroExpander::expand
  13: syntax::ext::expand::MacroExpander::expand_crate
  14: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
  15: rustc::util::common::time
  16: rustc_driver::driver::phase_2_configure_and_expand
  17: rustc_driver::driver::compile_input
  18: rustc_driver::run_compiler_impl
  19: <scoped_tls::ScopedKey<T>>::set
  20: syntax::with_globals
  21: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  22: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  23: rustc_driver::run
  24: rustc_driver::main
  25: std::rt::lang_start::{{closure}}
  26: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  27: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  28: std::rt::lang_start_internal
             at libstd/panicking.rs:289
             at libstd/panic.rs:374
             at libstd/rt.rs:58
  29: main
  30: __libc_start_main
  31: <unknown>
query stack during panic:
end of query stack

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.27.1 (5f2b325f6 2018-07-07) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `foobar`.

To learn more, run the command again with --verbose.

---

Content of the file was generated by base64 /dev/urandom | pv -Ss 16g > file (and subsequently truncated).
A file created using truncate yields only the error of ≥2³² as described above though, so the error might have to do with include_str!.

---

This is happening on:

tested rustc stable

% rustc --version --verbose
rustc 1.27.1 (5f2b325f6 2018-07-07)
binary: rustc
commit-hash: 5f2b325f64ed6caa7179f3e04913db437656ec7e
commit-date: 2018-07-07
host: x86_64-unknown-linux-gnu
release: 1.27.1
LLVM version: 6.0

tested rustc nightly

% rustup run nightly rustc --version --verbose
rustc 1.29.0-nightly (254f8796b 2018-07-13)
binary: rustc
commit-hash: 254f8796b729810846e2b97620032ecaf103db33
commit-date: 2018-07-13
host: x86_64-unknown-linux-gnu
release: 1.29.0-nightly
LLVM version: 7.0

---

Used code:

fn main()
{
	let _res = include_str!("../file");
}

---

Nightly yields a litte more info:

nightly with backtrace and empty (truncated) file

% RUST_BACKTRACE=1 rustup run nightly cargo build --verbose
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
     Running `rustc --crate-name foobar src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=0edabc38d37db739 -C extra-filename=-0edabc38d37db739 --out-dir /home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps -C incremental=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/incremental -L dependency=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps`
memory allocation of 8589934592 bytes failed                                                                           error: Could not compile `foobar`.

Caused by:
  process didn't exit successfully: `rustc --crate-name foobar src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=0edabc38d37db739 -C extra-filename=-0edabc38d37db739 --out-dir /home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps -C incremental=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/incremental -L dependency=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps` (signal: 6, SIGABRT: process abort signal)

---

Edit: seems as if the problem is happening due to running out of RAM because of doubling allocation sizes. Reported memory allocation of ${n} bytes failed messages on nightly are always powers of two and htop shows those allocations adding up pretty well. However this also happens with a 1g file. Running out of 16g of RAM when include_str!()ing a 1g file shouldn't happen, yet I can't tell where the problematic allocation(s) happen(s).

[saskenuba]

I also have the same error, but in my case, it is a tiny file in comparison(82mb).

In debug mode, it compiles and tests normally, but on release, it fails to build after my computer freezes for about 2 seconds.

Does the compiler include the file on the binary even on tests? I thought it would check only at runtime.

trace:

error: could not compile `steam-inventory-parser`.
Caused by:
  process didn't exit successfully: `rustc --crate-name steam_inventory_parser --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --test -C metadata=b07e9aa8ce28fcc6 -C extra-filename=-b07e9aa8ce28fcc6 --out-dir /home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps -L dependency=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps --extern boolinator=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libboolinator-14bdfbf816afc3d2.rlib --extern chrono=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libchrono-3f34e00f8db6b539.rlib --extern either=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libeither-bcbf9585182e52ab.rlib --extern flate2=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libflate2-3cacb3ce577b09ca.rlib --extern lazy_static=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/liblazy_static-b9b33625fe9d36fa.rlib --extern regex=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libregex-857ec56e74313868.rlib --extern serde=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libserde-de21e8f5e0bf3fb7.rlib --extern serde_json=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libserde_json-ace6174dc8fdd25c.rlib --extern serde_repr=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libserde_repr-387f73d0adeb3a32.so --extern serde_with=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libserde_with-2fa3235741b88b3f.rlib --extern thiserror=/home/martin/Documentos/Programming/Rust/steam-inventory-parser/target/release/deps/libthiserror-78f01c8033fedbb9.rlib` (signal: 9, SIGKILL: kill)

Process finished with exit code 101