Segmentation fault on nightly and beta when running with --release

[joe-hauns]

I encountered a segmentation fault when running a program on nightly or beta in release mode.
The segfault occurs somewhere in another crate but within safe code and only if --release is passed to cargo.
I created a crate which is minimal setup to encounter this error:
https://github.com/joeschman/tokio-timer-segfault
A more detailed description can be found in the readme.

[fu5ha]

I am also experiencing this in https://github.com/termhn/nano-rs on nightly-x86_64-pc-windows-msvc however not when running on my Mac also on nightly.

[fu5ha]

This fixes it tokio-rs/tokio-timer#40 ... not sure what that means or if it is helpful (some optimization to do with continue breaking things maybe?)

[michaelwoerister]

cc @rust-lang/compiler

[nikomatsakis]

Do we know if this is a regression? Can one of you try to reproduce on older nightly builds and see if you can find some point that worked?

[nikomatsakis]

My guess would be that it is some kind of optimization gone wrong in LLVM, but whether that's LLVM's fault or ours is sort of unclear.

[joe-hauns]

Can one of you try to reproduce on older nightly builds and see if you can find some point that worked?

I'd test that but pittily I could not find out how to revert to older nightly/beta versions using rustup :/

[oli-obk]

You can use rustup override set nightly-2018-01-15 to set to e.g. 15th January.

[joe-hauns]

I checked older nightly versions and the segfault appears first when using nightly-2018-02-11-x86_64-apple-darwin.

[nikomatsakis]

@joeschman Thanks! Btw, jfyi there is a new shiny tool by @Mark-Simulacrum that can do the bisection for you:

https://github.com/rust-lang-nursery/cargo-bisect-rustc

It will even find the problem down to the PR level if it arose in the last 90 days.

In any case, these are the diffs between those two nightlies, if I'm not mistaken:

3bcda48...45fba43

[nikomatsakis]

One thing jumps out at me:

6b7b6b6

Upgrade to LLVM 6 =)

[fu5ha]

Indeed. Definitely seems like the kind of bug a new LLVM version could introduce.

[nikomatsakis]

cc @rust-lang/compiler -- anybody want to try to track down a potential LLVM bug introduced by LLVM 6 transition?

@nagisa seems like your speciality :)

[nagisa]

Okay, I’ll try to take a look at this tomorrow.

[nagisa]

Ugh, I cannot reproduce the fault on Linux, so it is very hard for me to even start investigating this more seriously.

The supposed error location is within a huge inlined 1.5k line blob of IR and the function with and without the fix at tokio-rs/tokio-timer#40 appears to be optimised down into identical code, at least for that specific function.

[fu5ha]

Anything I could provide from the windows side that would help with this—artifacts from building etc? I’m not super knowledgeable about everything that happens but willing to help if I can :)

…

On Mar 21, 2018, at 11:21 AM, Simonas Kazlauskas ***@***.***> wrote: Ugh, I cannot reproduce the fault on Linux, so it is very hard for me to even start investigating this more seriously. The supposed error location is within a huge inlined 1.5k line blob of IR and the function with and without the fix at tokio-rs/tokio-timer#40 <tokio-rs/tokio-timer#40> appears to be optimised down into identical code, at least for that specific function. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#49010 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2Lny1pOIsY8d03rItFM7eY0JmiAkZVks5tgpoggaJpZM4SqHsX>.

[nagisa]

Can you see if the issue is reproducible on windows with the minimal sample provided in the first comment?

[joe-hauns]

If there‘s any more information i can provide for the problem ln macos, just tell me what ;)

[nagisa]

Wrote this comment yesterday but forgot to hit submit.

---

So here’s the thing that sticks out as a sore thumb to me in the optimised IR as far as differences between the "bad" and "fixed" versions goes:

  %min.0.ph32.i.i.i = phi i64* [ null, %bb8.lr.ph.i.lr.ph.lr.ph.i.i.i ], [ %.lcssa23.i164.pre-phi.i.pre-phi.i.pre-phi, %bb11.i165.i.i ]
  ; ...
  %328 = icmp ne i64* %min.0.ph32.i.i.i, null
  ; a while later
  call void @llvm.assume(i1 %328) #13, !noalias !213

I still haven’t checked the IR closely enough to confirm whether the CFG allows for this assume to ever called on falsy value, but this is really the only notable difference between the fixed and non-fixed version anyway.

Surprisingly, this %328 is never ever used again.

[nagisa]

Things that would be helpful and make it easier to debug this:

1. A single-file minimal reproducer with as few dependencies on system utilities (such as threads) as possible;
2. Backtrace from a debugger, register dump at the time of the fault and disassembly of the function.

A minimal reproducer that works on windows would be better as I have an access to a windows VM, though perhaps it is as good a time as ever to figure out how to make a mac VM… hmm.

[nikomatsakis]

Assigning to nagisa, but it seems like they are stuck without some way to reproduce.

[nikomatsakis]

Assigning to @pnkfelix to try and reproduce on mac

[joe-hauns]

I now minimized the example. All the code is now in the repository without dependencies to crates that are outside the repository. Pittily a one-file reproducer was not possible. It seems that the bug only occurs when the code is called from another crate. I minimized the size of that crate to 20 lines of code. When the same code is run from an internal module, no segfault occurs.
Both (calling the module & the crate function) is included in the last commit of the crate i linked above.

Furthermore I found out that with yesterday's nightly build ( nightly-2018-03-21-x86_64-apple-darwin) the segfault does not occur anymore. I get the segfaut with nightly-2018-02-11-x86_64-apple-darwin tonightly-2018-02-15-x86_64-apple-darwin. (Didn't test it with *-03-16-* to *-03-20-* ).

Backtrace from a debugger, register dump at the time of the fault and disassembly of the function.

Any recommendations for a debugger for rust on a mac? I have hardly any experience with using debuggers at all.

[fu5ha]

Alright so I was testing on my Windows machine and confirmed the same behavior as @joeschman has on mac. I also pinpointed that it is fixed in nightly-2018-03-03 and later, but broken in nightly-2018-03-02 and earlier.

[pnkfelix]

I have reproduced the problem on my mac, using the joeschman/tokio-timer-segfault repo and the nightly-2018-03-02 rustc. I'll report more as I dissect further.

[pnkfelix]

Here is a gist https://gist.github.com/pnkfelix/fdab1b374d49e8850073a357d4f492f4 with the things that @nagisa had asked for (a stack backtrace, dump of the register file, and a disassembly of the function).

[pnkfelix]

@joeschman by the way, the way I got the info in the above gist is I just ran the binary under lldb, like so:

% lldb target/release/tokio-timer-segfault

and then when I got the debugger prompt ((lldb)), I ran the program with r:

(lldb) r

and then after the program exception occurred, I used the following commands (included in the gist output above) bt, register read, and dis.

lldb has an online help mechanism you can use to explore its set of commands; or you can google for cheat sheets online.

[pnkfelix]

@nagisa pointed out that I was using an old head of the source repo.

I updated and re-ran the same lldb commands, and updated the gist (which I'll relink for convenience: https://gist.github.com/pnkfelix/fdab1b374d49e8850073a357d4f492f4 )

[pnkfelix]

Experimentation and Discussion with @nagisa and @alexcrichton has led us to the hypothesis that this is a bug injected by thinlto, which is on by default for --release builds ...

[nikomatsakis]

commit range from nightly-2018-03-02 to nightly-2018-03-03 is 3eeb5a6 .. 9cb18a9

[nikomatsakis]

git log 3eeb5a665..9cb18a92a --author=bors --oneline yields:

9cb18a92ad Auto merge of #48653 - Manishearth:rollup2, r=Manishearth
ddfbf2b0f4 Auto merge of #47861 - sgrif:sg-rebase-chalkify-universe-refactorings, r=nikomatsakis

[nikomatsakis]

I doubt these are the culprits, but skimming the commit log shows @alexcrichton's "rustc: Tweak funclet cleanups of ffi functions", which seems ... at least in the neighborhood.

[nagisa]

Funclets would only affect certain windows targets and nothing else.

…

On Thu, Apr 12, 2018, 17:18 Niko Matsakis ***@***.***> wrote: I doubt these are the culprits, but skimming the commit log shows @alexcrichton <https://github.com/alexcrichton>'s "rustc: Tweak funclet cleanups of ffi functions" <804666f>, which seems ... at least in the neighborhood. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49010 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApc0seepHT1rsJVRbxA-7sCUJd5idBZks5tn2IsgaJpZM4SqHsX> .

[pnkfelix]

We may need to appoint someone to be the official "LTO debugging" expert based on the number of bugs we currently have that have sort of stalled at the point of determining "oh, this appears to be injected by [Thin]LTO."

[nikomatsakis]

triage: P-medium

Next steps are to diagnose the LLVM problem. Filing under #50422.

[steveklabnik]

Triage: looks like the energy to reproduce this one kinda petered out. Can anyone reproduce this today?

[steveklabnik]

One year later, with no reproduction instructions, I'm going to give this one a close. If anyone can make a reproduction, please let me know and we can re-open, thanks!