prevent size overflows in the type system

[thestinger]

Type size calculations inside the compiler can overflow, resulting in memory unsafety. Types without a valid mem::size_of should be forbidden to prevent unsoundness. Types that are valid today would become invalid and this would interfere with an attempt to support integer type parameters. I think it's a serious backwards compatibility issue since the interaction with generics is very bad.

std::mem::size_of::<[[u8, ..!0u], ..!0u]>() returns 1, since !0u * !0u wraps to 1. The same thing can be done with other aggregate types like structs / tuples and an overflow could also occur from the tag added in an enum.

[zwarich]

Here's an example of a simple program that segfaults on x86_64 OS X:

fn main() {
    let n = 0u;
    let a = box [&n,..0xF000000000000000u];
    println!("{}", a[0xFFFFFFu]);
}

[thestinger]

This seems to indicate that Rust can't correctly type-check generics from the definition alone. It needs to be aware of the type parameters in order to check for size overflows. Sizes are platform specific due to issues like alignment so valid types are platform specific.

[arielb1]

@thestinger

If a type has a size overflow then it can't exist, as it can't fit in memory. If you try to place it on the stack, you would get a stack overflow. This is, and should, be handled within trans.

[thestinger]

@arielb1: You're misunderstanding the issue. A type will a size overflow can certainly be created, as demonstrated by the example @zwarich gave. The size of the type has overflowed, so it's possible to create it because not enough space will be reserved. The problem is not limited to fixed-size arrays and can't be handled solely during initialization. The existence of the types actually needs to be considered invalid.

[arielb1]

@thestinger

Certainly they can be created now (due to the bug) – however, they should not be creatable (given that they take all address space). I think making size_of::<T> give a trans error when sizeof(T)>INT_MAX would be fine (its not like its impossible to make trans error out with resource limitations anyway).

[thestinger]

The problem isn't limited to mem::size_of. It's a general issue with size overflows in the compiler. There's no size_of call in @zwarich's example and it's not hard to think up new ways of hitting this.

[thestinger]

The size calculation code inside of the compiler (underlying size_of and lots of other code paths) could error out in this case, but I expect that it wouldn't be enough to prevent this in all cases. There are other ways of initializing data (uninit and init among others) that are used to build higher level safe abstractions. There's an expectation that every type corresponds to a valid representation, whether or not it can actually be created in practice.

[nikomatsakis]

@thestinger I agree, we should be careful in the trans paths, but it seems like ultimately this can be handled by checking for overflow in various places when constructing the LLVM version of a Rust type. e.g., when constructing an array type, we'd check that the size of the base type * the length doesn't overflow and so forth. Do you disagree?

[thestinger]

@nikomatsakis: The logic can be in trans, but it's not as simple as checking at construction of an array (multiplication overflow) / struct (addition overflow) / enum (addition overflow from the tag). For example, consider the init and uninit intrinsics or creating a struct with box.

[arielb1]

The overflow actually seems to happen within LLVMSizeOfTypeInBits aka llvm::DataLayout::getTypeSizeInBits (which is called from rustc::middle::trans::machine::llsize_of_real) – which makes this LLVM's issue.

This can be observed by noticing that the following program prints 0:

fn main() {
    println!("{}", std::mem::size_of::<[u8, ..(1<<61)]>());
}

Note that the size is 1<<61 bytes, not 1<<64 as would be expected (also, on 32-bit architectures, trans::intrinsic L242 reduces sizes to a uint – which seems bad, especially because this is a compiler uint, not a target uint, but the 64-bit overflow is totally LLVM's fault).