Out of memory errors don't work properly

[shkodm]

Found on develop branch when trying to run large example (automatically generated).
Can, for instance, can be reproduced:

256 x 512 x 256: [ ] [0] Cumulative allocation of 168231424 b (30.2 GB) Works fine
256 x 800 x 256: Cumulative allocation of -24577536 b (18446744073.7 GB) Throws out of memory error correctly, but incorrect reporting of total allocated memory.
512 x 800 x 256: Tries to allocated incorect (much smaller) total amount of memory and throws a different error:

Cumulative allocation of -154486272 b (25.6 GB)
[  ] Initializing Lattice ...
[ ] FATAL ERROR: an illegal memory access was encountered in Lattice.hpp at line 445

Expected behaviour: correct memory allocation is reported, and correct error is throws (the behaviour of master branch). Probably related to some casting or overflow

[llaniewski]

Seems the problem was introduced here:
101e6c2#diff-d31965790d0025ccd455cffbe6c4c9fdcdcc33946b62479b634f87f9d8a574f9R197
when @kubagalecki deleted (size_t) conversion in the calculation of size. I'll make a pull request to fix this (and fix the printing at the same time)

[shkodm]

@llaniewski it is probably a different bug, but some things still don't work as expected (also the same on master branch). I run on 2 V100 on Bunya, each with 80GB GPUs, my case is large, so I split between 2.
I get:
Cumulative allocation of 63.GB)
and then
an illegal memory access was encountered in Lattice.hpp at line 279

The error is the same even if try I split between 3 GPUs (40GB each, so plenty of space even if there is some unaccounted memory)

[llaniewski]

@shkodm Just to clarify, do these large cases run on the master branch?

[shkodm]

@llaniewski no, they also don't work on master branch. The error happens in the Lattice.cu at the same place (CUDA kernel synchronisation). I ran with d3q27_pf_velocity model.

[llaniewski]

Closing this issue and moving the discussion of the size limitation to #499 . Addressing it is a bigger thing and wound need testing of performance.