Solution diverging while using GPU for running VEROS

[sb4233]

Opening a new issue which was first reported here - #669 , as the problem is still not solved.

While using GPU for running VEROS, the solution diverges -

RuntimeError: solution diverged at iteration 411577
srun: error: gpu1: task 0: Exited with exit code 1

I checked this with both the global_1deg.py and global_flexible.py setups.

[dionhaefner]

How can I reproduce this? Are you running those setups unmodified?

[sb4233]

How can I reproduce this? Are you running those setups unmodified?

Yeah, I am just using the default setups.
About the runs, I am doing a 1000-year simulation (10 x 100-year simulation) with the veros resubmit method on a single GPU.

[dionhaefner]

So how long can you run until it diverges?

[sb4233]

So how long can you run until it diverges?

Mostly it diverges at the end of first 100-year simulation.

[dionhaefner]

Could you please try again with this environment variable set?

$ export VEROS_LINEAR_SOLVER=scipy

(I suggest you write restart files in regular intervals, e.g. every 5 years or so, which should come in handy for future debugging.)