Reproducibility against archived CMIP6 data

[blimlim]

Divergence in each configuration

The released configurations fail to reproduce the CMIP6 simulations when using the same restart files. Confusingly, divergence only happens some way into the simulation.

Our release-preindustrial+concentrations simulation is based on PI-02, which has output located at /g/data/p73/archive/CMIP6/ACCESS-ESM1-5/PI-02. Surface temperature diverges in our release in the middle of May of the 20th year of simulation:

Our release-historical+concentrations simulation is based on HI-24. This diverges very early, as we are using input files with different timing (see here and here for background behind these differences), and so this difference is not a problem.

If we set up our inputs to match HI-24 however, the simulations match until May 1 of the 4th year of simulation. From there they diverge:

Our prerelease AMIP configuration likewise used different input data as the CMIP run AM-09 by default. When we make the input data match, the surface temperatures diverge on July 7 1980, the third year of the simulation

---

Details from the AMIP experiments

The differences appear to come from the vk83 UM executables, rather than anything in the payu configuration.

Using a vk83 executable in the script setup also diverges from the CMIP6 simulation. Looking in the restart files (which save fields at higher precision than the history files) @MartinDix found that carbon pools start to diverge in Jan 1979. These pools don't immediately impact the climate, but eventually might affect the leaf area index which then causes the more visible divergence in the climate.

Rebuilding an executable via the older scripts matches the vk83 spack executables. @MartinDix mentions that the original CMIP6 executables may have been built when a different operating system was used, but aside from that, the code looks identical.

[MartinDix]

Original AM-09 simulation was run on gadi using the executable /g/data/p66/txz599/ACCESS-ESM1p5/bin/um_hg3.exe-20200311_r343. The original source directories used for the build are no longer available but date and name suggest it was this revision

% svn log file:///g/data/access/access-svn/cmip5/branches/dev/jxs599/trunk_ESM1.5 | head
------------------------------------------------------------------------
r343 | txz599 | 2020-03-11 19:46:27 +1100 (Wed, 11 Mar 2020) | 1 line

TZ: bugfix, wood harvest pool must not be initialised to 0

With the original executable a rerun of the script version reproduces the original AM-09 for several years. However a rerun with a rebuilt executable (using either the original method or spack) gives differences as above.

The 1979-01-01 restart files are identical. However the 1980-01-01 restarts have differences in carbon pool variables only. It seems that these differences eventually get large enough to cause a difference in the LAI which causes differences in the atmosphere and the simulations diverge.

What's remarkable is that the initial carbon pool difference happens on only a couple of grid points in the middle of January 1979 and is stable for the rest of the month.

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352/5