ACCESS-OM3: use per-variable checksums from mom restart files

[dougiesquire]

This PR modifies the AccessOm3 test class to extract per-variable checksums from MOM restart files, rather than using the ocean.stats output. This required some minor modifications to the ExpTestHelper and base Model test classes to allow the archive restart directories to be easily used.

Relevant to #86, although doesn't actually use the Payu hashes as suggested in that issue. See COSIMA/access-om3#278 for some additional context/discussion.

[codecov]

Codecov Report

Attention: Patch coverage is 93.93939% with 2 lines in your changes missing coverage. Please review.

Project coverage is 74.88%. Comparing base (c4a40b3) to head (dfcaa49).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
src/model_config_tests/models/accessom3.py	96.15%	1 Missing ⚠️
src/model_config_tests/test_bit_reproducibility.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
+ Coverage   72.77%   74.88%   +2.11%     
==========================================
  Files          17       17              
  Lines         889      900      +11     
==========================================
+ Hits          647      674      +27     
+ Misses        242      226      -16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[anton-seaice]

I haven't run this yet but will soon. I left a bunch of fairly inconsequential comments on the tests

I think you could leave the issue open because this only addresses MOM repro, not other components

[jo-basevi]

Just confirming that I've looked at the code and everything added looks good to me. I haven't got any additional feedback that hasn't been brought up already

[anton-seaice]

I ran this with access-om3 at it works as expected.

Run 1: make checksums and then I copied them into testing/checksums
Run 2: pass
Run 3: change MOM_input and confirmed fail correctly

restart repro from cold start passed and 2x1day with restart in the middle compared to 1x2 day failed, consistent with other tests

[dougiesquire]

@anton-seaice (and/or anyone else) could you take another look please? Once you're happy I'll tidy up the history a little before merging

[anton-seaice]

Thanks @dougiesquire !

[dougiesquire]

@anton-seaice, I tidied up the history a little. Could you please approve again

[anton-seaice]

Thanks @dougiesquire