Configuration consistency/sanity checks?

[dougiesquire]

Often it is important that different parameters in a configuration are set consistently. These may be different but related parameters, or one parameter that needs to be set to the same value in multiple places (yuck, I know, but we're stuck with this unfortunately). There's an argument for including simple checks like these in Payu, to be run automatically as part of payu run. If you agree, where's the right place to do this:

• in the setup step of a model driver (there are some checks in some drivers already)?
• as a userscript?
• something new, e.g. a driver check method (and subcommand)?
• something else?

[aidanheerdegen]

Sorry to equivocate, but all the options you've presented sound reasonable, depending on the specifics of the use case.

To my mind setup and check are close to the same thing. The setup option would have to call a check routine, or something similar. Creating check as a CLI argument would just be a case of making that internal command callable separately.

The question of setup or userscript probably comes down to how specific are these checks to a specific configuration? If they are something that would be applicable in all cases for a model, then adding it to the model driver makes sense. If the checks are more configuration specific, if there are cases where the checks might not be necessary, then userscript makes more sense.

The downside of userscript is you don't have access to all the internal payu information. If this was necessary there might be a case for a check routine, which isn't called by default, but could be specified as a userscript option for specific configs.

Perhaps some specific examples would help to elucidate this?

As an extension I'd like to see checks like this be part of CI when updating released configurations.

[dougiesquire]

To my mind setup and check are close to the same thing. The setup option would have to call a check routine, or something similar. Creating check as a CLI argument would just be a case of making that internal command callable separately.

Agreed. I just wondered whether it might be cleanest to have the checks in their own method. It would make it clear to new driver developers, for example, that checks are a good idea and where they should live. But yes, it boils down to the same thing, and thinking about it more I can't see any value in having separate setup and check subcommands.

All of the checks I want to run at the moment make the most sense in the (access-om3) driver. But, I wanted to check there were no strong opinions about whether/where/how they should be included in Payu.

As an extension I'd like to see checks like this be part of CI when updating released configurations.

Yes! And if the checks are in Payu we could use Payu to do this

[aidanheerdegen]

All of the checks I want to run at the moment make the most sense in the (access-om3) driver. But, I wanted to check there were no strong opinions about whether/where/how they should be included in Payu.

If these are checks that one would always want to run for all versions of access-om3 then yeah, sounds like the driver is the right place for them.

Agreed a checks method is clean, and can be exposed as a CLI at a later date, or added as a --checks option to setup if it isn't something that should always be run (which it may be).

[aidanheerdegen]

We're implementing some QA checks in CI for the access-om2 configurations

ACCESS-NRI/access-om2-configs#32

They're an example of checks we want to do for released configs, but not routinely when running the model. Just FYI.

[dougiesquire]

This came up in a meeting today. The access-om3 Payu driver already includes some configuration checks and we will be adding more. We would like to run these checks as part of the CI QA step @aidanheerdegen mentions above. However, these checks are currently run as part of the Payu setup step and we can't run payu setup in the CI because the Github runners don't have the input files (I think @jo-basevi maybe mentioned this might change in the future?).

Question: can I add a payu check to the Payu CLI to allow us to easily run just the checks?

[jo-basevi]

I think @jo-basevi maybe mentioned this might change in the future?

I think I was referencing this issue (ACCESS-NRI/model-config-tests#9) where we could run payu setup as part of the reproducibility tests that run on gadi (or add QA tests that can only run on gadi).

Yeah if these checks can be run on a github runner, e.g. just need the payu configuration available, then I think adding payu check is a good idea

[dougiesquire]

I had a chat with the team and it sounds like there isn't really a demand to be able to run the checks quickly on the Github runners, so scratch my suggestion 🙂. I'll close this issue