Add a checker for the testlist to verify that the tests make sense

[ekluzek]

One of the reasons for problems in our testlist like #3048 is that we don't have a checker to verify that the testlist doesn't have common problems. #2979 is an example of a checker, but it just checks that syntax is correct. Which is a great thing to check. But, it would be helpful to have something that goes beyond that and does other checking for test validity.

The way we have managed the testlist is with manual examination, and trying to make sure the list makes sense. Which is hard to do with a long list. And it takes a lot of focused effort to do. So it's not something we can do very often.

Ideas for things to check:

1. For example, one of the things checked for could be length by testmod to prevent Tests that don't have CTSM history output #3048 with newly added tests.
2. Have it verify which testmods are used in the testlist -- show which ones aren't tested
3. Have it check the types of tests being used and show which ones aren't being used.
4. Have it show the balance between different tests. query_testlist shows by machine/compiler, but also sorting by things like this: resolution, DEBUG or not, test-type, mpi-serial or not, threaded tests, and test-length categories

[ekluzek]

Number "1" in the list above would be hard to check in a non-hacky way, unless the test checking was done after a test list was run, as then it could just check that CLM history files exist, or do things like query the history settings in the lnd_in file.

The other checking for balance in the testlist could be part of query_testlist in cime. Or it could be something where we add a parser script in CTSM on top of cime.

[wwieder]

@samsrabin and @ekluzek will meet and make a list of subprojects to start making progress here

[samsrabin]

We have a planning document here.

[ekluzek]

I added a cime issue here: ESMCI/cime#4776