add dev workflow for cam-sima

Author: peverwhee

docs/development/cam-sima-workflow.md

@@ -0,0 +1,98 @@
+# Development workflow for CAM-SIMA
+
+This page describes the general workflow for adding new development to the [CAM-SIMA repository](https://github.com/ESCOMP/CAM-SIMA).
+
+## Workflow summary
+
+The general workflow for adding a feature, bug-fix, or modification to CAM-SIMA is as follows:
+
+1. [**Open an issue**](#open-an-issue)
+1. [**Add your code modifications**](#add-your-code-modifications)
+1. [**Open a PR**](#open-a-pr)
+    - If you know that this PR will need an official tag, then also add the [tag name](git-basics.md/#tagging-a-commit) to the PR description.
+1. **Respond to any reviewer requests.**
+1. [**Fix any failing tests**](#fix-failing-tests)
+1. [**Run the manual regression tests**](cam-testing.md/#regression-testing)
+    - Update `$CAM-SIMA/test/existing-test-failures.txt` if there are new tests failing or if existing test failures have been fixed.
+1. **Confirm with the CAM-SIMA gatekeeper that you are OK to merge. Then squash the commits and merge the PR** (e.g. the "squash and merge" option).
+1. [**Make a tag (if regression test answers have changed)**](git-basics.md/#tagging-a-commit)
+1. [**Archive baselines (if tag has been made)**](cam-testing.md/#archiving-baselines)
+
+## Workflow details
+
+The following sections describe various workflow actions in more detail.
+
+### Open an issue
+
+It is generally recommended to [open an issue](https://github.com/ESCOMP/CAM-SIMA/issues/new) for any new features that will be added or bug that has been found that will need to be fixed. There is currently no offiical requirement on what should be contained within the issue text, so generally just put any information you think might be relevant.
+
+### Add your code modifications
+
+1. If you haven't already, create a fork. Specific instructions can be found [here](git-basics.md/#one-time-github-setup).
+1. Create a branch on your fork off of the head of the ESCOMP remote's `development` branch. Specific commands can be found [here](git-basics.md/#working-with-branches)
+1. Apply your code modifications and/or script additions, and perform at least one test making sure your modifications work as expected. See below for Git commands for committing to your branch.
+
+There are multiple ways to do commit your changes, but one of the safer ways is to first check your status:
+```
+git status
+```
+
+This will provide a list of all modified files. For each one of those files whose modifications you want to add to the main package, you will do the following:
+```
+git add updated_fortran_file.F90
+```
+
+Where `updated_fortran_file.F90` should be replaced with whatever your file name is. Do this for each file you want to include. If you are confident that every file listed by `git status` needs to be added, then you can do it all at once by doing:
+```
+git add -A
+```
+
+You can then type `git status` again, at which point all of the files you added should be "staged" for commit (green).
+
+To commit your changes to your local branch:
+```
+git commit -m "<message>"
+```
+
+where `<message>` is a short description on sentence stating what the commits are for, e.g. "fixed color bar bug" or "added significance hatching".
+
+To push your changes to your fork:
+```
+git push
+```
+
+or, if you want to be more explicit,
+
+```
+git push origin <branch name>
+```
+
+### Open a PR
+
+1. Go to the [ESCOMP CAM-SIMA repo](https://github.com/ESCOMP/CAM-SIMA), and click on the "Pull requests" tab.
+ ![text](figures/pr-workflow-1.png "PR tab")
+1. There, you should see a "New pull request" button, which you should click.
+ ![text](figures/pr-workflow-2.png "New PR button")
+1. On the new "Compare changes" page, you should see a "compare across forks" link, which you should click.
+ ![text](figures/pr-workflow-3.png "Compare across forks")
+1. You should now see two new pull-down boxes (to the right of an arrow). Using these pull-down boxes, select the "development" branch of the ESCOMP repo and select your fork (which should be `<username>/CAM-SIMA`)
+1. Then select the branch which contains the commits.
+ ![text](figures/pr-workflow-4.png "pull-down boxes")
+1. You should then see a list of all the different modifications. If they generally look correct, click the "Create pull request" button
+1. A new page should appear. In the first text box, add the title of your pull request. The second text box will contain additional fields that you should fill out to the best of your ability.
+1. If your code changes cause answer changes, you will need a new tag. Enter the expected tag name in the PR description. More on tag conventions [here](git-basics.md/#tagging-a-commit).
+1. Add any relevant labels to the PR, add yourself as the assignee, and add any reviewers you would like to have. Otherwise, the core SE team will add reviewers for you.
+1. Click "Create pull request"
+
+### Fix failing tests
+Failing unit tests will appear at the bottom of the PR page (as shown below). You will also get an email if any tests fail.
+ ![text](figures/pr-workflow-5.png "unit testing")
+
+Click through (via `Details` link) to see the test output of the failing tests. Refer to the table below for debugging failing tests.
+
+| Failing test | Description | Resources |
+| ------------ | ----------- | --------- |
+| `.github/workflows/fleximod_test.yaml / fleximod-test (<python version>) (pull_request)` | git-fleximod test failure. Likely an invalid external | [git-fleximod testing](cam-testing.md#git-fleximod-tests) |
+| `Python unit tests / python_unit_tests (<python version>) (pull_request)` | Python unit test failure. Likely need to update sample files | [Python unit testing](cam-testing.md/#python-unit-testing) |
+| `Source Code Linting / source_code_tests (pull_request)` | Python linting failure. Need to fix modified python code. | [Python source code linting](cam-testing.md/#python) |
+

docs/development/cam-testing.md

@@ -9,6 +9,9 @@ This page describes the various automated and manual tests that are run for CAM-
 
 Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations.  The tests can be run with a local copy of CAM-SIMA by using the `test_driver.sh` script under `$CAM-SIMA/test/system`.  To run the tests associated with a particular compiler option one can do the following commands:
 
+!!! Warning "run git-fleximod"
+    Make sure you have run `bin/git-fleximod update` before you run the tests!
+
 For running GNU tests*:
 ```
 env CAM_FC=gnu ./test_driver.sh -f
@@ -49,7 +52,7 @@ If you have familiarity with CAM cases, the setup and directory structure will b
 The tests themselves are listed in `<CAM-SIMA>/cime_config/testdefs/testlist_cam.xml`. Any files that need to be included in order for the tests to run properly are located in `<CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX`, where `XXX` is the name of the test.  Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found [online here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html). 
 
 ### Archiving baselines
-If your PR changes answers, then after you have run the tests, merged your PR, and created a tag (see [tag workflow](git-basics.md#tagging-a-commit)), you will need to archive your baselines for the next person.
+**If your PR changes answers**, then after you have run the tests, merged your PR, and created a tag (see [tag workflow](git-basics.md#tagging-a-commit)), you will need to archive your baselines for the next person.
 
 To do this, navigate to `$CAM-SIMA/test/system` on derecho and run:
 
@@ -129,10 +132,23 @@ All `doctest` tests are also run automatically as long as the scripts they are l
 
 To manually run all of the unit tests at any time, simply run the following shell script:
 
-`CAM-SIMA/test/run_tests.sh`
+`CAM-SIMA/test/run_unit_tests.sh`
 
 Finally, when adding new tests, determine if the test can be done in only a few lines with minimal interaction with external files or variables.  If so, then it would likely be best as a `doctest`.  Otherwise it should be a `unittest` test.  Failure to follow this rule of thumb could result in test failures in the Github Actions workflow.  Also remember to add your new tests to the `run_tests.sh` script so future users can easily run the tests manually.
 
+#### Updating sample files
+If you modified any python files in your code modifications (those in `cime_config` or in `src/data`), you may need to update the sample files to match what is now expected.
+
+1. Run the unit tests as described above
+1. If you have failures, quickly check that the diffs are expected
+
+    ```
+    diff <tmp file> <sample file>
+    ```
+
+1. Copy over the changed files from the `tmp` directory to the `sample_files` directory
+1. Rerun the tests to confirm they all pass now.
+
 ### Static Source Code Analysis
 
 #### Python

docs/development/figures/pr-workflow-1.png

@@ -22,6 +22,7 @@ nav:
     - design/history.md
     - design/sima-design-goals.md
   - Development:
+    - development/cam-sima-workflow.md
     - development/cam-coding-standards.md
     - development/cam-testing.md
     - development/debugging.md