08_PCAST/08_PCAST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "746d1042-9b52-4e9f-bce4-0bcd7eb15d48",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![NCAR UCAR Logo](../NCAR_CISL_NSF_banner.jpeg)\n",
    "# Verifying Code Correctness with PCAST\n",
    "\n",
    "By: Daniel Howard [dhoward@ucar.edu](mailto:dhoward@ucar.edu), Consulting Services Group, CISL & NCAR \n",
    "\n",
    "Date: May 19th, 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca1ca32-c6ce-4ac3-8aa4-0fa7771c6560",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "In this notebook, we return to the MiniWeather application to learn how to use PCAST, a tool specific to the NVIDIA HPC SDK for verifying code correctness. We will cover:\n",
    "\n",
    "* Benefits and Challenges of Validating Scientific Software\n",
    "* Usage of Parallel Compiler Assisted Software Testing (PCAST)\n",
    "    * Comparing CPU and GPU Code Execution\n",
    "    * PCAST with a Golden File  \n",
    "    * PCAST with OpenACC and Autocompare"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfdb6b56-f032-42a0-a6c5-84ccece39ad8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    " Head to the [NCAR JupyterHub portal](https://jupyterhub.hpc.ucar.edu/stable) and __start a JupyterHub session on Casper login__ (or batch nodes using 1 CPU, no GPUs) and open the notebook in `08_PCAST/08_PCAST.ipynb`. Be sure to clone (if needed) and update/pull the NCAR GPU_workshop directory.\n",
    "\n",
    "```shell\n",
    "# Use the JupyterHub GitHub GUI on the left panel or the below shell commands\n",
    "git clone git@github.com:NCAR/GPU_workshop.git\n",
    "git pull\n",
    "```\n",
    "\n",
    "# Workshop Etiquette\n",
    "* Please mute yourself and turn off video during the session.\n",
    "* Questions may be submitted in the chat and will be answered when appropriate. You may also raise your hand, unmute, and ask questions during Q&A at the end of the presentation.\n",
    "* By participating, you are agreeing to [UCAR’s Code of Conduct](https://www.ucar.edu/who-we-are/ethics-integrity/codes-conduct/participants)\n",
    "* Recordings & other material will be archived & shared publicly.\n",
    "* Feel free to follow up with the GPU workshop team via Slack or submit support requests to [support.ucar.edu](https://support.ucar.edu)\n",
    "    * Office Hours: Asynchronous support via [Slack](https://ncargpuusers.slack.com) or schedule a time with an organizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8594d82b-c91d-4f83-80e7-8df8865c16e4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Complete Mid-Workshop Series Survey\n",
    "In order to get feedback on and improve future GPU workshop series sessions, please complete the below survey. We will spend 3-5 minutes at the start of today's session to collect your feedback.\n",
    "\n",
    "Head to [https://forms.gle/RRkfwnHnDsqqe1zE9](https://forms.gle/RRkfwnHnDsqqe1zE9) or scan the below QR code.\n",
    "\n",
    "<img src=\"img/Survey_QR.png\" alt=\"GPU Workshop Mid-Series Survey QR Code\" style=\"width:300px;\"/>\n",
    "\n",
    "If you've finished the survey, feel free to ask questions about any past material or other GPU topics during this time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e08459-56ad-4ee5-bea7-9a988b0e15c0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Notebook Setup\n",
    "Set the `PROJECT` code to a currently active project, ie `UCIS0004` for the GPU workshop, and `QUEUE` to the appropriate routing queue depending on if during a live workshop session (`gpuworkshop`), during weekday 8am to 5:30pm MT (`gpudev`), or all other times (`casper`). Due to limited shared GPU resources, please use `GPU_TYPE=gp100` during the workshop. Otherwise, set `GPU_TYPE=v100` (required for `gpudev`) for independent work. See [Casper queue documentation](https://arc.ucar.edu/knowledge_base/72581396#StartingCasperjobswithPBS-Concurrentresourcelimits) for more info.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38ce23ae-c614-4c46-ab85-c80484f0655d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export PROJECT=UCIS0004\n",
    "export QUEUE=gpudev\n",
    "export GPU_TYPE=v100\n",
    "\n",
    "module load nvhpc/22.2 &> /dev/null\n",
    "export PNETCDF_INC=/glade/u/apps/dav/opt/pnetcdf/1.12.2/openmpi/4.1.1/nvhpc/22.2/include\n",
    "export PNETCDF_LIB=/glade/u/apps/dav/opt/pnetcdf/1.12.2/openmpi/4.1.1/nvhpc/22.2/lib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e25bb660-24a0-4b0c-9abd-e8b72839fa01",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Benefits of Validating Scientific Software\n",
    "Building __trust__ is paramount for the effective sharing and receiving of computational software. From NASA's [Open Source Science for Data Processing and Archives Workshop](https://www.openscapes.org/blog/2021/10/18/nasa-open-source-science/), __verification__ and __validation__ is a key component of building trust in scientific software. They contribute towards effective __transparency__ and __reproducibility__.\n",
    "![Trust in scientific software](img/TrustScientificSoftware_NASA.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "348d4e60-ad01-45a6-8fcf-7e7443043bbd",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Open Science and Reproducibility\n",
    "Including tools, methods, and documentation for validating software can meaningfully build greater __trust in the accuracy__ of scientific codes and __enhance reproducibility__.\n",
    "![Reproducibility Spectrum](img/ReproducibilitySpectrum.png)\n",
    "From Altuna Akalin, [Scientific Data Analysis Pipelines and Reproducibility](https://towardsdatascience.com/scientific-data-analysis-pipelines-and-reproducibility-75ff9df5b4c5)\n",
    "\n",
    "Promoting and utilizing open science best practices, including making data, code, documentation, and associated tools like validation suites open source, tends to lead to increased recognition and citation rates. See [How Open Science Helps Researchers Succeed](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4973366/) (McKiernan, et. al., NIH eLife) and [Reproducibility in Scientific Software](https://www.osti.gov/servlets/purl/1525948) (Heroux, PASC18)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56673f67-12cb-45d6-a9f6-aea341492f9d",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Validation and the Perils of Software Bug Mismanagment\n",
    "Invariably, __bugs WILL be added to your code__. On average in industry, this can range from 1-25 errors per 1,000 lines of code. Recounting from [Testing of HPC Scientific Software](https://figshare.com/articles/presentation/Testing_of_HPC_Scientific_Software-_Part_1/6453017) (ISC 2018, Anshu Dubey), let's look at the case of modeling protein structures by Geoffrey Chang.\n",
    "\n",
    "* New code inadvertently transposed columns of data for an electron-density map\n",
    "* Model code then produced an incorrect protein structure\n",
    "* Led to the retraction of 5 publications, one with 364 citations\n",
    "* Other papers and grants that conflicted with this result were rejected\n",
    "* Chang did find and report the error himself\n",
    "\n",
    "Being able to catch errors in code through __unit testing__ or other means is vitally important to avoid such issues."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ed5ee63-4977-4d56-a62b-310cfe1db4b3",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "PCAST does not specifically do \"unit testing\" but learn more about the broader landscape of software reliability under the scope of research software via the [Better Scientific Software](https://bssw.io/) (BSSw) organization:\n",
    "* BSSW's [Better Reliability blog post category](https://bssw.io/items?category=better-reliability)\n",
    "* BSSW's [Software Verification blog post](https://bssw.io/blog_posts/software-verification)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bbf3b06-f85d-4417-9ca8-1c1407480789",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Challenges of Validating Scientific Software\n",
    "Achieving appropriate validation workflows is often not easy, particularly in the case of constantly changing scientific software. The paper [Challenges for Verifying and Validating Scientific Software in Computational Materials Science](https://arxiv.org/abs/1906.09179) by Vogel, et. al. at Humboldt University and DLR highlights common issues while implementing testing frameworks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a795a6fa-1c25-449b-afcb-92f409952bba",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "1. Lack of Precise Oracles\n",
    "    * Knowing the precise output of a science/engineering model is typically impossible a priori\n",
    "    * IEEE floating point computations are inexact and change depending on operation ordering and type\n",
    "    * See [What every computer scientist should know about floating-point arithmetic](https://dl.acm.org/doi/10.1145/103162.103163) by David Goldberg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec1c3c4-2b18-4b7e-bfa3-b44c07760505",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "2. Large Configuration Space\n",
    "    * Experimental nature of scientific software promotes the selection of many different algorithms and approaches to problem solving"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "238ae9e5-8665-448a-b85f-ded4918cdbba",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "3. Large-Scale, Heterogeneous Data\n",
    "    * Selecting test and validation data at pre- and post-processing steps results in high data variablilty\n",
    "    * Test data is then inherently cumbersome and expensive to manage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67f59a68-93f2-4ecd-925c-2d08fe989f00",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "4. Global Software Development\n",
    "    * Modern large scale software projects, like in climate science, are difficult to manage across global teams\n",
    "    * Standardizing testing frameworks across disparate teams is a political process"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035b6af5-4c1c-4c6a-9b42-eb4eaf7df984",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## PCAST: Parallel Compiler Assisted Software Testing\n",
    "\n",
    "The PCAST tool serves as a __convenient compiler aided framework__ for quickly building in some level of __software testing__ into your development workflow, particularly to __compare CPU data to GPU data__.\n",
    "\n",
    "However, PCAST won't enable all the benefits of software validation or resolve all the challenges previously discussed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "576cb4ae-8ff2-4234-9dde-3a06a20e5c6c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Primarily, PCAST allows for you to check that...\n",
    "\n",
    "1. Output from previously run code is similar to or equal to the output of a minimally modified code\n",
    "2. Calculations performed on the CPU is similar to or equal to calulations performed on the GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a35a11d5-e9b8-4ecd-9852-b5eaf81e5e6f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### GPU Program Execution - Normal\n",
    "![Normal GPU Program](img/CPUtoGPU_normal.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd49dd02-7ea8-40d9-a33b-8fd78b0a5824",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### GPU Program Execution - PCAST\n",
    "![PCAST GPU Program](img/CPUtoGPU_PCAST.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ae967af-8ba7-4b02-81e5-32f389501137",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Usage of PCAST\n",
    "The main documentation for using PCAST can be found within NVIDIA's HPC SDK Documentation, [HPC Compiler's User Guide](https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#pcast). Alternatively, the NVIDIA blog post [Detecting Divergence Using PCAST to Compare CPU to GPU Results](https://developer.nvidia.com/blog/detecting-divergence-using-pcast-to-compare-gpu-to-cpu-results/) may be referenced as a more casual read.\n",
    "\n",
    "__Usage with a Golden File__:    Run PCAST using calls to `pcast_compare` or `!$nvf compare()` directives (latter requires compiler flag `-Mpcast`) for CPU resident comparisons to a __golden file__.\n",
    "\n",
    "__Usage with OpenACC__:    Run PCAST using calls to `acc_compare` or `!$acc compare()` directives alongside usage of compiler flag `-gpu=redundant` or `-gpu=autocompare`.\n",
    "\n",
    "Additional options can be set using the `PCAST_COMPARE={option-list}` environment variable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f208ea7-76f9-4d7f-9dce-f89447b87b2f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Usage of PCAST - Golden File\n",
    "In this first approach, PCAST can compare successive program runs against a ground truth golden file. Essentially, CPU results will be compared to CPU results, with the assumption that the results in the golden file are correct. \n",
    "\n",
    "NOTE: It's up to the programmer to determine if the results are in fact correct according to the model.\n",
    "\n",
    "With calls to `pcast_compare(...)` or `!$nvf compare(var-list)` directives, a data file named `pcast_compare.dat` by default will either be created if it does not exist or read to compare computed data with saved data from the data file. If using directives, PCAST will only be enabled if the program is compiled with the `-Mpcast` flag.\n",
    "\n",
    "Notably, the directive is much more portable and significantly easier to use, ie `!$nvf compare(a(1:N))`, only requiring the input to be a `var-list` with an inferred full size of the array or specified sub-slices."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c75e2e2f-35df-4970-92dc-75e5a3bffa15",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### `pcast_compare()` Prototype\n",
    "Example arguments for the `pcast_compare(...)` function is given below, where the last 4 arguments are flexible to the descriptiveness required:\n",
    "\n",
    "`pcast_compare(state,\"real(8)\",(2*hs+nx)*(2*hs+nz)*NUM_VARS,\"state\",\"miniweather_orig.F90\",\"main\",155)`\n",
    "\n",
    "1. The address of the data to be saved or compared.\n",
    "2. A string containing the data type, ie `real(2,4,8)` `integer(2,4,8)` `complex(4,8)`\n",
    "3. The number of elements to compare.\n",
    "4. A string treated as the variable name.\n",
    "5. A string treated as the source file name.\n",
    "6. A string treated as the function name.\n",
    "7. An integer treated as a line number."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab432178-ea0e-4b8e-b544-f72b94cee80b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Additional PCAST Options\n",
    "Additonal options can be supplied to the PCAST runtime by modifying the `PCAST_COMPARE={option-list}` environment variable.\n",
    "\n",
    "* `datafile=\"name.dat\"`: Change the name of the golden file\n",
    "* `create`: Explicitly force the creation of the golden file\n",
    "* `compare`: Explicitly force the comparison to the golden file\n",
    "* `disable`: Disable any PCAST actions from taking place and force PCAST functions to immediately return with no effect. This does not disable `-gpu=redundant` execution if enabled for OpenACC codes\n",
    "\n",
    "All `PCAST_COMPARE` environmental variable options can be found in the [NVIDIA Documentation](https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#pcast-env-vars)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee4a2479-501b-4d30-b7c5-47a0b1907b30",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## EXERCISE: Usage of PCAST - Create a Golden File\n",
    "We will first use the original OpenACC version of [MiniWeather](https://github.com/mrnorman/miniWeather/blob/master/fortran/miniWeather_mpi_openacc.F90) in order to create a __golden file__ from the `openacc_orig` executable. To do this, edit the file [miniWeather_mpi_openacc_orig.F90](miniWeather_mpi_openacc_orig.F90) and add either `call pcast_compare()` or directives `!$nvf compare()` where desired. If you use the function call, make sure to add `use openacc` to head of program (already done) and follow the function prototype given previously.\n",
    "\n",
    "A simple recommendation is to save results of the `state` variable into the golden file near the end of program execution. Of course, additional variables could be saved but keep in mind, if `pcast_compare()` or its associated directive is called too many times across program execution, the golden file can grow to very large sizes.\n",
    "\n",
    "Then, run the next two cells to compile and run the code on the GPU. For future exercises, be sure to keep the configuration of the model consistent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "255c79ed-4c01-4846-b273-4699c02f4e87",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export OPENACC_FLAGS=\"-acc -gpu=cc60,cc70 -Mpcast\"\n",
    "\n",
    "mpif90 -I${PNETCDF_INC} -Mextend -O0 -DNO_INFORM -c miniWeather_mpi_openacc_orig.F90 -o miniWeather_mpi_openacc_orig.F90.o \\\n",
    "-D_NX=200 -D_NZ=100 -D_SIM_TIME=2.01 -D_OUT_FREQ=1 -D_DATA_SPEC=DATA_SPEC_THERMAL ${OPENACC_FLAGS}\n",
    "\n",
    "mpif90 -Mextend -O3 -DNO_INFORM miniWeather_mpi_openacc_orig.F90.o -o openacc_orig -L${PNETCDF_LIB} -lpnetcdf ${OPENACC_FLAGS}\n",
    "rm -f miniWeather_mpi_openacc_orig.F90.o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f8ce88b-a2b0-4b98-84e9-86c4a6a8de43",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export PCAST_COMPARE=\"create\"\n",
    "qcmd -A $PROJECT -q $QUEUE -l select=1:ncpus=1:ngpus=1 -l gpu_type=$GPU_TYPE -l walltime=60 -v PCAST_COMPARE -- \\\n",
    "cd $PWD && ./openacc_orig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5ff5981-1dc0-4761-ba1c-a94ad81ed022",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## EXERCISE: Usage of PCAST - Test against a Golden File\n",
    "Add the same `pcast_comapre()` calls or directives to the non-original file [miniWeather_mpi_openacc.F90](miniWeather_mpi_openacc.F90). A bug has been introduced somewhere in the code.\n",
    "\n",
    "Make and run the non-original file and __use the PCAST report to try to determine where the bug is__.\n",
    "\n",
    "1. __Would testing additional variables via PCAST across the MiniWeather code fascilitate this search better?__\n",
    "2. __What would be an easier way to find the source of the bug?__ Hint: Use `diff` command.\n",
    "3. __How would you incorporate PCAST testing in a development workflow to minimize creation of bugs as changes are made to source files?__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef3cf348-561c-4844-be56-3860e3be942b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export OPENACC_FLAGS=\"-acc -gpu=cc60,cc70 -Mpcast\"\n",
    "\n",
    "mpif90 -I${PNETCDF_INC} -Mextend -O0 -DNO_INFORM -c miniWeather_mpi_openacc.F90 -o miniWeather_mpi_openacc.F90.o \\\n",
    "-D_NX=200 -D_NZ=100 -D_SIM_TIME=2.01 -D_OUT_FREQ=1 -D_DATA_SPEC=DATA_SPEC_THERMAL ${OPENACC_FLAGS}\n",
    "\n",
    "mpif90 -Mextend -O3 -DNO_INFORM miniWeather_mpi_openacc.F90.o -o openacc -L${PNETCDF_LIB} -lpnetcdf ${OPENACC_FLAGS}\n",
    "rm -f miniWeather_mpi_openacc.F90.o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "471cdf2c-9b37-4d64-b32e-0cad847db921",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export PCAST_COMPARE=\"compare,summary\"\n",
    "qcmd -A $PROJECT -q $QUEUE -l select=1:ncpus=1:ngpus=1 -l gpu_type=$GPU_TYPE -l walltime=60 -v PCAST_COMPARE -- \\\n",
    "cd $PWD && ./openacc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7941d0b-6d08-4b41-b729-405876d99dfe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Usage of PCAST - OpenACC and `-gpu=redundant` / `-gpu=autocompare`\n",
    "In this second aproach, PCAST can be run to directly compare the calculations between the CPU and GPU. \n",
    "\n",
    "In `redundant` mode, CPU code is generated alongside GPU code and ran redundantly. Then, every time a `call acc_compare()` or `!$acc compare()` directive is encountered, the compiler will verify the values between any specified variables.\n",
    "\n",
    "In `autocompare` mode, CPU code is again generated alongside GPU code and ran reduntantly. Then, every time data is moved between CPU and GPU, such as via an `!$acc update host()` directive or at the edges of data regions, the compiler will verify the values between the data that was to be moved.\n",
    "\n",
    "For a comprehensive comparison of all GPU resident data, you can also use `call acc_compare_all()` or `!$acc compare all`.\n",
    "\n",
    "`autocompare` is the easiest method to quickly test correctness between CPU and GPU code particularly when using OpenACC. The flag automaticaly implies `redundant`. Since the source file stays the same, ie directive comments are ignored when generating CPU code, this can highlight if the OpenACC runtime is introducing any divergence in the GPU target code either due to inappropriate loop directives, ie missing `private()` or `reduction()`, or bad managagement of data movement."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7c766a-8e77-497a-b0a5-ff8af92b89fb",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Additional PCAST Options\n",
    "Additonal options can be supplied to the PCAST runtime by modifying the `PCAST_COMPARE={option-list}` environment variable. These options are also relevant for golden file mode.\n",
    "\n",
    "* `outputfile=\"name.dat\"`: Specify the file to write comparison output. Default is `stderr`\n",
    "* `summary`: Print summary of comparisons at the end of execution\n",
    "* `abs=n`, `rel=n`, `ulp=n`, or `ieee`:  Specify the types and tolerance of comparisons performed, where `n` is the magnitude of relative 10^n, absolute 10^n, or number of units precision difference tolerated respectively. Add `ieee` to enable NaN checks\n",
    "* `report=n`: Modify the default number (50) differences reported at each comparison where `n` is the number of differences to report\n",
    "* `stop`: Stop at the first difference outside of tolerance\n",
    "\n",
    "All `PCAST_COMPARE` environmental variable options can be found in the [NVIDIA Documentation](https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#pcast-env-vars)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92adbb48-0629-487c-ae7c-6fb10cb0fa0e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## EXERCISE: Usage of PCAST - OpenACC and `-gpu=redundant` / `-gpu=autocompare`\n",
    "Try out PCAST with `autocompare`. Initially, you will notice that many \"errors\" are reported. However, all of them are within machine precision error. Add the `PCAST_COMPARE` flag `abs=12` in order to tolerate errors within a reasonable bounds. If you like, add additional `!$acc update host()` directives in different parts of MiniWeather to automatically compare in additional regions.\n",
    "\n",
    "1. __How does this impact your confidence in the correctness of the GPU code?__\n",
    "2. __Try adding the `tile(32,32,NUM_VARS)` clause like in a previous OpenACC exercise that produced incorrect results. Can you verify the incorrect results more clearly with PCAST?__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de7fd3f0-ab99-43a8-bd4f-76c8990966af",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export OPENACC_FLAGS=\"-acc -gpu=autocompare,cc60,cc70\"\n",
    "\n",
    "mpif90 -I${PNETCDF_INC} -Mextend -O0 -DNO_INFORM -c miniWeather_mpi_openacc.F90 -o miniWeather_mpi_openacc.F90.o \\\n",
    "-D_NX=1024 -D_NZ=512 -D_SIM_TIME=200.01 -D_OUT_FREQ=20 -D_DATA_SPEC=DATA_SPEC_THERMAL ${OPENACC_FLAGS}\n",
    "\n",
    "mpif90 -Mextend -O3 -DNO_INFORM miniWeather_mpi_openacc.F90.o -o openacc -L${PNETCDF_LIB} -lpnetcdf ${OPENACC_FLAGS}\n",
    "rm -f miniWeather_mpi_openacc.F90.o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adad05dd-6180-4d13-8468-361d702c3c69",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export PCAST_COMPARE=\"summary,report=2,stop\"\n",
    "qcmd -A $PROJECT -q $QUEUE -l select=1:ncpus=1:ngpus=1 -l gpu_type=$GPU_TYPE -l walltime=60 -v PCAST_COMPARE -- \\\n",
    "$PWD/openacc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7990a65-f240-45c7-a668-530a54271620",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Final Points\n",
    "1. __PCAST is not able to verify that the original CPU code is correct__ accoring to the science or model specification.\n",
    "2. However, you can __use PCAST to verify that CPU+GPU results stay consistent__ across minor refactoring edits using a golden file.\n",
    "3. __PCAST can verify that the results computed between CPU and GPU are in agreement and correct__, up to machine precision error.\n",
    "    * Differences between how GPU and CPU code is compiled can still introduce machine precision error.\n",
    "4. Implementing some form of __validation and verification in a development workflow establishes greater trust in the software and minimizes bugs__."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80240bea-4ca0-4e35-aee5-3f44b637bbe6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Suggested Resources\n",
    "* [HPC Compiler's User Guide - PCAST](https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#pcast)\n",
    "* [Detecting Divergence Using PCAST to Compare CPU to GPU Results](https://developer.nvidia.com/blog/detecting-divergence-using-pcast-to-compare-gpu-to-cpu-results/) - Wolfe & Smith, NVIDIA\n",
    "* [Challenges for Verifying and Validating Scientific Software in Computational Materials Science](https://arxiv.org/abs/1906.09179) - Vogel, et. al.\n",
    "* [Testing of HPC Scientific Software](https://figshare.com/articles/presentation/Testing_of_HPC_Scientific_Software-_Part_1/6453017) - ISC 2018, Anshu Dubey\n",
    "* BSSW's [Better Reliability blog post category](https://bssw.io/items?category=better-reliability)\n",
    "* BSSW's [Software Verification blog post](https://bssw.io/blog_posts/software-verification)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}