Merge pull request #79 from ESMValGroup/development

Release v2.0.0a1

Author: mattiarighi

README.md

@@ -1,38 +1,68 @@
 # ESMValTool Core package
+
 [![Documentation Status](https://readthedocs.org/projects/esmvaltool/badge/?version=version2_development)](https://esmvaltool.readthedocs.io/en/version2_development/?badge=version2_development)
 [![DOIBadge](https://img.shields.io/badge/DOI-10.17874%2Fac8548f0315-blue.svg)](https://doi.org/10.17874/ac8548f0315)
 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/ESMValGroup?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 [![CircleCI](https://circleci.com/gh/ESMValGroup/ESMValCore.svg?style=svg)](https://circleci.com/gh/ESMValGroup/ESMValCore)
-[![Codacy Coverage Badge](https://api.codacy.com/project/badge/Coverage/79bf6932c2e844eea15d0fb1ed7e415c)](https://www.codacy.com/app/ESMValGroup/ESMValCore?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValCore&utm_campaign=Badge_Coverage)
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/79bf6932c2e844eea15d0fb1ed7e415c)](https://www.codacy.com/app/ESMValGroup/ESMValTool?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValTool&utm_campaign=Badge_Grade)
+[![Codacy Badge](https://api.codacy.com/project/badge/Coverage/5d496dea9ef64ec68e448a6df5a65783)](https://www.codacy.com/app/ESMValGroup/ESMValCore?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValCore&utm_campaign=Badge_Coverage)
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/5d496dea9ef64ec68e448a6df5a65783)](https://www.codacy.com/app/ESMValGroup/ESMValCore?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValCore&utm_campaign=Badge_Grade)
 [![Docker Build Status](https://img.shields.io/docker/build/esmvalgroup/esmvaltool.svg)](https://hub.docker.com/r/esmvalgroup/esmvaltool/)
 [![Anaconda-Server Badge](https://anaconda.org/esmvalgroup/esmvalcore/badges/installer/conda.svg)](https://conda.anaconda.org/esmvalgroup)
 
-
-ESMValTool Core: core functionalities for the ESMValTool, a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
+ESMValTool Core: core functionalities for the ESMValTool, a community
+diagnostic and performance metrics tool for routine evaluation of Earth System
+Models in CMIP.
 
 # Getting started
-This is the development branch for ESMValTool Core. ESMValTool version 2 is under rapid development, an installation from source is recommended at the moment.
-
-## Installing from source [recommended]
-Please see [CONTRIBUTING.md](https://github.com/ESMValGroup/ESMValCore/blob/development/CONTRIBUTING.md) for instructions on installing ESMValTool from source.
 
 ## Installing from Anaconda
+
 The Anaconda package can be found on [ESMValGroup Anaconda Channel.](https://anaconda.org/ESMValGroup)
 
-If you already installed Anaconda, you can install ESMValTool by running:
-```
-conda install -c esmvalgroup -c conda-forge esmvalcore
-```
+If you already have
+[Anaconda installed](https://conda.io/projects/conda/en/latest/user-guide/install/index.html),
+you can install ESMValTool Core by running:
+
+    conda install -c esmvalgroup -c conda-forge esmvalcore
+
+## Installing from source
+
+If you need the very latest features, please see
+[CONTRIBUTING.md](https://github.com/ESMValGroup/ESMValCore/blob/development/CONTRIBUTING.md)
+for instructions on installing ESMValTool Core from source.
+
+## Using ESMValTool Core as a Python library
+
+The ESMValTool Core package provides various functions for:
+
+-   Finding data in a directory structure typically used for CMIP data
+
+-   Reading CMOR tables and using those to check model and observational data.
+
+-   ESMValTool preprocessor functions based on
+    [iris](https://scitools.org.uk/iris/docs/latest/) for e.g. regridding,
+    vertical interpolation, statistics, correcting (meta)data errors, extracting
+    a time range, etcetera.
 
 ## Running ESMValTool
-- Review `config-user.yml`. To customize for your system, create a copy, edit and use the command line option `-c` to instruct `esmvaltool` to use your custom configuration.
-- Optionally install the ESMValTool recipes and diagnostics
-- Run e.g. `esmvaltool -c ~/config-user.yml examples/recipe_python.yml
+
+-   Review `config-user.yml`. To customize for your system, create a copy, edit
+    and use the command line option `-c` to instruct `esmvaltool` to use your
+    custom configuration file.
+
+-   Install the [ESMValTool](https://github.com/ESMValGroup/ESMValTool)
+    to run [ESMValTool recipes and diagnostics](https://esmvaltool.readthedocs.io/en/latest/recipes/index.html)
+
+-   Run e.g. `esmvaltool -c ~/config-user.yml examples/recipe_python.yml` after
+    downloading the necessary data.
 
 ## Getting help
-The easiest way to get help if you cannot find the answer in the documentation on [readthedocs](https://esmvaltool.readthedocs.io), is to open an [issue on GitHub](https://github.com/ESMValGroup/ESMValCore/issues).
+
+The easiest way to get help if you cannot find the answer in the documentation
+on [readthedocs](https://esmvaltool.readthedocs.io), is to open an
+[issue on GitHub](https://github.com/ESMValGroup/ESMValCore/issues).
 
 ## Contributing
-If you would like to contribute a new diagnostic or feature, please have a look at [CONTRIBUTING.md](https://github.com/ESMValGroup/ESMValCore/blob/development/CONTRIBUTING.md).
 
+If you would like to contribute a new feature, please have a
+look at [CONTRIBUTING.md](https://github.com/ESMValGroup/ESMValCore/blob/development/CONTRIBUTING.md).

doc/sphinx/source/esmvalcore/preprocessor.inc

@@ -131,7 +131,7 @@ The _area.py module contains the following preprocessor functions:
 
 * extract_region: Extract a region from a cube based on lat/lon corners.
 * zonal_means: Calculates the zonal or meridional means.
-* average_region: Calculates the average value over a region.
+* area_statistics: Calculates the average value over a region.
 * extract_named_regions: Extract a specific region from in the region cooordinate.
 
 
@@ -166,26 +166,24 @@ This function takes two arguments:
 See also :func:`esmvalcore.preprocessor.zonal_means`.
 
 
-3. average_region
+3. area_statistics
 -----------------
 
 This function calculates the average value over a region - weighted by the
 cell areas of the region.
 
-This function takes three arguments:
-coord1: the name of the coordinate in the first direction.
-coord2: the name of the coordinate in the second dimension.
-operator: the name of the operation to apply (default: mean).
+This function takes the argument,
+operator: the name of the operation to apply.
 
-While this function is named `average_region`, it can be used to apply several
+This function can be used to apply several
 different operations in the horizonal plane: mean, standard deviation, median
 variance, minimum and maximum.
 
 Note that this function is applied over the entire dataset. If only a specific
 region, depth layer or time period is required, then those regions need to be
 removed using other preprocessor operations in advance.
 
-See also :func:`esmvalcore.preprocessor.average_region`.
+See also :func:`esmvalcore.preprocessor.area_statistics`.
 
 
 4. extract_named_regions
@@ -205,7 +203,7 @@ Volume manipulation
 The _volume.py module contains the following preprocessor functions:
 
 * extract_volume: Extract a specific depth range from a cube.
-* average_volume: Calculate the volume-weighted average.
+* volume_statistics: Calculate the volume-weighted average.
 * depth_integration: Integrate over the depth dimension.
 * extract_transect: Extract data along a line of constant latitude or longitude.
 * extract_trajectory: Extract data along a specified trajectory.
@@ -225,19 +223,19 @@ negative, then z_min and z_max need to be negative numbers.
 See also :func:`esmvalcore.preprocessor.extract_volume`.
 
 
-2. average_volume
+2. volume_statistics
 -----------------
 
 This function calculates the volume-weighted average across three dimensions,
 but maintains the time dimension. The following arguments are required:
 
-coord1: the name of the coordinate in the first direction.
-coord2: the name of the coordinate in the second dimension.
+This function takes the argument: operator, which defines the 
+operation to apply over the volume.
 
 No depth coordinate is required as this is determined by iris. This
 function works best when the fx_files provide the cell volume.
 
-See also :func:`esmvalcore.preprocessor.average_volume`.
+See also :func:`esmvalcore.preprocessor.volume_statistics`.
 
 
 3. depth_integration
@@ -294,15 +292,21 @@ Vertical interpolation
 ======================
 Documentation of _regrid.py (part 1)
 
-Land/Sea/Ice Masking
-====================
+Masking
+=======
 Documentation of _mask.py (part 1)
 
+1. Introduction to masking
+---------------------------
+
 Certain metrics and diagnostics need to be computed and performed on restricted regions of the Globe; ESMValTool supports subsetting the input data on land mass, oceans and seas, ice. This is achived by masking the model data and keeping only the values associated with grid points that correspond to e.g. land mass
 or oceans and seas; masking is done either by using standard mask files that have the same grid resolution as the model data (these files are usually produced
 at the same time with the model data and are called fx files) or, in the absence of these files, by using Natural Earth masks. Natural Earth masks, even if they are not model-specific, represent a good approximation since their grid resolution is almost always much higher than the model data, and they are constantly updated with changing
 geographical features.
 
+2. Land-sea masking
+-------------------
+
 In ESMValTool v2 land-seas-ice masking can be done in two places: in the preprocessor, to apply a mask on the data before any subsequent preprocessing step, and before
 running the diagnostic, or in the disgnostic phase. We present both these implementations below.
 
@@ -321,6 +325,9 @@ kept in the data after applying the mask; conversely, it will retrieve the `fx:
 type of files; observational data is missing them altogether), then the tool attempts to mask using Natural Earth mask files (that are vectorized rasters).
 Note that the resolutions for the Natural Earth masks are much higher than any usual CMIP model: 10m for land and 50m for ocean masks.
 
+3. Ice masking
+---------------
+
 Note that for masking out ice the preprocessor is using a different function, this so that both land and sea or ice can be masked out without
 losing generality. To mask ice out one needs to add the preprocessing step much as above:
 
@@ -334,6 +341,9 @@ losing generality. To mask ice out one needs to add the preprocessing step much
 To keep only the ice, one needs to mask out landsea, so use that as value for mask_out. As in the case of mask_landsea, the tool will automatically
 retrieve the `fx: sftgif` file corresponding the the used variable and extract the ice mask from it.
 
+4. mask files
+--------------
+
 At the core of the land/sea/ice masking in the preprocessor are the mask files (whether it be fx type or Natural Earth type of files); these files (bar Natural Earth)
 can be retrived and used in the diagnostic phase as well or solely. By specifying the `fx_files:` key in the variable in diagnostic in the recipe, and populating it
 with a list of desired files e.g.:
@@ -356,6 +366,42 @@ the 'config' diagnostic variable items e.g.:
         sftlf_file = attributes['fx_files']['sftlf']
         areacello_file = attributes['fx_files']['areacello']
 
+5. Missing values masks
+-----------------------
+
+Missing (masked) values can be a nuisance especially when dealing with multimodel ensembles and having to compute
+multimodel statistics; different numbers of missing data from dataset to datest may introduce biases and artifically
+assign more weight to the datasets that have less missing data. This is handled in ESMValTool via the missing values
+masks: two types of such masks are available: one for the multimodel case and another for the single model case.
+
+The multimodel missing values mask (mask_fillvalues) is a preprocessor step that usually comes after all the single-model
+steps (regridding, area selection etc) have been performed; in a nutshell, it combines missing values masks from individual
+models into a multimodel missing values mask; the individual model masks are built according to common criteria:
+the user chooses a time window in which missing data points are counted, and if the number of missing data points relative
+to the number of total data points in a window is less than a chosen fractional theshold, the window is discarded i.e.
+all the points in the window are masked (set to missing).
+
+.. code-block:: bash
+
+    preprocessors:
+      missing_values_preprocessor:
+        mask_fillvalues:
+          threshold_fraction: 0.95
+          min_value: 19.0
+          time_window: 10.0
+
+In the example above, the fractional threshold for missing data vs. total data is set to 95% and the time window is set to
+10.0 (units of the time coordinate units). Optionally, a minimum value threshold can be applied, in this case it is set
+to 19.0 (in units of the variable units).
+
+A similar preprocessor step exists for the single-dataset: mask_window_threshold (with the same arguments as mask_fillvalues).
+
+6. Min, max and interval masking
+--------------------------------
+
+Thresholding on minimum and maximum accepted data values can also be performed: masks are constructed based on the
+results of thresholding; inside and outside interval thresholding and masking can also be performed. These functions
+are mask_above_threshold, mask_below_threshold, mask_inside_range, and mask_outside_range.
 
 Horizontal regridding
 =====================

doc/sphinx/source/recipes/recipe_oceans.rst

@@ -488,9 +488,8 @@ An appropriate preprocessor for a 3D+time field would be:
 	      lat2:  30.
 	      z_min: 0.
 	      z_max: 3000.
-	    average_region:
-	      coord1: longitude
-	      coord2: latitude
+	    area_statistics:
+              operator: mean
 
 
 
@@ -524,18 +523,15 @@ For a global area-weighted average 2D field:
 
   .. code-block:: yaml
 
-	average_area:
-	  coord1: longitude
-	  coord2: latitude
+	area_statistics:
+          operator: mean
 
 For a global volume-weighted average 3D field:
 
   .. code-block:: yaml
 
-	average_volume:
-	  coord1: longitude
-	  coord2: latitude
-	  coordz: depth
+	volume_statistics:
+          operator: mean
 
 For a global area-weighted surface of a 3D field:
 
@@ -544,9 +540,8 @@ For a global area-weighted surface of a 3D field:
 	extract_levels:
 	  levels: [0., ]
 	  scheme: linear_horizontal_extrapolate_vertical
-	average_area:
-	  coord1: longitude
-	  coord2: latitude
+	area_statistics:
+          operator: mean
 
 
 An example of the multi-model time series plots can seen here:

doc/sphinx/source/requirements.txt

@@ -10,7 +10,6 @@ prov[dot]
 psutil
 pyyaml
 shapely
-six
 xarray
 yamale
 sklearn

esmvalcore/_config.py

@@ -4,9 +4,7 @@
 import logging.config
 import os
 import time
-from distutils.version import LooseVersion
 
-import iris
 import yaml
 
 from .cmor.table import read_cmor_tables
@@ -29,11 +27,6 @@ def find_diagnostics():
 DIAGNOSTICS_PATH = find_diagnostics()
 
 
-def use_legacy_iris():
-    """Return True if legacy iris is used."""
-    return LooseVersion(iris.__version__) < LooseVersion("2.0.0")
-
-
 def read_config_user_file(config_file, recipe_name):
     """Read config user file and store settings in a dictionary."""
     with open(config_file, 'r') as file:

esmvalcore/_data_finder.py

@@ -9,8 +9,6 @@
 import os
 import re
 
-import six
-
 from ._config import get_project_config, replace_mip_fx
 from .cmor.table import CMOR_TABLES
 
@@ -177,7 +175,7 @@ def _select_drs(input_type, drs, project):
     """Select the directory structure of input path."""
     cfg = get_project_config(project)
     input_path = cfg[input_type]
-    if isinstance(input_path, six.string_types):
+    if isinstance(input_path, str):
         return input_path
 
     structure = drs.get(project, 'default')

esmvalcore/_recipe.py

@@ -413,7 +413,7 @@ def _update_fx_settings(settings, variable, config_user):
             settings['mask_landseaice']['fx_files'].append(
                 fx_files_dict['sftgif'])
 
-    for step in ('average_region', 'average_volume'):
+    for step in ('area_statistics', 'volume_statistics'):
         if settings.get(step, {}).get('fx_files'):
             settings[step]['fx_files'] = get_input_fx_filelist(
                 variable=variable,
@@ -431,7 +431,7 @@ def _read_attributes(filename):
 
     with Dataset(filename, 'r') as dataset:
         for attr in dataset.ncattrs():
-            attributes[attr] = getattr(dataset, attr)
+            attributes[attr] = dataset.getncattr(attr)
     return attributes
 
 

esmvalcore/_version.py

@@ -1,2 +1,2 @@
 """ESMValCore version."""
-__version__ = '2.0.0a0'
+__version__ = '2.0.0a1'

esmvalcore/cmor/check.py

@@ -70,7 +70,8 @@ def __init__(self,
         self.automatic_fixes = automatic_fixes
 
     def check_metadata(self, logger=None):
-        """Check the cube metadata.
+        """
+        Check the cube metadata.
 
         Perform all the tests that do not require to have the data in memory.