Merge pull request #68 from andersy005/feature/era5

Add support for ERA5 Reanalysis
intake · May 10, 2019 · 9ed38df · 9ed38df
2 parents fafd02e + 3684f93
commit 9ed38df
Show file tree

Hide file tree

Showing 17 changed files with 177,082 additions and 715 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -12,6 +12,7 @@ Features
 - Add implementation for The Gridded Meteorological Ensemble Tool (GMET) data holdings (:pr:`61`) `Anderson Banihirwe`_
 - Allow users to specify exclude_dirs for CMIP collections (:pr:`63`) & (:issue:`62`) `Anderson Banihirwe`_
 - Keep CMIP6 ``tracking_id`` in merge_keys (:pr: `67`) `Anderson Banihirwe`_
+- Add implementation for ERA5 datasets (:pr: `68`) `Anderson Banihirwe`_
 
 
 Intake-esm v2019.4.26 (2019-04-26)

diff --git a/README.rst b/README.rst
@@ -1,6 +1,6 @@
-===============================
+===========
 Intake-esm
-===============================
+===========
 
 .. image:: https://img.shields.io/circleci/project/github/NCAR/intake-esm/master.svg?style=for-the-badge&logo=circleci
     :target: https://circleci.com/gh/NCAR/intake-esm/tree/master
@@ -29,7 +29,8 @@ Intake-esm supports data holdings from the following projects:
 - CMIP: Coupled Model Intercomparison Project (phase 5 and phase 6)
 - CESM: Community Earth System Model Large Ensemble (LENS), and Decadal Prediction Large Ensemble (DPLE)
 - MPI-GE: The Max Planck Institute for Meteorology (MPI-M) Grand Ensemble (MPI-GE)
-- GMET: The Gridded Meteorological Ensemble Tool
+- GMET: The Gridded Meteorological Ensemble Tool data
+- ERA5: ECWMF ERA5 Reanalysis dataset stored on NCAR's GLADE in ``/glade/collections/rda/data/ds630.0``
 
 See documentation_ for more information.
 

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -4,7 +4,7 @@ Intake-esm
 
 
 `Intake-esm` provides functionality to build and load `intake <https://intake.readthedocs.io/en/latest/>`_ catalogs
-for earth system data holdings produced by projects such as CMIP, CESM Large ensemble.
+for earth system data holdings produced by projects such as CMIP, CESM.
 
 
 Feedback
@@ -24,6 +24,7 @@ please open an issue at the GitHub `main repository <http://github.com/NCAR/inta
    notebooks/cesm.ipynb
    notebooks/mpige.ipynb
    notebooks/gmet.ipynb
+   notebooks/era5.ipynb
    api
    contributing
    changelog

diff --git a/docs/source/notebooks/cesm.ipynb b/docs/source/notebooks/cesm.ipynb
diff --git a/docs/source/notebooks/cmip5.ipynb b/docs/source/notebooks/cmip5.ipynb
diff --git a/docs/source/notebooks/era5.ipynb b/docs/source/notebooks/era5.ipynb
diff --git a/docs/source/notebooks/gmet.ipynb b/docs/source/notebooks/gmet.ipynb
@@ -4,7 +4,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Building a GMET Collection Catalog"
+    "# GMET (Gridded Meteorological Ensemble Tool) Data Holdings\n",
+    "## Building a GMET Collection Catalog"
    ]
   },
   {
@@ -13,11 +14,15 @@
    "source": [
     "Building a GMET (Gridded Meteorological Ensemble Tool) collection catalog follows the same steps that are used when building a CMIP or CESM collection catalog:\n",
     "\n",
-    "- Define a collection catalog in a YAML or nested dictionary\n",
+    "- Define a collection catalog in a YAML or nested dictionary.\n",
     "\n",
-    "- Pass this collection definition to `intake_open_esm_metadatastore()` class.\n",
+    "- Build the collection catalog by passing the collection definition to `intake_open_esm_metadatastore()` class.\n",
     "\n",
-    "- Use the built collection catalog"
+    "- Use the built collection catalog by:\n",
+    "\n",
+    "  - Querying the collection catalog\n",
+    "  - Getting YAML representation of a catalog entry\n",
+    "  - Loading data into an xarray dataset"
    ]
   },
   {
@@ -1307,17 +1312,11 @@
      },
      "metadata": {},
      "output_type": "display_data"
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tqdm/autonotebook/__init__.py:14: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
-      "  \" (e.g. in jupyter console)\", TqdmExperimentalWarning)\n"
-     ]
     }
    ],
    "source": [
+    "import warnings\n",
+    "warnings.simplefilter(\"ignore\") # Silence warnings\n",
     "import intake"
    ]
   },
@@ -1348,21 +1347,11 @@
       "        urlpath: /glade/p/ral/hap/anewman/conus_v1p2/eighth/v2_landmask/\n",
       "        exclude_dirs: ['*/mean/*', '*/summary_files/*']\n"
      ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "bash: module: line 1: syntax error: unexpected end of file\n",
-      "bash: error importing function definition for `BASH_FUNC_module'\n",
-      "bash: ml: line 1: syntax error: unexpected end of file\n",
-      "bash: error importing function definition for `BASH_FUNC_ml'\n"
-     ]
     }
    ],
    "source": [
     "%%bash\n",
-    "cat ./tests/gmet.yml"
+    "cat ../../../tests/gmet.yml"
    ]
   },
   {
@@ -1371,7 +1360,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "cdefinition = \"./tests/gmet.yml\""
+    "cdefinition = \"../../../tests/gmet.yml\""
    ]
   },
   {
@@ -1417,14 +1406,15 @@
     }
    ],
    "source": [
-    "col = intake.open_esm_metadatastore(collection_input_definition=cdefinition, overwrite_existing=True)"
+    "col = intake.open_esm_metadatastore(collection_input_definition=cdefinition, \n",
+    "                                    overwrite_existing=True)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Using the Built Collection\n",
+    "## Using the Built Collection\n",
     "\n",
     "Once the collection catalog is built, we can start querying it and using it to load data into xarray datasets:"
    ]
@@ -1433,6 +1423,15 @@
    "cell_type": "code",
    "execution_count": 5,
    "metadata": {},
+   "outputs": [],
+   "source": [
+    "col = intake.open_esm_metadatastore(collection_name='gmet')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -1580,7 +1579,7 @@
        "4  /glade/p/ral/hap/anewman/conus_v1p2/eighth/v2_...  "
       ]
      },
-     "execution_count": 5,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1589,9 +1588,16 @@
     "col.df.head()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Querying the collection catalog"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1610,7 +1616,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [
     {
@@ -1620,7 +1626,7 @@
        "      dtype=object)"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1629,9 +1635,16 @@
     "cat.query_results.time_range.unique()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Getting YAML representation of a catalog entry"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [
     {
@@ -1642,7 +1655,7 @@
       "  source:\n",
       "  - module: intake_esm.gmet\n",
       "sources:\n",
-      "  gmet_a647a5d1-a812-412c-aa00-89a359ceb9c6:\n",
+      "  gmet_1cf478d6-2f51-4b50-a1e5-c448fefa22bd:\n",
       "    args:\n",
       "      collection_name: gmet\n",
       "      query:\n",
@@ -1682,15 +1695,22 @@
     "print(cat.yaml(True))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Loading data into an xarray dataset"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "590a998df7b34e72a7bad4d8642acbc0",
+       "model_id": "8a351f9b650049109794a245271b52b0",
        "version_major": 2,
        "version_minor": 0
       },
@@ -1715,7 +1735,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [
     {
@@ -1734,15 +1754,15 @@
        "    t_mean     (member_id, time, lat, lon) float32 dask.array<shape=(9, 1096, 224, 464), chunksize=(1, 183, 224, 464)>\n",
        "    t_range    (member_id, time, lat, lon) float32 dask.array<shape=(9, 1096, 224, 464), chunksize=(1, 183, 224, 464)>\n",
        "Attributes:\n",
-       "    history:      2019-04-27 17:21:34.325435 xarray.open_dataset('/glade/p/ra...\n",
-       "    source:       Generated using version 1.1 of CONUS ensemble code base\n",
-       "    title:        CONUS daily 12-km gridded ensemble precipitation and temper...\n",
+       "    NCO:          netCDF Operators version 4.7.4 (http://nco.sf.net)\n",
+       "    history:      2019-05-09 22:02:24.956095 xarray.open_dataset('/glade/p/ra...\n",
        "    institution:  National Center fo Atmospheric Research (NCAR), Boulder, CO...\n",
-       "    references:   Newman et al. 2015: Gridded Ensemble Precipitation and Temp...\n",
-       "    NCO:          netCDF Operators version 4.7.4 (http://nco.sf.net)"
+       "    title:        CONUS daily 12-km gridded ensemble precipitation and temper...\n",
+       "    source:       Generated using version 1.1 of CONUS ensemble code base\n",
+       "    references:   Newman et al. 2015: Gridded Ensemble Precipitation and Temp..."
       ]
      },
-     "execution_count": 10,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1753,7 +1773,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1762,27 +1782,27 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 13,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "intake 0.4.4+130.gd1aa72c\n",
-      "last updated: 2019-04-27 \n",
+      "last updated: 2019-05-09 \n",
       "\n",
       "CPython 3.7.3\n",
       "IPython 7.1.1\n",
       "\n",
       "compiler   : GCC 7.3.0\n",
       "system     : Linux\n",
-      "release    : 3.12.62-60.64.8-default\n",
+      "release    : 3.10.0-693.21.1.el7.x86_64\n",
       "machine    : x86_64\n",
       "processor  : x86_64\n",
       "CPU cores  : 72\n",
       "interpreter: 64bit\n",
-      "Git hash   : 6488abc12b58366318e9c6883369bfdf0acffc3f\n"
+      "Git hash   : 62cd7380ad1660cd134a67e79f8be852cde66ba4\n"
      ]
     }
    ],
@@ -1793,9 +1813,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python [conda env:analysis]",
+   "display_name": "Python 3",
    "language": "python",
-   "name": "conda-env-analysis-py"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {