Patch: Dependency updates (NREL#262)

* address dependency stack items in NREL#259

* reduce configuration file count when toml option is available

* add easier way to run unit and/or regression

* remove optional from required

* restore dask requirement, but through xarray

* update installation instructions

* remove unnecessary requirements file

* remove unnecessary file and fix conditions test in options

* dont' forget to save your work

* ensure all test pathways are found with(out) flags

* update changelog

* add missing

* added h5pyd to examples extra, updated readme

* support python 3.11 and reorganize modifiers

* add note on Python 3.11 recommended install flow

* add additional description

* minor language updates and clarifications

* add in a python 3.11 for bleeding edge tests runner

* reinstate requests as required

* update changelog for python 3.11 support

---------

Co-authored-by: Jordan Perr-Sauer <[email protected]>

Author: RHammond2

.coveragerc

@@ -15,7 +15,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, windows-latest]
-        python-version: [3.9]
+        python-version: [3.9, 3.11]
 
     steps:
     - uses: actions/checkout@v3

.github/workflows/ci-tests.yml

@@ -1,6 +1,18 @@
 # Changelog
 All notable changes to this project will be documented in this file. If you make a notable change to the project, please add a line describing the change to the "unreleased" section. The maintainers will make an effort to keep the [Github Releases](https://github.com/NREL/OpenOA/releases) page up to date with this changelog. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## Unreleased - TBD
+
+- Python 3.11 is now supported.
+- Updates the dependency requirements to minimize the number of required packages, and have a more
+  expansive list of modifiers. Users can now use any combination of
+  `pip install openoa[examples, develop, docs, nrel-wind, reanalysis]` to ensure the appropriate
+  packages are installed for their workflow.
+- Adds a `--unit` and `--regression` flag for running pytest that works in addition to
+`pytest test/unit` or `pytest test/regression`.
+- Converts some configuration files into `pyproject.toml` settings to reduce visual clutter
+  at the top-level of the directory.
+
 ## v3.0 - 29 September 2023
 
 Please see the [updated documentation](https://openoa.readthedocs.io/en/latest/index.html) for a complete overview of the new and improved OpenOA. Much will look familiar, but using the library should now be much more streamlined, and usage should be significantly faster.

CHANGELOG.md

@@ -1,7 +1,6 @@
 
 # Contributing
 
-
 ## Getting Started
 
 These contributing guidelines should be read by software developers wishing to contribute code or
@@ -209,13 +208,13 @@ To run the tests you can use any of the following commands, depending on your ne
 3. Only the unit tests:
 
    ```bash
-   pytest test/unit
+   pytest --unit
    ```
 
 4. Only the regression (integration) tests:
 
    ```bash
-   pytest test/regression
+   pytest --regression
    ```
 
 ## Pull Request
@@ -239,6 +238,7 @@ base. Be sure to write a complete description of these changes in the pull reque
 All tests must pass. Pull requests will be rejected or have changes requested if tests do not pass,
 or cannot pass with changes. Tests are automatically run through Github Actions for any pull request
 or push to the main or develop branches, but should also be run locally before submission.
+
 #### Test Coverage
 
 The testing framework described below will generate a coverage report from the tests run through

contributing.md

@@ -1,6 +1,14 @@
+# Examples Requirements
+
 Please be sure to install the required packages for running the example notebooks by
-executing the following command:
+executing one of the following commands:
 
+```bash
+pip install "openoa[examples]"
 ```
-pip install -r requirements.txt
+
+or for a source installation:
+
+```bash
+pip install ".[examples]"
 ```

examples/02b_plant_aep_analysis_cubico.ipynb

@@ -35,4 +35,37 @@ length_sort_straight = "True"
 lines_after_imports = "2"
 sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"]
 known_first_party = "openoa"
-known_third_party = ["numpy","scipy","pandas","pygam","tqdm","statsmodels","scikit_learn","EIA-python","requests","pyproj","shapely"]
+known_third_party = [
+    "numpy",
+    "scipy",
+    "pandas",
+    "pygam",
+    "tqdm",
+    "statsmodels",
+    "scikit_learn",
+    "EIA-python",
+    "requests",
+    "pyproj",
+    "shapely"
+]
+
+[tool.pytest.ini_options]
+python_files = [
+  "test/*.py",
+]
+testpaths = [
+  "test/unit/*.py",
+  "test/regression/*.py",
+]
+filterwarnings = [
+  "ignore::DeprecationWarning:sklearn.*:",
+  "ignore::DeprecationWarning:patsy.*:",
+  "ignore::DeprecationWarning:pygam.*:",
+]
+
+[tool.coverage.report]
+exclude_lines = [
+    "# pragma: no cover",
+    "def .+plot",
+    "def plot_.+",
+]

examples/README.md

@@ -26,7 +26,7 @@ If you would like to try out the code before installation or simply explore the
 
 If you use this software in your work, please cite our JOSS article with the following BibTex:
 
-```
+```bibtex
 @article{Perr-Sauer2021,
   doi = {10.21105/joss.02171},
   url = {https://doi.org/10.21105/joss.02171},
@@ -43,7 +43,7 @@ If you use this software in your work, please cite our JOSS article with the fol
 
 ### Requirements
 
-- Python 3.8, 3.9, or 3.10 with pip.
+- Python 3.8-3.11 with pip.
 
 We strongly recommend using the Anaconda Python distribution and creating a new conda environment for OpenOA. You can download Anaconda through [their website.](https://www.anaconda.com/products/individual)
 
@@ -72,6 +72,22 @@ python
 >>> openoa.__version__
 ```
 
+#### Installation Options
+
+There are a number of installation options that can be used, depending on the use case, which can be
+installed with the following pattern `pip install "openoa[opt1,opt2]"` (`pip install .[opt1,opt2]` is also allowed).
+
+- `develop`: for linting, automated formatting, and testing
+- `docs`: for building the documentation
+- `examples`: for the full Jupyter Lab suite (also contains `reanalysis` and `nrel-wind`)
+- `renalysis`: for accessing and processing MERRA2 and ERA5 data
+- `nrel-wind`: for accessing the NREL WIND Toolkit
+- `all`: for the complete dependency stack
+
+> **Important**
+> If using Python 3.11, install `openoa` only, then reinstall adding the modifiers to reduce
+> the amount of time it takes for pip to resolve the dependency stack.
+
 #### Common Installation Issues
 
 - In Windows you may get an error regarding geos_c.dll. To fix this install Shapely using:
@@ -88,6 +104,8 @@ pip install --upgrade pywin32==255
 
 #### Example Notebooks and Data
 
+Be sure to install OpenOA using the `examples` modifier from [above](#installation-options). Such as: `pip install ".[examples]"`
+
 The example data will be automaticaly extracted as needed by the tests. To manually extract the example data for use with the example notebooks, use the following command:
 
 ```bash
@@ -100,6 +118,8 @@ The example notebooks are located in the `examples` directory. We suggest instal
 jupyter lab  # "jupyter notebook" is also ok if that's your preference
 ```
 
+Open the URL printed to your command prompt in your favorite browser. Once jupyter is open, navigate to the "examples" directory in the file explorer and open an example notebook.
+
 ### Development
 
 Please see the developer section of the contributing guide [here](contributing.md), or on the [documentation site](https://openoa.readthedocs.io/en/latest/getting_started/contributing.html) for complete details.
@@ -110,7 +130,7 @@ this second step must be done before committing any changes):
 
 ```bash
 cd OpenOA
-pip install -e ".[develop, docs]"
+pip install -e ".[develop, docs, examples]"
 pre-commit install
 ```
 
@@ -138,7 +158,7 @@ pytest
 To run unit tests only:
 
 ```bash
-pytest test/unit
+pytest --unit
 ```
 
 To run all tests and generate a code coverage report

examples/requirements.txt

@@ -19,22 +19,16 @@
     "pandas>=2.0",
     "pygam>=0.9.0",
     "scipy>=1.7",
-    "statsmodels>=0.11",
+    "statsmodels>=0.11; python_version<'3.11'",
+    "statsmodels>=0.13.3; python_version=='3.11'",
     "tqdm>=4.28.1",
     "matplotlib>=3.6",
     "bokeh>=2.4",
-    "attrs>=22",
+    "attrs>=22.2",
     "pytz",
-    "h5pyd",
     "pyyaml",
-    "pyspark",
     "tabulate",
-    "statsmodels",
-    "jupyterlab",
-    "xarray",
-    "dask",
-    "netcdf4",
-    "cdsapi",
+    "ipython",
 ]
 
 # Testing-only dependencies
@@ -43,7 +37,6 @@
 # All extra dependencies (see keys for breakdown by purpose)
 EXTRAS = {
     "docs": [
-        "ipython",
         "Sphinx>=5.0,!=7.2.0",
         "pydata-sphinx-theme",
         "sphinx_design>=0.3",
@@ -58,8 +51,20 @@
         "flake8",
         "flake8-docstrings",
     ],
+    "examples": [
+        "jupyterlab",
+    ],
+    "nrel-wind": ["h5pyd"],
+    "reanalysis": [
+        "cdsapi",
+        "xarray[parallel]",  # Dask required for loading data
+        "h5py",  # Required for netcdf4
+        "netcdf4",
+    ],
 }
 EXTRAS["develop"] += TESTS
+EXTRAS["examples"] += EXTRAS["reanalysis"] + EXTRAS["nrel-wind"]
+EXTRAS["all"] = EXTRAS["develop"] + EXTRAS["docs"] + EXTRAS["examples"]
 
 
 # Read the version from the __init__.py file without importing it
@@ -99,6 +104,5 @@ def read_file(filename):
     include_package_data=True,
     install_requires=REQUIRED,
     extras_require=EXTRAS,
-    tests_require=TESTS,
-    python_requires=">=3.8, <3.11",
+    python_requires=">=3.8, <3.12",
 )

pyproject.toml

@@ -3,6 +3,12 @@
 All notebooks are located at /examples in the OpenOA repository, and can be modified and run on
 [Binder](https://mybinder.org/v2/gh/NREL/OpenOA/develop_v3?filepath=examples).
 
+```{important}
+Be sure to install OpenOA using the `examples` modifier (`pip install "openoa[examples]"`). This
+will install all of the dependencies used by the additional methods contained within the examples
+notebooks.
+```
+
 ## Overview
 
 In each of the following examples we'll be providing more insight into the different functionalities
@@ -32,6 +38,11 @@ of OpenOA. In each notebook, OpenOA is demonstrated using two years of operation
 
 ## Gap Analysis Step 1b: Estimate the AEP and Its Uncertainty Using Cubico Open Data [[link]](02b_plant_aep_analysis_cubico.ipynb)
 
+```{important}
+Be sure to install OpenOA using the `examples` and `reanalysis` modifiers for this notebook
+(`pip install "openoa[examples,reanalysis]"`).
+```
+
 * Introduces the annual energy production (AEP) class, and how to estimate the uncertainty using
   a Monte Carlo approach
 * Demonstrates some of the supplementary tools to analysis, such as the plotting routines, and

pytest.ini

@@ -9,9 +9,9 @@ library on our `Binder page`_.
 Quick Start
 ***********
 
-For most users, getting started should be as easy as either below code blocks, but if you get hung up on running examples
-locally, or interacting with the code, you can always interact with our `Binder page`_
-
+For most users, getting started should be as easy as either below code blocks, but if you get hung
+up on running examples locally, or interacting with the code, you can always interact with our
+`Binder page`_
 
 Pip
 ---
@@ -29,6 +29,23 @@ Source
     cd openoa
     pip install -e .
 
+Additional Dependencies
+-----------------------
+
+.. important::
+    If using Python 3.11, install ``openoa`` only, then reinstall adding the modifiers to reduce
+    the amount of time it takes for pip to resolve the dependency stack.
+
+Whether installing from PyPI or source, any combination of the following can be used to install
+additional dependencies. For example, the examples requirements can be installed using
+`pip install "openoa[examples]"`.
+
+- `develop`: for linting, automated formatting, and testing
+- `docs`: for building the documentation
+- `examples`: for the full Jupyter Lab suite (also contains `reanalysis` and `nrel-wind`)
+- `renalysis`: for accessing and processing MERRA2 and ERA5 data
+- `nrel-wind`: for accessing the NREL WIND Toolkit
+- `all`: for the complete dependency stack
 
 Installation and Contributing
 *****************************
@@ -40,5 +57,4 @@ Installation and Contributing
     changelog
 
 
-
 .. _Binder page: https://mybinder.org/v2/gh/NREL/OpenOA/develop_v3?filepath=examples

readme.md

@@ -31,6 +31,18 @@ at the same time with the following.
 
     pip install "OpenOA[develop]"
 
+.. important::
+    If using Python 3.11, install ``openoa`` only, then reinstall adding the modifiers to reduce
+    the amount of time it takes for pip to resolve the dependency stack.
+
+Additional options:
+- `develop`: for linting, automated formatting, and testing
+- `docs`: for building the documentation
+- `examples`: for the full Jupyter Lab suite (also contains `reanalysis` and `nrel-wind`)
+- `renalysis`: for accessing and processing MERRA2 and ERA5 data
+- `nrel-wind`: for accessing the NREL WIND Toolkit
+- `all`: for the complete dependency stack
+
 
 Now you can verify the version that was installed
 
@@ -54,7 +66,7 @@ guidelines and processes, please see the :ref:`contributors guide <contributing>
     pip install -e .
 
     # Extras can also be installed as one or any combination of the following
-    pip install -e ".[develop,docs,extras]"
+    pip install -e ".[develop,docs]"
 
 Now you can verify the version that was installed