Add NSRDB GOES v4 to iotools #2378

williamhobbs · 2025-02-06T18:54:08Z

Closes Add NSRDB GOES V4 #2326
I am familiar with the contributing guidelines
Tests added
Updates entries in docs/sphinx/source/reference for API changes.
Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
Pull request is nearly complete and ready for detailed review.
Maintainer: Appropriate GitHub Labels (including remote-data) and Milestone are assigned to the Pull Request and linked Issue.

williamhobbs · 2025-02-06T20:34:16Z

What's the best practice for lines like this one that end up too long? Do I simply add # noqa or # noqa: F401? (I don't know what F401 is for, but I've seen it used here...)

GOES_URL = NSRDB_API_BASE + "/api/nsrdb/v2/solar/nsrdb-GOES-aggregated-v4-0-0-download.csv"

https://github.com/williamhobbs/pvlib-python/blob/4af81a76b4d8bb86b992b5a87ca81dbec6ecb8d5/pvlib/iotools/goes4.py#L15

mikofski · 2025-02-06T21:57:41Z

You could use parentheses

GOES_URL = NSRDB_API_BASE + ("/api/nsrdb/v2/solar/"
"nsrdb-GOES-aggregated-v4-0-0-download.csv")

or just break the string smaller and add them

API_STUB = "/api/nsrdb/v2/solar/"
ENDPOINT = "nsrdb-GOES-aggregated-v4-0-0-download.csv"
GOES_URL = (NSRDB_API_BASE
    + API_STUB + ENDPOINT)

imo avoid ignoring pep8 warnings or else they become pointless.

williamhobbs · 2025-02-06T22:41:08Z

Thanks, @mikofski!

docs/sphinx/source/whatsnew/v0.11.3.rst

Co-authored-by: Echedey Luis <[email protected]>

williamhobbs · 2025-02-07T00:33:52Z

I'm not 100% sure that I set up the tests in test_goes4.py correctly. I copied what was done in test_psm3.py and manually downloaded and reformatted as needed the test data csv files.

See test_read_goes4_map_variables() in particular. columns_mapped includes variables like 'Ozone' that are part of a manual download but are not addressed in pvlib.

I think it's correct, but I could be misunderstanding the goals of some of the tests.

…-python into nsrdb_goes_v4

williamhobbs · 2025-02-08T02:51:35Z

Ok, made the switch from "goes4" to "psm4".

I haven't done anything with @kandersolar's suggestion for different get_ functions for different endpoints (e.g., get_nsrdb_goes4_aggregated, get_nsrdb_goes4_conus, and get_nsrdb_goes4_tmy). I'll wait for some consensus on that one.

WIP: handful of minor things

…-python into nsrdb_goes_v4

williamhobbs · 2025-02-13T22:12:25Z

Ok, I made an effort to create separate get_ functions in a new branch: https://github.com/williamhobbs/pvlib-python/tree/nsrdb_psm4_dedicated_get_functions.

I'll hold off on merging that with the branch in this PR (https://github.com/williamhobbs/pvlib-python/tree/nsrdb_goes_v4) until someone suggests that I do it - I don't want to make a mess of things too quickly.

AdamRJensen

A few initial comments

pvlib/iotools/psm4.py

AdamRJensen · 2025-03-23T18:49:18Z

pvlib/iotools/psm4.py

+    return parse_psm4(fbuf, map_variables)
+
+
+def parse_psm4(fbuf, map_variables=True):


I personally think we should get rid of the parse functions as it creates a LOT of duplicate documentation and significantly more functions than needed. It's unnecessary clutter and it takes very little to modify read functions to also take filebuffers.

Alternatively, we could consider making the parse functions private.

I am generally +1 to ditching parse_ and having read_ work with buffers as well as filenames. But let's do it as a follow-up PR.

pvlib/iotools/psm4.py

AdamRJensen · 2025-03-23T18:57:32Z

pvlib/tests/iotools/test_psm4.py

+@pytest.fixture(scope="module")
+def nrel_api_key():
+    """Supplies pvlib-python's NREL Developer Network API key.
+
+    Azure Pipelines CI utilizes a secret variable set to NREL_API_KEY
+    to mitigate failures associated with using the default key of
+    "DEMO_KEY". A user is capable of using their own key this way if
+    desired however the default key should suffice for testing purposes.
+    """
+    try:
+        demo_key = os.environ["NREL_API_KEY"]
+    except KeyError:
+        warnings.warn(
+            "WARNING: NREL API KEY environment variable not set! "
+            "Using DEMO_KEY instead. Unexpected failures may occur."
+        )
+        demo_key = 'DEMO_KEY'
+    return demo_key


If it's the same key as for the PSM3, then I suggest importing it from that test module (should be possible?) to avoid duplicating code.

holding off on this, per a conversation with @kandersolar

I moved the fixture to conftest and imported it in both psmN test modules

AdamRJensen · 2025-03-23T22:16:39Z

Given the unanimous vote and the three weeks passed (see #2326 (comment)), you can go ahead with splitting the functions into individual ones.

Please still consider the comments above 😄

williamhobbs · 2025-04-02T22:33:44Z

Ok @AdamRJensen, I implemented many of your comments into https://github.com/williamhobbs/pvlib-python/tree/nsrdb_psm4_dedicated_get_functions. I may be at the limit of my git/GitHub knowledge, though. What do I do next to get those changes (along with the dedicated functions) from that branch into this PR? I can follow up offline if that's best...

Edit: I sorted this out, thanks to help from @kandersolar.

…tions Nsrdb psm4 dedicated get functions

williamhobbs · 2025-04-03T19:04:26Z

@AdamRJensen - I think I addressed all of your comments in one way or another. Thanks for the review - I think I forgot to say that earlier.

tests/conftest.py

Co-authored-by: Will Holmgren <[email protected]>

AdamRJensen · 2025-04-10T13:17:02Z

docs/sphinx/source/reference/iotools.rst

+   iotools.get_nsrdb_psm4_conus
+   iotools.get_nsrdb_psm4_full_disc
+   iotools.read_nsrdb_psm4
+   iotools.parse_nsrdb_psm4


Why don't we make this a private function for now so we don't have to deprecate it later?

Are we also going to add support for buffers to read_nsrdb_psm4? I recommend moving forward with this PR API as is and if you want to take that on before the next release then go for it.

Also need to edit the docstring returns sections if this function becomes private.

AdamRJensen · 2025-04-10T13:19:06Z

pvlib/iotools/psm4.py

+    names : str, default '2023'
+        PSM4 API parameter specifing year (e.g. ``2023``) to download. The
+        allowed values update periodically, so consult the NSRDB reference
+        below for the current set of options.


No need to follow the strange naming convention of PSM. I suggest renaming this to year.

AdamRJensen · 2025-04-10T13:20:58Z

pvlib/iotools/psm4.py

+    url : str, optional
+        Full API endpoint URL. If not specified, the PSM4 GOES Aggregated v4
+        URL is used.
+    timeout : int, default 30
+        time in seconds to wait for server response before timeout
+    utc: bool, default : False
+        retrieve data with timestamps converted to UTC. False returns
+        timestamps in local standard time of the selected location


I suggest changing the order such that the api specific parameters comes first, i.e., have utc come before map_variables, url, and timeout - this is a somewhat unwritten convention for the iotools functions.

AdamRJensen · 2025-04-10T13:23:48Z

pvlib/iotools/psm4.py

+        PSM4 API parameter specifing TMY variant to download (e.g. ``'tmy'``
+        or ``'tgy-2022'``).  The allowed values update periodically, so
+        consult the NSRDB references below for the current set of options.
+    time_period : int, {60}


Time period could be thought of as what time period is covered, i.e., start to end. I think time_step is a better name for this parameter (this is also what is used by the pvlib.iotools.get_cams function).

pvlib/iotools/psm4.py

AdamRJensen · 2025-04-10T13:27:47Z

pvlib/iotools/psm4.py

+    return data, metadata
+
+
+def read_nsrdb_psm4(filename, map_variables=True):


Could this function be called read_sam instead?

I think read_sam could be misinterpreted. SAM isn't the NSRDB, it is a user of that data; read_sam could be confused with retrieve_sam; and SAM is moving towards exporting a JSON file for a simulation, which could be of future interest to pvlib users.

Co-authored-by: Adam R. Jensen <[email protected]>

williamhobbs added 5 commits February 6, 2025 10:46

added goes4

485165b

added goes4 to iotools __init__.py

856abc7

change zenith to solar_zenith

026738e

fixed references to psm3

ee9d526

fixed some linter issues

4af81a7

uppdated whatsnew

547bf02

fixed more formating issues

fee3743

echedey-ls reviewed Feb 6, 2025

View reviewed changes

docs/sphinx/source/whatsnew/v0.11.3.rst Outdated Show resolved Hide resolved

williamhobbs and others added 3 commits February 6, 2025 18:00

added tests and test data

2664d03

fixed linter issues

0118317

Update docs/sphinx/source/whatsnew/v0.11.3.rst

2f6817c

Co-authored-by: Echedey Luis <[email protected]>

williamhobbs mentioned this pull request Feb 7, 2025

Add NSRDB GOES V4 #2326

Open

williamhobbs added 3 commits February 7, 2025 20:43

change "goes4" to "psm4"

fe8821d

Merge branch 'nsrdb_goes_v4' of https://github.com/williamhobbs/pvlib…

b3eb5a5

…-python into nsrdb_goes_v4

fix some missed goes4 instances

f5dec0a

wholmgren and others added 10 commits February 12, 2025 09:32

init file goes4 --> psm4

c23ce24

use urljoin

c857047

use str year in test

c216255

remove versionchanged doc warning

3144899

Merge pull request #1 from wholmgren/nsrdb_goes_v4

bd4f760

WIP: handful of minor things

fix function names in whatsnew

30bf1b1

Merge branch 'nsrdb_goes_v4' of https://github.com/williamhobbs/pvlib…

703685a

…-python into nsrdb_goes_v4

another whatsnew fix

d75c5a0

Merge branch 'main' into nsrdb_goes_v4

9c1230c

created separate get_nsrdb_psm4_ functions

4524551

williamhobbs added 3 commits February 13, 2025 16:07

update whatsnew

87d701c

update docs for API changes

5a13994

update docs for API changes

523877b

kandersolar added this to the v0.11.3 milestone Feb 20, 2025

kandersolar added the enhancement label Feb 20, 2025

kandersolar modified the milestones: v0.11.3, v0.11.4 Mar 14, 2025

AdamRJensen reviewed Mar 23, 2025

View reviewed changes

addressed many of adamrjensen's initial comments

af8ebd4

williamhobbs added 3 commits April 3, 2025 11:16

Merge branch 'nsrdb_goes_v4' into nsrdb_psm4_dedicated_get_functions

999bfa6

Merge pull request #2 from williamhobbs/nsrdb_psm4_dedicated_get_func…

44a9ef1

…tions Nsrdb psm4 dedicated get functions

addressed more of adam's comments

502b323

kandersolar and others added 6 commits April 4, 2025 13:25

move nrel api key fixture to conftest

cabe2fc

Merge remote-tracking branch 'upstream/main' into pr/2378

ae7b6e7

move whatsnew entry to 0.12.1

60ade0e

move data files to new test data location

3196b60

lint

8f5ae98

Merge branch 'main' into nsrdb_goes_v4

3fbd34a

wholmgren approved these changes Apr 9, 2025

View reviewed changes

tests/conftest.py Outdated Show resolved Hide resolved

williamhobbs and others added 2 commits April 9, 2025 13:57

Update tests/conftest.py

709e4a2

Co-authored-by: Will Holmgren <[email protected]>

Merge branch 'main' into nsrdb_goes_v4

8a5b814

AdamRJensen approved these changes Apr 10, 2025

View reviewed changes

Update pvlib/iotools/psm4.py

eab4f79

Co-authored-by: Adam R. Jensen <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add NSRDB GOES v4 to iotools #2378

Add NSRDB GOES v4 to iotools #2378

williamhobbs commented Feb 6, 2025 •

edited by kandersolar

Loading

williamhobbs commented Feb 6, 2025

mikofski commented Feb 6, 2025

williamhobbs commented Feb 6, 2025

williamhobbs commented Feb 7, 2025

williamhobbs commented Feb 8, 2025

williamhobbs commented Feb 13, 2025

AdamRJensen left a comment

AdamRJensen Mar 23, 2025

kandersolar Apr 3, 2025

AdamRJensen Mar 23, 2025

williamhobbs Apr 3, 2025

kandersolar Apr 4, 2025

AdamRJensen commented Mar 23, 2025

williamhobbs commented Apr 2, 2025 •

edited

Loading

williamhobbs commented Apr 3, 2025

AdamRJensen Apr 10, 2025

wholmgren Apr 10, 2025 •

edited

Loading

AdamRJensen Apr 10, 2025 •

edited

Loading

AdamRJensen Apr 10, 2025

AdamRJensen Apr 10, 2025

AdamRJensen Apr 10, 2025

cwhanse Apr 10, 2025 •

edited

Loading

		return parse_psm4(fbuf, map_variables)


		def parse_psm4(fbuf, map_variables=True):

		return data, metadata


		def read_nsrdb_psm4(filename, map_variables=True):

Add NSRDB GOES v4 to iotools #2378

Are you sure you want to change the base?

Add NSRDB GOES v4 to iotools #2378

Conversation

williamhobbs commented Feb 6, 2025 • edited by kandersolar Loading

williamhobbs commented Feb 6, 2025

mikofski commented Feb 6, 2025

williamhobbs commented Feb 6, 2025

williamhobbs commented Feb 7, 2025

williamhobbs commented Feb 8, 2025

williamhobbs commented Feb 13, 2025

AdamRJensen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdamRJensen commented Mar 23, 2025

williamhobbs commented Apr 2, 2025 • edited Loading

williamhobbs commented Apr 3, 2025

Choose a reason for hiding this comment

wholmgren Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

AdamRJensen Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwhanse Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

williamhobbs commented Feb 6, 2025 •

edited by kandersolar

Loading

williamhobbs commented Apr 2, 2025 •

edited

Loading

wholmgren Apr 10, 2025 •

edited

Loading

AdamRJensen Apr 10, 2025 •

edited

Loading

cwhanse Apr 10, 2025 •

edited

Loading