Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NREL Siting Lab dataset archiver #585

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

e-belfer
Copy link
Member

@e-belfer e-belfer commented Feb 12, 2025

Overview

Closes #584

What problem does this address?

  • Add an archiver to grab all NREL siting data. The links on the main page are dynamically generated through Javascript, so we POST to the API to get them before looping through each page and archiving the data. We end up with one zip file per dataset.

What did you change in this PR?

  • Added the archiver and the archiver source metadata

Questions for the reviewer:

  • I've left working_partitions blank in the datapackage.json file because I dynamically generate all the URLs. Is this a problem if we aren't planning to access the data through our datastore?

Testing

How did you make sure this worked? How can a reviewer verify this?
https://github.com/catalyst-cooperative/pudl-archiver/actions/runs/13296766499

To-do list

Tasks

Preview Give feedback

@e-belfer e-belfer requested a review from cmgosnell February 12, 2025 23:14
@e-belfer e-belfer self-assigned this Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In progress
Development

Successfully merging this pull request may close these issues.

Write an archiver for NREL Siting Datasets
1 participant