Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fill gsp values #375

Merged
merged 2 commits into from
Feb 12, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 24 additions & 7 deletions ocf_datapipes/load/gsp/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -239,13 +239,30 @@
gsp_nominal_capacity_df = empty_df.join(gsp_nominal_capacity_df)
gsp_effective_capacity_df = empty_df.join(gsp_effective_capacity_df)

# interpolate in between, maximum 'live_interpolate_minutes' mins
# note data is in 30 minutes chunks
limit = int(interpolate_minutes / 30)
if limit > 0:
gsp_power_df.interpolate(limit=limit, inplace=True, method="cubic")
gsp_nominal_capacity_df.interpolate(limit=limit, inplace=True, method="cubic")
gsp_effective_capacity_df.interpolate(limit=limit, inplace=True, method="cubic")
try:
# interpolate in between, maximum 'live_interpolate_minutes' mins
# note data is in 30 minutes chunks
limit = int(interpolate_minutes / 30)
if limit > 0:
gsp_power_df.interpolate(limit=limit, inplace=True, method="cubic")
gsp_nominal_capacity_df.interpolate(limit=limit, inplace=True, method="cubic")
gsp_effective_capacity_df.interpolate(limit=limit, inplace=True, method="cubic")
except Exception as e:
logger.warning(

Check warning on line 251 in ocf_datapipes/load/gsp/database.py

View check run for this annotation

Codecov / codecov/patch

ocf_datapipes/load/gsp/database.py#L250-L251

Added lines #L250 - L251 were not covered by tests
f"Tried to interpolate the data, but failed ({e}). "
"We will fill these values with 0s for the moment. "
"We also set the nominal_capacity and effective_capacity data frames to zero too."
"These shouldn't get used in pvnet_app, so its ok. "
)

# create a dataframe of zeros, with index datetimes, and columns gsp_ids
data_zeros = pd.DataFrame(

Check warning on line 259 in ocf_datapipes/load/gsp/database.py

View check run for this annotation

Codecov / codecov/patch

ocf_datapipes/load/gsp/database.py#L259

Added line #L259 was not covered by tests
np.zeros((len(empty_df), len(gsp_ids))),
index=pd.date_range(start=start_utc_extra, end=now, freq="30min", tz=timezone.utc),
columns=gsp_ids,
)

return data_zeros, data_zeros, data_zeros

Check warning on line 265 in ocf_datapipes/load/gsp/database.py

View check run for this annotation

Codecov / codecov/patch

ocf_datapipes/load/gsp/database.py#L265

Added line #L265 was not covered by tests

# filter out the extra minutes loaded
logger.debug(f"{len(gsp_power_df)} of datetimes before filter on {start_utc}")
Expand Down