Question on Sample Size for Training

[kwon-developer]

Hi, I hope this message finds you well.

Your current deep learning architecture includes 317 Great Britain Grids along with 2 years worth of data.

1. Would you say it was/is better to train all 317 at once than having separate models for different parts of regions with similar weathers like (for instance) 'North East England', ' West Midlands', 'Northern Ireland', 'London', etc?

2. Secondly, does your most current version include an extra yet separate architecture for past-PV generation output data?
   Based on your paper published at ICLR in 2024, you only had IFS, UKV, Satellite & Solar Position. Yet, have you included past/historical pv generation data? If so, since it has no spatial data, do you simply plug input directly into 'late fusion' block similar to solar coordinate input? (sorry...where may I find such code block?)

Apologies for lengthy questions
Thank you for your time! (seems very interesting :))

[AUdaltsova]

Hi @kwon-developer! Thanks for reaching out.

1. We do use 317 GSPs all at once, but we use ID encodings to help the model differentiate between them and learn things specific to the region. It has worked quite well so far! I can't say we've tried the framework you're proposing, with different models governing different regions, but off the top of my head I wouldn't expect it to be better: this way you will have to split the GSPs into groups and each model will get less data. This would be especially disadvantageous for smaller GSPs (e g in Scotland almost all of them are quite small), which tend to have noisier data, and I would expect being able to learn from other more stable GSPs helps the model to perform better on the smaller ones.
2. You're right, we do not use PV history in our GSP set-up! We do include it into other projects sometimes though (e g it can be very useful when making a model for a single site), and then it gets its own encoder, like this one:

"pv_encoder": 
        "_target_": "pvnet.models.multimodal.site_encoders.encoders.SingleAttentionNetwork",
        "_partial_": true,
        "num_sites": 1,
        "out_features": 64,
        "num_heads": 4,
        "kdim": 64,
        "id_embed_dim": 64

It doesn't have a spatial component, that's true, so the encoder we use is not a convolutional one (also, solar position for example doesn't have a spatial component either! It's just two 1D variables. It doesn't get an encoder though, if you want to see how it's handled I suggest you start here)

Hope that helps! Let me know if you have further questions :)

[AUdaltsova]

Glad this is helpful!

Sorry, I don't think I caught that first part. Are you saying you're using 1 hour of history and forecasting for up to 3 hours ahead? I'm also not sure what you mean when you say you can only use 3 hours of data due to the data being hourly - maybe because I usually think of "hourly" as time resolution (e g your datapoints are an hour away from each other) and the 3 hour thing is more about data availability, but maybe you mean something else?

Generally, we don't use GSP history, so no GSP data goes into an encoder. When we use pv history, it's typically for sites, we would get a couple of hours of history in each sample, and put it through the single-attention encoder (you can look at it here), and that would typically be one site at a time, so I think it should work fine in your case as well!

I think the diagram is still correct; we don't typically use GSP history, but PVNet is set up to handle it if we need it to.

[kwon-developer]

Sorry for the unclear question!

Just to clarify — for pv_history, I’m planning to use the past 3 hours of yield data. But since I only have 1-hour resolution, that ends up being a 1D vector with 3 time steps. (fyi - like yours my model will forecast up to 8 hours :) )

So after reading your code description — "A simple attention-based model with a single multihead attention layer" — I was wondering if that might be a bit overkill in my case. I’m training on a single site and for pv_history, I am using a 1D input of length 3. Setting target_id_dim=1, kdim=10, and num_heads=2 for such short and simple vector feels maybe excessive for this setup.

But if you think it’s still fine, I will go ahead and try it out! (just wanted some advice and thank you for your amazing advice and time🙂)

[AUdaltsova]

Thanks for clarifying! You might be right about it being a bit of overkill, but this is what we use with similar setups. If you find a better solution, do let us know! Can't say we've done any sort of proper parameter tuning around this part (to the best of my knowledge at least), so definitely useful to play around with that.

[AUdaltsova]

Hi @kwon-developer! Just wanted to let you know that one of my colleagues has done some work on pv history handling recently and your hunch was right, the encoder is an overkill. They've added an option to directly concatenate the data without encoding, you can take a look here.

[kwon-developer]

@AUdaltsova
Thanks as always for your amazing help! 🙏
Quick question—while revisiting the base model and exploring different architectures out of curiosity, I noticed that the solar coordinates are on a 30-minute interval, whereas the other NWPs use a 1-hour interval.

Was there a specific reason for choosing 30 minutes for the solar data?
I tried testing it with a 1-hour interval and didn’t see much difference, so I was curious to understand the rationale behind your original choice.

[AUdaltsova]

Hi @kwon-developer! I'm glad this is helpful :)

Solar coords are on a 30 min interval to mimic the target data, which is also 30-minutely. The idea is that for each point a model is predicting, it knows the exact position of the sun, which should be quite helpful for forecasting. Solar coordinates used to be calculated in training and use the timestamps of the target data provided, but we've fairly recently decoupled them for robustness & flexibility--though I'm fairly sure that since then we've continued to match them to target data. To be fair, re: NWPs are 1 hourly, if we could get our hands on a 30-minutely NWP I'm sure we would've preferred that!

It's super interesting to hear you've tried a coarser solar coords resolution, would you mind telling me a bit more about your experiment? We haven't run one like that ourselves, so knowing more about your results would be super useful!