[ENH] Add more `train_test_split` strategies by Muhammad-Rebaal · Pull Request #2087 · sktime/pytorch-forecasting

Muhammad-Rebaal · 2026-02-23T11:38:22Z

Closes #2064 which references #1974

I've resolved the issue to expand the train_test_split capabilities within the ptf-v2 DataModules. Could you please review the PR?

Here is a summary of the changes made:

Created Splitters: Added pytorch_forecasting/data/splitters.py which implements three new splitting functions: random_series_split (acts as group-based split), stratified_series_split, and temporal_window_split.
Integrated into DataModules: Added the split_strategy parameter ("random", "stratified", "temporal", "group") and delegated splitting logic to both EncoderDecoderTimeSeriesDataModule (pytorch_forecasting/data/data_module.py) and TslibDataModule (pytorch_forecasting/data/_tslib_data_module.py).
Backwards Compatibility: Ensured the previous randomized series split logic remains fully intact and acts as the default behavior when split_strategy="random".

Thank You !

…ecasting into train_test_split

…/pytorch-forecasting into train_test_split

codecov · 2026-03-02T17:02:17Z

Codecov Report

❌ Patch coverage is 12.63617% with 401 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@7bcf66c). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
pytorch_forecasting/data/tests/test_splitters.py	0.00%	190 Missing ⚠️
pytorch_forecasting/data/splitters.py	16.12%	104 Missing ⚠️
...g/data/data_module/_encoder_decoder_data_module.py	22.22%	63 Missing ⚠️
...forecasting/data/data_module/_tslib_data_module.py	31.25%	44 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2087   +/-   ##
=======================================
  Coverage        ?   84.28%           
=======================================
  Files           ?      177           
  Lines           ?    10587           
  Branches        ?        0           
=======================================
  Hits            ?     8923           
  Misses          ?     1664           
  Partials        ?        0

Flag	Coverage Δ
cpu	`84.28% <12.63%> (?)`
pytest	`84.28% <12.63%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…/pytorch-forecasting into train_test_split

phoeenniixx · 2026-03-05T18:36:43Z

-                self._train_size : self._train_size + self._val_size
-            ]
+        # Ensure categorical encoders are fitted
+        if getattr(self, "_categorical_encoders", "auto") == "auto":


Can you please specify where are you defining _categorical_encoders?

I think this should be handled sperately? or maybe you can stack both PRs?

Hi @phoeenniixx! You're absolutely right _categorical_encoders was never defined in it. But it is in this my PR #2082 . So, could you review that and we'd stack over after merging that I'll continue from here and make this categorical_encoder's train_test_split feature more stable. I just added that as a part of reference for future implementation.

I think categorical_encoders are out of scope for this PR? If you want to use them, I think you should stack on the the current PR using git merge?

phoeenniixx

Thanks!
There are some unrelated changes to layers (like changes to the documentation) please remove that

Muhammad-Rebaal · 2026-03-09T22:37:23Z

Thanks! There are some unrelated changes to layers (like changes to the documentation) please remove that

Thanks for guiding @phoeenniixx, Kindly review the PR. I've removed the un-related changes.

Muhammad-Rebaal · 2026-03-16T09:15:34Z

Hi @phoeenniixx , @PranavBhatP , @fkiraly

Kindly review this PR

Muhammad-Rebaal · 2026-07-17T13:06:38Z

Hi @agobbifbk @Muhammad-Rebaal Can we try something like sktime implementation: https://github.com/sktime/sktime/blob/main/sktime/split/temporal_train_test_split.py THere are multiple ways, we can start from there? FYI @fkiraly

I've analyzed sktime's approach, and I think we can implement a custom TemporalTrainTestSplitter class in splitters.py tailored for our window-based pipeline.

This approach allows us to support anchor ("start"/"end") semantics and 3-way splits (train/val/test) using global timestamp cutoffs. The main benefit is that it simplifies the setup() logic across all DataModules and gives us a clean, reusable API without adding an external dependency.

Does this direction look good to you? If so, I can start on a backward-compatible PR.

agobbifbk · 2026-07-30T08:21:46Z

What if we have 2 groups but with different lifespan? If we use temporal cutoff we may end with one series in train and the other in validation, but maybe we don't want this. We should also think to have a percentage approach that computes the cutoff PER GROUP and then use the temporal cutoff, what do you think?

Muhammad-Rebaal · 2026-07-30T09:14:55Z

What if we have 2 groups but with different lifespan? If we use temporal cutoff we may end with one series in train and the other in validation, but maybe we don't want this. We should also think to have a percentage approach that computes the cutoff PER GROUP and then use the temporal cutoff, what do you think?

Hi @agobbifbk Could you please explain to me this what do you mean by 2 groups. If there are 2 groups then they should be separated first into group 1 and group 2 and then the temporal split applies. So, what is the situation you're talking about that we end up?

agobbifbk · 2026-07-30T09:32:28Z

If group 1 goes from 2000 to 2010 and group 2 goes from 2005 to 2015 and you put a global cutoff at 2005, for the train/validation split you will end up with half of group 1 on train and half in test, the group 2 entirely in validation. But what if I want half of both in train and half in validation? I'm saying that global cutoff can be used if I want to end up in situation 1, but we need also to use the percentage and compute local cutoff if I want to end up with situation 2.

…ecasting into train_test_split

…/pytorch-forecasting into train_test_split

Muhammad-Rebaal · 2026-08-04T14:54:23Z

Hi @phoeenniixx, @agobbifbk, I've made the following changes, kindly have a look:

The user can now apply their own temporal cutoff.

If data is provided from 2020 to 2027 andthe user applied the temporal cutoff by themselves

dm = TslibDataModule(
    ds,
    context_length=30,
    prediction_length=7,
    split_strategy="temporal",
    temporal_cutoffs={"end_train": 2022.0, "start_test": 2023.0}
)

Train: [2020, 2022]
Validation: (2022, 2023)
Test: [2023, 2027]

There's a default global cutoff is available if the user don't specify their cutoff in that case : 70/15/15

agobbifbk · 2026-08-05T07:09:17Z

Nice one @Muhammad-Rebaal! Just one comment. Since we can pass both the percentage and the global cutoff, maybe it worth to print a warning that says, when both are not None, that global cutoff is checked first. Did you check if the temporal split works also for datetime, date, and other non-numerical timestamps? From a first review it seems that the percentage split is applied per series in case of temporal split instead of per group isn't it?

for s_idx, timestamps in series_timestamps.items():
        t_min = float(np.min(timestamps))
        t_max = float(np.max(timestamps))
        t_range = t_max - t_min
        if t_range == 0:
            series_cutoffs[s_idx] = None
        else:
            train_cutoff = t_min + train_val_test_split[0] * t_range
            val_cutoff = train_cutoff + train_val_test_split[1] * t_range
            series_cutoffs[s_idx] = (train_cutoff, val_cutoff)
    all_zero = all(v is None for v in series_cutoffs.values())

If you do it per series, it may be that one group has some future data in validation no? Or I'm reading it wrongly?

…passed both temporal cutoff value and split

Muhammad-Rebaal · 2026-08-05T18:04:59Z

Since we can pass both the percentage and the global cutoff, maybe it worth to print a warning that says, when both are not None, that global cutoff is checked first.

Yes I've added the warning for that.

and you're right for the rest I've added a fix to tackle that.

Muhammad-Rebaal · 2026-08-05T19:57:13Z

+    all_ts = np.concatenate(list(series_timestamps.values()))
+    global_timeline = np.unique(all_ts)
+
+    if len(global_timeline) <= 1:
+        total_w = len(windows)
+        train_end = int(np.round(train_val_test_split[0] * total_w))
+        if train_end == 0 and train_val_test_split[0] > 0 and total_w > 0:
+            train_end = 1
+        val_end = train_end + int(np.round(train_val_test_split[1] * total_w))
+        val_end = min(val_end, total_w)
+        return windows[:train_end], windows[train_end:val_end], windows[val_end:]
+
+    n = len(global_timeline)
+    train_pos = min(int(np.round(train_val_test_split[0] * n)), n - 1)
+    val_pos = min(
+        int(np.round((train_val_test_split[0] + train_val_test_split[1]) * n)),
+        n - 1,
+    )
+
+    train_cutoff = global_timeline[train_pos]
+    val_cutoff = global_timeline[val_pos]
+
+    train_windows, val_windows, test_windows = [], [], []
+    for w in windows:
+        end_time = _get_window_end_time(w, series_timestamps)
+        if end_time <= train_cutoff:
+            train_windows.append(w)
+        elif end_time <= val_cutoff:
+            val_windows.append(w)
+        else:
+            test_windows.append(w)
+    return train_windows, val_windows, test_windows


Going with default split (70,15,15)
We've 2 series :
S1: [1,2,3,4,5,6] , S2 : [4,6,7,8]
then combine that via np.concatenate() -> [1,2,3,4,5,6,7,8] -> length of 8
train_pos = global_timeline[6] -> [6]
val_pos = global_timeline[7] -> [7]

train_cutoff = 7
val_cutoff = 8

Which later by looping we get
end_time = 8

train_windows = [1-7)
val_windows = [8]
test_windows = [empty in this case]

Almost: this concatenation is performed by group or regardless the group? Your logic works if S1 and S2 are from the same group even if this is not possible (it can not be that the same group has the same timestamp in two series).
if S1 is a series of group 1 and S2 of group 2 you should not concatenate the TS you should perform the split per series (as it was at the beginning).

I give another example:
percentage: (80,10,10) (for convenience)

G1: S1: [1,2,3,4,5,6] , S2 : [7,8,9,10] --> 1-8 in train, 9 validation 10 test
G2: S3: [11,12,13,14,15,16] , S4 : [17,18,19,20] --> 11-18 in train, 19 validation 20 test

So the concatenation must be per group, hard for me to read if from the code, if it is already like this good :-)
Moreover, if the user goes for a temporal split we should suggest to use the group as categorical feature (known).

Since you are at this point, can we implement also the group-time split? In this case first you divide the groups into train, validation and test, then the groups in train are divided using the temporal logic into train validation and test! In this case the group CAN not be used as categorical variable otherwise the model can hardly generalize on new groups!

Hope this clarifies!

Actually its not that way implemented but I completely understand what you mean so I'd adjust that accordingly then we'd move towards the group-time split

Just one question :

In this case first you divide the groups into train, validation and test

It would be train_val_test_split: tuple = (0.8,0.1,0.1) or train_val_test_split: tuple = (0.7, 0.15, 0.15)

Maybe we should call train_val_test_split differently if is referred to the temporal split or group split, so we need one more parameter here

Yeah I agree, I've added a param for that called group-split to handle it.

phoeenniixx

Would it make sense to move the splitting logic to split folder and then you access it from there, in that way you dont have to implement the same thing twice

Muhammad-Rebaal · 2026-08-07T09:14:34Z

Would it make sense to move the splitting logic to split folder and then you access it from there, in that way you dont have to implement the same thing twice

Yeah, I think that's a good idea to do so but I think first the implementation should be done then we'd refactor.

…/pytorch-forecasting into train_test_split

phoeenniixx · 2026-08-07T15:17:47Z

Would it make sense to move the splitting logic to split folder and then you access it from there, in that way you dont have to implement the same thing twice

Yeah, I think that's a good idea to do so but I think first the implementation should be done then we'd refactor.

But rn (when the thing is being actually designed) - it is easier to refactor no? Once it is released, we would need deprecastions etc to make this same refactor. Also, this will overcomplicate the datamodule class once we want to add more splitting algos (which will happen soon ig)

phoeenniixx · 2026-08-07T15:18:01Z

Also, the code-quality is failing :)

Muhammad-Rebaal · 2026-08-07T15:26:37Z

Would it make sense to move the splitting logic to split folder and then you access it from there, in that way you dont have to implement the same thing twice

Yeah, I think that's a good idea to do so but I think first the implementation should be done then we'd refactor.

But rn (when the thing is being actually designed) - it is easier to refactor no? Once it is released, we would need deprecastions etc to make this same refactor. Also, this will overcomplicate the datamodule class once we want to add more splitting algos (which will happen soon ig)

Yes you're right I didn't mean the release thing ? I mean if there are split strategies or this is the last one for this PR then I'd refactor them all in this PR

let's say there 3 functions of splitting and there is another that needs to be done so I wrote the 4th one and then refactor in the same PR.

What is your suggestion?

phoeenniixx · 2026-08-07T16:00:33Z

Sorry, I dont understand what you mean? Did you plan to first reach to a consensus and then refactor in the same PR? That is also a good idea... Everything depends on you as you are the main person on this feat rn

Muhammad-Rebaal · 2026-08-07T16:20:28Z

Sorry, I dont understand what you mean? Did you plan to first reach to a consensus and then refactor in the same PR? That is also a good idea... Everything depends on you as you are the main person on this feat rn

No worries! Yes, I meant reaching a consensus first and then refactoring it all in the same PR. Thanks for understanding.

Muhammad-Rebaal · 2026-08-11T07:44:40Z

Hi @agobbifbk, these are the following changes:

Implemented group-time-split.
Fix the concatination issue in the _split_percentage method.

Is there any other split strategies that needs to be implemented. Kindly let me know.

feat:Add more train_test_split strategies

12c5e57

Muhammad-Rebaal requested review from PranavBhatP, benHeid, fkiraly, fnhirwa, jdb78, phoeenniixx and yarnabrina as code owners February 23, 2026 11:38

Muhammad-Rebaal and others added 6 commits March 2, 2026 17:26

Merge branch 'main' into train_test_split

d4e48b3

Merge branch 'main' of https://github.com/Muhammad-Rebaal/pytorch-for…

455472c

…ecasting into train_test_split

Merge branch 'train_test_split' of https://github.com/Muhammad-Rebaal…

1758ec3

…/pytorch-forecasting into train_test_split

fix: Failed PyTest

29734d7

fix: ruff formatting

263ae38

fix: failure of test_univariate_forecast

d089be4

Muhammad-Rebaal and others added 4 commits March 3, 2026 04:29

Merge branch 'main' into train_test_split

90d345a

fix: Module not found error

af59b4c

Merge branch 'train_test_split' of https://github.com/Muhammad-Rebaal…

35bf4c3

…/pytorch-forecasting into train_test_split

fix: formatting issue

aae1f96

phoeenniixx reviewed Mar 5, 2026

View reviewed changes

phoeenniixx requested changes Mar 5, 2026

View reviewed changes

phoeenniixx changed the title ~~feat: Add more train_test_split strategies~~ [ENH] Add more train_test_split strategies Mar 5, 2026

phoeenniixx added the enhancement New feature or request label Mar 5, 2026

Muhammad-Rebaal and others added 2 commits March 9, 2026 23:49

Merge branch 'main' into train_test_split

9248866

fix:Reverse the documentation related issues

ab5b4d0

Merge branch 'main' into train_test_split

b623d59

Muhammad-Rebaal added 2 commits March 16, 2026 14:17

Merge branch 'main' into train_test_split

573d1fb

Merge branch 'main' into train_test_split

2d0456c

Merge branch 'main' into train_test_split

5017d01

Muhammad-Rebaal and others added 5 commits August 2, 2026 22:40

Merge branch 'main' into train_test_split

2ee716e

Merge branch 'main' of https://github.com/Muhammad-Rebaal/pytorch-for…

a4a47b4

…ecasting into train_test_split

Merge branch 'main' into train_test_split

84a5bca

Merge branch 'train_test_split' of https://github.com/Muhammad-Rebaal…

d71376d

…/pytorch-forecasting into train_test_split

[ENH] Dynamic Cutoff is implemented

a6f3a24

train test split can handle timestamps & raise warning when the user …

54b5de8

…passed both temporal cutoff value and split

Muhammad-Rebaal commented Aug 5, 2026

View reviewed changes

phoeenniixx mentioned this pull request Aug 6, 2026

[ENH] Add preprocessing to tslib datamodule #2330

Open

Merge branch 'main' into train_test_split

a87d7e4

phoeenniixx requested changes Aug 7, 2026

View reviewed changes

Muhammad-Rebaal added 2 commits August 7, 2026 20:08

[ENH] Added Implementation for group-time-split

5144273

Merge branch 'train_test_split' of https://github.com/Muhammad-Rebaal…

f68b441

…/pytorch-forecasting into train_test_split

fix code quality

15543a9

phoeenniixx mentioned this pull request Aug 8, 2026

[ENH] Add train/test split strategy for v2 datamodules #2375

Open

3 tasks

Merge branch 'main' into train_test_split

ab5d862

Uh oh!

Conversation

Muhammad-Rebaal commented Feb 23, 2026

Uh oh!

codecov Bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Muhammad-Rebaal Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phoeenniixx left a comment

Choose a reason for hiding this comment

Uh oh!

Muhammad-Rebaal commented Mar 9, 2026

Uh oh!

Muhammad-Rebaal commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Muhammad-Rebaal commented Jul 17, 2026

Uh oh!

agobbifbk commented Jul 30, 2026

Uh oh!

Muhammad-Rebaal commented Jul 30, 2026

Uh oh!

agobbifbk commented Jul 30, 2026

Uh oh!

Muhammad-Rebaal commented Aug 4, 2026

Uh oh!

agobbifbk commented Aug 5, 2026

Uh oh!

Muhammad-Rebaal commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Muhammad-Rebaal Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phoeenniixx left a comment

Choose a reason for hiding this comment

Uh oh!

Muhammad-Rebaal commented Aug 7, 2026

Uh oh!

phoeenniixx commented Aug 7, 2026

Uh oh!

phoeenniixx commented Aug 7, 2026

Uh oh!

Muhammad-Rebaal commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phoeenniixx commented Aug 7, 2026

Uh oh!

Muhammad-Rebaal commented Aug 7, 2026

Uh oh!

Muhammad-Rebaal commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

codecov Bot commented Mar 2, 2026 •

edited

Loading

Muhammad-Rebaal Mar 9, 2026 •

edited

Loading

Muhammad-Rebaal commented Mar 16, 2026 •

edited

Loading

Muhammad-Rebaal Aug 7, 2026 •

edited

Loading

Muhammad-Rebaal commented Aug 7, 2026 •

edited

Loading

Muhammad-Rebaal commented Aug 11, 2026 •

edited

Loading