Severity: P1 — wrong answers / DoS-shaped runtime. From the 2026-07-26 sweep. Note: since commit fa26f5f (#524) these no longer masquerade as success: true with NaN, but they still run to completion instead of being rejected up front with a clear scitype error.
BUG-14 — evaluate accepts non-forecasters and non-estimators
evaluate(estimator_handle=<Deseasonalizer transformer>, y="airline")
evaluate(estimator_handle=<int handle from spec "42">, y="airline")
Both attempt cross-validation instead of rejecting with "not a forecaster".
Fix: check the object_type / scitype before _run_evaluate; reject anything that isn't a forecaster.
BUG-15 — evaluate accepts Panel data and burns ~2 min producing nonsense
evaluate(estimator_handle=<NaiveForecaster>, y="basic_motions", cv_folds=3)
-> success, test_MeanAbsolutePercentageError: 5875693866242.47, 23 folds, ~2 min wall clock
A forecaster on Panel-scitype y is both a wrong answer and a runtime hazard.
Fix: reject Panel-scitype y for forecasters up front.
(Catalogued as BUG-14 / BUG-15 in MCP_TEST_FINDINGS.md.)
Severity: P1 — wrong answers / DoS-shaped runtime. From the 2026-07-26 sweep. Note: since commit fa26f5f (#524) these no longer masquerade as
success: truewith NaN, but they still run to completion instead of being rejected up front with a clear scitype error.BUG-14 — evaluate accepts non-forecasters and non-estimators
Both attempt cross-validation instead of rejecting with "not a forecaster".
Fix: check the
object_type/ scitype before_run_evaluate; reject anything that isn't a forecaster.BUG-15 — evaluate accepts Panel data and burns ~2 min producing nonsense
A forecaster on Panel-scitype
yis both a wrong answer and a runtime hazard.Fix: reject Panel-scitype
yfor forecasters up front.(Catalogued as BUG-14 / BUG-15 in
MCP_TEST_FINDINGS.md.)