Skip to content

[BUG] evaluate accepts non-forecasters and Panel data (runs instead of rejecting) #535

Description

@Shashankss1205

Severity: P1 — wrong answers / DoS-shaped runtime. From the 2026-07-26 sweep. Note: since commit fa26f5f (#524) these no longer masquerade as success: true with NaN, but they still run to completion instead of being rejected up front with a clear scitype error.

BUG-14 — evaluate accepts non-forecasters and non-estimators

evaluate(estimator_handle=<Deseasonalizer transformer>, y="airline")
evaluate(estimator_handle=<int handle from spec "42">, y="airline")

Both attempt cross-validation instead of rejecting with "not a forecaster".
Fix: check the object_type / scitype before _run_evaluate; reject anything that isn't a forecaster.

BUG-15 — evaluate accepts Panel data and burns ~2 min producing nonsense

evaluate(estimator_handle=<NaiveForecaster>, y="basic_motions", cv_folds=3)
  -> success, test_MeanAbsolutePercentageError: 5875693866242.47, 23 folds, ~2 min wall clock

A forecaster on Panel-scitype y is both a wrong answer and a runtime hazard.
Fix: reject Panel-scitype y for forecasters up front.

(Catalogued as BUG-14 / BUG-15 in MCP_TEST_FINDINGS.md.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions