Scheduled scaler failing to update HPA with Object metric type and Average Value

[apratina]

We configured HPA with object metrics type along with average value. We are noting scheduled scaler is failing to update HPA configuration and it would fail with following error:

E0117 00:30:00.053071       1 scaling-controller.go:171] FAILED TO UPDATE HPA: <HPA_NAME> - HorizontalPodAutoscaler.autoscaling "<HPA_NAME>" is invalid: spec.metrics[1].object.target.value: Invalid value: resource.Quantity{i:resource.int64Amount{value:0, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"0", Format:"DecimalSI"}: must be positive

It seems to be related to this K8s HPA bug kubernetes/kubernetes#87733

Until K8s bug is fixed, we will run into issues.

Can we look into options to workaround the issue and add the necessary logic in scheduled-scaler code to handle this situation. Some work around has been mentioned in the issue kubernetes/kubernetes#87733

[ericuldall]

Did you try the workaround? Any luck ?

[ericuldall]

After a quick review of the code I imagine the issue may be that it's using an older version of the api. I'm not sure if that's going to be a problem in general or just for that workaround you shared. I'd be curious to see what if that workaround sticks after getting updated by the scheduled scaler.

[NareshKoduru]

@ericuldall,
We are noticing this on the 1.29 version of the k8s cluster with HPA of v2 apiVersion.
Since I can patch the HPA object(adding a label) manually but schedulerScaler unable to make changes to minReplicas I thought of informing this here kubernetes/kubernetes#87733 (comment)