How To Setup kubernetes Custom Metrics Scaling For HPA in GKE (Google Container Engine)

1

By default, kubernetes HPA supports only CPU metrics for pod autoscaling. However, now the autoscaling beta API version has the capability to scale HPA based on memory and other custom metrics.

I have been trying to set this up on GKE (Google Container engine). But I am stuck at HPA part where it is not able to get the custom metrics.

So here is what I did,

  1. Setup Prometheus operator for custom metrics aggregation.
  2. Custom metrics API registration
  3. Deployed a pod exposing custom metrics.
  4. Setup HPA for scaling using custom metrics. (Getting error here as the HPA is not receiving the custom metrics from the pod.)

Also, I see from one blog post that for custom metric scaling, you need to enable --horizontal-pod-auto-scaler-use-rest-clients=true for the kubernetes cluster. So one doubt I have is that, by default in the latest GKE clusters, is it enable?

Following are the resources I used for setting up this on GKE.

  1. https://blog.openshift.com/kubernetes-1-8-now-custom-metrics/
  2. https://github.com/kubeless/kubeless/tree/master/manifests/autoscaling
  3. https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/

Also, I have raised an issue kubeless repo for the same (https://github.com/kubeless/kubeless/issues/493)

2

Hi,

GKE have been supporting HPA with custom metrics since version 1.9. If you have a group of horizontally autoscaled pods inside your cluster each exporting a custom metric then you can set an average per pod target for that metric.

An example of that would be an autoscaled deployment of a frontend where each replica exports its current QPS. One could set the average target of QPS per frontend pod and use the HPA to scale the deployment up and down accordingly. You can find the documentation and a tutorial explaining how to set this up here: https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling

Kubernetes 1.10 becoming available on GKE will extend the support for custom metrics to include metrics not attached to any Kubernetes object. This will give you the ability to scale a deployment based on any metric listed here, for example number of messages in Google Pub/Sub queue.

Best,

1 Like
3

Thanks, @Maks_Osowski for the update. We have done a POC on the same… :slight_smile: