dynamically create node pools on hetzner?

[leblancmeneses]

[Cluster Autoscaler] : The Deployment "cluster-autoscaler" is invalid: metadata.annotations: Too long: must have at most 262144 bytes"]

This occurs when I try adding 23 total pools upfront.

It seems the config gets serialized and then k8s decides what to create/delete .

I wanted a generic system of pools. If I can't pre-create them is there a way I can dynamically create the pools? I'm trying to build a custom cost optimizer based on hetzner current pricing for my region.

I can create 12 pools (masters spin up but workers with instance: 1 do not):

• Limit: 262,144 bytes (256 KB)
• Used: 242,876 bytes (93% of limit) - with 12 pools how i have configured
• Remaining: 19,268 bytes (~19 KB) - I can put 1 extra pool but no more

At 4 pools things work like expected.

kubectl get deployment -n kube-system cluster-autoscaler -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      very long json also containing base64 HCLOUD_CLUSTER_CONFIG.
  
.....      
      
        env:
        - name: HCLOUD_CLUSTER_CONFIG
          value: long-base64-encoded-config          

If needed, can it be added as a configmap and mounted on cluster-autoscaler?

[vitobotta]

Wow! What would you do with 23 node pools? :D What's the point to have more node pools than even the available instance types? And anyway yes, there is a limit to the size of the whole autoscaler config, because it has to include al the code required to initialize the autoscaled node and set up k3s. Can you elaborate on what you are proposing re: configmap?

[leblancmeneses]

there is 11 in hil:

nodeGroups:
  # Skipping cax11 - not available in requested regions
  # Skipping cax21 - not available in requested regions
  # Skipping cax31 - not available in requested regions
  # Skipping cax41 - not available in requested regions
  - instanceType: ccx13
    family: ccx
    defaults:
      vcpu: 2
      ram: 8
      arch: amd64
      dedicated: true
      priceHourly: 0.0233000000
    regions:
      - { region: hil }

  - instanceType: ccx23
    family: ccx
    defaults:
      vcpu: 4
      ram: 16
      arch: amd64
      dedicated: true
      priceHourly: 0.0465000000
    regions:
      - { region: hil }

  - instanceType: ccx33
    family: ccx
    defaults:
      vcpu: 8
      ram: 32
      arch: amd64
      dedicated: true
      priceHourly: 0.0890000000
    regions:
      - { region: hil }

  - instanceType: ccx43
    family: ccx
    defaults:
      vcpu: 16
      ram: 64
      arch: amd64
      dedicated: true
      priceHourly: 0.1779000000
    regions:
      - { region: hil }

  - instanceType: ccx53
    family: ccx
    defaults:
      vcpu: 32
      ram: 128
      arch: amd64
      dedicated: true
      priceHourly: 0.3558000000
    regions:
      - { region: hil }

  - instanceType: ccx63
    family: ccx
    defaults:
      vcpu: 48
      ram: 192
      arch: amd64
      dedicated: true
      priceHourly: 0.5337000000
    regions:
      - { region: hil }

  - instanceType: cpx11
    family: cpx
    defaults:
      vcpu: 2
      ram: 2
      arch: amd64
      dedicated: false
      priceHourly: 0.0080000000
    regions:
      - { region: hil }

  # Skipping cpx12 - not available in requested regions
  - instanceType: cpx21
    family: cpx
    defaults:
      vcpu: 3
      ram: 4
      arch: amd64
      dedicated: false
      priceHourly: 0.0161000000
    regions:
      - { region: hil }

  # Skipping cpx22 - not available in requested regions
  - instanceType: cpx31
    family: cpx
    defaults:
      vcpu: 4
      ram: 8
      arch: amd64
      dedicated: false
      priceHourly: 0.0289000000
    regions:
      - { region: hil }

  # Skipping cpx32 - not available in requested regions
  - instanceType: cpx41
    family: cpx
    defaults:
      vcpu: 8
      ram: 16
      arch: amd64
      dedicated: false
      priceHourly: 0.0537000000
    regions:
      - { region: hil }

  # Skipping cpx42 - not available in requested regions
  - instanceType: cpx51
    family: cpx
    defaults:
      vcpu: 16
      ram: 32
      arch: amd64
      dedicated: false
      priceHourly: 0.1074000000
    regions:
      - { region: hil }

but I do have a cross join to add taints to all of this specific to test isolation.

                'labels': [{'key': 'rh-cost-optimizer-test', 'value': 'true'}],
                'taints': [{'key': 'rh-cost-optimizer-test', 'value': 'true', 'effect': 'NoSchedule'}],

gets me 22. Then 1 more for infra-core nodes to keep critical pods from everything else.

---

In practice, I haven't been able to get past configuring 5 pools, due to the implementation.

Instead of:

set_env_variable(env_vars, HCLOUD_CLUSTER_CONFIG_VAR, Base64.strict_encode(build_config_json))

that causes the issue due to the large size; have this point to a config map.
Does k3s require this approach or is this ENV specific to hetzner-k3s implementation?

AI created pr that hasn't been tested: https://github.com/leblancmeneses/hetzner-k3s/pull/1/files

[vitobotta]

The configmap approach would work, at the cost of not being able to customize the autoscaler manifest URL since we'd need to process it. But it's doable. I can take a look but I am very busy with work at the moment (so much so that I had to cancel my Christmas holidays) so I am not sure when i will be able to look at it.

[leblancmeneses]

worked on it a bit today because I need this. Using binary from main with my modifications for now.
#697