CKS Enhancements docs #458

nvazquez · 2024-12-02T11:09:48Z

📚 Documentation preview 📚: https://cloudstack-documentation--458.org.readthedocs.build/en/458/

DaanHoogland · 2024-12-18T15:35:45Z

source/plugins/cloudstack-kubernetes-service.rst

@@ -15,7 +15,10 @@ CloudStack Kubernetes Service

 The Kubernetes Service plugin adds Kubernetes integration to CloudStack. The plugin is disabled by default and an admin can enable it using a Global Setting. It enables users to run containerized services using Kubernetes clusters.

-With CoreOS having reached EOL, from 4.16 the Kubernetes Service Plugin will use the existing SystemVM Template for deploying kubernetes clusters. For installation of Kubernetes binaries on the cluster nodes, a binaries ISO is used for each Kubernetes version to be made available via CloudStack. This allows faster, offline installation of Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters.
+With CoreOS having reached EOL, from 4.16 the Kubernetes Service Plugin will use the existing SystemVM Template by default for deploying kubernetes clusters. For installation of Kubernetes binaries on the cluster nodes, a binaries ISO is used for each Kubernetes version to be made available via CloudStack. This allows faster, offline installation of Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters.


Suggested change

With CoreOS having reached EOL, from 4.16 the Kubernetes Service Plugin will use the existing SystemVM Template by default for deploying kubernetes clusters. For installation of Kubernetes binaries on the cluster nodes, a binaries ISO is used for each Kubernetes version to be made available via CloudStack. This allows faster, offline installation of Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters.

With CoreOS having reached EOL, from 4.16 on the Kubernetes Service Plugin will use

the existing SystemVM Template by default for deploying kubernetes clusters. For

installation of Kubernetes binaries on the cluster nodes, a binaries ISO is used for each

Kubernetes version to be made available via CloudStack. This allows faster, offline

installation of Kubernetes binaries and docker images along with support for adding

multiple versions of Kubernetes for upgrades and running different clusters.

DaanHoogland · 2024-12-18T15:36:44Z

source/plugins/cloudstack-kubernetes-service.rst

+With CoreOS having reached EOL, from 4.16 the Kubernetes Service Plugin will use the existing SystemVM Template by default for deploying kubernetes clusters. For installation of Kubernetes binaries on the cluster nodes, a binaries ISO is used for each Kubernetes version to be made available via CloudStack. This allows faster, offline installation of Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters.
+
+.. note::
+   Since version 4.21.0 users can choose different templates for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. These templates must be previously registered selecting the 'For CKS' option.


Suggested change

Since version 4.21.0 users can choose different templates for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. These templates must be previously registered selecting the 'For CKS' option.

Since version 4.21.0 users can choose different templates for different types of nodes

(worker, control, etcd nodes) for deploying Kubernetes clusters. These templates must

be previously registered selecting the 'For CKS' option.

Hi @DaanHoogland I see most of the suggested changes are splitting the single line into multiple lines. Is this a problem on having them as single lines?

source/plugins/cloudstack-kubernetes-service.rst

nvazquez · 2025-05-24T02:45:01Z

@blueorangutan docbuild

blueorangutan · 2025-05-24T02:46:05Z

@nvazquez a Jenkins job has been kicked to build the document. I'll keep you posted as I make progress.

blueorangutan · 2025-05-24T02:47:02Z

QA-Doc build preview: https://qa.cloudstack.cloud/builds/docs-build/pr/458. (QA-JID 349)

Pearl1594 · 2025-05-26T10:33:32Z

source/plugins/cloudstack-kubernetes-service.rst

+With CoreOS having reached EOL, from 4.16 the Kubernetes Service Plugin will use the existing SystemVM Template by default for deploying kubernetes clusters. For installation of Kubernetes binaries on the cluster nodes, a binaries ISO is used for each Kubernetes version to be made available via CloudStack. This allows faster, offline installation of Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters.
+
+.. note::
+   Since version 4.21.0 users can choose different templates and service offerings for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. The templates must be previously registered selecting the 'For CKS' option.


Suggested change

Since version 4.21.0 users can choose different templates and service offerings for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. The templates must be previously registered selecting the 'For CKS' option.

From version 4.21.0, users can choose different templates and service offerings for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. The templates must be previously registered selecting the 'For CKS' option.

Pearl1594 · 2025-05-26T10:37:46Z

source/plugins/cloudstack-kubernetes-service.rst

@@ -82,6 +88,17 @@ Eg:
 **NOTE:**
 From ACS 4.16 onwards, Kubernetes versions >= 1.20.x are only supported (https://endoflife.date/kubernetes).

+**NOTE:**
+From ACS 4.21 onwards, it is possible specify the version for etcd binaries on the create-kubernetes-binaries-iso.sh script as an optional parameter ETCD_VERSION. When the ETCD_VERSION parameter is set, the specified etcd version binaries are downloaded and stored on the /etcd directory on the Kubernetes ISO.


Suggested change

From ACS 4.21 onwards, it is possible specify the version for etcd binaries on the create-kubernetes-binaries-iso.sh script as an optional parameter ETCD_VERSION. When the ETCD_VERSION parameter is set, the specified etcd version binaries are downloaded and stored on the /etcd directory on the Kubernetes ISO.

From ACS 4.21 onwards, it is possible to specify the version for etcd binaries in the create-kubernetes-binaries-iso.sh script as an optional parameter - ETCD_VERSION. When the ETCD_VERSION parameter is set, the specified etcd version binaries are downloaded and stored in the Kubernetes ISO.

Pearl1594 · 2025-05-26T10:42:28Z

source/plugins/cloudstack-kubernetes-service.rst

+
+   # ./create-kubernetes-binaries-iso.sh ./ 1.27.2 1.3.0 1.27.0 https://raw.githubusercontent.com/weaveworks/weave/master/prog/weave-kube/weave-daemonset-k8s-1.11.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml setup-v1.27.2 3.5.1
+
+The Kubernetes ISOs generated with an specific etcd version are required to create separate etcd nodes on Kubernetes clusters. See :ref:`flexible-kubernetes-clusters`.


Suggested change

The Kubernetes ISOs generated with an specific etcd version are required to create separate etcd nodes on Kubernetes clusters. See :ref:`flexible-kubernetes-clusters`.

To deploy Kubernetes clusters with

Kubernetes ISOs built with a specified etcd version are necessary for creating Kubernetes clusters with separate etcd nodes. See :ref:`flexible-kubernetes-clusters`.

Pearl1594 · 2025-05-26T10:43:10Z

source/plugins/cloudstack-kubernetes-service.rst

@@ -92,6 +109,10 @@ Once the ISO has been built for a desired Kubernetes version, it can be added by

 |cks-add-version-form.png|

+.. note::
+   Since 4.21.0 it is possible to deploy separate dedicated etcd nodes. This requires


Suggested change

Since 4.21.0 it is possible to deploy separate dedicated etcd nodes. This requires

From 4.21.0, it is possible to deploy separate dedicated etcd nodes. This requires

Pearl1594 · 2025-05-26T10:46:06Z

source/plugins/cloudstack-kubernetes-service.rst

@@ -194,6 +215,17 @@ New Kubernetes clusters can be created using the API or via the UI. User will be

 |cks-create-cluster-form.png|

+Since 4.21.0, the Hypervisor selection is available for Kubernetes Cluster nodes. By default the Hypervisor selection is empty.


Suggested change

Since 4.21.0, the Hypervisor selection is available for Kubernetes Cluster nodes. By default the Hypervisor selection is empty.

From 4.21.0, you can select the hypervisor type for Kubernetes cluster nodes. By default, no hypervisor is selected.

Pearl1594 · 2025-05-26T11:02:23Z

source/plugins/cloudstack-kubernetes-service.rst

+
+- The Management Server’s SSH Public key must be added at the cloud user’s authorized_keys file at `~/.ssh/authorized_keys`.
+
+The VM Instances meeting the requirements above must be added to the Kubernetes cluster by the `addNodesToKubernetesCluster` API specifying:


Suggested change

The VM Instances meeting the requirements above must be added to the Kubernetes cluster by the `addNodesToKubernetesCluster` API specifying:

The VM Instances meeting the requirements above can be added to the Kubernetes cluster by the `addNodesToKubernetesCluster` API specifying:

Pearl1594 · 2025-05-26T11:04:39Z

source/plugins/cloudstack-kubernetes-service.rst

+- **manualupgrade** (optional parameter that indicates if the node is marked for manual upgrade and excluded from the Kubernetes cluster upgrade operation)
+
+.. note::
+   Users are able to add nodes to Kubernetes cluster and mark them for manual upgrade only. Once the nodes are marked for manual upgrade, the future cluster upgrade operations will exclude these nodes and their Kubernetes version won't be upgraded.


Suggested change

Users are able to add nodes to Kubernetes cluster and mark them for manual upgrade only. Once the nodes are marked for manual upgrade, the future cluster upgrade operations will exclude these nodes and their Kubernetes version won't be upgraded.

Users will have the ability to add nodes to the Kubernetes cluster and mark them for manual upgrade. Once the nodes are marked for manual upgrade, the future cluster upgrade operations will exclude these nodes i.e., the Kubernetes version won't be upgraded.

Pearl1594 · 2025-05-26T11:05:16Z

source/plugins/cloudstack-kubernetes-service.rst

+
+- Addition of port-forwarding rules and firewall rules (for isolated networks)
+
+- VM is rebooted with the Kubernetes config passed as userdata


Suggested change

- VM is rebooted with the Kubernetes config passed as userdata

- VM is rebooted with the Kubernetes configuration passed as userdata

Pearl1594 · 2025-05-26T11:06:02Z

source/plugins/cloudstack-kubernetes-service.rst

+Removing an external worker node from a Kubernetes cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+External worker nodes must be removed from a Kubernetes cluster by the `removeNodesFromKubernetesCluster` API specifying:


Suggested change

External worker nodes must be removed from a Kubernetes cluster by the `removeNodesFromKubernetesCluster` API specifying:

External worker nodes can be removed from a Kubernetes cluster by the `removeNodesFromKubernetesCluster` API specifying:

Pearl1594 · 2025-05-26T11:06:45Z

source/plugins/cloudstack-kubernetes-service.rst

+Dedicate specific hosts/clusters to a specific domain for CKS cluster deployment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Administratos are able to dedicate hosts to a domain or account. CloudStack will take the host dedication in consideration when deploying Kubernetes clusters.


Suggested change

Administratos are able to dedicate hosts to a domain or account. CloudStack will take the host dedication in consideration when deploying Kubernetes clusters.

Administrators are able to dedicate hosts to a domain or account. CloudStack will take the host dedication into consideration when deploying Kubernetes clusters.

Pearl1594 · 2025-05-26T11:11:59Z

@nvazquez Should we add some images for:

Custom template registration - where for CKS is highlighted
CNI Configuration
Thanks.

kiranchavala · 2025-05-28T05:51:00Z

@nvazquez Should we add some images for:

* Custom template registration - where `for CKS` is highlighted

* CNI Configuration
  Thanks.

Yes, screenshots about registering cni userdata would be helpful

also sample kubectl commands to verify


root@cksclusteradditon-control-190ca0ce253:~# kubectl get pods -A

NAMESPACE             NAME                                                           READY  STATUS                  RESTARTS         AGE

kube-system           calico-kube-controllers-8d76c5f9b-pkhcv                        1/1    Running                 6 (44m ago)      2d21h

kube-system           calico-node-n4msg                                              1/1    Running                 0                2d21h

kube-system           calico-node-pdz2w                                              1/1    Running                 0                2d18h

kube-system           calico-node-slmg2                                              1/1    Running                 0                2d21h



root@cksclusteradditon-control-190ca0ce253:~# kubectl get bgppeer

NAME        AGE

bgp-peer-1  2d22h



root@cksclusteradditon-control-190ca0ce253:~# kubectl get bgpconfiguration

NAME     AGE

default  2d22h


root@cksclusteradditon-control-190ca0ce253:~# kubectl describe bgpconfiguration

Name:        default

Namespace:

Labels:      <none>

Annotations: <none>

API Version: crd.projectcalico.org/v1

Kind:        BGPConfiguration

Metadata:

 Creation Timestamp: 2024-07-19T08:25:14Z

 Generation:         1

 Resource Version:   580

 UID:                2b927b4e-82d3-4200-a3c1-9bf0cd5f5824

Spec:

 As Number:           65145

 Log Severity Screen: Debug

Events:                <none>

kiranchavala · 2025-05-28T08:00:59Z

@nvazquez also can we update one test iso which has external etcd

https://download.cloudstack.org/testing/custom_templates/ubuntu/22.04/

nvazquez added 2 commits December 2, 2024 08:08

Add note about node templtes

359f414

Add API params for Advanced Settings

4eb88ed

rohityadavcloud added this to the 4.21.0 milestone Dec 2, 2024

DaanHoogland reviewed Dec 18, 2024

View reviewed changes

source/plugins/cloudstack-kubernetes-service.rst Outdated Show resolved Hide resolved

DaanHoogland reviewed Dec 18, 2024

View reviewed changes

source/plugins/cloudstack-kubernetes-service.rst Outdated Show resolved Hide resolved

Extend documentation

7b04580

nvazquez marked this pull request as ready for review May 24, 2025 02:44

Pearl1594 reviewed May 26, 2025

View reviewed changes

nvazquez mentioned this pull request May 28, 2025

CKS Enhancements apache/cloudstack#9102

Open

22 tasks

	Since version 4.21.0 users can choose different templates and service offerings for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. The templates must be previously registered selecting the 'For CKS' option.
	From version 4.21.0, users can choose different templates and service offerings for different types of nodes (worker, control, etcd nodes) for deploying Kubernetes clusters. The templates must be previously registered selecting the 'For CKS' option.

	From ACS 4.21 onwards, it is possible specify the version for etcd binaries on the create-kubernetes-binaries-iso.sh script as an optional parameter ETCD_VERSION. When the ETCD_VERSION parameter is set, the specified etcd version binaries are downloaded and stored on the /etcd directory on the Kubernetes ISO.
	From ACS 4.21 onwards, it is possible to specify the version for etcd binaries in the create-kubernetes-binaries-iso.sh script as an optional parameter - ETCD_VERSION. When the ETCD_VERSION parameter is set, the specified etcd version binaries are downloaded and stored in the Kubernetes ISO.


		# ./create-kubernetes-binaries-iso.sh ./ 1.27.2 1.3.0 1.27.0 https://raw.githubusercontent.com/weaveworks/weave/master/prog/weave-kube/weave-daemonset-k8s-1.11.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml setup-v1.27.2 3.5.1

		The Kubernetes ISOs generated with an specific etcd version are required to create separate etcd nodes on Kubernetes clusters. See :ref:`flexible-kubernetes-clusters`.

	The Kubernetes ISOs generated with an specific etcd version are required to create separate etcd nodes on Kubernetes clusters. See :ref:`flexible-kubernetes-clusters`.
	To deploy Kubernetes clusters with
	Kubernetes ISOs built with a specified etcd version are necessary for creating Kubernetes clusters with separate etcd nodes. See :ref:`flexible-kubernetes-clusters`.

	Since 4.21.0 it is possible to deploy separate dedicated etcd nodes. This requires
	From 4.21.0, it is possible to deploy separate dedicated etcd nodes. This requires

		@@ -194,6 +215,17 @@ New Kubernetes clusters can be created using the API or via the UI. User will be

		\|cks-create-cluster-form.png\|

		Since 4.21.0, the Hypervisor selection is available for Kubernetes Cluster nodes. By default the Hypervisor selection is empty.

	Since 4.21.0, the Hypervisor selection is available for Kubernetes Cluster nodes. By default the Hypervisor selection is empty.
	From 4.21.0, you can select the hypervisor type for Kubernetes cluster nodes. By default, no hypervisor is selected.


		- The Management Server’s SSH Public key must be added at the cloud user’s authorized_keys file at `~/.ssh/authorized_keys`.

		The VM Instances meeting the requirements above must be added to the Kubernetes cluster by the `addNodesToKubernetesCluster` API specifying:

	Users are able to add nodes to Kubernetes cluster and mark them for manual upgrade only. Once the nodes are marked for manual upgrade, the future cluster upgrade operations will exclude these nodes and their Kubernetes version won't be upgraded.
	Users will have the ability to add nodes to the Kubernetes cluster and mark them for manual upgrade. Once the nodes are marked for manual upgrade, the future cluster upgrade operations will exclude these nodes i.e., the Kubernetes version won't be upgraded.


		- Addition of port-forwarding rules and firewall rules (for isolated networks)

		- VM is rebooted with the Kubernetes config passed as userdata

	External worker nodes must be removed from a Kubernetes cluster by the `removeNodesFromKubernetesCluster` API specifying:
	External worker nodes can be removed from a Kubernetes cluster by the `removeNodesFromKubernetesCluster` API specifying:

	Administratos are able to dedicate hosts to a domain or account. CloudStack will take the host dedication in consideration when deploying Kubernetes clusters.
	Administrators are able to dedicate hosts to a domain or account. CloudStack will take the host dedication into consideration when deploying Kubernetes clusters.

CKS Enhancements docs #458

Are you sure you want to change the base?

CKS Enhancements docs #458

Conversation

nvazquez commented Dec 2, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nvazquez commented May 24, 2025

Uh oh!

blueorangutan commented May 24, 2025

Uh oh!

blueorangutan commented May 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Pearl1594 commented May 26, 2025

Uh oh!

kiranchavala commented May 28, 2025

Uh oh!

kiranchavala commented May 28, 2025

Uh oh!

Uh oh!

nvazquez commented Dec 2, 2024 •

edited by github-actions bot

Loading