karmada v1.0.3 release

Changes since v1.0.2

Bug Fixes

• karmadactl: fixed the karmada-systemnamespace already existing issue when deployingkarmada-agent` issue. (#1609, @hanweisen)
• karmadactl: Fixed karmada-controller-manager args not honor customized namespace issue. (#1690, @prodanlabs)
• karmada-controller-manager: Fixed the FullyApplied condition of ResourceBinding/ClusterResourceBinding mislabeling issue in case of non-scheduled. (#1518, @huone1)
• karmada-controller-manager: Fixed RB/CRB controller can't aggregate status in case of work condition update issue. (#1524, @mrlihanbo)
• karmada-controller-manager: Fixed panic in case of interpreter webhook returns nil patch. (#1591, @CharlesQQ)
• karmada-aggregate-apiserver: Fixed timeout issue when request cluster/proxy with options -w or logs -f from karmadactl get. (#1631, @XiShanYongYe-Chang)
• karmada-aggregate-apiserver: Fixed exec failed: error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr. (#1641, @pangsq)

Other

• The base image alpine has been promoted to v3.15.1. (#1582, @RainbowMango)

karmada v1.0.2 release

Changes since v1.0.1

Bug Fixes

• karmadactl: Fixed --namespace flag of init command not work issue. (#1452, @sayaoailun)
• karmadactl: Fixed init failure due to data path not clean issue. (#1473, @prodanlabs)
• karmadactl: Fixed init can not select default release version issue. (#1495, @prodanlabs)
• karmadactl: Fixed init can not read KUBECONFIG environment variable issue. (#1482, @lonelyCZ)
• helm chart: Fixed version constraints skip the pre-releases issue. (#1466, @pigletfly)
• karmada-controller-manager: Fixed a bug where resource binding is not created occasionally. (#1384, @dddddai)

karmada v1.1.1 release

Changes since v1.1.0

Bug Fixes

• karmadactl: Fixed --namespace flag of init command not work issue. (#1452, @sayaoailun)
• karmadactl: Fixed init failure due to data path not clean issue. (#1473, @prodanlabs)
• karmadactl: Fixed init can not read KUBECONFIG environment variable issue. (#1482, @lonelyCZ)
• karmadactl: Fixed init can not select default release version issue. (#1496, @prodanlabs)
• helm chart: Fixed version constraints skip the pre-releases issue. (#1466, @pigletfly)

karmada v1.1.0 release

What's New

Multi-Cluster Ingress

The newly introduced MultiClusterIngress API exposes HTTP and HTTPS routes that target multi-cluster services within the Karmada control plane. The specification of MultiClusterIngress is compatible with Kubernetes Ingress.

Traffic routing is controlled by rules defined on the MultiClusterIngress resource, an MultiClusterIngress controller is responsible for fulfilling the ingress. The Multi-Cluster-Nginx Ingress Controller is one of the MultiClusterIngress controller implementations maintained by the community.

(Feature contributors: @GitHubxsy @XiShanYongYe-Chang)

Federated ResourceQuota

The newly introduced FederatedResourceQuota provides constraints that limit total resource consumption per namespace across all clusters. It can limit the number of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

(Feature contributors: @RainbowMango @XiShanYongYe-Chang)

Configurability improvement for performance tuning

The default number of reconciling workers has been enlarged and configurable. A larger number of workers means higher responsiveness but heavier CPU and network load. The number of concurrent workers could be configured by the flags introduced to karmada-controller-manager and karmada-agent.

Flags introduced to karmada-controller-manager:

• --concurrent-work-syncs
• --concurrent-namespace-syncs
• --concurrent-resource-template-syncs
• --concurrent-cluster-syncs
• --concurrent-clusterresourcebinding-syncs
• --concurrent-resourcebinding-syncs

Flags introduced to karmada-agent:

• --concurrent-work-syncs
• --concurrent-cluster-syncs

(Feature contributor: @pigletfly)

Resource Interpreter Webhook Enhancement

Introduced AggregateStatus support for the Resource Interpreter Webhook framework, which enables customized resource status aggregating.

Introduced InterpreterOperationInterpretDependency support for the Resource Interpreter Webhook framework, which enables propagating workload's dependencies automatically.

Refer to Customizing Resource Interpreter for more details.

(Feature contributors: @iawia002 @mrlihanbo)

Other Notable Changes

Bug Fixes

• karmadactl and kubectl-karmada: Fixed that init cannot update the APIService. (@prodanlabs, #1207)
• karmada-controller-manager: Fixed ApplyPolicySucceed event type mistake (should be Normal but not Warning). (@Garrybest, #1267)
• karmada-controller-manager and karmada-agent: Fixed that resync slows down reconciliation. (@Garrybest, #1265)
• karmada-controller-manager/karmada-agent: Fixed continually updating cluster status due to unordered apiEnablements. (@pigletfly, #1304)
• karmada-controller-manager: Fixed that Replicas set by OverridePolicy will be reset by the ReviseReplica interpreterhook. (@likakuli, #1352)
• karmada-controller-manager: Fixed that ResourceBinding couldn't be created in a corner case. (@dddddai, #1368)
• karmada-scheduler: Fixed inaccuracy in requested resources in the case that pod limits are specified but requests are not. (@Garrybest, #1225)
• karmada-scheduler: Fixed spreadconstraints[i].MaxGroups is invalidated in some scenarios. (@huone1, #1324)

Features & Enhancements

• karmadactl: Introduced --tls-min-version flag to specify the minimum TLS version. (@carlory, #1278)
• karmadactl: Improved the get command to show more useful information. (@lonelyCZ, #1270)
• karmada-controller-manager/karmada-agent: Introduced --resync-period flag to specify reflector resync period (defaults to 0, meaning no resync). (@Garrybest, #1261)
• karmada-controller-manager: Introduced --metrics-bind-address flag to specify the customized address for metrics. (@pigletfly, #1341)
• karmada-webhook: Introduced --metrics-bind-address and --health-probe-bind-address flags. (@mrlihanbo, #1346)

Instrumentation (Metrics and Events)

• karmada-controller-manager: Fixed ApplyPolicySucceed event type mistake (should be Normal but not Warning). (@Garrybest, #1267)

Deprecation

• OverridePolicy/ClusterOverridePolicy: The .spec.targetCluster and spec.overriders have been deprecated in favor of spec.overrideRules. (@RainbowMango #1238)
• karmada-aggregate-apiserver: Deprecated --master and --karmada-config flags. Please use --kubeconfig instead. (@carlory, #1336)

Contributors

Thank you to everyone who contributed to this release!

Users whose commits are in this release (alphabetically by user name)

• @AllenZMC
• @ashley-rongfang
• @carlory
• @CuiDengdeng
• @dddddai
• @EvaDD
• @Fish-pro
• @Garrybest
• @helen-frank
• @huone1
• @iawia002
• @jinglina
• @kerthcet
• @liangyongzhenya
• @likakuli
• @lonelyCZ
• @mrlihanbo
• @pigletfly
• @prodanlabs
• @RainbowMango
• @RishiKumarRay
• @Tingtal
• @viniciuspietscher
• @weilaaa
• @wlp1153468871
• @XiShanYongYe-Chang
• @zach593
• @zgfh

karmada v1.0.1 release

Changes since v1.0.0

Bug Fixes

• karmadactl and kubectl-karmada: Fixed init can not update the APIService issue. (#1207, @prodanlabs )
• karmada-controller-manager: Fixed ApplyPolicySucceed event type mistake(should be Normal but Warning). (#1267, @Garrybest )
• karmada-controller-manager and karmada-agent: Fixed resync slow down reconcile issue. (#1265, @Garrybest )

karmada v1.0.0 release

What's New

Aggregated Kubernetes API Endpoint

The newly introduced karmada-aggregated-apiserver component aggregates all registered clusters and allows users to access member clusters through Karmada by the proxy endpoint, e.g.

- Retrieve `Node` from `member1`:  /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes
- Retrieve `Pod` from `member2`: /apis/cluster.karmada.io/v1alpha1/clusters/member2/proxy/api/v1/namespaces/default/pods

Please refer to user guide for more details.

(Feature contributor: @kevin-wangzefeng @GitHubxsy @XiShanYongYe-Chang @mrlihanbo @jrkeen @prodanlabs @carlory @RainbowMango)

Promoting Workloads from Legacy Clusters to Karmada

Legacy workloads running in Kubernetes now can be promoted to Karmada smoothly without container restart.
In favor of promote commands added to Karmada CLI, any kind of Kubernetes resources can be promoted to Karmada easily, e.g.

# Promote deployment(default/nginx) from cluster1 to Karmada
kubectl karmada promote deployment nginx -n default -c cluster1

(Feature contributor: @lonelyCZ @iawia002 @dddddai)

Verified Integration with Ecosystem

Benefiting from the Kubernetes native API support, Karmada can easily integrate the single cluster ecosystem for multi-cluster, multi-cloud purpose. The following components have been verified by the Karmada community:

• argo-cd: refer to working with argo-cd
• Flux: refer to propagating helm charts with flux
• Istio: refer to working with Istio
• Filebeat: refer to working with Filebeat
• Submariner: refer to working with Submariner
• Velero: refer to working with Velero
• Prometheus: refer to working with Prometheus

(Feature contributor: @lfbear @learner0810 @zirain @Rains6 @gy95 @XiShanYongYe-Chang )

OverridePolicy Improvements

By leverage of the new-introduced RuleWithCluster fields to OverridePolicy and ClusterOverridePolicy, users are now able to define override policies with a single policy for specified workloads.

(Feature contributor: @iawia002 @lfbear @RainbowMango @lonelyCZ @jameszhangyukun )

Karmada Installation Improvements

Introduced init command to Karmada CLI. Users are now able to install Karmada by a single command.

Please refer to Installing Karmada for more details.

(Feature contributor: @prodanlabs @lonelyCZ @jrkeen )

Configuring Karmada Controllers

Now all controllers provided by Karmada work as plug-ins. Users are now able to turn off any of them from the default enabled list.
See --controllers flag of karmada-controller-manager and karmada-agent for more details.

(Feature contributor: @snowplayfire @iawia002 @jameszhangyukun )

Resource Interpreter Webhook Enhancement

Introduced ReviseReplica support for the Resource Interpreter Webhook framework, which enables scheduling all customized workloads just like Kubernetes native ones.

Refer to Resource Interpreter Webhook Proposal for more design details.

(Feature contributor: @iawia002)

Other Notable Changes

Bug Fixes

• karmada-controller-manager: Fixed the issue that the annotation of resource template cannot be updated. (@mrlihanbo #1012)
• karmada-controller-manager: Fixed the issue of generating binding reference key. (@JarHMJ #1003)
• karmada-controller-manager: Fixed the inefficiency of en-queue failed task issue. (@Garrybest #1068)

Features & Enhancements

• Karmada CLI: Introduced --cluster-provider flag to join command to specify provider of joining cluster. (@2hangchen #1025)
• Karmada CLI: Introduced taint command to set taints for clusters. (@lonelyCZ #889)
• Karmada CLI: The Applied condition of Work and Scheduled/FullyApplied of ResourceBinding are available for kubectl get. (@lonelyCZ #1110)
• karmada-controller-manager: The cluster discovery feature now supports v1beta1 of cluster-api. (@iawia002 #1029)
• karmada-controller-manager: The Job's startTime and completionTime now available at resource template. (@Garrybest #1034)
• karmada-controller-manager: introduced --controllers flag to enable or disable controllers. (@snowplayfire #1083)
• karmada-controller-manager: Support retain ownerReference from observed objects. (@snowplayfire #1116)
• karmada-controller-manager and karmada-agent: Introduced cluster-cache-sync-timeout flag to specify the time waiting for cache sync. (@snowplayfire #1112)

Instrumentation (Metrics and Events)

• karmada-scheduler-estimator: Introduced /metrics endpoint to emit metrics. (@Garrybest #1030)
• Introduced ApplyPolicy and ScheduleBinding events for resource template. (@mrlihanbo #1070)

Deprecation

• The ReplicaSchedulingPolicy API deprecated at v0.9.0 now has been removed in favor of ReplicaScheduling of PropagationPolicy. (@iawia002 #1161)

Contributors

Thank you to everyone who contributed to this release!

Users whose commits are in this release (alphabetically by user name)

• @2hangchen
• @aven-ai
• @BDXGD
• @carlory
• @dddddai
• @eightzero
• @fanzhihai0215
• @feeltimeQ
• @fleeto
• @Garrybest
• @ghl116
• @gy95
• @haiker2011
• @Haleygo
• @iawia002
• @imroc
• @JackZxj
• @jameszhangyukun
• @JarHMJ
• @jrkeen
• @kevin-wangzefeng
• @leonharetd
• @lfbear
• @lonelyCZ
• @mrlihanbo
• @Phil-sun
• @pigletfly
• @prodanlabs
• @RainbowMango
• @Rains6
• @Shike-Ada
• @snowplayfire
• @wawa0210
• @XiShanYongYe-Chang
• @zirain

karmada v0.10.1 release

Changes since v0.10.0

Bug Fixes

• karmada-controller-manager: fixed the issue of generating binding reference key. (#1003, @JarHMJ)
• karmada-controller-manager: fixed resource template annotations can not update issue. (#1012, @mrlihanbo)

karmada v0.10.0 release

What's New

Resource Interpreter Webhook

The newly introduced Resource Interpreter Webhook framework allows users to implement their own CRD plugins that will be consulted at all parts of propagation process. With this feature, CRDs and CRs will be propagated just like Kubernetes native resources, which means all scheduling primitives also support custom resources. An example as well as some helpful utilities are provided to help users better understand how this framework works.

Refer to Proposal for more details.

(Feature contributor: @RainbowMango, @XiShanYongYe-Chang, @gy95)

Significant Scheduling Enhancement

1. Introduced dynamicWeight primitive to PropagationPolicy and ClusterPropagationPolicy. With this feature, replicas could be divided by a dynamic weight list, and the weight of each cluster will be calculated based on the available replicas during scheduling.
   This feature can balance the cluster's utilization significantly. (#841)

2. Introduced Job schedule (divide) support. A Job that desires many replicas now could be divided into many clusters just like Deployment.
   This feature makes it possible to run huge Jobs across small clusters. (#898)

(Feature contributor: @Garrybest )

Workloads Observation from Karmada Control Plane

After workloads (e.g. Deployments) are propagated to member clusters, users may also want to get the overall workload status across many clusters, especially the status of each pod. In this release, a get subcommand was introduced to the kubectl-karmada. With this command, user are now able get all kinds of resources deployed in member clusters from the Karmada control plane.

For example (get deployment and pods across clusters):

$ kubectl karmada get deployment
NAME    CLUSTER   READY   UP-TO-DATE   AVAILABLE   AGE   ADOPTION
nginx   member2   1/1     1            1           19m   Y
nginx   member1   1/1     1            1           19m   Y
$ kubectl karmada get pods
NAME                     CLUSTER   READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-vzdvt   member1   1/1     Running   0          31m
nginx-6799fc88d8-l55kk   member2   1/1     Running   0          31m

(Feature contributor: @lfbear @QAQ-rookie)

Other Notable Changes

• karmada-scheduler-estimator: The number of pods becomes an important reference when calculating available replicas for the cluster. (@Garrybest, #777)
• The labels (resourcebinding.karmada.io/namespace, resourcebinding.karmada.io/name, clusterresourcebinding.karmada.io/name) which were previously added on the Work object now have been moved to annotations. (@XiShanYongYe-Chang, #752)
• Bugfix: Fixed the impact of cluster unjoining on resource status aggregation. (@dddddai, #817)
• Instrumentation: Introduced events (SyncFailed and SyncSucceed) to the Work object. (@wawa0210, #800)
• Instrumentation: Introduced condition (Scheduled) to the ResourceBinding and ClusterResourceBinding. (@dddddai, #823)
• Instrumentation: Introduced events (CreateExecutionNamespaceFailed and RemoveExecutionNamespaceFailed) to the Cluster object. (@pigletfly, #749)
• Instrumentation: Introduced several metrics (workqueue_adds_total, workqueue_depth, workqueue_longest_running_processor_seconds, workqueue_queue_duration_seconds_bucket) for karmada-agent and karmada-controller-manager. (@Garrybest, #831)
• Instrumentation: Introduced condition (FullyApplied) to the ResourceBinding and ClusterResourceBinding. (@lonelyCZ, #825)
• karmada-scheduler: Introduced feature gates. (@iawia002, #805)
• karmada-controller-manager: Deleted resources from member clusters that use "Background" as the default delete option. (@RainbowMango, #970)

Contributors

Thank you to everyone who contributed to this release!

Users whose commits are in this release (alphabetically by user name)

• @2hangchen
• @algebra2k
• @benjaminhuo
• @Charlie17Li
• @ctripcloud
• @dddddai
• @duguhaotian
• @fleeto
• @Garrybest
• @gf457832386
• @gy95
• @hyschumi
• @iawia002
• @jameszhangyukun
• @kerthcet
• @kevin-wangzefeng
• @learner0810
• @lfbear
• @lonelyCZ
• @mrlihanbo
• @penghuima
• @Phil-sun
• @pigletfly
• @QAQ-rookie
• @RainbowMango
• @snowplayfire
• @TeodoraBoros
• @wawa0210
• @wzshiming
• @XiShanYongYe-Chang
• @youhonglian
• @yvoilee

karmada v0.9.0 release

What's New

Upgrading support
Users are now able to upgrade from the previous version smoothly. With the multiple version feature of CRD, objects with different schemas can be automatically converted between versions. Karmada uses the semantic versioning and will provide workarounds for inevitable breaking changes.

In this release, the ResourceBining and ClusterResourceBinding promote to v1alpha2 and the previous v1alpha1 version is still available for one more release. With the upgrading instruction, the previous version of Karmada can promote smoothly.

(Feature contributor: @RainbowMango )

Introduced karmada-scheduler-estimator to facilitate end-to-end scheduling accuracy
Karmada scheduler aims to assign workload to clusters according to constraints and available resources of each member cluster. The kube-scheduler working on each cluster takes the responsibility to assign Pods to Nodes.
Even though Karmada has the capacity to reschedule failure workload between member clusters, but the community still commits lots of effort to improve the accuracy of the end-to-end scheduling.

The karmada-scheduler-estimator is the effective assistant of karmada-scheduler, it provides prediction-based scheduling decisions that can significantly improve the scheduling efficiency and avoid the wave of rescheduling among clusters. Note that this feature is implemented as a pluggable add-on. For the instructions please refer to scheduler estimator guideline.

(Feature contributor: @Garrybest )

Maintainability improvements
A bunch of significant maintainability improvements were added to this release, including:

• Simplified Karmada installation with helm chart.
  (Feature contributor: @algebra2k @jrkeen )

• Provided metrics to observe scheduler status, the metrics API now served at /metrics of karmada-scheduler.
  With these metrics, users are now able to evaluate the scheduler's performance and identify the bottlenecks.
  (Feature contributor: @qianjun1993 )

• Provided events to Karmada API objects as supplemental information to debug problems.
  (Feature contributor: @pigletfly )

Other Notable Changes

• karmada-controller-manager: The ResourceBinding/ClusterResourceBinding won't be deleted after associate PropagationPolicy/ClusterPropagationPolicy is removed and is still available until resource template is removed.(@qianjun1993, #601)
• Introduced --leader-elect-resource-namespace which is used to specify the namespace of election object to components karmada-controller-manager/karmada-scheduler/karmada-agent`. (@XiShanYongYe-Chang #698)
• Deprecation: The API ReplicaSchedulingPolicy has been deprecated and will be removed from the following release. The feature now has been integrated into ReplicaScheduling.
• Introduced kubectl-karmada commands as the extensions for kubectl. (@XiShanYongYe-Chang #686)
• karmada-controller-manager introduced a version command to represent version information. (@RainbowMango #717 )
• karmada-scheduler/karmada-webhook/karmada-agent/karmada-scheduler-estimator introduced a version command to represent version information. (@lonelyCZ #719 )
• Provided instructions about how to use the Submariner to connect the network between member clusters. (@XiShanYongYe-Chang #737 )
• Added four metrics to the karmada-scheduler to monitor scheduler performance. (@qianjun1993 #747)

Contributors

Thank you to everyone who contributed to this release!

Users whose commits are in this release (alphabetically by user name)

• @2hangchen
• @CKchen0726
• @dddddai
• @ESonata
• @fleeto
• @Garrybest
• @Hrishikesh156
• @iawia002
• @jrkeen
• @just1900
• @kerthcet
• @lfbear
• @lonelyCZ
• @mrlihanbo
• @MukulKolpe
• @Onyinye91-ctrl
• @phantooom
• @pigletfly
• @qianjun1993
• @RainbowMango
• @saiyan86
• @smartding
• @XiShanYongYe-Chang
• @zhuyaguang

karmada v0.8.0 release

What's New

Automatic cluster discovery with cluster-api
For users who are using cluster-api (sigs.k8s.io/cluster-api), Karmada is now able to automatically discover & join clusters when provisioned, and unjoin them in case of destroyed.

Note that this features is implemented as a built-in plugin. To enbale it, simply indicate the following to flags in karmada-controller-manager config:

--cluster-api-kubeconfig string        Path to the cluster-api management cluster kubeconfig file.
--cluster-api-context string           Name of the cluster context in cluster-api management cluster kubeconfig file.

(Feature contributor: @XiShanYongYe-Chang )

Introduced CommandOverrider and ArgsOverrider to simplify commands customization per cluster
For multi-cluster applications, it's quite common to set different arguments when running on different clusters or environments.
In this release, two overrider plugins: CommandOverrider and ArgsOverrider are introduced, based on industry best practices. These two handy tools allow users to declare complex declarations and avoid configuration mistakes.

Workload types supported now are: Deployment, ReplicaSet, DaemonSet, StatefulSet and Pod, more types including CRDs will be supported in later releases.

(Feature contributor: @lfbear @betaincao )

Better integration support with Kubernetes ecosystem
The Kubernetes native APIs support and patterns to run cloud-native applications of Karmada make it quite easy to quickly integrate with other projects in the Kubernetes ecosystem.

In release, several useful features that will help Karmada work seamlessly with other systems.

• ResourceBinding and ClusterResourceBinding now supports present the applied status. (@pigletfly #595)
• More types of resources now support aggregating status to the resource template, inlcuding Job, Service, and Ingress. (@mrlihanbo #609)
• argo-cd is also verified to run full featured with Karmada to achieve multi-cluster GitOps.

Other Notable Changes

• karmadactl: introduced cordon and uncordon commands to mark a cluster schedulable and un-schedulable. (#464, @algebra2k )
• karmada-controller-manager: introduced --skipped-propagating-namespaces flag to skip resources in certain namespaces from propagating. (#533, @pigletfly )
• karmada-controller-manager/karmada-agent/karmada-scheduler: Introduced flags to config the QPS and burst which are used to control the client traffic interacting with Karmada or member cluster's kube-apiserver. (#611, @Garrybest )
  • --cluster-api-qps QPS to use while talking with cluster kube-apiserver.
  • --cluster-api-burst Burst to use while talking with cluster kube-apiserver.
  • --kube-api-qps QPS to use while talking with karmada-apiserver.
  • --kube-api-burst Burst to use while talking with karmada-apiserver.
• Karmada quick-start scripts now support running on Mac OS. (#538, @lfbear )

Contributors

Thank you to everyone who contributed to this release!

Users whose commits are in this release (alphabetically by user name)

• @algebra2k
• @betaincao
• @garfcat
• @Garrybest
• @gy95
• @Iceber
• @lfbear
• @lushenle
• @mrlihanbo
• @phantooom
• @pigletfly
• @qianjun1993
• @RainbowMango
• @wawa0210
• @weilaaa
• @XiShanYongYe-Chang