Add e2e testing for the customized control panel of Karmadactl init. #6728
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,76 @@ | ||||||||
| #!/usr/bin/env bash | ||||||||
| # Copyright 2021 The Karmada Authors. | ||||||||
| # | ||||||||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||||||||
| # you may not use this file except in compliance with the License. | ||||||||
| # You may obtain a copy of the License at | ||||||||
| # | ||||||||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||||||||
| # | ||||||||
| # Unless required by applicable law or agreed to in writing, software | ||||||||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||||||||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||||||||
| # See the License for the specific language governing permissions and | ||||||||
| # limitations under the License. | ||||||||
|
|
||||||||
| set -o errexit | ||||||||
| set -o nounset | ||||||||
| set -o pipefail | ||||||||
|
|
||||||||
| # This script is used to create a blank cluster for karmadactl init e2e testing. | ||||||||
| # It creates a host cluster and prepares necessary components for testing. | ||||||||
| # This script depends on utils in: ${REPO_ROOT}/hack/util.sh | ||||||||
| # 1. Used by developers to test karmadactl init with custom control plane setup | ||||||||
|
|
||||||||
| function usage() { | ||||||||
| echo "Usage:" | ||||||||
| echo " hack/init-e2e-environment.sh [-h]" | ||||||||
| echo " h: print help information" | ||||||||
| } | ||||||||
|
|
||||||||
| while getopts 'h' OPT; do | ||||||||
| case $OPT in | ||||||||
| h) | ||||||||
| usage | ||||||||
| exit 0 | ||||||||
| ;; | ||||||||
| ?) | ||||||||
| usage | ||||||||
| exit 1 | ||||||||
| ;; | ||||||||
| esac | ||||||||
| done | ||||||||
|
|
||||||||
| REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. | ||||||||
| source "${REPO_ROOT}"/hack/util.sh | ||||||||
|
|
||||||||
| # variable define | ||||||||
| export KUBECONFIG_PATH=${KUBECONFIG_PATH:-"${HOME}/.kube"} | ||||||||
| export MAIN_KUBECONFIG=${MAIN_KUBECONFIG:-"${KUBECONFIG_PATH}/karmada.config"} | ||||||||
| export HOST_CLUSTER_NAME=${HOST_CLUSTER_NAME:-"karmada-host"} | ||||||||
|
|
||||||||
| # step1: set up a base development environment | ||||||||
| "${REPO_ROOT}"/hack/setup-dev-base.sh | ||||||||
| export KUBECONFIG="${MAIN_KUBECONFIG}" | ||||||||
|
|
||||||||
| # step2: prepare and copy the newest crds for karmadactl init | ||||||||
| # step2.1: prepare the newest crds for karmadactl init | ||||||||
| echo "Prepare the newest crds for karmadactl init" | ||||||||
| cd charts/karmada/ | ||||||||
| cp -r _crds crds | ||||||||
| tar -zcvf ../../crds.tar.gz crds | ||||||||
|
There was a problem hiding this comment. The temporary
Suggested change
|
||||||||
| cd - | ||||||||
|
|
||||||||
| # step2.2: Verify CRDs package is created successfully | ||||||||
| if [ ! -f "./crds.tar.gz" ]; then | ||||||||
| echo "ERROR: Failed to create CRDs package at ./crds.tar.gz" | ||||||||
| exit 1 | ||||||||
| fi | ||||||||
| echo "CRDs package created successfully: ./crds.tar.gz" | ||||||||
|
|
||||||||
| # step2.3: Copy the local crds.tar.gz file to the specified path | ||||||||
| DATA_DIR="${HOME}/karmada-data" | ||||||||
| mkdir -p "${DATA_DIR}" | ||||||||
| cp ./crds.tar.gz "${DATA_DIR}/crds.tar.gz" | ||||||||
|
There was a problem hiding this comment. The
Suggested change
|
||||||||
|
|
||||||||
| echo "Environment is ready for running karmadactl init e2e tests." | ||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| #!/usr/bin/env bash | ||
| # Copyright 2020 The Karmada Authors. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| set -o errexit | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| # This script runs init e2e test against a blank cluster to test karmadactl init functionality. | ||
| # You should prepare your environment in advance and following environment may be you need to set or use default one. | ||
| # - HOST_CLUSTER_KUBECONFIG: absolute path of host cluster KUBECONFIG file. | ||
| # | ||
| # Usage: hack/run-e2e-init.sh | ||
| # Example 1: hack/run-e2e-init.sh (run init e2e with default config) | ||
| # Example 2: export HOST_CLUSTER_KUBECONFIG=<KUBECONFIG PATH> hack/run-e2e-init.sh (run init e2e with your KUBECONFIG) | ||
|
|
||
|
|
||
| KUBECONFIG_PATH=${KUBECONFIG_PATH:-"${HOME}/.kube"} | ||
| HOST_CLUSTER_KUBECONFIG=${HOST_CLUSTER_KUBECONFIG:-"$KUBECONFIG_PATH/karmada.config"} | ||
| DATA_DIR=${DATA_DIR:-"${HOME}/karmada-data"} | ||
| VERSION="latest" | ||
| REGISTRY="docker.io/karmada" | ||
|
|
||
| # KARMADA_RUNNING_ON_KIND indicates if current testing against on karmada that installed on a kind cluster. | ||
| # Defaults to true. | ||
| # For kind cluster, the kind related logs will be collected after the testing. | ||
| KARMADA_RUNNING_ON_KIND=${KARMADA_RUNNING_ON_KIND:-true} | ||
|
|
||
| KARMADA_HOST_CLUSTER_NAME=${KARMADA_HOST_CLUSTER_NAME:-"karmada-host"} | ||
|
|
||
| ARTIFACTS_PATH=${ARTIFACTS_PATH:-"${HOME}/karmada-e2e-init-logs"} | ||
| mkdir -p "$ARTIFACTS_PATH" | ||
|
|
||
| # Install ginkgo | ||
| GO111MODULE=on go install github.com/onsi/ginkgo/v2/ginkgo | ||
|
|
||
| # Run init e2e | ||
| export KUBECONFIG=${HOST_CLUSTER_KUBECONFIG} | ||
| export CRDs_PATH=${CRDs_PATH:-"${DATA_DIR}/crds.tar.gz"} | ||
| export KARMADA_AGGREGATED_APISERVER_IMAGE=${KARMADA_AGGREGATED_APISERVER_IMAGE:-"${REGISTRY}/karmada-aggregated-apiserver:${VERSION}"} | ||
| export KARMADA_CONTROLLER_MANAGER_IMAGE=${KARMADA_CONTROLLER_MANAGER_IMAGE:-"${REGISTRY}/karmada-controller-manager:${VERSION}"} | ||
| export KARMADA_SCHEDULER_IMAGE=${KARMADA_SCHEDULER_IMAGE:-"${REGISTRY}/karmada-scheduler:${VERSION}"} | ||
| export KARMADA_WEBHOOK_IMAGE=${KARMADA_WEBHOOK_IMAGE:-"${REGISTRY}/karmada-webhook:${VERSION}"} | ||
| set +e | ||
| ginkgo -v --race --trace --fail-fast -p --randomize-all ./test/e2e/suites/init -- --host-context=${KARMADA_HOST_CLUSTER_NAME} | ||
| TESTING_RESULT=$? | ||
|
|
||
| # Collect logs | ||
| echo "Collect logs to $ARTIFACTS_PATH..." | ||
| cp "$HOST_CLUSTER_KUBECONFIG" "$ARTIFACTS_PATH" | ||
|
|
||
| if [ "$KARMADA_RUNNING_ON_KIND" = true ]; then | ||
| echo "Collecting $KARMADA_HOST_CLUSTER_NAME logs..." | ||
| mkdir -p "$ARTIFACTS_PATH/$KARMADA_HOST_CLUSTER_NAME" | ||
| kind export logs --name="$KARMADA_HOST_CLUSTER_NAME" "$ARTIFACTS_PATH/$KARMADA_HOST_CLUSTER_NAME" | ||
| fi | ||
|
|
||
| echo "Collected logs at $ARTIFACTS_PATH:" | ||
| ls -al "$ARTIFACTS_PATH" | ||
|
|
||
| # If E2E test failed, exit directly with the test result | ||
| if [ $TESTING_RESULT -ne 0 ]; then | ||
| echo "Init E2E test failed with exit code $TESTING_RESULT, skipping component restart check." | ||
| exit $TESTING_RESULT | ||
| fi | ||
|
|
||
| # Check if Karmada components have restarted, if any has, it means that OOM or panic has occurred | ||
| # due to memory modification, and needs to be investigated. | ||
| echo "Init E2E run successfully." | ||
| echo "Checking if Karmada components have restarted after init..." | ||
|
|
||
| # Function to check pod restart count for a given component | ||
| check_component_restart() { | ||
| local component_label=$1 | ||
| local component_name=$2 | ||
|
|
||
| echo "Checking ${component_name} pods..." | ||
|
|
||
| # Get pod information in a single call, including both name and restart count | ||
| # Use a template that handles missing containerStatuses gracefully | ||
| local pod_info | ||
| pod_info=$(kubectl --context="${KARMADA_HOST_CLUSTER_NAME}" get pod -n karmada-system -l "${component_label}" \ | ||
| -o go-template='{{range .items}}{{.metadata.name}}:{{if .status.containerStatuses}}{{(index .status.containerStatuses 0).restartCount}}{{else}}0{{end}}{{"\n"}}{{end}}' 2>/dev/null) | ||
|
|
||
| if [ -z "$pod_info" ]; then | ||
| echo "No pods found for ${component_name}, skipping..." | ||
| return 0 | ||
| fi | ||
|
|
||
| # Process each pod's information | ||
| while IFS=: read -r pod_name restart_count; do | ||
| # Skip empty lines | ||
| [ -z "$pod_name" ] && continue | ||
|
|
||
| # Ensure restart_count is a number (default to 0 if empty or invalid) | ||
| if ! [[ "$restart_count" =~ ^[0-9]+$ ]]; then | ||
| echo "Warning: Unable to get restart count for pod $pod_name, assuming 0" | ||
| restart_count=0 | ||
| fi | ||
|
|
||
| if [ "$restart_count" -gt 0 ]; then | ||
| echo "ERROR: ${component_name} pod $pod_name has restarted $restart_count times." | ||
| echo "This indicates OOM or panic occurred and needs to be investigated." | ||
| return 1 # Return failure to stop checking | ||
| else | ||
| echo "${component_name} pod $pod_name: no restarts" | ||
| fi | ||
| done <<< "$pod_info" | ||
|
|
||
| return 0 | ||
| } | ||
|
|
||
| # List of basic components that should be running after karmadactl init | ||
| components=( | ||
| "app=etcd:etcd" | ||
| "app=karmada-apiserver:karmada-apiserver" | ||
| "app=karmada-aggregated-apiserver:karmada-aggregated-apiserver" | ||
| "app=kube-controller-manager:kube-controller-manager" | ||
| "app=karmada-controller-manager:karmada-controller-manager" | ||
| "app=karmada-scheduler:karmada-scheduler" | ||
| "app=karmada-webhook:karmada-webhook" | ||
| ) | ||
|
|
||
| # Check each component, stop at first failure | ||
| for component in "${components[@]}"; do | ||
| IFS=':' read -r label name <<< "$component" | ||
| if ! check_component_restart "$label" "$name"; then | ||
| echo "COMPONENT RESTART CHECK FAILED: Component $name has restarted, stopping further checks." | ||
| exit 1 | ||
| fi | ||
| done | ||
|
|
||
| echo "All component restart checks passed." | ||
| exit $TESTING_RESULT |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For robustness, it's better to use an absolute path constructed from
${REPO_ROOT}to avoid issues with the script's execution directory. This makes the script less dependent on the current working directory.