Issues/84 - getting started with e2e testing - can run promotion pull request scenarios (#126)

* e2e for promotions pull request

* bump version

* review suggestions

Author: enekofb

CONTRIBUTING.md

@@ -17,6 +17,9 @@ by Go to download dependencies from private repositories (replace `USERNAME` wit
 
 ## Running Tests
 
+The project has unit, integration and end-to-end testing. For all of them we are using `go test` for all test types
+(including e2e) due to its simplicity.
+
 ### Unit Testing
 
 This project has unit tests
@@ -25,7 +28,8 @@ This project has unit tests
 make test
 ```
 
-Testing on this controller follows closely what is done by default in `kubebuilder` although we ditched the use of `Gomega` in favor of native Go testing structure. As an example on how to write tests for this controller you can take a look at Kubebuilder [docs](https://book.kubebuilder.io/cronjob-tutorial/writing-tests.html#writing-controller-tests) for reference.
+Testing on this controller follows closely what is done by default in `kubebuilder` although we ditched the use of `Gomega` in favor of native Go testing structure. 
+As an example on how to write tests for this controller you can take a look at Kubebuilder [docs](https://book.kubebuilder.io/cronjob-tutorial/writing-tests.html#writing-controller-tests) for reference.
 
 ### Integration Testing
 
@@ -53,8 +57,66 @@ where you would need to add the build tag `integration`
 
 ```go
 //go:build integration
+```
 
+### End-to-End testing 
+
+This project has a test suite for end to end pipeline controller journeys. You could find them
+within [e2e folder](./e2e). 
+
+#### Configuration
+
+A set of [test pipelines](/e2e/testdata/pipelines) exists to run the tests. They depend on a set 
+of common resources.
+
+A set of git repos are required to hold the manifests to use during pull request promotion strategy.
+
+- [github repo](https://github.com/weaveworks/pipeline-controller-e2e)
+- [gitlab repo](https://gitlab.git.dev.weave.works/pipeline-controller/pipeline-controller-e2e)
+
+A [set of credentials](https://docs.gitops.weave.works/docs/pipelines/promoting-applications/#create-credentials-secret) 
+per git provider needs to exist.
+
+```yaml
+apiVersion: v1
+data:
+  password: <my-github-pat>
+  token: <my-github-pat>
+  username: <my-github-username>
+kind: Secret
+metadata:
+  name: github-promotion-credentials
+  namespace: flux-system
+type: Opaque
+---
+apiVersion: v1
+data:
+  password: <my-gitlab-pat>
+  token: <my-gitlab-pat>
+  username: <my-gitlab-username>
+kind: Secret
+metadata:
+  name: gitlab-promotion-credentials
+  namespace: flux-system
+type: Opaque
 ```
+#### Running the tests
+
+Execute `make e2e` to exercise the test.  
+
+In order to interact with them the following targets exists in [Makefile](./Makefile):
+
+- `e2e-setup`: provisions a local environment with kind, flux and pipeline controller to run the tests.
+- `e2e-test`: runs the tests against the running environment.
+- `e2e-clean`: tears down the created environment.
+- `e2e`: single target that executes setup + test + clean targets
+
+One of the concerns with end-to-end testing is that tend to be unstable. A simple script to test how stable
+is the test suite could be used [stability script](/hack/e2e-suite-stability.sh).
+
+#### Adding a test
+
+Add your test case within the e2e folder. An example is [promotion_pull_request_test.go](e2e/promotion_pull_request_test.go)
 
 ## Working with promotions
 

Makefile

@@ -92,7 +92,7 @@ test: manifests generate fmt vet envtest ## Run tests.
 
 .PHONY: integration-test
 integration-test:
-	go test -tags integration -v ./...
+	go test -tags integration -cover -v ./...
 
 .PHONY: lint-and-install-chart
 lint-and-install-chart:
@@ -207,3 +207,26 @@ $(GOLANGCI_LINT): $(LOCALBIN)
 helm: $(HELM)
 $(HELM): $(LOCALBIN)
 	test -s $(LOCALBIN)/helm || { curl -s https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | HELM_INSTALL_DIR=$(LOCALBIN) bash -s -- --no-sudo --version $(HELM_VERSION); }
+
+.PHONY: e2e
+e2e: e2e-setup e2e-test e2e-clean
+
+.PHONY: e2e-setup
+e2e-setup: docker-build kind flux deploy
+	kubectl wait -n pipeline-system deployment/pipeline-controller --for=condition=available
+	kubectl apply -f e2e/testdata --recursive
+
+kind:
+	kind create cluster --name=pipeline-controller || kubectx kind-pipeline-controller
+	kind load docker-image --name=pipeline-controller $(IMG)
+
+flux:
+	flux install
+
+.PHONY: e2e-test
+e2e-test:
+	go test -tags e2e -count=1 -v ./e2e
+
+.PHONY: e2e-clean
+e2e-clean:
+	kind delete cluster --name=pipeline-controller

api/v1alpha1/pipeline_types.go

@@ -61,7 +61,7 @@ type PipelineSpec struct {
 // Promotion defines all the available promotion strategies. All of the fields in here are mutually exclusive, i.e. you can only select one
 // promotion strategy per Pipeline. Failure to do so will result in undefined behaviour.
 type Promotion struct {
-	// PullRequest defines a promotion through a GitHub Pull Request.
+	// PullRequest defines a promotion through a pull request
 	// +optional
 	PullRequest *PullRequestPromotion `json:"pull-request,omitempty"`
 	// Notification defines a promotion where an event is emitted through Flux's notification-controller each time an app is to be promoted.
@@ -98,7 +98,7 @@ type PullRequestPromotion struct {
 	// +optional
 	Branch string `json:"branch"`
 	// SecretRef specifies the Secret containing authentication credentials for
-	// the git repository and for the GitHub API.
+	// the git repository and for the git provider API.
 	// For HTTPS repositories the Secret must contain 'username' and 'password'
 	// fields.
 	// For SSH repositories the Secret must contain 'identity'

charts/pipeline-controller/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: pipeline-controller
 description: Pipeline-controller Helm chart for Weave GitOps Enterprise
 type: application
-version: 0.9.0
+version: 0.10.0
 appVersion: "v0.0.26"

charts/pipeline-controller/crds/pipelines.weave.works_pipelines.yaml

@@ -132,8 +132,7 @@ spec:
                       app is to be promoted.
                     type: object
                   pull-request:
-                    description: PullRequest defines a promotion through a GitHub
-                      Pull Request.
+                    description: PullRequest defines a promotion through a pull request
                     properties:
                       branch:
                         description: 'The branch to checkout after cloning. Note:
@@ -144,8 +143,8 @@ spec:
                         type: string
                       secretRef:
                         description: SecretRef specifies the Secret containing authentication
-                          credentials for the git repository and for the GitHub API.
-                          For HTTPS repositories the Secret must contain 'username'
+                          credentials for the git repository and for the git provider
+                          API. For HTTPS repositories the Secret must contain 'username'
                           and 'password' fields. For SSH repositories the Secret must
                           contain 'identity' and 'known_hosts' fields. For Git Provider
                           API to manage pull requests, it must contain a 'token' field.

config/crd/bases/pipelines.weave.works_pipelines.yaml

@@ -132,8 +132,7 @@ spec:
                       app is to be promoted.
                     type: object
                   pull-request:
-                    description: PullRequest defines a promotion through a GitHub
-                      Pull Request.
+                    description: PullRequest defines a promotion through a pull request
                     properties:
                       branch:
                         description: 'The branch to checkout after cloning. Note:
@@ -144,8 +143,8 @@ spec:
                         type: string
                       secretRef:
                         description: SecretRef specifies the Secret containing authentication
-                          credentials for the git repository and for the GitHub API.
-                          For HTTPS repositories the Secret must contain 'username'
+                          credentials for the git repository and for the git provider
+                          API. For HTTPS repositories the Secret must contain 'username'
                           and 'password' fields. For SSH repositories the Secret must
                           contain 'identity' and 'known_hosts' fields. For Git Provider
                           API to manage pull requests, it must contain a 'token' field.