Skip to content

Commit f347b55

Browse files
corrierilucaAlan
andauthored
feat(runner): refactor runner + use tenv to install binaries (#319)
* refactor(runner): move network mirror creation into utils * refactor(runner): move getDiff to utils * refactor(runner): move clone to FetchRepositoryContent * feat(runner): use tenv library to install binaries * fix(runner): correct name for hermitcrab env vars * refactor(runner): rework exec and init functions * refactor(runner): split the Exec function * test(runner): add first E2E test * test(runner): add tf apply E2E test * test(runner): add tests for hermitcrab and resources * test(runner): refactor tests to not be parallel * test(runner): add cases for tg and bin versions * test(runner): add error cases * fix(runner): check error from defered os.Chdir * chore(deps): run go mod tidy after runner refactor * docs(runner): update docs for tenv and runner refacto * docs: fix info callouts * fix(runner): merge messed with datastore initialization --------- Co-authored-by: Alan <[email protected]>
1 parent 34db87c commit f347b55

File tree

29 files changed

+1446
-1417
lines changed

29 files changed

+1446
-1417
lines changed

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ Dockerfile.cross
1616

1717
.terraform/
1818

19+
# Artifact generated by tests
20+
test.out/*
21+
1922
# Test binary, build with `go test -c`
2023
*.test
2124

cmd/runner/start.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,5 +21,6 @@ func buildRunnerStartCmd(app *burrito.App) *cobra.Command {
2121

2222
cmd.Flags().StringVar(&app.Config.Runner.SSHKnownHostsConfigMapName, "ssh-known-hosts-cm-name", "burrito-ssh-known-hosts", "configmap name to get known hosts file from")
2323
cmd.Flags().StringVar(&app.Config.Runner.RunnerBinaryPath, "runner-binary-path", "/runner/bin", "binary path where the runner can expect to find terraform or terragrunt binaries")
24+
cmd.Flags().StringVar(&app.Config.Runner.RepositoryPath, "repository-path", "/runner/repository", "path where the runner fetches the Git repository to work on")
2425
return cmd
2526
}

docs/examples/terraform-layer.yaml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,3 @@ spec:
1111
repository:
1212
name: my-repository
1313
namespace: burrito-project
14-
terraform:
15-
version: 1.3.1

docs/getting-started.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ spec:
6262
```
6363

6464
!!! info
65-
You can also connect to a public repository by omitting `spec.repository.secretName` in your `TerraformRepository` definition.
65+
You can also connect to a public repository by omitting `spec.repository.secretName` in your `TerraformRepository` definition.
6666

6767
## 3. Synchronize a terraform layer
6868

docs/guides/iac-drift-detection.md

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -72,14 +72,14 @@ Here is the content of the `TerraformRepository` resource that you have created
7272
apiVersion: config.terraform.padok.cloud/v1alpha1
7373
kind: TerraformRepository
7474
metadata:
75-
name: my-repo
75+
name: my-repository
7676
namespace: burrito-project
7777
spec:
7878
repository:
79-
url: git@github.com:padok-team/burrito-examples.git
79+
url: https://github.com/padok-team/burrito-examples
8080
```
8181

82-
Create a `TerraformLayer` resource in the `burrito-system` namespace, referencing the `TerraformRepository` you just created. For now, the autoApply is set to false, so the layer will only plan the Terraform code and not apply it.
82+
Create a `TerraformLayer` resource in the `burrito-system` namespace, referencing the `TerraformRepository` you just created. For now, the `autoApply` is set to false, so the layer will only plan the Terraform code and not apply it.
8383

8484
```bash
8585
kubectl apply -f https://raw.githubusercontent.com/padok-team/burrito/main/docs/examples/terraform-layer.yaml
@@ -99,8 +99,6 @@ spec:
9999
repository:
100100
name: my-repository
101101
namespace: burrito-project
102-
terraform:
103-
version: 1.3.1
104102
```
105103

106104
Check that your Terraform code is being planned by Burrito:
@@ -135,7 +133,7 @@ kubectl port-forward svc/burrito-server -n burrito-system 8080:80
135133

136134
![Burrito drift example](../assets/demo/drift-example.png)
137135

138-
Activate the autoApply feature by updating the `TerraformLayer` resource:
136+
Activate the `autoApply` feature by updating the `TerraformLayer` resource:
139137

140138
```bash
141139
kubectl patch tfl my-layer -n burrito-project --type merge --patch '{"spec":{"remediationStrategy":{"autoApply":true}}}'

docs/guides/pr-mr-workflow.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ Configure a webhook in your GitHub repository to point to the exposed `burrito-s
4444
kind: Secret
4545
metadata:
4646
name: burrito-webhook-secret
47-
namespace: burrito
47+
namespace: burrito-system
4848
type: Opaque
4949
stringData:
5050
burrito-webhook-secret: <my-webhook-secret>
@@ -86,7 +86,7 @@ my-layer-pr-1-rxvrt ApplyNeeded my-repository update-tf-code terraform
8686

8787
The Burrito server should have commented on the PR/MR with the plan output. You can check the comments on the PR/MR to see the plan output.
8888

89-
![Pul request comment demo](../assets/demo/pr-demo.png)
89+
![Pull request comment demo](../assets/demo/pr-demo.png)
9090

9191
Set the `autoApply` field to `true` in the TerraformLayer resource to automatically apply the Terraform code upon merging the PR/MR.
9292

docs/operator-manual/architecture.md

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,8 @@ The `TerraformRun` controller also creates and deletes the [Kubernetes leases](h
8686

8787
### The runners
8888

89-
The runner image implementation heavily relies on Golang libraries provided by Hashicorp such as [`tfexec`](https://github.com/hashicorp/terraform-exec) and [`hc-install`](https://github.com/hashicorp/hc-install) which allows us to dynamically download and use any version of the Terraform binary.
90-
Thus, we support any existing version of Terraform.
89+
The runner implementation relies on [`tenv`](https://github.com/tofuutils/tenv), a tool from the community which allows us to dynamically download and use any version of Terraform, Terragrunt or OpenTofu (coming soon). Thus, we support any existing version of Terraform.
9190

92-
The runners also support any existing version of [Terragrunt](https://terragrunt.gruntwork.io/).
91+
If no version constraint is set in the TerraformLayer resource or in the TerraformRepository resource, `tenv` will detect which version of Terraform/Terragrunt/OpenTofu to use by looking at the version constraints in your code.
9392

94-
The runner is responsible to update the annotations of the layer it is associated with to store information about what commit was plan/apply and when.
93+
The runner is responsible to update the annotations of the layer it is associated to to store information about what commit was planned/applied and when.

docs/operator-manual/pr-mr-workflow.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,14 @@
33
<p align="center"><img src="../../assets/design/pr-mr-workflow.excalidraw.png" width="1000px" /></p>
44

55
!!! info
6-
In this documentation all references to pull requests can be change to merge requests for GitLab. However, the resulting Kubernetes object will still be named `TerraformPullRequest`.
6+
In this documentation all references to pull requests can be change to merge requests for GitLab. However, the resulting Kubernetes object will still be named `TerraformPullRequest`.
77

88
## Components
99

1010
### The server
1111

1212
!!! info
13-
For more information about the server, see the [architectural overview](./architecture.md) documentation.
13+
For more information about the server, see the [architectural overview](./architecture.md) documentation.
1414

1515
Upon receiving a Pull Request creation event, the server creates a `TerraformPullRequest` resource.
1616

@@ -35,7 +35,7 @@ The status of a `TerraformPulLRequest` is defined using the [conditions standard
3535
- `IsCommentUpToDate`. This condition is used to check if the controller needs to send a comment to a pull request. This is checked by comparing the last discovered commit and the last commit for which a comment was already sent.
3636

3737
!!! info
38-
We use annotations to store information.
38+
We use annotations to store information.
3939

4040
With those 3 conditions, we defined 3 states:
4141

docs/user-guide/terraform-version.md

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,20 @@
1-
# Choose a terraform/terragrunt version
1+
# Configure a Terraform/Terragrunt/OpenTofu version
22

3-
For now, Burrito doesn't auto-detect the terraform or terragrunt version used in your repository. You have to specify it in the `TerraformRepository` or `TerraformLayer` resource.
3+
By leveraging [`tenv`](https://github.com/tofuutils/tenv), Burrito auto-detects the Terraform, Terragrunt or OpenTofu version used in your repository, with version constraints set in your code (see [`tenv`'s README](https://github.com/tofuutils/tenv/blob/main/README.md)).
44

5-
## Choose terraform version
5+
Additionally, you can to specify version constraints in the `TerraformRepository` or `TerraformLayer` resource as described below.
66

7-
Both `TerraformRepository` and `TerraformLayer` expose a `spec.terrafrom.version` map field.
7+
## Choose Terraform version
8+
9+
Both `TerraformRepository` and `TerraformLayer` expose a `spec.terraform.version` map field that support version constraints as described in the [Terraform documentation](https://www.terraform.io/docs/language/expressions/version-constraints.html).
810

911
If the field is specified for a given `TerraformRepository` it will be applied by default to all `TerraformLayer` linked to it.
1012

1113
If the field is specified for a given `TerraformLayer` it will take precedence over the `TerraformRepository` configuration.
1214

1315
## Enable Terragrunt
1416

15-
You can specify usage of terragrunt as follow:
17+
You can specify usage of Terragrunt with the `spec.terraform.terragrunt` map as follow:
1618

1719
```yaml
1820
apiVersion: config.terraform.padok.cloud/v1alpha1
@@ -21,7 +23,7 @@ metadata:
2123
name: random-pets-terragrunt
2224
spec:
2325
terraform:
24-
version: "1.3.1"
26+
version: "~> 1.3.0"
2527
terragrunt:
2628
enabled: true
2729
version: "0.44.5"
@@ -36,3 +38,7 @@ spec:
3638
3739
!!! info
3840
This configuration can be specified at the `TerraformRepository` level to be enabled by default in each of its layers.
41+
42+
## Use OpenTofu instead of Terraform
43+
44+
(coming soon)

go.mod

Lines changed: 24 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,13 @@ require (
1313
github.com/ghodss/yaml v1.0.0
1414
github.com/gruntwork-io/go-commons v0.17.1
1515
github.com/hashicorp/go-multierror v1.1.1
16+
github.com/hashicorp/hcl/v2 v2.21.0
1617
github.com/hashicorp/terraform-json v0.17.1
1718
github.com/onsi/ginkgo/v2 v2.17.3
1819
github.com/onsi/gomega v1.33.1
1920
github.com/sirupsen/logrus v1.9.3
2021
github.com/stretchr/testify v1.9.0
22+
github.com/tofuutils/tenv/v3 v3.1.0
2123
google.golang.org/api v0.178.0
2224
k8s.io/apimachinery v0.30.2
2325
k8s.io/client-go v0.30.2
@@ -34,8 +36,14 @@ require (
3436
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1 // indirect
3537
github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0 // indirect
3638
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
39+
github.com/BurntSushi/toml v1.4.0 // indirect
3740
github.com/Microsoft/go-winio v0.6.1 // indirect
3841
github.com/ProtonMail/go-crypto v1.1.0-alpha.2 // indirect
42+
github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f // indirect
43+
github.com/ProtonMail/gopenpgp/v2 v2.7.5 // indirect
44+
github.com/PuerkitoBio/goquery v1.9.2 // indirect
45+
github.com/agext/levenshtein v1.2.3 // indirect
46+
github.com/andybalholm/cascadia v1.3.2 // indirect
3947
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
4048
github.com/aws/aws-sdk-go-v2 v1.29.0 // indirect
4149
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
@@ -53,22 +61,21 @@ require (
5361
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.25.0 // indirect
5462
github.com/aws/aws-sdk-go-v2/service/sts v1.29.0 // indirect
5563
github.com/aws/smithy-go v1.20.2 // indirect
56-
github.com/cloudflare/circl v1.3.7 // indirect
57-
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
64+
github.com/cloudflare/circl v1.3.9 // indirect
65+
github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
5866
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
5967
github.com/emirpasic/gods v1.18.1 // indirect
68+
github.com/fatih/color v1.17.0 // indirect
6069
github.com/felixge/httpsnoop v1.0.4 // indirect
6170
github.com/go-errors/errors v1.4.2 // indirect
6271
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
6372
github.com/go-git/go-billy/v5 v5.5.0 // indirect
6473
github.com/go-logr/stdr v1.2.2 // indirect
65-
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect
6674
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
6775
github.com/golang-jwt/jwt v3.2.2+incompatible // indirect
6876
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
6977
github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
7078
github.com/google/gnostic-models v0.6.8 // indirect
71-
github.com/google/go-github/v56 v56.0.0 // indirect
7279
github.com/google/go-github/v62 v62.0.0 // indirect
7380
github.com/google/go-querystring v1.1.0 // indirect
7481
github.com/google/pprof v0.0.0-20240424215950-a892ee059fd6 // indirect
@@ -77,13 +84,15 @@ require (
7784
github.com/googleapis/gax-go/v2 v2.12.4 // indirect
7885
github.com/hashicorp/errwrap v1.0.0 // indirect
7986
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
87+
github.com/hashicorp/go-hclog v1.6.3 // indirect
8088
github.com/hashicorp/go-retryablehttp v0.7.4 // indirect
8189
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
8290
github.com/kevinburke/ssh_config v1.2.0 // indirect
8391
github.com/kylelemons/godebug v1.1.0 // indirect
8492
github.com/labstack/gommon v0.4.2 // indirect
8593
github.com/mattn/go-colorable v0.1.13 // indirect
8694
github.com/mattn/go-isatty v0.0.20 // indirect
95+
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
8796
github.com/pjbgf/sha1cd v0.3.0 // indirect
8897
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
8998
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
@@ -93,23 +102,22 @@ require (
93102
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
94103
github.com/skeema/knownhosts v1.2.2 // indirect
95104
github.com/sourcegraph/conc v0.3.0 // indirect
96-
github.com/urfave/cli v1.22.4 // indirect
97105
github.com/urfave/cli/v2 v2.10.3 // indirect
98106
github.com/valyala/bytebufferpool v1.0.0 // indirect
99107
github.com/valyala/fasttemplate v1.2.2 // indirect
100108
github.com/xanzy/ssh-agent v0.3.3 // indirect
101109
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
102-
github.com/zclconf/go-cty v1.14.0 // indirect
110+
github.com/zclconf/go-cty v1.15.0 // indirect
103111
go.opencensus.io v0.24.0 // indirect
104112
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
105113
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
106114
go.opentelemetry.io/otel v1.24.0 // indirect
107115
go.opentelemetry.io/otel/metric v1.24.0 // indirect
108116
go.opentelemetry.io/otel/trace v1.24.0 // indirect
109117
golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
110-
golang.org/x/mod v0.17.0 // indirect
111-
golang.org/x/sync v0.7.0 // indirect
112-
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
118+
golang.org/x/mod v0.20.0 // indirect
119+
golang.org/x/sync v0.8.0 // indirect
120+
golang.org/x/tools v0.24.0 // indirect
113121
google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda // indirect
114122
google.golang.org/genproto/googleapis/api v0.0.0-20240506185236-b8a5c65736ae // indirect
115123
google.golang.org/genproto/googleapis/rpc v0.0.0-20240429193739-8cf5692501f6 // indirect
@@ -141,8 +149,7 @@ require (
141149
github.com/google/go-github/v50 v50.2.0
142150
github.com/google/gofuzz v1.2.0 // indirect
143151
github.com/google/uuid v1.6.0 // indirect
144-
github.com/hashicorp/go-version v1.7.0
145-
github.com/hashicorp/hc-install v0.6.4
152+
github.com/hashicorp/go-version v1.7.0 // indirect
146153
github.com/hashicorp/hcl v1.0.0 // indirect
147154
github.com/hashicorp/terraform-exec v0.19.0
148155
github.com/imdario/mergo v0.3.15 // indirect
@@ -166,19 +173,19 @@ require (
166173
github.com/prometheus/procfs v0.12.0 // indirect
167174
github.com/spf13/afero v1.11.0 // indirect
168175
github.com/spf13/cast v1.6.0 // indirect
169-
github.com/spf13/cobra v1.7.0
176+
github.com/spf13/cobra v1.8.1
170177
github.com/spf13/pflag v1.0.5
171178
github.com/spf13/viper v1.18.2
172179
github.com/subosito/gotenv v1.6.0 // indirect
173180
github.com/xanzy/go-gitlab v0.93.2
174181
go.uber.org/multierr v1.11.0 // indirect
175182
go.uber.org/zap v1.26.0 // indirect
176-
golang.org/x/crypto v0.24.0 // indirect
177-
golang.org/x/net v0.26.0 // indirect
183+
golang.org/x/crypto v0.26.0 // indirect
184+
golang.org/x/net v0.28.0 // indirect
178185
golang.org/x/oauth2 v0.20.0
179-
golang.org/x/sys v0.21.0 // indirect
180-
golang.org/x/term v0.21.0 // indirect
181-
golang.org/x/text v0.16.0 // indirect
186+
golang.org/x/sys v0.24.0 // indirect
187+
golang.org/x/term v0.23.0 // indirect
188+
golang.org/x/text v0.17.0 // indirect
182189
golang.org/x/time v0.5.0 // indirect
183190
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
184191
google.golang.org/protobuf v1.34.1 // indirect

0 commit comments

Comments
 (0)