Skip to content

Commit d11bba6

Browse files
arybolovlevarybolovlev
authored
Fix Pod spec toleration when the spec is a template (#2380)
1 parent 299de2f commit d11bba6

11 files changed

+158
-21
lines changed

‎.changelog/2380.txt

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
```release-note:note
2+
We have updated the logic of resources that use the Pod specification template, such as `kubernetes_deployment_v1`, `kubernetes_stateful_set_v1`, etc, and now the provider will keep all tolerations(`spec.toleration`) returned by Kubernetes. The same is applicable for the data sources `kubernetes_pod_v1` and `kubernetes_pod`. The behavior of resources `kubernetes_pod_v1` and `kubernetes_pod` remains unchanged, i.e. the provider will keep removing tolerations with well-known [taints](https://kubernetes.io/docs/reference/labels-annotations-taints/) since they might be attached to the object by Kubernetes controller and could lead to a perpetual diff.
3+
```
4+
5+
```release-note:bug
6+
`resource/kubernetes_replication_controller`: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
7+
```
8+
9+
```release-note:bug
10+
`resource/kubernetes_replication_controller_v1`: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
11+
```
12+
13+
```release-note:bug
14+
`resource/kubernetes_stateful_set: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
15+
```
16+
17+
```release-note:bug
18+
`resource/kubernetes_stateful_set_v1: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
19+
```
20+
21+
```release-note:bug
22+
`resource/kubernetes_deployment: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
23+
```
24+
25+
```release-note:bug
26+
`resource/kubernetes_deployment_v1: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
27+
```
28+
29+
```release-note:bug
30+
`resource/kubernetes_daemonset: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
31+
```
32+
33+
```release-note:bug
34+
`resource/kubernetes_daemon_set_v1: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
35+
```
36+
37+
```release-note:bug
38+
`resource/kubernetes_cron_job: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
39+
```
40+
41+
```release-note:bug
42+
`resource/kubernetes_cron_job_v1: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
43+
```
44+
45+
```release-note:bug
46+
`resource/kubernetes_job: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
47+
```
48+
49+
```release-note:bug
50+
`resource/kubernetes_job_v1: fix an issue when the provider cuts out toleration under pod spec template(`*.template.spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/). That could lead to a perpetual diff behavior.
51+
```
52+
53+
```release-note:bug
54+
`data_source/kubernetes_pod`: fix an issue when the provider cuts out toleration under pod spec(`spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/).
55+
```
56+
57+
```release-note:bug
58+
`data_source/kubernetes_pod_v1`: fix an issue when the provider cuts out toleration under pod spec(`spec.toleration`) if it uses a well-known [taint](https://kubernetes.io/docs/reference/labels-annotations-taints/).
59+
```

‎kubernetes/data_source_kubernetes_pod_v1.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,8 @@ func dataSourceKubernetesPodV1Read(ctx context.Context, d *schema.ResourceData,
7070
return diag.FromErr(err)
7171
}
7272

73-
podSpec, err := flattenPodSpec(pod.Spec)
73+
// isTemplate argument here is equal to 'true' because we want to keep all attributes that Kubernetes unchanged.
74+
podSpec, err := flattenPodSpec(pod.Spec, true)
7475
if err != nil {
7576
return diag.FromErr(err)
7677
}

‎kubernetes/data_source_kubernetes_pod_v1_test.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ package kubernetes
55

66
import (
77
"fmt"
8+
"regexp"
89
"testing"
910

1011
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/acctest"
@@ -16,6 +17,7 @@ func TestAccKubernetesDataSourcePodV1_basic(t *testing.T) {
1617
dataSourceName := "data.kubernetes_pod_v1.test"
1718
name := fmt.Sprintf("tf-acc-test-%s", acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum))
1819
imageName := busyboxImage
20+
oneOrMore := regexp.MustCompile(`^[1-9][0-9]*$`)
1921

2022
resource.ParallelTest(t, resource.TestCase{
2123
PreCheck: func() { testAccPreCheck(t) },
@@ -34,6 +36,7 @@ func TestAccKubernetesDataSourcePodV1_basic(t *testing.T) {
3436
Check: resource.ComposeAggregateTestCheckFunc(
3537
resource.TestCheckResourceAttr(dataSourceName, "metadata.0.name", name),
3638
resource.TestCheckResourceAttr(dataSourceName, "spec.0.container.0.image", imageName),
39+
resource.TestMatchResourceAttr(dataSourceName, "spec.0.toleration.#", oneOrMore),
3740
),
3841
},
3942
},

‎kubernetes/resource_kubernetes_deployment_v1_test.go

Lines changed: 42 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,7 @@ func TestAccKubernetesDeploymentV1_with_tolerations(t *testing.T) {
316316
deploymentName := fmt.Sprintf("tf-acc-test-%s", acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum))
317317
resourceName := "kubernetes_deployment_v1.test"
318318
imageName := busyboxImage
319+
key := "myKey"
319320
tolerationSeconds := 6000
320321
operator := "Equal"
321322

@@ -325,11 +326,11 @@ func TestAccKubernetesDeploymentV1_with_tolerations(t *testing.T) {
325326
CheckDestroy: testAccCheckKubernetesDeploymentV1Destroy,
326327
Steps: []resource.TestStep{
327328
{
328-
Config: testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName, &tolerationSeconds, operator, nil),
329+
Config: testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName, key, operator, "", &tolerationSeconds),
329330
Check: resource.ComposeAggregateTestCheckFunc(
330331
testAccCheckKubernetesDeploymentV1Exists(resourceName, &conf),
331332
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.effect", "NoExecute"),
332-
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.key", "myKey"),
333+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.key", key),
333334
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.operator", operator),
334335
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.toleration_seconds", "6000"),
335336
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.value", ""),
@@ -345,6 +346,7 @@ func TestAccKubernetesDeploymentV1_with_tolerations_unset_toleration_seconds(t *
345346
deploymentName := fmt.Sprintf("tf-acc-test-%s", acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum))
346347
resourceName := "kubernetes_deployment_v1.test"
347348
imageName := busyboxImage
349+
key := "myKey"
348350
operator := "Equal"
349351
value := "value"
350352

@@ -354,11 +356,11 @@ func TestAccKubernetesDeploymentV1_with_tolerations_unset_toleration_seconds(t *
354356
CheckDestroy: testAccCheckKubernetesDeploymentV1Destroy,
355357
Steps: []resource.TestStep{
356358
{
357-
Config: testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName, nil, operator, &value),
359+
Config: testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName, key, operator, value, nil),
358360
Check: resource.ComposeAggregateTestCheckFunc(
359361
testAccCheckKubernetesDeploymentV1Exists(resourceName, &conf),
360362
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.effect", "NoExecute"),
361-
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.key", "myKey"),
363+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.key", key),
362364
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.operator", operator),
363365
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.value", "value"),
364366
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.toleration_seconds", ""),
@@ -368,6 +370,36 @@ func TestAccKubernetesDeploymentV1_with_tolerations_unset_toleration_seconds(t *
368370
})
369371
}
370372

373+
func TestAccKubernetesDeploymentV1_with_well_known_tolerations(t *testing.T) {
374+
var conf appsv1.Deployment
375+
376+
deploymentName := fmt.Sprintf("tf-acc-test-%s", acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum))
377+
resourceName := "kubernetes_deployment_v1.test"
378+
imageName := busyboxImage
379+
key := "node.kubernetes.io/unreachable"
380+
tolerationSeconds := 6000
381+
operator := "Exists"
382+
383+
resource.ParallelTest(t, resource.TestCase{
384+
PreCheck: func() { testAccPreCheck(t) },
385+
ProviderFactories: testAccProviderFactories,
386+
CheckDestroy: testAccCheckKubernetesDeploymentV1Destroy,
387+
Steps: []resource.TestStep{
388+
{
389+
Config: testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName, key, operator, "", &tolerationSeconds),
390+
Check: resource.ComposeAggregateTestCheckFunc(
391+
testAccCheckKubernetesDeploymentV1Exists(resourceName, &conf),
392+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.effect", "NoExecute"),
393+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.key", key),
394+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.operator", operator),
395+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.toleration_seconds", "6000"),
396+
resource.TestCheckResourceAttr(resourceName, "spec.0.template.0.spec.0.toleration.0.value", ""),
397+
),
398+
},
399+
},
400+
})
401+
}
402+
371403
func TestAccKubernetesDeploymentV1_with_container_liveness_probe_using_exec(t *testing.T) {
372404
var conf appsv1.Deployment
373405

@@ -1851,14 +1883,14 @@ func testAccKubernetesDeploymentV1ConfigWithSecurityContextSysctl(deploymentName
18511883
`, deploymentName, imageName)
18521884
}
18531885

1854-
func testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName string, tolerationSeconds *int, operator string, value *string) string {
1886+
func testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageName, key, operator, value string, tolerationSeconds *int) string {
18551887
tolerationDuration := ""
18561888
if tolerationSeconds != nil {
18571889
tolerationDuration = fmt.Sprintf("toleration_seconds = %d", *tolerationSeconds)
18581890
}
18591891
valueString := ""
1860-
if value != nil {
1861-
valueString = fmt.Sprintf("value = \"%s\"", *value)
1892+
if value != "" {
1893+
valueString = fmt.Sprintf("value = \"%s\"", value)
18621894
}
18631895

18641896
return fmt.Sprintf(`resource "kubernetes_deployment_v1" "test" {
@@ -1887,7 +1919,7 @@ func testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageNam
18871919
spec {
18881920
toleration {
18891921
effect = "NoExecute"
1890-
key = "myKey"
1922+
key = "%s"
18911923
operator = "%s"
18921924
%s
18931925
%s
@@ -1896,14 +1928,14 @@ func testAccKubernetesDeploymentV1ConfigWithTolerations(deploymentName, imageNam
18961928
container {
18971929
image = "%s"
18981930
name = "containername"
1899-
command = ["sleep", "300"]
1931+
command = ["sleep", "infinity"]
19001932
}
19011933
termination_grace_period_seconds = 1
19021934
}
19031935
}
19041936
}
19051937
}
1906-
`, deploymentName, operator, valueString, tolerationDuration, imageName)
1938+
`, deploymentName, key, operator, valueString, tolerationDuration, imageName)
19071939
}
19081940

19091941
func testAccKubernetesDeploymentV1ConfigWithLivenessProbeUsingExec(deploymentName, imageName string) string {

‎kubernetes/resource_kubernetes_pod_v1.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,7 @@ func resourceKubernetesPodV1Read(ctx context.Context, d *schema.ResourceData, me
201201
return diag.FromErr(err)
202202
}
203203

204-
podSpec, err := flattenPodSpec(pod.Spec)
204+
podSpec, err := flattenPodSpec(pod.Spec, false)
205205
if err != nil {
206206
return diag.FromErr(err)
207207
}

‎kubernetes/structures_daemonset.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ func flattenDaemonSetSpec(in appsv1.DaemonSetSpec, d *schema.ResourceData, meta
2424
att["selector"] = flattenLabelSelector(in.Selector)
2525
}
2626

27-
podSpec, err := flattenPodSpec(in.Template.Spec)
27+
podSpec, err := flattenPodSpec(in.Template.Spec, true)
2828
if err != nil {
2929
return nil, err
3030
}

‎kubernetes/structures_deployment.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ func flattenDeploymentSpec(in appsv1.DeploymentSpec, d *schema.ResourceData, met
3535

3636
att["strategy"] = flattenDeploymentStrategy(in.Strategy)
3737

38-
podSpec, err := flattenPodSpec(in.Template.Spec)
38+
podSpec, err := flattenPodSpec(in.Template.Spec, true)
3939
if err != nil {
4040
return nil, err
4141
}

‎kubernetes/structures_pod.go

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ func flattenOS(in v1.PodOS) []interface{} {
3838
return []interface{}{att}
3939
}
4040

41-
func flattenPodSpec(in v1.PodSpec) ([]interface{}, error) {
41+
func flattenPodSpec(in v1.PodSpec, isTemplate bool) ([]interface{}, error) {
4242
att := make(map[string]interface{})
4343
if in.ActiveDeadlineSeconds != nil {
4444
att["active_deadline_seconds"] = *in.ActiveDeadlineSeconds
@@ -141,7 +141,7 @@ func flattenPodSpec(in v1.PodSpec) ([]interface{}, error) {
141141
}
142142

143143
if len(in.Tolerations) > 0 {
144-
att["toleration"] = flattenTolerations(in.Tolerations)
144+
att["toleration"] = flattenTolerations(in.Tolerations, isTemplate)
145145
}
146146

147147
if len(in.TopologySpreadConstraints) > 0 {
@@ -293,11 +293,12 @@ func flattenSysctls(sysctls []v1.Sysctl) []interface{} {
293293
return att
294294
}
295295

296-
func flattenTolerations(tolerations []v1.Toleration) []interface{} {
296+
func flattenTolerations(tolerations []v1.Toleration, isTemplate bool) []interface{} {
297297
att := []interface{}{}
298298
for _, v := range tolerations {
299299
// The API Server may automatically add several Tolerations to pods, strip these to avoid TF diff.
300-
if _, ok := builtInTolerations[v.Key]; ok {
300+
// Keep all tolerations if we flatten the Pod spec template.
301+
if _, ok := builtInTolerations[v.Key]; ok && !isTemplate {
301302
log.Printf("[INFO] ignoring toleration with key: %s", v.Key)
302303
continue
303304
}

‎kubernetes/structures_pod_test.go

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import (
1616
func TestFlattenTolerations(t *testing.T) {
1717
cases := []struct {
1818
Input []corev1.Toleration
19+
isTemplate bool
1920
ExpectedOutput []interface{}
2021
}{
2122
{
@@ -25,6 +26,7 @@ func TestFlattenTolerations(t *testing.T) {
2526
Value: "true",
2627
},
2728
},
29+
false,
2830
[]interface{}{
2931
map[string]interface{}{
3032
"key": "node-role.kubernetes.io/spot-worker",
@@ -43,6 +45,7 @@ func TestFlattenTolerations(t *testing.T) {
4345
Value: "true",
4446
},
4547
},
48+
false,
4649
[]interface{}{
4750
map[string]interface{}{
4851
"key": "node-role.kubernetes.io/other-worker",
@@ -61,21 +64,59 @@ func TestFlattenTolerations(t *testing.T) {
6164
TolerationSeconds: ptr.To(int64(120)),
6265
},
6366
},
67+
false,
6468
[]interface{}{
6569
map[string]interface{}{
6670
"effect": "NoExecute",
6771
"toleration_seconds": "120",
6872
},
6973
},
7074
},
75+
{
76+
[]corev1.Toleration{
77+
{
78+
Effect: "NoExecute",
79+
Key: "node.kubernetes.io/unreachable",
80+
Operator: "Exists",
81+
TolerationSeconds: ptr.To(int64(120)),
82+
},
83+
},
84+
false,
85+
[]interface{}{},
86+
},
7187
{
7288
[]corev1.Toleration{},
89+
false,
7390
[]interface{}{},
7491
},
92+
{
93+
[]corev1.Toleration{},
94+
true,
95+
[]interface{}{},
96+
},
97+
{
98+
[]corev1.Toleration{
99+
{
100+
Effect: "NoExecute",
101+
Key: "node.kubernetes.io/unreachable",
102+
Operator: "Exists",
103+
TolerationSeconds: ptr.To(int64(120)),
104+
},
105+
},
106+
true,
107+
[]interface{}{
108+
map[string]interface{}{
109+
"effect": "NoExecute",
110+
"key": "node.kubernetes.io/unreachable",
111+
"operator": "Exists",
112+
"toleration_seconds": "120",
113+
},
114+
},
115+
},
75116
}
76117

77118
for _, tc := range cases {
78-
output := flattenTolerations(tc.Input)
119+
output := flattenTolerations(tc.Input, tc.isTemplate)
79120
if !reflect.DeepEqual(output, tc.ExpectedOutput) {
80121
t.Fatalf("Unexpected output from flattener.\nExpected: %#v\nGiven: %#v",
81122
tc.ExpectedOutput, output)

‎kubernetes/structures_replication_controller.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ func flattenReplicationControllerSpec(in corev1.ReplicationControllerSpec, d *sc
2222
}
2323

2424
if in.Template != nil {
25-
podSpec, err := flattenPodSpec(in.Template.Spec)
25+
podSpec, err := flattenPodSpec(in.Template.Spec, true)
2626
if err != nil {
2727
return nil, err
2828
}

‎kubernetes/structures_stateful_set.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ func flattenPodTemplateSpec(t corev1.PodTemplateSpec) ([]interface{}, error) {
193193
template := make(map[string]interface{})
194194

195195
template["metadata"] = flattenMetadataFields(t.ObjectMeta)
196-
spec, err := flattenPodSpec(t.Spec)
196+
spec, err := flattenPodSpec(t.Spec, true)
197197
if err != nil {
198198
return []interface{}{template}, err
199199
}

0 commit comments

Comments
 (0)