Terraform destroy with dbaas-postgresql returns internal server error

[superflo22]

Description

Running terraform destroy and planning to delete all resources does not succeed when a manged postgres is used.

Expected behavior

The pipeline should not terminate and destroy all resources

Environment

Terraform version:

registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:d5b621ec092dd6ec67d2192b914af33e15db1fe5b7c9d543c1151008b4132e0c

Provider version:

ionos-cloud/ionoscloud v6.5.7

OS:

gitlab-runner 16.11.1

Configuration Files

How to Reproduce

Steps to reproduce the behavior:

1. Create repo with the tf resources where a postgres cluster is configured
2. Run the gitlab template for terraform destroy
3. ...

Error and Debug Output

Using docker image sha256:abb83ff04190f9822ffd23e776138e00e954643fbef3d1073a0c765109019c2c for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:d5b621ec092dd6ec67d2192b914af33e15db1fe5b7c9d543c1151008b4132e0c ...
$ gitlab-terraform destroy
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
Initializing provider plugins...
- Finding hashicorp/kubernetes versions matching "2.32.0"...
- Finding latest version of hashicorp/random...
- Finding ionos-cloud/ionoscloud versions matching ">= 6.4.0, 6.5.7"...
- Installing hashicorp/kubernetes v2.32.0...
- Installed hashicorp/kubernetes v2.32.0 (signed by HashiCorp)
- Installing hashicorp/random v3.6.3...
- Installed hashicorp/random v3.6.3 (signed by HashiCorp)
- Installing ionos-cloud/ionoscloud v6.5.7...
- Installed ionos-cloud/ionoscloud v6.5.7 (signed by a HashiCorp partner, key ID 2D2E9201D5B7747D)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
random_password.keyclaok_password: Refreshing state... [id=none]
random_password.pg_initial_user_password: Refreshing state... [id=none]
ionoscloud_k8s_cluster.k8s_cluster: Refreshing state... [id=584e6a8d-4839-4b7e-aca4-fa7e5c5d9b2b]
ionoscloud_datacenter.DC: Refreshing state... [id=e69c0069-8261-431d-81d1-7b14b3254042]
ionoscloud_lan.dc_lan: Refreshing state... [id=1]
data.ionoscloud_k8s_cluster.k8s_cluster_data: Reading...
ionoscloud_k8s_node_pool.nodepool: Refreshing state... [id=be04883d-9515-4eae-8e15-e1234a2b621e]
module.ip_postgres.data.ionoscloud_k8s_node_pool_nodes.this: Reading...
module.ip_postgres.data.ionoscloud_k8s_node_pool_nodes.this: Read complete after 0s [id=be04883d-9515-4eae-8e15-e1234a2b621e]
module.ip_postgres.data.ionoscloud_server.first_node: Reading...
module.ip_postgres.data.ionoscloud_server.first_node: Read complete after 1s [id=dc1518a6-b1c0-41ef-bf93-dad4d579227f]
ionoscloud_pg_cluster.pg_cluster: Refreshing state... [id=6303eb83-0556-421c-b9cb-0d89ed7cf42a]
ionoscloud_pg_user.keyclaok_user: Refreshing state... [id=8fd5d228-3613-5cb8-804c-0a4a4b99edae]
ionoscloud_pg_database.keyclaok_pg_database: Refreshing state... [id=e2674557-0458-547c-89ca-5a85935f29ad]
data.ionoscloud_k8s_cluster.k8s_cluster_data: Read complete after 1s [id=584e6a8d-4839-4b7e-aca4-fa7e5c5d9b2b]
kubernetes_secret.argocd_cluster: Refreshing state... [id=argocd/argocd-cluster-test]
kubernetes_namespace.terraform: Refreshing state... [id=terraform]
kubernetes_secret.postgres_credentials: Refreshing state... [id=terraform/postgres-credentials-secret]
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy
Terraform will perform the following actions:
  # ionoscloud_datacenter.DC will be destroyed
  - resource "ionoscloud_datacenter" "DC" {
      - cpu_architecture    = [
          - {
              - cpu_family = "INTEL_SKYLAKE"
              - max_cores  = 8
              - max_ram    = 20480
              - vendor     = "GenuineIntel"
            },
          - {
              - cpu_family = "AMD_EPYC"
              - max_cores  = 8
              - max_ram    = 20480
              - vendor     = "AuthenticAMD"
            },
          - {
              - cpu_family = "INTEL_ICELAKE"
              - max_cores  = 8
              - max_ram    = 20480
              - vendor     = "GenuineIntel"
            },
        ] -> null
      - description         = "VDC managed by Terraform - do not edit manually" -> null
      - features            = [
          - "acronis-api-v2",
          - "allow-update-expose-serial",
          - "amd-epyc",
          - "backup-service-with-feign",
          - "cloud-init",
          - "cloud-init-private-image",
          - "contract-identities",
          - "core-vps",
          - "cpu-hot-plug",
          - "disk-vio-hot-plug",
          - "disk-vio-hot-unplug",
          - "enable-cache-volume",
          - "flow-logs",
          - "intel-icelake",
          - "k8s",
          - "mem-hot-plug",
          - "monitoring",
          - "nic-hot-plug",
          - "nic-hot-unplug",
          - "os-pool-optimised",
          - "pcc",
          - "pjd-include-vnics-section",
          - "private-k8s-cluster",
          - "pservers-dont-support-mix-of-os-types",
          - "ssd",
          - "ssd-storage-zoning",
          - "token-management-mfa-claim-required",
          - "use-backup-service",
          - "use-current-logged-identity-for-s3",
          - "use-platform-s3-for-flow-log",
          - "use-s3-service",
          - "v-cpu-instance",
          - "vm-autoscaling",
          - "vnf-alb",
          - "vnf-lb",
          - "vnf-nat",
        ] -> null
      - id                  = "e69c0069-8261-431d-81d1-7b14b3254042" -> null
      - ipv6_cidr_block     = "2a01:239:240:e600::/56" -> null
      - location            = "de/txl" -> null
      - name                = "TEST" -> null
      - sec_auth_protection = true -> null
      - version             = 51 -> null
    }
  # ionoscloud_k8s_cluster.k8s_cluster will be destroyed
  - resource "ionoscloud_k8s_cluster" "k8s_cluster" {
      - allow_replace             = false -> null
      - id                        = "584e6a8d-4839-4b7e-aca4-fa7e5c5d9b2b" -> null
      - k8s_version               = "1.30.2" -> null
      - name                      = "TEST" -> null
      - public                    = true -> null
      - viable_node_pool_versions = [
          - "1.30.5",
          - "1.30.4",
          - "1.30.3",
          - "1.30.2",
          - "1.29.9",
          - "1.29.8",
          - "1.29.7",
          - "1.29.6",
          - "1.29.5",
          - "1.29.4",
        ] -> null
      - maintenance_window {
          - day_of_the_week = "Friday" -> null
          - time            = "23:40:58Z" -> null
        }
    }
  # ionoscloud_k8s_node_pool.nodepool will be destroyed
  - resource "ionoscloud_k8s_node_pool" "nodepool" {
      - allow_replace     = false -> null
      - annotations       = {} -> null
      - availability_zone = "AUTO" -> null
      - cores_count       = 3 -> null
      - cpu_family        = "INTEL_SKYLAKE" -> null
      - datacenter_id     = "e69c0069-8261-431d-81d1-7b14b3254042" -> null
      - id                = "be04883d-9515-4eae-8e15-e1234a2b621e" -> null
      - k8s_cluster_id    = "584e6a8d-4839-4b7e-aca4-fa7e5c5d9b2b" -> null
      - k8s_version       = "1.30.2" -> null
      - labels            = {} -> null
      - name              = "test-cluster-nodepool-02" -> null
      - node_count        = 1 -> null
      - ram_size          = 18432 -> null
      - storage_size      = 20 -> null
      - storage_type      = "HDD" -> null
      - lans {
          - dhcp = true -> null
          - id   = 1 -> null
          - routes {
              - gateway_ip = "192.168.1.100" -> null
              - network    = "192.168.1.100/24" -> null
            }
        }
      - maintenance_window {
          - day_of_the_week = "Sunday" -> null
          - time            = "16:59:19Z" -> null
        }
    }
  # ionoscloud_lan.dc_lan will be destroyed
  - resource "ionoscloud_lan" "dc_lan" {
      - datacenter_id = "e69c0069-8261-431d-81d1-7b14b3254042" -> null
      - id            = "1" -> null
      - name          = "Lan" -> null
      - public        = false -> null
    }
  # ionoscloud_pg_cluster.pg_cluster will be destroyed
  - resource "ionoscloud_pg_cluster" "pg_cluster" {
      - cores                = 1 -> null
      - display_name         = "test-postgres-cluster" -> null
      - dns_name             = "pg-l1t1ggnsuu49n03p.postgresql.de-txl.ionos.com" -> null
      - id                   = "6303eb83-0556-421c-b9cb-0d89ed7cf42a" -> null
      - instances            = 1 -> null
      - location             = "de/txl" -> null
      - postgres_version     = "15" -> null
      - ram                  = 2048 -> null
      - storage_size         = 2048 -> null
      - storage_type         = "HDD" -> null
      - synchronization_mode = "ASYNCHRONOUS" -> null
      - connection_pooler {
          - enabled   = false -> null
          - pool_mode = "transaction" -> null
        }
      - connections {
          - cidr          = "10.7.222.3/24" -> null
          - datacenter_id = "e69c0069-8261-431d-81d1-7b14b3254042" -> null
          - lan_id        = "1" -> null
        }
      - credentials {
          - password = (sensitive value) -> null
          - username = "test-postgres-user" -> null
        }
      - maintenance_window {
          - day_of_the_week = "Sunday" -> null
          - time            = "09:00:00" -> null
        }
    }
  # ionoscloud_pg_database.keyclaok_pg_database will be destroyed
  - resource "ionoscloud_pg_database" "keyclaok_pg_database" {
      - cluster_id = "6303eb83-0556-421c-b9cb-0d89ed7cf42a" -> null
      - id         = "e2674557-0458-547c-89ca-5a85935f29ad" -> null
      - name       = "keycloak" -> null
      - owner      = "testkeycloak" -> null
    }
  # ionoscloud_pg_user.keyclaok_user will be destroyed
  - resource "ionoscloud_pg_user" "keyclaok_user" {
      - cluster_id     = "6303eb83-0556-421c-b9cb-0d89ed7cf42a" -> null
      - id             = "8fd5d228-3613-5cb8-804c-0a4a4b99edae" -> null
      - is_system_user = false -> null
      - password       = (sensitive value) -> null
      - username       = "testkeycloak" -> null
    }
  # kubernetes_namespace.terraform will be destroyed
  - resource "kubernetes_namespace" "terraform" {
      - id                               = "terraform" -> null
      - wait_for_default_service_account = false -> null
      - metadata {
          - annotations      = {} -> null
          - generation       = 0 -> null
          - labels           = {} -> null
          - name             = "terraform" -> null
          - resource_version = "34519586551" -> null
          - uid              = "f7332a72-f678-46f1-80a9-e29d6f9673d0" -> null
        }
    }
  # kubernetes_secret.argocd_cluster will be destroyed
  - resource "kubernetes_secret" "argocd_cluster" {
      - data                           = (sensitive value) -> null
      - id                             = "argocd/argocd-cluster-test" -> null
      - immutable                      = false -> null
      - type                           = "Opaque" -> null
      - wait_for_service_account_token = true -> null
      - metadata {
          - annotations      = {} -> null
          - generation       = 0 -> null
          - labels           = {
              - "argocd.argoproj.io/secret-type" = "cluster"
              - "orchideo-connect.de/env-name"   = "test"
              - "orchideo-connect.de/is-worker"  = "true"
            } -> null
          - name             = "argocd-cluster-test" -> null
          - namespace        = "argocd" -> null
          - resource_version = "35077526929" -> null
          - uid              = "5e42e3ff-2e48-43d0-9e02-2e1b0d4a1c9b" -> null
        }
    }
  # kubernetes_secret.postgres_credentials will be destroyed
  - resource "kubernetes_secret" "postgres_credentials" {
      - data                           = (sensitive value) -> null
      - id                             = "terraform/postgres-credentials-secret" -> null
      - immutable                      = false -> null
      - type                           = "Opaque" -> null
      - wait_for_service_account_token = true -> null
      - metadata {
          - annotations      = {} -> null
          - generation       = 0 -> null
          - labels           = {} -> null
          - name             = "postgres-credentials-secret" -> null
          - namespace        = "terraform" -> null
          - resource_version = "34519623935" -> null
          - uid              = "d97ed494-534e-4f1d-8352-a5fde6408098" -> null
        }
    }
  # random_password.keyclaok_password will be destroyed
  - resource "random_password" "keyclaok_password" {
      - bcrypt_hash      = (sensitive value) -> null
      - id               = "none" -> null
      - length           = 16 -> null
      - lower            = true -> null
      - min_lower        = 0 -> null
      - min_numeric      = 0 -> null
      - min_special      = 0 -> null
      - min_upper        = 0 -> null
      - number           = true -> null
      - numeric          = true -> null
      - override_special = "!#$%&*()-_=+[]{}<>:?" -> null
      - result           = (sensitive value) -> null
      - special          = true -> null
      - upper            = true -> null
    }
  # random_password.pg_initial_user_password will be destroyed
  - resource "random_password" "pg_initial_user_password" {
      - bcrypt_hash      = (sensitive value) -> null
      - id               = "none" -> null
      - length           = 30 -> null
      - lower            = true -> null
      - min_lower        = 0 -> null
      - min_numeric      = 1 -> null
      - min_special      = 1 -> null
      - min_upper        = 1 -> null
      - number           = true -> null
      - numeric          = true -> null
      - override_special = "!+" -> null
      - result           = (sensitive value) -> null
      - special          = true -> null
      - upper            = true -> null
    }
Plan: 0 to add, 0 to change, 18 to destroy.
ionoscloud_pg_database.keyclaok_pg_database: Destroying... [id=e2674557-0458-547c-89ca-5a85935f29ad]

kubernetes_secret.postgres_credentials: Destroying... [id=terraform/postgres-credentials-secret]
kubernetes_secret.argocd_cluster: Destroying... [id=argocd/argocd-cluster-test]
kubernetes_secret.postgres_credentials: Destruction complete after 0s
kubernetes_namespace.terraform: Destroying... [id=terraform]
kubernetes_secret.argocd_cluster: Destruction complete after 0s
ionoscloud_pg_database.keyclaok_pg_database: Destruction complete after 0s
ionoscloud_pg_user.keyclaok_user: Destroying... [id=8fd5d228-3613-5cb8-804c-0a4a4b99edae]
ionoscloud_pg_user.keyclaok_user: Destruction complete after 0s
random_password.keyclaok_password: Destroying... [id=none]
random_password.keyclaok_password: Destruction complete after 0s
kubernetes_namespace.terraform: Still destroying... [id=terraform, 10s elapsed]
kubernetes_namespace.terraform: Destruction complete after 13s
╷
│ Error: 500 Internal Server Error: {"httpStatus":500,"messages":[{"errorCode":"dbaas-postgresql-01","message":"Internal server error."}]}

Additional Notes

The counter of planned resources is off because of removed project sprecifics.

Found that deleting the Postgres Cluster in DCD before restarting the destruction pipeline "fixes" this.

References

none

[cristiGuranIonos]

This is a known issue, that is being worked. We will let you know when it is fixed, but it might take a while.
Meanwhile, as a workaround maybe you can try:

1. Set a 10s sleep after each user destruction. This example sets one(20s sleep) after cluster creation also.

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    ionoscloud = {
       source = "ionos-cloud/ionoscloud"
        version = "6.5.9"
    }
  }
}
resource "ionoscloud_datacenter" "example" {
  name                    = "example"
  location                = "de/txl"
  description             = "Datacenter for testing dbaas cluster"
}

resource "ionoscloud_lan"  "example" {
  datacenter_id           = ionoscloud_datacenter.example.id
  public                  = false
  name                    = "example"
}

resource "ionoscloud_pg_cluster" "example" {
  postgres_version        = "15"
  instances               = 1
  cores                   = 4
  ram                     = 2048
  storage_size            = 2048
  storage_type            = "SSD"
  connections   {
    datacenter_id         =  ionoscloud_datacenter.example.id
    lan_id                =  ionoscloud_lan.example.id
    cidr                  =  "192.168.100.1/24"
  }
  location                = ionoscloud_datacenter.example.location
  display_name            = "PostgreSQL_cluster"
  maintenance_window {
    day_of_the_week       = "Sunday"
    time                  = "09:00:00"
  }
  synchronization_mode    = "ASYNCHRONOUS"
}

resource "random_password" "user_password" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "ionoscloud_pg_user" "example_pg_user2" {
  cluster_id = ionoscloud_pg_cluster.example.id
  username = "exampleuser2"
  password = random_password.user_password2.result
  depends_on = [time_sleep.wait_10_seconds, time_sleep.wait_20_seconds]
}

resource "ionoscloud_pg_user" "example_pg_user3" {
  cluster_id = ionoscloud_pg_cluster.example.id
  username = "exampleuser3"
  password = random_password.user_password2.result
  depends_on = [time_sleep.wait_10_seconds, time_sleep.wait_20_seconds]
}

resource "random_password" "user_password2" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "time_sleep" "wait_10_seconds" {
  depends_on = [ionoscloud_pg_cluster.example]
  destroy_duration = "10s"
}

resource "time_sleep" "wait_20_seconds" {
  depends_on = [ionoscloud_pg_cluster.example]
  create_duration = "20s"
}

2. Set parallelism=1 .