Skip to content

Kubernetes aggregation connection deadlocks #370

@markchalloner

Description

@markchalloner

Describe the bug
Using the following aggregation:

connection "kubernetes_all" {
  plugin      = "kubernetes@1.5.2
  type        = "aggregator"
  connections = ["kubernetes_*"]
}

causes a deadlock

2026-04-27 11:22:30.778 UTC [161] ERROR:  deadlock detected
2026-04-27 11:22:30.778 UTC [161] DETAIL:  Process 161 waits for ShareUpdateExclusiveLock on relation 19035 of database 16384; blocked by process 154.
        Process 154 waits for ShareUpdateExclusiveLock on relation 18963 of database
        Process 161: COMMENT ON FOREIGN TABLE "kubernetes_all"."kubernetes_horizontal_pod_autoscaler" is $steampipe_escape$Kubernetes HorizontalPodAutoscaler is the configuration for a horizontal pod autoscaler, which automatically mana
ges the replica count of any resource implementing the scale subresource based on the metrics specified.$steampipe_escape$;
        COMMENT ON COLUMN "kubernetes_all"."kubernetes_horizontal_pod_autoscaler"."name" is $steampipe_escape$Name of the object.  Name must be unique within a namespace.$steampipe_escape$;

and the connection to never enter the ready state:

$ steampipe query --output=csv "select * from steampipe_connection_state where state != 'ready';"
name,state,type,connections,import_schema,error,plugin,plugin_instance,schema_mode,schema_hash,comments_set,connection_mod_time,plugin_mod_time,file_name,start_line_number,end_line_number kubernetes_all,error,aggregator,kubernetes_*,enabled,ERROR: deadlock detected (SQLSTATE 40P01),hub.steampipe.io/plugins/turbot/kubernetes@1.5.2,hub.steampipe.io/plugins/turbot/kubernetes@1.5.2,dynamic,bfe54a76630d3d3c2666c24d704b6e6a,tr
ue,2026-04-27T15:59:52Z,1970-01-01T00:00:00Z,/home/tm-app/.steampipe/config/kubernetes.spc,6,10

The database log also shows that goFdwImport is run twice for this schema

2026-04-28 10:06:46.234 UTC [INFO]  hub: goFdwImportForeignSchema remote 'hub.steampipe.io/plugins/turbot/kubernetes@1.5.2' local 'kubernetes_all'
2026-04-28 10:06:46.558 UTC [INFO]  hub: goFdwImportForeignSchema remote 'hub.steampipe.io/plugins/turbot/kubernetes@1.5.2' local 'kubernetes_all'

Steampipe version (steampipe -v)
v2.4.2

Plugin version (steampipe plugin list)
1.5.2

To reproduce

  • Create a config with an aggregate connection and a kubernetes manifest connection:
connection "kubernetes_all" {
  plugin      = "kubernetes@1.5.2
  type        = "aggregator"
  connections = ["kubernetes_*"]
}
connection "kubernetes_foo" {
  plugin  = "kubernetes@1.5.2
  manifest_file_paths = [
    "foo-pods.yaml"
  ]
  source_types = ["manifest"]
}
  • Run steampipe service start
  • Observe the state of the kubernetes_all schema never becomes ready
steampipe query --output=csv "select * from steampipe_connection_state where state != 'ready';"

Expected behavior
No duplicate goFdwImportForeignSchema and no deadlock

Additional context
The ordering of connections seems to matter, interestingly this only seems to happen if the aggregate connection is the first connection alphabetically (i.e. something like kubernetes_x which calls goFdwImportForeignSchema after the individual kubernetes_cluster connections will not deadlock).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions