no matches for kind "Plan" in version "upgrade.cattle.io/v1" #298

StefanSa · 2024-03-07T13:08:21Z

Version
v0.13.4

Platform/Architecture
openSUSE MicroOS 20240221

Describe the bug
When i create a new plan, i get this error message:

resource mapping not found for name: "k3s-server" namespace: "system-upgrade" from "k3s-upgrade.yaml": no matches for kind "Plan" in version "upgrade.cattle.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "k3s-agent" namespace: "system-upgrade" from "k3s-upgrade.yaml": no matches for kind "Plan" in version "upgrade.cattle.io/v1"
ensure CRDs are installed first

To Reproduce

kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/v0.13.4/manifests/system-upgrade-controller.yaml

kubectl label node master-01 master-02 worker-01 worker-02 worker-03 k3s-upgrade=true
kubectl apply -f k3s-upgrade.yaml

k3s-upgrade.yaml:

---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-server
  namespace: system-upgrade
  labels:
    k3s-upgrade: server
spec:
  concurrency: 1
  version: v1.29.2+k3s1
  nodeSelector:
    matchExpressions:
      - {key: k3s-upgrade, operator: Exists}
      - {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
      - {key: k3s.io/hostname, operator: Exists}
      - {key: k3os.io/mode, operator: DoesNotExist}
      - {key: node-role.kubernetes.io/master, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  cordon: true
#  drain:
#    force: true
  upgrade:
    image: rancher/k3s-upgrade
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-agent
  namespace: system-upgrade
  labels:
    k3s-upgrade: agent
spec:
  concurrency: 2 # in general, this should be the number of workers - 1
  version: v1.29.2+k3s1
  nodeSelector:
    matchExpressions:
      - {key: k3s-upgrade, operator: Exists}
      - {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
      - {key: k3s.io/hostname, operator: Exists}
      - {key: k3os.io/mode, operator: DoesNotExist}
      - {key: node-role.kubernetes.io/master, operator: NotIn, values: ["true"]}
  serviceAccountName: system-upgrade
  prepare:
  prepare:
    # Since v0.5.0-m1 SUC will use the resolved version of the plan for the tag on the prepare container.
    # image: rancher/k3s-upgrade:v1.17.4-k3s1
    image:rancher/k3s-upgrade
    args: ["prepare", "k3s-server"]
  drain:
    force: true
    skipWaitForDeleteTimeout: 60 # set this to prevent upgrades from hanging on small clusters since k8s v1.18
  upgrade:
image:rancher/k3s-upgrade

Expected behavior
Upgrade plan without error message.

Actual behavior
Error message:
no matches for kind "Plan" in version "upgrade.cattle.io/v1"

Additional context
log in pod:

E0307 13:10:47.489933       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Plan: failed to list *v1.Plan: the server could not find the requested resource (get plans.meta.k8s.io)
E0307 13:11:29.290633       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Plan: failed to list *v1.Plan: the server could not find the requested resource (get plans.meta.k8s.io)
E0307 13:11:34.471197       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0307 13:12:03.206769       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Plan: failed to list *v1.Plan: the server could not find the requested resource (get plans.meta.k8s.io)
E0307 13:12:10.178176       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0307 13:12:55.404339       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Plan: failed to list *v1.Plan: the server could not find the requested resource (get plans.meta.k8s.io)
E0307 13:12:59.802270       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"

The text was updated successfully, but these errors were encountered:

brandond · 2024-03-07T16:36:17Z

You should also apply the CRD manifest: https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/crd.yaml

StefanSa · 2024-03-08T08:22:37Z

@brandond Hi Brad, thanks for the hint, that at least fixed this error.
But now i don't get any active jobs displayed and there are still these error messages in the pod.

kubectl get jobs -n system-upgrade
No resources found in system-upgrade namespace.

E0308 08:11:43.584498       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
2024-03-08T08:12:36.630633052Z E0308 08:12:36.630500       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
2024-03-08T08:13:27.715830430Z E0308 08:13:27.715712       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:14:25.235647       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:15:21.877204       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:16:20.665483       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:16:57.158737       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:17:27.865155       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:18:03.792967       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:18:41.345443       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:19:23.898774       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"
E0308 08:20:14.628402       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:system-upgrade:system-upgrade" cannot list resource "secrets" in API group "" in the namespace "system-upgrade"

SISheogorath · 2024-03-08T11:24:38Z

Do you see the system-upgrade-controller role in the namespace?

StefanSa · 2024-03-08T11:35:42Z

@SISheogorath
There are two cluster roles here.
One is system-upgrade-controller and the other is system-upgrade-controller-drainer.

StefanSa · 2024-03-08T12:02:51Z

@SISheogorath @brandond
Error found. The role were incomplete.
The authorization to read secrets and all rights for the jobs were missing.
With this role it works without a problem:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"system-upgrade-controller"},"rules":[{"apiGroups":["batch"],"resources":["jobs"],"verbs":["get","list","watch"]},{"apiGroups":[""],"resources":["namespaces","nodes"],"verbs":["get","list","watch"]},{"apiGroups":[""],"resources":["nodes"],"verbs":["update"]},{"apiGroups":["upgrade.cattle.io"],"resources":["plans","plans/status"],"verbs":["get","list","watch","create","patch","update","delete"]}]}
    objectset.rio.cattle.io/applied: >-
      H4sIAAAAAAAA/6yQQa/TMBCE/8uenbykTvyaXDlw58AFvcPa3ry4de3IXhdB1f+OHCohUQQXTtaMdj6P5ga4uc+UsosBZkgaTYuF15jcd2QXQ3s+5tbFl2sPAs4uWJjhgy+ZKX2KnkDAhRgtMsJ8Awwh8p7LVUZ9IsOZuE0utgaZPVWYqxRCjaqbpgbHQTXDKJdGv8qhIbsMh15bPZoR7gI8avJ/xa2YV5hBvSqlOnWUYyclyYPuD91x0kpKuwzGDLKfrFULVmjAC8EM+VtmujRle09oqTExcIreU6o3qXjKMH/ZN/qYYtmqAo1sVngTkCjHksx+A6eoczWvlPRuvBODAO9yfb7+zNzFb6wnTO2VN6xKQIiW/gv0iVM2i0x/yD6W+DXuE2zzGGq5/X3JjFz+2VGASVQ/FLA9jEcDAZY87VXe7j8CAAD///Ycri6NAgAA
    objectset.rio.cattle.io/id: eaba6099-a546-453f-b734-edf421bdb5c5
  creationTimestamp: '2024-03-07T10:42:28Z'
  finalizers:
    - wrangler.cattle.io/auth-prov-v2-crole
  labels:
    objectset.rio.cattle.io/hash: 6766606835033e32b12089b633df4cc4319dd6fa
  managedFields:
    - apiVersion: rbac.authorization.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-03-07T10:42:28Z'
    - apiVersion: rbac.authorization.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:objectset.rio.cattle.io/applied: {}
            f:objectset.rio.cattle.io/id: {}
          f:finalizers:
            .: {}
            v:"wrangler.cattle.io/auth-prov-v2-crole": {}
          f:labels:
            .: {}
            f:objectset.rio.cattle.io/hash: {}
        f:rules: {}
      manager: rancher
      operation: Update
      time: '2024-03-08T11:50:34Z'
  name: system-upgrade-controller
  resourceVersion: '23869128'
  uid: d3802361-ab53-4cbe-92a9-aaffcd61f325
rules:
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - get
      - list
      - watch
      - create
      - delete
      - patch
      - update
  - apiGroups:
      - ''
    resources:
      - namespaces
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - update
  - apiGroups:
      - upgrade.cattle.io
    resources:
      - plans
      - plans/status
    verbs:
      - get
      - list
      - watch
      - create
      - patch
      - update
      - delete
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - list

SISheogorath · 2024-03-08T12:08:49Z

There should be two clusterroles and one role. When I adjusted the roles for the controller, I decided to limit secrets and job creation to the namespace of the controller.

system-upgrade-controller/manifests/clusterrole.yaml

Lines 53 to 79 in 4a64353

    
           --- 
        
           apiVersion: rbac.authorization.k8s.io/v1 
        
           kind: Role 
        
           metadata: 
        
             name: system-upgrade-controller 
        
           rules: 
        
           - apiGroups: 
        
             - batch 
        
             resources: 
        
             - jobs 
        
             verbs: 
        
             - create 
        
             - delete 
        
             - deletecollection 
        
             - patch 
        
             - update 
        
             - get 
        
             - list 
        
             - watch 
        
           - apiGroups: 
        
             - "" 
        
             resources: 
        
             - secrets 
        
             verbs: 
        
             - get 
        
             - list 
        
             - watch

Maybe this was too restrictive. I just double checked my setup, the controller is functional here with these roles.

The reason I asked for its existence is that it might be related to object ordering: #296

StefanSa · 2024-03-08T12:16:46Z

Here there are only clusterroles and no role

StefanSa · 2024-03-08T12:21:24Z

watch missing also on secret

Failed to watch *v1.Secret: unknown (get secrets.meta.k8s.io)

SISheogorath · 2024-03-08T12:39:18Z

If you apply the release manifest a second time (now that the namespace exists), does it fix the issue?

StefanSa · 2024-03-08T12:51:15Z

all objects unchanged.

kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/v0.13.4/manifests/system-upgrade-controller.yaml
namespace/system-upgrade unchanged
serviceaccount/system-upgrade unchanged
configmap/default-controller-env unchanged
deployment.apps/system-upgrade-controller unchanged

SISheogorath · 2024-03-08T12:57:50Z

If you use the manifests directory from the tag in the git repository, you have to apply all manifests.

The release manifest I referred to is attached to the Release on GitHub: https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/system-upgrade-controller.yaml

StefanSa · 2024-03-08T13:09:55Z

That looks good.

kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/system-upgrade-controller.yaml
clusterrole.rbac.authorization.k8s.io/system-upgrade-controller configured
role.rbac.authorization.k8s.io/system-upgrade-controller created
clusterrole.rbac.authorization.k8s.io/system-upgrade-controller-drainer unchanged
clusterrolebinding.rbac.authorization.k8s.io/system-upgrade-drainer unchanged
clusterrolebinding.rbac.authorization.k8s.io/system-upgrade unchanged
rolebinding.rbac.authorization.k8s.io/system-upgrade created
namespace/system-upgrade unchanged
serviceaccount/system-upgrade unchanged
configmap/default-controller-env unchanged
deployment.apps/system-upgrade-controller configured

And no more error messages, a role has also been created.

1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2024-03-08T13:00:42Z" level=info msg="No access to list CRDs, assuming CRDs are pre-created."
2024-03-08T13:00:43.076970550Z time="2024-03-08T13:00:43Z" level=info msg="Starting /v1, Kind=Node controller"
2024-03-08T13:00:43.077002976Z time="2024-03-08T13:00:43Z" level=info msg="Starting /v1, Kind=Secret controller"
2024-03-08T13:00:43.099580197Z time="2024-03-08T13:00:43Z" level=info msg="Starting batch/v1, Kind=Job controller"
2024-03-08T13:00:43.146037841Z time="2024-03-08T13:00:43Z" level=info msg="Starting upgrade.cattle.io/v1, Kind=Plan controller"

SISheogorath · 2024-03-08T13:11:17Z

Well, now we know, that #296 actually fixed a problem 🙌🏻

gravufo · 2024-04-14T18:31:39Z

The permissions are not correct, the upgrade pod spews errors that it is not allowed to delete pods.
I tried completely removing all SUC resources (including CRDs) and reinstalling from scratch and the issue persists. Going back to version 0.13.2 makes it work.
There is definitely something wrong in the roles or cluster roles. Didn't dig deeper yet.

damdo · 2024-07-21T13:13:53Z

I faced similar issues as @gravufo
and dug deeper. I created an issue out of this #319
and a PR to fix that #320

harmathy mentioned this issue Mar 23, 2024

Installation of CRD for system-upgrade-controller is missing k3s-io/docs#255

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no matches for kind "Plan" in version "upgrade.cattle.io/v1" #298

no matches for kind "Plan" in version "upgrade.cattle.io/v1" #298

StefanSa commented Mar 7, 2024 •

edited

Loading

brandond commented Mar 7, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

gravufo commented Apr 14, 2024

damdo commented Jul 21, 2024

no matches for kind "Plan" in version "upgrade.cattle.io/v1" #298

no matches for kind "Plan" in version "upgrade.cattle.io/v1" #298

Comments

StefanSa commented Mar 7, 2024 • edited Loading

brandond commented Mar 7, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

StefanSa commented Mar 8, 2024

SISheogorath commented Mar 8, 2024

gravufo commented Apr 14, 2024

damdo commented Jul 21, 2024

StefanSa commented Mar 7, 2024 •

edited

Loading