Skip to content

plugin.gatewayAPI: panic: runtime error: slice bounds out of range [:2:1] #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
erikwoo opened this issue Dec 19, 2024 · 5 comments
Open

plugin.gatewayAPI: panic: runtime error: slice bounds out of range [:2:1] #95

erikwoo opened this issue Dec 19, 2024 · 5 comments
Assignees
@Philipp-Plotnikov
Labels
bug Something isn't working

Comments

@erikwoo
Copy link

erikwoo commented Dec 19, 2024 •
edited
Loading

Checklist:

  • [*] I've included steps to reproduce the bug.
  • [*] I've included the version of argo rollouts.
  • [*] I've included the version of the gateway API plugin
  • [* ] I've included the name and version of the gateway API provider I am using
  • [*] I've included my Rollout Spec (anonymized if needed)

Describe the bug

To Reproduce

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: '1'
  labels:
    gitops/apps.git: demo
  name: demo
  namespace: demo
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: demo
  strategy:
    canary:
      analysis:
        args:
          - name: service
            value: demo
          - name: namespace
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        startingStep: 0
        templates:
          - templateName: istio-success-rate
      antiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution: {}
      canaryService: demo-canary
      maxSurge: 50%
      maxUnavailable: 0
      stableService: demo
      steps:
        - setCanaryScale:
            replicas: 1
        - setHeaderRoute:
            match:
              - headerName: cookie
                headerValue:
                  regex: company_id=.*(666|111)$
            name: route-by-cookie
        - pause:
            duration: 30m
      trafficRouting:
        managedRoutes:
          - name: route-by-cookie
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: demo
            namespace: demo
  template:
    metadata:
      annotations:
        configmap-md5: ''
        descheduler.alpha.kubernetes.io/evictable: 'true'
      labels:
        app: demo
    spec:
      containers:
        - env:
            - name: NAMESPACE
              value: demo
            - name: APP
              value: demo
            - name: CLUSTER_TYPE
              value: test
          image: >-
            registry-vpc.cn-hangzhou.aliyuncs.com/gitops-tools/rollouts-demo:yellow
          imagePullPolicy: Always
          name: rollouts-demo
          ports:
            - containerPort: 8080
              protocol: TCP
      enableServiceLinks: false
      serviceAccountName: demo

Version

v1.7.2 argo-rollouts
v0.4.0 rollouts-plugin-trafficrouter-gatewayapi

Logs
time="2024-12-19T12:17:15Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=demo rollout=demo
time="2024-12-19T12:17:15Z" level=info msg="Reconciling TrafficRouting with type 'GatewayAPI'" namespace=demo rollout=demo
2024-12-19T12:17:15.863Z [DEBUG] plugin.gatewayAPI: time="2024-12-19T12:17:15Z" level=info msg="[RemoveManagedRoutes] plugin "argoproj-labs/gatewayAPI" controls HTTPRoutes: [demo]" plugin=trafficrouter
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: panic: runtime error: slice bounds out of range [:2:1]
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI:
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: goroutine 40 [running]:
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: slices.Delete...
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /opt/hostedtoolcache/go/1.22.6/x64/src/slices/slices.go:219
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.removeManagedHTTPRouteEntry(0xc000747380, {0xc0003bedc0, 0x1, 0x4}, {0xc00036f060, 0xf}, {0xc00036f63c, 0x4})
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/httproute.go:393 +0x4f1
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).removeHTTPManagedRoutes(0xc00046a500, {0xc00075e040, 0x1, 0x0?}, 0xc0005bda20)
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/httproute.go:296 +0x597
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).RemoveManagedRoutes.func1({{0xc00036f63c?, 0xc0004b39d0?}, 0x4?})
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/plugin.go:153 +0x65
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.forEachGatewayAPIRoute[...]({0xc0007279b0?, 0xc0006876d0?, 0x2?}, 0xc0005716e0?)
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/plugin.go:276 +0x56
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).RemoveManagedRoutes(0xc00046a500, 0xc000711108)
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/plugin.go:148 +0x1a5
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: github.com/argoproj/argo-rollouts/rollout/trafficrouting/plugin/rpc.(*TrafficRouterRPCServer).RemoveManagedRoutes(0x1?, {0x1875500?, 0xc000711108?}, 0xc00075e090)
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /home/runner/go/pkg/mod/github.com/argoproj/[email protected]/rollout/trafficrouting/plugin/rpc/rpc.go:255 +0x3f
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: reflect.Value.call({0xc00053c5a0?, 0xc0004f0030?, 0x13?}, {0x1c4e6db, 0x4}, {0xc0004ebef8, 0x3, 0x3?})
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /opt/hostedtoolcache/go/1.22.6/x64/src/reflect/value.go:596 +0xca6
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: reflect.Value.Call({0xc00053c5a0?, 0xc0004f0030?, 0x0?}, {0xc000505ef8?, 0x0?, 0x0?})
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /opt/hostedtoolcache/go/1.22.6/x64/src/reflect/value.go:380 +0xb9
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: net/rpc.(*service).call(0xc0002fc0c0, 0xc0002fc080, 0xc0002240a0, 0xc0002240d0, 0xc0004ee200, 0xc0002da0c0, {0x198bb40?, 0xc0002e46c0?, 0x0?}, {0x19ee700, ...}, ...)
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /opt/hostedtoolcache/go/1.22.6/x64/src/net/rpc/server.go:381 +0x20e
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: created by net/rpc.(*Server).ServeCodec in goroutine 38
2024-12-19T12:17:15.888Z [DEBUG] plugin.gatewayAPI: /opt/hostedtoolcache/go/1.22.6/x64/src/net/rpc/server.go:478 +0x3d7
2024-12-19T12:17:15.890Z [ERROR] plugin: plugin process exited: plugin=/home/argo-rollouts/plugin-bin/argoproj-labs/gatewayAPI id=146609 error="exit status 2"
time="2024-12-19T12:17:15Z" level=error msg="roCtx.reconcile err failed to remove managed routes via plugin: RemoveManagedRoutes rpc call error: unexpected EOF" generation=2 namespace=demo resourceVersion=30188502 rollout=demo
time="2024-12-19T12:17:15Z" level=info msg="Reconciliation completed" generation=2 namespace=demo resourceVersion=30188502 rollout=demo time_ms=87.60934099999999
time="2024-12-19T12:17:15Z" level=error msg="rollout syncHandler error: failed to remove managed routes via plugin: RemoveManagedRoutes rpc call error: unexpected EOF" namespace=demo rollout=demo
time="2024-12-19T12:17:15Z" level=info msg="rollout syncHandler queue retries: 19 : key "demo/demo"" namespace=demo rollout=demo
time="2024-12-19T12:17:15Z" level=error msg="failed to remove managed routes via plugin: RemoveManagedRoutes rpc call error: unexpected EOF" error=""


Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
@kostis-codefresh kostis-codefresh added the bug Something isn't working label Dec 20, 2024
@guy-frontegg
Copy link

We are affected by this issue also

time="2025-01-02T09:48:18Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=frontegg rollout=identity-v2-hp
time="2025-01-02T09:48:18Z" level=info msg="Reconciling TrafficRouting with type 'GatewayAPI'" namespace=frontegg rollout=identity-v2-hp
2025-01-02T09:48:18.869Z [DEBUG] plugin.gatewayAPI: time="2025-01-02T09:48:18Z" level=info msg="[SetHeaderRoute] plugin \"argoproj-labs/gatewayAPI\" controls HTTPRoutes: [identity-v2-hp-route]" plugin=trafficrouter
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: panic: runtime error: slice bounds out of range [:2:1]
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI:
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: goroutine 27 [running]:
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: slices.Delete[...](...)
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/opt/hostedtoolcache/go/1.22.6/x64/src/slices/slices.go:219
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.removeManagedHTTPRouteEntry(0xc000651ad0, {0xc00039a000, 0x1, 0x4}, {0xc00037d390, 0xe}, {0xc00004a750, 0x14})
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/httproute.go:393 +0x4f1
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).removeHTTPManagedRoutes(0xc000427000, {0xc00067b590, 0x1, 0x2d11aa0?}, 0xc0005a9600)
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/httproute.go:296 +0x597
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).setHTTPHeaderRoute(0x0?, 0x1000100?, 0x0?, 0xffffffffffffffff?)
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/httproute.go:74 +0x57a
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).SetHeaderRoute.func1({{0xc00004a750?, 0xc0000c5810?}, 0x4?})
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/plugin.go:103 +0x55
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.forEachGatewayAPIRoute[...]({0xc000524690?, 0xc0006716c8?, 0x2?}, 0xc00067b6d8?)
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/plugin.go:276 +0x56
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin.(*RpcPlugin).SetHeaderRoute(0xc000427000, 0xc0005cae08, 0xc0005cb418)
2025-01-02T09:48:18.955Z [DEBUG] plugin.gatewayAPI: 	/home/runner/work/rollouts-plugin-trafficrouter-gatewayapi/rollouts-plugin-trafficrouter-gatewayapi/pkg/plugin/plugin.go:98 +0x1cd
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: github.com/argoproj/argo-rollouts/rollout/trafficrouting/plugin/rpc.(*TrafficRouterRPCServer).SetHeaderRoute(0x1?, {0x1875540?, 0xc0005cae08?}, 0xc0003e2190)
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: 	/home/runner/go/pkg/mod/github.com/argoproj/[email protected]/rollout/trafficrouting/plugin/rpc/rpc.go:214 +0x49
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: reflect.Value.call({0xc00008b7a0?, 0xc0004e0438?, 0x0?}, {0x1c4e6db, 0x4}, {0xc0005ebef8, 0x3, 0x0?})
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: 	/opt/hostedtoolcache/go/1.22.6/x64/src/reflect/value.go:596 +0xca6
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: reflect.Value.Call({0xc00008b7a0?, 0xc0004e0438?, 0x0?}, {0xc0005ebef8?, 0x0?, 0x0?})
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: 	/opt/hostedtoolcache/go/1.22.6/x64/src/reflect/value.go:380 +0xb9
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: net/rpc.(*service).call(0xc0000ab2c0, 0xc0000ab280, 0xc00045a210, 0xc00045a220, 0xc000427480, 0xc00021a780, {0x198bb40?, 0xc0003360d0?, 0x0?}, {0x19ee700, ...}, ...)
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: 	/opt/hostedtoolcache/go/1.22.6/x64/src/net/rpc/server.go:381 +0x20e
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: created by net/rpc.(*Server).ServeCodec in goroutine 12
2025-01-02T09:48:18.956Z [DEBUG] plugin.gatewayAPI: 	/opt/hostedtoolcache/go/1.22.6/x64/src/net/rpc/server.go:478 +0x3d7
time="2025-01-02T09:48:18Z" level=error msg="roCtx.reconcile err failed to set header route via plugin: SetHeaderRoute rpc call error: unexpected EOF" generation=9 namespace=frontegg resourceVersion=331489727 rollout=identity-v2-hp
time="2025-01-02T09:48:18Z" level=info msg="Reconciliation completed" generation=9 namespace=frontegg resourceVersion=331489727 rollout=identity-v2-hp time_ms=172.71424399999998
time="2025-01-02T09:48:18Z" level=error msg="rollout syncHandler error: failed to set header route via plugin: SetHeaderRoute rpc call error: unexpected EOF" namespace=frontegg rollout=identity-v2-hp
time="2025-01-02T09:48:18Z" level=info msg="rollout syncHandler queue retries: 273 : key \"frontegg/identity-v2-hp\"" namespace=frontegg rollout=identity-v2-hp
time="2025-01-02T09:48:18Z" level=error msg="failed to set header route via plugin: SetHeaderRoute rpc call error: unexpected EOF" error="<nil>"

@rtagliabue
Copy link

rtagliabue commented Mar 6, 2025 •
edited
Loading

we also have this issue, are there any updates?

time="2025-03-06T18:00:16Z" level=error msg="rollout syncHandler error: failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:16Z" level=info msg="rollout syncHandler queue retries: 110 : key \"rollouts/rollout-header\"" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:16Z" level=error msg="failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" error="<nil>"
time="2025-03-06T18:00:19Z" level=info msg="invalidated cache for resource in namespace: argo-rollouts with the name: argo-rollouts-notification-configmap"
time="2025-03-06T18:00:19Z" level=info msg="Start processing" resource=rollouts/rollout-header
time="2025-03-06T18:00:19Z" level=info msg="Processing completed" resource=rollouts/rollout-header
time="2025-03-06T18:00:26Z" level=info msg="Started syncing rollout" generation=1 namespace=rollouts resourceVersion=203793 rollout=rollout-header
time="2025-03-06T18:00:26Z" level=info msg="delaying service switch from  to fd59f4c46: ReplicaSet not fully available" namespace=rollouts rollout=rollout-header service=canary-svc-header
time="2025-03-06T18:00:26Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:26Z" level=info msg="Reconciling TrafficRouting with type 'GatewayAPI'" namespace=rollouts rollout=rollout-header
2025-03-06T18:00:26.539Z [DEBUG] plugin.gatewayAPI: time="2025-03-06T18:00:26Z" level=info msg="[RemoveManagedRoutes] plugin \"argoproj-labs/gatewayAPI\" controls HTTPRoutes: [http-route]" plugin=trafficrouter
time="2025-03-06T18:00:26Z" level=error msg="roCtx.reconcile err failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" generation=1 namespace=rollouts resourceVersion=203793 rollout=rollout-header
time="2025-03-06T18:00:26Z" level=info msg="Reconciliation completed" generation=1 namespace=rollouts resourceVersion=203793 rollout=rollout-header time_ms=11.165773999999999
time="2025-03-06T18:00:26Z" level=error msg="rollout syncHandler error: failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:26Z" level=info msg="rollout syncHandler queue retries: 112 : key \"rollouts/rollout-header\"" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:26Z" level=error msg="failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" error="<nil>"
time="2025-03-06T18:00:36Z" level=info msg="Started syncing rollout" generation=1 namespace=rollouts resourceVersion=203793 rollout=rollout-header
time="2025-03-06T18:00:36Z" level=info msg="delaying service switch from  to fd59f4c46: ReplicaSet not fully available" namespace=rollouts rollout=rollout-header service=canary-svc-header
time="2025-03-06T18:00:36Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:36Z" level=info msg="Reconciling TrafficRouting with type 'GatewayAPI'" namespace=rollouts rollout=rollout-header
2025-03-06T18:00:36.550Z [DEBUG] plugin.gatewayAPI: time="2025-03-06T18:00:36Z" level=info msg="[RemoveManagedRoutes] plugin \"argoproj-labs/gatewayAPI\" controls HTTPRoutes: [http-route]" plugin=trafficrouter
time="2025-03-06T18:00:36Z" level=error msg="roCtx.reconcile err failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" generation=1 namespace=rollouts resourceVersion=203793 rollout=rollout-header
time="2025-03-06T18:00:36Z" level=info msg="Reconciliation completed" generation=1 namespace=rollouts resourceVersion=203793 rollout=rollout-header time_ms=15.670358999999998
time="2025-03-06T18:00:36Z" level=error msg="rollout syncHandler error: failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:36Z" level=info msg="rollout syncHandler queue retries: 113 : key \"rollouts/rollout-header\"" namespace=rollouts rollout=rollout-header
time="2025-03-06T18:00:36Z" level=error msg="failed to remove managed routes via plugin: httproutes.gateway.networking.k8s.io \"http-route\" not found" error="<nil>"

@kostis-codefresh
Copy link
Collaborator

For those affected by this issue could you elaborate a bit on WHEN it happens?

Just when starting a rollout? Upgrading the controller? Scaling back? Doing something else?

@chukmunnlee
Copy link

I found that this can be caused by entries in the argo-gatewayapi-configmap ConfigMap in your namespace. Check the httpManagedRoutes key to see if there is any stale data relating to the managedRoutes. If you can just delete the ConfigMap and let Argo recreate it on the next deployment.

@schlags
Copy link
Contributor

schlags commented Apr 7, 2025 •
edited
Loading

The best way to make this happen is removing the rule in the HTTPRoute that the configmap has marked as managed. It does not seem to be able to understand how to handle the situation where the rule it tries to delete is no longer present (and no other rule at that index exists).

This invites a pretty nasty situation where a rollout in that constant retry loop might end up deleting a rule that it shouldn't.

Either way, one easy way to get into this situation is by using ArgoCD since the added rules put the object Out of Sync if you don't ignoreDIfferences correctly. This means any sync event on that route DURING an active canary can remove the rules that the plugin expects to exist and remove itself.

I've noticed that if I disable autosync on my application running a canary where header routes are added, I do not experience this issue. I've also now configured the following on my argocd application and expect these to stay silent

  - group: gateway.networking.k8s.io
    kind: '*'
    managedFieldsManagers:
      - gatewayAPI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Philipp-Plotnikov Philipp-Plotnikov

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@erikwoo @chukmunnlee @kostis-codefresh @guy-frontegg @schlags @Philipp-Plotnikov @rtagliabue