Data Plane routes traffic to stale Pod IPs after backend rollout, causing 504 timeouts

[trantandat18pm]

Summary

We are experiencing intermittent 504 Gateway Timeout errors when deploying new versions of backend applications behind NGINX Gateway Fabric.

The issue occurs after a backend rollout, where the NGINX data plane continues routing traffic to stale Pod IPs that no longer exist, even though:

The new backend Pods are running and healthy

Control plane logs show successful configuration updates

Direct access via NodePort / ClusterIP works correctly

Restarting the data plane pods immediately resolves the issue.

Environment

• NGINX Gateway Fabric version: v2.3.0
• Gateway API version: v1.4.1
• Deployment mode:
  • Control Plane replicas: 1
  • Data Plane replicas: 5

Gateway Configuration (simplified)

• Single Gateway
• Multiple HTTPS listeners
• Wildcard hostnames
• Routes attached from multiple namespaces

listeners:
- protocol: HTTP
  port: 80
- protocol: HTTPS
  port: 443
  hostname: "*.xxxtest.com"
- protocol: HTTPS
  port: 443
  hostname: "*.xxx.com"

Symptoms

During a backend rollout:

• Requests through the Gateway intermittently return 504 Gateway Timeout
• Requests to the same Service via NodePort succeed
• Gateway access logs show traffic being forwarded to unexpected backend IPs
  • These IPs correspond to Pods from the previous version, already terminated
• Restarting the NGF data plane clears the stale IPs and restores normal traffic

[nginx-bot]

Hi @trantandat18pm! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[sjberman]

@trantandat18pm Thanks for reporting this. We've seen similar issues in the past, and thought we had fixed them.

If you run into this situation again, before you restart the NGINX data plane pods, can you verify if the old or new backend IP addresses are referenced in the nginx config, by running kubectl exec <nginx-pod> -- nginx -T? Hoping to understand if the issue is that nginx wasn't reloaded when it should've been, or if the config itself wasn't written properly.

I'd also be curious if nginx -s reload fixes the problem instead of a full pod restart.

In addition, setting debug logs and including those would be helpful. That can be done in the NginxProxy CRD,

kubectl -n nginx-gateway edit nginxproxies.gateway.nginx.org ngf-proxy-config

spec:
  logging:
      agentLevel: debug