Skip to content

Flakey test: disapp-deploy-cephfs failover fails — VolSync sync never completes (Submariner Globalnet double IP allocation) #2699

Open
Bug
Open
Flakey test: disapp-deploy-cephfs failover fails — VolSync sync never completes (Submariner Globalnet double IP allocation)#2699
Bug

Description

@nirs
nirs
opened on Aug 9, 2026

Summary

Ramen e2e test disapp-deploy-cephfs fails randomly during Failover while waiting for the DRPC to become ready. The DRPC reaches phase FailedOver / progression Completed, but lastGroupSyncTime remains unset because VolSync never successfully syncs from the new primary (dr2) to the secondary (dr1).

Root cause analysis of gathered artifacts points to a Submariner Globalnet double-allocation race: Globalnet allocates two global IPs for the same VolSync destination ServiceExport; Lighthouse keeps advertising the first (stale) IP while GlobalIngressIP ends on the second. VolSync then connects to the stale IP and sync fails.

This matches a known Globalnet issue previously reproduced on OpenShift (same service getting two global IPs). This issue documents the Ramen-side failure and evidence; a follow-up Submariner issue will track the Globalnet bug.

Note

This is not a test issue but an actual issue with cephfs based apps in upstream and downstream.

Environment / artifacts

  • CI run / gather tarball: e2e.31308426814-1
  • Clusters: hub, dr1, dr2 (minikube / drenv)
  • Storage: Rook CephFS + VolSync (rsync-TLS) + Submariner Globalnet
  • Other CephFS e2e cases in the same run (subscr-deploy-cephfs, appset-deploy-cephfs) completed successfully

Paths below are relative to the tarball root.

1. Failure at the Ramen e2e level

Which test / step failed

  • Test: disapp-deploy-cephfs
  • Failed step: Failover from dr1 → dr2, while waiting for DRPC readiness after phase FailedOver
  • Timeout: FailoverTimeout (15m)

Timeline from e2e log

File: e2e/dr.log

Failover starts:

2026-08-09T10:39:57.927Z	INFO	disapp-deploy-cephfs	dractions/actions.go:165	Failing over workload from cluster "dr1" to cluster "dr2"
2026-08-09T10:39:57.951Z	DEBUG	disapp-deploy-cephfs	dractions/actions.go:308	Updated drpc "ramen-ops/disapp-deploy-cephfs" with action "Failover" to target cluster "dr2"
2026-08-09T10:39:57.951Z	DEBUG	disapp-deploy-cephfs	dractions/retry.go:127	Waiting until drpc "ramen-ops/disapp-deploy-cephfs" reach progression "WaitOnUserToCleanUp" in cluster "hub"

Phase reaches FailedOver, then the test waits for DRPC ready:

2026-08-09T10:42:54.161Z	DEBUG	disapp-deploy-cephfs	dractions/retry.go:74	Waiting until drpc "ramen-ops/disapp-deploy-cephfs" reach phase "FailedOver" in cluster "hub"
2026-08-09T10:42:54.168Z	DEBUG	disapp-deploy-cephfs	dractions/retry.go:85	drpc "ramen-ops/disapp-deploy-cephfs" phase is "FailedOver" in cluster "hub" in 0.007 seconds
2026-08-09T10:42:54.169Z	DEBUG	disapp-deploy-cephfs	dractions/retry.go:28	Waiting until drpc "ramen-ops/disapp-deploy-cephfs" is ready in cluster "hub"

Final failure (~12 minutes later):

2026-08-09T10:54:57.918Z	ERROR	disapp-deploy-cephfs	test/testing.go:55	Failed to failover workload: drpc not ready in cluster "hub" (Available: true, PeerReady: true, ProgressionCompleted: true, lastGroupSyncTime: <nil>): context deadline exceeded
github.com/ramendr/ramen/e2e/test.(*T).Fatalf
	/home/github/actions-runner/_work/ramen/ramen/e2e/test/testing.go:55
github.com/ramendr/ramen/e2e/test.(*Context).Failover
	/home/github/actions-runner/_work/ramen/ramen/e2e/test/context.go:156

Key e2e observation: DRPC reports Available=true, PeerReady=true, progression completed, but lastGroupSyncTime is nil — no successful VolSync group sync after failover.

2. Failure analysis at the Ramen / VolSync level

After failover, Ramen expects VolSync on the new primary (dr2) to sync to the destination service exported from the secondary (dr1) via Submariner (*.svc.clusterset.local). Sync never succeeds, so DRPC never becomes ready for the e2e readiness check.

2.1 DRPC stuck without sync time / DataProtected

File: test/gather.rdr/hub/namespaces/ramen-ops/ramendr.openshift.io/drplacementcontrols/disapp-deploy-cephfs.yaml

status:
  actionStartTime: "2026-08-09T10:39:57Z"
  conditions:
  - reason: FailedOver
    status: "True"
    type: Available
  - reason: Success
    status: "True"
    type: PeerReady
  - message: VolumeReplicationGroup (ramen-ops/disapp-deploy-cephfs) on cluster dr2
      is progressing on protecting workload data (Not all VolSync PVCs are protected),
      retrying till DataProtected condition is met
    reason: Progressing
    status: "False"
    type: Protected
  phase: FailedOver
  progression: Completed
  resourceConditions:
    conditions:
    - message: All VolSync PVCs are ready
      reason: Ready
      status: "True"
      type: DataReady
    - message: Not all VolSync PVCs are protected
      reason: Progressing
      status: "False"
      type: DataProtected

Notes:

  • No lastGroupSyncTime is set on the DRPC status (matches e2e lastGroupSyncTime: <nil>).
  • DataReady=True but DataProtected=False / Protected=False.

2.2 VRG on primary (dr2) also not DataProtected

File: test/gather.rdr/dr2/namespaces/ramen-ops/ramendr.openshift.io/volumereplicationgroups/disapp-deploy-cephfs.yaml

status:
  conditions:
  - message: All VolSync PVCs are ready
    reason: Ready
    status: "True"
    type: DataReady
  - message: Not all VolSync PVCs are protected
    reason: Progressing
    status: "False"
    type: DataProtected

2.3 VolSync ReplicationSource on dr2 (sender) fails to connect

File: test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/volsync.backube/replicationsources/busybox-pvc.yaml

spec:
  rsyncTLS:
    address: volsync-rsync-tls-dst-busybox-pvc.test-disapp-deploy-cephfs.svc.clusterset.local
    copyMethod: Snapshot
    keySecret: disapp-deploy-cephfs-vs-secret
  sourcePVC: busybox-pvc
  trigger:
    schedule: '*/1 * * * *'
status:
  conditions:
  - message: Synchronization in-progress
    reason: SyncInProgress
    status: "True"
    type: Synchronizing
  latestMoverStatus:
    result: Failed
    logs: |-
      rsync: [sender] safe_read failed to read 1 bytes: Connection reset by peer (104)
      rsync error: error in rsync protocol data stream (code 12) at io.c(279) [sender=3.2.5]
      Synchronization failed. rsync returned: 1212

2.4 VolSync mover logs: DNS/clusterset name resolves to stale global IP 242.0.255.253

File: test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/pods/volsync-rsync-tls-src-busybox-pvc-msmv2/rsync-tls/current.log

Syncing data to volsync-rsync-tls-dst-busybox-pvc.test-disapp-deploy-cephfs.svc.clusterset.local:8000 ...
2026.08.09 10:54:49 LOG6[0]: s_connect: connecting 242.0.255.253:8000
2026.08.09 10:54:49 LOG7[0]: s_connect: s_poll_wait 242.0.255.253:8000: waiting 10 seconds
rsync: [sender] safe_read failed to read 1 bytes: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(279) [sender=3.2.5]
2026.08.09 10:54:59 LOG3[0]: s_connect: s_poll_wait 242.0.255.253:8000: TIMEOUTconnect exceeded

2.5 ServiceExport on destination cluster (dr1) looks healthy

File: test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/multicluster.x-k8s.io/serviceexports/volsync-rsync-tls-dst-busybox-pvc.yaml

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: volsync-rsync-tls-dst-busybox-pvc
  namespace: test-disapp-deploy-cephfs
  labels:
    ramendr.openshift.io/created-by-ramen: "true"
  ownerReferences:
  - apiVersion: volsync.backube/v1alpha1
    kind: ReplicationDestination
    name: busybox-pvc
status:
  conditions:
  - reason: Valid
    status: "True"
    type: Valid
  - message: Service was successfully exported to the broker
    reason: Exported
    status: "True"
    type: Ready

2.6 ServiceImport on dr2 is Ready (but does not embed the global IP)

File: test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/multicluster.x-k8s.io/serviceimports/volsync-rsync-tls-dst-busybox-pvc.yaml

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceImport
metadata:
  name: volsync-rsync-tls-dst-busybox-pvc
  namespace: test-disapp-deploy-cephfs
  annotations:
    lighthouse.submariner.io/use-clusterset-ip: "false"
spec:
  ports:
  - name: rsync-tls
    port: 8000
    protocol: TCP
  type: ClusterSetIP
status:
  clusters:
  - cluster: dr1
  conditions:
  - reason: Ready
    status: "True"
    type: Ready

2.7 Lighthouse EndpointSlice on dr2 advertises stale IP 242.0.255.253

File: test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/discovery.k8s.io/endpointslices/volsync-rsync-tls-dst-busybox-pvc-lnwbq.yaml

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: volsync-rsync-tls-dst-busybox-pvc-lnwbq
  namespace: test-disapp-deploy-cephfs
  labels:
    endpointslice.kubernetes.io/managed-by: lighthouse-agent.submariner.io
    multicluster.kubernetes.io/service-name: volsync-rsync-tls-dst-busybox-pvc
    multicluster.kubernetes.io/source-cluster: dr1
endpoints:
- addresses:
  - 242.0.255.253
  conditions:
    ready: true
ports:
- name: rsync-tls
  port: 8000
  protocol: TCP

(Same stale address is present on dr1 as well:
test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/discovery.k8s.io/endpointslices/volsync-rsync-tls-dst-busybox-pvc-lnwbq.yaml)

2.8 GlobalIngressIP final allocated IP is 242.0.255.252 (mismatch)

File: test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/submariner.io/globalingressips/volsync-rsync-tls-dst-busybox-pvc.yaml

apiVersion: submariner.io/v1
kind: GlobalIngressIP
metadata:
  name: volsync-rsync-tls-dst-busybox-pvc
  namespace: test-disapp-deploy-cephfs
spec:
  serviceRef:
    name: volsync-rsync-tls-dst-busybox-pvc
  target: ClusterIPService
status:
  allocatedIP: 242.0.255.252
  conditions:
  - message: Allocated global IP
    reason: Success
    status: "True"
    type: Allocated

IP mismatch (smoking gun)

Source IP
Final GlobalIngressIP.status.allocatedIP 242.0.255.252
Lighthouse EndpointSlice advertised address 242.0.255.253
VolSync mover connect target 242.0.255.253:8000

Ramen/VolSync export/import objects are present and Ready; the cross-cluster address published for the exported VolSync destination service is wrong relative to the final Globalnet allocation.

2.9 ReplicationDestination on dr1 (receiver)

File: test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/volsync.backube/replicationdestinations/busybox-pvc.yaml

apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: busybox-pvc
  namespace: test-disapp-deploy-cephfs
spec:
  rsyncTLS:
    destinationPVC: busybox-pvc
    serviceType: ClusterIP
    keySecret: disapp-deploy-cephfs-vs-secret
status:
  conditions:
  - message: Synchronization in-progress
    reason: SyncInProgress
    status: "True"
    type: Synchronizing
  rsyncTLS:
    address: 10.109.247.141

Destination side is waiting for sync; the failure is on the path to the destination via the stale global IP.

3. Submariner Globalnet evidence (double allocation)

File: test/gather.rdr/dr1/namespaces/submariner-operator/pods/submariner-globalnet-7vvr6/submariner-globalnet/current.log

For service test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc, Globalnet processes the ServiceExport twice and allocates two different global IPs within ~140ms:

2026-08-09T10:42:54.512Z INF ..rvice_controller.go:117 Globalnet            Service "test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc" created
2026-08-09T10:42:54.527Z INF ..xport_controller.go:164 Globalnet            Processing ServiceExport "test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc"
2026-08-09T10:42:54.528Z INF ..xport_controller.go:199 Globalnet            Creating GlobalIngressIP object test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc, TargetRef: "ClusterIPService", "volsync-rsync-tls-dst-busybox-pvc" 
2026-08-09T10:42:54.549Z INF ..essip_controller.go:150 Globalnet            Processing created test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc, TargetRef: "ClusterIPService", "volsync-rsync-tls-dst-busybox-pvc", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}
2026-08-09T10:42:54.551Z INF ..essip_controller.go:193 Globalnet            Allocated global IP ["242.0.255.253"] for "test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc"
2026-08-09T10:42:54.563Z INF ..xport_controller.go:164 Globalnet            Processing ServiceExport "test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc"
2026-08-09T10:42:54.563Z INF ..xport_controller.go:199 Globalnet            Creating GlobalIngressIP object test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc, TargetRef: "ClusterIPService", "volsync-rsync-tls-dst-busybox-pvc" 
2026-08-09T10:42:54.614Z INF ..rvice_controller.go:117 Globalnet            Service "test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc" updated
2026-08-09T10:42:54.691Z INF ..essip_controller.go:150 Globalnet            Processing created test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc, TargetRef: "ClusterIPService", "volsync-rsync-tls-dst-busybox-pvc", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}
2026-08-09T10:42:54.691Z INF ..essip_controller.go:193 Globalnet            Allocated global IP ["242.0.255.252"] for "test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc"
2026-08-09T10:42:55.311Z INF ..essip_controller.go:150 Globalnet            Processing updated test-disapp-deploy-cephfs/volsync-rsync-tls-dst-busybox-pvc, TargetRef: "ClusterIPService", "volsync-rsync-tls-dst-busybox-pvc", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2026, time.August, 9, 10, 42, 54, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.0.255.252"}

Interpretation

  1. ServiceExport create/reconcile races inside Globalnet (Creating GlobalIngressIP twice).
  2. First allocation: 242.0.255.253 (this is what Lighthouse publishes / VolSync connects to).
  3. Second allocation overwrites GlobalIngressIP status to 242.0.255.252 (final gathered CR).
  4. EndpointSlice remains on the first IP → cross-cluster connect to a non-live global IP → VolSync sync fails → Ramen DRPC never gets lastGroupSyncTime → e2e failover times out.

This is the same class of Globalnet “two IPs for one service” failure previously seen on OpenShift.

Related files in the tarball (quick index)

Purpose Path
E2E failure log e2e/dr.log
DRPC test/gather.rdr/hub/namespaces/ramen-ops/ramendr.openshift.io/drplacementcontrols/disapp-deploy-cephfs.yaml
VRG (dr2 primary) test/gather.rdr/dr2/namespaces/ramen-ops/ramendr.openshift.io/volumereplicationgroups/disapp-deploy-cephfs.yaml
VolSync ReplicationSource test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/volsync.backube/replicationsources/busybox-pvc.yaml
VolSync ReplicationDestination test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/volsync.backube/replicationdestinations/busybox-pvc.yaml
ServiceExport test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/multicluster.x-k8s.io/serviceexports/volsync-rsync-tls-dst-busybox-pvc.yaml
ServiceImport (dr2) test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/multicluster.x-k8s.io/serviceimports/volsync-rsync-tls-dst-busybox-pvc.yaml
EndpointSlice (stale IP) test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/discovery.k8s.io/endpointslices/volsync-rsync-tls-dst-busybox-pvc-lnwbq.yaml
GlobalIngressIP (final IP) test/gather.rdr/dr1/namespaces/test-disapp-deploy-cephfs/submariner.io/globalingressips/volsync-rsync-tls-dst-busybox-pvc.yaml
VolSync mover log test/gather.rdr/dr2/namespaces/test-disapp-deploy-cephfs/pods/volsync-rsync-tls-src-busybox-pvc-msmv2/rsync-tls/current.log
Globalnet log (double alloc) test/gather.rdr/dr1/namespaces/submariner-operator/pods/submariner-globalnet-7vvr6/submariner-globalnet/current.log

Logs

Logs: e2e.31308426814-1.tar.gz
Failed build: https://github.com/RamenDR/ramen/actions/runs/31308426814/attempts/1
Successful build: https://github.com/RamenDR/ramen/actions/runs/31308426814/attempts/2

Dpependencies

This is an issue in submariner: submariner-io/submariner#4122

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions