Two consecutive live migrations of a VM on a Linstor storage pool fail

[ykazakov]

Is there an existing issue for this?

• There is no existing issue for this bug

Is this happening on an up to date version of Incus?

• This is happening on a supported version of Incus

Incus system details

config:
  cluster.https_address: 10.10.0.12:8443
  core.https_address: :8443
  storage.backups_volume: local/backups
  storage.images_volume: local/images
  storage.linstor.controller_connection: http://10.20.0.101:3370
  user.ui.title: KI
api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - network_sriov
  - console
  - restrict_dev_incus
  - migration_pre_copy
  - infiniband
  - dev_incus_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - dev_incus_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - backup_compression
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - snapshot_schedule_aliases
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  - resources_network_firmware
  - backup_compression_algorithm
  - ceph_data_pool_name
  - container_syscall_intercept_mount
  - compression_squashfs
  - container_raw_mount
  - container_nic_routed
  - container_syscall_intercept_mount_fuse
  - container_disk_ceph
  - virtual-machines
  - image_profiles
  - clustering_architecture
  - resources_disk_id
  - storage_lvm_stripes
  - vm_boot_priority
  - unix_hotplug_devices
  - api_filtering
  - instance_nic_network
  - clustering_sizing
  - firewall_driver
  - projects_limits
  - container_syscall_intercept_hugetlbfs
  - limits_hugepages
  - container_nic_routed_gateway
  - projects_restrictions
  - custom_volume_snapshot_expiry
  - volume_snapshot_scheduling
  - trust_ca_certificates
  - snapshot_disk_usage
  - clustering_edit_roles
  - container_nic_routed_host_address
  - container_nic_ipvlan_gateway
  - resources_usb_pci
  - resources_cpu_threads_numa
  - resources_cpu_core_die
  - api_os
  - container_nic_routed_host_table
  - container_nic_ipvlan_host_table
  - container_nic_ipvlan_mode
  - resources_system
  - images_push_relay
  - network_dns_search
  - container_nic_routed_limits
  - instance_nic_bridged_vlan
  - network_state_bond_bridge
  - usedby_consistency
  - custom_block_volumes
  - clustering_failure_domains
  - resources_gpu_mdev
  - console_vga_type
  - projects_limits_disk
  - network_type_macvlan
  - network_type_sriov
  - container_syscall_intercept_bpf_devices
  - network_type_ovn
  - projects_networks
  - projects_networks_restricted_uplinks
  - custom_volume_backup
  - backup_override_name
  - storage_rsync_compression
  - network_type_physical
  - network_ovn_external_subnets
  - network_ovn_nat
  - network_ovn_external_routes_remove
  - tpm_device_type
  - storage_zfs_clone_copy_rebase
  - gpu_mdev
  - resources_pci_iommu
  - resources_network_usb
  - resources_disk_address
  - network_physical_ovn_ingress_mode
  - network_ovn_dhcp
  - network_physical_routes_anycast
  - projects_limits_instances
  - network_state_vlan
  - instance_nic_bridged_port_isolation
  - instance_bulk_state_change
  - network_gvrp
  - instance_pool_move
  - gpu_sriov
  - pci_device_type
  - storage_volume_state
  - network_acl
  - migration_stateful
  - disk_state_quota
  - storage_ceph_features
  - projects_compression
  - projects_images_remote_cache_expiry
  - certificate_project
  - network_ovn_acl
  - projects_images_auto_update
  - projects_restricted_cluster_target
  - images_default_architecture
  - network_ovn_acl_defaults
  - gpu_mig
  - project_usage
  - network_bridge_acl
  - warnings
  - projects_restricted_backups_and_snapshots
  - clustering_join_token
  - clustering_description
  - server_trusted_proxy
  - clustering_update_cert
  - storage_api_project
  - server_instance_driver_operational
  - server_supported_storage_drivers
  - event_lifecycle_requestor_address
  - resources_gpu_usb
  - clustering_evacuation
  - network_ovn_nat_address
  - network_bgp
  - network_forward
  - custom_volume_refresh
  - network_counters_errors_dropped
  - metrics
  - image_source_project
  - clustering_config
  - network_peer
  - linux_sysctl
  - network_dns
  - ovn_nic_acceleration
  - certificate_self_renewal
  - instance_project_move
  - storage_volume_project_move
  - cloud_init
  - network_dns_nat
  - database_leader
  - instance_all_projects
  - clustering_groups
  - ceph_rbd_du
  - instance_get_full
  - qemu_metrics
  - gpu_mig_uuid
  - event_project
  - clustering_evacuation_live
  - instance_allow_inconsistent_copy
  - network_state_ovn
  - storage_volume_api_filtering
  - image_restrictions
  - storage_zfs_export
  - network_dns_records
  - storage_zfs_reserve_space
  - network_acl_log
  - storage_zfs_blocksize
  - metrics_cpu_seconds
  - instance_snapshot_never
  - certificate_token
  - instance_nic_routed_neighbor_probe
  - event_hub
  - agent_nic_config
  - projects_restricted_intercept
  - metrics_authentication
  - images_target_project
  - images_all_projects
  - cluster_migration_inconsistent_copy
  - cluster_ovn_chassis
  - container_syscall_intercept_sched_setscheduler
  - storage_lvm_thinpool_metadata_size
  - storage_volume_state_total
  - instance_file_head
  - instances_nic_host_name
  - image_copy_profile
  - container_syscall_intercept_sysinfo
  - clustering_evacuation_mode
  - resources_pci_vpd
  - qemu_raw_conf
  - storage_cephfs_fscache
  - network_load_balancer
  - vsock_api
  - instance_ready_state
  - network_bgp_holdtime
  - storage_volumes_all_projects
  - metrics_memory_oom_total
  - storage_buckets
  - storage_buckets_create_credentials
  - metrics_cpu_effective_total
  - projects_networks_restricted_access
  - storage_buckets_local
  - loki
  - acme
  - internal_metrics
  - cluster_join_token_expiry
  - remote_token_expiry
  - init_preseed
  - storage_volumes_created_at
  - cpu_hotplug
  - projects_networks_zones
  - network_txqueuelen
  - cluster_member_state
  - instances_placement_scriptlet
  - storage_pool_source_wipe
  - zfs_block_mode
  - instance_generation_id
  - disk_io_cache
  - amd_sev
  - storage_pool_loop_resize
  - migration_vm_live
  - ovn_nic_nesting
  - oidc
  - network_ovn_l3only
  - ovn_nic_acceleration_vdpa
  - cluster_healing
  - instances_state_total
  - auth_user
  - security_csm
  - instances_rebuild
  - numa_cpu_placement
  - custom_volume_iso
  - network_allocations
  - zfs_delegate
  - storage_api_remote_volume_snapshot_copy
  - operations_get_query_all_projects
  - metadata_configuration
  - syslog_socket
  - event_lifecycle_name_and_project
  - instances_nic_limits_priority
  - disk_initial_volume_configuration
  - operation_wait
  - image_restriction_privileged
  - cluster_internal_custom_volume_copy
  - disk_io_bus
  - storage_cephfs_create_missing
  - instance_move_config
  - ovn_ssl_config
  - certificate_description
  - disk_io_bus_virtio_blk
  - loki_config_instance
  - instance_create_start
  - clustering_evacuation_stop_options
  - boot_host_shutdown_action
  - agent_config_drive
  - network_state_ovn_lr
  - image_template_permissions
  - storage_bucket_backup
  - storage_lvm_cluster
  - shared_custom_block_volumes
  - auth_tls_jwt
  - oidc_claim
  - device_usb_serial
  - numa_cpu_balanced
  - image_restriction_nesting
  - network_integrations
  - instance_memory_swap_bytes
  - network_bridge_external_create
  - network_zones_all_projects
  - storage_zfs_vdev
  - container_migration_stateful
  - profiles_all_projects
  - instances_scriptlet_get_instances
  - instances_scriptlet_get_cluster_members
  - instances_scriptlet_get_project
  - network_acl_stateless
  - instance_state_started_at
  - networks_all_projects
  - network_acls_all_projects
  - storage_buckets_all_projects
  - resources_load
  - instance_access
  - project_access
  - projects_force_delete
  - resources_cpu_flags
  - disk_io_bus_cache_filesystem
  - instance_oci
  - clustering_groups_config
  - instances_lxcfs_per_instance
  - clustering_groups_vm_cpu_definition
  - disk_volume_subpath
  - projects_limits_disk_pool
  - network_ovn_isolated
  - qemu_raw_qmp
  - network_load_balancer_health_check
  - oidc_scopes
  - network_integrations_peer_name
  - qemu_scriptlet
  - instance_auto_restart
  - storage_lvm_metadatasize
  - ovn_nic_promiscuous
  - ovn_nic_ip_address_none
  - instances_state_os_info
  - network_load_balancer_state
  - instance_nic_macvlan_mode
  - storage_lvm_cluster_create
  - network_ovn_external_interfaces
  - instances_scriptlet_get_instances_count
  - cluster_rebalance
  - custom_volume_refresh_exclude_older_snapshots
  - storage_initial_owner
  - storage_live_migration
  - instance_console_screenshot
  - image_import_alias
  - authorization_scriptlet
  - console_force
  - network_ovn_state_addresses
  - network_bridge_acl_devices
  - instance_debug_memory
  - init_preseed_storage_volumes
  - init_preseed_profile_project
  - instance_nic_routed_host_address
  - instance_smbios11
  - api_filtering_extended
  - acme_dns01
  - security_iommu
  - network_ipv4_dhcp_routes
  - network_state_ovn_ls
  - network_dns_nameservers
  - acme_http01_port
  - network_ovn_ipv4_dhcp_expiry
  - instance_state_cpu_time
  - network_io_bus
  - disk_io_bus_usb
  - storage_driver_linstor
  - instance_oci_entrypoint
  - network_address_set
  - server_logging
  - network_forward_snat
  - memory_hotplug
  - instance_nic_routed_host_tables
  - instance_publish_split
  - init_preseed_certificates
  - custom_volume_sftp
  - network_ovn_external_nic_address
  - network_physical_gateway_hwaddr
  - backup_s3_upload
  - snapshot_manual_expiry
  - resources_cpu_address_sizes
  - disk_attached
  - limits_memory_hotplug
  - disk_wwn
  - server_logging_webhook
  - storage_driver_truenas
  - container_disk_tmpfs
  - instance_limits_oom
  - backup_override_config
  - network_ovn_tunnels
  - init_preseed_cluster_groups
  - usb_attached
  - backup_iso
  - instance_systemd_credentials
  - cluster_group_usedby
  - bpf_token_delegation
  - file_storage_volume
  - network_hwaddr_pattern
  - storage_volume_full
  - storage_bucket_full
  - device_pci_firmware
  - resources_serial
  - ovn_nic_limits
  - storage_lvmcluster_qcow2
  - oidc_allowed_subnets
  - file_delete_force
  - nic_sriov_select_ext
  - network_zones_dns_contact
  - nic_attached_connected
  - nic_sriov_security_trusted
  - direct_backup
  - instance_snapshot_disk_only_restore
  - unix_hotplug_pci
  - cluster_evacuating_restoring
  - projects_restricted_image_servers
  - storage_lvmcluster_size
  - authorization_scriptlet_cert
  - lvmcluster_remove_snapshots
  - daemon_storage_logs
  - instances_debug_repair
  - network_io_bus_ovn
  - dependent
  - metrics_project_resources
  - storage_volume_nbd
  - projects_restricted_storage_pool_access
  - server_shutdown_action
  - instances_placement_scriptlet_rebalance
  - network_bridge_multicast_snooping
  - storage_create_options
  - instances_tpm_platform_cert
  - linstor_raw
  - network_address_set_ip_ranges
  - storage_volumes_rebuild
  - api_fragments
  - instance_limits_cpu_topology
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
  - tls
auth_user_name: [redacted]
auth_user_method: tls
environment:
  addresses:
    - XXX.XX.XX.XXX:8443
    - '[XXXX:XXX:XXXX:XXX::XXX]:8443'
    - 10.10.0.12:8443
    - 10.20.0.12:8443
    - 10.30.0.12:8443
    - 10.7.15.1:8443
    - '[fd42:5a7d:cff3:18da::1]:8443'
    - '[fd42:287e:53a1:849e:1266:6aff:fe8a:6b21]:8443'
  architectures:
    - x86_64
    - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  certificate_fingerprint: [redacted]
  driver: lxc | qemu
  driver_version: 7.0.0 | 11.0.1
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features: {}
  kernel_version: 7.0.12-zabbly+
  lxc_features: {}
  os_name: IncusOS
  os_version: "202606110200"
  project: default
  server: incus
  server_clustered: true
  server_event_mode: full-mesh
  server_name: node1
  server_pid: 3008
  server_version: "7.1"
  storage: zfs | linstor
  storage_version: 2.4.2-1 | 1.33.3 / 9.3.2
  storage_supported_drivers:
    - name: btrfs
      version: "6.14"
      remote: false
    - name: lvmcluster
      version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0
      remote: true
    - name: zfs
      version: 2.4.2-1
      remote: false
    - name: dir
      version: "1"
      remote: false
    - name: lvm
      version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0
      remote: false
    - name: linstor
      version: 1.33.3 / 9.3.2
      remote: true

Instance details

No response

Instance log

No response

Current behavior

I reliably obtain the following error when restoring an evacuated Incus cluster node with a VM on the Linstor remote storage pool with live migration enabled:

Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message

Expected behavior

No response

Steps to reproduce

1. Setup a Linstor remote storage pool that uses LVM Thin backend just like in the tutorial
2. Create and start a test VM:

incus launch images:ubuntu/24.04 test-vm --vm -c migration.stateful=true --storage remote --target node1

3. Wait until the VM fully starts and live migrate it to another cluster node:

incus mv test-vm --target node2

4. Live migrate the VM to the original node:

incus mv test-vm --target node1

5. Observe the error:

Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message

6. Check the kernel logs of node1:

incus admin os debug log --target node1

7. Observe the EXT4-fs error:

[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Preparing cluster-wide state change 960940965: 1->all role( Primary )
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: State change 960940965: primary_nodes=6, weak_nodes=FFFFFFFFFFFFFFF8
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Committing cluster-wide state change 960940965 (0ms)
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: role( Secondary -> Primary ) [incusd:3055 auto-promote]
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs (drbd1003): mounted filesystem cf387762-ee19-4225-b7aa-ae0ed0c84f64 r/w with ordered data mode. Quota mode: none.
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs warning (device drbd1003): ext4_dirblock_csum_verify:375: inode #33: comm incusd: No space for directory leaf checksum. Please run e2fsck -D.
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs error (device drbd1003): ext4_empty_dir:3078: inode #33: comm incusd: Directory block failed checksum
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs warning (device drbd1003): ext4_dirblock_csum_verify:375: inode #33: comm incusd: No space for directory leaf checksum. Please run e2fsck -D.
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs error (device drbd1003): htree_dirblock_to_tree:1051: inode #33: comm incusd: Directory block failed checksum
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs warning (device drbd1003): ext4_dirblock_csum_verify:375: inode #33: comm incusd: No space for directory leaf checksum. Please run e2fsck -D.
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs error (device drbd1003): __ext4_find_entry:1624: inode #33: comm incusd: checksumming directory block 0
[2026/06/12 12:07:26 CEST] systemd: var-lib-incus-storage\x2dpools-remote-virtual\x2dmachines-test\x2dvm.mount: Deactivated successfully.
[2026/06/12 12:07:26 CEST] kernel: EXT4-fs (drbd1003): unmounting filesystem cf387762-ee19-4225-b7aa-ae0ed0c84f64.
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Preparing cluster-wide state change 2250084524: 1->all role( Secondary )
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: State change 2250084524: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF8
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Committing cluster-wide state change 2250084524 (1ms)
[2026/06/12 12:07:26 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: role( Primary -> Secondary ) [incusd:6714 auto-demote]
[2026/06/12 12:07:26 CEST] incusd: time="2026-06-12T12:07:26+02:00" level=error msg="Failed migration on target" clusterMoveSourceName=test-vm err="open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message" instance=test-vm live=true project=default push=false

Additional Info

I performed the file system check from a separate container, but the file system appears to be fine:

incus stop test-vm
incus launch images:ubuntu/24.04 diag-node1 --project default --target node1
incus config device add diag-node1 drbd1003 unix-block source=/dev/drbd1003 path=/dev/xvdb --project default
incus exec diag-node1 -- e2fsck -f -n /dev/xvdb

The output is:

e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvdb: 40/129024 files (0.0% non-contiguous), 12556/128987 blocks

So, perhaps, the problem is caused by some stale file system caches left when the VM was running on node1?

Also there is some strange message in the kernel log on node2 during the first migration:

[2026/06/12 12:07:14 CEST] kernel: EXT4-fs (drbd1003): recovery complete

Here is a more complete log from node2 during the first migration:

[2026/06/12 12:06:27 CEST] Satellite: 2026-06-12 12:06:27.390 [DeviceManager] INFO  LINSTOR/Satellite/722238 SYSTEM - Begin DeviceManager cycle 91
[2026/06/12 12:06:27 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Preparing remote state change 3753169887: 1->all role( Primary )
[2026/06/12 12:06:27 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d 4c4c4544-0044-5410-8033-b2c04f503034: Committing remote state change 3753169887 (primary_nodes=2)
[2026/06/12 12:06:27 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d 4c4c4544-0044-5410-8033-b2c04f503034: peer( Secondary -> Primary ) [remote]
[2026/06/12 12:06:27 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Preparing remote state change 94184223: 1->all role( Secondary )
[2026/06/12 12:06:27 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d 4c4c4544-0044-5410-8033-b2c04f503034: Committing remote state change 94184223 (primary_nodes=0)
[2026/06/12 12:06:27 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d 4c4c4544-0044-5410-8033-b2c04f503034: peer( Primary -> Secondary ) [remote]
[2026/06/12 12:06:28 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Preparing remote state change 4011869940: 1->all role( Primary )
[2026/06/12 12:06:28 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d 4c4c4544-0044-5410-8033-b2c04f503034: Committing remote state change 4011869940 (primary_nodes=2)
[2026/06/12 12:06:28 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d 4c4c4544-0044-5410-8033-b2c04f503034: peer( Secondary -> Primary ) [remote]
[2026/06/12 12:07:14 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Preparing cluster-wide state change 1391564923: 2->all role( Primary )
[2026/06/12 12:07:14 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: State change 1391564923: primary_nodes=6, weak_nodes=FFFFFFFFFFFFFFF8
[2026/06/12 12:07:14 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: Committing cluster-wide state change 1391564923 (1ms)
[2026/06/12 12:07:14 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d: role( Secondary -> Primary ) [incusd:3149 auto-promote]
[2026/06/12 12:07:14 CEST] kernel: drbd incus-volume-f6842afd662d487699ca4addf1e4a81d/1 drbd1003: sending new current UUID: 2061C4FB99D6DBC5
[2026/06/12 12:07:14 CEST] kernel: EXT4-fs (drbd1003): recovery complete
[2026/06/12 12:07:14 CEST] kernel: EXT4-fs (drbd1003): mounted filesystem cf387762-ee19-4225-b7aa-ae0ed0c84f64 r/w with ordered data mode. Quota mode: none.

[bensmrs]

I’m really confused, as no on-disk data is moved, so it’s really activating the disk on the other node before migrating runtime state, then deactivating it on the first one afterwards…
So maybe there are cases where both machines write to the activated disk, which surely could make EXT4 unhappy.

[ykazakov]

What is this file: /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml? I looked what is on the disk /dev/drbd1003 and it seems that it also has config/files/metadata.yaml. Could it be that the metadata is mounted on both hosts and both try to modify the metadata during live migration?

[ykazakov]

Indeed, looks like the disk /dev/drbd1003 is not a VM filesystem but its metadata:

.
├── OVMF_VARS.4MB.ms.fd
├── agent-client.crt
├── agent-client.key
├── agent.crt
├── agent.key
├── backup.yaml
├── config
│   ├── agent-mounts.json
│   ├── agent.conf
│   ├── agent.crt
│   ├── agent.key
│   ├── files
│   │   └── metadata.yaml
│   ├── incus-agent
│   ├── install.sh
│   ├── lxd-agent -> incus-agent
│   ├── nics
│   ├── server.crt
│   ├── systemd
│   │   ├── incus-agent-setup
│   │   └── incus-agent.service
│   └── udev
│       └── 99-incus-agent.rules
├── lost+found
├── metadata.yaml
├── qemu.nvram -> OVMF_VARS.4MB.ms.fd
└── templates
    ├── hostname.tpl
    └── hosts.tpl

8 directories, 22 files

[bensmrs]

Maybe we’re missing a flush somewhere, I’ll dig into that next week.

[ykazakov]

I think this is unsafe for EXT4 file systems and might be the cause for the issue here:

incus/internal/server/storage/drivers/driver_linstor_utils.go

Line 1027 in 60c8df3

"DrbdOptions/Net/allow-two-primaries": "yes", // Required for mounting volumes simultaneously on two nodes when live migrating

[bensmrs]

We kinda need that for live migration. As long as we can guarantee that there’s a single writer…
Although the metadata (i.e. the non-VM data stuff) could maybe work without, but I’m really not sure. Do you have an idea @luissimas?

[bensmrs]

Can you reproduce a corruption on an additional FS volume mounted on the instance? Trying to better identify the issue.

[ykazakov]

We kinda need that for live migration. As long as we can guarantee that there’s a single writer…

I think this is unsafe. The problem appears to be not the file system corruption (as I verified) but the cache rot (where the Kernel holds page caches that have changed on disk).

Google tells me that even read-only mounting of EXT4 file systems from different locations is unsafe.

Can you reproduce a corruption on an additional FS volume mounted on the instance?

I can try but I do not think there will be a difference. As mentioned, the problem is not with the VM file system but with the metadata volume.

[luissimas]

Yes DrbdOptions/Net/allow-two-primaries is required to enable live migration. This lifts much of the responsibility to Incus (and QEMU), but AFAIK is the only way to enable live migration of VMs with attached DRBD devices.

I think we could get away with not doing that for the metadata volume, but I'd have to test. In any case, this might be a more general issue. We treat the metadata volume much like any other filesystem volume, so this error might also happen with those.

I can also take a closer look at this next week and try out some other combinations of volumes and migrations to see if I can pin point at which layer this is happening.

@ykazakov this only happens when moving the VM back to the previous node, correct? Did you by any chance tried this with a 3-node setup and a place_count=3? Something like node1->node2, node2->node3 and finally node3->node1.

[ykazakov]

Did you by any chance tried this with a 3-node setup and a place_count=3?

I just tried with 3 nodes (no change in place_count). The move fails already for node2->node3, so no roundtrip is necessary:

% incus launch images:ubuntu/24.04 test-vm --vm -c migration.stateful=true --storage remote --target node1
% incus mv test-vm --target node2
% incus mv test-vm --target node3
Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message

Logs on the nodes:

node1

[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 2008908579: 3->all role( Primary )
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Duplicate prepare for remote state change 2008908579
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Committing remote state change 2008908579 (primary_nodes=C)
[2026/06/13 16:12:23 CEST] kernel: incusbr0: port 1(tapb8093e34) entered disabled state
[2026/06/13 16:12:23 CEST] systemd-networkd: tapb8093e34: Lost carrier
[2026/06/13 16:12:23 CEST] systemd: var-lib-incus-devices-test\x2dvm-config.mount.mount: Deactivated successfully.
[2026/06/13 16:12:24 CEST] kernel: tapb8093e34: left allmulticast mode
[2026/06/13 16:12:24 CEST] kernel: tapb8093e34: left promiscuous mode
[2026/06/13 16:12:24 CEST] kernel: incusbr0: port 1(tapb8093e34) entered disabled state
[2026/06/13 16:12:24 CEST] kernel: audit: type=1700 audit(1781359944.018:2666): dev=tapb8093e34 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[2026/06/13 16:12:24 CEST] systemd-networkd: tapb8093e34: Link UP
[2026/06/13 16:12:24 CEST] kernel: audit: type=1300 audit(1781359944.018:2666): arch=c000003e syscall=44 success=yes exit=40 a0=29 a1=549289bcc90 a2=28 a3=0 items=0 ppid=1 pid=2862 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="incusd" exe="/opt/incus/bin/incusd" subj=unconfined key=(null)
[2026/06/13 16:12:24 CEST] kernel: audit: type=1327 audit(1781359944.018:2666): proctitle=696E63757364002D2D67726F757000696E6375732D61646D696E002D2D6C6F6766696C65
[2026/06/13 16:12:24 CEST] systemd-networkd: tapb8093e34: Link DOWN
[2026/06/13 16:12:24 CEST] kernel: EXT4-fs (drbd1001): unmounting filesystem aef6d332-520f-406e-a890-13adbc42d9f9.
[2026/06/13 16:12:24 CEST] systemd: var-lib-incus-storage\x2dpools-remote-virtual\x2dmachines-test\x2dvm.mount: Deactivated successfully.
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing cluster-wide state change 1012653420: 2->all role( Secondary )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: State change 1012653420: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Committing cluster-wide state change 1012653420 (1ms)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: role( Primary -> Secondary ) [incusd:155151 auto-demote]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing cluster-wide state change 3792571131: 2->0 conn( Disconnecting )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: State change 3792571131: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Committing cluster-wide state change 3792571131 (1ms)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/0 drbd1000 4c4c4544-0043-5410-8033-c8c04f503034: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/1 drbd1001 4c4c4544-0043-5410-8033-c8c04f503034: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: Starting sender thread (peer-node-id 0)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: Connection closed
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: helper command: /sbin/drbdadm disconnected
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: helper command: /sbin/drbdadm disconnected exit code 0
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: conn( Disconnecting -> StandAlone ) [disconnected]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: Terminating receiver thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing cluster-wide state change 2799660704: 2->1 conn( Disconnecting )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: State change 2799660704: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Cluster is now split
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Committing cluster-wide state change 2799660704 (1ms)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: susp-io( no -> quorum ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/0 drbd1000: quorum( yes -> no ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/0 drbd1000 4c4c4544-0044-5410-8033-b1c04f503034: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/1 drbd1001: quorum( yes -> no ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/1 drbd1001 4c4c4544-0044-5410-8033-b1c04f503034: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [down]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Starting sender thread (peer-node-id 1)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Connection closed
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: helper command: /sbin/drbdadm disconnected
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: helper command: /sbin/drbdadm disconnected exit code 0
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: conn( Disconnecting -> StandAlone ) [disconnected]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Terminating receiver thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] 55-scsi-sg3_id.rules: WARNING: SCSI device drbd1000 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2026/06/13 16:12:24 CEST] 55-scsi-sg3_id.rules: WARNING: SCSI device drbd1001 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.225 [DeviceManager] INFO  LINSTOR/Satellite/5f98f8 SYSTEM - Resource 'incus-volume-bf13309b7b5b433b9b52a70793eaf0c4' [DRBD] deleted.
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.227 [DeviceManager] INFO  LINSTOR/Satellite/ SYSTEM - End DeviceManager cycle 286
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.227 [DeviceManager] INFO  LINSTOR/Satellite/a85f5f SYSTEM - Begin DeviceManager cycle 287
[2026/06/13 16:12:24 CEST] kernel: drbd /unregistered/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Terminating worker thread
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.393 [DeviceManager] INFO  LINSTOR/Satellite/ SYSTEM - End DeviceManager cycle 287
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.393 [DeviceManager] INFO  LINSTOR/Satellite/15a34c SYSTEM - Begin DeviceManager cycle 288
[2026/06/13 16:12:24 CEST] kernel: audit: type=1400 audit(1781359944.571:2667): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-test-vm_</var/lib/incus>" pid=501077 comm="apparmor_parser"
[2026/06/13 16:12:24 CEST] kernel: audit: type=1300 audit(1781359944.571:2667): arch=c000003e syscall=1 success=yes exit=31 a0=4 a1=55cfbb7df730 a2=1f a3=0 items=0 ppid=501076 pid=501077 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="apparmor_parser" exe="/usr/sbin/apparmor_parser" subj=unconfined key=(null)
[2026/06/13 16:12:24 CEST] kernel: audit: type=1327 audit(1781359944.571:2667): proctitle=61707061726D6F725F706172736572002D52574C002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F6361636865002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F70726F66696C65732F696E6375732D746573742D766D
[2026/06/13 16:12:24 CEST] systemd-networkd: incusbr0: Lost carrier

node2

[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing cluster-wide state change 2008908579: 3->all role( Primary )
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: State change 2008908579: primary_nodes=C, weak_nodes=FFFFFFFFFFFFFFFC
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Committing cluster-wide state change 2008908579 (1ms)
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: role( Secondary -> Primary ) [incusd:4026 auto-promote]
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/1 drbd1001: sending new current UUID: 9D84E1B2F4ADEA61
[2026/06/13 16:12:19 CEST] kernel: EXT4-fs (drbd1001): recovery complete
[2026/06/13 16:12:19 CEST] kernel: EXT4-fs (drbd1001): mounted filesystem aef6d332-520f-406e-a890-13adbc42d9f9 r/w with ordered data mode. Quota mode: none.
[2026/06/13 16:12:19 CEST] systemd-networkd: tapc18cb113: Link UP
[2026/06/13 16:12:19 CEST] kernel: incusbr0: port 2(tapc18cb113) entered blocking state
[2026/06/13 16:12:19 CEST] kernel: incusbr0: port 2(tapc18cb113) entered disabled state
[2026/06/13 16:12:19 CEST] kernel: tapc18cb113: entered allmulticast mode
[2026/06/13 16:12:19 CEST] kernel: tapc18cb113: entered promiscuous mode
[2026/06/13 16:12:19 CEST] kernel: audit: type=1700 audit(1781359939.962:2667): dev=tapc18cb113 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[2026/06/13 16:12:19 CEST] kernel: audit: type=1300 audit(1781359939.962:2667): arch=c000003e syscall=44 success=yes exit=40 a0=74 a1=1a8d03c8eae0 a2=28 a3=0 items=0 ppid=1 pid=3008 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="incusd" exe="/opt/incus/bin/incusd" subj=unconfined key=(null)
[2026/06/13 16:12:19 CEST] kernel: audit: type=1327 audit(1781359939.962:2667): proctitle=696E63757364002D2D67726F757000696E6375732D61646D696E002D2D6C6F6766696C65
[2026/06/13 16:12:20 CEST] kernel: audit: type=1400 audit(1781359940.049:2668): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus-test-vm_</var/lib/incus>" pid=145995 comm="apparmor_parser"
[2026/06/13 16:12:20 CEST] kernel: audit: type=1300 audit(1781359940.049:2668): arch=c000003e syscall=1 success=yes exit=314337 a0=6 a1=56191f25eb30 a2=4cbe1 a3=0 items=0 ppid=145994 pid=145995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="apparmor_parser" exe="/usr/sbin/apparmor_parser" subj=unconfined key=(null)
[2026/06/13 16:12:20 CEST] kernel: audit: type=1327 audit(1781359940.049:2668): proctitle=61707061726D6F725F706172736572002D72574C002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F6361636865002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F70726F66696C65732F696E6375732D746573742D766D
[2026/06/13 16:12:20 CEST] systemd-networkd: tapc18cb113: Gained carrier
[2026/06/13 16:12:20 CEST] systemd-networkd: incusbr0: Gained carrier
[2026/06/13 16:12:20 CEST] kernel: incusbr0: port 2(tapc18cb113) entered blocking state
[2026/06/13 16:12:20 CEST] kernel: incusbr0: port 2(tapc18cb113) entered forwarding state
[2026/06/13 16:12:20 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/0 drbd1000: sending new current UUID: 2D7C9BA209237723
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 1012653420: 2->all role( Secondary )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Duplicate prepare for remote state change 1012653420
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-c8c04f503034: Committing remote state change 1012653420 (primary_nodes=8)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Ignoring redundant P_TWOPC_COMMIT packet 1012653420.
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 3792571131: 2->0 conn( Disconnecting )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Duplicate prepare for remote state change 3792571131
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Committing remote state change 3792571131 (primary_nodes=8)
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.144 [DeviceManager] INFO  LINSTOR/Satellite/19b90c SYSTEM - Resource 'incus-volume-bf13309b7b5b433b9b52a70793eaf0c4' [DRBD] adjusted.
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 2799660704: 2->1 conn( Disconnecting )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Committing remote state change 2799660704 (primary_nodes=8)
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.167 [DeviceManager] INFO  LINSTOR/Satellite/ SYSTEM - End DeviceManager cycle 183
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.167 [DeviceManager] INFO  LINSTOR/Satellite/2491e6 SYSTEM - Begin DeviceManager cycle 184
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.385 [DeviceManager] INFO  LINSTOR/Satellite/2491e6 SYSTEM - DRBD regenerated resource file: /var/lib/linstor.d/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4.res
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.407 [DeviceManager] INFO  LINSTOR/Satellite/2491e6 SYSTEM - Resource 'incus-volume-bf13309b7b5b433b9b52a70793eaf0c4' [DRBD] adjusted.
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.428 [DeviceManager] INFO  LINSTOR/Satellite/ SYSTEM - End DeviceManager cycle 184
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.428 [DeviceManager] INFO  LINSTOR/Satellite/eafbba SYSTEM - Begin DeviceManager cycle 185
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 1750603182: 1->all role( Primary )
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Committing remote state change 1750603182 (primary_nodes=A)
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: peer( Secondary -> Primary ) [remote]
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 2215496607: 1->all role( Secondary )
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: Committing remote state change 2215496607 (primary_nodes=8)
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b1c04f503034: peer( Primary -> Secondary ) [remote]
[2026/06/13 16:12:31 CEST] incusd: time="2026-06-13T16:12:31+02:00" level=error msg="Failed migration on source" clusterMoveSourceName=test-vm err="Error from migration control target: open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message" instance=test-vm live=true project=default push=false

node3

[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 2008908579: 3->all role( Primary )
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b2c04f503034: Committing remote state change 2008908579 (primary_nodes=C)
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0044-5410-8033-b2c04f503034: peer( Secondary -> Primary ) [remote]
[2026/06/13 16:12:19 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/1 drbd1001 4c4c4544-0044-5410-8033-b2c04f503034: received new current UUID: 9D84E1B2F4ADEA61 weak_nodes=FFFFFFFFFFFFFFFC
[2026/06/13 16:12:20 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/0 drbd1000 4c4c4544-0044-5410-8033-b2c04f503034: received new current UUID: 2D7C9BA209237723 weak_nodes=FFFFFFFFFFFFFFFC
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 1012653420: 2->all role( Secondary )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Committing remote state change 1012653420 (primary_nodes=8)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: peer( Primary -> Secondary ) [remote]
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.108 [DeviceManager] INFO  LINSTOR/Satellite/856d92 SYSTEM - Aligning /dev/linstor_bcache_pool/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4_00000 size from 10488040 KiB to 10489856 KiB to be a multiple of extent size 4096 KiB (from Storage Pool)
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.108 [DeviceManager] INFO  LINSTOR/Satellite/856d92 SYSTEM - Aligning /dev/linstor_bcache_pool/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4_00001 size from 512148 KiB to 516096 KiB to be a multiple of extent size 4096 KiB (from Storage Pool)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 3792571131: 2->0 conn( Disconnecting )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Committing remote state change 3792571131 (primary_nodes=8)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing remote state change 2799660704: 2->1 conn( Disconnecting )
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Committing remote state change 2799660704 (primary_nodes=8)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Connected -> TearDown ) peer( Secondary -> Unknown ) [remote]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/0 drbd1000 4c4c4544-0043-5410-8033-cac04f503034: pdsk( Diskless -> DUnknown ) repl( Established -> Off ) [remote]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4/1 drbd1001 4c4c4544-0043-5410-8033-cac04f503034: pdsk( Diskless -> DUnknown ) repl( Established -> Off ) [remote]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Starting sender thread (peer-node-id 2)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Connection closed
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: helper command: /sbin/drbdadm disconnected
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: helper command: /sbin/drbdadm disconnected exit code 0
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( TearDown -> Unconnected ) [disconnected]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Restarting receiver thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Unconnected -> Connecting ) [connecting]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Connecting -> Disconnecting ) [del-peer]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Starting sender thread (peer-node-id 2)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Connection closed
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: helper command: /sbin/drbdadm disconnected
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: helper command: /sbin/drbdadm disconnected exit code 0
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Disconnecting -> StandAlone ) [disconnected]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating receiver thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Starting sender thread (peer-node-id 2)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( StandAlone -> Unconnected ) [connect]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Starting receiver thread (peer-node-id 2)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Unconnected -> Connecting ) [connecting]
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.298 [DeviceManager] INFO  LINSTOR/Satellite/856d92 SYSTEM - Resource 'incus-volume-bf13309b7b5b433b9b52a70793eaf0c4' [DRBD] adjusted.
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.358 [DeviceManager] INFO  LINSTOR/Satellite/ SYSTEM - End DeviceManager cycle 171
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.358 [DeviceManager] INFO  LINSTOR/Satellite/cacb34 SYSTEM - Begin DeviceManager cycle 172
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.379 [DeviceManager] INFO  LINSTOR/Satellite/cacb34 SYSTEM - Aligning /dev/linstor_bcache_pool/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4_00000 size from 10488040 KiB to 10489856 KiB to be a multiple of extent size 4096 KiB (from Storage Pool)
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.379 [DeviceManager] INFO  LINSTOR/Satellite/cacb34 SYSTEM - Aligning /dev/linstor_bcache_pool/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4_00001 size from 512148 KiB to 516096 KiB to be a multiple of extent size 4096 KiB (from Storage Pool)
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.393 [DeviceManager] INFO  LINSTOR/Satellite/cacb34 SYSTEM - DRBD regenerated resource file: /var/lib/linstor.d/incus-volume-bf13309b7b5b433b9b52a70793eaf0c4.res
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Connecting -> Disconnecting ) [del-peer]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Starting sender thread (peer-node-id 2)
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Connection closed
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: helper command: /sbin/drbdadm disconnected
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: helper command: /sbin/drbdadm disconnected exit code 0
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: conn( Disconnecting -> StandAlone ) [disconnected]
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating receiver thread
[2026/06/13 16:12:24 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 4c4c4544-0043-5410-8033-cac04f503034: Terminating sender thread
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.462 [DeviceManager] INFO  LINSTOR/Satellite/cacb34 SYSTEM - Resource 'incus-volume-bf13309b7b5b433b9b52a70793eaf0c4' [DRBD] adjusted.
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.485 [DeviceManager] INFO  LINSTOR/Satellite/ SYSTEM - End DeviceManager cycle 172
[2026/06/13 16:12:24 CEST] Satellite: 2026-06-13 16:12:24.485 [DeviceManager] INFO  LINSTOR/Satellite/725aca SYSTEM - Begin DeviceManager cycle 173
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing cluster-wide state change 1750603182: 1->all role( Primary )
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: State change 1750603182: primary_nodes=A, weak_nodes=FFFFFFFFFFFFFFF4
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Committing cluster-wide state change 1750603182 (1ms)
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: role( Secondary -> Primary ) [incusd:3140 auto-promote]
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs (drbd1001): mounted filesystem aef6d332-520f-406e-a890-13adbc42d9f9 r/w with ordered data mode. Quota mode: none.
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs warning (device drbd1001): ext4_dirblock_csum_verify:375: inode #33: comm incusd: No space for directory leaf checksum. Please run e2fsck -D.
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs error (device drbd1001): ext4_empty_dir:3078: inode #33: comm incusd: Directory block failed checksum
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs warning (device drbd1001): ext4_dirblock_csum_verify:375: inode #33: comm incusd: No space for directory leaf checksum. Please run e2fsck -D.
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs error (device drbd1001): htree_dirblock_to_tree:1051: inode #33: comm incusd: Directory block failed checksum
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs warning (device drbd1001): ext4_dirblock_csum_verify:375: inode #33: comm incusd: No space for directory leaf checksum. Please run e2fsck -D.
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs error (device drbd1001): __ext4_find_entry:1624: inode #33: comm incusd: checksumming directory block 0
[2026/06/13 16:12:30 CEST] systemd: var-lib-incus-storage\x2dpools-remote-virtual\x2dmachines-test\x2dvm.mount: Deactivated successfully.
[2026/06/13 16:12:30 CEST] kernel: EXT4-fs (drbd1001): unmounting filesystem aef6d332-520f-406e-a890-13adbc42d9f9.
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Preparing cluster-wide state change 2215496607: 1->all role( Secondary )
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: State change 2215496607: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: Committing cluster-wide state change 2215496607 (1ms)
[2026/06/13 16:12:30 CEST] kernel: drbd incus-volume-bf13309b7b5b433b9b52a70793eaf0c4: role( Primary -> Secondary ) [incusd:154173 auto-demote]
[2026/06/13 16:12:30 CEST] incusd: time="2026-06-13T16:12:30+02:00" level=error msg="Failed migration on target" clusterMoveSourceName=test-vm err="open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message" instance=test-vm live=true project=default push=false

Linstor resources and volumes

# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                                  ┊ Node  ┊ Layers       ┊ Usage  ┊ Conns ┊    State ┊ Vote ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node2 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊ Diskless ┊ Yes  ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node4 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ Diskless ┊ Yes  ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node2 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ Diskless ┊ Yes  ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node4 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
# linstor r lv 
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Resource                                      ┊ Node  ┊ StoragePool          ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊  Allocated ┊ InUse  ┊    State ┊ Repl           ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node2 ┊ DfltDisklessStorPool ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊            ┊ InUse  ┊ Diskless ┊ Established(2) ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node2 ┊ DfltDisklessStorPool ┊     1 ┊    1001 ┊ /dev/drbd1001 ┊            ┊ InUse  ┊ Diskless ┊ Established(2) ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node4 ┊ bcache_pool          ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊   1.19 GiB ┊ Unused ┊ UpToDate ┊ Established(2) ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node4 ┊ bcache_pool          ┊     1 ┊    1001 ┊ /dev/drbd1001 ┊ 128.02 MiB ┊ Unused ┊ UpToDate ┊ Established(2) ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node3 ┊ bcache_pool          ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊   1.19 GiB ┊ Unused ┊ UpToDate ┊ Established(2) ┊
┊ incus-volume-bf13309b7b5b433b9b52a70793eaf0c4 ┊ node3 ┊ bcache_pool          ┊     1 ┊    1001 ┊ /dev/drbd1001 ┊ 128.02 MiB ┊ Unused ┊ UpToDate ┊ Established(2) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node1 ┊ DfltDisklessStorPool ┊     0 ┊    1004 ┊ /dev/drbd1004 ┊            ┊ Unused ┊ Diskless ┊ Established(2) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node1 ┊ DfltDisklessStorPool ┊     1 ┊    1005 ┊ /dev/drbd1005 ┊            ┊ Unused ┊ Diskless ┊ Established(2) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node2 ┊ DfltDisklessStorPool ┊     0 ┊    1004 ┊ /dev/drbd1004 ┊            ┊ Unused ┊ Diskless ┊ Established(2) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node2 ┊ DfltDisklessStorPool ┊     1 ┊    1005 ┊ /dev/drbd1005 ┊            ┊ Unused ┊ Diskless ┊ Established(2) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node4 ┊ bcache_pool          ┊     0 ┊    1004 ┊ /dev/drbd1004 ┊ 983.42 MiB ┊ Unused ┊ UpToDate ┊ Established(3) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node4 ┊ bcache_pool          ┊     1 ┊    1005 ┊ /dev/drbd1005 ┊  79.98 MiB ┊ Unused ┊ UpToDate ┊ Established(3) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node3 ┊ bcache_pool          ┊     0 ┊    1004 ┊ /dev/drbd1004 ┊ 983.42 MiB ┊ Unused ┊ UpToDate ┊ Established(3) ┊
┊ incus-volume-d6b8185a1274406e87035263c551e14f ┊ node3 ┊ bcache_pool          ┊     1 ┊    1005 ┊ /dev/drbd1005 ┊  79.98 MiB ┊ Unused ┊ UpToDate ┊ Established(3) ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[ykazakov]

We treat the metadata volume much like any other filesystem volume, so this error might also happen with those.

Do other filesystem volumes get mounted on the host during migration or they are treaded like block devices and are simply passed to QEMU? If the file systems are not mounted on the host (or they are safe for multiple mounts), I think, there should be no danger of cache rot.

[ykazakov]

Did you by any chance tried this with a 3-node setup and a place_count=3?

I now tried with place_count=3 and result was the same:

% incus storage set remote linstor.resource_group.place_count=3
% incus launch images:ubuntu/24.04 test-vm --vm -c migration.stateful=true --storage remote --target node1
Launching test-vm
% incus mv test-vm --target node3
% incus mv test-vm --target node4
Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: open /var/lib/incus/virtual-machines/test-vm/config/files/metadata.yaml: bad message 

I used node3 and node4 because these were the nodes with resource replicas as reported by linstor-controller.

# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                                  ┊ Node  ┊ Layers       ┊ Usage  ┊ Conns ┊    State ┊ Vote ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-8ba6ea96450d48ee9baf29e05adc2346 ┊ node1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-8ba6ea96450d48ee9baf29e05adc2346 ┊ node2 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ Diskless ┊ Yes  ┊
┊ incus-volume-8ba6ea96450d48ee9baf29e05adc2346 ┊ node3 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-8ba6ea96450d48ee9baf29e05adc2346 ┊ node4 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-92ad2644f7e84af3bdc8008384bda4a6 ┊ node1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-92ad2644f7e84af3bdc8008384bda4a6 ┊ node2 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ Diskless ┊ Yes  ┊
┊ incus-volume-92ad2644f7e84af3bdc8008384bda4a6 ┊ node3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
┊ incus-volume-92ad2644f7e84af3bdc8008384bda4a6 ┊ node4 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ Yes  ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[ykazakov]

I have a hypothesis about what is going on.
When I launch a fresh VM and mount its metadata volume on another node, the file system check reports that the file system is corrupted:

% incus launch images:ubuntu/24.04 test-vm --vm -c migration.stateful=true --storage remote --target node1
Launching test-vm
% incus launch images:ubuntu/24.04 diag-node3 --project default --target node3
Launching diag-node1
% incus config device add diag-node3 drbd1001 unix-block source=/dev/drbd1001 path=/dev/xvdb --project default

Device drbd1001 added to diag-node3
% incus exec diag-node3 -- e2fsck -f -n /dev/xvdb
e2fsck 1.47.0 (5-Feb-2023)
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (116584, counted=116430).
Fix? no

Free inodes count wrong (129007, counted=128982).
Fix? no

Feature orphan_present is set but orphan file is clean.
Clear? no


/dev/xvdb: ********** WARNING: Filesystem still has errors **********

/dev/xvdb: 17/129024 files (0.0% non-contiguous), 12403/128987 blocks

I think, the same happens during the live migration:

1. the destination node attempts to mount the metadata filesystem,
2. it notices that the file system is corrupted (because some changed data still sits in caches of the initial node),
3. it "repairs" the file system. That is why we see the message:

[2026/06/13 16:12:19 CEST] kernel: EXT4-fs (drbd1001): recovery complete

4. the initial node flushes the remaining changes to the disk before unmounting the volume

Result: the data on the disk is modified from 2 different nodes and could no longer be automatically repaired upon the second move.