DPE-2959 async replication (#352)

* poc k8s-k8s async repl

* isort had mismatched line_length config

* (WIP) improved support

* (WIP) support more cases

* scale up/down for repl clusters
* idle state for 2nds
* instance label when creating repl cluster
* more typing

* (wip) replica secondaries support

* allow replica secondaries to rejoin

* (WIP) sync cluster-set-name and better deal with repl 2ndaries

* (WIP) fix secondaries join and messaging

* fix: removed unneeded flag

* support for relation broken

* dissolved replica cluster stays blocked
* added the promote standby action
* support for cluster set name config
* minor refactors and typing

* async common methods, single module/file

* removed dup action and lint fixes

* check for user data on replica side

* moved async_replication to a library (owned by vm charm)

* sync with vm code

* lint fixes

* scale-in locks write to global primary

* address pr comments

* second batch of PR comments adressing

* fence/unfence actions

* fix for single unit replica cluster

* workaround: secrets in relation data

* partial unit test fixes

* normalized cluster name for dict reference

* remove test due to be refactored

* allows rejoin after unrelate

* fix lock instance reference

* allow unrelated cluster rejoin

* reset cluster-set name

* dealing with secret not found

* ensure secrets are shared with full uri

* fix recreation and unrelation after promotion

* ensure all online unis

* unset read only after unfence

* refactor remove instance to acommodate changing

* fix race condition

* automatic deal with clusters with the same name

* bump

* test for mysql version and cluster-set-name on rejoin

* avoid handling on unit removal

* sync local root

* use method from lib

* lint fixes

* addressing pr feedback

* using node mode instead of role

* covering edge cases for recovery/failover

* add rejoin invalidated cluster action

* the integration test

* workaround for issue #399

* bump libpatch

* add group marks

* missing default

* set flag to avoid lock release

* chore: fixes old issue on test

* fix markers

* fix marker import

* fix retry process on removal

* fix user creation after refactor

`root@%` created only when required.

* lint/grammar fixes

* pr comment

Author: paulomach

actions.yaml

@@ -2,7 +2,13 @@
 # See LICENSE file for licensing details.
 
 get-cluster-status:
-  description: Get cluster status information without topology
+  description: Get cluster status information
+  params:
+    cluster-set:
+      type: boolean
+      default: False
+      description: Whether to fetch the cluster or cluster-set status.
+        Possible values are False (default) or True.
 
 get-password:
   description: Fetch the system user's password, which is used by charm.
@@ -28,7 +34,7 @@ set-password:
 set-tls-private-key:
   description:
     Set the privates key, which will be used for certificate signing requests (CSR). Run
-      for each unit separately.
+    for each unit separately.
   params:
     internal-key:
       type: string
@@ -55,3 +61,54 @@ pre-upgrade-check:
 
 resume-upgrade:
   description: Resume a rolling upgrade after asserting successful upgrade of a new revision.
+
+promote-standby-cluster:
+  description: |
+    Promotes this cluster to become the leader in the cluster-set. Used for safe switchover or failover.
+    Must be run against the charm leader unit of a standby cluster.
+  params:
+    cluster-set-name:
+      type: string
+      description: |
+        The name of the cluster-set. Mandatory option, used for confirmation.
+    force:
+      type: boolean
+      default: False
+      description: |
+        Use force when previous primary is unreachable (failover). Will invalidate previous
+        primary.
+
+recreate-cluster:
+  description: |
+    Recreates cluster on one or more standalone units that were previously part of a standby cluster.
+
+    When a standby cluster is removed from an async replication relation, the cluster will be dissolved and
+    each unit will be kept in blocked status. Recreating the cluster allows to rejoin the async replication
+    relation, or usage as a standalone cluster.
+
+fence-writes:
+  description: |
+    Stops write traffic to a primary cluster of a ClusterSet.
+  params:
+    cluster-set-name:
+      type: string
+      description: |
+        The name of the cluster-set. Mandatory option, used for confirmation.
+
+unfence-writes:
+  description: |
+    Resumes write traffic to a primary cluster of a ClusterSet.
+  params:
+    cluster-set-name:
+      type: string
+      description: |
+        The name of the cluster-set. Mandatory option, used for confirmation.
+
+rejoin-cluster:
+  description: |
+    Rejoins an invalidated cluster to the cluster-set, after a previous failover or switchover.
+  params:
+    cluster-name:
+      type: string
+      description: |
+        The name of the cluster to be rejoined.

config.yaml

@@ -3,7 +3,12 @@
 
 options:
   cluster-name:
-    description: "Optional - Name of the MySQL InnoDB cluster"
+    description: "Optional - Name of the MySQL InnoDB cluster, set once at deployment"
+    type: "string"
+  cluster-set-name:
+    description: |
+      Optional - Name for async replication cluster set, set once at deployment.
+      On `recreate-clster` action call, the cluster set name will be re-generated automatically.
     type: "string"
   profile:
     description: |

lib/charms/data_platform_libs/v0/data_secrets.py

@@ -97,7 +97,7 @@ def get_content(self) -> Dict[str, str]:
         """Getting cached secret content."""
         if not self._secret_content:
             if self.meta:
-                self._secret_content = self.meta.get_content()
+                self._secret_content = self.meta.get_content(refresh=True)
         return self._secret_content
 
     def set_content(self, content: Dict[str, str]) -> None: