hopefully finishing touches on kubernetes

bradymiller · bradymiller · commit 04dd9b22c0f2 · 2023-01-13T00:44:30.000-08:00
diff --git a/kubernetes/README.md b/kubernetes/README.md
@@ -5,14 +5,10 @@ OpenEMR Kubernetes orchestration. Orchestration included OpenEMR, MariaDB, Redis
   - Redis - Configured to support failover. There is 1 master and 2 slaves (no read access on slaves) for a statefulset, 3 sentinels for another statefulset, and then 2 proxies deployment. The proxies ensure that redis traffic is always directed towards master. The proxy replications can be increased/decreased. However the primary/slaves and sentinels would require script changes if wish to increase/decrease replicates for these since these are hard-coded several place in the scripts. There are 3 users/passwords (`default` (nopass), `replication` (replicationpassword), `admin` (adminpassword)) used in this redis scheme, and the passwords should be set to something else if use this scheme in production. The main place the passwords are set is in kubernetes/redis/configmap-acl.yaml script. Other places where passwords are used include the following: `replication` in kubernetes/redis/configmap-main.yaml, `admin` in kubernetes/redis/configmap-pipy.yaml, `admin` in kubernetes/redis/statefulset-sentinel.yaml. The `default` is the typical worker/app/client user, which will plan to assign a password when OpenEMR docker is updated to support redis username/password.
   - phpMyAdmin - There is 1 deployment instance of phpMyAdmin.
 
-Would not consider this production quality, but will be a good working, starting point, and hopefully open the door to a myriad of other kubernetes based solutions. Note this is supported by 6.0.0 and higher dockers. If wish to use the most recent development codebase, then can change from openemr/openemr:7.0.0 to openemr/openemr:flex in the openemr/deployment.yaml script (note this will take much longer to start up (probably at least 10 minutes and up to 90 minutes) and is more cpu intensive since each instance of OpenEMR will download codebase and build separately).
+Would not consider this production quality, but will be a good working, starting point, and hopefully open the door to a myriad of other kubernetes based solutions. Note this is supported by 7.0.0 and higher dockers. If wish to use the most recent development codebase, then can change from openemr/openemr:7.0.0 to openemr/openemr:dev (in the openemr/deployment.yaml script), which is built nightly from the development codebase. If you wish to build dynamically from a branch/tag from a github repo or other git repo, then can change from openemr/openemr:7.0.0 to openemr/openemr:flex (in the openemr/deployment.yaml script) (note this will take much longer to start up (probably at least 10 minutes and up to 90 minutes) and is more cpu intensive since each instance of OpenEMR will download codebase and build separately).
 
 (Quick note: Development in progress, minikube or kind not required for deployment. :8080 for http, :8090 for https, grab the NodePort for phpmyadmin)
 
-You should drop down to one OpenEMR instance-node before trying to pull in an updated image.
-
-TODO (optimizing some things). Add support for redis password in the OpenEMR dockers; then add password to the default user in the redis acl (it is now set to work without a password); then turn on protected mode in redis config.
-
 # Use
 1. Install (and then start) Kubernetes with Minikube or Kind or other.
     - For Minikube or other, can find online documentation.
@@ -39,36 +35,53 @@ TODO (optimizing some things). Add support for redis password in the OpenEMR doc
     ```bash
     bash kub-up
     ```
-3. Can see pod progress with following command:
-    ```bash
-    kubectl get pod
-    ```
-      - It will look something like this:
-          ```console
-          NAME                          READY   STATUS    RESTARTS   AGE
-          mysql-565f988976-np9zs        1/1     Running   0          133m
-          openemr-5f6db6c87c-8xgzq      1/1     Running   0          133m
-          openemr-5f6db6c87c-bfktf      1/1     Running   0          133m
-          openemr-5f6db6c87c-bwzdr      1/1     Running   0          133m
-          openemr-5f6db6c87c-qn5ll      1/1     Running   0          133m
-          openemr-5f6db6c87c-znq8h      1/1     Running   0          133m
-          phpmyadmin-78968d6cfb-cdfmq   1/1     Running   0          133m
-          redis-74cc9d667-5ltbq         1/1     Running   0          133m
-          ```
-4. Can see the service listing with following command:
+3. Can see overall progress with following command:
     ```bash
-    kubectl get svc
+    kubectl get all
     ```
-      - It will look something like this:
+      - It will look something like this when completed:
           ```console
-          NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
-          kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP                         151m
-          mysql        ClusterIP   10.109.255.180   <none>        3306/TCP                        147m
-          openemr      NodePort    10.104.48.53     <none>        8080:31314/TCP,8090:30613/TCP   147m
-          phpmyadmin   NodePort    10.97.77.18      <none>        8081:30571/TCP                  147m
-          redis        ClusterIP   10.98.24.148     <none>        6379/TCP                        147m
+          NAME                              READY   STATUS    RESTARTS   AGE
+          pod/mysql-sts-0                   1/1     Running   0          111s
+          pod/mysql-sts-1                   1/1     Running   0          91s
+          pod/openemr-7889cf48d8-9jdfl      1/1     Running   0          111s
+          pod/openemr-7889cf48d8-qphrw      1/1     Running   0          111s
+          pod/openemr-7889cf48d8-zlx9f      1/1     Running   0          111s
+          pod/phpmyadmin-f4d9bfc69-rx82d    1/1     Running   0          111s
+          pod/redis-0                       1/1     Running   0          111s
+          pod/redis-1                       1/1     Running   0          77s
+          pod/redis-2                       1/1     Running   0          55s
+          pod/redisproxy-744b7749dc-c6pkw   1/1     Running   0          111s
+          pod/redisproxy-744b7749dc-k8rzp   1/1     Running   0          111s
+          pod/sentinel-0                    1/1     Running   0          111s
+          pod/sentinel-1                    1/1     Running   0          34s
+          pod/sentinel-2                    1/1     Running   0          30s
+
+          NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                         AGE
+          service/kubernetes   ClusterIP      10.96.0.1      <none>        443/TCP                         3m40s
+          service/mysql        ClusterIP      None           <none>        3306/TCP                        111s
+          service/openemr      LoadBalancer   10.96.6.51     <pending>     8080:32561/TCP,8090:32468/TCP   111s
+          service/phpmyadmin   NodePort       10.96.64.163   <none>        8081:31466/TCP                  111s
+          service/redis        ClusterIP      None           <none>        6379/TCP                        111s
+          service/redisproxy   ClusterIP      None           <none>        6379/TCP                        111s
+          service/sentinel     ClusterIP      None           <none>        5000/TCP                        111s
+
+          NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
+          deployment.apps/openemr      3/3     3            3           111s
+          deployment.apps/phpmyadmin   1/1     1            1           111s
+          deployment.apps/redisproxy   2/2     2            2           111s
+
+          NAME                                    DESIRED   CURRENT   READY   AGE
+          replicaset.apps/openemr-7889cf48d8      3         3         3       111s
+          replicaset.apps/phpmyadmin-f4d9bfc69    1         1         1       111s
+          replicaset.apps/redisproxy-744b7749dc   2         2         2       111s
+
+          NAME                         READY   AGE
+          statefulset.apps/mysql-sts   2/2     111s
+          statefulset.apps/redis       3/3     111s
+          statefulset.apps/sentinel    3/3     111s
           ```
-5. Getting the url link to OpenEMR:
+4. Getting the url link to OpenEMR:
     - If using minikube, can get the link to go to OpenEMR with following command (use the top link for http and bottom link for https):
         ```bash
         minikube service openemr --url
@@ -78,11 +91,11 @@ TODO (optimizing some things). Add support for redis password in the OpenEMR doc
             http://192.168.99.100:31314
             http://192.168.99.100:30613
             ```
-    - If using kind, then can use the 3***** port shown in step 4 above with the ip address obtained from following command:
+    - If using kind, then can use the 3***** port shown in step 3 (at `service/openemr`) above with the ip address obtained from following command:
         ```bash
         docker inspect kind-control-plane | grep "IPAddress"
         ```
-6. Getting the url link to phpMyAdmin:
+5. Getting the url link to phpMyAdmin:
     - If using minikube, can get the link to go to phpMyAdmin with following command:
         ```bash
         minikube service phpmyadmin --url
@@ -91,22 +104,21 @@ TODO (optimizing some things). Add support for redis password in the OpenEMR doc
             ```console
             http://192.168.99.100:30571
             ```
-    - If using kind, then can use the 3***** port shown in step 4 above with the ip address obtained from following command:
+    - If using kind, then can use the 3***** port shown in step 3 (at `service/phpmyadmin`) above with the ip address obtained from following command:
         ```bash
         docker inspect kind-control-plane | grep "IPAddress"
         ```
-7. Some cool replicas stuff with OpenEMR. The OpenEMR docker pods are run as a replica set (since it is set to 3 replicas in this OpenEMR deployment script). Gonna cover how to view the replica set and how to change the number of replicas on the fly in this step.
+6. Some cool replicas stuff with OpenEMR. The OpenEMR docker pods are run as a replica set (since it is set to 3 replicas in this OpenEMR deployment script). Gonna cover how to view the replica set and how to change the number of replicas on the fly in this step.
     - First. lets list the replica set like this:
         ```bash
         kubectl get rs
         ```
         - It will look something like this (note OpenEMR has 3 desired and 3 current replicas going):
             ```console
             NAME                    DESIRED   CURRENT   READY   AGE
-            mysql-64449b8cf7        1         1         1       4m5s
-            openemr-5f6db6c87c      3         3         3       4m5s
-            phpmyadmin-78968d6cfb   1         1         1       4m5s
-            redis-74cc9d667         1         1         1       4m5s
+            openemr-7889cf48d8      3         3         3       9m22s
+            phpmyadmin-f4d9bfc69    1         1         1       9m22s
+            redisproxy-744b7749dc   2         2         2       9m22s
             ```
     - Second, lets increase OpenEMR's replicas from 3 to 10 (ie. pretend in an environment where a huge number of OpenEMR users are using the system at the same time)
         ```bash
@@ -127,7 +139,7 @@ TODO (optimizing some things). Add support for redis password in the OpenEMR doc
             ```
         - Now, there are 5 replicas of OpenEMR instead of 10. Enter the `kubectl get rs` and `kubectl get pod` to see what happened.
     - This is just a quick overview of scaling. Note we just did manual scaling in the example above, but there are also options of automatic scaling for example depending on cpu use etc.
-8. Some cool replicas stuff with MariaDB. 2 statefulset replications of MariaDB (1 primary/master with 1 replica/slave) are created by default. The number of replicas can be increased or decreased.
+7. Some cool replicas stuff with MariaDB. 2 statefulset replications of MariaDB (1 primary/master with 1 replica/slave) are created by default. The number of replicas can be increased or decreased.
     - Increase replicas (after this command will have the 1 primary/master with 3 replicas/slaves).
         ```bash
         kubectl scale sts mysql-sts --replicas=4
@@ -136,7 +148,7 @@ TODO (optimizing some things). Add support for redis password in the OpenEMR doc
         ```bash
         kubectl scale sts mysql-sts --replicas=3
         ```
-9. To stop and remove OpenEMR orchestration (this will delete everything):
+8. To stop and remove OpenEMR orchestration (this will delete everything):
     ```bash
     bash kub-down
     ```
diff --git a/kubernetes/openemr/deployment.yaml b/kubernetes/openemr/deployment.yaml
@@ -48,6 +48,8 @@ spec:
           value: "admin"
         - name: REDIS_SERVER
           value: "redisproxy"
+        - name: REDIS_PASSWORD
+          value: "defaultpassword"
         - name: SWARM_MODE
           value: "yes"
         image: openemr/openemr:7.0.0
diff --git a/kubernetes/redis/configmap-acl.yaml b/kubernetes/redis/configmap-acl.yaml
@@ -6,4 +6,4 @@ data:
   users.acl: |
     user admin on >adminpassword ~* &* +@all
     user replication on >replicationpassword +psync +replconf +ping
-    user default on nopass ~* &* +@all -@dangerous
+    user default on >defaultpassword ~* &* +@all -@dangerous
diff --git a/kubernetes/redis/configmap-main.yaml b/kubernetes/redis/configmap-main.yaml
@@ -17,16 +17,14 @@ data:
     #appendfilename "appendonly.aof"
     
     # Enabled ACL based auth. 
-    # TODO - support protected mode after have support for a redis password in the openemr docker
-    #        then can set password for the default user in the acl conf
-    protected-mode no
+    protected-mode yes
     
     # This is used by the replics nodes to communicate with master to replicate the data.
     # we are using a user called "replication" for this, and the a strong pwd for the same is given in masterauth
     masterauth replicationpassword
     masteruser replication
     
-    # this is the second ConfiMap will be mounted to. it has the list of uses needed.
+    # this is the second ConfigMap will be mounted to. it has the list of users needed.
     aclfile /conf/acl/users.acl
     
     # port, each redis nodes will be used
diff --git a/kubernetes/redis/configmap-pipy.yaml b/kubernetes/redis/configmap-pipy.yaml
@@ -36,6 +36,7 @@ data:
             unhealthy_nodes.remove(_target),
             data = resp.shift(40).toString().split('\r\n'),
             role = data[3].split(':')[1],
+            config.debug && console.log(`Role is ${role} for ${_target}`),
             role === 'master' && unhealthy_master.remove(_target)
           ))()
       })
