-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
GITBOOK-163: Deployment Documentation 1.2 Organize
- Loading branch information
1 parent
a36904e
commit 75a373c
Showing
39 changed files
with
800 additions
and
421 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # External Components Setup | ||
|
|
||
| ## Introduction | ||
|
|
||
| This guide provides instructions to deploy external components on the Kubernetes (K8s) cluster upon which OpenG2P components reply (Refer to [Deployment Architecture](../deployment-architecture.md)). | ||
|
|
||
| | Module/Component | Comments | | ||
| | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | ||
| | [PostgreSQL](postgresql-server-deployment.md) | Required for all components. A single server instance may be used housing all databases. | | ||
| | [Keycloak](keycloak-deployment.md) | Required for PBMS, Social Registry | | ||
| | [MinIO](minio-deployment.md) | Required for PBMS and GCTB only | | ||
| | [ODK Central](odk-central-deployment.md) | Required for Registration Toolkit | | ||
| | [Kafka](kafka-deployment.md) | Required for Monitoring & Reporting | | ||
| | [Logging & OpenSearch](logging-and-opensearch-deployment.md) | Required for Monitoring & Reporting | | ||
| | [MOSIP Key Manager](keymanager-deployment.md) | Required for PBMS, Social Registry | | ||
| | [e-Signet](e-signet-deployment.md) | Required for SPAR and optionally for PBMS | |
35 changes: 35 additions & 0 deletions
35
deployment/external-components-setup/e-signet-deployment.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # e-Signet Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| This doc provides instructions on installing e-Signet on the OpenG2P cluster. | ||
|
|
||
| This is only required for sandbox/pilot environments. Or when e-Signet is not present or is not provided by the ID Provider. If an e-Signet instance is already available, OpenG2P Modules can just connect to that instance. | ||
|
|
||
| This doc only provides instructions to install e-Signet with Mock ID System (for integration with real ID system, refer to [e-Signet docs](https://docs.esignet.io)). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
| * [PostgreSQL](postgresql-server-deployment.md) | ||
| * [Keycloak](keycloak-deployment.md) for API Authentication | ||
| * [Keymanager](keymanager-deployment.md) | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/esignet](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/esignet) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
|
|
||
| ## Post-installation | ||
|
|
||
| After installation is successful, e-Signet can be accessed at https://esignet.openg2p.sandbox.net, depending on the hostname given above. | ||
|
|
||
| To seed more data of beneficiaries into the mock ID system APIs, use the APIs available at https://esignet.openg2p.sandbox.net/v1/mock-identity-system/swagger-ui/index.html.  | ||
|
|
||
| Or edit and use this script [https://github.com/OpenG2P/openg2p-data/blob/develop/scripts/upload\_data\_to\_mock\_esignet.py](https://github.com/OpenG2P/openg2p-data/blob/develop/scripts/upload\_data\_to\_mock\_esignet.py) to upload data. TODO: elaborate. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # Kafka Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| Skip this if the [realtime reporting framework](https://github.com/openg2p/openg2p-reporting) is not being used. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/kafka](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/kafka) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
|
|
||
| ## Post-installation | ||
|
|
||
| After installation is successful, Kafka UI can be accessed at https://kafka.openg2p.sandbox.net, depending on the hostname given above. |
25 changes: 25 additions & 0 deletions
25
deployment/external-components-setup/keycloak-deployment.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Keycloak Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| Keycloak is used in OpenG2P to provide single sign-on to some of the apps. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
| * [PostgreSQL](postgresql-server-deployment.md) | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/keycloak](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/keycloak) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
|
|
||
| ## Post-installation | ||
|
|
||
| After installation is successful, Keycloak Admin console will be accessible at https://keycloak.openg2p.sandbox.net, depending on the hostname given above. |
27 changes: 27 additions & 0 deletions
27
deployment/external-components-setup/keymanager-deployment.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Keymanager Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| MOSIP's Keymanager component is used by some OpenG2P modules (like PBMS and social registry) to store keys and perform cryptography operations. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
| * [PostgreSQL](postgresql-server-deployment.md) | ||
| * [Keycloak](keycloak-deployment.md) for API Authentication | ||
| * HSM. By default, Softhsm will be installed, unless real HSM is available. | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/keymanager](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/keymanager) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
|
|
||
| ## Post-installation | ||
|
|
||
| After installation is successful, Keymanager APIs will be accessible at https://openg2p.sandbox.net/v1/keymanager, depending on the hostname given above. |
67 changes: 67 additions & 0 deletions
67
deployment/external-components-setup/logging-and-opensearch-deployment.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Logging & OpenSearch Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| Logs from different components present on the cluster will be pulled into OpenSearch to display dashboards and compute reports. Fluentd is used to pull capture logs and put into OpenSearch. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
| * [Keycloak](keycloak-deployment.md) for Authentication and Sign-in to UI | ||
|
|
||
| ## Installation | ||
|
|
||
| Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/logging](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/logging) directory. | ||
|
|
||
| ### Install OpenSearch (and related components) | ||
|
|
||
| * Run this to install OpenSearch and related components. | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
| * After installation is successful, OpenSearch Dashboards will be accessible at https://opensearch.openg2p.sandbox.net, depending on the hostname given above. | ||
|
|
||
| ### Install Rancher Logging (Fluentd) | ||
|
|
||
| 1. On Rancher UI, navigate to Apps (or Apps & Marketplace) -> Charts | ||
| 2. Search and install Logging from the list, with default values. | ||
|
|
||
| ### Add _Index State Policy_ on OpenSearch | ||
|
|
||
| * Run this to add ISM Policy (This is responsible for automatically deleting logstash indices after 3 days. Configure the minimum age to delete indices, in the same script below.) | ||
|
|
||
| ``` | ||
| ./opensearch-ism-script.sh | ||
| ``` | ||
|
|
||
| ### Configure Rancher FluentD | ||
|
|
||
| * Run this to create _ClusterOutput_ (This is responsible for redirecting all logs to OpenSearch.) | ||
|
|
||
| ``` | ||
| kubectl apply -f clusterflow-opensearch.yaml | ||
| ``` | ||
| * Run this to create a _ClusterFlow_ (This is responsible for filtering OpenG2P service logs, from the logs of all pods.) | ||
|
|
||
| ``` | ||
| kubectl apply -f clusterflow-all.yaml | ||
| ``` | ||
|
|
||
| ### Filters | ||
|
|
||
| Note the filters applied in [clusterflow-all.yaml](https://github.com/OpenG2P/openg2p-deployment/blob/main/kubernetes/logging/clusterflow-all.yaml). You may update the same for your install if required, and rerun the apply command. | ||
|
|
||
| ### Dashboards | ||
|
|
||
| * TODO | ||
|
|
||
| ### TraceId | ||
|
|
||
| * TODO | ||
|
|
||
| ### Troubleshooting | ||
|
|
||
| * TODO |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Minio Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| MinIO is used by some components of OpenG2P store documents. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
| * [Keycloak](keycloak-deployment.md) for Authentication and Sign-in to UI | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/minio](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/minio) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
|
|
||
| ## Post-installation | ||
|
|
||
| After installation is successful, MinIO console will be accessible at https://minio.openg2p.sandbox.net, depending on the hostname given above. | ||
|
|
||
| Once OpenG2P PBMS is installed, do the following: | ||
|
|
||
| * Navigate to OpenG2P Documents (From OpenG2P Menu) -> Document Store. | ||
| * Configure URL and password for this backend service (Like `http://minio.minio:9000`). Password and account-id/username can be obtained from the secrets in minio namespace. |
42 changes: 42 additions & 0 deletions
42
deployment/external-components-setup/odk-central-deployment.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # ODK Central Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| ODK is used mainly by the Registration Toolkit to collect data offline and online. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
| * [PostgreSQL](postgresql-server-deployment.md) | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/odk-central](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/odk-central) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| SANDBOX_HOSTNAME="openg2p.sandbox.net" \ | ||
| ./install.sh | ||
| ``` | ||
| * Note: The above helm chart uses the following docker images built from [https://github.com/getodk/central/tree/v2023.1.0](https://github.com/getodk/central/tree/v2023.1.0), since ODK Central doesn't provide pre-built docker images for these. | ||
| ``` | ||
| openg2p/odk-central-backend:v2023.1.0 | ||
| openg2p/odk-central-frontend:v2023.1.0 | ||
| openg2p/odk-central-enketo:v2023.1.0 | ||
| ``` | ||
| ## Post-installation | ||
| After installation is successful, ODK Central will be accessible at https://odk.openg2p.sandbox.net, depending on the hostname given above. | ||
| To create the first user, do this (Subsequent users can be created through UI.): | ||
| * Exec into the service pod, and create a user (and promote if required). | ||
| ```bash | ||
| kubectl exec -it <service-pod> -- odk-cmd -u <email> user-create | ||
| kubectl exec -it <service-pod> -- odk-cmd -u <email> user-promote | ||
| ``` | ||
25 changes: 25 additions & 0 deletions
25
deployment/external-components-setup/postgresql-server-deployment.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # PostgreSQL Server Deployment | ||
|
|
||
| ## Introduction | ||
|
|
||
| This guide provides instructions to install PostgreSQL Server on the Kubernetes cluster. However, if you already have PostgresSQL server installed, or are using Cloud hosted Postgres, then you may skip the server installation. The instructions to initialize OpenG2P component databases are provided as part of the component installation instructions. | ||
|
|
||
| ## Databases | ||
|
|
||
| Module/component-wise listing of databases is given below | ||
|
|
||
| <table><thead><tr><th width="349">Module/Component</th><th>Database Name</th></tr></thead><tbody><tr><td>PBMS</td><td><code>openg2pdb</code></td></tr><tr><td>Keycloak</td><td><code>keycloakdb</code></td></tr><tr><td>ODK</td><td><code>odkdb</code></td></tr><tr><td>SPAR</td><td><code>spardb</code></td></tr><tr><td>G2P Cash Transfer Bridge</td><td><code>gctbdb</code></td></tr><tr><td>MOSIP Key Manager</td><td><code>mosip_keymgr</code></td></tr></tbody></table> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * The following utilities/tools must be present on the user's machine. | ||
| * `kubectl`, `istioctl`, `helm`, `jq`, `curl`, `wget`, `git`, `bash`, `envsubst`. | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone the [https://github.com/openg2p/openg2p-deployment](https://github.com/openg2p/openg2p-deployment) repo and navigate to [kubernetes/postgresql](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/postgresql) directory. | ||
| * Run: | ||
|
|
||
| ```bash | ||
| ./install.sh | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Infrastructure Setup | ||
|
|
||
| | Infra | Comments | | ||
| | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | ||
| | [Wireguard](wireguard-server-setup.md) | Only one per all the environments | | ||
| | [Rancher](rancher.md) | Only one per all the environments | | ||
| | [NFS Server](nfs-server.md) | One for each environment like sandbox, pilot, staging, production | | ||
| | [OpenG2P K8s Cluster](k8s-cluster.md) | One for each environment | | ||
| | [Loadbalancer](loadbalancer-setup.md) | One for each environment. For non cloud-native Kubernetes clusters either create a VM with Nginx or create a cloud LB. | |
File renamed without changes.
Oops, something went wrong.