| Component | Build Job | Build Tag | Deploy Job | Deployment | Comment |
|---|---|---|---|---|---|
| Batch Service | Build/Core/Lms | release-5.3.1_RC1 | Deploy/Kubernetes/Lms | release-5.3.1_RC1 | QR Codes Image download Issue fix |
| Project | Release Date | Version |
|---|---|---|
| Lern | 27-May-2023 | V 5.3.0 |
| Lern | 23-Jun-2023 | V 5.3.1 |
| Components | Build Jenkins Job | Build Tag | Deploy Jenkins Job | Deploy Tag | Comment |
|---|---|---|---|---|---|
| Kafka Setup | Deploy/Lern/KafkaSetup | verify if kafka topic = programuser.info is created or not | |||
| Data pipeline | Build/Lern/FlinkJobs | release-5.3.0_RC5 | Deploy/Lern/FlinkJobs | release-5.3.0_RC5 | Add program-user-info into job list and deploy it. |
| Data Products | Build/Lern/LernDataProducts | release-5.3.0_RC6 | Deploy/Lern/LernDataProducts | release-5.3.0_RC6 | Add program-user-exhaust into job list of Deploy/Lern/LernAnalyticsReplayJobs for running it. |
| Cassandra Migration | Build/Core/Cassandra | release-5.2.0_RC1 | Deploy/Kubernetes/Cassandra | release-5.2.0-lern | add the sunbird_programs keyspace in Deploy Jenkins jobs |
| Analytics | Deploy/Kubernetes/Analytics | Deploy with release-6.0.0 branch |
Summary of the Changes
Details of the Changes:
LR-491 User detail (PII) report for ML programs - Data Product
LR-285 User detail (PII) report for ML programs - Flink Job
default config for services
sunbird.program.report.keyspace="{{ program_keyspace }}"
ml.exhaust.store.prefix="ml_reports"
Please define below variables
program_keyspace: "sunbird_programs"
ml.exhaust.store.prefix="ml_reports"
https://github.com/shikshalokam/sunbird-utils/blob/release-5.2.0/sunbird-cassandra-migration/cassandra-migration/src/main/resources/db/migration/cassandra/sunbird_programs/V1.1_cassandra.cql
| Name of the Flink Job added |
|---|
| program-user-info |
LR-285 - User detail flink job for ML-programs - setup/configuration details:
For this ticket, we have only done unit testing with the help of simulated events. Integration testing has not been done as the required workflows concerning this will only be enabled after Ed 6.0 release. As part of this ticket we have enabled new Flink jobs and they in no way impact any existing workflows
Job name: program-user-info
The purpose of this job is to record the user's information when the user submits the program. Whenever a program is submitted, this job receives an event with the user's information as JSON data and then it parses and stores it as respective key-value pairs in Cassandra.
Keyspace name: sunbird_program
Schema of the Kafka Topic:
Kafka Topic Name: {{envName}}.programuser.info
Event Structure:-
{
programId: {
type : "ObjectId",
required : true,
index: true
},
programName: String,
programExternalId: String,
noOfResourcesStarted: {
type:Number,
index: true
}
userId: {
type: String,
index: true
},
requestForPIIConsent:true/false
userProfile: Object,
userRoleInformation: Object,
appInformation: Object,
createdAt: Date,
updatedAt: Date,
deleted:Boolean
}
Job Configurations:
kafka {
input.topic = ${job.env}".programuser.info"
groupId = ${job.env}"-programuser-group"
}
task {
consumer.parallelism = 1
downstream.parallelism = 1
programUser{
parallelism = 1
}
}
ml-cassandra {
keyspace = "sunbird_programs"
table = "program_enrollment"
port = "9042"
host =
}
Flink build Jenkins job name: /Build/job/Lern/job/FlinkJobs
Flink deploy Jenkins job name: /Deploy/job/<environment>/job/Lern/job/FlinkJobs/program-user-info
Jenkins job for building Cassandra: /Build/job/Core/job/Cassandra/
Jenkins job for deploying Cassandra: /Deploy/job/<environment>/job/Kubernetes/job/Cassandra
Configurations to be done by System admin:
- Setup default 'Data Security Policy' settings using tenant preference API.
curl --location --request PATCH '{{host}}/api/org/v2/preferences/update' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
"request": {
"orgId": "default",
"key": "dataSecurityPolicy",
"data": {
"level": "PLAIN_DATASET",
"dataEncrypted": "No",
"comments": "Data is not encrypted",
"job": {
"userinfo-exhaust": {
"level": "PASSWORD_PROTECTED_DATASET",
"dataEncrypted": "No",
"comments": "Password protected file."
},
"program-user-exhaust": {
"level": "PASSWORD_PROTECTED_DATASET",
"dataEncrypted": "No",
"comments": "Password protected file."
}
},
"securityLevels": {
"PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
"PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
"TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
"PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
}
}
}
}'
| Components | Build Jenkins Job | Build Tag | Deploy Jenkins Job | Deploy Tag | Comment |
|---|---|---|---|---|---|
| Kafka Setup | Deploy/Lern/KafkaSetup | release-5.3.0_RC3 | |||
| Data pipeline | Build/Lern/FlinkJobs | release-5.3.0_RC5 | Deploy/Lern/FlinkJobs | release-5.3.0_RC5 | Add legacy-certificate-migrator into job list and deploy it. |
| Data Products | Build/Lern/LernDataProducts | release-5.3.0_RC6 | Deploy/Lern/LernDataProducts | release-5.3.0_RC6 | |
| Batch Service | Build/Core/Lms | release-5.3.0_RC1 | Deploy/Kubernetes/Lms | release-5.3.0-lern | |
| User&Org Service | Build/Core/Learner | release-5.3.0_RC2 | Deploy/Kubernetes/Learner | release-5.3.0-lern | |
| Analytics | Deploy/Kubernetes/Analytics | Deploy with release-6.0.0 branch |
Details of the Changes:
LR-436 OldCertificateMigration spark data-product
LR-437 LegacyCertificateMigrator Flink job
LR-438 Sunbird RC changes for updating schema for issued date
LR-330 Certificate template font url migration
LR-395, LR-465 PII data security
LR-451 Local setup of Data-pipeline - Ubuntu & Mac - Github and Microsite update
LR-443 Local setup of UserOrg - Ubuntu & Mac - Github and Microsite update
LR-445 Local setup of LMS - Ubuntu & Mac - Github and Microsite update
LR-422 Point the channel create API to content-service instead of learning-service
LR-519 Textbook APIs code cleanup from Course-Batch service
LR-486 Microsite update with Certificate generation flow diagram
LR-520 Group service - activity type should be case insensitive
LR-556 Local setup of LMS - Ubuntu & Mac - Mock service setup
LR-456 Local setup of Sunbird-utils - Ubuntu & Mac - Github and Microsite update\
- name: exhaustSubmitProxyAPI
uris: "{{ course_service_prefix }}/v1/jobrequest/submit"
upstream_url: "{{ lms_service_url }}/v1/jobrequest/submit"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- courseAccess
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ small_request_size_limit }}"
- name: opa-checks
config.required: false
config.enabled: false
- name: exhaustListProxyAPI
uris: "{{ course_service_prefix }}/v1/jobrequest/list"
upstream_url: "{{ lms_service_url }}/v1/jobrequest/list"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- courseAccess
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ small_request_size_limit }}"
- name: opa-checks
config.required: false
config.enabled: false
- name: orgAddEncryptionKey
uris: "{{ org_service_prefix }}/v1/update/encryptionkey"
upstream_url: "{{ learning_service_url }}/v1/org/update/encryptionkey"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- orgSuperAdmin
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ small_request_size_limit }}"
- name: opa-checks
config.required: false
config.enabled: false
The below environment variable needs to be configured in the 'sunbird-lms-service.env' file dev ops repo. Ref: https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/stack-sunbird/templates/sunbird_lms-service.env
| Variable Name | Values | Comments |
|---|---|---|
| exhaust_api_base_url | {{ analytics_service_url | default('http://analytics-service:9000') }} | Obsrv exhaust API endpoint for batch service |
| exhaust_api_submit_endpoint | /request/submit | To submit job request from batch service |
| exhaust_api_list_endpoint | /request/list/ | To list job request from batch service |
| sunbird_api_auth_token | "{{ core_vault_sunbird_api_auth_token }}" | Authentication token for APIs |
Configurations to be done by System admin:
- Execute CURL for providing link to download "Decryption Tool". Tool reference: https://github.com/Sunbird-Lern/sunbird-utils/blob/release-5.3.0/decryption-tool/decryption-tool.zip
{% hint style="info" %} Please upload the tool to your public cloud location or to your repository and provide the link to the same in below system setting variable value. {% endhint %}
curl --location --request POST '{{host}}/api/data/v1/system/settings/set' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{api_key}}' \
--header 'x-authenticated-user-token: {{user_token}}' \
--data-raw '{
"request": {
"id": "decryptionToolLink",
"field": "decryptionToolLink",
"value": "{\"link\":\"<link to download decryption tool>\", \"Comments\": \"To use this tool, run the command with encrypted file and key to decrypt\"}"
}
}'
- Setup default 'Data Security Policy' settings using tenant preference API.
curl --location --request POST '{{host}}/api/org/v2/preferences/create' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
"request": {
"orgId": "default",
"key": "dataSecurityPolicy",
"data": {
"level": "PLAIN_DATASET",
"dataEncrypted": "No",
"comments": "Data is not encrypted",
"job": {
"userinfo-exhaust": {
"level": "PASSWORD_PROTECTED_DATASET",
"dataEncrypted": "No",
"comments": "Password protected file."
}
},
"securityLevels": {
"PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
"PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
"TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
"PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
}
}
}
}'
- Setup **default '**PII data security settings' using tenant preference API.
curl --location --request POST '{{host}}/api/org/v2/preferences/create' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
"request": {
"orgId": "default",
"key": "userPrivateFields",
"data": {
"PIIFields": [
"email",
"phone",
"userName",
"prevUsedEmail",
"prevUsedPhone",
"recoveryEmail",
"recoveryPhone"
]
}
}
}'
Configurations that can be done by Tenants:
- Use Tenant preference create API to create tenant specific 'Data Security Policy' settings similar to 'default' Data Security Policy settings but with tenant orgId.
Note:
a. Tenant level security cannot be lower than 'default' Data Security Policy'.
b. Job Level security Policy in a Tenant specific configuration cannot be lower than Tenant Level configuration and cannot be lower than job level configuration in 'default' Data Security Policy'.
c. Below mapping shows the priority/grade of security policies
"PLAIN_DATASET" < "PASSWORD_PROTECTED_DATASET" < "TEXT_KEY_ENCRYPTED_DATASET" < "PUBLIC_KEY_ENCRYPTED_DATASET"
- In order to use "PUBLIC_KEY_ENCRYPTED_DATASET" security configuration for an exhaust report, tenant admin should have uploaded public pem key file using below API.
curl --location --request PATCH '{{host}}/api/org/v1/update/encryptionkey' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--form 'organisationId={{org_id}}' \
--form 'encryptionKey=@path_to_public_pem_file'
- To generate Private Key
openssl genrsa -out private.pem 4096
- To generate Public Key
openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem
Please install GitBash: The Git installation package comes with SSH. Using Git Bash, which is the Git command line tool, you can generate SSH key pairs. Git Bash has an SSH client that enables you to connect to and interact with Triton containers on Windows.
To install Git:
- Download and initiate the Git installer.
- When prompted, accept the default components by clicking Next.
- Choose the default text editor. If you have Notepad++ installed, select Notepad++ and click Next.
- Select to Use Git from the Windows Command Prompt and click Next.
- Select to Use OpenSSL library and click Next.
- Select to Checkout Windows-style, commit Unix-style line endings and click Next.
- Select to Use MinTTY (The default terminal of mYSYS2) and click Next.
- Accept the default extra option configuration by clicking Install. When the installation completes, you may need to restart Windows.
Launching GitBash:
- press Start+R to launch the Run dialog.
- Type C:\Program Files\Git\bin\bash.exe and press Enter.
Generating Key pair:
- To generate Private Key
openssl genrsa -out private.pem 4096
- To generate Public Key
openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem
| Name of the Flink Job added |
|---|
| legacy-certificate-migrator |
LR-436 - Deploy Data-product
Data-product build Jenkins job: Build/Lern/LernDataProducts
Deploy Jenkins job: Deploy/{{env}}/Lern/LernDataProducts
LR-437 - Deploy legacy-certificate-migrator Flink job
Build Jenkins job: /Build/job/Lern/job/FlinkJobs
Deploy Jenkins job: /Deploy/job/<environment>/job/Lern/job/FlinkJobs
LR-438 - Update RC schema
Step 1 : Upload updated schema files.
Deploy Jenkins job: Deploy/dev/Sunbird-RC/Upload_RC_Schema
Note: Since certificate signer service will cache the credential template. please make sure the credential template is updated in the respective path as per below file.
Step 2 : Deploy certificate signer service
Jenkins Job: Deploy/dev/Sunbird-RC/CertificateSign
Sunbird Lern BB is using Sunbird RC for generating & issuing e-credentials in its use cases (e.g.: course completion certificate) for all the latest completed courses (post March-2022). All the old certificates were custom generated and stored in Cassandra and cloud storage.
Once we migrate these certificates then we no longer need to store certificates in Cassandra and all the certificates will be using Sunbird RC going forward.
Reference Link: https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3117416449/LR-4+Design+of+migrating+existing+certificate+in+to+RC
Note: After migrating old certificates to RC, certificate verification of old certificates will become invalid. To support to old certificate verification, Sunbird ED building block is implementating in portal service in release 6.0. Kindly find the ticket in this link. So recommended to migrate the certificates after getting the old certification verification support as well.
Step 1
Create Kafka topic for only the purpose of this migration process
Topic name: {{env}}.legacy.certificate.migrate
Step 2
In the spark machine, update the old-certificate-migration-job model config in mount/data/analytics/scripts/lern-model-config.sh with correct values.
Sample model config:
{% code overflow="wrap" %}
{"search":{"type":"none"},"model":"org.sunbird.lms.audit.OldCertificateMigrationJob","modelParams":{"mode":"execute","store":"azure","sparkCassandraConnectionHost":"10.5.3.17", "cert_base_path": "https://dev.lern.sunbird.org", "cloud_storage_base_url": "https://sunbirddev.blob.core.windows.net", "cloud_store_base_path_placeholder": "CLOUD_BASE_PATH","content_cloud_storage_container": "sunbird-content-staging", "cloud_storage_cname_url": "https://obj.stage.sunbirded.org", "batchId": "01320961460024934435", "kafka_broker": "localhost:9092", "kafka_topic": "sunbirddevlern.legacy.certificate.migrate","output_file_path":"./reports/"},"parallelization":8,"appName":"OldCertificateMigrationJob"}
{% endcode %}
Note: migration job can be run single batch with "batchId": "01320961460024934435" and multiple batches with "batchId": "01320961460024934435,01220961460024934536" and for all batches with "batchId": "all" .
Step 3
Run the job with the below command in the spark machine.
/mount/data/analytics/scripts/lern-run-job.sh old-certificate-migration-job &
Note: logs can be found in below locations,
Joblog: /mount/data/analytics/scripts/logs/joblog.log
Execution log: /mount/data/analytics/logs/lern-data-products/{current_date}-job-execution.log
Note:
Verification steps can be found in the design page: https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3117416449/LR-4+Design+of+migrating+existing+certificate+in+to+RC#Verification-steps-for-the-certificate-migration-process
All the templates are having dev URLs configured for Fonts in all the environments as per our observation. All these font URLs have to be migrated to the new cname URL
Note: Before font url migration, make sure all the font files are available at cname mapped account or cloud storage container. To verify, where the font files are available, open any svg template file in editor and check the font URL's host.
Please use java 11 for running the scripts
Download SVG file migrator and uploader jars by below command,
cd ~
mkdir svg_template_migration
cd svg_template_migration
wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-migration/svg-migrator.jar"
wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-upload/svg-uploader.jar"
Download the svg template files and update the font URLs in the template files.
java -jar svg-migrator.jar "{{ content search host }}" "0" "1000" "font_migration" "{{ Old URL }}" "{{ cname url }}"
#EXAMPLE
#java -jar svg-migrator.jar "dev.lern.sunbird.org" "0" "1000" "font_migration" "https://sunbirddev.blob.core.windows.net" "https://obj.diksha.gov.in"
Note: Before moving to next step, please verify atleast one svg file for whether the font URL got updated.
Upload the svg template files back to the cloud storage by below command.
java -jar svg-uploader.jar "{{ content search host }}" "0" "1000" "{{ storage key}}" "{{ storage secret }}" "{{svg file path}}" "{{storage type: (azure,..)}}" "{{ CSP endpoint (based on CSP it is optional) }}" "{{ region (based on CSP it is optional) }}"
#EXAMPLE
#java -jar svg-uploader.jar "dev.lern.sunbird.org" "0" "5" "sunbirddevbbpublic" "{{ secret }}" "/Users/{{username}}/svg_template_migration" "azure"