Update Documentation with Learnings from Cleanup Exercise (#312)

* Updating documentation to include learnings from the cleanup exercercise

Author: nman-hmcts

source/Camunda/howto.html.md.erb

@@ -1,6 +1,6 @@
 ---
 title: Camunda How To Guides
-last_reviewed_on: 2024-04-27
+last_reviewed_on: 2024-07-25
 review_in: 12 months
 weight: 50
 ---
@@ -86,7 +86,7 @@ If you know the name of the definition you can search for it via the Camunda Coc
 
 <img src="images/processInstances.png" style="height:300px;">
 
-Once you've found it you can select the process to see more details about it including a list of process instances for this specific process definition.
+Once you have found it, you can select the process to see more details about it including a list of process instances for this specific process definition.
 
 Within the definition page there is a `delete running instances` option
 
@@ -100,13 +100,62 @@ Enter a reason for the deletion e.g. the Jira ticket number and then use the sea
 
 Its possible to select only specific records in the list of results or by selecting `Query` it will select all records found, **ALL** records means across all pages shown now just the current page, remember these are SQL queries in the background!
 
-## Update removal time on a record
+## Removing Historic Process and Decision Instances
+
+There are two types of instances that needs to be cleared out in Camunda, historic process instances and historic decision instances. Removing historic process and decision instances will improve the performance of Camunda and prevent the database from becoming too large. Going forward, manual clearing should be less necessary as new process and decision instances will automatically have a TTL (Time to Live) value set. This TTL is 90 days for production environments and 30 days for all non-production environments.
+
+However, TTL values have not been set for all existing instances, and only some teams have adopted the practice of setting TTLs. As a result, manual cleanups may still be required until all teams have updated existing instances and fully embraced the process.
+
+There are two ways to remove historic process and decision instances:
+
+- Set removal time on process or decision instance (preferable)
+- Delete the process or decision instance
+
+## Set Removal Time on a Process or Decision Instance
 
 Camunda removal time updates are a batch operation that can be carried out via the Camunda Cockpit UI.
 
-Updating a records removal time will control when that record will be removed from the Camunda database by the nightly Camunda clean up task.
+Setting a records removal time will control when that record will be removed from the Camunda database by the Camunda clean up task.
+
+### Process Instance
+
+To run a removal time operation on process instances you can:
+
+- Log into Camunda and select the `Cockpit` option
+
+    <img src="images/cockpit.png" style="height:100px;">
+
+- Navigate to the menu and select `more` then `batch operation`
+
+    <img src="images/batchOperation.png" style="width:150px;">
+
+- On this page you will find a number of fields, selecting `operation` will show a dropdown of available options
+
+    <img src="images/batchOperationOptions.png" style="width:300px;">
+
+- Select `Set removal time to historic process instances`. Ensure you select all options as per the below screenshot. Double check that Absolute, Hierarchical and Query are selected.
 
-To run a removal time operation you can:
+    <img src="images/processInstanceRemovalTime.png">
+
+- Initially this will search the database for all historic records of process instances. You can reduce the time to return results by using search criteria.
+
+- To narrow down these results you can add search criteria. Clicking in the `add criteria` search box will show a list of available options such as start date, finish date, `incident status` and a host of `IDs` options. For the purposes of cleanup, you can chose 'Finished Only' and 'Finish Date' with value set to before a particular date. If you are getting lot of results or if the results are taking a long time to load, simply set the finish date to an older date to reduce the number of results being returned.
+
+    <img src="images/processInstanceFilters.png">
+
+- You can also save your query as shown in the screenshot below for later use. However, this query is not available across different users as it is stored in the browser cache of the user.
+
+    <img src="images/saveQuery.png">
+
+- Ensure you have the right dates and filters applied and then select "Set removal time to historic process instances all Instances in Filter".
+
+- On the next screen select "Execute Operation" to proceed with setting the removal times.
+
+- Finally you can track progress of the operation by navigating to the menu and select `more` then `batches`
+
+### Decision Instance
+
+To run a removal time operation on decision instances you can:
 
 - Log into Camunda and select the `Cockpit` option
 
@@ -120,19 +169,34 @@ To run a removal time operation you can:
 
     <img src="images/batchOperationOptions.png" style="width:300px;">
 
-- Select one of the `Set removal time to historic X` options that applies to your need.
+- Select `Set removal time to historic decision instances`. Ensure you select all options as per the below screenshot. Double check that Absolute and Query are selected and importantly Hierarchical is deselected. This differs slightly from the steps from process decisions. If you select Hierarchical the whole operation will run extremely slowly, so ensure it is not selected as it is selected by default.
+
+    <img src="images/decisionInstanceRemovalTime.png">
+
+- Initially this will search the database for all historic records of decision instances. You can reduce the time to return results by using search criteria.
+
+- To narrow down these results you can add search criteria. Clicking in the `add criteria` search box will show a list of available options. For the purposes of cleanup, you can chose 'Evaluation Date' with value set to before a particular date. Evaluation Date is the equivalent of Finished Date when it comes to decision instances. If you are getting lot of results or if the results are taking a long time to load, simply set the evaluation date to an older date to reduce the number of results being returned.
+
+    <img src="images/decisionInstanceFilters.png">
+
+- You can also save your query as shown in the screenshot below for later use. However, this query is not available across different users as it is stored in the browser cache of the user.
+
+    <img src="images/saveQuery.png">
+
+- Ensure you have the right dates and filters applied and then select "Set removal time to historic decision instances all Instances in Filter".
+
+- On the next screen select "Execute Operation" to proceed with setting the removal times.
+
+- Finally you can track progress of the operation by navigating to the menu and select `more` then `batches`
+
+### Cleanup Job
 
-- Initially this will search the database for all historic records for your chosen option e.g. all completed process instances. You can reduce the time to return results by using search criteria.
+Once the removal times are set, you can confirm this by checking the batches to see if the set removal time jobs have completed. Now you can either wait for the next schedule clean up to complete or you can manually run the cleanup job by navigating to the menu and selecting `more` then `cleanup`. Click on cleanup to trigger the job. Once the cleanup is complete which may take a while depending on the number of instances to clean up, you can run the same queries as before from the batch operation menu. If the cleanup job has completed correctly, the queries should return empty results.
 
-- To narrow down these results you can add search criteria. Clicking in the `add criteria` search box will show a list of available options such as start date, finish date, `incident status` and a host of `IDs` options.
+## Delete the Process or Decision Instance
 
-    <img src="images/searchCriteria.png" style="width:600px;">
+Instead of setting a removal time and waiting for cleanup jobs, you can directly delete a historic process or decision instance. However, this approach is riskier because there is no option to revert in the event of an incorrect execution. This direct deletion is a single operation on the database, whereas the other option (Setting Removal Times) involves first updating and then deleting the instance.
 
-- For set removal time batch operations there are a number of additional options provided:
+- Running a Delete operation is pretty similar to running a Set Removal Time operation, except you should select "Delete finished process instances" or "Delete decision instances"
 
-    - **Cleared** - removes the removal time of the select records, there should be no reason to use this.
-    - **Calculated** - Uses either the Time To Live of the definition that created this record OR the system default of 30 days where no TTL is available. The value is calculated by Camunda an will likely be in the future by at least 30 days.
-    - **Absolute** - An absolute date chosen by the user to set for each record selected, a calendar input appears for this option and it can be set to any date past or present. Past simply means the record will be removed during the next clean up task.
-    - **Hierarchical** - Sets the removal time across all instances of the hierarchy
-    - **Update in chunks** - Will stream the changes slowly to ensure the database is not overloaded. It is very advisable to use this option.
-    - **Chunk Size** - this option will appear when `Update in chunks` is select and should be set to a reasonable number to maintain normal functionality of the service but still achieve the desired outcome.
+    <img src="images/deleteOperations.png">

source/Camunda/images/decisionInstanceFilters.png

@@ -1,6 +1,6 @@
 ---
 title: Camunda Setup
-last_reviewed_on: 2024-04-27
+last_reviewed_on: 2024-07-25
 review_in: 12 months
 weight: 45
 ---
@@ -124,16 +124,15 @@ The application components are deployed to AKS via [Flux](https://github.com/hmc
 
 Whilst the deployments are architecturally the same they differ in configuration. UI is similar across all environments but API has custom configuration specifically for Production and Demo which are the ones like to see the most end user traffic.
 
-For Demo and Production we have setup controlled clean up configuration.
-The application has default values for these built in however due to the nature of Production these defaults have been overridden.
+For all non-production environments, we have set up controlled cleanup configuration via [Flux](https://github.com/hmcts/cnp-flux-config/tree/master/apps/camunda/camunda-api). It is important to note that regarding the cleanup start and end times, any configuration specified via Flux is an additional schedule, not an override. A default schedule is already specified in the source files [here](https://github.com/hmcts/camunda-bpm/blob/master/src/main/resources/application.yaml). An additional schedule, like the one below, has been configured for non-production environments since these environments are usually offline or shut down outside of working hours, which is when the default schedule runs for production.
 
 ```YAML
   CAMUNDA_BATCH_HISTORY_TTL: "P30D"
   CAMUNDA_BATCH_OPERATIONS: "P30D"
   CAMUNDA_HISTORY_TTL: "P30D"
   CAMUNDA_HISTORY_JOBLOG__TTL: "P30D"
-  CAMUNDA_HISTORY_CLEANUP_START_TIME: "00:30+0100"
-  CAMUNDA_HISTORY_CLEANUP_END_TIME: "06:00+0100"
+  CAMUNDA_HISTORY_CLEANUP_START_TIME: "12:00+0100"
+  CAMUNDA_HISTORY_CLEANUP_END_TIME: "14:00+0100"
   CAMUNDA_HISTORY_CLEANUP_STRATEGY: "removalTimeBased"
   CAMUNDA_PARALLELISM: 1
 ```
@@ -147,4 +146,4 @@ As an example, we have this value in the [application.yaml](https://github.com/h
 historyTimeToLive: ${CAMUNDA_HISTORY_TTL:P5D}
 ```
 
-The `historyTimeToLive` configuration can be set by using `CAMUNDA_HISTORY_TTL` or it will be assigned a default of `P5D (5 Days)`. This allows us to override any value in the application.yaml file from Flux on a per environment basis.
+The `historyTimeToLive` configuration can be set by using `CAMUNDA_HISTORY_TTL` or it will be assigned a default of `P5D (5 Days)`. This allows us to override any value in the application.yaml file from Flux on a per environment basis.

source/Camunda/images/decisionInstanceRemovalTime.png

@@ -1,6 +1,6 @@
 ---
 title: Camunda Troubleshooting
-last_reviewed_on: 2024-04-27
+last_reviewed_on: 2024-07-25
 review_in: 12 months
 weight: 55
 ---
@@ -23,11 +23,11 @@ Its important to remember that this list is hierarchical i.e. Flux will override
 
 ## Database Performance
 
-There are many reasons a database can't become a bottleneck for any service so we cannot document all of them here but we can document those issues that we've seen and resolved.
+There are many reasons a database cannot become a bottleneck for any service so we cannot document all of them here but we can document those issues that we have seen and resolved.
 
 ### Database size
 
-We've observed that the Camunda interface can become sluggish and unresponsive, leading to service timeouts and eventual 504 responses.
+We have observed that the Camunda interface can become sluggish and unresponsive, leading to service timeouts and eventual 504 responses.
 
 Our initial solution was to increase the DB SKU, thereby allocating more resources for the service to use. However, this is a temporary fix, not a long-term solution.
 
@@ -65,8 +65,8 @@ Its possible to find the number of historical records held in the database in th
 
         <img src="images/differentCleanableTypes.png" style="width:300px;">
 
-    - If `History Time To Live` for a definition is `Null` that is a **problem**, those records will never be clean up by Camunda because Camunda cannot calculate a `removal time` value. See [this](howto.html#update-removal-time-on-a-record) guide on adding removal times to records.
-    - If the `finished` count is large and the `cleanable` count is 0 that doesn't mean the records will not be removed, if there is a `History Time To Live` value set then it means that those finished records cannot be cleaned up **yet** but will be in future.
+    - If `History Time To Live` for a definition is `Null` that is a **problem**, those records will never be clean up by Camunda because Camunda cannot calculate a `removal time` value. See [this](howto.html#set-removal-time-on-a-process-or-decision-instance) guide on adding removal times to records.
+    - If the `finished` count is large and the `cleanable` count is 0 that does not mean the records will not be removed, if there is a `History Time To Live` value set then it means that those finished records cannot be cleaned up **yet** but will be in future.
 
          <img src="images/cleanableRecords.png" style="width:200px;">