Merge branch 'ZH-261' of github.com:GenomicDataInfrastructure/standar…

…d-operating-procedures into ZH-261

CHANGELOG.md

@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - [``sop_index.py``](scripts/sop_index.py) - Script to automatically create the SOP index table
 - [``utils.py``](scripts/utils.py) - General functions used by other scripts
 - [``sops/README.md``](sops/README.md) - Markdown containing the SOP index table
+- [``GDI-SOP_review-checklist.md``](docs/GDI-SOP_review-checklist.md) - Documentation guidelines in the form of checklists, for reviewers, approvers and authorizers of SOPs.
+- [``GDI-SOP_charter.md``](docs/GDI-SOP_charter.md) - Documentation Charter of the task 4.3
 - [``tests/``](tests/) - Directory containing tests to run GH repo's code
 - [``requirements.txt``](requirements.txt) - Needed modules to run GH repo's code
 - [``lint_sops.yml``](.github/workflows/lint_sops.yml) - GH Workflow to trigger linter on PRs

README.md

@@ -28,38 +28,38 @@ flowchart TB
     B2 --> E
     F[General GDI SOP Template]
     F -->|Is used by| E
-    E[Operations Committee - OC\nSecurity and Data Protection Committee - SDPC]
+    E[Operations Committee - OC<br>Security and Data Protection Committee - SDPC]
     E -->|Prepares template| G
     G[SOP Template]
     G --> I
 
     %% Boxes
     subgraph European-level SOPs
         S[SOP Instance]
-        V[Finished European-level\nSOP Instance]
-        ZA[Released European-level\nSOP Instance]
+        V[Finished European-level<br>SOP Instance]
+        ZA[Released European-level<br>SOP Instance]
     end
 
-    ZA -->|Enters periodic\nreview cycle| ZA
+    ZA -->|Enters periodic<br>review cycle| ZA
 
     subgraph Node-specific SOPs
         Q[SOP Template]
-        W[Finished Node-specific\nSOP Template]
-        Z[Released Node-specific\nSOP Template]
-        nodeRep[OC/SDPC Representatives\nfrom each node]
+        W[Finished Node-specific<br>SOP Template]
+        Z[Released Node-specific<br>SOP Template]
+        nodeRep[OC/SDPC Representatives<br>from each node]
         Z -->|is used by| nodeRep
         subgraph Node's-GitHub
-            nodeTem[Node-specific\nSOP Template]
-            nodeSOP[Node-specific\nSOP Instance]
+            nodeTem[Node-specific<br>SOP Template]
+            nodeSOP[Node-specific<br>SOP Instance]
         end
         nodeRep -->|Copy SOP Template| Node's-GitHub
         subgraph Node's-Roles
-            nodeRep[Node's OC/SDPC \n representative]
+            nodeRep[Node's OC/SDPC <br> representative]
             nodeRep --> |Nominate| nodeExp
             nodeExp[Nominated experts]
         end
         subgraph Node's-SOP-development-process
-            nodeDev(Template gets adapted\n with the node's needs)
+            nodeDev(Template gets adapted<br> with the node's needs)
             nodeDev --> nodeRev
             nodeRev(Review)
             nodeRev --> nodeApp
@@ -69,14 +69,14 @@ flowchart TB
         nodeTem --> nodeDev
         nodeApp --> |Produces| nodeSOP
     end
-    rev2[OC/SDPC] -..->|Approves \n changes| nodeApp
+    rev2[OC/SDPC] -..->|Approves <br> changes| nodeApp
 
     I --> |Start development process| R
     Authors -..-> |Fill in SOP content| L
     Reviewers -..->|Review SOP| M
     Approvers -..->|Approve SOP|N
     Authorizers -..->|Authorize SOP| O
-    R{Is SOP\na template?}
+    R{Is SOP<br>a template?}
     R -->|Yes| Q
     R -->|No| S
     Q -->|Enters cycle|SOP-development-cycle
@@ -96,11 +96,11 @@ flowchart TB
     aos-merge -->|Produces| W
 
     subgraph Main-GitHub-repository
-        aos-accessioning(SOP accessioning\n and formatting)
-        aos-merge(PR against \n `dev` branch)
+        aos-accessioning(SOP accessioning<br> and formatting)
+        aos-merge(PR against <br> &apos;dev&apos; branch)
         subgraph SOP-release-process
-            git1(Pull Request\nto `main` branch)
-            git1 -->|Automatically \n triggers| git2
+            git1(Pull Request<br>to &apos;main&apos; branch)
+            git1 -->|Automatically <br> triggers| git2
             git2(Zenodo release)
         end
     end

docs/GDI-SOP_github-management.md

@@ -8,13 +8,13 @@ Given that the base repository is public, forks of this repository cannot be pri
 ````mermaid
 flowchart TD
     A[Source Repository] 
-    dec{Node's repo\nprivacy}
+    dec{Node's repo<br>privacy}
     A --> dec
     dec -->|Public| B(Fork public repository)
     dec -->|Private| C(Import)    
-    H[Work with node's\n repository]
+    H[Work with node's<br> repository]
     F --> H
-    B --> H[Work with node's\n repository]
+    B --> H
     subgraph Public Path
         B
     end

docs/GDI-SOP_review-checklist.md

@@ -0,0 +1,81 @@
+# SOP Review Checklist
+
+This checklist serves as a guide for **SOP reviewers, approvers, and authorizers**. It outlines the steps and key points to ensure that the SOPs are complete, clear, and compliant with the GDI framework. All reviewers should ensure each point is met before moving on to the next step. Approvers and authorizers have additional responsibilities as noted below.
+
+## General SOP Review Checklist
+
+### 1. **Document Structure & Formatting**
+- [ ] **Compliance with SOP template**: Ensure that the SOP complies with the latest SOP template (see [``GDI-SOP_sop-template.md``](./GDI-SOP_sop-template.md)). Annotations in the template (between ``<>``) contain useful information and formatting rules, including:
+    - **Required Sections**: Ensure the SOP includes the required sections (e.g., ``Index`` or ``Procedures``).
+    - **Linting Process**: Check that key values (e.g., the SOP title, the SOP number...) make sense and comply with the expected format.
+- [ ] **Document Consistency**: Check for consistent formatting, structure, and numbering throughout the SOP.
+- [ ] **Metadata**: Verify that metadata in all tables (e.g., versioning, dates, authors...) is included and accurate.
+
+#### Linting Script
+To ease the document linting process, the script ``sop_linter.py`` was created. This script **checks multiple parts of the format of an SOP**, and retrieves any given errors.
+
+You can **manually execute** the script to ensure a file (i.e., the SOP document) passes the linting checks as follows:
+````
+cd standard-operating-procedures
+pip install --upgrade --no-deps -r requirements.txt
+python3 scripts/sop_linter.py tests/GDI-SOP0000_sop-template_for_linting.md -v 1
+````
+
+Replace the test SOP (``tests/GDI-SOP0000_sop-template_for_linting.md``) with the file path of the SOP you are reviewing.
+
+If there are no linting errors, the output will look like the following:
+````
+{
+  "tests/GDI-SOP0000_sop-template_for_linting.md": {
+    "errors": [],
+    "warnings": []
+  }
+}
+````
+
+For your convenience, the script is automatically executed (via [``lint_sops.yml``](../.github/workflows/lint_sops.yml) workflow) whenever a [Pull Request](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/pulls) (PR) containing SOPs is created. Therefore, if you are reviewing a PR, you can **easily see if the SOPs comply with the expected format** by ensuring that the **latest commit has the green check mark** (✔️) next to it.
+
+![Valid: workflow check in Pull Request](images/GDI-SOP_review-checklist-workflows.png)
+
+If, on the contrary, the latest commit has a red cross (❌), some of the checks did not pass and it requires further inspection.
+
+### 2. **Content Completeness**
+- [ ] **Purpose and Scope**: Ensure the SOP clearly states its purpose and scope, with no ambiguity about the intended outcomes.
+- [ ] **Roles and Responsibilities**: Confirm that the roles are clearly defined and all stakeholders (OC, SDPC, authors, reviewers, etc.) are properly referenced.
+- [ ] **Procedures**: Review all procedures for clarity and completeness. Procedures should include step-by-step instructions that are easy to follow.
+- [ ] **References**: Ensure any external references, tools, or resources are correctly cited and linked if applicable. Make sure that all hyperlinks work as expected.
+
+### 3. **Clarity & Usability**
+- [ ] **Language**: Verify the SOP is written in clear, simple language, avoiding jargon where possible.
+- [ ] **Consistency**: Ensure terms and abbreviations are used consistently throughout the document, as defined in the glossary, and also across the related SOPs, where appropriate.
+- [ ] **Glossary**: Check that the glossary includes all acronyms that are being used throughout the document.
+- [ ] **Examples/Diagrams**: Check for the presence of any necessary examples, diagrams, or clarifying notes to aid understanding.
+
+### 4. **Operational Feasibility**
+- [ ] **Applicability**: Evaluate whether the content of SOP is relevant to the intended operational environment and aligned with GDI objectives.
+- [ ] **Scalability**: Depending on the type of SOP, ie "European-level" or "Node-specific", review how the SOP is applicable to all GDI nodes, or specifies any node-specific requirements, respectively.
+- [ ] **Feasibility**: Ensure that the procedures are feasible for the assigned roles and aligned with the technological capabilities of the nodes.
+
+### 5. **Cross-Referencing and SOP Compliance**
+- [ ] **Referencing Existing SOPs**: Check that the SOP does not significantly overlap or conflict with other existing SOPs, and ensure that references to any related SOPs are included. See the full [list of released SOPs](../sops/README.md).
+- [ ] **Compliance with SOP Development SOP**: Verify that the SOP has followed the procedure outlined in the [``GDI-SOP0007_SOP-template-creation.md``](../sops/european-level/GDI-SOP0007_SOP-template-creation.md), including review stages and deadlines.
+
+## Reporting and Escalation
+- [ ] **Document your feedback** in the appropriate review platform: either the Google Document or Pull Request, depending on the SOP development phase.
+- [ ] **Escalate comments to both the authors and the OC/SDPC member** in charge of this SOP development. These stakeholders are listed in the SOP's "Roles and Responsibilities" section. There are numerous ways to reach out to these roles, depending on the platform of review: ideally either Google Doc comments or GitHub Pull Request reviews; and secondarily through emails or directly GDI's slack workspace.
+
+## Additional Checklist for Approvers
+
+- [ ] **Impact Assessment**: Evaluate the potential impact of the SOP on operations, and confirm that it enhances standardization and efficiency across nodes.
+- [ ] **Node Representation**: Confirm that the SOP has been reviewed by relevant representatives from GDI nodes, and their input has been incorporated.
+- [ ] **Feedback Integration**: Ensure that all feedback from reviewers has been appropriately addressed in the final draft.
+
+## Additional Checklist for Authorizers
+
+- [ ] **Formal Authorization**: Confirm that the SOP is ready for final release, and provide authorization (or veto) for its inclusion in the GitHub repository. This veto period is defined in the SOP ``SOP Template creation``(../sops/european-level/GDI-SOP0007_SOP-template-creation.md) as **4 weeks** after the authorization request. If no comment is received before the veto can be exercised, the **SOP will automatically be considered authorized**.
+
+## Tools & References
+
+- **GitHub Repository**: [GDI SOP Repository](https://github.com/GenomicDataInfrastructure/standard-operating-procedures) (Check linting status, review, and approve PRs)
+- [**SOP Template creation SOP**](../sops/european-level/GDI-SOP0007_SOP-template-creation.md)
+- **ZenHub Board**: [T4.3 GDI SOPs](https://app.zenhub.com/workspaces/t43-gdi-sops-667c1c5532726a00b93d51e4/board)

docs/GDI-SOP_sop-accessioning.md

@@ -75,9 +75,9 @@ flowchart TB
             sopt_m2("`**Identifier**: GDI-SOP0001.v1`")
         end
     end
-    sopt_m1 -->|+| sopt
-    title1 -->|+| sopt
-    sopt_m2 -..->|+| sopt
+    sopt_m1 -->|#43;| sopt
+    title1 -->|#43;| sopt
+    sopt_m2 -..->|#43;| sopt
     sopt -->|Node adapts template| sopi
 
     subgraph Node's-GitHub
@@ -87,12 +87,11 @@ flowchart TB
             sopi_m2("`**Identifier**: GDI-SOP0001.v1_SWE.v1`")
         end
     end
-    sopi_m2 -..->|+|sopi
+    sopi_m2 -..->|#43;| sopi
 
     %% Styles
     class sopt Template
     class sopi SOP
-
 ````
 
 ### Supporting documents
@@ -109,13 +108,13 @@ The repository is divided in two main directories:
   - `docs/`: Contains all other supporting documents.
 
 ## SOP Life-cycle
-In this section we briefly describe the common life-cycle of an SOP within the GDI platform. For further details, take a look at the [**summary diagram**](../README.md#summary-diagram).
+In this section we briefly describe the common life-cycle of an SOP within the GDI platform. For further details, take a look at the [**summary diagram**](../README.md#summary-diagram) and the [SOP Template creation SOP](../sops/european-level/GDI-SOP0007_SOP-template-creation.md#7-summary-or-context-diagram).
 
 1. SOP is **requested** by any GDI member by making use of the [**"New SOP request"**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/new/choose) GitHub Issue Template.
-2. SOP Request is evaluated by the 1+MG Working Group, which approves or denies it.
+2. SOP Request is evaluated by the OC/SDPC, who approves or denies it.
 3. The SOP starts its development phase (content filling, review, approval...). This may have multiple steps, based on the chose initial method (e.g. Google Docs, GitHub...). Regardless of the method, the [general SOP template](GDI-SOP_sop-template.md) is used to construct it.
 4. The SOP gets to a level of maturity where it can be added through a Pull Request to the source GitHub repository after approval from authorizers (e.g. Management Board, 1+MG Working Group).
-5. PR is merged with the source repository, representing the end of its development cycle, and triggering the SOP release process. This includes steps like the SOP accessioning, where it receives its ``<Identifier>``.
+5. PR is merged with the source repository, representing the end of its development cycle, soon to be released. 
 
 If the SOP is a European-level SOP instance, it enters its own periodic review cycle. On the other hand, if it is a Node-specific SOP Template, the SOP can be then adapted by nodes once these fork/import (see [rubric](/GDI-SOP_github-management.md)) the source repository. See below an example of how a Node-specific SOP Template could be adapted between the source and a GDI's node repositories.
 ````mermaid
@@ -145,7 +144,7 @@ gitGraph
 - To create something similar to a version manifest, making use of GitHub actions to:
   - Check that the accessioning and naming conventions are correct
   - Check that the SOP identifier is correct, and we place it into the SOP manifest/browser/table somewhere in the repo.
-  - Metadata in the file-content of each SOP is correct and adecuate.
+  - Metadata in the file-content of each SOP is correct and adequate.
     - We should have a table with all metadata of the SOPs, with their topics, location, id, title, type, etc. This would be used in Changelogs.
   - Generate DOIs for GDI SOPs.
   - Prepare documentation (Changelog) for tags/releases.

docs/GDI-SOP_sop-template.md

@@ -87,6 +87,7 @@ _< 1-2 sentence summary describing what is covered within this SOP, defining wha
 ### 6. Introduction and Background Information
 
 _< Include if additional information is needed to ensure the SOP can be understood as a standalone document. Include definitions of SOP-specific concepts, and also include relevant definitions in the Glossary section above. >_
+For a broader context of GDI SOPs, please refer to the [Charter](./GDI-SOP_charter.md#4-introduction).
 
 ### 7. Summary or Context Diagram
 

requirements.txt

@@ -1,4 +1,4 @@
-beautifulsoup4==4.12.3
-markdown==3.6
-packaging==24
-pandas==2.2
+beautifulsoup4>=4.12.3
+markdown>=3.6
+packaging>=24
+pandas>=2.0,<2.2

sops/node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md

@@ -36,7 +36,7 @@ Find GDI SOPs common Glossary at the [**charter document**](https://github.com/G
 | 1+MG         | 1+ Million Genomes                                 |
 | DAC          | Data Access Committee                              |
 | EDIC         | European Digital Infrastructure Consortium         |
-| NCP          | Node Contact Point                                 |
+| NCP          | National Coordination Point                                 |
 | SOP          | Standard Operating Procedure                       |
 
 | Term          | Definition                                         |

tests/GDI-SOP0000_sop-template_for_linting.md

@@ -4,6 +4,7 @@
 |-------------------|---------------------|
 | Template SOP number  | GDI-SOP0001 |
 | Template SOP version | ``v1``          |
+| Topic                | Data protection & security |
 | Template SOP Type    | European-level SOP |
 | GDI Node             |  |
 | Instance version     |  |