feat(schematic): fds 828 merge main into feature branch (#2465)

* feat(schematic): integration testing (#2398) * changed authenticication so that only endpoints that need it have it * updated schematic * add patch for access token * schema endpoints no longer mockeed * added tests for handle exceptions * added integration tests * marked synapse tests * added error handling for bad schema urls * fix error message * add workflow for end to end testing * fix some test results * add unit mark * add unit mark * add workflow for testing with secrets * rename file * fix synapse test file when secrets file doesnt exists * fix test workflows * turned synapse ids into secrets in workflow * turned synapse ids into secrets in workflow * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * add paging, and split connected noeds into two endpoints * paginated preoject datasets query * paginated preoject datasets query * paginated dataset files endpoint * pagniate project manifests endpoint * paginate get node dependencies * paginate get node dependencies * paginate node properties endpoint * paginate validation rules endpoint * paginate get projects endpoint * update schematic and other packages * remove uneeded type ignores * added generate manifest endpoints * temp commit * delete manifest file * add generate manifest endpoint * fix broken tests * ci(sage-monorepo): create a GH composite action to setup the dev container (#2438) * Create a composite action to setup the dev container * Add shell * Add shell * Fix indent * Add shell * Temporarily comment out sonar scan for pushes to main (#2443) * Touch one file (#2444) * Restore sonar scan in CI workflow (#2446) * Undo openchallenges-app touch (#2447) * Add name when running `.github/actions/setup-dev-container` (#2451) * ci(sage-monorepo): experiment with `pull_request_target` (#2453) * Add ssonar scan workflow * Check if secret can be read * ci(sage-monorepo): try sonar scan trigger (#2454) * Touch OC app file * Update workflow to run sonar scans * ci(sage-monorepo): test sonar scan from PR from fork (#2455) * Touch OC web app * Touch the file again * Fetch correct branch * Undo changes to web app * ci(sage-monorepo): try new sonar scan workflow for PRs (#2456) * Touch oc web app * Update sonar scan workflow * ci(sage-monorepo): try sonar scan workflow triggered by fork PR (#2457) * Touch OC web app * Trigger again the workflow * Create prototype of new sonar scanner for pr * Cleanup * Cleanup * Update scanner parameters * Disable sonar scanner in main CI workflow * ci(sage-monorepo): parametrize sonar scanner tool (#2458) * Touch oc web app * Ignore .nx * Save before trying Bard suggestion * Upgrade sonar scanner to support PR * Simplify command * Cleanpu * Set working dir * Specify PR number to SonarScanner for Gradle * Remove scanner tool proto * Sort Ci env vars in alphanum * Update message * ci(sage-monorepo): fix how env vars are passed to the dev container (#2459) * Touch two projects with scanner * Pass env var to CI * Add SONAR_PULL_REQUEST_NUMBER to dev container definition * ci(sage-monorepo): try sonar scan again for PRs originating from forks (#2460) * Touch two OC projects * Require label `sonar-scan-appproved` to run Sonar * Remove hard coded PR number * ci(sage-monorepo): test Sonar workflow with label-based approval (#2461) * Touch two oc projects * Throw error if sonar label is missing * Update mesage * Touch two oc projects (#2462) --------- Co-authored-by: Thomas Schaffter <[email protected]>
Sage-Bionetworks · Jan 26, 2024 · c11c1f9 · c11c1f9
1 parent 06269d9
commit c11c1f9
Show file tree

Hide file tree

Showing 12 changed files with 375 additions and 206 deletions.
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -1,7 +1,6 @@
 {
   "name": "Sage Dev Container",
   "image": "ghcr.io/sage-bionetworks/sage-devcontainer:be08406",
-
   "containerEnv": {
     "NX_BASE": "${localEnv:NX_BASE}",
     "NX_BRANCH": "${localEnv:NX_BRANCH}",
@@ -11,20 +10,21 @@
     "NX_HEAD": "${localEnv:NX_HEAD}",
     "NX_RUN_GROUP": "${localEnv:NX_RUN_GROUP}",
     "SONAR_TOKEN": "${localEnv:SONAR_TOKEN}",
+    "SONAR_PULL_REQUEST_NUMBER": "${localEnv:SONAR_PULL_REQUEST_NUMBER}",
     "DOCKER_USERNAME": "${localEnv:DOCKER_USERNAME}",
     "DOCKER_PASSWORD": "${localEnv:DOCKER_PASSWORD}"
   },
-
   "features": {
     "ghcr.io/devcontainers/features/docker-in-docker:2.4.0": {
       "version": "24.0.5",
       "moby": false
     }
   },
-
   "customizations": {
     "codespaces": {
-      "openFiles": ["README.md"]
+      "openFiles": [
+        "README.md"
+      ]
     },
     "vscode": {
       "extensions": [
@@ -66,13 +66,43 @@
       }
     }
   },
-
   "forwardPorts": [
-    2432, 3000, 3306, 4200, 4211, 5017, 5200, 5432, 5601, 7010, 7080, 7200, 7888, 8010, 8071, 8000,
-    8080, 8081, 8082, 8083, 8084, 8085, 8086, 8090, 8091, 8092, 8200, 8787, 8888, 8889, 9090, 9104,
-    9200, 9411, 27017
+    2432,
+    3000,
+    3306,
+    4200,
+    4211,
+    5017,
+    5200,
+    5432,
+    5601,
+    7010,
+    7080,
+    7200,
+    7888,
+    8010,
+    8071,
+    8000,
+    8080,
+    8081,
+    8082,
+    8083,
+    8084,
+    8085,
+    8086,
+    8090,
+    8091,
+    8092,
+    8200,
+    8787,
+    8888,
+    8889,
+    9090,
+    9104,
+    9200,
+    9411,
+    27017
   ],
-
   "portsAttributes": {
     "2432": {
       "label": "itlas-postgres",
@@ -219,8 +249,10 @@
       "onAutoForward": "silent"
     }
   },
-
   "remoteUser": "vscode",
   "shutdownAction": "stopContainer",
-  "runArgs": ["--name", "sage_devcontainer"]
-}
+  "runArgs": [
+    "--name",
+    "sage_devcontainer"
+  ]
+}
diff --git a/.github/actions/setup-dev-container/action.yml b/.github/actions/setup-dev-container/action.yml
@@ -0,0 +1,80 @@
+name: 'Set up the dev container'
+description: 'Installs the dev container CLI, fetches caches (if exist), and starts the dev container'
+runs:
+  using: 'composite'
+  steps:
+    - name: Set up Yarn cache
+      uses: actions/cache@v3
+      with:
+        path: '/tmp/.yarn/cache'
+        key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
+        restore-keys: |
+          ${{ runner.os }}-yarn-
+
+    - name: Set up Renv cache
+      uses: actions/cache@v3
+      with:
+        path: '/tmp/.cache/R/renv/cache'
+        key: ${{ runner.os }}-renv-cache-${{ hashFiles('**/renv.lock') }}
+        restore-keys: |
+          ${{ runner.os }}-renv-cache-
+
+    - name: Set up Poetry cache
+      uses: actions/cache@v3
+      with:
+        path: '/tmp/.cache/pypoetry'
+        key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
+
+    - name: Set up venv cache
+      uses: actions/cache@v3
+      with:
+        path: |
+          /tmp/.local/share/virtualenv
+          **/.venv
+        key: ${{ runner.os }}-venv-${{ hashFiles('**/poetry.lock') }}
+
+    - name: Set up Gradle cache
+      uses: actions/cache@v3
+      with:
+        path: |
+          /tmp/.gradle/caches
+          /tmp/.gradle/wrapper
+        key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
+        restore-keys: |
+          ${{ runner.os }}-gradle-
+
+    - name: Install the Dev Container CLI
+      shell: bash
+      run: npm install -g @devcontainers/[email protected]
+
+    - name: Start the dev container
+      shell: bash
+      run: |
+        mkdir -p \
+          /tmp/.yarn/cache \
+          /tmp/.cache/R/renv/cache \
+          /tmp/.cache/pypoetry \
+          /tmp/.local/share/virtualenv \
+          /tmp/.gradle/caches \
+          /tmp/.gradle/wrapper
+
+        devcontainer up \
+          --mount type=bind,source=/tmp/.yarn/cache,target=/workspaces/sage-monorepo/.yarn/cache \
+          --mount type=bind,source=/tmp/.cache/R/renv/cache,target=/home/vscode/.cache/R/renv/cache \
+          --mount type=bind,source=/tmp/.cache/pypoetry,target=/home/vscode/.cache/pypoetry \
+          --mount type=bind,source=/tmp/.local/share/virtualenv,target=/home/vscode/.local/share/virtualenv \
+          --mount type=bind,source=/tmp/.gradle/caches,target=/home/vscode/.gradle/caches \
+          --mount type=bind,source=/tmp/.gradle/wrapper,target=/home/vscode/.gradle/wrapper \
+          --workspace-folder ../sage-monorepo
+
+    - name: Prepare the workspace
+      shell: bash
+      run: |
+        devcontainer exec --workspace-folder ../sage-monorepo bash -c "
+          sudo chown -R vscode:vscode \
+            /workspaces/sage-monorepo \
+            /home/vscode/.cache \
+            /home/vscode/.local \
+            /home/vscode/.gradle \
+          && . ./dev-env.sh \
+          && workspace-install-affected"
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -12,7 +12,7 @@ env:
   NX_CLOUD_AUTH_TOKEN: ${{ secrets.NX_CLOUD_AUTH_TOKEN }}
   NX_CLOUD_ENCRYPTION_KEY: ${{ secrets.NX_CLOUD_ENCRYPTION_KEY }}
   NX_CLOUD_ENV_NAME: 'linux'
-  SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+  # SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
   DOCKER_USERNAME: ${{ github.actor }}
   DOCKER_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
 
@@ -31,78 +31,8 @@ jobs:
       - name: Derive appropriate SHAs for base and head for `nx affected` commands
         uses: nrwl/nx-set-shas@v3
 
-      - name: Set up Yarn cache
-        uses: actions/cache@v3
-        with:
-          path: '/tmp/.yarn/cache'
-          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-yarn-
-
-      - name: Set up Renv cache
-        uses: actions/cache@v3
-        with:
-          path: '/tmp/.cache/R/renv/cache'
-          key: ${{ runner.os }}-renv-cache-${{ hashFiles('**/renv.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-renv-cache-
-
-      - name: Set up Poetry cache
-        uses: actions/cache@v3
-        with:
-          path: '/tmp/.cache/pypoetry'
-          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
-
-      - name: Set up venv cache
-        uses: actions/cache@v3
-        with:
-          path: |
-            /tmp/.local/share/virtualenv
-            **/.venv
-          key: ${{ runner.os }}-venv-${{ hashFiles('**/poetry.lock') }}
-
-      - name: Set up Gradle cache
-        uses: actions/cache@v3
-        with:
-          path: |
-            /tmp/.gradle/caches
-            /tmp/.gradle/wrapper
-          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
-          restore-keys: |
-            ${{ runner.os }}-gradle-
-
-      - name: Install the Dev Container CLI
-        run: npm install -g @devcontainers/[email protected]
-
-      - name: Start the dev container
-        run: |
-          mkdir -p \
-            /tmp/.yarn/cache \
-            /tmp/.cache/R/renv/cache \
-            /tmp/.cache/pypoetry \
-            /tmp/.local/share/virtualenv \
-            /tmp/.gradle/caches \
-            /tmp/.gradle/wrapper
-
-          devcontainer up \
-            --mount type=bind,source=/tmp/.yarn/cache,target=/workspaces/sage-monorepo/.yarn/cache \
-            --mount type=bind,source=/tmp/.cache/R/renv/cache,target=/home/vscode/.cache/R/renv/cache \
-            --mount type=bind,source=/tmp/.cache/pypoetry,target=/home/vscode/.cache/pypoetry \
-            --mount type=bind,source=/tmp/.local/share/virtualenv,target=/home/vscode/.local/share/virtualenv \
-            --mount type=bind,source=/tmp/.gradle/caches,target=/home/vscode/.gradle/caches \
-            --mount type=bind,source=/tmp/.gradle/wrapper,target=/home/vscode/.gradle/wrapper \
-            --workspace-folder ../sage-monorepo
-
-      - name: Prepare the workspace
-        run: |
-          devcontainer exec --workspace-folder ../sage-monorepo bash -c "
-            sudo chown -R vscode:vscode \
-              /workspaces/sage-monorepo \
-              /home/vscode/.cache \
-              /home/vscode/.local \
-              /home/vscode/.gradle \
-            && . ./dev-env.sh \
-            && workspace-install-affected"
+      - name: Set up the dev container
+        uses: ./.github/actions/setup-dev-container
 
       - name: Lint the affected projects
         run: |
@@ -124,10 +54,10 @@ jobs:
           devcontainer exec --workspace-folder ../sage-monorepo bash -c ". ./dev-env.sh \
             && nx affected --target=integration-test"
 
-      - name: Scan the affected projects with Sonar
-        run: |
-          devcontainer exec --workspace-folder ../sage-monorepo bash -c ". ./dev-env.sh \
-            && nx affected --target=sonar"
+      # - name: Scan the affected projects with Sonar
+      #   run: |
+      #     devcontainer exec --workspace-folder ../sage-monorepo bash -c ". ./dev-env.sh \
+      #       && nx affected --target=sonar"
 
       - name: Publish the images of the affected projects
         run: |
@@ -161,78 +91,8 @@ jobs:
       - name: Derive appropriate SHAs for base and head for `nx affected` commands
         uses: nrwl/nx-set-shas@v3
 
-      - name: Set up Yarn cache
-        uses: actions/cache@v3
-        with:
-          path: '/tmp/.yarn/cache'
-          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-yarn-
-
-      - name: Set up Renv cache
-        uses: actions/cache@v3
-        with:
-          path: '/tmp/.cache/R/renv/cache'
-          key: ${{ runner.os }}-renv-cache-${{ hashFiles('**/renv.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-renv-cache-
-
-      - name: Set up Poetry cache
-        uses: actions/cache@v3
-        with:
-          path: '/tmp/.cache/pypoetry'
-          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
-
-      - name: Set up venv cache
-        uses: actions/cache@v3
-        with:
-          path: |
-            /tmp/.local/share/virtualenv
-            **/.venv
-          key: ${{ runner.os }}-venv-${{ hashFiles('**/poetry.lock') }}
-
-      - name: Set up Gradle cache
-        uses: actions/cache@v3
-        with:
-          path: |
-            /tmp/.gradle/caches
-            /tmp/.gradle/wrapper
-          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
-          restore-keys: |
-            ${{ runner.os }}-gradle-
-
-      - name: Install the Dev Container CLI
-        run: npm install -g @devcontainers/[email protected]
-
-      - name: Start the dev container
-        run: |
-          mkdir -p \
-            /tmp/.yarn/cache \
-            /tmp/.cache/R/renv/cache \
-            /tmp/.cache/pypoetry \
-            /tmp/.local/share/virtualenv \
-            /tmp/.gradle/caches \
-            /tmp/.gradle/wrapper
-
-          devcontainer up \
-            --mount type=bind,source=/tmp/.yarn/cache,target=/workspaces/sage-monorepo/.yarn/cache \
-            --mount type=bind,source=/tmp/.cache/R/renv/cache,target=/home/vscode/.cache/R/renv/cache \
-            --mount type=bind,source=/tmp/.cache/pypoetry,target=/home/vscode/.cache/pypoetry \
-            --mount type=bind,source=/tmp/.local/share/virtualenv,target=/home/vscode/.local/share/virtualenv \
-            --mount type=bind,source=/tmp/.gradle/caches,target=/home/vscode/.gradle/caches \
-            --mount type=bind,source=/tmp/.gradle/wrapper,target=/home/vscode/.gradle/wrapper \
-            --workspace-folder ../sage-monorepo
-
-      - name: Prepare the workspace
-        run: |
-          devcontainer exec --workspace-folder ../sage-monorepo bash -c "
-            sudo chown -R vscode:vscode \
-              /workspaces/sage-monorepo \
-              /home/vscode/.cache \
-              /home/vscode/.local \
-              /home/vscode/.gradle \
-            && . ./dev-env.sh \
-            && workspace-install"
+      - name: Set up the dev container
+        uses: ./.github/actions/setup-dev-container
 
       - name: Lint the affected projects
         run: |