Skip to content

Commit c11c1f9

Browse files
andrewelambtschaffter
andrewelamb
and
tschaffter
authored
feat(schematic): fds 828 merge main into feature branch (#2465)
* feat(schematic): integration testing (#2398) * changed authenticication so that only endpoints that need it have it * updated schematic * add patch for access token * schema endpoints no longer mockeed * added tests for handle exceptions * added integration tests * marked synapse tests * added error handling for bad schema urls * fix error message * add workflow for end to end testing * fix some test results * add unit mark * add unit mark * add workflow for testing with secrets * rename file * fix synapse test file when secrets file doesnt exists * fix test workflows * turned synapse ids into secrets in workflow * turned synapse ids into secrets in workflow * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * Update schematic-api-ci.yml * add paging, and split connected noeds into two endpoints * paginated preoject datasets query * paginated preoject datasets query * paginated dataset files endpoint * pagniate project manifests endpoint * paginate get node dependencies * paginate get node dependencies * paginate node properties endpoint * paginate validation rules endpoint * paginate get projects endpoint * update schematic and other packages * remove uneeded type ignores * added generate manifest endpoints * temp commit * delete manifest file * add generate manifest endpoint * fix broken tests * ci(sage-monorepo): create a GH composite action to setup the dev container (#2438) * Create a composite action to setup the dev container * Add shell * Add shell * Fix indent * Add shell * Temporarily comment out sonar scan for pushes to main (#2443) * Touch one file (#2444) * Restore sonar scan in CI workflow (#2446) * Undo openchallenges-app touch (#2447) * Add name when running `.github/actions/setup-dev-container` (#2451) * ci(sage-monorepo): experiment with `pull_request_target` (#2453) * Add ssonar scan workflow * Check if secret can be read * ci(sage-monorepo): try sonar scan trigger (#2454) * Touch OC app file * Update workflow to run sonar scans * ci(sage-monorepo): test sonar scan from PR from fork (#2455) * Touch OC web app * Touch the file again * Fetch correct branch * Undo changes to web app * ci(sage-monorepo): try new sonar scan workflow for PRs (#2456) * Touch oc web app * Update sonar scan workflow * ci(sage-monorepo): try sonar scan workflow triggered by fork PR (#2457) * Touch OC web app * Trigger again the workflow * Create prototype of new sonar scanner for pr * Cleanup * Cleanup * Update scanner parameters * Disable sonar scanner in main CI workflow * ci(sage-monorepo): parametrize sonar scanner tool (#2458) * Touch oc web app * Ignore .nx * Save before trying Bard suggestion * Upgrade sonar scanner to support PR * Simplify command * Cleanpu * Set working dir * Specify PR number to SonarScanner for Gradle * Remove scanner tool proto * Sort Ci env vars in alphanum * Update message * ci(sage-monorepo): fix how env vars are passed to the dev container (#2459) * Touch two projects with scanner * Pass env var to CI * Add SONAR_PULL_REQUEST_NUMBER to dev container definition * ci(sage-monorepo): try sonar scan again for PRs originating from forks (#2460) * Touch two OC projects * Require label `sonar-scan-appproved` to run Sonar * Remove hard coded PR number * ci(sage-monorepo): test Sonar workflow with label-based approval (#2461) * Touch two oc projects * Throw error if sonar label is missing * Update mesage * Touch two oc projects (#2462) --------- Co-authored-by: Thomas Schaffter <thomas.schaffter@gmail.com>
1 parent 06269d9 commit c11c1f9

File tree

12 files changed

+375
-206
lines changed

12 files changed

+375
-206
lines changed

‎.devcontainer/devcontainer.json‎

Lines changed: 44 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
{
22
"name": "Sage Dev Container",
33
"image": "ghcr.io/sage-bionetworks/sage-devcontainer:be08406",
4-
54
"containerEnv": {
65
"NX_BASE": "${localEnv:NX_BASE}",
76
"NX_BRANCH": "${localEnv:NX_BRANCH}",
@@ -11,20 +10,21 @@
1110
"NX_HEAD": "${localEnv:NX_HEAD}",
1211
"NX_RUN_GROUP": "${localEnv:NX_RUN_GROUP}",
1312
"SONAR_TOKEN": "${localEnv:SONAR_TOKEN}",
13+
"SONAR_PULL_REQUEST_NUMBER": "${localEnv:SONAR_PULL_REQUEST_NUMBER}",
1414
"DOCKER_USERNAME": "${localEnv:DOCKER_USERNAME}",
1515
"DOCKER_PASSWORD": "${localEnv:DOCKER_PASSWORD}"
1616
},
17-
1817
"features": {
1918
"ghcr.io/devcontainers/features/docker-in-docker:2.4.0": {
2019
"version": "24.0.5",
2120
"moby": false
2221
}
2322
},
24-
2523
"customizations": {
2624
"codespaces": {
27-
"openFiles": ["README.md"]
25+
"openFiles": [
26+
"README.md"
27+
]
2828
},
2929
"vscode": {
3030
"extensions": [
@@ -66,13 +66,43 @@
6666
}
6767
}
6868
},
69-
7069
"forwardPorts": [
71-
2432, 3000, 3306, 4200, 4211, 5017, 5200, 5432, 5601, 7010, 7080, 7200, 7888, 8010, 8071, 8000,
72-
8080, 8081, 8082, 8083, 8084, 8085, 8086, 8090, 8091, 8092, 8200, 8787, 8888, 8889, 9090, 9104,
73-
9200, 9411, 27017
70+
2432,
71+
3000,
72+
3306,
73+
4200,
74+
4211,
75+
5017,
76+
5200,
77+
5432,
78+
5601,
79+
7010,
80+
7080,
81+
7200,
82+
7888,
83+
8010,
84+
8071,
85+
8000,
86+
8080,
87+
8081,
88+
8082,
89+
8083,
90+
8084,
91+
8085,
92+
8086,
93+
8090,
94+
8091,
95+
8092,
96+
8200,
97+
8787,
98+
8888,
99+
8889,
100+
9090,
101+
9104,
102+
9200,
103+
9411,
104+
27017
74105
],
75-
76106
"portsAttributes": {
77107
"2432": {
78108
"label": "itlas-postgres",
@@ -219,8 +249,10 @@
219249
"onAutoForward": "silent"
220250
}
221251
},
222-
223252
"remoteUser": "vscode",
224253
"shutdownAction": "stopContainer",
225-
"runArgs": ["--name", "sage_devcontainer"]
226-
}
254+
"runArgs": [
255+
"--name",
256+
"sage_devcontainer"
257+
]
258+
}

‎.github/actions/setup-dev-container/action.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
name: 'Set up the dev container'
2+
description: 'Installs the dev container CLI, fetches caches (if exist), and starts the dev container'
3+
runs:
4+
using: 'composite'
5+
steps:
6+
- name: Set up Yarn cache
7+
uses: actions/cache@v3
8+
with:
9+
path: '/tmp/.yarn/cache'
10+
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
11+
restore-keys: |
12+
${{ runner.os }}-yarn-
13+
14+
- name: Set up Renv cache
15+
uses: actions/cache@v3
16+
with:
17+
path: '/tmp/.cache/R/renv/cache'
18+
key: ${{ runner.os }}-renv-cache-${{ hashFiles('**/renv.lock') }}
19+
restore-keys: |
20+
${{ runner.os }}-renv-cache-
21+
22+
- name: Set up Poetry cache
23+
uses: actions/cache@v3
24+
with:
25+
path: '/tmp/.cache/pypoetry'
26+
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
27+
28+
- name: Set up venv cache
29+
uses: actions/cache@v3
30+
with:
31+
path: |
32+
/tmp/.local/share/virtualenv
33+
**/.venv
34+
key: ${{ runner.os }}-venv-${{ hashFiles('**/poetry.lock') }}
35+
36+
- name: Set up Gradle cache
37+
uses: actions/cache@v3
38+
with:
39+
path: |
40+
/tmp/.gradle/caches
41+
/tmp/.gradle/wrapper
42+
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
43+
restore-keys: |
44+
${{ runner.os }}-gradle-
45+
46+
- name: Install the Dev Container CLI
47+
shell: bash
48+
run: npm install -g @devcontainers/cli@0.49.0
49+
50+
- name: Start the dev container
51+
shell: bash
52+
run: |
53+
mkdir -p \
54+
/tmp/.yarn/cache \
55+
/tmp/.cache/R/renv/cache \
56+
/tmp/.cache/pypoetry \
57+
/tmp/.local/share/virtualenv \
58+
/tmp/.gradle/caches \
59+
/tmp/.gradle/wrapper
60+
61+
devcontainer up \
62+
--mount type=bind,source=/tmp/.yarn/cache,target=/workspaces/sage-monorepo/.yarn/cache \
63+
--mount type=bind,source=/tmp/.cache/R/renv/cache,target=/home/vscode/.cache/R/renv/cache \
64+
--mount type=bind,source=/tmp/.cache/pypoetry,target=/home/vscode/.cache/pypoetry \
65+
--mount type=bind,source=/tmp/.local/share/virtualenv,target=/home/vscode/.local/share/virtualenv \
66+
--mount type=bind,source=/tmp/.gradle/caches,target=/home/vscode/.gradle/caches \
67+
--mount type=bind,source=/tmp/.gradle/wrapper,target=/home/vscode/.gradle/wrapper \
68+
--workspace-folder ../sage-monorepo
69+
70+
- name: Prepare the workspace
71+
shell: bash
72+
run: |
73+
devcontainer exec --workspace-folder ../sage-monorepo bash -c "
74+
sudo chown -R vscode:vscode \
75+
/workspaces/sage-monorepo \
76+
/home/vscode/.cache \
77+
/home/vscode/.local \
78+
/home/vscode/.gradle \
79+
&& . ./dev-env.sh \
80+
&& workspace-install-affected"

‎.github/workflows/ci.yml‎

Lines changed: 9 additions & 149 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ env:
1212
NX_CLOUD_AUTH_TOKEN: ${{ secrets.NX_CLOUD_AUTH_TOKEN }}
1313
NX_CLOUD_ENCRYPTION_KEY: ${{ secrets.NX_CLOUD_ENCRYPTION_KEY }}
1414
NX_CLOUD_ENV_NAME: 'linux'
15-
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
15+
# SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
1616
DOCKER_USERNAME: ${{ github.actor }}
1717
DOCKER_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
1818

@@ -31,78 +31,8 @@ jobs:
3131
- name: Derive appropriate SHAs for base and head for `nx affected` commands
3232
uses: nrwl/nx-set-shas@v3
3333

34-
- name: Set up Yarn cache
35-
uses: actions/cache@v3
36-
with:
37-
path: '/tmp/.yarn/cache'
38-
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
39-
restore-keys: |
40-
${{ runner.os }}-yarn-
41-
42-
- name: Set up Renv cache
43-
uses: actions/cache@v3
44-
with:
45-
path: '/tmp/.cache/R/renv/cache'
46-
key: ${{ runner.os }}-renv-cache-${{ hashFiles('**/renv.lock') }}
47-
restore-keys: |
48-
${{ runner.os }}-renv-cache-
49-
50-
- name: Set up Poetry cache
51-
uses: actions/cache@v3
52-
with:
53-
path: '/tmp/.cache/pypoetry'
54-
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
55-
56-
- name: Set up venv cache
57-
uses: actions/cache@v3
58-
with:
59-
path: |
60-
/tmp/.local/share/virtualenv
61-
**/.venv
62-
key: ${{ runner.os }}-venv-${{ hashFiles('**/poetry.lock') }}
63-
64-
- name: Set up Gradle cache
65-
uses: actions/cache@v3
66-
with:
67-
path: |
68-
/tmp/.gradle/caches
69-
/tmp/.gradle/wrapper
70-
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
71-
restore-keys: |
72-
${{ runner.os }}-gradle-
73-
74-
- name: Install the Dev Container CLI
75-
run: npm install -g @devcontainers/cli@0.49.0
76-
77-
- name: Start the dev container
78-
run: |
79-
mkdir -p \
80-
/tmp/.yarn/cache \
81-
/tmp/.cache/R/renv/cache \
82-
/tmp/.cache/pypoetry \
83-
/tmp/.local/share/virtualenv \
84-
/tmp/.gradle/caches \
85-
/tmp/.gradle/wrapper
86-
87-
devcontainer up \
88-
--mount type=bind,source=/tmp/.yarn/cache,target=/workspaces/sage-monorepo/.yarn/cache \
89-
--mount type=bind,source=/tmp/.cache/R/renv/cache,target=/home/vscode/.cache/R/renv/cache \
90-
--mount type=bind,source=/tmp/.cache/pypoetry,target=/home/vscode/.cache/pypoetry \
91-
--mount type=bind,source=/tmp/.local/share/virtualenv,target=/home/vscode/.local/share/virtualenv \
92-
--mount type=bind,source=/tmp/.gradle/caches,target=/home/vscode/.gradle/caches \
93-
--mount type=bind,source=/tmp/.gradle/wrapper,target=/home/vscode/.gradle/wrapper \
94-
--workspace-folder ../sage-monorepo
95-
96-
- name: Prepare the workspace
97-
run: |
98-
devcontainer exec --workspace-folder ../sage-monorepo bash -c "
99-
sudo chown -R vscode:vscode \
100-
/workspaces/sage-monorepo \
101-
/home/vscode/.cache \
102-
/home/vscode/.local \
103-
/home/vscode/.gradle \
104-
&& . ./dev-env.sh \
105-
&& workspace-install-affected"
34+
- name: Set up the dev container
35+
uses: ./.github/actions/setup-dev-container
10636

10737
- name: Lint the affected projects
10838
run: |
@@ -124,10 +54,10 @@ jobs:
12454
devcontainer exec --workspace-folder ../sage-monorepo bash -c ". ./dev-env.sh \
12555
&& nx affected --target=integration-test"
12656
127-
- name: Scan the affected projects with Sonar
128-
run: |
129-
devcontainer exec --workspace-folder ../sage-monorepo bash -c ". ./dev-env.sh \
130-
&& nx affected --target=sonar"
57+
# - name: Scan the affected projects with Sonar
58+
# run: |
59+
# devcontainer exec --workspace-folder ../sage-monorepo bash -c ". ./dev-env.sh \
60+
# && nx affected --target=sonar"
13161

13262
- name: Publish the images of the affected projects
13363
run: |
@@ -161,78 +91,8 @@ jobs:
16191
- name: Derive appropriate SHAs for base and head for `nx affected` commands
16292
uses: nrwl/nx-set-shas@v3
16393

164-
- name: Set up Yarn cache
165-
uses: actions/cache@v3
166-
with:
167-
path: '/tmp/.yarn/cache'
168-
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
169-
restore-keys: |
170-
${{ runner.os }}-yarn-
171-
172-
- name: Set up Renv cache
173-
uses: actions/cache@v3
174-
with:
175-
path: '/tmp/.cache/R/renv/cache'
176-
key: ${{ runner.os }}-renv-cache-${{ hashFiles('**/renv.lock') }}
177-
restore-keys: |
178-
${{ runner.os }}-renv-cache-
179-
180-
- name: Set up Poetry cache
181-
uses: actions/cache@v3
182-
with:
183-
path: '/tmp/.cache/pypoetry'
184-
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
185-
186-
- name: Set up venv cache
187-
uses: actions/cache@v3
188-
with:
189-
path: |
190-
/tmp/.local/share/virtualenv
191-
**/.venv
192-
key: ${{ runner.os }}-venv-${{ hashFiles('**/poetry.lock') }}
193-
194-
- name: Set up Gradle cache
195-
uses: actions/cache@v3
196-
with:
197-
path: |
198-
/tmp/.gradle/caches
199-
/tmp/.gradle/wrapper
200-
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
201-
restore-keys: |
202-
${{ runner.os }}-gradle-
203-
204-
- name: Install the Dev Container CLI
205-
run: npm install -g @devcontainers/cli@0.49.0
206-
207-
- name: Start the dev container
208-
run: |
209-
mkdir -p \
210-
/tmp/.yarn/cache \
211-
/tmp/.cache/R/renv/cache \
212-
/tmp/.cache/pypoetry \
213-
/tmp/.local/share/virtualenv \
214-
/tmp/.gradle/caches \
215-
/tmp/.gradle/wrapper
216-
217-
devcontainer up \
218-
--mount type=bind,source=/tmp/.yarn/cache,target=/workspaces/sage-monorepo/.yarn/cache \
219-
--mount type=bind,source=/tmp/.cache/R/renv/cache,target=/home/vscode/.cache/R/renv/cache \
220-
--mount type=bind,source=/tmp/.cache/pypoetry,target=/home/vscode/.cache/pypoetry \
221-
--mount type=bind,source=/tmp/.local/share/virtualenv,target=/home/vscode/.local/share/virtualenv \
222-
--mount type=bind,source=/tmp/.gradle/caches,target=/home/vscode/.gradle/caches \
223-
--mount type=bind,source=/tmp/.gradle/wrapper,target=/home/vscode/.gradle/wrapper \
224-
--workspace-folder ../sage-monorepo
225-
226-
- name: Prepare the workspace
227-
run: |
228-
devcontainer exec --workspace-folder ../sage-monorepo bash -c "
229-
sudo chown -R vscode:vscode \
230-
/workspaces/sage-monorepo \
231-
/home/vscode/.cache \
232-
/home/vscode/.local \
233-
/home/vscode/.gradle \
234-
&& . ./dev-env.sh \
235-
&& workspace-install"
94+
- name: Set up the dev container
95+
uses: ./.github/actions/setup-dev-container
23696

23797
- name: Lint the affected projects
23898
run: |

0 commit comments

Comments
 (0)