Skip to content

Commit d79c129

Browse files
EbonsignoriEbonsignori
authored
Fixes for docs-staging-x repos workflows (#53864)
1 parent 6a8499c commit d79c129

File tree

12 files changed

+264
-69
lines changed

12 files changed

+264
-69
lines changed

‎.github/workflows/sync-staging-repo-files.yml

+44-15
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@ on:
1313
- 'src/deployments/staging/.github/**'
1414
- 'src/deployments/staging/Dockerfile'
1515
- 'src/deployments/staging/.env.example'
16-
- 'src/deployments/staging/README.example.md'
16+
- 'src/deployments/staging/README.staging.md'
17+
- 'src/deployments/staging/ownership.yaml'
1718
- 'src/deployments/staging/config/**'
1819

1920
permissions:
@@ -47,18 +48,21 @@ jobs:
4748
id: generate-repos
4849
run: |
4950
NUMBER_OF_REPOS=${{ steps.read-config.outputs.number_of_repos }}
51+
# Since we use 0-based index e.g. docs-staging-0, we need to subtract 1
52+
END=$((NUMBER_OF_REPOS - 1))
5053
repos=()
51-
for i in $(seq 0 $NUMBER_OF_REPOS); do
54+
for i in $(seq 0 $END); do
5255
repos+=("{\"repo\": \"github/docs-staging-$i\", \"index\": $i}")
5356
done
5457
json_repos=$(printf '%s\n' "${repos[@]}" | jq -s -c .)
5558
echo "repos=$json_repos" >> $GITHUB_OUTPUT
5659
57-
- name: Set matrix output with repo and index
60+
- name: Set matrix
5861
id: set-matrix
5962
run: |
60-
repos=${{ steps.generate-repos.outputs.repos }}
61-
echo "matrix={\"include\": $repos}" >> $GITHUB_OUTPUT
63+
echo "matrix={\"include\": $REPOS}" >> $GITHUB_OUTPUT
64+
env:
65+
REPOS: ${{ steps.generate-repos.outputs.repos }}
6266

6367
- uses: ./.github/actions/slack-alert
6468
if: ${{ failure() && github.event_name != 'workflow_dispatch' }}
@@ -77,17 +81,20 @@ jobs:
7781
- name: Checkout source repository
7882
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
7983
with:
80-
fetch-depth: 0
84+
# Only need latest commits to sync with
85+
fetch-depth: 2
8186

8287
- name: Checkout target repository
8388
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
8489
with:
8590
repository: ${{ matrix.repo }}
86-
token: ${{ secrets.DOCS_BOT_PAT_READPUBLICKEY }}
91+
token: ${{ secrets.DOCS_BOT_PAT_WORKFLOW }}
8792
path: target_repo
88-
fetch-depth: 0
93+
fetch-depth: 2
8994

9095
- name: Synchronize files to target repo
96+
env:
97+
INDEX: ${{ matrix.index }}
9198
run: |
9299
# Create necessary directories if they DNE
93100
mkdir -p target_repo/build-scripts
@@ -111,11 +118,14 @@ jobs:
111118
cp src/deployments/staging/.env.example target_repo/.env
112119
fi
113120
114-
# Conditional copy for README.md if not present
115-
if [ ! -f target_repo/README.md ]; then
116-
cp src/deployments/staging/README.example.md target_repo/README.md
121+
# Only copy README.md for non-review servers e.g. index >= 2
122+
if [ "$INDEX" -ge 2 ]; then
123+
cp src/deployments/staging/README.staging.md target_repo/README.md
117124
fi
118125
126+
# Copy ownership.yaml
127+
cp src/deployments/staging/ownership.yaml target_repo/ownership.yaml
128+
119129
- name: Install jq
120130
run: sudo apt-get update && sudo apt-get install -y jq
121131

@@ -146,17 +156,36 @@ jobs:
146156
# If any files still contain {{x}}, replace them with the current index
147157
find target_repo -type f -exec sed -i "s/{{x}}/$INDEX/g" {} +
148158
149-
- name: Commit and push changes
159+
- name: Commit Changes to new branch
160+
id: commit_changes
150161
run: |
162+
BRANCH_NAME=sync-staging-files-${{ github.run_id }}
151163
cd target_repo
152164
git config user.name "github-actions[bot]"
153165
git config user.email "github-actions[bot]@users.noreply.github.com"
166+
git checkout -b $BRANCH_NAME
154167
git add .
155-
# If there are changes, commit and push
156168
if ! git diff --cached --quiet; then
157-
git commit -m "Synchronize files from source repository with index ${{ matrix.index }}"
158-
git push
169+
git commit -m "Synchronize files from source repository with docs-staging-${{ matrix.index }}"
170+
git push origin $BRANCH_NAME
159171
fi
172+
echo "BRANCH_NAME=${BRANCH_NAME}" >> $GITHUB_OUTPUT
173+
174+
# We want to create a PR instead of committing directly in order to trigger a deployment
175+
- name: Create Pull Request
176+
id: create_pr
177+
env:
178+
GH_TOKEN: ${{ secrets.DOCS_BOT_PAT_READPUBLICKEY }}
179+
run: |
180+
cd target_repo
181+
PR_URL=$(gh pr create \
182+
--title "Sync files from docs-internal" \
183+
--body "This PR synchronized the files of this repo with the source of truth files in doc-internal. The PR should automatically merge, if it doesn't please fix files in docs-internal so that the fix is applied to every docs-staging-x repo." \
184+
--base main \
185+
--head ${{ steps.commit_changes.outputs.BRANCH_NAME }} \
186+
)
187+
# Enable auto-merge on PR
188+
gh pr merge $PR_URL --auto --squash
160189
161190
- uses: ./.github/actions/slack-alert
162191
if: ${{ failure() && github.event_name != 'workflow_dispatch' }}

‎.github/workflows/update-docs-staging-x-repo.yml

+8-1
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ on:
1818
types: [labeled]
1919

2020
permissions:
21-
contents: read
21+
contents: write
2222

2323
jobs:
2424
dispatch-sha:
@@ -72,6 +72,13 @@ jobs:
7272
# Get the commit SHA from the pull request head
7373
COMMIT_SHA="${{ github.event.pull_request.head.sha }}"
7474
75+
# Update the docs-staging-x branch to the latest SHA from the PR branch
76+
git config user.name "github-actions[bot]"
77+
git config user.email "github-actions[bot]@users.noreply.github.com"
78+
git checkout docs-staging-$STAGING_NUMBER || { echo "Failed to checkout docs-staging-$STAGING_NUMBER"; exit 1; }
79+
git reset --hard $COMMIT_SHA || { echo "Git reset failed"; exit 1; }
80+
git push origin docs-staging-$STAGING_NUMBER --force || { echo "Git push failed"; exit 1; }
81+
7582
else
7683
echo "Event type $EVENT_TYPE not supported."
7784
echo "should_dispatch=false" >> $GITHUB_OUTPUT

‎src/deployments/staging/.github/workflows/deploy-on-repo-dispatch.yml

+18-14
Original file line numberDiff line numberDiff line change
@@ -22,51 +22,55 @@ jobs:
2222
- name: Checkout Repository
2323
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
2424
with:
25-
fetch-depth: 0 # Ensure full history for PR creation
25+
fetch-depth: 2
2626

27-
# Extract SHA from the dispatch payload
27+
# Extract SHA from the dispatch payload and set it as an output
2828
- name: Set SHA from Payload
2929
id: set_sha
30-
run: echo "SHA=${{ github.event.client_payload.SHA }}" >> $GITHUB_ENV
30+
run: echo "SHA=${{ github.event.client_payload.SHA }}" >> $GITHUB_OUTPUT
3131

3232
# Update the .env file with the new SHA
3333
- name: Update .env File
3434
run: |
35+
SHA=${{ steps.set_sha.outputs.SHA }}
3536
if grep -q "^SHA=" .env; then
3637
sed -i "s/^SHA=.*/SHA=${SHA}/" .env
3738
else
3839
echo "SHA=${SHA}" >> .env
40+
fi
3941
4042
- name: Commit Changes to new branch
43+
id: commit_changes
4144
run: |
4245
BRANCH_NAME=update-sha-${{ github.run_id }}
4346
git config user.name "github-actions[bot]"
4447
git config user.email "github-actions[bot]@users.noreply.github.com"
4548
git checkout -b $BRANCH_NAME
4649
git add .env
47-
git commit -m "Update SHA to ${{ env.SHA }}"
48-
echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
50+
git commit -m "Update SHA to ${{ steps.set_sha.outputs.SHA }}"
51+
echo "BRANCH_NAME=${BRANCH_NAME}" >> $GITHUB_OUTPUT
4952
5053
- name: Push Branch
51-
run: git push origin ${{ env.BRANCH_NAME }}
54+
run: git push origin ${{ steps.commit_changes.outputs.BRANCH_NAME }}
5255

56+
# Create a Pull Request and set the PR URL as an output
5357
- name: Create Pull Request
5458
id: create_pr
5559
env:
5660
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
5761
run: |
5862
PR_URL=$(gh pr create \
59-
--title "Update SHA to ${{ env.SHA }}" \
60-
--body "This PR updates the SHA in the \`.env\` file to \`${{ env.SHA }}\`." \
63+
--title "Update SHA to ${{ steps.set_sha.outputs.SHA }}" \
64+
--body "This PR updates the SHA in the \`.env\` file to \`${{ steps.set_sha.outputs.SHA }}\`." \
6165
--base main \
62-
--head ${{ env.BRANCH_NAME }} \
63-
--json url \
64-
--jq .url)
65-
echo "PR_URL=$PR_URL" >> $GITHUB_ENV
66+
--head ${{ steps.commit_changes.outputs.BRANCH_NAME }} \
67+
)
68+
echo "PR_URL=${PR_URL}" >> $GITHUB_OUTPUT
6669
70+
# Merge the Pull Request
6771
- name: Merge Pull Request
6872
env:
6973
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
7074
run: |
71-
PR_NUMBER=$(gh pr view $PR_URL --json number --jq .number)
72-
gh pr merge $PR_NUMBER --merge --delete-branch --auto --squash --yes
75+
PR_NUMBER=$(gh pr view ${{ steps.create_pr.outputs.PR_URL }} --json number --jq .number)
76+
gh pr merge $PR_NUMBER --auto --squash

‎src/deployments/staging/.github/workflows/moda-ci.yaml

+95
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
name: docs-staging-{{x}} Moda CI
2+
3+
# More info on CI actions setup can be found here:
4+
# https://github.com/github/ops/blob/master/docs/playbooks/build-systems/moving-moda-apps-from-bp-to-actions.md
5+
6+
on:
7+
workflow_dispatch:
8+
push:
9+
branches-ignore:
10+
- 'gh-readonly-queue/**'
11+
merge_group:
12+
types: [checks_requested]
13+
14+
jobs:
15+
##########################
16+
# Add DOCS_BOT_PAT_READPUBLICKEY to vault-keys
17+
##########################
18+
set-vault-keys:
19+
runs-on: ubuntu-latest
20+
outputs:
21+
modified_vault_keys: ${{ steps.modify_vault_keys.outputs.modified }}
22+
steps:
23+
- name: Set vault-keys output
24+
id: modify_vault_keys
25+
run: |
26+
if [ -z "${{ vars.VAULT_KEYS }}" ]; then
27+
echo "modified=DOCS_BOT_PAT_READPUBLICKEY" >> $GITHUB_OUTPUT
28+
else
29+
echo "modified=${{ vars.VAULT_KEYS }},DOCS_BOT_PAT_READPUBLICKEY" >> $GITHUB_OUTPUT
30+
fi
31+
32+
#############
33+
# Moda jobs
34+
#############
35+
moda-config-bundle:
36+
name: ${{ matrix.ci_job.job }}
37+
needs: set-vault-keys
38+
strategy:
39+
fail-fast: false
40+
matrix:
41+
ci_job: [{ 'job': 'docs-staging-{{x}}-moda-config-bundle' }]
42+
uses: github/internal-actions/.github/workflows/moda.yml@main
43+
with:
44+
ci-formatted-job-name: ${{ matrix.ci_job.job }}
45+
vault-keys: ${{ needs.set-vault-keys.outputs.modified_vault_keys }}
46+
secrets:
47+
dx-bot-token: ${{ secrets.INTERNAL_ACTIONS_DX_BOT_ACCOUNT_TOKEN }}
48+
datadog-api-key: ${{ secrets.DATADOG_API_KEY }}
49+
50+
#############
51+
# Docker Image jobs
52+
#############
53+
docker-image:
54+
name: ${{ matrix.ci_job.job }}
55+
needs: set-vault-keys
56+
strategy:
57+
fail-fast: false
58+
matrix:
59+
ci_job: [{ 'job': 'docs-staging-{{x}}-docker-image' }]
60+
uses: github/internal-actions/.github/workflows/kube.yml@main
61+
with:
62+
ci-formatted-job-name: ${{ matrix.ci_job.job }}
63+
# Fetches the 'DOCS_BOT_PAT_READPUBLICKEY' secret from Vault to pass to the docker build environment as --secret id=DOCS_BOT_PAT_READPUBLICKEY,src=$(cat DOCS_BOT_PAT_READPUBLICKEY)
64+
vault-keys: ${{ needs.set-vault-keys.outputs.modified_vault_keys }}
65+
docker-build-env-secrets: 'DOCS_BOT_PAT_READPUBLICKEY'
66+
secrets:
67+
dx-bot-token: ${{ secrets.INTERNAL_ACTIONS_DX_BOT_ACCOUNT_TOKEN }}
68+
datadog-api-key: ${{ secrets.DATADOG_API_KEY }}
69+
70+
#############
71+
# Docker Security jobs
72+
#############
73+
docker-security:
74+
name: ${{ matrix.ci_job.job }}
75+
needs: set-vault-keys
76+
strategy:
77+
fail-fast: false
78+
matrix:
79+
ci_job: [{ 'job': 'docs-staging-{{x}}-docker-security' }]
80+
uses: github/internal-actions/.github/workflows/docker_security.yml@main
81+
with:
82+
ci-formatted-job-name: ${{ matrix.ci_job.job }}
83+
# Fetches the 'DOCS_BOT_PAT_READPUBLICKEY' secret from Vault to pass to the docker build environment as --secret id=DOCS_BOT_PAT_READPUBLICKEY,src=$(cat DOCS_BOT_PAT_READPUBLICKEY)
84+
vault-keys: ${{ needs.set-vault-keys.outputs.modified_vault_keys }}
85+
docker-build-env-secrets: 'DOCS_BOT_PAT_READPUBLICKEY'
86+
secrets:
87+
dx-bot-token: ${{ secrets.INTERNAL_ACTIONS_DX_BOT_ACCOUNT_TOKEN }}
88+
datadog-api-key: ${{ secrets.DATADOG_API_KEY }}
89+
90+
permissions:
91+
actions: read
92+
checks: read
93+
contents: read
94+
statuses: read
95+
id-token: write

‎src/deployments/staging/README.example.md

-12
This file was deleted.

‎src/deployments/staging/README.md

+10-6
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ The dedicated review servers are deployed in a similar fashion that the staging
2626

2727
Each staging server requires its own `github/` repo in order to deploy to Moda in the form of `github/docs-staging-X` where X is the number of that staging server e.g. `github/docs-staging-0` or `github/docs-staging-1`.
2828

29-
The URLs of the staging servers also follow this pattern, `docs-staging-x.github.net`, e.g. `docs-staging-2.github.net`
29+
The URLs of the staging servers also follow this pattern, `https://docs-staging-{{x}}.service.iad.github.net`, e.g. `https://docs-staging-2.service.iad.github.net`
3030

3131
With the exception of the first 2 which are our review servers:
3232

@@ -37,10 +37,13 @@ Ideally there should always be enough staging servers for each developer on the
3737

3838
So we have 8 dedicated staging servers, `docs-staging-{2-9}`:
3939

40-
- `docs-staging-2` -> https://docs-staging-2.github.net
41-
- `docs-staging-3` -> https://docs-staging-3.github.net
40+
- `docs-staging-2` -> https://docs-staging-2.service.iad.github.net
41+
- `docs-staging-3` -> https://docs-staging-3.service.iad.github.net
4242
- etc
43-
- `docs-staging-9` -> https://docs-staging-9.github.net
43+
- `docs-staging-9` -> https://docs-staging-9.service.iad.github.net
44+
45+
> [!NOTE]
46+
> [Developer VPN](https://thehub.github.com/security/security-operations/developer-vpn-access/) access is required to view a staging server. Initial set up takes some work, but connecting to it after it's configured is rather simple.
4447
4548
## How do staging deploys work from docs-internal?
4649

@@ -60,7 +63,7 @@ sequenceDiagram
6063
WF2->WF2: 1. Extracts SHA from `repository_dispatch` event <br />2. Updates `.env` in docs-staging-x with SHA value<br />3. Auto-merges the PR into docs-staging-x
6164
WF2->MD: Auto-merge kicks off Moda deploy
6265
MD->MD: Dockerfile build clones docs-internal code from SHA target set in `.env`
63-
note over MD: Deployed to <br/> `docs-staging-X.github.net`
66+
note over MD: Deployed to <br/> `https://docs-staging-{{x}}.service.iad.github.net`
6467
```
6568

6669
Whenever a developer pushes code to a staging branch in `docs-internal`, e.g. `docs-staging-2`, a pipeline begins with the final result being a staging server running with the latest changes from that branch. See the above diagram, or read below for a textual explanation.
@@ -81,13 +84,14 @@ The pipeline is as follows:
8184

8285
1. The PR merge kicks off an automatic Moda deploy for the `docs-staging-X` server.
8386

84-
1. At build time, the [Dockerfile](./Dockerfile) clones the `SHA` from `docs-internal` and builds, runs, and deploys it to https://docs-staging-X.github.net which is only accessible behind the devvpn.
87+
1. At build time, the [Dockerfile](./Dockerfile) clones the `SHA` from `docs-internal` and builds, runs, and deploys it to `https://docs-staging-{{x}}.service.iad.github.net` which is only accessible behind the [Developer VPN](https://thehub.github.com/security/security-operations/developer-vpn-access/).
8588

8689
## How do review server deploys work from docs-internal?
8790

8891
The process is very similar to the process in the previous section for staging servers. The differences are as follows:
8992

9093
1. Review servers live in:
94+
9195
1. Repo [docs-staging-0](https://github.com/github/doc-staging-0) (internal) @ https://docs-review.github.com
9296
1. Repo [docs-staging-1](https://gthub.com/github/doc-staging-1) (external) @ https://os-docs-review.github.com
9397

0 commit comments

Comments
 (0)