Skip to content

Commit cf8e994

Browse files
gavindidrichsengavindidrichsen
authored
Merge pull request #106 from puppetlabs/cat_1820
(CAT-1820) Move workflow-restarter to cat_github_actions
2 parents c6a80f3 + 395b3cb commit cf8e994

File tree

9 files changed

+269
-0
lines changed

9 files changed

+269
-0
lines changed

‎.github/actions/workflow-restarter-proxy/action.yml

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
name: 'Workflow Restarter Proxy'
3+
description: |
4+
This custom action acts as a proxy to trigger the reusable workflow that restarts a failed job.
5+
NOTE: This action cannot itself do the re-start because in effect it's composite steps get folded
6+
into the source workflow, the one that "uses" this custom action. Since github does not allow a workflow
7+
to retrigger itself, then the source workflow must be triggered not by this but by another workflow.
8+
Therefore, this custom action triggers that other workflow.
9+
inputs:
10+
repository:
11+
description: 'Should be set to github.repository via the calling workflow'
12+
required: true
13+
run_id:
14+
description: 'Should be set to github.run_id via the calling workflow'
15+
required: true
16+
runs:
17+
using: 'composite'
18+
steps:
19+
# ABORT if not SOURCE_GITHUB_TOKEN environment variable set
20+
- name: Check for presence of SOURCE_GITHUB_TOKEN environment variable
21+
shell: bash
22+
run: |
23+
if [[ -z "${{ env.SOURCE_GITHUB_TOKEN }}" ]]; then
24+
echo "ERROR: \$SOURCE_GITHUB_TOKEN must be set by the calling workflow" 1>&2 && exit 1
25+
fi
26+
27+
# checkout the repository because I want bundler to have access to my Gemfile
28+
- name: Checkout repository
29+
uses: actions/checkout@v4
30+
31+
# setup ruby including a bundle install of my Gemfile
32+
- name: Set up Ruby and install Octokit
33+
uses: ruby/setup-ruby@v1
34+
with:
35+
ruby-version: '3'
36+
bundler-cache: true # 'bundle install' will be run and gems cached for faster workflow runs
37+
38+
# Trigger the reusable workflow
39+
- name: Trigger reusable workflow
40+
shell: bash
41+
run: |
42+
gem install octokit
43+
ruby -e "
44+
require 'octokit'
45+
client = Octokit::Client.new(:access_token => '${{ env.SOURCE_GITHUB_TOKEN }}')
46+
client.post(
47+
'/repos/${{ inputs.repository }}/actions/workflows/workflow-restarter.yml/dispatches',
48+
{
49+
ref: 'main',
50+
inputs: {
51+
repo: '${{ inputs.repository }}',
52+
run_id: '${{ inputs.run_id }}'
53+
}
54+
}
55+
)
56+
"

‎.github/workflows/workflow-restarter-test.yml

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Workflow Restarter TEST
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
fail:
7+
description: >
8+
For (acceptance, unit) jobs:
9+
'true' = (fail, succeed) and
10+
'false' = (succeed, fail)
11+
required: true
12+
default: 'true'
13+
env:
14+
SOURCE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
15+
16+
jobs:
17+
unit:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- name: Check outcome
21+
run: |
22+
if [ "${{ github.event.inputs.fail }}" = "true" ]; then
23+
echo "'unit' job succeeded"
24+
exit 0
25+
else
26+
echo "'unit' job failed"
27+
exit 1
28+
fi
29+
acceptance:
30+
runs-on: ubuntu-latest
31+
steps:
32+
- name: Check outcome
33+
run: |
34+
if [ "${{ github.event.inputs.fail }}" = "true" ]; then
35+
echo "'acceptance' job failed"
36+
exit 1
37+
else
38+
echo "'acceptance' job succeeded"
39+
exit 0
40+
fi
41+
42+
on-failure-workflow-restarter-proxy:
43+
# (1) run this job after the "acceptance" job and...
44+
needs: [acceptance, unit]
45+
# (2) continue ONLY IF "acceptance" fails
46+
if: always() && needs.acceptance.result == 'failure' || needs.unit.result == 'failure'
47+
runs-on: ubuntu-latest
48+
steps:
49+
# (3) checkout this repository in order to "see" the following custom action
50+
- name: Checkout repository
51+
uses: actions/checkout@v4
52+
53+
- name: Trigger reusable workflow
54+
uses: "puppetlabs/cat-github-actions/.github/actions/workflow-restarter-proxy@main"
55+
env:
56+
SOURCE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
57+
with:
58+
repository: ${{ github.repository }}
59+
run_id: ${{ github.run_id }}

‎.github/workflows/workflow-restarter.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
name: Workflow Restarter
3+
on:
4+
workflow_call:
5+
inputs:
6+
repo:
7+
description: "GitHub repository name."
8+
required: true
9+
type: string
10+
run_id:
11+
description: "The ID of the workflow run to rerun."
12+
required: true
13+
type: string
14+
retries:
15+
description: "The number of times to retry the workflow run."
16+
required: false
17+
type: string
18+
default: "3"
19+
20+
jobs:
21+
rerun:
22+
runs-on: ubuntu-latest
23+
steps:
24+
- name: Checkout code
25+
uses: actions/checkout@v4
26+
27+
- name: Check retry count
28+
id: check-retry
29+
run: |
30+
# IF `--attempts` returns a non-zero exit code, then keep retrying
31+
status_code=$(gh run view ${{ inputs.run_id }} --repo ${{ inputs.repo }} --attempt ${{ inputs.retries }} --json status) || {
32+
echo "Retry count is within limit"
33+
echo "::set-output name=should_retry::true"
34+
exit 0
35+
}
36+
37+
# ELSE `--attempts` returns a zero exit code, so stop retrying
38+
echo "Retry count has reached the limit"
39+
echo "::set-output name=should_retry::false"
40+
env:
41+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
42+
43+
- name: Re-run failed jobs
44+
if: ${{ steps.check-retry.outputs.should_retry == 'true' }}
45+
run: gh run rerun --failed ${{ inputs.run_id }} --repo ${{ inputs.repo }}
46+
continue-on-error: true
47+
env:
48+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎docs/image-1.png

97.7 KB
Loading

‎docs/image-2.png

59.7 KB
Loading

‎docs/image-3.png

76.5 KB
Loading

‎docs/image-4.png

98.4 KB
Loading

‎docs/image.png

82.8 KB
Loading

‎docs/workflow-restarter.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# workflow-restarter
2+
3+
## Description
4+
5+
Although GitHub provides built-in programatic mechanisms for retrying individual steps within a workflow, it doesn't provide one for retrying entire workflows. One possible reason for this limitation may be to prevent accidental infinite retry loops around failing workflows. Any workflow that fails, however, can be manually re-started from the failed workflow on the `Actions` tab of the repository. For more information on restarting github worklows see [Re-running workflows and jobs](https://docs.github.com/en/actions/managing-workflow-runs/re-running-workflows-and-jobs).
6+
7+
Nevertheless, it is possible to programmatically restart a workflow after it fails and the section below shows how to restart a failing workflow 3 times using the `workflow-restarter` re-usable workflow.
8+
9+
## Usage
10+
11+
If setting up the the `workflow-restarter` for the first time, then make sure to initialize it first and then configure another workflow to programmatically restart on failure.
12+
13+
### Initialize the `Workflow Restarter`
14+
15+
First, configure the `workflow-restarter-proxy` custom action by creating a `workflow-restarter.yml` file beneath the `.github/workflows` directory in your repository.
16+
17+
Second, configure the `workflow-restarter` re-usable workflow:
18+
19+
```yaml
20+
name: Workflow Restarter
21+
on:
22+
workflow_dispatch:
23+
inputs:
24+
repo:
25+
description: "GitHub repository name."
26+
required: true
27+
type: string
28+
run_id:
29+
description: "The ID of the workflow run to rerun."
30+
required: true
31+
type: string
32+
retries:
33+
description: "The number of times to retry the workflow run."
34+
required: false
35+
type: string
36+
default: "3"
37+
38+
jobs:
39+
call-reusable-workflow:
40+
uses: puppetlabs/cat-github-actions/.github/workflows/workflow-restarter.yml@main
41+
with:
42+
repo: ${{ inputs.repo }}
43+
run_id: ${{ inputs.run_id }}
44+
retries: ${{ inputs.retries }}
45+
```
46+
47+
Finally, verify that the `workflow-restarter.yml` performs as expected:
48+
1. Add a `workflow-restarter-test.yml` file to `.github/workflows`, copy the contents of `./github/workflows/workflow-restarter-test` from this repository
49+
2. Kick off the `workflow-restarter-test` and it should fail and be re-started 3 times. For example output see the [appendix below](#verify-workflow-restarter-with-workflow-restarter-test).
50+
51+
### Configure an existing workflow to use `on-failure-workflow-restarter`
52+
53+
Now add something like the following `yaml` job at the end of your workflow, changing only the `needs` section to suit.
54+
55+
For example, the following will trigger a restart if either the `acceptance` or the `unit` jobs preceeding it fail. A restart of the failing jobs will be attempted 3 times at which point if the failing jobs continue to fail, then the workflow will be marked as failed. If, however, at any point the `acceptance` and `unit` both pass fine then the restarted workflow will be marked as successful
56+
57+
```yaml
58+
on-failure-workflow-restarter-proxy:
59+
# (1) run this job after the "acceptance" job and...
60+
needs: [acceptance, unit]
61+
# (2) continue ONLY IF "acceptance" fails
62+
if: always() && needs.acceptance.result == 'failure' || needs.unit.result == 'failure'
63+
runs-on: ubuntu-latest
64+
steps:
65+
# (3) checkout this repository in order to "see" the following custom action
66+
- name: Checkout repository
67+
uses: actions/checkout@v2
68+
69+
# (4) "use" the custom action to retrigger the failed "acceptance job" above
70+
# NOTE: pass the SOURCE_GITHUB_TOKEN to the custom action because (a) it must have
71+
# this to trigger the reusable workflow that restarts the failed job; and
72+
# (b) custom actions do not have access to the calling workflow's secrets
73+
- name: Trigger reusable workflow
74+
uses: ./.github/actions/workflow-restarter-proxy
75+
env:
76+
SOURCE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
77+
with:
78+
repository: ${{ github.repository }}
79+
run_id: ${{ github.run_id }}
80+
```
81+
82+
## Appendix
83+
84+
### Verify `Workflow Restarter` with `Workflow Restarter TEST`
85+
86+
The following shows 3 `Workflow Restarter` occuring after the `Workflow Restarter TEST`, which is set to fail continuously.
87+
88+
![alt text](image.png)
89+
90+
Looking closer at the `Workflow Restarter TEST` reveals
91+
92+
* that the workflow includes 2 jobs `unit` and `acceptance`; and
93+
* that the workflow has been re-run 3 times, e.g.,
94+
95+
![alt text](image-1.png)
96+
97+
Further, the following sequence of screenshots shows that only failed jobs are re-run.
98+
99+
* The `on-failure-workflow-restarter` job **(1)** is triggered by the failure of the `unit` job and **(2)** successfully calls the `workflow-restarter` workflow
100+
* The `workflow-restarter` in turn triggers a re-run of the `unit` job **(3)** and the `Workflow Restarter TEST` shows this as an incremented attempt count at **(4)**.
101+
102+
![alt text](image-2.png)
103+
104+
![alt text](image-3.png)
105+
106+
![alt text](image-4.png)

0 commit comments

Comments
 (0)