Skip to content

Commit 474e148

Browse files
dshemetovdshemetov
committed
Merge branch 'nancodes' into nans_chng
2 parents 5a60716 + 19e8be4 commit 474e148

File tree

580 files changed

+48965
-3391
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

580 files changed

+48965
-3391
lines changed

‎.bumpversion.cfg

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
[bumpversion]
2+
current_version = 0.1.13
3+
commit = True
4+
message = chore: bump covidcast-indicators to {new_version}
5+
tag = False

‎.github/CONTRIBUTING.md

+46-11
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Admins will assign issues to one or more people based on balancing expediency, e
2626

2727
So, how does one go about developing a pipeline for a new data source?
2828

29-
**tl;dr**
29+
### tl;dr
3030

3131
1. Create your new indicator branch from `main`.
3232
2. Build it using the appropriate template, following the guidelines in the included README.md and REVIEW.md files.
@@ -52,7 +52,7 @@ git checkout -b dev-my-feature-branch
5252

5353
Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. If using Python, add the name of the directory to the list found in `jobs > build > strategy > matrix > packages` in `.github/workflows/python-ci.yml`, which will enable automated checks for your indicator when you make PRs. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
5454

55-
- Consult your peers with questions! :handshake:
55+
* Consult your peers with questions! :handshake:
5656

5757
Once you have something that runs locally and passes tests you set up your remote branch eventual review and production deployment.
5858

@@ -75,9 +75,9 @@ becomes available to the public.
7575

7676
Once you have your branch set up you should get in touch with a platform engineer to pair up on the remaining production needs. These include:
7777

78-
- Adding the necessary Jenkins scripts for your indicator.
79-
- Preparing the runtime host with any Automation configuration necessities.
80-
- Reviewing the workflow to make sure it meets the general guidelines and will run as expected on the runtime host.
78+
* Adding the necessary Jenkins scripts for your indicator.
79+
* Preparing the runtime host with any Automation configuration necessities.
80+
* Reviewing the workflow to make sure it meets the general guidelines and will run as expected on the runtime host.
8181

8282
Once all the last mile configuration is in place you can create a pull request against `prod` to initiate the CI/CD pipeline which will build, test, and package your indicator for deployment.
8383

@@ -97,11 +97,46 @@ Currently, the production indicators all live and run on the venerable and peren
9797

9898
We use a branch-based git workflow coupled with [Jenkins](https://www.jenkins.io/) and [Ansible](https://www.ansible.com/) to build, test, package, and deploy each indicator individually to the runtime host.
9999

100-
- Jenkins dutifully manages the whole process for us by executing several "stages" in the context of a [CI/CD pipeline](https://dzone.com/articles/learn-how-to-setup-a-cicd-pipeline-from-scratch). Each stage does something unique, building on the previous stage. The stages are:
101-
- Environment - Sets up some environment-specific needs that the other stages depend on.
102-
- Build - Create the Python venv on the Jenkins host.
103-
- Test - Run linting and unit tests.
104-
- Package - Tar and gzip the built environment.
105-
- Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary).
100+
* Jenkins dutifully manages the whole process for us by executing several "stages" in the context of a [CI/CD pipeline](https://dzone.com/articles/learn-how-to-setup-a-cicd-pipeline-from-scratch). Each stage does something unique, building on the previous stage. The stages are:
101+
* Environment - Sets up some environment-specific needs that the other stages depend on.
102+
* Build - Create the Python venv on the Jenkins host.
103+
* Test - Run linting and unit tests.
104+
* Package - Tar and gzip the built environment.
105+
* Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary).
106106

107107
There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host. It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release.
108+
109+
### Preparing container images of indicators
110+
111+
It may be desirable to build a container image from an indicator. To do this:
112+
113+
* Edit the `.github/workflows/build-container-images.yml` file and add your indicator directory name to the `matrix.packages` section of the `jobs:` block:
114+
115+
```yaml
116+
...
117+
jobs:
118+
build:
119+
runs-on: ubuntu-latest
120+
strategy:
121+
matrix:
122+
packages: [ new_indicator ] # indicator directory name
123+
...
124+
```
125+
126+
* Create a suitable Dockerfile in the root of your indicator directory.
127+
128+
* GitHub Actions will try to build this for you and register it in our private repo.
129+
130+
Currently we will build container images off of `main` and `prod` branches. These can be pulled by systems or humans that have access to the registry.
131+
132+
* `main` builds create a registered image of:
133+
134+
```text
135+
ghcr.io/cmu-delphi/covidcast-indicators-${indicator_name}:dev
136+
```
137+
138+
* `prod` builds create a registered image of:
139+
140+
```text
141+
ghcr.io/cmu-delphi/covidcast-indicators-${indicator_name}:latest
142+
```

‎.github/ISSUE_TEMPLATE/data_quality_issue.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: Data quality issue
33
about: Missing data, weird data, broken data
44
title: ''
55
labels: 'data quality'
6-
assignees: 'nmdefries'
6+
assignees: 'krivard'
77
---
88

99
**Actual Behavior:**

‎.github/ISSUE_TEMPLATE/feature_release.md

-31
This file was deleted.

‎.github/workflows/build-container-images.yml

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Build indicator container images and upload to registry
2+
3+
on:
4+
push:
5+
branches: [ main, prod ]
6+
7+
jobs:
8+
build:
9+
runs-on: ubuntu-latest
10+
strategy:
11+
matrix:
12+
packages: [ "" ]
13+
steps:
14+
- name: Checkout code
15+
uses: actions/checkout@v2
16+
17+
- name: Login to GitHub Container Registry
18+
uses: docker/login-action@v1
19+
with:
20+
registry: ghcr.io
21+
username: cmu-delphi-deploy-machine
22+
password: ${{ secrets.CMU_DELPHI_DEPLOY_MACHINE_PAT }}
23+
24+
- name: Build, tag, and push image to Github
25+
run: |
26+
baseRef="${GITHUB_REF#*/}"
27+
baseRef="${baseRef#*/}"
28+
case "${baseRef}" in
29+
main)
30+
imageTag="dev"
31+
;;
32+
prod)
33+
imageTag="latest"
34+
;;
35+
*)
36+
imageTag="${baseRef//\//_}" # replace `/` with `_` in branch name
37+
;;
38+
esac
39+
if [ -z ${{ matrix.packages }} ]; then
40+
echo "The matrix list is empty so we will not build any images."
41+
else
42+
cd ${{ github.workspace }}/${{ matrix.packages }}
43+
echo "using tag: --${imageTag}--"
44+
docker build -t ghcr.io/${{ github.repository }}-${{ matrix.packages }}:$imageTag --file Dockerfile .
45+
docker push ghcr.io/${{ github.repository }}-${{ matrix.packages }}:$imageTag
46+
fi

‎.github/workflows/create-delphi-utils-release.yml

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: (Deprecated) Create Delphi Utils Release
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
versionName:
7+
description: 'Semantic Version Number (i.e., 5.5.0 or patch, minor, major, prepatch, preminor, premajor, prerelease)'
8+
required: true
9+
default: patch
10+
11+
jobs:
12+
create_release:
13+
runs-on: ubuntu-latest
14+
defaults:
15+
run:
16+
working-directory: _delphi_utils_python
17+
steps:
18+
- name: Check out code
19+
uses: actions/checkout@v2
20+
with:
21+
ref: prod
22+
ssh-key: ${{ secrets.CMU_DELPHI_DEPLOY_MACHINE_SSH }}
23+
- name: Reset prod branch
24+
run: |
25+
git fetch origin main:main
26+
git reset --hard main
27+
- name: Set up Python 3.8
28+
uses: actions/setup-python@v2
29+
with:
30+
python-version: 3.8
31+
- name: Change version number
32+
id: version
33+
run: |
34+
python -m pip install bump2version
35+
echo -n "::set-output name=next_tag::"
36+
bump2version --list ${{ github.event.inputs.versionName }} | grep ^new_version | sed -r s,"^.*=",,
37+
- name: Create pull request into prod
38+
uses: peter-evans/create-pull-request@v3
39+
with:
40+
branch: release/${{ steps.version.outputs.next_tag }}
41+
commit-message: 'chore: release ${{ steps.version.outputs.next_tag }}'
42+
base: prod
43+
title: Release ${{ steps.version.outputs.next_tag }}
44+
labels: chore
45+
reviewers: krivard
46+
assignees: krivard
47+
body: |
48+
Releasing ${{ steps.version.outputs.next_tag }}.

‎.github/workflows/create-release.yml

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: Create Release
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
versionName:
7+
description: 'Semantic Version Number (i.e., 5.5.0 or patch, minor, major, prepatch, preminor, premajor, prerelease)'
8+
required: true
9+
default: patch
10+
11+
jobs:
12+
create-release:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- name: Check out code
16+
uses: actions/checkout@v2
17+
with:
18+
ref: prod
19+
ssh-key: ${{ secrets.CMU_DELPHI_DEPLOY_MACHINE_SSH }}
20+
- name: Reset prod branch
21+
run: |
22+
git fetch origin main:main
23+
git reset --hard main
24+
git config --global user.email [email protected]
25+
git config --global user.name "Delphi Deploy Bot"
26+
- name: Set up Python 3.8
27+
uses: actions/setup-python@v2
28+
with:
29+
python-version: 3.8
30+
- name: Install bump2version
31+
run: python -m pip install bump2version
32+
- name: Check for delphi-utils changes
33+
uses: dorny/paths-filter@v2
34+
id: changes
35+
with:
36+
base: 'prod'
37+
ref: 'main'
38+
filters: |
39+
utils:
40+
- '_delphi_utils_python/**'
41+
- name: Bump delphi-utils version
42+
id: utils-changed
43+
if: steps.changes.outputs.utils == 'true'
44+
working-directory: ./_delphi_utils_python
45+
run: |
46+
echo -n "::set-output name=version::"
47+
bump2version --list ${{ github.event.inputs.versionName }} | grep ^new_version | sed -r s,"^.*=",,
48+
echo -e "\n::set-output name=msg::(*new*)"
49+
- name: Detect delphi-utils version
50+
id: utils-unchanged
51+
if: steps.changes.outputs.utils == 'false'
52+
working-directory: ./_delphi_utils_python
53+
run: |
54+
echo -n "::set-output name=version::"
55+
bump2version --list -n ${{ github.event.inputs.versionName }} | grep ^current_version | sed -r s,"^.*=",,
56+
echo -e "\n::set-output name=msg::(same as it was)"
57+
- name: Bump covidcast-indicators version
58+
id: indicators
59+
run: |
60+
echo -n "::set-output name=version::"
61+
bump2version --list ${{ github.event.inputs.versionName }} | grep ^new_version | sed -r s,"^.*=",,
62+
- name: Create pull request into prod
63+
uses: peter-evans/create-pull-request@v3
64+
with:
65+
branch: release/indicators_v${{ steps.indicators.outputs.version }}_utils_v${{ steps.utils-changed.outputs.version }}${{ steps.utils-unchanged.outputs.version }}
66+
base: prod
67+
title: Release covidcast-indicators ${{ steps.indicators.outputs.version }}
68+
labels: chore
69+
reviewers: krivard
70+
assignees: krivard
71+
body: |
72+
Releasing:
73+
* covidcast-indicators ${{ steps.indicators.outputs.version }}
74+
* delphi-utils ${{ steps.utils-changed.outputs.version }}${{ steps.utils-unchanged.outputs.version }} ${{steps.utils-changed.outputs.msg }}${{ steps.utils-unchanged.outputs.msg }}

0 commit comments

Comments
 (0)