Skip to content

Commit

Permalink
Mitigate race condition (#3)
Browse files Browse the repository at this point in the history 
Try to ensure in-order execution when different workflows race for deployments.
  • Loading branch information
@mpdude
mpdude authored Sep 15, 2020
1 parent 62d30ec commit fc0c006
Show file tree
Hide file tree
Showing 5 changed files with 88 additions and 5 deletions.
11 changes: 11 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ The IAM User that is used to run the action requires the following IAM permissio
"codedeploy:CreateDeployment",
"codedeploy:RegisterApplicationRevision",
"codedeploy:GetDeploymentConfig",
"codedeploy:GetDeploymentGroup",
"codedeploy:UpdateDeploymentGroup",
"codedeploy:CreateDeploymentGroup"
],
Expand All @@ -148,6 +149,16 @@ The IAM User that is used to run the action requires the following IAM permissio
}
```

## Race Conditions

As of writing, the AWS CodeDeploy API does not accept new deployment requests for an application and deployment group as long as another deployment is still in progress. So, this action will retry a few times and eventually (hopefully) succeed.

There might be situations where several workflow runs are triggered in quick succession - for example, when merging several approved pull requests in a short time. Since your test suites or workflow runs might take a varying amount of time to finish and to reach the deployment phase (_this_ action), you cannot be sure that the triggered deployments will happen in the order you merged the pull requests (to stick with the example). You could not even be sure that the last deployment made was based on the last commit in your repository.

To work around this, this action includes the GitHub Actions "[run id](https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context)" in the `description` field for created deployments. Before creating a new deployment, it will fetch the _last attempted deployment_ from the AWS API and compare its run id with the current run. If the current run has a _lower_ id than the last attempted deployment, the deployment will be aborted.

This workaround should catch a good share of possible out-of-order deployments. There is a slight chance for mishaps, however: If a _newer_ deployment happens to start _after_ we checked the run id and finishes _before_ we commence our own deployment (just a few lines of code later), this might go unnoticed. To really prevent this from happening, ordering deployments probably needs to be supported on the AWS API side, see https://github.com/aws/aws-codedeploy-agent/issues/248.

## Action Input and Output Parameters

### Input
Expand Down
2 changes: 1 addition & 1 deletion cli.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@

const action = require('./create-deployment');
try {
await action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
await action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, null, core);
} catch (e) {
console.log(`👉🏻 ${e.message}`);
process.exit(1);
Expand Down
36 changes: 35 additions & 1 deletion create-deployment.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ function fetchBranchConfig(branchName) {
process.exit();
}

exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, core) {
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, runNumber, core) {
const branchConfig = fetchBranchConfig(branchName);
const safeBranchName = branchName.replace(/[^a-z0-9-/]+/gi, '-').replace(/\/+/, '--');
const deploymentGroupName = branchConfig.deploymentGroupName ? branchConfig.deploymentGroupName.replace('$BRANCH', safeBranchName) : safeBranchName;
Expand Down Expand Up @@ -63,19 +63,53 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
}

let tries = 0;
const description = runNumber ? `Created by webfactory/create-aws-codedeploy-deployment (run_number=${runNumber})` : '';

while (true) {

if (++tries > 5) {
core.setFailed('🤥 Unable to create a new deployment (too much concurrency?)');
return;
}

if (runNumber) {
var {deploymentGroupInfo: {lastAttemptedDeployment: {deploymentId: lastAttemptedDeploymentId}}} = await codeDeploy.getDeploymentGroup({
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
}).promise();

var {deploymentInfo: {description: lastAttemptedDeploymentDescription}} = await codeDeploy.getDeployment({
deploymentId: lastAttemptedDeploymentId,
}).promise();

var matches, lastAttemptedDeploymentRunNumber;

if (matches = lastAttemptedDeploymentDescription.match(/run_number=(\d+)/)) {
lastAttemptedDeploymentRunNumber = matches[1];
if (parseInt(lastAttemptedDeploymentRunNumber) > parseInt(runNumber)) {
core.setFailed(`🙅‍♂️ The last attempted deployment as returned by the AWS API has been created by a higher run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber}. Aborting.`);
return;
} else {
console.log(`🔎 Last attempted deployment was from run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber} - proceeding.`);
}
}

/*
There's a slight remaining chance that the above check does not suffice: If we just
passed the check, but another (newer) build creates AND finishes a deployment
BEFORE we reach the next lines, an out-of-order deployment might happen. This is a
race condition that requires an extension on the AWS API side in order to be resolved,
see https://github.com/aws/aws-codedeploy-agent/issues/248.
*/
}

try {
var {deploymentId: deploymentId} = await codeDeploy.createDeployment({
...deploymentConfig,
...{
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
description: description,
revision: {
revisionType: 'GitHub',
gitHubLocation: {
Expand Down
40 changes: 38 additions & 2 deletions dist/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ function fetchBranchConfig(branchName) {
process.exit();
}

exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, core) {
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, runNumber, core) {
const branchConfig = fetchBranchConfig(branchName);
const safeBranchName = branchName.replace(/[^a-z0-9-/]+/gi, '-').replace(/\/+/, '--');
const deploymentGroupName = branchConfig.deploymentGroupName ? branchConfig.deploymentGroupName.replace('$BRANCH', safeBranchName) : safeBranchName;
Expand Down Expand Up @@ -79,19 +79,53 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
}

let tries = 0;
const description = runNumber ? `Created by webfactory/create-aws-codedeploy-deployment (run_number=${runNumber})` : '';

while (true) {

if (++tries > 5) {
core.setFailed('🤥 Unable to create a new deployment (too much concurrency?)');
return;
}

if (runNumber) {
var {deploymentGroupInfo: {lastAttemptedDeployment: {deploymentId: lastAttemptedDeploymentId}}} = await codeDeploy.getDeploymentGroup({
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
}).promise();

var {deploymentInfo: {description: lastAttemptedDeploymentDescription}} = await codeDeploy.getDeployment({
deploymentId: lastAttemptedDeploymentId,
}).promise();

var matches, lastAttemptedDeploymentRunNumber;

if (matches = lastAttemptedDeploymentDescription.match(/run_number=(\d+)/)) {
lastAttemptedDeploymentRunNumber = matches[1];
if (parseInt(lastAttemptedDeploymentRunNumber) > parseInt(runNumber)) {
core.setFailed(`🙅‍♂️ The last attempted deployment as returned by the AWS API has been created by a higher run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber}. Aborting.`);
return;
} else {
console.log(`🔎 Last attempted deployment was from run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber} - proceeding.`);
}
}

/*
There's a slight remaining chance that the above check does not suffice: If we just
passed the check, but another (newer) build creates AND finishes a deployment
BEFORE we reach the next lines, an out-of-order deployment might happen. This is a
race condition that requires an extension on the AWS API side in order to be resolved,
see https://github.com/aws/aws-codedeploy-agent/issues/248.
*/
}

try {
var {deploymentId: deploymentId} = await codeDeploy.createDeployment({
...deploymentConfig,
...{
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
description: description,
revision: {
revisionType: 'GitHub',
gitHubLocation: {
Expand Down Expand Up @@ -157,8 +191,10 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
const branchName = isPullRequest ? payload.pull_request.head.ref : payload.ref.replace(/^refs\/heads\//, ''); // like "my/branch_name"
console.log(`🎋 On branch '${branchName}', head commit ${commitId}`);

const runNumber = process.env['github_run_number'] || process.env['GITHUB_RUN_NUMBER'];

try {
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, runNumber, core);
} catch (e) {}
})();

Expand Down
4 changes: 3 additions & 1 deletion index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@
const branchName = isPullRequest ? payload.pull_request.head.ref : payload.ref.replace(/^refs\/heads\//, ''); // like "my/branch_name"
console.log(`🎋 On branch '${branchName}', head commit ${commitId}`);

const runNumber = process.env['github_run_number'] || process.env['GITHUB_RUN_NUMBER'];

try {
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, runNumber, core);
} catch (e) {}
})();

0 comments on commit fc0c006

Please sign in to comment.