4-projects

4-projects

This repo is part of a multi-part guide that shows how to configure and deploy the example.com reference architecture described in Google Cloud security foundations guide. The following table lists the parts of the guide.

0-bootstrap	Bootstraps a Google Cloud organization, creating all the required resources and permissions to start using the Cloud Foundation Toolkit (CFT). This step also configures a CI/CD Pipeline for foundations code in subsequent stages.
1-org	Sets up top level shared folders, networking projects, and organization-level logging, and sets baseline security settings through organizational policy.
2-environments	Sets up development, nonproduction, and production environments within the Google Cloud organization that you've created.
3-networks-svpc	Sets up shared VPCs with default DNS, NAT (optional), Private Service networking, VPC service controls, on-premises Dedicated Interconnect, and baseline firewall rules for each environment. It also sets up the global DNS hub.
3-networks-hub-and-spoke	Sets up shared VPCs with all the default configuration found on step 3-networks-svpc, but here the architecture will be based on the Hub and Spoke network model. It also sets up the global DNS hub
4-projects (this file)	Sets up a folder structure, projects, and application infrastructure pipeline for applications, which are connected as service projects to the shared VPC created in the previous stage.
5-app-infra	Deploy a simple Compute Engine instance in one of the business unit projects using the infra pipeline set up in 4-projects.

For an overview of the architecture and the parts, see the terraform-example-foundation README.

Purpose

The purpose of this step is to set up the folder structure, projects, and infrastructure pipelines for applications that are connected as service projects to the shared VPC created in the previous stage.

For each business unit, a shared infra-pipeline project is created along with Cloud Build triggers, CSRs for application infrastructure code, Google Cloud Storage buckets for state storage, and a new Docker image will be built for the Confidential Space environment, which will be used in the 5-app-infra step.

This step follows the same conventions as the Foundation pipeline deployed in 0-bootstrap. A custom workspace (bu1-example-app) is created by this pipeline and necessary roles are granted to the Terraform Service Account of this workspace by enabling variable sa_roles as shown in this example.

This pipeline is utilized to deploy resources in projects across development/nonproduction/production in step 5-app-infra. Other Workspaces can also be created to isolate deployments if needed.

Prerequisites

1. 0-bootstrap executed successfully.

2. 1-org executed successfully.

3. 2-environments executed successfully.

4. 3-networks executed successfully.

5. For the manual step described in this document, you need to use the same Terraform version used on the build pipeline. Otherwise, you might experience Terraform state snapshot lock errors.

   Note: As mentioned in 0-bootstrap README note 2 at the end of Cloud Build deploy section, make sure that you have requested at least 50 additional projects for the projects step service account, otherwise you may face a project quota exceeded error message during the following steps and you will need to apply the fix from this entry of the Troubleshooting guide in order to continue.

Troubleshooting

Please refer to troubleshooting if you run into issues during this step.

Usage

Note: If you are using MacOS, replace cp -RT with cp -R in the relevant commands. The -T flag is needed for Linux, but causes problems for MacOS.

Deploying with Cloud Build

1. Clone the gcp-projects repo based on the Terraform output from the 0-bootstrap step. Clone the repo at the same level of the terraform-example-foundation folder, the following instructions assume this layout. Run terraform output cloudbuild_project_id in the 0-bootstrap folder to get the Cloud Build Project ID.

   export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id)
   echo ${CLOUD_BUILD_PROJECT_ID}
   
   gcloud source repos clone gcp-projects --project=${CLOUD_BUILD_PROJECT_ID}

2. Change to the freshly cloned repo, change to the non-main branch and copy contents of foundation to new repo.

   cd gcp-projects
   git checkout -b plan
   
   cp -RT ../terraform-example-foundation/4-projects/ .
   cp ../terraform-example-foundation/build/cloudbuild-tf-* .
   cp ../terraform-example-foundation/build/tf-wrapper.sh .
   chmod 755 ./tf-wrapper.sh

3. Rename auto.example.tfvars files to auto.tfvars.

   mv common.auto.example.tfvars common.auto.tfvars
   mv shared.auto.example.tfvars shared.auto.tfvars
   mv development.auto.example.tfvars development.auto.tfvars
   mv nonproduction.auto.example.tfvars nonproduction.auto.tfvars
   mv production.auto.example.tfvars production.auto.tfvars

4. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars, development.auto.tfvars, nonproduction.auto.tfvars, and production.auto.tfvars files.

5. See any of the shared folder README.md files for additional information on the values in the shared.auto.tfvars file.

6. Use terraform output to get the backend bucket value from 0-bootstrap output.

   export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw gcs_bucket_tfstate)
   echo "remote_state_bucket = ${remote_state_bucket}"
   
   sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars

7. (Optional) If you want additional subfolders for separate business units or entities, make additional copies of the folder business_unit_1 and modify any values that vary across business unit like business_code, business_unit, or subnet_ip_range.

For example, to create a new business unit similar to business_unit_1, run the following:

#copy the business_unit_1 folder and it's contents to a new folder business_unit_2
cp -r  business_unit_1 business_unit_2

# search all files under the folder `business_unit_2` and replace strings for business_unit_1 with strings for business_unit_2
grep -rl bu1 business_unit_2/ | xargs sed -i 's/bu1/bu2/g'
grep -rl business_unit_1 business_unit_2/ | xargs sed -i 's/business_unit_1/business_unit_2/g'
# search subnet_ip_range 10.3.64.0 and replace for the new range 10.4.64.0
grep -rl 10.3.64.0 business_unit_2/ | xargs sed -i 's/10.3.64.0/10.4.64.0/g'

1. Commit changes.

   git add .
   git commit -m 'Initialize projects repo'

2. You need to manually plan and apply only once the business_unit_1/shared and business_unit_2/shared environments since development, nonproduction, and production depend on them.

3. Use terraform output to get the Cloud Build project ID and the projects step Terraform Service Account from 0-bootstrap output. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set using the Terraform Service Account to enable impersonation.

   export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id)
   echo ${CLOUD_BUILD_PROJECT_ID}
   
   export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_step_terraform_service_account_email)
   echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

4. Run init and plan and review output for environment shared.

   ./tf-wrapper.sh init shared
   ./tf-wrapper.sh plan shared

5. Run apply shared.

   ./tf-wrapper.sh apply shared

6. Push your plan branch to trigger a plan for all environments. Because the plan branch is not a named environment branch), pushing your plan branch triggers terraform plan but not terraform apply. Review the plan output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

   git push --set-upstream origin plan

7. Merge changes to production. Because this is a named environment branch, pushing to this branch triggers both terraform plan and terraform apply. Review the apply output in your Cloud Build project. https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

   git checkout -b production
   git push origin production

8. After production has been applied, apply development.

9. Merge changes to development. Because this is a named environment branch, pushing to this branch triggers both terraform plan and terraform apply. Review the apply output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

   git checkout -b development
   git push origin development

10. After development has been applied, apply nonproduction.

11. Merge changes to nonproduction. Because this is a named environment branch, pushing to this branch triggers both terraform plan and terraform apply. Review the apply output in your Cloud Build project. https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

    git checkout -b nonproduction
    git push origin nonproduction

12. Unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

    unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

13. Before move to the next step, configure directional policies for your perimeter.

14. Navigate to gcp-org.

    cd ../gcp-org/

15. If you are deploying with VPC Service Controls in dry run mode, update the required_egress_rule_app_infra_dry_run and required_ingress_rule_app_infra_dry_run variables to true, if you are deploying with VPC Service Controls in enforced mode, update the required_egress_rule_app_infra and required_ingress_rule_app_infra variables to true in service_control.tf file.

    export enforce_vpcsc=$(terraform -chdir="envs/shared/" output -raw enforce_vpcsc); \
    echo "enforce_vpcsc" = $enforce_vpcsc
    if [[ "$enforce_vpcsc" == "false" ]]; then \
    sed -i -E '/^[[:space:]]*\/\/required_ingress_rules_app_infra_dry_run[[:space:]]*=/ s|^[[:space:]]*//||' ./envs/shared/terraform.tfvars; \
    else \
    sed -i -E '/^[[:space:]]*\/\/required_ingress_rules_app_infra[[:space:]]*=/ s|^[[:space:]]*//||' ./envs/shared/terraform.tfvars; \
    fi
    if [[ "$enforce_vpcsc" == "false" ]]; then \
    sed -i -E '/^[[:space:]]*\/\/required_egress_rules_app_infra_dry_run[[:space:]]*=/ s|^[[:space:]]*//||' ./envs/shared/terraform.tfvars; \
    else \
    sed -i -E '/^[[:space:]]*\/\/required_egress_rules_app_infra[[:space:]]*=/ s|^[[:space:]]*//||' ./envs/shared/terraform.tfvars; \
    fi

16. Commit and push the changes.

    git add .
    git commit -m "Add infra pipeline directional policies"
    git push
    
    cd ..

17. You can now move to the instructions in the 5-app-infra step.

Deploying with GitHub Actions

See 0-bootstrap README-GitHub.md.

Run Terraform locally

1. The next instructions assume that you are at the same level of the terraform-example-foundation folder. Create and change into gcp-projects folder, copy the code, Terraform wrapper script and ensure it can be executed.

   mkdir gcp-projects
   cp -R  terraform-example-foundation/4-projects/* gcp-projects/
   cp terraform-example-foundation/build/tf-wrapper.sh gcp-projects/
   cp terraform-example-foundation/.gitignore gcp-projects/
   chmod 755 ./gcp-projects/tf-wrapper.sh

2. Navigate to gcp-projects and initialize a local Git repository to manage versions locally. Then, create the environment branches.

   cd gcp-projects
   git init
   git commit -m "initialize empty directory" --allow-empty
   git checkout -b shared
   git checkout -b development
   git checkout -b nonproduction
   git checkout -b production

3. Rename auto.example.tfvars files to auto.tfvars.

   mv common.auto.example.tfvars common.auto.tfvars
   mv shared.auto.example.tfvars shared.auto.tfvars
   mv development.auto.example.tfvars development.auto.tfvars
   mv nonproduction.auto.example.tfvars nonproduction.auto.tfvars
   mv production.auto.example.tfvars production.auto.tfvars

4. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars, development.auto.tfvars, nonproduction.auto.tfvars, and production.auto.tfvars files. See any of the shared folder README.md files for additional information on the values in the shared.auto.tfvars file. Use terraform output to get the remote state bucket (the backend bucket used by previous steps) value from gcp-bootstrap output.

   export remote_state_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw gcs_bucket_tfstate)
   echo "remote_state_bucket = ${remote_state_bucket}"
   
   sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars

We will now deploy each of our environments(development/production/nonproduction) using the tf-wrapper.sh script.

1. Use terraform output to get the Seed project ID and the organization step Terraform service account from gcp-bootstrap output. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set using the Terraform Service Account to enable impersonation.

   export SEED_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw seed_project_id)
   echo ${SEED_PROJECT_ID}
   
   export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw projects_step_terraform_service_account_email)
   echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

2. (Optional) If you want additional subfolders for separate business units or entities, make additional copies of the folder business_unit_1 and modify any values that vary across business unit like business_code, business_unit, or subnet_ip_range.

For example, to create a new business unit similar to business_unit_1, run the following:

#copy the business_unit_1 folder and it's contents to a new folder business_unit_2
cp -r  business_unit_1 business_unit_2

# search all files under the folder `business_unit_2` and replace strings for business_unit_1 with strings for business_unit_2
grep -rl bu1 business_unit_2/ | xargs sed -i 's/bu1/bu2/g'
grep -rl business_unit_1 business_unit_2/ | xargs sed -i 's/business_unit_1/business_unit_2/g'
# search subnet_ip_range 10.3.64.0 and replace for the new range 10.4.64.0
grep -rl 10.3.64.0 business_unit_2/ | xargs sed -i 's/10.3.64.0/10.4.64.0/g'

1. Checkout shared branch. Run init and plan and review output for environment shared.

   git checkout shared
   ./tf-wrapper.sh init shared
   ./tf-wrapper.sh plan shared

2. Run apply shared.

   ./tf-wrapper.sh apply shared
   git add .
   git commit -m "Initial shared commit."

3. Checkout development branch and merge shared into it. Run init and plan and review output for environment production.

   git checkout development
   git merge shared
   ./tf-wrapper.sh init development
   ./tf-wrapper.sh plan development

4. Run apply development.

   ./tf-wrapper.sh apply development
   git add .
   git commit -m "Initial development commit."

5. Checkout nonproduction and merge development into it. Run init and plan and review output for environment nonproduction.

   git checkout nonproduction
   git merge development
   ./tf-wrapper.sh init nonproduction
   ./tf-wrapper.sh plan nonproduction

6. Run apply nonproduction.

   ./tf-wrapper.sh apply nonproduction
   git add .
   git commit -m "Initial nonproduction commit."

7. Checkout shared production. Run init and plan and review output for production development.

   git checkout production
   git merge nonproduction
   ./tf-wrapper.sh init production
   ./tf-wrapper.sh plan production

8. Run apply production.

   ./tf-wrapper.sh apply production
   git add .
   git commit -m "Initial production commit."
   cd ../

If you received any errors or made any changes to the Terraform config or any .tfvars, you must re-run ./tf-wrapper.sh plan <env> before run ./tf-wrapper.sh apply <env>.

1. Unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

   unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

2. Before move to the next step, configure directional policies for your perimeter.

3. Use terraform output to get the Seed project ID and the organization step Terraform service account from gcp-bootstrap output. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set using the Terraform Service Account to enable impersonation.

   export SEED_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw seed_project_id)
   echo ${SEED_PROJECT_ID}
   
   export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw organization_step_terraform_service_account_email)
   echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

4. Before move to the next step, configure directional policies for your perimeter.

5. Navigate to gcp-org.

6. Use terraform output to get the APP Infra Pipeline cloud build project id and project number.

   cd ../gcp-org/
   
   export cloudbuild_project_number=$(terraform -chdir="../gcp-projects/business_unit_1/shared/" output -raw cloudbuild_project_number)
   echo "cloud build project number = $cloudbuild_project_number"
   export cloudbuild_project_id=$(terraform -chdir="../gcp-projects/business_unit_1/shared/" output -raw cloudbuild_project_id)
   echo "cloud build project id = $cloudbuild_project_id"
   sed -i'' -e "s/PRJ_APP_INFRA_PIPELINE_NUMBER/${cloudbuild_project_number}/" /envs/shared/service_control.tf
   sed -i'' -e "s/PRJ_APP_INFRA_ID/${cloudbuild_project_id}/" /envs/shared/service_control.tf

7. If you are deploying with VPC Service Controls in dry run mode, update the required_egress_rule_app_infra_dry_run and required_ingress_rule_app_infra_dry_run variables to true, if you are deploying with VPC Service Controls in enforced mode, update the required_egress_rule_app_infra and required_ingress_rule_app_infra variables to true in service_control.tf file.

   export enforce_vpcsc=$(terraform -chdir="envs/shared/" output -raw enforce_vpcsc); \
   echo "enforce_vpcsc" = $enforce_vpcsc
   if [[ "$enforce_vpcsc" == "false" ]]; then \
   sed -i -E '/^[[:space:]]*\/\/required_ingress_rules_app_infra_dry_run[[:space:]]*=/ s|^[[:space:]]*//||' envs/shared/terraform.tfvars; \
   else \
   sed -i -E '/^[[:space:]]*\/\/required_ingress_rules_app_infra[[:space:]]*=/ s|^[[:space:]]*//||' envs/shared/terraform.tfvars; \
   fi
   if [[ "$enforce_vpcsc" == "false" ]]; then \
   sed -i -E '/^[[:space:]]*\/\/required_egress_rules_app_infra_dry_run[[:space:]]*=/ s|^[[:space:]]*//||' envs/shared/terraform.tfvars; \
   else \
   sed -i -E '/^[[:space:]]*\/\/required_egress_rules_app_infra[[:space:]]*=/ s|^[[:space:]]*//||' envs/shared/terraform.tfvars; \
   fi

8. Run plan and apply and output for production development. Commit and save the changes.

   ./tf-wrapper plan production
   ./tf-wrapper apply production
   git commit -m "Add infra pipeline directional policies"
   
   cd ..

9. Unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

   unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

10. You can now move to the instructions in the 5-app-infra step.

Warning

If you previously enabled the dry-run variables required_ingress_rules_app_infra_dry_run and required_egress_rules_app_infra_dry_run, you must disable both of them before running the 4-projects step in enforced mode. Once enforced mode is activated and the 4-projects step is executed, the projects created in that step are removed from the dry-run perimeter. Keeping these dry-run variables active creates conflicts, because they define rules for projects that will no longer be part of the dry-run perimeter after this transition.