Prepare Stage Fails with Control Tower launch issues

[richardkeit]

Describe the bug
Prepare stage fails on new AWS account

2024-09-11 07:15:52.150 | info | index | The Landing Zone deployment operation successfully started, operation identifier is - cadf33dd-5e2a-4498-bd64-61098b6464c7
2024-09-11 07:15:52.527 | info | index | The AWS Control Tower Landing Zone operation with identifier cadf33dd-5e2a-4498-bd64-61098b6464c7 is currently in IN_PROGRESS state. After 5 minutes delay, the status will be rechecked.
2024-09-11 07:20:52.923 | info | index | The AWS Control Tower Landing Zone operation with identifier cadf33dd-5e2a-4498-bd64-61098b6464c7 is currently in IN_PROGRESS state. After 5 minutes delay, the status will be rechecked.
2024-09-11 07:25:53.291 | warn | index | AWS Control Tower Landing Zone operation with identifier cadf33dd-5e2a-4498-bd64-61098b6464c7 in FAILED state !!!!. Please investigate CT operation before executing pipeline
2024-09-11 07:25:53.299 | error | runner | AWS Control Tower Landing Zone operation with identifier cadf33dd-5e2a-4498-bd64-61098b6464c7 in FAILED state !!!!. Please investigate CT operation before executing pipeline
Error: AWS Control Tower Landing Zone operation with identifier cadf33dd-5e2a-4498-bd64-61098b6464c7 in FAILED state !!!!. Please investigate CT operation before executing pipeline

To Reproduce

• Create a new account
• Install the LZA solution

Note: Generated default config:
global-config.yaml:

controlTower:
  enable: true
  landingZone:
    version: '3.3'
    logging:
      loggingBucketRetentionDays: 365
      accessLoggingBucketRetentionDays: 3650
      organizationTrail: true
    security:
      enableIdentityCenterAccess: true
  controls: []

Expected behavior
Solution works without issue. Currently this is a 66% pass rate (eg 2/3 installations have succeeded)

Please complete the following information about the solution:

• Version: v1.9.1
• Region: ap-southeast-2
• Was the solution modified from the version published on this repository?
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the services this solution uses?
• Were there any errors in the CloudWatch Logs?

Screenshots

Additional context
Operation log:

❯ aws controltower get-landing-zone-operation  --operation-identifier cadf33dd-5e2a-4498-bd64-61098b6464c7
{
    "operationDetails": {
        "endTime": "2024-09-11T07:21:13+00:00",
        "operationIdentifier": "cadf33dd-5e2a-4498-bd64-61098b6464c7",
        "operationType": "CREATE",
        "startTime": "2024-09-11T07:15:51+00:00",
        "status": "FAILED",
        "statusMessage": "AWS IAM Identity Center can't complete your request because another request is already in progress. Try again later."
    }
}

CreateLandingZoneCall:

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAWCZC6GMLN4JXCUM36:AWSCodeBuild-ecd4b66a-e232-4bcf-98e9-47f671fcfbfc",
        "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-PipelineSt-AdminCdkToolkitRole292E16-DLnwnTIHTWTs/AWSCodeBuild-ecd4b66a-e232-4bcf-98e9-47f671fcfbfc",
        "accountId": "XXXXXXXXX",
        "accessKeyId": "ASIAWCZC6GMLDIBOJTN6",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROAWCZC6GMLN4JXCUM36",
                "arn": "arn:aws:iam::XXXXXXXXX:role/AWSAccelerator-PipelineSt-AdminCdkToolkitRole292E16-DLnwnTIHTWTs",
                "accountId": "XXXXXXXXX",
                "userName": "AWSAccelerator-PipelineSt-AdminCdkToolkitRole292E16-DLnwnTIHTWTs"
            },
            "attributes": {
                "creationDate": "2024-09-11T07:07:45Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2024-09-11T07:15:52Z",
    "eventSource": "controltower.amazonaws.com",
    "eventName": "CreateLandingZone",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "13.55.255.220",
    "userAgent": "aws-sdk-js/3.556.0 ua/2.0 os/linux#4.14.291-218.527.amzn2.x86_64 lang/js md/nodejs#18.20.3 api/controltower#3.556.0 exec-env/AWS_ECS_EC2 AwsSolution/SO0199/1.9.2",
    "requestParameters": {
        "version": "3.3",
        "manifest": {
            "governedRegions": [
                "ap-southeast-2",
                "us-east-1"
            ],
            "organizationStructure": {
                "security": {
                    "name": "Security"
                },
                "sandbox": {
                    "name": "Infrastructure"
                }
            },
            "centralizedLogging": {
                "accountId": "XXXX",
                "configurations": {
                    "loggingBucket": {
                        "retentionDays": 365
                    },
                    "accessLoggingBucket": {
                        "retentionDays": 3650
                    },
                    "kmsKeyArn": "arn:aws:kms:ap-southeast-2:XXXXXXXXX:key/YYYYYYY-YYYYYYY-YYYYYYY-YYYYYYY"
                },
                "enabled": true
            },
            "securityRoles": {
                "accountId": "AAAAAAA"
            },
            "accessManagement": {
                "enabled": true
            }
        }
    },
    "responseElements": {
        "arn": "arn:aws:controltower:ap-southeast-2:XXXXXXXXX:landingzone/AAAAAAA",
        "operationIdentifier": "cadf33dd-5e2a-4498-bd64-61098b6464c7"
    },
    "requestID": "984a61c5-4304-46b7-b4c8-1fb8a3986e1a",
    "eventID": "5db2088b-9511-4b6f-b2fe-398bf5e0098a",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management"
}

[twist-jputro]

This happened very often to me. The workaround is to retry LZ deployment from Control Tower console and it always worked, but it is better to not having this problem.

[olafpasveer]

This is really annoying and prevents launching a new LZ through automation

[erwaxler]

Hi @richardkeit , thank you for your code contribution! This will be fixed in the next major release thanks to your pull request #567 , I will keep this issue open until that release is made public. Thank you for your support of the Landing Zone Accelerator!