Add Retry Mechanism to E2E EKS Terraform Deployment

[harrryr]

Issue #, if available:
The EKS Canary occasionally fails due to transitivity issues. One of the recurring errors are Max attempts reached in the step : Wait for Endpoint to Come Online. This occurs due to the endpoint sometimes taking longer than expected to become ready.

Description of changes:

• Change the endpoint wait time from 5 minutes to 10 minutes.
• Add a retry mechanism such that if either the terraform deployment or the endpoint connection fails, then destroy the terraform deployment and try again. This will ensure that the endpoint is deleted and recreated.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

Attention: 102 lines in your changes are missing coverage. Please review.

Comparison is base (09e6487) 85.71% compared to head (d355dc8) 50.73%.
Report is 175 commits behind head on main.

Files	Patch %	Lines
...ent/providers/AwsAppSignalsCustomizerProvider.java	24.00%	35 Missing and 3 partials ⚠️
...gent/providers/AwsSpanMetricsProcessorBuilder.java	0.00%	20 Missing ⚠️
...ders/AttributePropagatingSpanProcessorBuilder.java	0.00%	16 Missing ⚠️
...viders/AwsMetricAttributesSpanExporterBuilder.java	0.00%	11 Missing ⚠️
...try/javaagent/providers/AwsSpanProcessingUtil.java	90.16%	1 Missing and 5 partials ⚠️
...vaagent/providers/AwsMetricAttributeGenerator.java	96.89%	2 Missing and 3 partials ⚠️
...y/javaagent/providers/AwsSpanMetricsProcessor.java	91.48%	0 Missing and 4 partials ⚠️
...t/providers/AttributePropagatingSpanProcessor.java	94.59%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #634       +/-   ##
=============================================
- Coverage     85.71%   50.73%   -34.99%     
- Complexity       19      264      +245     
=============================================
  Files             3       39       +36     
  Lines            49     1301     +1252     
  Branches          5      141      +136     
=============================================
+ Hits             42      660      +618     
- Misses            3      609      +606     
- Partials          4       32       +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[majanjua-amzn]

Some small questions but overall should be fine

[majanjua-amzn]

nit: Can we just pull the enablement script into the folder it would be useful in?

[majanjua-amzn]

Do we need to add this to the other validation steps?

[srprash]

+1

[srprash]

Is there a sample run where we can see this change in action?

[srprash]

Success should be made 0 after the terraform apply command has completed. That's is the assumption in the following code.

[harrryr]

Success of 1 indicates that the setting up App Signals on the sample app failed, while 0 indicates that everything ran successfully.

The logic is:

1. Set Success to 1 (Set initial value to 1 so that the while loop runs)
2. While Success is 1 (Indicates that terraform deployment/endpoint connection failed and will try again):
    2a: Set Success to 0 (Set the value to 0 and if there were any failures change it to 1)
    2b: Run Terraform apply (If the deployment failed, then success will change to 1)
    2c: If Success is still 0, then install App Signals and check endpoint connection
    2d: If endpoint connection failed, change success to 1
    2e: If Success is 1 at this point, then either the deployment or connection failed and run the while loop again. If it is still 0, then the code ran successfully and exit the while loop

If the success is made 0 after the terraform apply, then it will override whether terraform deployment succeeded or not. If after the terraform deployment the success is 1, we want to skip the endpoint connection step and redeploy the terraform again.

[majanjua-amzn]

I think it's a little confusing because 1 usually means true and in this case setting success to 1 means the previous step failed, i.e. success is not true.

Also, could we not just wrap the retry in a while look that checks the retry counter instead of the success variable? Technically, success is a variable specific to a single attempt and shouldn't even be used outside the loop or as a condition for it. Instead, we should rely on the retry counter, i.e. if attempt # < attempt limit, run retry