Amplify Hosting console (default) Build broken for "api/team" at latest TEAM IDC APP v1.2.2 ?

[gbayfield-em]

Describe the bug
The TEAM Amplify/Hosting Console initial (fresh new) deployment using default Amplify Build Spec (Unchanged) now fails across TEAM IDC APP v1.2.2 & our automated prior build of the v1.1.2 version. Was running fine for months, suddenly and consistently now fails to build the "api/team" resource using the default amplify.yaml file (not overridden) which is unchanged for v1.1.2 through v1.2.2. Amplify Hosting console Build gets as far as "Executing command: /usr/local/bin/pip3.9 install --user pipenv==2023.6.12" then fails with the following Build Output shown below...

In terms of the build content, its the unchanged AWS TEAM IDC APP Codebase at latest v1.2.2 here, no Build or Source file changes on the default 'main' codebase (run from CodeCommit Repo per TEAM setup instructions). This Amplify Build Failure is really early on, just attempting to setup the pipenv env (so we dont need to worry about the complexities associated with setting up the IdC APP, we dont get that far now ?)

From the attached amplify.yaml Build Spec, the default build is now failing on the line

... [INFO]: # Executing command: /usr/local/bin/pip3.9 install --user pipenv==2023.6.12
...
"Deployment 1"
2025-01-08T14:18:24.167Z [WARNING]: - Uploading files.
2025-01-08T14:18:25.205Z [WARNING]: - Initializing your environment: main
2025-01-08T14:18:25.311Z [WARNING]: - Building resource api/team
2025-01-08T14:18:36.470Z [WARNING]: ✖ Initializing your environment: main
2025-01-08T14:18:36.477Z [WARNING]: ✖ There was an error initializing your environment.
2025-01-08T14:18:36.477Z [INFO]: 🛑 Packaging overrides failed.
Learn more at: https://docs.amplify.aws/cli/graphql/override/
2025-01-08T14:18:36.489Z [INFO]:
2025-01-08T14:18:36.490Z [INFO]: Session Identifier: 1ad6c553-8c16-45db-8d9d-728d23b43a92
2025-01-08T14:18:36.553Z [ERROR]: !!! Build failed
2025-01-08T14:18:36.553Z [ERROR]: !!! Error: Command failed with exit code 1
2025-01-08T14:18:36.553Z [INFO]: # Starting environment caching...
2025-01-08T14:18:36.553Z [INFO]: # Environment caching completed
...
"Deployment 2"
Attempting to redeploy using Amplify console (button) produces same error (with another msg to fetch something "from the cloud" >?! These messages are not helpful :/ )
....
2025-01-08T09:10:46.653Z [INFO]: �[0mBackend environment main found in Amplify Console app: TEAM-IDC-APP�[0m
2025-01-08T09:10:47.688Z [WARNING]: - Fetching updates to backend environment: main from the cloud.
2025-01-08T09:10:48.005Z [WARNING]: - Building resource api/team
2025-01-08T09:10:57.277Z [WARNING]: ✖ Fetching updates to backend environment: main from the cloud.
2025-01-08T09:10:57.282Z [WARNING]: ✖ There was an error initializing your environment.
2025-01-08T09:10:57.295Z [INFO]: 🛑 Packaging overrides failed.
2025-01-08T09:10:57.296Z [INFO]: Learn more at: https://docs.amplify.aws/cli/graphql/override/

amplify.yaml attached (same for v1.1.2 thru v1.2.2)

To Reproduce
Steps to reproduce the behavior:

1. Attempt first deploy of TEAM IDC APP following documented sample project instructions - this amplify failure occurs early-on on Amplify Hosting build, following the successful completion of the TEAM-IDC-APP CloudFormation Stack (which launches this Amplify Hosting Build which can be tracked on the Amplify console).

Expected behavior
Previously the Amplify Build successfully completed.

Screenshots

Desktop (please complete the following information):
AWS Console- Amplify Hosting env build output...

Smartphone (please complete the following information):

Additional context
Build was working thru to Dec timeframes

All comments welcome ...

[KMPhaniKumar]

I am also facing same issue

[KMPhaniKumar]

[gbayfield-em]

By my reckoning, all sample builds of this TEAM IDC APP (from v1.1.2+) using Amplify Hosting now fail with this dependency error !? Can we have a response to confirm or correct this assertion please ?

[ashwinkonireddy]

Attempting to upgrade the TEAM IDC app from version 1.1.0 to 1.2.2, but encountering the same issue.

[tawoyinfa]

Acknowledged. Investigating the cause of the issue.

[gbayfield-em]

Hi Taiwo, much appreciated, I imagine this build breakage will impact a lot of folks, including those attempting first time setup of sample app...

[gbayfield-em]

Hi there @tawoyinfa any progress or timeframes on this AWS Amplify Hosting build-breakage pls ? I suspect this may be due to broken depedencies on the api/team resource graphql-ttl-transformer but have no evidence thus far... am trying the amplify init/push (build) from VSCode env to see if this sheds any light ?

[tawoyinfa]

I am working actively with the AWS Amplify team on this issue. I will provide updates as soon as I make progress.

[tawoyinfa]

There seems to be a similar Amplify hosting issue aws-amplify/amplify-cli#13931 which is yet to be resolved.

[gbayfield-em]

There seems to be a similar Amplify hosting issue aws-amplify/amplify-cli#13931 which is yet to be resolved.

Interesting, we use npm - the standard TEAM IDC APP v1.2.2 release has a couple of yarn.lock files present in the amplify directory - will try removing the 3 JIT function yarn.lock files & retry our build ...
e.g.

[gbayfield-em]

@tawoyinfa - Was a long-shot but tried removing the yarn.lock files from the three functions ^^ but the Amplify Hosting build still fails the exact same way. Should these function yarn.lock files be part of the build ?
Regarding aws-amplify/amplify-cli#13931 that Amplify error mentions yarn, this one #343 does not, though we share the same generic environment initialization error messages. We might be barking up the wrong tree, any more feedback from AWS Amplify team pls ?

[tawoyinfa]

@gbayfield-em can you add the AMPLIFY_ENABLE_DEBUG_OUTPUT and set the value to true in the environment variables on the Amplify hosting settings. It should give us more info on the root cause as opposed to a vague output

[mcfadden]

I just ran a deploy (new deploy, not an upgrade) with AMPLIFY_ENABLE_DEBUG_OUTPUT set to true. Here is the relevant output:

2025-01-13T16:47:45.035Z [WARNING]: - Fetching updates to backend environment: main from the cloud.
2025-01-13T16:47:45.382Z [INFO]: Overrides functionality is not implemented for this category
2025-01-13T16:47:45.383Z [WARNING]: - Building resource api/team
2025-01-13T16:47:55.368Z [WARNING]: ✖ Fetching updates to backend environment: main from the cloud.
2025-01-13T16:47:55.374Z [WARNING]: ✖ There was an error initializing your environment.
2025-01-13T16:47:55.387Z [INFO]: 🛑 Packaging overrides failed.
2025-01-13T16:47:55.387Z [INFO]: Learn more at: https://docs.amplify.aws/cli/graphql/override/
InvalidOverrideError: Packaging overrides failed.
at /snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-category-api/lib/index.js:309:23
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async transformCategoryStack (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-category-api/lib/index.js:308:29)
at async transformResourceWithOverrides (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/override-manager/transform-resource.js:66:9)
at async Object.buildOverrides (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/utility-functions.js:88:7)
at async buildOverridesEnabledResources (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/build-override-enabled-resources.js:27:3)
at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/initialize-env.js:100:3)
2025-01-13T16:47:55.409Z [INFO]: Session Identifier: 623d4f17-3ec5-4e2c-8b81-d73dde996d5e
2025-01-13T16:47:55.546Z [ERROR]: !!! Build failed
2025-01-13T16:47:55.547Z [ERROR]: !!! Error: Command failed with exit code 1

[gbayfield-em]

Ok set env var on clean new build via deployment/template.yml (TEAM IDC APP v1.2.2) ....

and i get the exact same stack trace ^^ as mcfadden (thanks!)

so hopefully this will help the AWS Amplify guys pin down the issue !? What's weird about this build failure is "nothings changed" from our perspective with a TEAM IDC APP automated build that was running & repeatable for literally months (from v1.1.2) ...

[lukasgabi]

We also are facing the same issue on our side. The main culprit seems to be the override.ts that configures the logging of the AppSync API. Removing this config (i.e. renaming) makes the build run successfully on our setup again, it's however less than ideal to lose the logging config. I tested with different versions of Amplify CLI and couldn't get it running with the override.

[tawoyinfa]

@lukasgabi you are right. The override.ts file is the root cause of the issue. I am investigating if there is a workaround. Alternatively we can remove the override (in the mean time) and add instructions in the documentation on how to re-enable logging manually on the console.

[gbayfield-em]

Ok awesome, so the (temp) workaround, pending further progress, is to remove the offending override.ts file from the build, namely iam-identity-center-team/amplify/backend/api/team/override.ts

[tawoyinfa]

Hi all, I have released a new version 1.3.0 that resolves this issue by removing the override.ts configuration.
The release note also provides instruction on how to enable logging on the Appsync API once the deployment is complete.