Incorrect Metadata-Flavor header when calling universe_domain from a Storage/PubSub client

[ernestoalejo]

Please make sure you have searched for information in the following guides.

• Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
• Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
• Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
• Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
• Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
• Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
• Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples

A screenshot that you have tested with "Try this API".

It's the GCE internal metadata service.

Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.

https://gist.github.com/ernestoalejo/4e998384aed6f075277569c687bdc6d8

A step-by-step description of how to reproduce the issue, based on the linked reproduction.

I started to receive an unexpected error when initializing a storage client. The client calls the metadata server to retrieve the application default credentials but one of the calls is to universe_domain which doesn't returns any Metadata-Flavor header at all.

The full stacktrace:

          Error: Invalid response from metadata service: incorrect Metadata-Flavor header.
              at metadataAccessor (/app/node_modules/.pnpm/[email protected]/node_modules/gcp-metadata/build/src/index.js:120:19)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async GoogleAuth.getUniverseDomainFromMetadataServer (/app/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:169:30)
              at async GoogleAuth.getApplicationDefaultAsync (/app/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:266:21)
              at async GoogleAuth.getClient (/app/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:695:17)
              at async GoogleAuth.authorizeRequest (/app/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:736:24)
              at async Promise.all (index 1)
              at async prepareRequest (file:///app/node_modules/.pnpm/@[email protected]/node_modules/@google-cloud/storage/build/esm/src/nodejs-common/util.js:463:61)

I can't update the library version, is a third party software application which is not under our control.

When I curl the endpoint manually from the machine it doesn't return the header. In fact it accepts any header in the request or even no header at all and never returns its value like other endpoints:

curl -v -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/universe/universe_domain
* Host metadata.google.internal:80 was resolved.
* IPv6: (none)
* IPv4: 169.254.169.254
*   Trying 169.254.169.254:80...
* Connected to metadata.google.internal (169.254.169.254) port 80
> GET /computeMetadata/v1/universe/universe_domain HTTP/1.1
> Host: metadata.google.internal
> User-Agent: curl/8.5.0
> Accept: */*
> Metadata-Flavor: Google
>
< HTTP/1.1 200 OK
< Content-Type: application/text
< ETag: 23d0ca25faab0f19
< Date: Mon, 10 Mar 2025 15:23:31 GMT
< Server: Metadata Server for VM
< Content-Length: 14
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
<
* Connection #0 to host metadata.google.internal left intact
googleapis.com

A clear and concise description of what the bug is, and what you expected to happen.

I know 9.7.0 is not the latest version of this library, but it's a third party software and I would totally expect the metadata server to keep working in GCE for existing software. Also it's not too old and I would expect a recent version of this library to keep working.

A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

The issue is with internal universe_domain call and the metadata service, there isn't public information to reference.

[mssodhi]

Experiencing the same when using pubsub on AppEngine. Randomly started throwing Error: Invalid response from metadata service: incorrect Metadata-Flavor header.

[DmitrijBuckovsky]

Hi,
I had the same issue with @google-cloud/secret-manager (using [email protected])

Error: Invalid response from metadata service: incorrect Metadata-Flavor header.
    at metadataAccessor (/node_modules/gcp-metadata/build/src/index.js:120:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async GoogleAuth.getUniverseDomainFromMetadataServer (/node_modules/google-auth-library/build/src/auth/googleauth.js:169:30)
    at async GoogleAuth.getApplicationDefaultAsync (/node_modules/google-auth-library/build/src/auth/googleauth.js:266:21)
    at async GoogleAuth.getClient (/node_modules/google-auth-library/build/src/auth/googleauth.js:695:17)
    at async GrpcClient._getCredentials (/node_modules/google-gax/build/src/grpc.js:145:24)
    at async GrpcClient.createStub (/node_modules/google-gax/build/src/grpc.js:318:23)

and @google-cloud/storage using (using [email protected]) - !!! Solution not tested yet !!!

Error: Invalid response from metadata service: incorrect Metadata-Flavor header.
    at metadataAccessor (/usr/src/qapline/node_modules/gcp-metadata/build/src/index.js:120:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async GoogleAuth.getUniverseDomainFromMetadataServer (/usr/src/qapline/node_modules/google-auth-library/build/src/auth/googleauth.js:169:30)
    at async GoogleAuth.getApplicationDefaultAsync (/usr/src/qapline/node_modules/google-auth-library/build/src/auth/googleauth.js:266:21)
    at async GoogleAuth.getClient (/usr/src/qapline/node_modules/google-auth-library/build/src/auth/googleauth.js:695:17)
    at async GoogleAuth.authorizeRequest (/usr/src/qapline/node_modules/google-auth-library/build/src/auth/googleauth.js:736:24)
    at async Promise.all (index 1)
    at async prepareRequest (/usr/src/qapline/node_modules/@google-cloud/storage/build/src/nodejs-common/util.js:459:61)

Updating google-auth-library to latest (9.15.1) woked for me. Also version 9.12.0 is working.

Solution:
Add "google-auth-library": "^9.15.1" to dependencies in package.json.
Do not forget to npm i to update package-lock.json.

  "dependencies": {
    "@google-cloud/secret-manager": "^5.6.0",
    "google-auth-library": "^9.15.1",
...

[saikatharryc]

Same, on pub-sub,

Error:

{"level":"error","message":"Bootstrap failed. Closing. Invalid response from metadata service: incorrect Metadata-Flavor header.","scope":"main","stack":"Error: Invalid response from metadata service: incorrect Metadata-Flavor header.\n    at metadataAccessor (/app/node_modules/gcp-metadata/build/src/index.js:120:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async GoogleAuth._GoogleAuth_getUniverseFromMetadataServer (/app/node_modules/google-auth-library/build/src/auth/googleauth.js:777:26)\n    at async GoogleAuth.getUniverseDomain (/app/node_modules/google-auth-library/build/src/auth/googleauth.js:185:168)\n    at async GoogleAuth.getApplicationDefaultAsync (/app/node_modules/google-auth-library/build/src/auth/googleauth.js:251:42)\n    at async GoogleAuth.getClient (/app/node_modules/google-auth-library/build/src/auth/googleauth.js:673:17)\n    at async GrpcClient._getCredentials (/app/node_modules/google-gax/build/src/grpc.js:145:24)\n    at async GrpcClient.createStub (/app/node_modules/google-gax/build/src/grpc.js:308:23)","timestamp":"2025-03-11 18:23:03"}
stream closed EOF for gtm/pbp-service-5bcdf6dfd-t67mw (main)

versions, package lock:

    "node_modules/@google-cloud/pubsub": {
      "version": "4.1.0",
      "license": "Apache-2.0",
      "dependencies": {
        "@google-cloud/paginator": "^5.0.0",
        "@google-cloud/precise-date": "^4.0.0",
        "@google-cloud/projectify": "^4.0.0",
        "@google-cloud/promisify": "^4.0.0",
        "@opentelemetry/api": "^1.6.0",
        "@opentelemetry/semantic-conventions": "~1.18.0",
        "@types/duplexify": "^3.6.0",
        "@types/long": "^4.0.0",
        "arrify": "^2.0.0",
        "extend": "^3.0.2",
        "google-auth-library": "^9.0.0",
        "google-gax": "^4.0.4",
        "heap-js": "^2.2.0",
        "is-stream-ended": "^0.1.4",
        "lodash.snakecase": "^4.1.1",
        "p-defer": "^3.0.0"
      },
      "engines": {
        "node": ">=14.0.0"
      }
    },

This is on GKE, and curl to metadata api from within the machine/pod resolves just fine, except i'm not sure what elements suppose to be expected for this response to be valid enough.

[ernestoalejo]

Note that in our case it's a third party software that can't be updated easily. Also I would expect Compute Engine not to break functionality unexpectedly with recent versions of this library.

The library expects the Metadata-Flavor header to be echoed back, as other endpoints do. However, this no longer seems to be the case with universe_domain.

[julienlafont-shine]

We're facing the same issue, with GAE, and only in the C zone of europe-west1 (B & D zone are not affected).

The bump of the google-auth-library solve the issue 👍

[twistedpair]

Started running into this on GCE VMs using @google-cloud/secret-manager Monday.
These VMs and code have been running without issue for years, and the code was unchanged.