Bug: Claude Code (Pro/Max Subscribers) - Incorrectly Requires ANTHROPIC_API_KEY

[wsimmonds]

Background

For subscribers to Claude Pro or Claude Max, Claude Code does not require an API key (in fact, an API key would override the token associated with the user's subscription and result in API pricing).

Additionally, it would incorrectly fail for those running Claude Code against Amazon Bedrock (via AWS_BEARER_TOKEN_BEDROCK token as seen on documentation page: https://docs.claude.com/en/docs/claude-code/settings)

Problem

Both cli.ts and claude-code-cli.ts have validation logic which checks for the presence of an API key, and if missing will log an error and call process.exit(1).

cli.ts:

next-evals-oss/cli.ts

Line 1460 in eaf7596

if (!apiKey) {

claude-code-cli.ts:

next-evals-oss/claude-code-cli.ts

Line 282 in eaf7596

const apiKey = values["api-key"] || process.env.ANTHROPIC_API_KEY;

Expected Behaviour

If a user is a subscriber to Claude Pro or Max/is logged in to Claude, it should allow them to run evals instead of exiting with an error

Potential Implementation(s)

I haven't researched exactly how Claude Code stores the token, the documentation shows that it does respect an ANTHROPIC_AUTH_TOKEN environment variable: However, this is not present on my system despite being logged in.

The simplest implementation might be to remove the checks and potentially handle authentication failures upon launching claude code as error that will result in terminating the script.

The same logic is also duplicated in two places, so there's also an opportunity to refactor this.

[mclenhard]

Hey @wsimmonds , thanks for pointing this out. Will do some more research tomorrow on the best way to handle the auth errors, but agree we don't need to bail unnecessarily here.

[wsimmonds]

Shout if you'd like some help, as I'm on Max and am happy to validate any fixes for you.