Skip to content

codex-auth.md

Latest commit

 

History

History
324 lines (225 loc) · 13.4 KB

codex-auth.md

File metadata and controls

324 lines (225 loc) · 13.4 KB
Raw

Codex Auth Architecture

This document describes how Codex authentication currently works in CoCalc, in both cocalc-plus and launchpad/project-host deployments.

Scope

This covers:

  • Auth source resolution for Codex turns
  • Credential storage and replication
  • Project-host runtime materialization
  • Site-funded Codex admission, proxy enforcement, and accounting

Current Auth Sources and Precedence

For each Codex turn, project-host resolves auth in this order:

  1. ChatGPT subscription (subscription)
  2. Workspace OpenAI API key (project-api-key)
  3. Account OpenAI API key (account-api-key)
  4. Site OpenAI API key (site-api-key)
  5. Shared ~/.codex fallback (shared-home) only when explicitly enabled

Launchpad default:

  • shared-home auth is disabled by default (COCALC_PRODUCT=launchpad)
  • this avoids accidental collaborator auth leakage from workspace ~/.codex/auth.json

Resolution code:

Global Architecture (Hub + DB)

The central hub stores external credentials encrypted at rest, and exposes two API surfaces:

  • User-facing system RPCs for settings UI (hub.system.*)
  • Host-facing RPCs for project-hosts (hub.hosts.*)
flowchart TD
  U[User in Browser] --> SYS[hub.system RPC]
  SYS --> DB[(external_credentials table)]

  PH[Project Host] --> HOSTS[hub.hosts RPC]
  HOSTS --> DB

  DB --> ENC[encrypted_payload via server crypto settings]
Loading

Primary server modules:

Local Runtime Architecture (Project Host)

Project-host resolves auth at turn start, then runs upstream codex inside a per-project/per-auth-context podman runtime.

flowchart TD
  FE[Frontend chat turn] --> ACP[ACP evaluate]
  ACP --> CE[CodexAppServerAgent]
  CE --> SP[project-host codex spawner]
  SP --> AR[resolve auth runtime]
  AR -->|subscription/api key/shared-home| CT[podman codex container]
  AR -->|site-funded| FP[host-local funded proxy]
  FP --> OAI[OpenAI Responses API]
  FP --> SEED[seed funding ledger]
  CT --> CODEX[upstream codex app-server]

  AR --> REG[host RPC: credential pull/check]
  REG --> HUB[central hub]
Loading

Key modules:

ACP Conat Authorization Boundary

Interactive ACP requests execute directly on the project host. Their Conat subjects bind both security identities:

acp.project-<project_id>.account-<account_id>.<operation>

The project remains the second subject segment so normal project-host routing can resolve the destination. Project-host authorization then requires all of the following:

  • the authenticated principal is an account, not a project identity;
  • the subject account equals the authenticated account;
  • the account is a locally mirrored owner or collaborator of the subject project;
  • the operation is a publication; replies use the caller's private inbox.

Viewer and public-share grants do not authorize ACP. The ACP server derives both IDs from the subject and rejects any payload or nested chat project ID that disagrees with it. This is important because account_id selects credential, approval, admission, and attribution paths; it must never be trusted solely from request data.

Legacy subjects of the form acp.project-<project_id>.<operation> are accepted only for authenticated collaborators to reach a compatibility listener. That listener returns ACP_CLIENT_REFRESH_REQUIRED, terminates the request, and never calls ACP execution, session, control, or automation handlers.

Key modules:

Credential Lifecycle

ChatGPT subscription auth

Two supported paths:

  1. Device auth from project-host runtime (codex login --device-auth)
  2. Fallback upload of auth.json generated on user machine

Project-host handlers:

Frontend controls:

OpenAI API keys (account/workspace)

Managed in account settings UI and stored as external credentials:

Where Data Lives

Central (authoritative)

  • external_credentials.encrypted_payload in Postgres
  • encrypted/decrypted by server credential helpers

Project-host cache

Subscription auth cache directory:

  • default root: codexSubscriptionsPath from src/packages/backend/data.ts
  • default location on hosts: /btrfs/data/secrets/codex-subscriptions/<account_id>/

Typical files:

  • auth.json
  • config.toml (cli_auth_credentials_store = "file")
  • .last_used marker for GC

.codex Semantics in Project-Host

In project-host mode, CoCalc intentionally separates auth material from session history.

  • Auth source-of-truth:
    • subscription auth comes from host secrets (/btrfs/data/secrets/codex-subscriptions/...)
    • API-key auth comes from credential resolution and env injection
  • Session history source-of-truth:
    • Codex session JSONL files live under workspace storage (/root/.codex/sessions in the runtime container, i.e. project volume)

Important behavior:

  • Project-host ignores workspace ~/.codex/auth.json for auth resolution in launchpad mode.
  • For subscription auth, project-host mounts only auth files (auth.json, config.toml) from secrets into /root/.codex, while keeping /root/.codex/sessions in the workspace.
  • Shared-home auth (shared-home) can still be explicitly enabled via COCALC_CODEX_AUTH_SHARED_HOME_MODE for single-user/plus-style deployments.

Subscription Cache GC

Project-host runs periodic cleanup of stale local subscription caches.

Defaults:

  • TTL: 72h (COCALC_CODEX_SUBSCRIPTION_CACHE_TTL_MS)
  • Sweep interval: 1h (COCALC_CODEX_SUBSCRIPTION_CACHE_SWEEP_MS)

GC skips currently mounted /root/.codex paths from active codex-* containers.

Site-Funded Codex

When auth source is site-api-key, each turn uses the funded path:

  • the project host requests an atomic seed-authoritative reservation;
  • the project runtime receives a short-lived proxy token, never the site API key;
  • a host-local OpenAI-compatible proxy forces the configured model, reasoning, service tier, output limit, request count, duration, and maximum cost;
  • every provider response emits an idempotent exact usage event;
  • completion, interruption, failure, and expiration all settle observed cost;
  • unreported usage and finish events remain in a durable host SQLite outbox.

The initial policy is GPT-5.6 Luna, low reasoning, standard speed, no OpenAI hosted paid tools, and a five-cent maximum reservation. All limits are dynamic site settings. site_funded_codex_enabled disables only included access; launch_disable_ai remains the complete AI kill switch.

Per-account 5-hour and 7-day allowances come exclusively from the account's resolved membership ai_limits, including any account entitlement override. There are no separate free/member fallback allowances in site settings. One AI unit represents one cent of provider spend, so 100 units equals US$1; either limit being zero disables included Codex for that account. The separate free and paid weekly pools are aggregate site-wide spending circuit breakers.

Modules:

Exact funded host RPCs:

  • hosts.reserveSiteFundedCodexTurn
  • hosts.heartbeatSiteFundedCodexTurn
  • hosts.recordSiteFundedCodexUsageEvent
  • hosts.finishSiteFundedCodexTurn
  • hosts.getSiteFundedCodexPoolStatus
  • hosts.getSiteOpenAiApiKey

Legacy aggregate allowance/report RPCs remain only for compatibility with old site-key runtimes and are deliberately bypassed by exact funded turns.

Accounting and reconciliation

Funded costs use integer micro-US-dollars and a versioned Luna price catalog. The ledger separately records ordinary input, cached input, cache writes, output/reasoning, long-context pricing, provider request identity, and tool fees. Prompt and response content is not stored in the funded ledger.

The admin Site Settings page shows free/member pool committed and reserved exposure. For provider reconciliation, use a dedicated OpenAI project and set:

  • site_funded_codex_openai_project_id;
  • site_funded_codex_openai_admin_key (an organization admin key, not the normal provider API key).

Refresh the pool card to compare current-period local committed cost with the OpenAI Costs API. OpenAI billing data can lag, so short-lived discrepancies are expected; persistent discrepancies require investigation before widening a rollout.

Site Key Refresh Behavior

Project-host caches site key fetches and refreshes on:

  • cache expiry
  • explicit force refresh after an auth failure retry path

Current intent:

  • avoid frequent polling across many hosts
  • recover quickly when key rotates or cache is stale

Important Runtime Notes

  • The user does not get shell access to the Codex runtime container.
  • For personal API-key auth, project-host injects provider config and OPENAI_API_KEY into the Codex runtime.
  • For site-funded auth, the runtime only receives a turn-scoped local proxy token. The host forwards the real API key to OpenAI.
  • For subscription auth, Codex reads file-based auth from mounted /root/.codex/auth.json and /root/.codex/config.toml sourced from host secrets, not from workspace files.

Known Limitations / Future Work

  • OpenAI's Costs API requires a separate organization admin key and reports daily buckets, so reconciliation is not instantaneous.
  • Account-home compatibility projection into the legacy ai_usage_log UI is separate from the exact seed ledger; the Codex UI reads exact funded remaining allowance directly.
  • Strong end-to-end project-host websocket authz is still a separate hardening project.

Fast Ban / Kill-Switch Approach

When an account must be disabled quickly (abuse, ToS violations, security response), we use a layered approach:

Current behavior

  • New funded turns can be blocked immediately with site_funded_codex_enabled, the global pool, an account hold, or launch_disable_ai.
  • Running funded turns remain bounded by their local signed reservation, duration, request count, and cost limit even if the seed is unavailable.
  • Credential-backed turns (ChatGPT plan / user API key) are not billed to CoCalc, so the immediate financial risk is lower, but account-level enforcement still depends on connection/session auth controls.

Intended end state (with project-host auth hardening)

  • Central hub marks account as banned and stops issuing/refreshing project-host auth leases.
  • Hub pushes a kick event to project-hosts to disconnect live sockets for that account.
  • Project-host closes active Codex sessions/streams for the banned account.
  • Reconnect and token refresh attempts fail, so access does not resume.

This gives both:

  • low-friction long-lived sessions for legitimate users, and
  • fast forced eviction for abuse cases without waiting for long token expiry windows.

Quick Debug Checklist

If a turn uses the wrong auth source or fails unexpectedly:

  1. Check current payment source in account settings UI panel.
  2. Check host logs for resolved auth source (project-host:codex-auth and project-host:codex-project).
  3. Verify presence/absence of local subscription auth files in codex-subscriptions/<account_id>.
  4. Verify central credential existence via hub.system.listExternalCredentials.
  5. For site-funded mode, verify the Site Settings pool card, reservation denial code, project-host outbox, and project-host:site-funded-codex-proxy logs.