Skip to content

k8s-ee-config-reference.md

Latest commit

 

History

History
819 lines (596 loc) · 20.3 KB

k8s-ee-config-reference.md

File metadata and controls

819 lines (596 loc) · 20.3 KB

k8s-ee Configuration Reference

Complete reference for the k8s-ee.yaml configuration file used by ephemeral PR environments.

Quick Start

Minimal configuration requires only projectId:

# k8s-ee.yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/koder-cat/k8s-ephemeral-environments/main/.github/actions/validate-config/schema.json
projectId: myapp

Tip: Add the schema comment for IDE autocompletion and validation.

This creates PR environments at myapp-pr-{number}.k8s-ee.genesluna.dev with sensible defaults.

Full Configuration Example

# k8s-ee.yaml - Full example with all options
projectId: myapp
trigger: automatic    # or "on-demand" for /deploy-preview command

app:
  port: 3000
  healthPath: /health
  metricsPath: /metrics

image:
  context: .
  dockerfile: Dockerfile
  repository: ghcr.io/myorg/myapp  # Optional, auto-generated if not set

resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 384Mi

env:
  NODE_ENV: production
  LOG_LEVEL: info

envFrom:
  - secretRef:
      name: my-secret
  - configMapRef:
      name: my-config

databases:
  postgresql: true
  mongodb: false
  redis: false
  minio: false
  mariadb: false

ingress:
  enabled: true
  annotations:
    custom.annotation/key: value

metrics:
  enabled: false
  interval: 30s

Field Reference

projectId (required)

Unique identifier for this project in multi-tenant clusters.

Property Value
Type string
Required Yes
Min Length 1
Max Length 20
Pattern ^[a-z0-9]([a-z0-9-]{0,18}[a-z0-9])?$

Validation Rules:

  • Must be lowercase alphanumeric with hyphens
  • Must start and end with alphanumeric character
  • Maximum 20 characters (leaves room for -pr-{number} suffix)

Examples:

projectId: myapp           # Valid
projectId: my-cool-app     # Valid
projectId: my-app-123      # Valid
projectId: MyApp           # Invalid: uppercase
projectId: my_app          # Invalid: underscore
projectId: -myapp          # Invalid: starts with hyphen
projectId: this-is-a-very-long-project-name  # Invalid: too long

trigger

Controls how PR environments are created.

Property Value
Type string
Required No
Default automatic
Values automatic, on-demand

automatic (default): Environment is created automatically when a PR is opened or updated. This is the standard behavior.

on-demand: Environment is only created when someone comments /deploy-preview on the PR. After creation, subsequent pushes auto-redeploy. Use /destroy-preview to tear down the environment early.

trigger: on-demand

Note: On-demand mode requires trigger: on-demand in your k8s-ee.yaml. The universal workflow template from the Onboarding Guide handles both modes — once you have it, switching is a one-line config change. See On-Demand Environments for details.

app

Application settings for the deployed container.

app.port

Property Value
Type integer
Default 3000
Minimum 1
Maximum 65535

Container port the application listens on. The platform automatically configures:

  • Deployment: Sets the container port
  • NetworkPolicy: Allows ingress traffic on this port from Traefik

Common Configurations:

app:
  port: 3000  # Node.js, Express, NestJS (default)
  port: 8080  # .NET, Go, Java Spring Boot
  port: 8000  # Python FastAPI, Django

app.healthPath

Property Value
Type string
Default "/health"
Pattern ^/.*

Health check endpoint path used for liveness and readiness probes.

app:
  healthPath: /api/health

app.metricsPath

Property Value
Type string
Default (none)
Pattern ^/.*

Metrics endpoint path for Prometheus scraping. Only needed if metrics.enabled: true.

app:
  metricsPath: /metrics

image

Docker image build configuration.

image.context

Property Value
Type string
Default "."

Docker build context path relative to repository root.

image:
  context: ./backend

image.dockerfile

Property Value
Type string
Default "Dockerfile"

Path to Dockerfile relative to the build context.

image:
  dockerfile: Dockerfile.prod

image.repository

Property Value
Type string
Default (auto-generated)

Custom image repository URL. If not set, auto-generated based on the registry type:

  • GHCR (default): ghcr.io/{owner}/{repo}/{project-id}
  • ECR: <account-id>.dkr.ecr.<region>.amazonaws.com/{owner}/{repo}/{project-id}
image:
  repository: ghcr.io/myorg/custom-image

resources

Container resource requests and limits. Must fit within cluster LimitRange (max 512Mi memory per container).

resources.requests.cpu

Property Value
Type string
Default "50m"
Pattern ^[0-9]+(m|[0-9]*)?$

CPU request in millicores (e.g., 50m, 100m, 0.5).

resources.requests.memory

Property Value
Type string
Default "128Mi"
Pattern ^[0-9]+(Mi|Gi)$

Memory request (e.g., 128Mi, 256Mi, 1Gi).

resources.limits.cpu

Property Value
Type string
Default "200m"
Pattern ^[0-9]+(m|[0-9]*)?$

CPU limit in millicores.

resources.limits.memory

Property Value
Type string
Default "384Mi"
Pattern ^[0-9]+(Mi|Gi)$

Memory limit. Maximum allowed: 512Mi (cluster LimitRange).

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

env

User-defined environment variables injected into the application pod as key-value pairs. These are stored in a Kubernetes ConfigMap ({namespace}-app-config) and mounted via envFrom.

Property Value
Type object
Default {}
Key Pattern ^[A-Za-z_][A-Za-z0-9_]*$ 
env:
  NODE_ENV: staging
  LOG_LEVEL: info
  JWT_SECRET: "my-ephemeral-secret-not-for-production"
  AUTH_BYPASS_LDAP: "true"

How it works: The env values flow through the deployment pipeline:

  1. validate-config parses k8s-ee.yaml and outputs them as env-json
  2. The reusable workflow passes env-json to the deploy-app action
  3. deploy-app passes them to Helm via --set-json env=...
  4. The Helm chart injects them into the app ConfigMap alongside platform variables (PORT, PR_NUMBER, etc.)

Verification: To confirm your env vars reached the pod:

kubectl get configmap {namespace}-app-config -n {namespace} -o yaml

Note: Database connection variables (DATABASE_URL, PGHOST, MINIO_ENDPOINT, etc.) are injected separately by database charts and do not need to be listed in env. Variables like MINIO_BUCKET that are configured via databases.minio.bucket are also injected automatically.

envFrom

Environment variables from Kubernetes secrets or configmaps.

Property Value
Type array
Default []

Each item must have exactly one of secretRef or configMapRef (not both).

envFrom:
  - secretRef:
      name: database-credentials
  - configMapRef:
      name: my-shared-config

writableDirs

Directories that need to be writable at runtime. Each directory is mounted as an emptyDir volume. Required because containers run with readOnlyRootFilesystem: true for security.

Property Value
Type array of strings
Default []
Path Pattern Must start with / 
writableDirs:
  - /app/upload
  - /app/data

Note: /tmp is always writable (mounted automatically). Only declare directories outside of /tmp that your app needs to write to.

databases

Database configuration. All databases are disabled by default (opt-in).

When enabled, databases are automatically deployed to your PR environment and connection details are injected as environment variables (e.g., DATABASE_URL for PostgreSQL).

Each database can be configured as:

  • Boolean: true to enable with defaults, false to disable
  • Object: Enable with custom configuration

databases.postgresql

Property Value
Type boolean | object
Default false

PostgreSQL database using CloudNativePG operator.

# Simple enable
databases:
  postgresql: true

# With custom configuration
databases:
  postgresql:
    enabled: true
    version: "16"
    storage: 2Gi

# With bootstrap SQL (for table creation)
databases:
  postgresql:
    enabled: true
    bootstrap:
      postInitApplicationSQL:
        - |
          CREATE TABLE users (
            id SERIAL PRIMARY KEY,
            name VARCHAR(255)
          );
          GRANT ALL PRIVILEGES ON users TO app;
          GRANT USAGE, SELECT ON SEQUENCE users_id_seq TO app;

Object properties:

  • enabled: boolean (default: true)
  • version: string (default: "16")
  • storage: string (default: "1Gi", pattern: ^[0-9]+(Mi|Gi|Ti)$)
  • bootstrap.postInitApplicationSQL: array of SQL strings to run after database creation
  • bootstrap.initSQL: array of SQL strings to run on postgres database (for extensions)

Important: Bootstrap SQL runs as postgres superuser, but your app connects as app user. You must include GRANT statements for table access. Use $func$ instead of $$ for function delimiters.

Note: For production applications with evolving schemas, use Drizzle ORM migrations instead of bootstrap SQL. Migrations provide versioning, are reversible, and work with existing data.

databases.mongodb

Property Value
Type boolean | object
Default false

MongoDB database using MongoDB Community Operator.

databases:
  mongodb:
    enabled: true
    storage: 2Gi

Object properties:

  • enabled: boolean (default: true)
  • version: string
  • storage: string (default: "1Gi", pattern: ^[0-9]+(Mi|Gi|Ti)$)

databases.redis

Property Value
Type boolean | object
Default false

Redis cache using simple deployment.

databases:
  redis: true

Object properties:

  • enabled: boolean (default: true)

databases.minio

Property Value
Type boolean | object
Default false

MinIO object storage using MinIO Operator.

databases:
  minio:
    enabled: true
    storage: 5Gi

Object properties:

  • enabled: boolean (default: true)
  • storage: string (default: "1Gi", pattern: ^[0-9]+(Mi|Gi|Ti)$)

databases.mariadb

Property Value
Type boolean | object
Default false

MariaDB database using simple deployment.

databases:
  mariadb:
    enabled: true
    version: "11.4"
    storage: 2Gi

Object properties:

  • enabled: boolean (default: true)
  • version: string (default: "11.4")
  • storage: string (default: "1Gi", pattern: ^[0-9]+(Mi|Gi|Ti)$)

Resource Requirements by Database

The platform automatically calculates ResourceQuota based on enabled databases. Each PR namespace receives a quota sized for its specific configuration.

Service CPU Limit Memory Limit Storage
Application (base) 300m 512Mi 1Gi
PostgreSQL +500m +512Mi +2Gi
MongoDB +500m +512Mi +2Gi
Redis +200m +128Mi -
MinIO +500m +512Mi +2Gi
MariaDB +300m +256Mi +2Gi

Example Calculated Quotas:

Configuration CPU Limit Memory Limit Storage
App only 300m 512Mi 1Gi
App + PostgreSQL 800m 1Gi 3Gi
App + PostgreSQL + Redis 1000m 1.1Gi 3Gi
App + PostgreSQL + MongoDB 1300m 1.5Gi 5Gi
All databases enabled 2100m 2.4Gi 9Gi

The quota is calculated at namespace creation time based on the databases section in your k8s-ee.yaml. No manual intervention is required.

Note: Quotas are calculated once at namespace creation. If you add databases to an existing PR environment, close and reopen the PR to recalculate quotas, or manually patch the ResourceQuota.

Note: These are approximate values. Actual consumption varies based on workload and operator versions.

ingress

Ingress configuration for external access.

ingress.enabled

Property Value
Type boolean
Default true

Enable/disable ingress creation.

ingress.annotations

Property Value
Type object
Default {}

Additional annotations for the ingress resource.

ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
    traefik.ingress.kubernetes.io/rate-limit: "100"

metrics

Prometheus metrics configuration.

metrics.enabled

Property Value
Type boolean
Default false

Enable ServiceMonitor for Prometheus scraping.

metrics.interval

Property Value
Type string
Default "30s"
Pattern ^[0-9]+(s|m|h)$

Scrape interval (e.g., 15s, 1m, 5m).

metrics:
  enabled: true
  interval: 15s

Automatic Namespace Labeling

When metrics are enabled, the ServiceMonitor automatically adds a namespace label to all scraped metrics using Prometheus relabeling. This enables Grafana dashboards to filter metrics by namespace without requiring your application to add this label.

Your application metrics will include namespace="myapp-pr-42" automatically, matching the PR environment namespace.

Metrics Instrumentation Guide

When metrics.enabled: true, the platform deploys a ServiceMonitor that tells Prometheus to scrape GET /metrics on your app. Your app must expose this endpoint returning Prometheus text format — otherwise the Grafana "PR Developer Insights" dashboard will show App Status = DOWN and DB Connected = NO.

Required Metrics

These metrics power the Grafana dashboard panels:

Metric Name Type Labels Dashboard Panels
http_requests_total Counter method, route, status_code Request Rate, Error Rate, 5xx by Endpoint, Requests by Status
http_request_duration_seconds Histogram method, route, status_code P95 Latency, P95 by Endpoint, Slowest Endpoints
db_pool_connections_total Gauge — DB Connected, Connection Pool
db_pool_connections_idle Gauge — Connection Pool
db_pool_connections_waiting Gauge — Connection Pool
db_query_duration_seconds Histogram operation, success Query Duration, Failed Queries

Minimum requirements: A /metrics endpoint returning Prometheus text format, plus http_requests_total and http_request_duration_seconds. The database metrics are optional but recommended if your app uses a database.

Automatic metrics: The up metric (App Status panel) and kube_pod_status_phase (Pods Running panel) are provided automatically by Prometheus and kube-state-metrics — no app instrumentation needed.

Recommended Histogram Buckets
HTTP duration: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] (seconds)
DB query duration: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1] (seconds)
Node.js/Express Example (prom-client)
import * as client from 'prom-client';

const registry = new client.Registry();
registry.setDefaultLabels({
  app: 'my-app',
  pr: process.env.PR_NUMBER || 'unknown',
});
client.collectDefaultMetrics({ register: registry });

const httpRequestTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
  registers: [registry],
});

const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
  registers: [registry],
});

// GET /metrics endpoint
app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', registry.contentType);
  res.send(await registry.metrics());
});

Reference implementation: See demo-app/apps/api/src/metrics/ for a complete example with HTTP middleware, database metrics, and pool monitoring.

Common Scenarios

Node.js API with PostgreSQL

projectId: my-api

app:
  port: 3000
  healthPath: /health

env:
  NODE_ENV: production

databases:
  postgresql: true

Python Flask with Redis Cache

projectId: flask-app

app:
  port: 5000
  healthPath: /healthz

resources:
  requests:
    memory: 256Mi
  limits:
    memory: 512Mi

databases:
  redis: true

Full-Stack with Multiple Databases

projectId: fullstack

app:
  port: 8080
  healthPath: /api/health
  metricsPath: /metrics

databases:
  postgresql: true
  redis: true
  minio:
    enabled: true
    storage: 2Gi

metrics:
  enabled: true
  interval: 30s

On-Demand Environments

Save resources by creating environments only when needed:

projectId: myapp
trigger: on-demand

app:
  port: 3000
  healthPath: /health

databases:
  postgresql: true

With this configuration, environments are only created when someone comments /deploy-preview on the PR. Use /destroy-preview to tear down the environment early. Requires the universal workflow template from the Onboarding Guide.

Monorepo Backend Service

projectId: backend-svc

image:
  context: ./services/backend
  dockerfile: Dockerfile

app:
  port: 4000
  healthPath: /ready

envFrom:
  - secretRef:
      name: shared-secrets

Validation Errors

The schema validation provides clear error messages:

Error Cause Fix
projectId must match pattern Invalid characters or format Use lowercase alphanumeric + hyphens only
projectId must be <= 20 characters ID too long Shorten the project ID
app.port must be >= 1 Invalid port number Use port between 1-65535
resources.limits.memory must match pattern Invalid memory format Use format like 256Mi or 1Gi
databases.*.storage must match pattern Invalid storage format Use format like 1Gi, 500Mi, or 2Ti
env property name is invalid Invalid env var name Use pattern [A-Za-z_][A-Za-z0-9_]*
envFrom item must have secretRef or configMapRef Empty envFrom entry Specify either secretRef or configMapRef

Computed Values

These values are automatically computed and added to the configuration:

Field Formula Example
_computed.namespace {projectId}-pr-{prNumber} myapp-pr-42
_computed.previewUrl https://{namespace}.{domain} https://myapp-pr-42.k8s-ee.genesluna.dev
_computed.prNumber From workflow input 42

Platform-Injected Environment Variables

In addition to user-defined env values, the platform automatically injects these environment variables into every app pod:

Variable Source Example
PORT app.port config 3000
PR_NUMBER Workflow input 42
COMMIT_SHA Git HEAD SHA a1b2c3d4...
BRANCH_NAME PR head branch feat-my-feature
APP_VERSION Image tag pr-42
PREVIEW_URL Computed preview URL https://myapp-pr-42.k8s-ee.genesluna.dev

Database charts inject additional variables when databases are enabled (e.g., DATABASE_URL, PGHOST, MINIO_ENDPOINT).

CORS tip: If your application has a CORS allowlist, add process.env.PREVIEW_URL to it so the frontend served from the preview domain can make API calls. Example (Express.js):

const allowedOrigins = [
  'http://localhost:3000',
  process.env.PREVIEW_URL,  // k8s-ee preview domain
].filter(Boolean);

See Also