PROMETHEUS

PROMETHEUS is an event-driven Java framework for building multimodal digital agents with explicit state-machine control, a developing regulation layer, and structured behaviour output.

Why

Many agent systems still treat interaction as turn-based chat: a user says something and the agent replies. That is too narrow for digital agents that must work with voice, facial expression, hand signs, group context, weather, and other signals while deciding when to speak, gesture, stay silent, or yield.

PROMETHEUS focuses on the mapping problem behind multimodal agents: how sensed events from humans and environments become inspectable agent state, and how that state becomes coordinated speech, nonverbal behaviour, motion intent, or display output.

What

PROMETHEUS provides an engineering framework for digital agents that may remain screen-based, be embodied as VR avatars, or be connected to physical robots. Agents are explicit state machines, not opaque chat loops. Their task logic is implemented with states, transitions, guards, actions, prompts, and storage.

The framework treats multimodality as a first-class contract:

• Perception clients publish observations as Event objects.
• Agents declare accepted observations and emitted behaviour modalities through an AgentInteractionProfile.
• Agent responses are persisted as BehaviourPlan events with speech, nonVerbal, motion, and display channels.
• Streaming clients subscribe to behaviour events and render the channels that their target avatar, robot, or UI can support.

How

Every runtime input is normalized as an event. A user utterance, facial emotion sample, social grouping report, hand sign, weather context, system tick, and internal regulation signal use the same event pipeline. The current state decides whether the event changes control flow, updates storage, or triggers a new behaviour plan.

The regulation foundation runs alongside task control, can maintain persisted variables, and can emit internal opportunities back through the state machine. Direct multimodal motivation, arbitration, and modulation of generated behaviour are not complete yet. The state machine therefore remains explicit and authoritative while the regulation layer develops.

Bundled Clients

Valerian Access Management

URL: http://localhost:8080/valerian-admin/

Use this client to create new access codes and assign the agent types made available in the scope of each access code. A valid access code must be entered before using Valerian Cockpit.

Valerian Cockpit

URL: http://localhost:8080/valerian/

Valerian Cockpit is the primary all-in-one client for trying the core agents that ship with PROMETHEUS. After entering an access code, open the drawer on the right with the heartbeat button, choose an agent type, create an instance, connect to it, and reset or delete it when needed. The drawer's diagnostics tab shows runtime events and agent state.

The cockpit is organised into three columns: sensing, verbal interaction by text or speech, and behaviour. Each column can be maximised or opened in a separate window when an experiment needs more screen space.

Social Context Sensitivity

This core agent demonstrates social-context sensing.

• Sensing: visual detection of human presence, groups of humans, and whether people are attentive toward the agent.
• Interaction: the agent comments on the social situation; users can also enter utterances or run the interaction in speech-to-speech mode.
• Behaviour: the full behaviour spectrum, including speech, gesture, facial expression, gaze, motion, hand signs, and display output where supported.

Facial Expression Sensitivity

This core agent demonstrates facial-expression sensing.

• Sensing: visual detection of faces and emotion, valence, and arousal.
• Interaction: the agent comments on the social situation; users can also enter utterances or run the interaction in speech-to-speech mode.
• Behaviour: the full behaviour spectrum, including speech, gesture, facial expression, gaze, motion, hand signs, and display output where supported.

Rock-Scissor-Paper

This core agent demonstrates a hand-sign game loop.

• Sensing: visual detection of the user's hand sign: rock, scissor, or paper.
• Interaction: ready, draw a sign, and receive the evaluation of who won; text- and speech-based interaction remain available.
• Behaviour: the full behaviour spectrum, plus the additional hand sign drawn by the agent.

Talk to Me

URL: http://localhost:8080/public/talktome

Talk to Me is a reduced, public-facing Valerian-style output-only speech client. An administrator first assigns core.talk_to_me to an access code. The user then enters that code and explicitly creates, selects, and deletes their own scoped speech instances.

Enter up to 2,000 Unicode code points and choose Speak to persist and speak that exact text without language-model rewriting. The client exposes the OpenAI Speech voices, output speed, browser speaker selection, and speaker refresh; alloy is the fresh-user voice default while an explicitly saved choice is retained. Voice and speed are locked only while a synthesis/playback request is active. Speaker selection remains available and applies immediately. The client does not request microphone access.

The supplied sample is loaded into the speech field on every page load. The icon controls above the field restore that sample or clear the field; they become available with the textarea after selecting an instance.

The browser sends one scoped speech request. PROMETHEUS first persists the canonical BehaviourPlan, then sends that plan's unchanged speech channel to OpenAI's output-only Speech API. The browser buffers the returned MP3 before playback and reports completion only when the audio element emits ended. Stop aborts an in-flight request or stops current playback. The 2,000-code-point boundary remains an application policy.

API Workbench

URL: http://localhost:8080/apiworkbench/

The API Workbench is a guided developer client for learning and testing the PROMETHEUS REST and SSE API. Start with the lifecycle column: open a scoped session, list allowed agent definitions, create or select an agent, inspect its interaction profile, subscribe to behaviour or monitor streams, and publish observation events. The endpoint catalog can also be used directly to inspect resolved URLs, path variables, headers, query parameters, JSON request bodies, copyable fetch, curl, and EventSource snippets, HTTP responses, SSE events, and the active agent profile.

Multilateral Displays

URLs:

• Listener display: http://localhost:8080/multilateral/listen/
• Reports display: http://localhost:8080/multilateral/reports/

The multilateral screens are separate meeting/group displays. They are not part of the Valerian cockpit workflow, but remain bundled for situations where a larger audience should see live listening state or generated meeting reports.

Current Agent Catalog

Production agent definitions live under src/main/java/ch/zhaw/prometheus/agentdefs. Definitions implement AgentDefinition, expose a stable key, and are discovered as Spring beans.

The main branch ships the Valerian baseline catalog:

Key	Purpose
core.facial_expression_sensitivity	Core demo for facial-expression observations.
core.multimodal_behaviour	Core demo for coordinated multimodal output.
core.rock_scissor_paper	Core hand-sign rock-scissor-paper demo.
core.role_clarification_guessing_game	Core guessing game focused on agent/user role clarity.
core.social_context_sensitivity	Core demo for social grouping and rich social context.
core.talk_to_me	Deterministic exact-text Realtime speech utility.
usecases.healthcare.guessing_game	Healthcare guessing game where Valerian guesses.
usecases.healthcare.guessing_game_user_guess	Healthcare guessing game where the user guesses.
usecases.healthcare.healthcare_conversation	Open healthcare conversation use case.
usecases.healthcare.smart_goal_coaching	Healthcare SMART-goal coaching use case.
usecases.healthcare.therapy_appointment_reminder	Single-state therapy appointment reminder.
usecases.healthcare.therapy_appointment_reminder_intro	Two-state therapy appointment reminder with introduction.

Event- or experiment-specific agents should live in application branches, separate modules, or deployment-specific code rather than being added to the main baseline catalog.

Requirements

• Java 21 or newer.
• MySQL.
• Maven Wrapper from this repository.
• Node.js only when running the Playwright visual smoke tests.
• OpenAI or Azure OpenAI configuration for prompt generation, Realtime speech, and output-only Speech synthesis.

Local Setup

Copy the template files and adjust them for your machine:

Copy-Item src/main/resources/application.properties.template src/main/resources/application.properties
Copy-Item src/main/resources/openai.properties.template src/main/resources/openai.properties

Minimum configuration:

• spring.datasource.url
• spring.datasource.username
• spring.datasource.password
• openai.openaivsazureopenai
• openai.url
• openai.key
• prometheus.talktome.speech.model and prometheus.talktome.speech.url when overriding Talk to Me's output-only Speech defaults
• prometheus.admin.token for Valerian Access Management

Run the application:

.\mvnw.cmd spring-boot:run

The default local URL is http://localhost:8080; it redirects to Valerian Cockpit.

Open the main surfaces:

• Valerian Cockpit: http://localhost:8080/valerian/
• Valerian Access Management: http://localhost:8080/valerian-admin/
• Talk to Me: http://localhost:8080/public/talktome
• API Workbench: http://localhost:8080/apiworkbench/
• Multilateral listener: http://localhost:8080/multilateral/listen/
• Multilateral reports: http://localhost:8080/multilateral/reports/

The standalone German/English SIRA/PROMETHEUS public page lives under .web/. Open .web/index.html directly in a browser, or host the .web/ directory as static files.

The standalone German study participation site lives under .web/participate/. It is self-contained for deployment as a separate host root. Before deploying, copy .web/participate/.env.example to .web/participate/.env, fill in the database and mail values, and execute .web/participate/database/schema.sql followed by .web/participate/database/seed.sql in the target MySQL database. ADMIN_NOTIFY_EMAIL may contain a comma-separated list of addresses that are added as BCC recipients on participant confirmation mails.

The participation admin view is available at /admin/ below that deployment root, for example https://participate.siralab.ch/admin/. It intentionally has no built-in authentication; protect or obscure the deployed folder name at the hosting level if needed. The table supports client-side search, sortable columns, deletion of registrations, and CSV export of the full loaded registration set. Deleting a registration removes its e-mail reservation and server-token summary, so the participant can register again from the same browser.

For local backend testing, use the provided .env.test and reset the local MySQL test database:

$env:PARTICIPATE_ENV_FILE = (Resolve-Path .web/participate/.env.test).Path
php .web/participate/tests/setup_test_db.php
php -S 127.0.0.1:8091 -t .web/participate

Testing

Run the Java regression suite:

.\mvnw.cmd test

Run JavaScript syntax checks for the bundled clients:

node --check src/main/resources/public/valerian/script.js
node --check src/main/resources/public/apiworkbench/script.js
node --check src/main/resources/public/talktome/script.js
node --check .web/participate/assets/app.js
node --check tests/playwright/valerian-column-expansion.spec.mjs
node --check tests/playwright/apiworkbench.spec.mjs
node --check tests/playwright/talktome.spec.mjs
node --check tests/playwright/participate.spec.mjs
node --check playwright.participate.config.mjs
php -l .web/participate/index.php
php -l .web/participate/config/bootstrap.php
php -l .web/participate/api/register.php
php -l .web/participate/api/registration.php
php -l .web/participate/admin/delete.php
php -l .web/participate/admin/index.php
php -l .web/participate/tests/setup_test_db.php
node --check .web/participate/admin/admin.js

Run the Playwright visual smoke tests:

npm install
npx playwright install chromium
npm run test:valerian:visual
npm run test:apiworkbench:visual
npm run test:talktome:visual
npm run test:participate:visual

The Valerian Playwright test starts or reuses http://127.0.0.1:8080, creates or re-enables access code VX102 through the admin API, and checks the facial expression report, social context report, and behaviour board. The API Workbench Playwright test uses deterministic mocked API responses to verify the guided lifecycle, snippets, request execution, and SSE viewer. Set PROMETHEUS_ADMIN_TOKEN when your local prometheus.admin.token differs from the test default. Set PROMETHEUS_SKIP_WEBSERVER=true when the app is already running.

The Talk to Me Playwright test uses the running Spring application and its configured test database for access-code assignment, scoped agent lifecycle, exact event/behaviour persistence, synthesis request mapping, audio completion, Stop, and deletion. It replaces only the external OpenAI Speech and physical speaker boundary with deterministic browser fakes, then checks the light desktop and dark mobile layouts. It uses access code TTM31 and the same admin-token environment override.

The participate Playwright test resets sira_participate_test through .web/participate/tests/setup_test_db.php, starts PHP's built-in server for .web/participate/ with .env.test, and verifies the landing page, registration wizard, validation, privacy modal, MySQL-backed registration, logged confirmation mail, duplicate e-mail rejection, returning-summary lookup, mobile layout, and the /admin/ registration table with search, sorting, and CSV export. It also verifies admin deletion and same-browser re-registration after deletion.

Connecting External Clients

External clients usually use the scoped demo API. It keeps agent instances behind an access code and mirrors what the Valerian cockpit does. Trusted backend tools can use the global agent endpoints shown later.

1. Open a scoped session

Create an access code and assign agent types in Valerian Access Management, or use the admin API. Then validate the code:

POST /demo/session
Content-Type: application/json

{
  "accessCode": "VX102"
}

The response contains the enabled agent types and existing scoped agents.

2. Create or select an agent

List agent types available to the code:

GET /demo/agent-types
X-Prometheus-Access-Code: VX102

Create an assigned agent type:

POST /demo/agents
X-Prometheus-Access-Code: VX102
Content-Type: application/json

{
  "agentDefinitionKey": "core.social_context_sensitivity"
}

Read the agent metadata before enabling perception or rendering controls:

GET /demo/agents/{agentId}/info
X-Prometheus-Access-Code: VX102

Relevant response fields:

{
  "id": "11111111-1111-1111-1111-111111111111",
  "name": "Valerian Core social context sensitivity",
  "description": "English Valerian Core agent for social-context sensing.",
  "active": true,
  "languageCode": "en",
  "interactionProfile": {
    "supportedObservations": [
      "obs.user_utterance",
      "obs.human.presence",
      "obs.social.grouping",
      "obs.social.context"
    ],
    "supportedBehaviourModalities": [
      "speech",
      "nonVerbal.gesture",
      "nonVerbal.facialExpression",
      "nonVerbal.gaze",
      "nonVerbal.motion"
    ],
    "profileTags": []
  }
}

Clients should use supportedObservations to decide which sensing UI or sensors to enable, and supportedBehaviourModalities to decide which behaviour channels to render. An empty profile means "unknown"; fall back conservatively.

3. Publish perception events

Send observations to the agent with acknowledge:

POST /demo/agents/{agentId}/acknowledge
X-Prometheus-Access-Code: VX102
Content-Type: application/json

{
  "type": "obs.user_utterance",
  "actor": "user",
  "kind": "observation",
  "payload": "What do you notice about this group?"
}

payload is always a string. For structured observations, encode the structured payload as JSON inside the string:

{
  "type": "obs.hand.sign",
  "actor": "user",
  "kind": "observation",
  "payload": "{\"source\":\"external.camera\",\"hand\":\"right\",\"sign\":\"rock\",\"confidence\":0.93,\"detectionMode\":\"client_camera\",\"ts\":\"2026-07-08T09:00:00Z\"}"
}

The response is:

{
  "responseEvent": {
    "type": "resp.behaviour_plan",
    "actor": "assistant",
    "kind": "response",
    "payload": "{\"speech\":\"I saw rock. I will reveal mine now.\",\"motion\":{\"handSign\":\"paper\"}}",
    "createdDate": "2026-07-08T09:00:01Z",
    "statePath": ["Valerian Core RPS Reveal Sign"]
  },
  "active": true
}

responseEvent may be null when the event updates context but does not trigger new behaviour.

Supported observation event types in the current public contract:

Event type	Actor	Payload
obs.user_utterance	user	Plain utterance text.
obs.emotion.face	user	JSON string with emotion, confidence, valence, arousal, optional expressions, source, and timestamp.
obs.human.presence	user	JSON string with aggregate human/tracked counts and source.
obs.social.grouping	user	JSON string with group count, singleton count, largest group size, groups, and source.
obs.social.context	user	JSON string with schemaVersion: 1, aggregate group fields, and per-person movement/attention fields.
obs.hand.sign	user	JSON string with sign as rock, scissor, or paper, plus confidence/source fields.
obs.weather.current	system	JSON string with location, condition, intensity, wind, temperature, precipitation, and timestamp.
obs.weather.forecast	system	JSON string with location and days[] forecast entries.

Use only observations declared by the agent profile unless you are deliberately testing fallback behaviour.

4. Subscribe to behaviour

Behaviour clients subscribe with Server-Sent Events:

GET /demo/agents/{agentId}/behaviour/stream?accessCode=VX102
Accept: text/event-stream

Browser EventSource cannot set custom headers, so scoped browser clients pass accessCode as a query parameter. Non-browser clients may use the X-Prometheus-Access-Code header instead.

Each behaviour event has:

• SSE event name: behaviour
• SSE id: persisted event id when available
• SSE data: an Event object with type resp.behaviour_plan

The event payload is a JSON string containing a BehaviourPlan:

{
  "speech": "That looks like a small group.",
  "nonVerbal": {
    "gesture": "ACKNOWLEDGE",
    "facialExpression": { "type": "attentive", "intensity": 0.55 },
    "gaze": { "direction": "toward_group", "focus": "group" },
    "motion": { "stillness": 0.75, "energy": 0.35 }
  },
  "motion": {
    "handSign": "paper"
  },
  "display": {
    "title": "Social context",
    "summary": "Two people nearby"
  }
}

Clients should ignore channels they cannot render. Reconnect with either the standard Last-Event-ID header or ?lastEventId=<id> to replay missed behaviour events.

5. Request generated behaviour without a new perception event

Some clients need backend complement behaviour after realtime speech, or want to ask the current state to generate again:

POST /demo/agents/{agentId}/behaviour/generate
X-Prometheus-Access-Code: VX102
Content-Type: application/json

{
  "outputProfile": "BACKEND_COMPLEMENT",
  "omitModalities": ["speech"]
}

Valid output profiles are:

• FULL_PLAN: speech and supported non-speech channels.
• REALTIME_SPEECH: speech-only prompt contract for realtime speech.
• BACKEND_COMPLEMENT: non-speech complement derived from the current context.

The endpoint returns 200 when behaviour was generated, 409 when no behaviour was produced, 404 when the agent is missing, and 400 for an unknown profile.

6. Monitor state and storage

Use these endpoints for diagnostics and operator UI:

GET /demo/agents/{agentId}/state
GET /demo/agents/{agentId}/states
GET /demo/agents/{agentId}/storage
GET /demo/agents/{agentId}/eventhistory
GET /demo/agents/{agentId}/monitor/stream?accessCode=VX102

The monitor SSE stream emits snapshot events with current state, inner state chain, active flag, known states, and storage entries.

Global Agent API

The global API is useful for trusted tools, tests, or internal services. It is not access-code scoped.

Method	Path	Purpose
GET	/agent	List persisted agents.
POST	/agent/singlestate	Create an ad-hoc single-state agent.
GET	/{agentId}/info	Agent metadata and interaction profile.
POST	/{agentId}/start	Start the current state.
DELETE	/{agentId}/reset	Reset the agent.
POST	/{agentId}/acknowledge	Publish an event.
GET	/{agentId}/prompt?profile=...	Inspect the prompt contract.
POST	/{agentId}/behaviour/generate	Generate behaviour from current state.
GET	/{agentId}/behaviour/stream	Subscribe to behaviour SSE.
GET	/{agentId}/monitor/stream	Subscribe to monitor SSE.

The request and response shapes are the same as the scoped demo API, without the access-code header.

Realtime Speech

Realtime speech clients do not talk directly to OpenAI. They post a WebRTC SDP offer to PROMETHEUS:

POST /demo/agents/{agentId}/realtime/call?voice=cedar&turnDetection=server_vad
X-Prometheus-Access-Code: VX102
Content-Type: application/sdp

v=0
...

PROMETHEUS creates the OpenAI Realtime call with the agent's current REALTIME_SPEECH prompt, keeps a backend sideband connection, accepts transcripts into the normal event pipeline, persists canonical assistant speech as resp.behaviour_plan, and asks Realtime to speak that exact text.

Close calls with:

DELETE /realtime/calls/{callId}

Transcription-only clients can request a session with:

POST /realtime/transcription/session?agentId={agentId}

Output-only Speech

Talk to Me does not create a Realtime call. It sends the observation and speech options to a scoped backend endpoint:

POST /demo/talktome/agents/{agentId}/speech?voice=cedar&speed=1.25
X-Prometheus-Access-Code: TTM31
Content-Type: application/json

{"type":"obs.user_utterance","actor":"user","kind":"observation","payload":"Exact text"}

The dedicated endpoint first verifies that the scoped agent carries the utility.talk_to_me profile tag. PROMETHEUS then acknowledges the observation with the REALTIME_SPEECH output profile, persists the deterministic core.talk_to_me speech plan, and passes that canonical speech string to prometheus.talktome.speech.url using prometheus.talktome.speech.model (default: gpt-4o-mini-tts). The endpoint returns uncached MP3 audio. Unsupported voices or speeds outside 0.25 through 4.0 are rejected before synthesis. The shared scoped-agent controller, Realtime configuration, and Realtime speech selection remain independent of this output-only path.

Admin API

Admin endpoints require:

X-Prometheus-Admin-Token: <prometheus.admin.token>

Method	Path	Purpose
GET	/admin/agent-types	List registered AgentDefinition types with package metadata.
GET	/admin/access-code-presets	List backend-defined access-code presets.
POST	/admin/access-code-presets/{presetKey}/apply	Create a preset bundle transactionally.
POST	/admin/access-codes	Create an access code.
GET	/admin/access-codes	List access codes.
PATCH	/admin/access-codes/{id}	Enable or disable a code.
PUT	/admin/access-codes/{id}/agent-types	Replace assigned agent type keys.
GET	/admin/access-codes/{id}/agents	List agents linked to a code.

Access codes must be exactly five ASCII letters or digits. The backend treats them as case-sensitive.

CORS

Bundled clients are same-origin. External browser clients need an explicit allowlist:

prometheus.cors.allowed-origins=http://127.0.0.1:5010,http://localhost:5010
prometheus.cors.allowed-origin-patterns=http://127.0.0.1:*,http://localhost:*

Keep this narrow because the scoped access code acts as a bearer-style client credential.

Repository Structure

src/main/java/ch/zhaw/prometheus
  agentdefs/        Registered Valerian agent definitions.
  application/      Application services for agents, access codes, Realtime, and scoped demos.
  controllers/      HTTP, SSE, admin, scoped demo, and static-client endpoints.
  logging/          SSE broadcasters.
  model/            Agent, state machine, event, behaviour, policy, regulation, and RPS domain model.
  spi/              OpenAI and Realtime integration boundary.

src/main/resources/public
  apiworkbench/     Guided REST/SSE API workbench for client developers.
  multilateral/     Meeting/group listener and report displays.
  talktome/         Public exact-text output-only Speech client.
  valerian/         Valerian cockpit.
  valerian-admin/   Valerian access management.

.web/
  index.html        Standalone German/English SIRA/PROMETHEUS public page.
  participate/      Standalone German study participation site.
    admin/          Unprotected registration overview with search, sort, and CSV export.
    api/            PHP JSON endpoints for registration and returning-summary lookup.
    assets/         Plain CSS and JavaScript for the landing page and wizard.
    config/         Local `.env` loading, PDO, JSON, cookie, and mail helpers.
    database/       MySQL schema and seed files for phpMyAdmin deployment.
    tests/          Local MySQL reset helper for integration smoke tests.

tests/playwright    Browser-level Valerian, Talk to Me, API Workbench, and participate smoke tests.

Developing New Agents

1. Start from an existing definition in agentdefs/core or agentdefs/usecases/healthcare.
2. Define prompts for state entry, response generation, transitions, actions, and outcome extraction.
3. Declare an AgentInteractionProfile that lists required observations and emitted behaviour modalities.
4. Implement AgentDefinition with a stable key and expose it as a Spring bean.
5. Use createInstance(...) for startup behaviour that should run immediately after scoped creation.
6. Add prompt/profile contract tests and update README/API examples if the public contract changes.

Prefer clear replacement over compatibility shims while the framework remains prototype-oriented.

Deployment Notes

The repository contains Heroku/container-oriented resources:

• Dockerfile
• src/main/resources/application-prod.properties
• src/main/resources/openai-prod.properties
• .github/workflows/deployment.yml

Production deployments must provide database credentials and OpenAI credentials through environment variables or platform config vars.

Project Notes

• .agents/messageinabottle.txt is the compact session bootstrap prompt for a new coding agent.
• .agents/CODEX.md contains reusable, project-neutral engineering and milestone practices.
• .agents/CONTEXT.MD describes PROMETHEUS's purpose, current capabilities, regulation gaps, architecture, and repository boundaries.
• The top of PROJECT.md is the current engineering snapshot. The remaining milestone records are a historical audit to search selectively, not required startup reading.