RFC 006: Agentic RL through Harness Interception — token-faithful traces for TRL

[rycerzes]

Summary

Adds an RFC for training the policy driving a black-box agent harness (Pi, OpenCode, Claude Code, …) inside OpenEnv by intercepting its LLM calls at an OpenAI-compatible boundary and recording a token-faithful trace — (prompt_ids, response_ids, loss_mask, response_logprobs, reward) — that TRL consumes via rollout_func.

This turns RFC 005's CLIHarnessAdapter.run_white_box stub (currently NotImplementedError) into a real seam. It is the installed-agent training path — the counterpart to the external-agent pattern TRL's Harbor integration already covers.

Design-only RFC PR — no runtime behavior changes.

What's in it

• Ownership boundary with TRL (OpenEnv owns interception + trace; TRL owns generation, weight sync, advantages, IS correction).
• 15 design decisions with rationale/trade-offs, each backed by cross-framework evidence (NVIDIA Polar/ProRL, verifiers, AReaL, Agent Lightning, rLLM) and recent literature (TIM, SAO, TITO, loss-mask/provenance work).
• Token-level trace contract, TITO discipline via TRL's chat_template_utils, prefix-break forking, session-ID-as-API-key auth, synthetic SSE replay, weight-sync fencing, proxy-enforced budgets.
• Mermaid diagrams: architecture, prefix-break forking, request lifecycle.

Relationship to other work

• Tracking issue: Agentic RL with black-box harnesses in OpenEnv via LLM-call interception #940
• Extends RFC 005 (rfcs/005-agentic-harnesses.md) — reconciles with it, does not parallel it.
• Supersedes draft [Draft] core: harness-rollout contract + runnable example for online RL with TRL #864 (message-level contract) with the token-level contract.

Note on numbering

Filed as 006, but 006 is also used by open PRs #794 and #624 — happy to renumber to whatever's free at merge time.

cc @burtenshaw @adithya-s-k @sergiopaniego

---

Note

Low Risk
Documentation-only RFC with no runtime, auth, or training behavior changes in the diff.

Overview
Adds RFC 006 (rfcs/006-agentic-rl-harness-interception.md), a design-only spec for training black-box CLI harnesses without modifying them.

The RFC proposes an interception server that sits on the harness’s OpenAI-compatible HTTP boundary, drives generation on TRL-controlled vLLM with return_token_ids and rollout logprobs, and emits token-faithful traces (prompt_ids, response_ids, loss_mask, response_logprobs, env reward) for TRL’s rollout_func via CLIHarnessAdapter.run_white_box and build_harness_rollout_func. It spells out the OpenEnv vs TRL ownership split, a HarnessTrace contract, 15 design decisions (TITO via TRL chat_template_utils, prefix-break forking, session bearer auth, synthetic SSE, weight-sync fencing, rollout budgets, δ diagnostics, config provenance), plus architecture/request-lifecycle Mermaid diagrams and a thin GRPOTrainer wiring example.

It extends RFC 005 (fills the run_white_box stub) and supersedes draft #864’s message-level seam with a token-level contract. No application code changes in this PR.

^{Reviewed by Cursor Bugbot for commit 45c93d7. Bugbot is set up for automated code reviews on this repo. Configure here.}