Skip to content

AGENTS.md

Latest commit

 

History

History
432 lines (320 loc) · 42.5 KB

AGENTS.md

File metadata and controls

432 lines (320 loc) · 42.5 KB
Raw

AGENTS.md - Training Hub

Guidelines for AI agents working in this codebase.

Project Overview

Training Hub is an algorithm-focused interface for common LLM training, continual learning, and reinforcement learning techniques. The goal is to expose common training algorithms in an intuitive and easy-to-use way, abstracting away backend complexity. Training Hub is designed to support as many backends as necessary—the current implementations are just the starting point.

  • Language: Python 3.11+
  • License: Apache-2.0
  • Primary Author: Red Hat AI Innovation Team

For the current list of supported algorithms, backends, and dependencies, see:

  • pyproject.toml - Dependencies and optional extras
  • src/training_hub/__init__.py - Public API exports
  • README.md - User-facing documentation and support matrix

Quick Commands

# Install in editable mode (development)
pip install -e .

# Install with LoRA support
pip install -e .[lora]

# Install with GRPO support (includes ART + verl backends)
pip install -e .[grpo,lora]

# Install with CUDA support (install sequentially after other extras)
pip install -e .[cuda] --no-build-isolation

# Install with development dependencies
pip install -e .[dev]

# Run tests
pytest tests/

# Serve documentation locally (requires docsify-cli)
cd docs && docsify serve

See pyproject.toml for the full list of optional dependency groups.

Code Organization

src/training_hub/
├── __init__.py              # Public API exports
├── hub_core.py              # Core utilities
├── utils.py                 # Shared utilities (torchrun params, type formatting)
├── visualization.py         # plot_loss() for training curves
├── algorithms/
│   ├── __init__.py          # Base classes: Algorithm, Backend, AlgorithmRegistry
│   ├── sft.py               # Supervised Fine-Tuning
│   ├── osft.py              # Orthogonal Subspace Fine-Tuning
│   ├── lora.py              # LoRA + SFT
│   ├── lora_grpo.py         # LoRA + GRPO and GRPO (ART backend, algorithm, convenience fns)
│   ├── lora_grpo_verl.py    # verl backend for LoRA + GRPO and GRPO
│   ├── rewards.py           # Reward functions (tool_call_reward, binary_reward)
│   ├── verl_tool_agent.py   # Custom verl agent loop for tool-call training
│   └── peft_extender.py     # PEFT parameter handling for LoRA
└── profiling/
    └── memory_estimator.py  # GPU memory estimation for training

To see current algorithms and backends, check AlgorithmRegistry usage in src/training_hub/algorithms/*.py.

Architecture Pattern

The codebase follows a Strategy + Registry pattern:

  1. Algorithm (abstract base class): Defines train(), get_required_params(), get_optional_params()
  2. Backend (abstract base class): Defines execute_training(params) - actual training implementation
  3. AlgorithmRegistry: Maps algorithm names to classes, and backends to algorithms
  4. Convenience functions: Top-level functions like sft(), osft() wrap the registry

See src/training_hub/algorithms/__init__.py for the base class definitions.

Adding a New Algorithm

  1. Create algorithm class inheriting from Algorithm in src/training_hub/algorithms/
  2. Create backend class inheriting from Backend
  3. Register both with AlgorithmRegistry: 
    AlgorithmRegistry.register_algorithm('my_algo', MyAlgorithm)
AlgorithmRegistry.register_backend('my_algo', 'my-backend', MyBackend)
  4. Add convenience function wrapper
  5. Export in __init__.py
  6. Add documentation in docs/algorithms/ and docs/api/

Follow existing implementations in sft.py, osft.py, or lora.py as templates.

Adding a New Backend

Training Hub is designed to support multiple backends per algorithm:

  1. Create backend class inheriting from Backend
  2. Implement execute_training(algorithm_params: dict) -> Any
  3. Register with existing algorithm: AlgorithmRegistry.register_backend('sft', 'new-backend', NewBackend)
  4. Users can then select your backend via the backend parameter: sft(..., backend='new-backend')

Naming Conventions

  • Files: snake_case.py
  • Classes: PascalCase (e.g., SFTAlgorithm, MiniTrainerOSFTBackend)
  • Functions: snake_case (e.g., sft(), osft(), lora_sft(), lora_grpo(), grpo())
  • Constants: UPPER_SNAKE_CASE (e.g., FLOAT32_BYTES_N)
  • Backend names: kebab-case strings (e.g., "instructlab-training", "mini-trainer", "verl", "art")
  • Algorithm names: lowercase (e.g., "sft", "osft", "lora_sft", "lora_grpo", "grpo")

Code Style

  • Follow Google Python Style Guide
  • Type hints throughout (Python 3.11+ style: list[str] not List[str])
  • Use Optional[T] for optional parameters with None default
  • Docstrings use Google format with Args/Returns sections
  • Training parameters use keyword-only syntax with defaults

Data Formats

Training data is expected in JSONL format. See docs/api/data-formats.md for full documentation.

Common formats:

  • Messages format: {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
  • Pretraining: {"document": "Raw text content..."}
  • Alpaca (LoRA): {"instruction": "...", "input": "...", "output": "..."}

Parameter Translation

Different backends use different parameter names. Translation happens in each backend's execute_training() method.

To understand parameter mappings, read the relevant backend class:

  • SFT: InstructLabTrainingSFTBackend in src/training_hub/algorithms/sft.py
  • OSFT: MiniTrainerOSFTBackend in src/training_hub/algorithms/osft.py
  • LoRA: UnslothLoRABackend in src/training_hub/algorithms/lora.py

Each backend class has a renames dict or inline translation showing the mapping.

Torchrun Integration

Multi-GPU/multi-node training uses torchrun. See utils.get_torchrun_params() in src/training_hub/utils.py for:

  • Precedence handling: args dict > environment variables > defaults
  • Mutual exclusivity between master_addr and rdzv_endpoint
  • Validation of nproc_per_node values

Memory Estimation

The profiling/memory_estimator.py module provides VRAM estimation. See the module docstrings for usage:

from training_hub import estimate

low, mid, high = estimate(
    training_method="osft",  # Check module for supported methods
    model_path="...",
    num_gpus=8,
    ...
)

Visualization

See src/training_hub/visualization.py for the plot_loss() function:

from training_hub import plot_loss

plot_loss("./checkpoints")  # Single run
plot_loss(["./run1", "./run2"], labels=["A", "B"], ema=True)  # Compare runs

Important Gotchas

Algorithm-specific constraints

Each algorithm has validation logic in its train() method. Read the method docstrings and validation code for current constraints:

  • OSFT: See OSFTAlgorithm.train() in src/training_hub/algorithms/osft.py
  • LoRA: See LoRASFTAlgorithm.train() in src/training_hub/algorithms/lora.py
  • SFT: See SFTAlgorithm.train() in src/training_hub/algorithms/sft.py

Installation

Install extras sequentially — [grpo] and [cuda] have conflicting transitive dependencies that the solver can resolve when installed in order:

pip install -e .[grpo,lora]           # GRPO backends (ART + verl)
pip install -e .[cuda] --no-build-isolation  # CUDA kernels (after grpo)

Testing

  • Manual testing with example scripts in examples/scripts/
  • Jupyter notebooks in examples/notebooks/ for interactive testing

Documentation

docs/
├── README.md            # Home page
├── _sidebar.md          # Navigation sidebar
├── algorithms/          # Algorithm overviews
├── api/                 # API reference (functions, classes, backends)
├── guides/              # How-to guides
└── examples/            # Examples overview
  • Uses Docsify
  • Use absolute paths for internal links: /api/functions/sft not ../functions/sft.md
  • See docs/DEVELOPING.md for documentation contribution guidelines

CI/CD

See .github/workflows/pypi.yaml for the current GitHub CI pipeline:

  • Build and package validation
  • Test PyPI publishing on main branch pushes
  • Production PyPI publishing on GitHub releases
  • Sigstore package signing

Version Management

Uses setuptools_scm for automatic versioning from git tags. See pyproject.toml [tool.setuptools_scm] section for configuration.

Examples

Example scripts are in examples/scripts/. All examples accept --help for argument documentation.

See examples/README.md for an overview of available examples.

Operational Practices for Development and Testing

These are hard-won lessons from development on this codebase. Follow them carefully.

Environment Management

  • Use uv for venv creation and package management when available. Prefer uv venv, uv pip install, and uv pip compile over python -m venv and pip install. uv is significantly faster and has better dependency resolution. Only fall back to basic python -m venv / pip if uv is not installed.
  • Track which venv you are using. This project may have multiple venvs (e.g., .venv, .venv-test) for different purposes. Before running anything, confirm you are in the correct one. If you create a new venv for a workstream, note it explicitly so future sessions know it exists and what it's for.
  • The main branch has correctly resolving dependency versions unless something upstream has changed. Do not preemptively downgrade packages to "fix" version conflicts. If something breaks after a new install, the problem is more likely a build artifact issue (see flash-attn below) than a fundamental version incompatibility.
  • Test install order in a fresh venv when adding or changing dependencies. Development iteration can mask dependency resolution issues — packages may already be installed from prior work, hiding the fact that a clean pip install would fail. Always verify the install path documented in README/CLAUDE.md works from scratch before merging dependency changes.

Debugging Build and Runtime Failures

  • Read the actual error before acting. When a build or import fails, read the full traceback and identify the root cause. Do not guess based on the package name alone.
  • flash-attn must be rebuilt when torch changes. If you see flash-attn errors after a torch upgrade, the fix is uv pip install flash-attn --no-build-isolation --force-reinstall (rebuilding against the new torch), NOT downgrading torch. The versions are not incompatible — the compiled extension just needs to match the torch it was built with. Run uv cache clean flash-attn first if needed.
  • Do not downgrade packages as a first resort. If something worked before and breaks now, investigate what changed rather than assuming version incompatibility. Common actual causes: stale build cache, wrong venv, missing rebuild of compiled extensions, or something else already running on the GPU.

GPU and Process Management

  • Check what is already running on GPUs before launching work AND after any failure. Run nvidia-smi --query-compute-apps=pid,used_memory,name --format=csv,noheader first. If GPUs are occupied, identify whether the processes are active work or leftover dead processes before deciding what to do. This applies to every GPU error — not just explicit OOM. NCCL errors, vLLM engine core crashes, pure virtual method called aborts, and FSDP init failures can all be caused by insufficient memory from stale processes, even when the error message doesn't mention OOM.
  • ray stop is unreliable for cleanup. It often fails to kill worker subprocesses (vLLM workers, FSDP workers). After calling ray stop, always verify with nvidia-smi that GPU memory is actually freed. If processes remain, kill them directly: nvidia-smi --query-compute-apps=pid --format=csv,noheader | sort -u | xargs -r kill -9.
  • Never run broad kill commands like pkill -f vllm or kill -9 on process patterns without first checking what matches. Other users or workstreams may have active processes. Use targeted kills on specific PIDs after confirming they are yours and no longer needed.
  • When you see any GPU error, check GPU state first. An OOM, NCCL error, or engine crash on a training run may not be the run's fault — another process (or a zombie from a prior run) may already be consuming GPU memory. Check nvidia-smi before changing configs, downgrading packages, or reducing batch sizes.

Running and Monitoring Training

  • Run long-running training in the background so the user can provide input and you can do other work. Use run_in_background=true on Bash commands for training launches.
  • Always do a short initial check (30-60 seconds) to verify a run has started successfully before switching to a longer monitoring cadence. Check for: process is alive, GPU utilization is nonzero, no immediate crash in logs. Do not sleep for 5-10 minutes only to discover the run failed on startup.
  • After confirming startup, check periodically but not excessively. Read the last few lines of the log file rather than sleeping blindly. If the run produces metrics to a file (e.g., training_metrics.jsonl), read that for progress.

Dependency Security

  • Check for known supply chain compromises when relaxing dependency version ranges. litellm 1.82.7-1.82.8 were compromised by the TeamPCP attack (March 24, 2026) — they harvest credentials at import time. The upper cap <1.82.7 exists specifically for this. When widening version ranges, search for CVEs and supply chain advisories in the expanded range before merging.
  • When a dependency is transitive (not imported directly), match the floor to the library that imports it. Training Hub doesn't import litellm — openpipe-art does. The floor should match ART's declared minimum (>=1.71.1), not an arbitrary recent version.
  • Test dependency floors against RHOAI AIPCC indexes. RHOAI ships pinned package versions (e.g., litellm 1.74.15/1.75.8, numba 0.61.2). If training_hub's floor is higher than what RHOAI ships, pip resolution fails at install time. When setting minimum versions, check what the target deployment platform actually provides.

Kubeflow Integration

  • New algorithms must emit max_steps in training_metrics.jsonl. Without it, the Kubeflow SDK reports progress as "unknown" (None) because it calculates progressPercentage from max_steps. For GRPO, set max_steps to num_iterations in both rollout and train phase entries.

Avoiding Regressions

  • Track speed and quality baselines. Before making changes, note the current performance (e.g., reward values, training time per step, memory usage). After changes, verify these haven't degraded. If a "fix" makes things slower or reduces reward, the fix introduced a regression.
  • Build quick test harnesses for bug fixes and new features. Write a small script that exercises the specific broken or new path, run it, and confirm the fix. Also run (or at minimum read) any existing tests to make sure prior functionality isn't broken.
  • Save test results and note what was tested. When a test passes, record: what was run, what config, what the output was, what commit it was on. This creates a reference point for future work. Use memory files or commit messages for this, not just conversation context.

Validating Before Shipping

These are process lessons — not what broke, but what could have been done better to catch problems earlier.

  • Reproduce the bug before writing the fix. When a bug is reported, the first step is to build a minimal reproduction on the current code — not to start coding the fix. If you can't reproduce the bug, you can't verify the fix. If the "fix" passes tests that also pass without the fix, the tests prove nothing. Use fresh output directories (not leftover state from prior runs), match the reporter's environment as closely as possible (same model, same ART/vLLM versions), and confirm you see the same error. Only then write the fix, and verify the reproduction now passes.
  • Do not create PRs before functionally testing the change. Code review and syntax checks are not sufficient. If a fix touches a runtime code path (especially one involving subprocesses, GPU operations, or third-party libraries), run it end-to-end before opening a PR. A fix that passes python -c "import ast; ast.parse(...)" but crashes at runtime wastes reviewer time and erodes trust. Set up a test environment first, then write the fix, then test, then PR.
  • Read how the downstream library actually consumes your inputs. When passing config to a third-party library (ART, vLLM, PEFT), read the library's source to understand which config dict it reads from and how. Do not assume two config sections (init_args vs engine_args) are merged. Do not assume a parameter set in one place is forwarded to another. grep for the parameter name in the library's installed source (site-packages/art/, etc.) to trace how it flows.
  • When a subprocess fails opaquely, find the real error before trying fixes. RuntimeError: Engine core initialization failed. Failed core proc(s): {} tells you nothing. Do not start upgrading/downgrading packages or changing configs until you've found the actual exception. For vLLM, grep full output for EngineCore_DP0.*ERROR. For ART subprocess errors, read the .training_error file. Time spent on wrong fixes compounds — each attempt changes the environment and may introduce new problems.
  • Test with the actual return format, not assumed keys. Before writing test assertions about a function's return value, call the function once and inspect what it actually returns. Writing assertions like result["algorithm"] based on reading an intermediate internal dict (not the return value) wastes a test cycle.
  • Test serialization of third-party objects before saving them. When saving configs or state dicts from libraries like PEFT, verify the object is JSON/safetensors-serializable before building a pipeline around it. peft_config.to_dict() returning a set for target_modules is not obvious from the API — only a quick json.dumps(config.to_dict()) test reveals it.

Working With This Codebase Specifically

  • lora_grpo has two backends (verl and art) and three data modes (tool_call, generic, custom). Changes to shared code paths (the LoRAGRPOAlgorithm.train() method, the convenience function, parameter forwarding) must be verified against both backends. A parameter added to the API must be forwarded in optional_params AND get_optional_params().
  • verl backend launches training via torchrun subprocess. Parameters are passed as Hydra-style CLI overrides. If you add a new parameter, it must be appended to the cmd list in VeRLLoRAGRPOBackend.execute_training().
  • Batch divisibility matters for verl. prompt_batch_size * group_size must be divisible by n_gpus * nnodes * micro_batch_size. ppo_mini_batch_size must be divisible by n_gpus * nnodes. Validation exists but verify it covers new configurations.
  • ART backend runs in-process. It uses asyncio.run() and LocalBackend(in_process=True). Errors propagate directly. The os._exit() issue in shutdown was fixed — do not re-introduce it, as it kills multi-iteration training silently.
  • ART subprocess requires os._exit() to prevent vLLM 0.21+ hang. vLLM V1's EngineCore and APIServer spawn non-daemon threads that prevent asyncio.run() from completing its shutdown. Without os._exit(), proc.join() in the parent hangs indefinitely after training completes. The fix uses os._exit(0) on success (in _run_training's finally block) and os._exit(1) on failure (in _subprocess_entry's except block), with an exit_code variable to track which path was taken. Results are saved to disk before exit, so the parent reads them via training_results.json/.training_error files after proc.join() returns.
  • ART post-training cleanup errors can mask successful runs. ART's background health-check task throws ConnectionRefusedError after vLLM shuts down at end of training. If not handled, this writes to error_path and causes Kubeflow to restart the pod even though training succeeded. Check if results_path exists before treating a post-training exception as a failure — if training completed, log the error as a warning instead.
  • ART backend runs in a spawned subprocess. execute_training() uses mp.get_context("spawn"). Functions passed as parameters (e.g., reward_fn, rollout_fn, iteration_callback) must be picklable — they must be defined at module level, not as closures or lambdas. Local functions defined inside test methods or if __name__ == "__main__" blocks will fail with AttributeError: Can't get attribute .... The same applies to python -c scripts: define functions in importable files instead.
  • ART has two sets of config: init_args (Unsloth) and engine_args (vLLM). Parameters that affect vLLM (like gpu_memory_utilization) must be set in engine_args, not just init_args. ART reads engine_args directly when constructing AsyncEngineArgs for vLLM. If a parameter is missing from engine_args, vLLM uses its default (e.g., gpu_memory_utilization=0.9), which will OOM when Unsloth already holds model weights on the same GPU.
  • vLLM V1 is the only engine in vLLM 0.15+. The VLLM_USE_V1=0 env var was removed in vLLM ~0.12. Do not attempt to force the V0 engine — it does not exist. vLLM V1 eagerly validates LoRA adapter paths on add_lora; if a checkpoint doesn't exist yet, create a seed adapter (see _create_seed_lora_checkpoint).
  • vLLM engine core errors are hidden in subprocess logs. When AsyncLLM.from_engine_args fails with RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}, the actual error is in the EngineCore subprocess stderr, NOT the parent process. To find it, grep for EngineCore_DP0.*ERROR in the full output. Common causes: flashinfer version mismatch, GPU memory exhaustion, incompatible torch version for compiled extensions.

Past Incidents (reference for future debugging)

These are real issues encountered during development. They illustrate why the rules above exist.

  • flash-attn "wrong torch" failure: After torch was upgraded, flash-attn import failed. The attempted fix was downgrading torch, which would have broken other things. Actual fix: uv cache clean flash-attn && uv pip install flash-attn --no-build-isolation --force-reinstall. The extension just needed rebuilding.
  • ART multi-iteration silent failure: os._exit(0) in _shutdown_art_backend was killing the process after iteration 1. A NameError on art_project (removed from method signature thinking closure would capture it — closures don't work across method boundaries) was the root cause, but os._exit masked the error. Multi-iteration training had worked before; the question was "what changed?" not "is this possible?"
  • Eval regression from history truncation: A change to truncate observation history (h_obs[:80]) during an eval refactor caused scores to drop from ~43% to ~16%. The truncation was introduced incidentally during a different change and wasn't caught until explicit comparison with prior results.
  • Broad ray stop killed other users: Running ray stop to clean up after training killed another user's active Ray cluster on the shared machine. Should have checked ps aux first and killed specific PIDs.
  • OOM blamed on new workload: Training OOM'd and the response was to reduce batch size. Actual cause: stale GPU processes from a prior run were still consuming memory. nvidia-smi would have shown this immediately.
  • NCCL error misdiagnosed as driver incompatibility: FSDP init failed with ncclUnhandledCudaError: Cuda failure 'out of memory'. This was misread as a CUDA driver/toolkit version mismatch, leading to attempted torch downgrade from cu128 to cu126. Actual cause: zombie ray/vLLM workers from prior failed runs were holding 79GB across all GPUs. ray stop had been called between runs but didn't kill the worker subprocesses. Fix: nvidia-smi --query-compute-apps=pid --format=csv,noheader | sort -u | xargs -r kill -9. Lesson: any GPU error — not just OOM — should start with checking what's on the GPUs. NCCL errors are often memory errors in disguise.
  • nnodes silently dropped: When hiding ART-internal params from the public API, nnodes was accidentally removed from the optional_params dict. Multi-node training silently fell back to 1 node. Caught by CodeRabbit review. Lesson: when adding parameters to the convenience function, they must also appear in optional_params in train() AND in get_optional_params().
  • Install order dependency: [grpo] and [cuda] extras must be installed sequentially. In development this worked because packages accumulated over time; in a fresh venv the solver couldn't resolve both simultaneously. Only caught when testing in a clean environment.
  • VLLM_USE_V1=0 became a no-op: The env var was set in _subprocess_entry to force vLLM's legacy V0 engine. vLLM ~0.12 removed the env var, and ART 0.5.17 pins vLLM 0.17.0 which only has V1. vLLM V1 eagerly validates LoRA adapter paths on add_lora, but ART calls add_lora before training creates the first checkpoint — causing FileNotFoundError on adapter_config.json. Lesson: env var workarounds for third-party behavior silently break when the dependency upgrades. Prefer structural fixes (creating the expected files) over environment hacks.
  • Seed checkpoint placed before model.register() was wiped by ART: The initial fix created a seed LoRA checkpoint at the expected path before calling model.register(backend). This worked locally because ART's save_model happened to produce valid adapter files for Qwen2.5-7B — so the seed was irrelevant and the test passed for the wrong reason. On RHOAI with Qwen3-4B, ART's registration recreated the project directory structure (deleting the seed), and then save_model did not produce a valid adapter_config.json. The fix had to be moved to a monkey-patch on convert_checkpoint_if_needed, which runs AFTER ART's directory setup + save_model but BEFORE vLLM's add_lora. Lesson: when a test passes, verify it's passing because of your fix, not because the bug doesn't reproduce in your environment. Reproduce the bug first, then fix it.
  • gpu_memory_utilization not forwarded to vLLM: The parameter was passed in init_args (Unsloth initialization) but not in engine_args (vLLM engine args). ART reads engine_args when constructing AsyncEngineArgs, so vLLM defaulted to 0.9 and OOM'd because Unsloth already held ~14GB of model weights on the same GPU. Worse, engine_args was conditionally omitted entirely (**({"engine_args": ...} if engine_kwargs else {})) — when max_lora_rank wasn't set, no engine args were forwarded at all. Fix: always pass engine_args with at least gpu_memory_utilization. Lesson: read how the downstream library (ART) actually consumes config — don't assume init_args and engine_args are merged.
  • PEFT target_modules returned as a set: peft_config.to_dict() returns target_modules as a Python set, which json.dump cannot serialize. This crashed the seed checkpoint creation with TypeError: Object of type set is not JSON serializable. Fix: convert to sorted list before serializing. Lesson: always test serialization of configs from third-party libraries — they may use non-JSON-safe types internally.
  • flashinfer-cubin version mismatch: vLLM 0.17.0 installed flashinfer-python==0.6.4 but flashinfer-cubin==0.6.8.post1 was already present from a prior install. The version mismatch caused the vLLM engine core subprocess to crash silently with RuntimeError: flashinfer-cubin version (0.6.8.post1) does not match flashinfer version (0.6.4). The parent process only showed Engine core initialization failed. Failed core proc(s): {} with no root cause. Fix: uv pip install flashinfer-cubin==0.6.4. Lesson: when vLLM engine core fails silently, grep for EngineCore_DP0.*ERROR to find the real error in subprocess logs.
  • vLLM engine core error invisible in spawned subprocess: When ART runs inside Training Hub's mp.spawn subprocess, vLLM launches its engine core as a sub-subprocess. Errors in that sub-subprocess are printed to stderr but not captured by the parent. The parent only sees RuntimeError: Engine core initialization failed with an empty proc set. Spent significant time trying to install compatible versions and upgrade torch before discovering the actual error (flashinfer mismatch, then GPU memory). Lesson: always capture and grep the full stderr, including lines prefixed with (EngineCore_DP0 pid=...), before trying to fix the problem.
  • ART cleanup ConnectionRefusedError restarted Kubeflow pods after successful training: ART's background health-check task tried to reach vLLM after it had already shut down, throwing ConnectionRefusedError. The exception handler wrote to error_path unconditionally, which Kubeflow interpreted as a training failure and restarted the pod. Fix: check if results_path exists (indicating training completed) before writing to error_path; if training succeeded, log the cleanup error as a warning. Lesson: post-training cleanup errors should not overwrite successful results — always check completion state before marking a run as failed.
  • litellm supply chain attack (TeamPCP, March 2026): Attackers compromised litellm's CI/CD pipeline and published malicious versions 1.82.7 and 1.82.8 that harvested credentials at import time. Upper cap <1.82.7 in pyproject.toml prevents accidental upgrade. Lesson: when relaxing dependency version ranges, always check for known supply chain compromises and CVEs in the expanded range.
  • vLLM 0.21+ subprocess hang after successful training: With vLLM 0.21+, the lora_grpo training subprocess hung indefinitely after training completed. The V1 engine's EngineCore and APIServer spawn non-daemon threads that prevent asyncio.run() from completing shutdown, blocking the parent at proc.join(). This was a progression from the ConnectionRefusedError fix (#78) — the failure mode changed from an exception to a silent hang. Fix: add os._exit(0) after results are saved and backend shutdown is attempted, with os._exit(1) on the error path so proc.exitcode correctly reflects failure. Lesson: vLLM's shutdown behavior changes across versions — when upgrading vLLM, verify that training subprocesses still exit cleanly after completion.

Testing the ART Backend

There are no automated tests checked into the repo yet. When testing ART GRPO locally:

  • Use generic data mode (not tool-call) for quick validation. Create a JSONL file with {"question": "...", "ground_truth": "..."} entries and pass a reward_fn that checks if ground_truth appears in the model's response. This avoids downloading the Toucan dataset.
  • Make questions hard enough to produce reward variance. If the model gets 100% reward on trivial questions (e.g., "What is 2+2?"), ART skips the GRPO training step ("Skipping tuning as there is no suitable data") because all trajectories have the same reward and there's no advantage signal. Use questions the model sometimes gets wrong to exercise the actual training path.
  • Use minimal configs for speed: num_iterations=1, group_size=2, prompt_batch_size=5, lora_r=8, max_tokens=64, gpu_memory_utilization=0.3. A single iteration takes ~90s on H100.
  • reward_fn must be picklable. Define it at module level in an importable file. Do not define it as a closure, lambda, or inside python -c. The ART subprocess uses mp.get_context("spawn") which pickles all parameters.
  • Set CUDA_VISIBLE_DEVICES=0 to constrain to a single GPU (ART is single-GPU only). Without this, vLLM may try to use all visible GPUs.
  • Install vllm separately after [grpo]. openpipe-art declares vllm as a dependency but the prebuilt wheel may need manual installation to match CUDA/torch versions. After install, verify: python -c "from vllm import _C; print('OK')". If _C import fails, the wheel was built for a different torch version.
  • Verify flashinfer versions match: uv pip list | grep flashinfer — flashinfer-python and flashinfer-cubin must be the same version. Mismatches crash the vLLM engine core subprocess silently.

Disk and Cache Management

  • Set TMPDIR to a large filesystem (e.g., NVMe) before launching training. Flash-attn and flashinfer JIT compilation write hundreds of MB of temp files to /tmp. If /tmp is on a small root partition, the build fills the disk silently and the compilation fails with no useful error. Also set RAY_TMPDIR for Ray socket paths (AF_UNIX paths have a 107-byte limit — deep NVMe paths can exceed this, use a short path like /mnt/nvme0n1/tmp).
  • Clear flashinfer JIT cache when changing CUDA toolkit versions. The cache at ~/.cache/flashinfer/ contains compiled kernels for a specific CUDA version. After upgrading nvcc, delete the cache: rm -rf ~/.cache/flashinfer/.
  • Clear uv and pip caches periodically. They can grow to 60-100GB on root. uv cache clean and rm -rf ~/.cache/pip/.
  • vLLM compile cache at ~/.cache/vllm/torch_compile_cache/ can also grow large. Safe to delete between runs.

Training Larger Models with verl (8B+)

These lessons apply to models larger than Qwen3-4B on 80GB H100s with co-located FSDP+vLLM.

  • Use load_format=dummy instead of safetensors for the verl rollout config. With safetensors, vLLM loads the full model from disk during init on top of FSDP's copy, doubling GPU memory. With dummy, vLLM allocates empty tensors for profiling and gets real weights via the checkpoint engine's weight sync. This is verl's own default — we were overriding it unnecessarily.
  • Increase update_weights_bucket_megabytes to 3072. The default 2048MB bucket can't fit the embedding weight for models with large vocabularies (e.g., Qwen3's 152K vocab × 4096 hidden × 4 bytes = 2.36GB in float32). Without this, the FSDP→vLLM weight sync fails.
  • load_format=safetensors is needed for LoRA on very large models (9B+). With dummy format, the first weight sync calls FSDP.summon_full_params which reconstructs the full unsharded model on each GPU — OOM for 9B+. With safetensors, vLLM preloads base weights from disk, so subsequent syncs only transfer LoRA params (base_sync_done=True). Use with layered_summon=True for efficient LoRA-only syncs.
  • Full fine-tuning works by setting lora_r=0. verl treats lora_rank=0 as "no LoRA, train all parameters." No other changes needed — the same FSDP+vLLM pipeline handles full fine-tuning. Full FT is faster per step (no LoRA adapter management overhead) but uses more memory and may overfit earlier.
  • verl's FSDP→vLLM weight sync is the memory bottleneck. During update_weights, FSDP gathers the full model in float32 onto each GPU to sync to vLLM. For 9B models in float32 this is ~36GB per GPU on top of FSDP's shards. This is why Qwen3-8B (8.2B params, 152K vocab) barely fits but Qwen3.5-9B (9.5B params, 248K vocab) doesn't without safetensors + layered_summon.

OSFT (mini_trainer backend) Specific

  • AdamW leaks into the frozen subspace without post-step re-projection. AdamW's element-wise moment rescaling (m̂_t / (√v̂_t + ε)) destroys the orthogonal structure of projected gradients, causing parameter updates to drift into the frozen subspace over time. The fix (mini_trainer PR #91) adds project_parameters() after each optimizer step. If you see gradual quality degradation during OSFT training with AdamW, check whether parameter re-projection is active.
  • OSFT V-projection uses factored form (not Gram matrix). The original dV -= dV @ (V_high^T @ V_high) was replaced with dV -= (dV @ V_high^T) @ V_high (mini_trainer PR #74). Under FSDP2, this replaces an (M, M) all-reduce with a (k_high, M) all-gather — up to 25% speedup on 8B models. Optional caching of the all-gathered V_high is available via OSFT_CACHE_V=1 (off by default).
  • OSFT supports on-demand checkpointing and auto-resume. With on_demand_checkpointing=True, the training loop responds to termination signals (SIGTERM, SIGUSR1/2, etc.) by saving full state via torch.distributed.checkpoint. On restart, if no explicit resume_from_full_state_checkpoint is provided, it auto-detects the latest valid checkpoint from {output_dir}/full_state_checkpoints/step_*/training_state.pt.
  • The checkpoint trigger file is now product-agnostic. Changed from mini_trainer_checkpoint_trigger to checkpoint_requested. Customizable via CHECKPOINT_TRIGGER_FILENAME env var.

Qwen3.5 (GatedDeltaNet) Specific

Qwen3.5 uses a hybrid architecture: 75% GatedDeltaNet linear attention layers + 25% full attention. This requires specific setup:

  • CUDA toolkit >= 12.5 required. flashinfer's GDN kernels use PTX intrinsics (tensormap_replace_global_dim) added in CUDA 12.5. CUDA 12.4 fails at JIT compile time. Install the toolkit only — no driver upgrade needed: sudo dnf install -y cuda-toolkit-12-8.
  • Install flash-linear-attention and tilelang. Without FLA's fused Triton kernels, GDN layers decompose into 15-20 separate GPU ops per step — 10-50x slower. tilelang is required on Hopper GPUs with Triton >= 3.4.0 to avoid incorrect gradients. Set use_fused_kernels=True in the verl model config.
  • vLLM >= 0.19.0 required. Older versions don't support Qwen3_5ForConditionalGeneration.
  • Pass language_model_only=True. Qwen3.5-9B is text-only but registers as a multimodal architecture. Without this flag, vLLM loads the full VL model with vision encoder. Install qwen-vl-utils as well (imported during model init even in text-only mode).
  • Do NOT use tensor_parallel_size > 1. Qwen3.5 has known TP issues with vLLM 0.19 (NCCL hangs during inference).
  • Set VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200. Default is 300s. GDN inference on long prompts (20K+ tokens) with TP=1 can exceed 5 minutes, causing TimeoutError: RPC call to sample_tokens timed out.
  • Qwen3.5 chat template requires dict arguments in tool_calls. Unlike Qwen3, the Qwen3.5 Jinja template calls .items() on tool call arguments. If arguments is a JSON string instead of a dict, tokenization fails with "Can only get item pairs from a mapping". The data prep code must json.loads() string arguments before passing them to the tokenizer.
  • transformers >= 4.58 required for the qwen3_5 model type. Also upgrade huggingface_hub >= 1.0.
  • Force-reinstall nvidia-nccl-cu12 after installing vLLM 0.19. vLLM may leave a stale cu13 NCCL .so file even though pip shows cu12. Verify with python -c "import ctypes; lib=ctypes.CDLL('path/to/libnccl.so.2'); v=ctypes.c_int(); lib.ncclGetVersion(ctypes.byref(v)); print(v.value)".

Past Incidents (continued)

  • Root filesystem filled by flash-attn JIT compilation. CUDA compilation of flash-attn wrote hundreds of 125-140MB temp files to /tmp on a 100GB root partition. The build failed silently with "Error compiling objects for extension" and no disk space error. Fix: set TMPDIR=/mnt/nvme0n1/tmp before building. Lesson: always set TMPDIR to a large filesystem for CUDA compilation.
  • Stale libnccl.so version after package reinstall. pip list showed nvidia-nccl-cu12==2.27.5 but the actual .so file was version 2.28.9 (from a prior cu13 install). Caused NCCL errors at runtime. Fix: pip install nvidia-nccl-cu12==2.27.5 --force-reinstall. Lesson: pip metadata and actual library files can diverge — verify with ctypes or ldd when debugging NCCL issues.
  • vLLM sample_tokens timeout on GDN models. Qwen3.5-9B with 29K+ token prompts exceeded the 300s default VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS. The vLLM worker appeared to hang and the engine reported EngineDeadError. Fix: increase timeout to 1200s. Lesson: GDN linear attention is slower per token than standard attention — timeout values calibrated for transformer models may be too low.
  • FLA fused kernels crash without tilelang on Hopper. RuntimeError: Triton >= 3.4.0 on Hopper GPUs produces incorrect results for gated chunk_bwd_dqkwg (see #640). Please install tilelang. The error message is clear and actionable — just pip install tilelang.
  • 21 min/step without FLA vs 3.3 min/step with FLA. The [transformers] The fast path is not available because one of the required library is not installed warning is easy to miss. Without FLA, GDN layers use a decomposed fallback that is 10-50x slower. Always install flash-linear-attention when training GDN/Qwen3.5 models.

verl Backend Reference Configuration

These are known-good configuration values from validated training runs. Use as a starting point when debugging or configuring new runs.

# Validated on 1x H100, Qwen2.5-7B-Instruct, generic Q&A data (ART backend)
lora_r=8, lora_alpha=4
gpu_memory_utilization=0.3
num_iterations=1, group_size=2, prompt_batch_size=5
max_tokens=64, concurrency=4
Result: successful completion, 10 rollouts, reward=1.0 (trivial data)
Training time: ~90s total (including model load + vLLM warmup)
Environment: openpipe-art==0.5.17, vllm==0.17.0, torch==2.10.0

# Validated on 4x H100, Qwen3-4B, OpenShift tool-call data (1886 traces)
lora_r=128, lora_alpha=256
gpu_memory_utilization=0.3
micro_batch_size=2  (needed for large vocab models — seq_len * 152K vocab causes OOM on logits)
max_prompt_length=32768
prompt_batch_size=50, group_size=8
learning_rate=5e-6
Step 1 reward: 0.154 (baseline reference for OpenShift v5 data)
Training time: ~7 min/step

# Validated on 4x H100, Qwen3-4B, AIME math data (custom reward)
lora_r=16, lora_alpha=8
gpu_memory_utilization=0.3
num_iterations=2, group_size=8, prompt_batch_size=50
learning_rate=1e-5
Result: successful completion, both iterations

# Validated on 8x H100, Qwen3-8B, OpenShift v4 data (LoRA)
lora_r=128, lora_alpha=256
gpu_memory_utilization=0.5, load_format=dummy
update_weights_bucket_megabytes=3072
prompt_batch_size=48, group_size=8
learning_rate=5e-6, max_prompt_length=32768
Final reward: 0.80 avg, 0.89 peak (8 epochs, 41 hours)

# Validated on 8x H100, Qwen3-8B, OpenShift v4 data (full fine-tuning)
lora_r=0 (full fine-tuning)
gpu_memory_utilization=0.5, load_format=dummy
Peak reward: 0.804 at epoch 5, then overfitting (8 epochs, 26.5 hours)

# Validated on 8x H100, Qwen3.5-9B, OpenShift v4 data (LoRA)
lora_r=128, lora_alpha=256
gpu_memory_utilization=0.3, load_format=safetensors, layered_summon=True
language_model_only=True, use_fused_kernels=True
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200
Requires: vllm==0.19.0, flash-linear-attention, tilelang, CUDA toolkit 12.5+
Step time: ~3.3 min/step, reward 0.85+ by epoch 2