test(recorded): add rails library coverage (5/5)

[Pouyanpi]

Summary

Adds recorded coverage for library rails and their composition order.

Why

Built-in rails need replayable end-to-end coverage so future rail changes can be reviewed against stable behavior.

What Changed

• Adds library rail configs, cassettes, and tests for regex, injection, self-check, content safety, jailbreak, topic control, and composition.
• Adds library README guidance requiring recorded coverage for new rails.

Review Notes

This is the final stack PR and includes no additional runtime changes.

Stack Position

Part 5 of 5.

• Previous: test(recorded): add rails public API coverage (4/5) #1977
• Next: none, final stack PR

Stack Context

This stack decomposes recorded end-to-end replay coverage into reviewable slices. The PRs should be reviewed against their parent branch in the stack.

Please review each PR against its parent branch, not directly against the root base branch, except for part 1.

Order	PR	Branch	Base
1	#1974	stack/recorded-tests-01-harness	develop
2	#1975	stack/recorded-tests-02-deterministic-library-load	stack/recorded-tests-01-harness
3	#1976	stack/recorded-tests-03-clients	stack/recorded-tests-02-deterministic-library-load
4	#1977	stack/recorded-tests-04-public-api	stack/recorded-tests-03-clients
5	#1978	stack/recorded-tests-05-library-rails	stack/recorded-tests-04-public-api

Validation

poetry check --lock
poetry lock --no-update
poetry install --with dev
poetry run pytest tests/recorded --block-network -q
pre-commit hooks passed during commit creation

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This is the final PR in a 5-part stack adding recorded end-to-end test coverage for NeMo Guardrails library rails. It also ships a runtime fix to llmrails.py that corrects the streaming path's user-input extraction.

• Adds seven test modules (regex, injection, self-check, content safety, jailbreak, topic control, composition) with VCR cassettes and supporting configs that pin the public contract of every built-in rail.
• Fixes _get_latest_user_message in llmrails.py to return the message's .content string instead of the full dict, so the streaming output-rail path renders user_input correctly in prompt templates (the cassette for test_content_safety_output_blocks_fake_main_stream now shows user: hello instead of user: {'role': 'user', 'content': 'hello'}).
• Adds contribution guidance to nemoguardrails/library/README.md requiring recorded coverage for new rails.

Confidence Score: 5/5

Safe to merge — adds test infrastructure only and a well-evidenced single-function fix in the streaming path.

The runtime change is narrow: one inner function inside the streaming output-rail handler now returns a content string instead of a full message dict. The cassettes before and after both verify the correct prompt is sent to the provider. All 45 changed files are either new test fixtures, configs, or VCR cassettes; no existing logic outside the streaming path is affected.

No files require special attention. The cassettes, configs, and test helpers are internally consistent.

Important Files Changed

Filename	Overview
nemoguardrails/rails/llm/llmrails.py	Fixes _get_latest_user_message to return the message content string instead of the full dict; cassettes confirm the streaming prompt now renders correctly.
tests/recorded/rails/library/helpers.py	Provides check_rails, generate_with_fake_main, and stream_with_fake_main helpers; streaming helper intentionally passes no FakeLLMModel and relies on options={"rails": ["output"]} to prevent main-model calls.
tests/recorded/rails/library/test_composition.py	Composition-order tests covering regex→self-check→jailbreak→content safety→topic control ordering; cassettes confirm each rail short-circuits correctly before the next rail runs.
tests/recorded/rails/library/configs/full_stack/config.yml	Full-stack config now has consistent jailbreak-before-content-safety ordering matching full_stack_no_topic (previous ordering discrepancy resolved).
tests/recorded/rails/library/test_content_safety.py	Covers safe-pass, block, streaming block, generation block, and provider-error scenarios for NIM content safety; streaming cassette confirms the user_input fix.
tests/recorded/rails/library/test_injection.py	Tests injection detection (block and omit modes) plus exception propagation when enable_rails_exceptions=True; all regex-based, no cassettes needed.
tests/recorded/rails/library/configs.py	Centralises RailsConfigSource constants and the shared JAILBREAK_PROMPT test fixture.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant T as Test
    participant R as LLMRails
    participant Regex as regex check
    participant SC as self check (OpenAI)
    participant JB as jailbreak detect (NIM)
    participant CS as content safety (NIM)
    participant TC as topic safety (NIM)

    T->>R: "check_async(messages, rail_types=[INPUT])"
    R->>Regex: detect_regex_pattern
    Regex-->>R: passed
    R->>SC: self_check_input prompt
    SC-->>R: safe → passed
    R->>JB: nemoguard-jailbreak-detect
    JB-->>R: "jailbreak=true → BLOCKED (stops here)"
    note over CS,TC: not reached when jailbreak blocks

    T->>R: "stream_async(messages, generator, rails=[output])"
    note over R: generator provides main content
    R->>CS: content_safety_check_output (NIM)
    CS-->>R: unsafe → stream error chunk emitted

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant T as Test
    participant R as LLMRails
    participant Regex as regex check
    participant SC as self check (OpenAI)
    participant JB as jailbreak detect (NIM)
    participant CS as content safety (NIM)
    participant TC as topic safety (NIM)

    T->>R: "check_async(messages, rail_types=[INPUT])"
    R->>Regex: detect_regex_pattern
    Regex-->>R: passed
    R->>SC: self_check_input prompt
    SC-->>R: safe → passed
    R->>JB: nemoguard-jailbreak-detect
    JB-->>R: "jailbreak=true → BLOCKED (stops here)"
    note over CS,TC: not reached when jailbreak blocks

    T->>R: "stream_async(messages, generator, rails=[output])"
    note over R: generator provides main content
    R->>CS: content_safety_check_output (NIM)
    CS-->>R: unsafe → stream error chunk emitted

Loading

_{Reviews (10): Last reviewed commit: "test(recorded): fix self-check-facts fal..." | Re-trigger Greptile}

[tgasser-nv]

Only question is on the changes to llmrails.py which seem to be out-of-scope for an integration-testing PR. The description says there were no runtime changes

[tgasser-nv]

Why is this change to the core runtime in the PR? It seems like a bugfix and out-of-scope for adding rails library coverage. It contradicts the PR description of "no runtime changes"

[Pouyanpi]

the test surfaced a pre-existing bug. I've extracted it to #2081

this PR now targets #2081

[Pouyanpi]

see #1978 (comment)