Deterministic policy-ranked tool fallback routing with auditable reason codes (Vibe Kanban)

[jbax1899]

Summary

This PR introduces deterministic, policy-driven fallback behavior when the selected response profile cannot execute requested search/tool behavior, and makes fallback outcomes auditable through stable execution reason codes.

What Changed

• Added an explicit fallback policy by profile-selection source in chat orchestration:
  • planner source can reroute to a tool-capable fallback profile.
  • request and default sources do not reroute and instead skip search deterministically.
• Added a deterministic fallback ranking strategy for tool-capable profiles:
  • prefer same provider,
  • then shared tier bindings,
  • then lower latency class,
  • then lower cost class,
  • then profile ID lexical tie-break.
• Replaced implicit first-match fallback behavior with policy-ranked candidate selection.
• Extended execution reason codes in contracts and schema with stable fallback outcomes:
  • search_rerouted_to_fallback_profile
  • search_reroute_not_permitted_by_selection_source
  • search_reroute_no_tool_capable_fallback_available
• Updated metadata normalization so executed tool events can preserve explicit policy reason codes (for reroute auditability).
• Updated and expanded tests across backend/contracts to cover:
  • planner-selected reroute,
  • planner-selected no-fallback skip,
  • request-selected skip-without-reroute,
  • executed tool reason-code validation and metadata preservation.

Why

The existing behavior depended on implicit first-match fallback, which could be surprising and harder to review in traces. This change makes fallback decisions intentional, deterministic, and explainable in both logs and execution metadata, while preserving fail-open behavior and keeping web/Discord behavior consistent through the shared backend orchestration path.

Important Implementation Details

• Fallback policy is centralized in chatOrchestrator and keyed by selection source (request | planner | default).
• Candidate ranking is deterministic and independent of catalog iteration order once the ranking tuple is applied.
• Upstream orchestrator tool context is preserved in chatService for skip decisions, and reroute reason codes are retained when runtime confirms tool execution.
• ResponseMetadata schema now allows reasonCode on executed events for policy-audit cases.

This PR was written using Vibe Kanban

Summary by CodeRabbit

• New Features

  • Policy-driven fallback routing for web search: ranks compatible alternatives (provider, tier overlap, latency, cost, tie-break) and reroutes only when permitted by selection-source policy; skipped searches are marked with policy reason codes.
  • Response metadata and schemas now preserve and accept explicit reason codes for executed tool events; new reason codes added for reroute outcomes.

• Tests

  • Added/updated tests covering reroute, skip, and no-fallback search scenarios and metadata behavior.

[coderabbitai]

Warning

Rate limit exceeded

@jbax1899 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 21 minutes and 0 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 21 minutes and 0 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 316f850d-d677-4448-b6db-f6cd67930884

📥 Commits

Reviewing files that changed from the base of the PR and between 3fbc79c and ba25db1.

📒 Files selected for processing (2)

• .github/workflows/ci.yml
• .github/workflows/codeql.yml

📝 Walkthrough

Walkthrough

Implements a policy-driven, deterministic ranking for search-capable fallback profiles, adds selection-source policies to control reroute vs skip, propagates policy-derived reason codes into tool execution metadata, and updates types/schemas and tests to accept these reason codes.

Changes

Cohort / File(s)	Summary
Search Fallback Orchestration packages/backend/src/services/chatOrchestrator.ts	Replaced prior deterministic fallback helper with policy-driven deterministic ranking (provider, tier-binding overlap, latency class, cost class, id tie-break). Added searchFallbackPolicyBySelectionSource, conditional reroute vs skip logic, policy-derived reasonCode assignment, and expanded fallback observability and logging with ranking step metadata and ranked candidate IDs.
Tool Execution Context & Metadata packages/backend/src/services/chatService.ts, packages/backend/src/services/openaiService.ts	Use upstreamToolExecution as the basis for effective tool execution context; when search intent exists, overlay status/reasonCode based on retrievalUsed. buildResponseMetadata now preserves explicit toolExecution.reasonCode for status: 'executed' events.
Contracts & Schemas packages/contracts/src/ethics-core/types.ts, packages/contracts/src/web/schemas.ts	Added three new ExecutionReasonCode literals (search_rerouted_to_fallback_profile, search_reroute_not_permitted_by_selection_source, search_reroute_no_tool_capable_fallback_available). Allowed reasonCode on executed ExecutionEvent entries in schema validation.
Tests packages/backend/test/chatOrchestrator.test.ts, packages/backend/test/openaiService.metadata.test.ts, packages/contracts/test/webSchemas.test.ts	Updated reroute tests to expect policy-derived messages and reason codes; split catalog mutation setup and added new test for skipping search when no tool-capable fallback exists; added metadata test ensuring executed tool events preserve reasonCode; adjusted schema test expectations.

Sequence Diagram

sequenceDiagram
    participant Orchestrator as Chat Orchestrator
    participant Ranker as Profile Ranker
    participant PolicyMap as SelectionSource Policy
    participant SearchExec as Search Executor
    participant Metadata as Metadata Builder

    Orchestrator->>Ranker: compute ranked search-capable fallbacks\n(exclude selected profile)
    Ranker-->>Orchestrator: rankedFallbackCandidates

    Orchestrator->>PolicyMap: query policy for selection source\n(allowReroute, rerouteCode, skipCode)
    PolicyMap-->>Orchestrator: policyDecision

    alt allowReroute == true
        Orchestrator->>SearchExec: execute search on top-ranked fallback
        SearchExec-->>Orchestrator: search results
        Orchestrator->>Metadata: set toolExecutionContext\n(status: executed, reasonCode: rerouteCode)
    else allowReroute == false
        Orchestrator->>Metadata: remove search from generation\nset toolExecutionContext\n(status: skipped, reasonCode: skipCode)
    end

    Metadata-->>Orchestrator: updated execution context

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• Unify execution timeline formatting across web and Discord #204: Modified execution timeline / ResponseMetadata surface and schema exports that interact with these new reasonCode and metadata changes.
• feat: planner-aware profile routing, execution timeline metadata, and Discord /chat model switching (with Ollama support) #200: Introduced execution-timeline and model profile/catalog plumbing that this change builds upon.
• Implement planner-only search degradation and user signals #201: Adjusted chatOrchestrator reroute/drop logic and execution metadata; closely related implementations and tests.

Suggested labels

enhancement

Poem

🐰 I hopped through catalogs, sniffed each profile's trail,
Ranked by provider, tier, latency, and tale,
Policies whispered reroute or graceful skip,
I left a tiny reasonCode on each tool's tip,
Logs sing my hops as I bound down the trail.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: deterministic, policy-ranked tool fallback routing with auditable reason codes, reflecting the core objectives of the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch vk/a9a7-deterministic-fa

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}