Enhance dspy.Refine with support for Hard/Soft Constraint Handling

[gilad12-coder]

Description:

This PR introduces a significantly enhanced dspy.Refine module, replacing the previous implementation with a more robust, flexible, and controllable system for iteratively improving module predictions.

Motivation:

The previous Refine module primarily focused on retrying a module with varying temperatures and generating LLM-based feedback when a reward_fn score fell below a threshold. While useful, it lacked:

1. Support for Hard Constraints: No built-in mechanism to enforce programmatic validation rules (e.g., output format, specific content requirements) beyond the soft constraints of a reward function.
2. Granular Feedback: Feedback was solely LLM-generated based on reward scores, lacking the ability to provide direct, programmatic feedback for specific validation failures.
3. Configurability: Limited options to control behavior beyond N attempts and the reward threshold.

Changes Introduced:

The new Refine module addresses these limitations by introducing several key features:

1. Validators (Hard Constraints):

   • Accepts an optional validators argument: a list of functions (Prediction) -> (bool, str).
   • Each validator checks the prediction. If it returns False, the accompanying string provides specific feedback on the failure.
   • Failed validations trigger retry attempts with the collected feedback messages incorporated into the prompt context (as the previous_attempts input field within a signature).

2. Distinct Handling of Constraints:

   • Hard Constraints (validators): Must pass for a prediction to be considered valid. Failure triggers retries with specific error messages.
   • Soft Constraints (reward_fn, threshold): Evaluated only if validators pass (or if no validators are provided). Falling below the threshold can trigger retries aimed at improving quality, potentially using LLM-generated feedback (OfferFeedback).

3. Improved Retry Logic & State Management:

   • Retries up to N times with varying temperatures.
   • Tracks the best_prediction based on a clear hierarchy:
     • Prefers predictions passing all validators.
     • Among those passing validators, prefers higher reward_fn scores (if applicable).
     • Among those failing validators, prefers predictions with fewer validation errors.
   • Uses an internal dspy.Predict or dspy.ChainOfThought (controlled by use_cot) to handle retries when validators are used, incorporating the previous_attempts log directly into the signature.

4. Enhanced Configuration & Control:

   • verbose: Enables detailed logging of the refinement process (validation checks, reward scores, feedback).
   • fail_on_invalid: If True, raises a ValueError if no prediction meets all constraints (validators and reward threshold) after N attempts. If False (default), returns the best prediction found according to the hierarchy above.
   • use_cot: Allows using Chain-of-Thought for the internal prediction steps during refinement when validator feedback is being processed.

5. Observability:

   • The returned Prediction object includes a Refine_metadata attribute containing details about the refinement process (iterations, success status, final reward, validation status, attempts log).

Breaking Changes:

• The constructor signature has changed significantly. Users migrating will need to update initialization calls.
• The core behavior is different due to the introduction of validators and the prioritized execution flow (validators first, then reward).
• The fail_count parameter is removed, replaced by fail_on_invalid.

[gilad12-coder]

Heads Up: Be aware that the LLM sometimes struggles to reliably format the complex advice dictionary requested by OfferFeedback signature (used in Refine). There's a try-except fallback to "N/A", but it means the generated feedback might not always be effective. See _get_feedback in Refine.

[zbambergerNLP]

Looks excellent! Though worth retaining the per-submodule approach of OfferFeedback rather than globalizing its scope to the entire program. Perhaps worth adding a TODO to add support for a global-level feedback signature, and allow users to select which they'd prefer. Also, worth allowing support for Boolean thresholds/rewards, and specifying clearly that rewards/verifier scores are bounded in the range [0, 1].

[zbambergerNLP]

We may want this to remain a dict[str, str] and maintain the functionality of advice -- notably the breakdown of feedback to specific sub-modules within the program. The existing description seems good for evoking feedback from the LLM, but we should ask for this type of feedback for every sub-module, and not the program as a whole.

[zbambergerNLP]

When calling a module that wraps the OfferFeedback signature, you'll likely need to format the dictionary into a single string, which will then in turn be formatted into the previous_attempts field in Refine.

[gilad12-coder]

Done

[zbambergerNLP]

We should still aim to offer feedback at the per-submodule level of the program. I would retain the existing functionality of OfferFeedback wherever possible, perhaps adjusting the wording of descriptions slightly how the LM should provide feedback.

[gilad12-coder]

Done

[zbambergerNLP]

Nit: can likely fit on one line. Worth supporting booleans as well.

[gilad12-coder]

Done

[gilad12-coder]

Done

[chenmoneygithub]

This style breaks on mkdocs (our doc site), let's use the same style as:

dspy/dspy/streaming/streamify.py

Line 51 in 5cd355b

Example:

[gilad12-coder]

Done

[chenmoneygithub]

nit: just 4 indents after linebreak instead of vertical alignment.

[gilad12-coder]

Done

[AriMKatz]

@gilad12-coder is this gonna go into 3.0? Anything blocking it? thanks for your work