Fix metadata processing in chat adapter #2040

gilad12-coder · 2025-01-11T15:53:08Z

This PR enhances the ChatAdapter implementation with two improvements:

1. Added Field Metadata Processing:

Introduced new metadata handling capabilities for numeric field constraints with PERMITTED_CONSTRAINTS
Added new utility functions to process field metadata:
- _format_constraint(): Formats numeric constraints (gt, lt, ge, le, multiple_of, allow_inf_nan) based on the constraints that can be passed in Pydantic's fields class (https://docs.pydantic.dev/latest/concepts/fields/#field-aliases)
- format_metadata_summary(): Creates human-readable metadata summaries for each field
- format_metadata_constraints(): Formats field constraints for prompt generation
Enhanced field descriptions in prompts to include metadata information (using the summary and the paragraphical format).
Added support for numeric field constraints in field metadata processing

example:
relevance_and_significance (float): The score of the argument based on its relevance and significance.
When evaluating relevance and significance, consider the following questions:

How significant is the central claim the argument is trying to establish?
How relevant is the claim to the topic?
The score should be a floating-point number between the specified lower and upper bounds. [Metadata: Ge(ge=0.0); Le(le=1.0)] (this is formatted using format_metadata_summary())
[[ ## relevance_and_significance ## ]]
{relevance_and_significance} # note: the value you produce must be a single float value that is greater than or equal to 0.0 and less than or equal to 1.0. (this is formatted using format_metadata_constraints())

2. Improved Code Documentation and Type Hints:

Added comprehensive docstring to all functions and classes
Added proper type hints throughout the file
Improved function signatures with explicit type annotations
Enhanced error messages for better debugging
Added descriptive comments for complex logic sections

… the prompt and added docstring throughtout the file

chenmoneygithub

thanks for the PR!

Both the metadata formatting and docstring improvements are pretty useful, but could you split them into 2 PRs? We also need to add the metadata formatting to the json adapter for consistency.

zbambergerNLP

Nice work here! I think some functions that are duplicated between the chat and json adapter should move to Base in order to reduce code duplication. Otherwise, most of my comments are formatting nits (mostly in favor of type annotations, which should really help developers that have built in IDE tools that use these).

zbambergerNLP · 2025-03-11T11:12:09Z

dspy/adapters/chat_adapter.py

-    def __call__(self, lm: LM, lm_kwargs: dict[str, Any], signature: Type[Signature], demos: list[dict[str, Any]], inputs: dict[str, Any]) -> list[dict[str, Any]]:
+    def __call__(self, lm: LM, lm_kwargs: dict[str, Any], signature: Type[Signature], demos: list[dict[str, Any]],
+                 inputs: dict[str, Any]) -> list[dict[str, Any]]:


nit: format as follows:

def __call__( self, lm: LM, lm_kwargs: Dict[str, Any], ... ) -> List[Dict[str, Any]]

zbambergerNLP · 2025-03-11T11:13:34Z

dspy/adapters/chat_adapter.py

+    def format(self, signature: Type[Signature], demos: list[dict[str, Any]], inputs: dict[str, Any]) -> list[
+        dict[str, Any]]:


Apply format I suggested in nit above.
Docstring I presume will go in a separate PR.

zbambergerNLP · 2025-03-11T11:15:10Z

dspy/adapters/chat_adapter.py

@@ -99,7 +85,7 @@ def format(self, signature: Type[Signature], demos: list[dict[str, Any]], inputs
        messages = try_expand_image_tags(messages)
        return messages

-    def parse(self, signature: Signature, completion: str, _parse_values: bool = True):
+    def parse(self, signature: Type[Signature], completion: str) -> dict[str, Any]:


nit: Dict instead of dict. Apply here and elsewhere.

zbambergerNLP · 2025-03-11T11:16:30Z

dspy/adapters/chat_adapter.py

+    def format_finetune_data(self, signature: Type[Signature], demos: list[dict[str, Any]], inputs: dict[str, Any],
+                             outputs: dict[str, Any]) -> dict[str, list[Any]]:


Revert to previous formatting of the function definition signature (multi-line and in the style I suggested in my nits above). Presumably docstring will go in separate PR. Note that we should use "Parameters" instead of "Args" (also in other function definitions where you'd added docstrings).

zbambergerNLP · 2025-03-11T11:19:21Z

dspy/adapters/chat_adapter.py

+    def format_turn(self, signature: Type[Signature], values: dict[str, Any], role: str, incomplete: bool = False,
+                    is_conversation_history: bool = False) -> dict[str, Any]:


See suggested format for function definitions in my first nit in this file.

zbambergerNLP · 2025-03-11T11:36:31Z

dspy/adapters/chat_adapter.py

    return "\n".join(parts).strip()


-def move_type_to_front(d: Union[Dict, List, Any]) -> Union[Dict, List, Any]:
-    """Moves the 'type' key to the front of the dictionary, recursively, for LLM readability/adherence."""
+def move_type_to_front(d):


Keep the type annotations here IMO. Makes the function easier to parse.

zbambergerNLP · 2025-03-11T11:37:13Z

dspy/adapters/chat_adapter.py

-def prepare_schema(type_: Type) -> Dict[str, Any]:
-    """Prepares a JSON schema for a given type."""
-    schema: Dict[str, Any] = pydantic.TypeAdapter(type_).json_schema()
+def prepare_schema(field_type):


Keep type annotations.

zbambergerNLP · 2025-03-11T11:37:37Z

dspy/adapters/chat_adapter.py

    parts = []
    parts.append("Your input fields are:\n" + enumerate_fields(signature.input_fields))
    parts.append("Your output fields are:\n" + enumerate_fields(signature.output_fields))
    parts.append("All interactions will be structured in the following way, with the appropriate values filled in.")

-    def field_metadata(field_name: str, field_info: FieldInfo) -> str:
-        """Creates a formatted representation of a field's information and metadata."""
+    def field_metadata(field_name, field_info):


Keep type annotations.

zbambergerNLP · 2025-03-11T11:39:16Z

dspy/adapters/json_adapter.py

+def _format_constraint(name: str, value: Union[str, float]) -> str:
+    constraints = {
+        'gt': f"greater than {value}",
+        'lt': f"less than {value}",
+        'ge': f"greater than or equal to {value}",
+        'le': f"less than or equal to {value}",
+        'multiple_of': f"a multiple of {value}",
+        'allow_inf_nan': "allows infinite and NaN values" if value else "no infinite or NaN values allowed"
+    }
+    return constraints.get(name, f"{name}={value}")


Worth moving these into the base class for adapter for the sake of inheritance.

zbambergerNLP · 2025-03-11T11:47:28Z

dspy/adapters/chat_adapter.py

+    """
+    Formats the constraints for a field.
+
+    Args:


nit: Replace with Parameters:.

chenmoneygithub

Thanks for the PR!

Two high level feedbacks:

We should move the constraint formatting logic into dspy/signatures/field.py since it is irrelevant to adapter types. We don't want to introduce new complexity to adapter, which is already unnecessarily complex now.
We are in the middle of an effort to standardize the style, and we use Google style guide as the guideline: https://google.github.io/styleguide/pyguide.html#383-functions-and-methods. Most of formatting in the PR doesn't follow the rules there.

chenmoneygithub · 2025-03-12T20:22:19Z

dspy/adapters/chat_adapter.py

+        lm_kwargs: Dict[str, Any],
+        signature: Type[Signature],
+        demos: List[Dict[str, Any]],
+        inputs: Dict[str, Any]


missing comma at the end, also don't use capital case Dict, using primitive types like dict is preferred: https://google.github.io/styleguide/pyguide.html#221-type-annotated-code

chenmoneygithub · 2025-03-12T20:22:40Z

dspy/adapters/chat_adapter.py

+        signature: Type[Signature],
+        demos: List[Dict[str, Any]],
+        inputs: Dict[str, Any]
+    ) -> List[


chenmoneygithub · 2025-03-12T20:23:03Z

dspy/adapters/chat_adapter.py

@@ -118,10 +140,15 @@ def format_finetune_data(self, signature: Type[Signature], demos: list[dict[str,
        assistant_message = self.format_turn(signature, outputs, role, incomplete)
        messages.append(assistant_message)

-        # Wrap the messages in a dictionary with a "messages" key
+        # Wrap the messages in a Dictionary with a "messages" key


no need to capitalize this

chenmoneygithub · 2025-03-12T20:23:35Z

dspy/adapters/chat_adapter.py

-    Args:
-      fields_with_values: A dictionary mapping information about a field to its corresponding
-                          value.
+    Parameters:


Revert this, Args is the standard: https://google.github.io/styleguide/pyguide.html#383-functions-and-methods

chenmoneygithub · 2025-03-12T20:25:00Z

dspy/adapters/chat_adapter.py

@@ -228,26 +267,96 @@ def type_info(v):
    return {"role": role, "content": joined_messages}


-def enumerate_fields(fields: dict) -> str:
+def _format_constraint(name: str, value: Union[str, float]) -> str:


We can move this function and other related ones into dspy/signatures/field and import in the adapter to keep adapter code simple.

gilad12-coder and others added 4 commits January 11, 2025 17:34

added the ability to process metadata feilds and add them directly to…

436d3c8

… the prompt and added docstring throughtout the file

quick fix to get_annotation_name

cc2e206

fixed some small formating issues

0a30136

Merge branch 'main' into fix-metadata-processing-in-ChatAdapter

b7c03cf

chenmoneygithub requested changes Mar 10, 2025

View reviewed changes

gilad12-coder added 2 commits March 10, 2025 20:02

adapted the chat adapter to add metadata processing

4001b85

added the metadata formatting to the json adapter

4ec5c74

zbambergerNLP reviewed Mar 11, 2025

View reviewed changes

addressed Zachs comments

4277af5

gilad12-coder requested a review from chenmoneygithub March 11, 2025 19:25

chenmoneygithub requested changes Mar 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix metadata processing in chat adapter #2040

Fix metadata processing in chat adapter #2040

gilad12-coder commented Jan 11, 2025

chenmoneygithub left a comment

zbambergerNLP left a comment

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

zbambergerNLP Mar 11, 2025

zbambergerNLP Mar 11, 2025

gilad12-coder Mar 11, 2025

chenmoneygithub left a comment

chenmoneygithub Mar 12, 2025

gilad12-coder Mar 12, 2025

chenmoneygithub Mar 12, 2025

gilad12-coder Mar 12, 2025

chenmoneygithub Mar 12, 2025

gilad12-coder Mar 12, 2025

chenmoneygithub Mar 12, 2025

gilad12-coder Mar 12, 2025

chenmoneygithub Mar 12, 2025

		def format(self, signature: Type[Signature], demos: list[dict[str, Any]], inputs: dict[str, Any]) -> list[
		dict[str, Any]]:

		def format_finetune_data(self, signature: Type[Signature], demos: list[dict[str, Any]], inputs: dict[str, Any],
		outputs: dict[str, Any]) -> dict[str, list[Any]]:

		def format_turn(self, signature: Type[Signature], values: dict[str, Any], role: str, incomplete: bool = False,
		is_conversation_history: bool = False) -> dict[str, Any]:

Fix metadata processing in chat adapter #2040

Are you sure you want to change the base?

Fix metadata processing in chat adapter #2040

Conversation

gilad12-coder commented Jan 11, 2025

chenmoneygithub left a comment

Choose a reason for hiding this comment

zbambergerNLP left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenmoneygithub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment