From 8f8711c862a487596d9695b5eeee0a89380479af Mon Sep 17 00:00:00 2001 From: Marc Udoff Date: Fri, 7 Feb 2025 16:56:58 -0500 Subject: [PATCH 1/6] AI Representation In this JEP we propose: 1. A way for objects to represent themselves at runtime for AI 2. A registry for users to define representations for objects that do not have them 3. A new messaging protocol to query for this data JEP for #128 --- 129-ai-representation/ai-representation.md | 208 +++++++++++++++++++++ 1 file changed, 208 insertions(+) create mode 100644 129-ai-representation/ai-representation.md diff --git a/129-ai-representation/ai-representation.md b/129-ai-representation/ai-representation.md new file mode 100644 index 00000000..732c7c47 --- /dev/null +++ b/129-ai-representation/ai-representation.md @@ -0,0 +1,208 @@ +--- +title: AI Representation Protocol for Jupyter +authors: Marc Udoff, Govinda Totla +issue-number: 128 +pr-number: +date-started: 2025-02-07 +--- + +### Summary + +This proposal introduces a standardized method, `_ai_repr_(**kwargs) -> Dict[str, Any]`, for objects in Jupyter environments to provide context-sensitive representations optimized for AI interactions. The protocol allows flexibility for multimodal or text-only representations and supports user-defined registries for objects lacking native implementations. Additionally, we propose a new messaging protocol for retrieving these representations, facilitating seamless integration into Jupyter AI tools. + +### Motivation + +Jupyter’s potential to deeply integrate AI is hindered by the lack of a standard mechanism for objects to provide rich, dynamic representations. Users interacting with AI tools should be able to reference variables and objects meaningfully within their notebooks, e.g., “Given `@myvar`, how do I add another row?” or “What’s the best way to do X with `@myvar`?” Existing mechanisms like `_repr_*` are insufficient due to performance concerns and their inability to adapt representations based on user preferences, model requirements, or contextual factors. This proposal addresses these gaps by: + +1. Defining a flexible, extensible protocol for AI representations. +2. Enabling custom object representations via registries without requiring upstream changes. +3. Standardizing a messaging protocol for the frontend to retrieve representations. + +Expected outcomes include enhanced AI-driven workflows, better support for multimodal interactions, and improved user experience in Jupyter-based AI applications. + +While this JEP focuses on Jupyter, it's understood that others may choose to reuse this same function in any python +process. While that would be a good outcome for the community, it is not the goal of this repr to set standards in +Python in general. + +### Guide-Level Explanation + +#### Introducing the `_ai_repr_` Protocol + +The `_ai_repr_` method allows objects to define representations tailored for AI interactions. This method returns a dictionary (`Dict[str, Any]`), where keys are MIME types (e.g., `text/plain`, `image/png`) and values are the corresponding representations. + +Example: + +```python +class CustomChart: + async def _ai_repr_(self, **kwargs): + return { + "text/plain": f"A chart titled {self.title} with series {self.series_names}", + "image/png": await self.render_image() + } +``` + +This allows flexibility: +- For text-only models, the `text/plain` key provides a concise description. +- For multimodal models, `image/png` can render the chart visually. + +For specific extensions, they can define new mime-types that give the required data. + +In practice, the kwargs should guides the `_ai_repr_` determine what to include in the mime bundle. For a text-only +model, we should not waste time generatin the image. + +Now I can talk with my an mutli-modal LLM. For example, I may say (assuming my plugin substitutes @ for the value form `_ai_repr_`): +> Please generate a caption for @my_chart. + +It should be assumed that callers may choose to inject type infromation about these objects as well, making it +even more powerful: +> In @my_tbl, can you help me turn the columns labeled with dates into accumulated monthly columns + +In this case, `@my_tbl` would not only give the LLM data about its schema, but we'd know this is Pandas or Polars without +a user having to specify this. + +It's possible that we'd want both an async and non-async version supported (or even just a sync version). If so, we can default one to the other: +``` +class CustomChart: + async def _ai_repr_(self, **kwargs): + return _ai_repr_(**kwargs) +``` + +#### Using Registries for Unsupported Objects + +To support objects that do not implement `_ai_repr_`, users can define custom representations in a registry: + +```python +from ai_repr import formatter + +def custom_repr(obj): + return {"text/plain": f"Custom representation of {type(obj).__name__}"} + +formatter.register(type(SomeExternalClass), custom_repr) + +formatter(SomeExternalClass()) # returns custom_repr +formatter(CustomChart()) # returns _ai_repr_ as above +formatter(ObjectWithNoAIReprOrRegisteredFormatter()) # str(obj) +``` + +This allows users to make useful representations of all objects they care about now, and extend the mime type of +existing objects without needing to monkey patch anything. This layering is imporant for empowering users quickly in this +space, without forcing any project to adopt nor forcing any user to upgrade a library. + +The registry should be free of any dependency to minimize conflicts. + +Both the registry and repr on objects are intended to be fully backwards compatible. + +#### Messaging Protocol + +A new message type, `ai_repr_request`, allows the frontend to retrieve representations dynamically. The content of a `ai_repr_request` message includes the following: + +```json +{ + "type": "ai_repr_request", + "content": { + "object_name": "", # The name of the object whose representation is requested + "kwargs": { # Optional context-specific parameters for representation customization + "": "" + } + } +} +``` +- **`object_name`**: (Required) The name of the variable or object in the kernel namespace for which the representation is being requested. +- **`kwargs`**: (Optional) Key-value pairs allowing customization of the representation (e.g., toggling multimodal outputs, adjusting verbosity). + +The kernel returns a response containing the representation of the requested object. The response format is as follows: + +```json +{ + "type": "ai_repr_reply", + "content": { + "data": { + "text/plain": "", # Plain-text representation + "image/png": "" # Optional, other MIME types as needed + }, + "metadata": { + "": { # Metadata associated with specific MIME types + "": "" + } + }, + "transient": { + "display_id": "" # Optional runtime-specific metadata + } + } +} +- **`data`**: A dictionary of MIME-typed representations for the object. +- **`metadata`**: A dictionary of metadata associated with the data, including global metadata and MIME-type-specific sub-dictionaries. +- **`transient`**: A dictionary of transient metadata, not intended to be persisted, such as a `display_id` for session-specific tracking. + + +##### Example + +```python +msg = { + "type": "ai_repr_request", + "content": { + "object_name": "myvar", + "kwargs": {"multimodal": False} + } +} +``` + +This allows for front end plugins enable the variable reference features (`@myvar`) noted above. Because it can pass kwargs, +it has fine-grained control over passsing relevant information to reprs. + +### Reference-Level Explanation + +#### Design Details + +- **Method Signature**: `_ai_repr_(**kwargs) -> Dict[str, Any]` + + - Must return at least one MIME type. + - Supports context-specific hints via `kwargs` (e.g., `multimodal`, `context_window`). + +- **Registries**: Allow user-defined representations for unsupported objects via `ai_repr`. + + That is, in the kernel, we run `await ai_repr.formatter(myvar)` to get the representation. + +- **Messaging Protocol**: + + - Define `ai_repr_request` as a new message type. + - Ensure it supports asynchronous processing for slow operations (e.g., rendering images). + +#### Interactions with Existing Features + +- Builds on Jupyter’s existing `_repr_*` ecosystem but focuses on AI-specific needs. +- Compatible with existing kernel and frontend architectures. + +### Rationale and Alternatives + +#### Rationale + +- Standardization ensures consistent, predictable behavior across libraries. +- Flexible MIME bundles cater to diverse AI model requirements. +- Registries accelerate adoption without waiting for upstream changes. + +#### Alternatives + +- **Extend `_repr_*` methods**: Lacks flexibility for passing model specific features (e.g., multimodal) and context-specific needs (e.g., using more/less of a context window). +- **Functional API only**: A single top-level function could work but limits extensibility and modularity. +- **Ignore standardization**: Risks fragmented, incompatible implementations. + +### Prior Art + +- **IPython’s `_repr_*` methods**: Provides the foundation but lacks AI-specific extensions. +- **Multimodal frameworks**: Other ecosystems (e.g., Hugging Face) emphasize context-aware representations. + +### Future Possibilities + +- Develop AI-specific registries for popular libraries (e.g., pandas, matplotlib) or make PRs to add this natively. + +This proposal lays the groundwork for seamless AI integration in Jupyter, empowering users to derive greater insights and efficiencies from their notebooks. + +# Unresolved questions + +- Should we support both or one of async/sync `_repr_ai_` +- What are good reccomended kwargs to pass +- How should this related to `repr_*`, if at all. +- What is the right default for objects without reprs/formatters defined? `str(obj)`, `None`, or `_repr_`? +- Should thread-saftey be required so that this can be called via a comm +- Can `ai_repr_request` be canceled? From b278d7fe5dcd616479c67df81d17797393ea5d92 Mon Sep 17 00:00:00 2001 From: Marc Udoff Date: Tue, 11 Feb 2025 18:10:25 -0500 Subject: [PATCH 2/6] Update 129-ai-representation/ai-representation.md Co-authored-by: David L. Qiu --- 129-ai-representation/ai-representation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/129-ai-representation/ai-representation.md b/129-ai-representation/ai-representation.md index 732c7c47..0f29523a 100644 --- a/129-ai-representation/ai-representation.md +++ b/129-ai-representation/ai-representation.md @@ -48,7 +48,7 @@ This allows flexibility: For specific extensions, they can define new mime-types that give the required data. In practice, the kwargs should guides the `_ai_repr_` determine what to include in the mime bundle. For a text-only -model, we should not waste time generatin the image. +model, we should not waste time generating the image. Now I can talk with my an mutli-modal LLM. For example, I may say (assuming my plugin substitutes @ for the value form `_ai_repr_`): > Please generate a caption for @my_chart. From 90d20bed1140eb2085f75e036a0d058f58fb7b77 Mon Sep 17 00:00:00 2001 From: Marc Udoff Date: Tue, 11 Feb 2025 18:10:31 -0500 Subject: [PATCH 3/6] Update 129-ai-representation/ai-representation.md Co-authored-by: David L. Qiu --- 129-ai-representation/ai-representation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/129-ai-representation/ai-representation.md b/129-ai-representation/ai-representation.md index 0f29523a..0b914fef 100644 --- a/129-ai-representation/ai-representation.md +++ b/129-ai-representation/ai-representation.md @@ -2,7 +2,7 @@ title: AI Representation Protocol for Jupyter authors: Marc Udoff, Govinda Totla issue-number: 128 -pr-number: +pr-number: 129 date-started: 2025-02-07 --- From 7e4fccfffbce663395443c18d6311927e81f086c Mon Sep 17 00:00:00 2001 From: Marc Udoff Date: Tue, 11 Feb 2025 18:11:44 -0500 Subject: [PATCH 4/6] Update 129-ai-representation/ai-representation.md Co-authored-by: David L. Qiu --- 129-ai-representation/ai-representation.md | 1 + 1 file changed, 1 insertion(+) diff --git a/129-ai-representation/ai-representation.md b/129-ai-representation/ai-representation.md index 0b914fef..e501c289 100644 --- a/129-ai-representation/ai-representation.md +++ b/129-ai-representation/ai-representation.md @@ -130,6 +130,7 @@ The kernel returns a response containing the representation of the requested obj } } } +``` - **`data`**: A dictionary of MIME-typed representations for the object. - **`metadata`**: A dictionary of metadata associated with the data, including global metadata and MIME-type-specific sub-dictionaries. - **`transient`**: A dictionary of transient metadata, not intended to be persisted, such as a `display_id` for session-specific tracking. From f805b4e5d1eabf45774507c42aff7430c1335698 Mon Sep 17 00:00:00 2001 From: Marc Udoff Date: Tue, 11 Feb 2025 18:12:19 -0500 Subject: [PATCH 5/6] Update 129-ai-representation/ai-representation.md Co-authored-by: David L. Qiu --- 129-ai-representation/ai-representation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/129-ai-representation/ai-representation.md b/129-ai-representation/ai-representation.md index e501c289..8f07d8bb 100644 --- a/129-ai-representation/ai-representation.md +++ b/129-ai-representation/ai-representation.md @@ -201,9 +201,9 @@ This proposal lays the groundwork for seamless AI integration in Jupyter, empowe # Unresolved questions -- Should we support both or one of async/sync `_repr_ai_` -- What are good reccomended kwargs to pass -- How should this related to `repr_*`, if at all. +- Should we support both or one of async/sync `_ai_repr_()`? +- What are good recommended `kwargs` to pass in `ai_repr_request`? +- How should this relate to `repr_*`, if at all? - What is the right default for objects without reprs/formatters defined? `str(obj)`, `None`, or `_repr_`? -- Should thread-saftey be required so that this can be called via a comm +- Should thread-safety be required so that this can be called via a comm? - Can `ai_repr_request` be canceled? From 1282304ce6870ffd2e3d94430d1104589e872fcb Mon Sep 17 00:00:00 2001 From: Marc Udoff Date: Wed, 12 Feb 2025 09:45:47 -0500 Subject: [PATCH 6/6] Update 129-ai-representation/ai-representation.md Co-authored-by: M Bussonnier --- 129-ai-representation/ai-representation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/129-ai-representation/ai-representation.md b/129-ai-representation/ai-representation.md index 8f07d8bb..c882da51 100644 --- a/129-ai-representation/ai-representation.md +++ b/129-ai-representation/ai-representation.md @@ -8,7 +8,7 @@ date-started: 2025-02-07 ### Summary -This proposal introduces a standardized method, `_ai_repr_(**kwargs) -> Dict[str, Any]`, for objects in Jupyter environments to provide context-sensitive representations optimized for AI interactions. The protocol allows flexibility for multimodal or text-only representations and supports user-defined registries for objects lacking native implementations. Additionally, we propose a new messaging protocol for retrieving these representations, facilitating seamless integration into Jupyter AI tools. +This proposal introduces a standardized method, `_ai_repr_(self, **kwargs) -> Dict[str, Any]`, for objects in Jupyter environments to provide context-sensitive representations optimized for AI interactions. The protocol allows flexibility for multimodal or text-only representations and supports user-defined registries for objects lacking native implementations. Additionally, we propose a new messaging protocol for retrieving these representations, facilitating seamless integration into Jupyter AI tools. ### Motivation