Skip to content

Commit 5c04bbe

Browse files
glorious-beardrogerbarreto
glorious-beard
and
rogerbarreto
authored
.Net: Add support for audio and binary tags to chat prompt parser (#11919)
### Motivation and Context #### Why is this change required? This template parsers like the YAML parser to embed content types other than just text and images for LLMs that support additional content types, like PDFs for OpenAI and DOCXs for Claude. Without this capability, functions with prompts that have attachments would have to manually build it's chat history in code. #### What problem does it solve? See above #### What scenario does it contribute to? Usage additional content types beyond visuals and audio for user messages #### Open Issues Addressed - Fixes #11044 ### Description #### Chat Prompt Parser To preserve backward compatibility, rather than consolidating binary content types, I chose to go with adding additional content types so that LLM chat service providers could opt-in to new content types. It also reduces the chances of breaking existing code. 3 new content types are created: * `PdfContent` for PDF files. Uses the tag "&lt;pdf&gt;". Allows for Base64 data URIs or standard URIs, similar to `ImageContent`. * `DocContent` for MS Word .doc files. Uses the tag "&lt;doc&gt;". Allows for Base64 data URIs or standard URIs, similar to `ImageContent`. * `DocxContent` for MS Word .docx files. Uses the tag "&lt;docx&gt;". Allows for Base64 data URIs or standard URIs, similar to `ImageContent`. (**NOTE**: `DocContent` and `DocxContent` are mainly separate because they have different MIME types and different content formats, though they could easily be consolidated into a single tag and just let the LLM provider handle distinguishing between "doc" and "docx" files. Alternately, I could also see the case for dropping ".doc" support and requiring the caller to only use ".docx".) In addition, the following 2 contents are now parsed from the XML: * `AudioContent` - Parses the tag "&lt;audio&gt;" with either Base64 data URIs or standard URIs, similar to `ImageContent`. * `BinaryContent` - Parses the tag "&lt;file&gt;" with either Base64 data URIs or standard URIs, similar to `ImageContent`. Here is a sample: ```xml <message role='user'> This part will be discarded upon parsing <text>Make sense of this random assortment of stuff.</text> <image>https://fake-link-to-image/</image> <audio>data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAIlYAAACABAAZGF0YVgAAAAA</audio> <pdf>data:application/pdf;base64,JVBERi0xLjQKJeLjz9MKMyAwIG9iago8PC9UeXBlL1hSZWYvUGFnZXMgNiAwIFIKL1R5cGUvUGFnZS9NZWRpYUJveCBbMCAwIDQ4MCA1MF0KL0NvbnRlbnRzIDw8L0V4dEdTdGF0ZSA8PC9JRCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GMSA8PC9GMiA8PC9GMyA8PC9GNCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GNSA8PC9GNiA8PC9GNyBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GOCAvPj4KZW5kb2JqCjEwIDAgb2JqCjw8L1R5cGUvUGFnZS9NYWRlYUJveCBbMCAwIDQ4MCA1MF0KL0NvbnRlbnRzIDw8L0V4dEdTdGF0ZSA8PC9JRCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GMSA8PC9GMiA8PC9GMyA8PC9GNCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GNSA8PC9GNiA8PC9GNyBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GOCAvPj4KZW5kb2JqCjEwIDAgb2JqCjw8L1R5cGUvUGFnZS9NYWRlYUJveCBbMCAwIDQ4MCA1MF0KL0NvbnRlbnRzIDw8L0V4dEdTdGF0ZSA8PC9JRCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GMSA8PC9G</pdf> <pdf>https://fake-link-to-pdf/</pdf> <doc>data:application/msword;base64,UEsDBBQAAAAIAI+Q1k5a2gAAABQAAAAIAAAAbmFtZS5kb2N4VVQJAAD9AAAACwAAAB4AAAAAA==</doc> <doc>https://fake-link-to-doc/</doc> <docx>data:application/vnd.openxmlformats-officedocument.wordprocessingml.document;base64,UEsDBBQAAAAIAI+Q1k5a2gAAABQAAAAIAAAAbmFtZS5kb2N4VVQJAAD9AAAACwAAAB4AAAAAA==</docx> <docx>https://fake-link-to-docx/</docx> <file>data:application/octet-stream;base64,UEsDBBQAAAAIAI+Q1k5a2gAAABQAAAAIAAAAbmFtZS5kb2N4VVQJAAD9AAAACwAAAB4AAAAAA==</file> <file>https://fake-link-to-binary/</file> This part will also be discarded upon parsing </message> ``` #### Amazon Bedrock Modified the `Converse` API request generator to handle the subset of binary content supported by Amazon Bedrock (PDF, DOC, DOCX, and Image), as documented [here](https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/BedrockRuntime/TContentBlock.html). #### OpenAI Modified the client to handle PDF content, audio content, and file references when generating a request to an OpenAI (or OpenAI compatible) client. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Roger Barreto <[email protected]>
1 parent da1ad23 commit 5c04bbe

File tree

5 files changed

+345
-27
lines changed

5 files changed

+345
-27
lines changed

‎dotnet/samples/Concepts/PromptTemplates/ChatPromptWithAudio.cs

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
// Copyright (c) Microsoft. All rights reserved.
2+
3+
using Microsoft.SemanticKernel;
4+
using Resources;
5+
6+
namespace PromptTemplates;
7+
8+
/// <summary>
9+
/// This example demonstrates how to use ChatPrompt XML format with Audio content types.
10+
/// The new ChatPrompt parser supports &lt;audio&gt; tags for various audio formats like WAV, MP3, etc.
11+
/// </summary>
12+
public class OpenAI_ChatPromptWithAudio(ITestOutputHelper output) : BaseTest(output)
13+
{
14+
/// <summary>
15+
/// Demonstrates using audio content in ChatPrompt XML format with data URI.
16+
/// </summary>
17+
[Fact]
18+
public async Task ChatPromptWithAudioContentDataUri()
19+
{
20+
// Load an audio file and convert to base64 data URI
21+
var audioBytes = await EmbeddedResource.ReadAllAsync("test_audio.wav");
22+
var audioBase64 = Convert.ToBase64String(audioBytes.ToArray());
23+
var dataUri = $"data:audio/wav;base64,{audioBase64}";
24+
25+
var chatPrompt = $"""
26+
<message role="system">You are a helpful assistant that can analyze audio content.</message>
27+
<message role="user">
28+
<text>Please transcribe and analyze this audio file.</text>
29+
<audio>{dataUri}</audio>
30+
</message>
31+
""";
32+
33+
var kernel = Kernel.CreateBuilder()
34+
.AddOpenAIChatCompletion(
35+
modelId: "gpt-4o-audio-preview", // Use audio-capable model
36+
apiKey: TestConfiguration.OpenAI.ApiKey)
37+
.Build();
38+
39+
var chatFunction = kernel.CreateFunctionFromPrompt(chatPrompt);
40+
var result = await kernel.InvokeAsync(chatFunction);
41+
42+
Console.WriteLine("=== ChatPrompt with Audio Content (Data URI) ===");
43+
Console.WriteLine("Prompt:");
44+
Console.WriteLine(chatPrompt);
45+
Console.WriteLine("\nResult:");
46+
Console.WriteLine(result);
47+
}
48+
49+
/// <summary>
50+
/// Demonstrates a conversation flow using ChatPrompt with audio content across multiple messages.
51+
/// </summary>
52+
[Fact]
53+
public async Task ChatPromptConversationWithAudioContent()
54+
{
55+
var audioBytes = await EmbeddedResource.ReadAllAsync("test_audio.wav");
56+
var audioBase64 = Convert.ToBase64String(audioBytes.ToArray());
57+
var audioDataUri = $"data:audio/wav;base64,{audioBase64}";
58+
59+
var chatPrompt = $"""
60+
<message role="system">You are a helpful assistant that specializes in audio analysis and transcription.</message>
61+
<message role="user">
62+
<text>I have an audio recording that I need help with. Can you analyze it?</text>
63+
<audio>{audioDataUri}</audio>
64+
</message>
65+
<message role="assistant">I can help you analyze this audio recording. Let me transcribe and examine its content for you. What specific information are you looking for from this audio?</message>
66+
<message role="user">
67+
<text>Can you provide a full transcription and also identify any background sounds or audio quality issues?</text>
68+
</message>
69+
""";
70+
71+
var kernel = Kernel.CreateBuilder()
72+
.AddOpenAIChatCompletion(
73+
modelId: "gpt-4o-audio-preview", // Use audio-capable model
74+
apiKey: TestConfiguration.OpenAI.ApiKey)
75+
.Build();
76+
77+
var chatFunction = kernel.CreateFunctionFromPrompt(chatPrompt);
78+
var result = await kernel.InvokeAsync(chatFunction);
79+
80+
Console.WriteLine("=== ChatPrompt Conversation with Audio Content ===");
81+
Console.WriteLine("Prompt (showing conversation flow):");
82+
Console.WriteLine(chatPrompt[..Math.Min(800, chatPrompt.Length)] + "...");
83+
Console.WriteLine("\nResult:");
84+
Console.WriteLine(result);
85+
}
86+
}

‎dotnet/samples/Concepts/PromptTemplates/ChatPromptWithBinary.cs

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
// Copyright (c) Microsoft. All rights reserved.
2+
3+
using Microsoft.SemanticKernel;
4+
using Resources;
5+
6+
namespace PromptTemplates;
7+
8+
/// <summary>
9+
/// This example demonstrates how to use ChatPrompt XML format with Binary content types.
10+
/// The new ChatPrompt parser supports &lt;binary&gt; tags for various document formats like PDF, Word, CSV, etc.
11+
/// </summary>
12+
public class ChatPromptWithBinary(ITestOutputHelper output) : BaseTest(output)
13+
{
14+
/// <summary>
15+
/// Demonstrates using binary content (PDF file) in ChatPrompt XML format with data URI.
16+
/// </summary>
17+
[Fact]
18+
public async Task ChatPromptWithBinaryContentDataUri()
19+
{
20+
// Load a PDF file and convert to base64 data URI
21+
var fileBytes = await EmbeddedResource.ReadAllAsync("employees.pdf");
22+
var fileBase64 = Convert.ToBase64String(fileBytes.ToArray());
23+
var dataUri = $"data:application/pdf;base64,{fileBase64}";
24+
25+
var chatPrompt = $"""
26+
<message role="system">You are a helpful assistant that can analyze documents.</message>
27+
<message role="user">
28+
<text>Please analyze this PDF document and provide a summary of its contents.</text>
29+
<binary>{dataUri}</binary>
30+
</message>
31+
""";
32+
33+
var kernel = Kernel.CreateBuilder()
34+
.AddOpenAIChatCompletion(
35+
modelId: TestConfiguration.OpenAI.ChatModelId,
36+
apiKey: TestConfiguration.OpenAI.ApiKey)
37+
.Build();
38+
39+
var chatFunction = kernel.CreateFunctionFromPrompt(chatPrompt);
40+
var result = await kernel.InvokeAsync(chatFunction);
41+
42+
Console.WriteLine("=== ChatPrompt with Binary Content (Data URI) ===");
43+
Console.WriteLine("Prompt:");
44+
Console.WriteLine(chatPrompt);
45+
Console.WriteLine("\nResult:");
46+
Console.WriteLine(result);
47+
}
48+
49+
/// <summary>
50+
/// Demonstrates a conversation flow using ChatPrompt with binary content across multiple messages.
51+
/// </summary>
52+
[Fact]
53+
public async Task ChatPromptConversationWithBinaryContent()
54+
{
55+
var pdfBytes = await EmbeddedResource.ReadAllAsync("employees.pdf");
56+
var pdfBase64 = Convert.ToBase64String(pdfBytes.ToArray());
57+
var pdfDataUri = $"data:application/pdf;base64,{pdfBase64}";
58+
59+
var chatPrompt = $"""
60+
<message role="system">You are a helpful assistant that can analyze documents and provide insights.</message>
61+
<message role="user">
62+
<text>I have a document that I need help understanding. Can you analyze it?</text>
63+
<binary>{pdfDataUri}</binary>
64+
</message>
65+
<message role="assistant">I can see this is a PDF document about employees. Let me analyze its contents for you. The document appears to contain employee information and organizational data. What specific aspects would you like me to focus on?</message>
66+
<message role="user">
67+
<text>Can you extract the key information and create a summary? Also, what format would be best for sharing this information with my team?</text>
68+
</message>
69+
""";
70+
71+
var kernel = Kernel.CreateBuilder()
72+
.AddOpenAIChatCompletion(
73+
modelId: TestConfiguration.OpenAI.ChatModelId,
74+
apiKey: TestConfiguration.OpenAI.ApiKey)
75+
.Build();
76+
77+
var chatFunction = kernel.CreateFunctionFromPrompt(chatPrompt);
78+
var result = await kernel.InvokeAsync(chatFunction);
79+
80+
Console.WriteLine("=== ChatPrompt Conversation with Binary Content ===");
81+
Console.WriteLine("Prompt (showing conversation flow):");
82+
Console.WriteLine(chatPrompt[..Math.Min(800, chatPrompt.Length)] + "...");
83+
Console.WriteLine("\nResult:");
84+
Console.WriteLine(result);
85+
}
86+
}

‎dotnet/samples/Concepts/README.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -192,16 +192,18 @@ dotnet test -l "console;verbosity=detailed" --filter "FullyQualifiedName=ChatCom
192192
### PromptTemplates - Using [`Templates`](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/PromptTemplate/IPromptTemplate.cs) with parametrization for `Prompt` rendering
193193

194194
- [ChatCompletionPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/ChatCompletionPrompts.cs)
195+
- [ChatLoopWithPrompt](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/ChatLoopWithPrompt.cs)
196+
- [ChatPromptWithAudio](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/ChatPromptWithAudio.cs)
197+
- [ChatPromptWithBinary](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/ChatPromptWithBinary.cs)
195198
- [ChatWithPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/ChatWithPrompts.cs)
196199
- [HandlebarsPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/HandlebarsPrompts.cs)
200+
- [HandlebarsVisionPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/HandlebarsVisionPrompts.cs)
197201
- [LiquidPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/LiquidPrompts.cs)
198202
- [MultiplePromptTemplates](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/MultiplePromptTemplates.cs)
199203
- [PromptFunctionsWithChatGPT](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/PromptFunctionsWithChatGPT.cs)
200-
- [TemplateLanguage](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/TemplateLanguage.cs)
201204
- [PromptyFunction](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/PromptyFunction.cs)
202-
- [HandlebarsVisionPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/HandlebarsVisionPrompts.cs)
203205
- [SafeChatPrompts](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/SafeChatPrompts.cs)
204-
- [ChatLoopWithPrompt](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/ChatLoopWithPrompt.cs)
206+
- [TemplateLanguage](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/PromptTemplates/TemplateLanguage.cs)
205207

206208
### RAG - Retrieval-Augmented Generation
207209

‎dotnet/src/SemanticKernel.Abstractions/AI/ChatCompletion/ChatPromptParser.cs

Lines changed: 42 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,6 @@ namespace Microsoft.SemanticKernel.ChatCompletion;
1212
/// </summary>
1313
internal static class ChatPromptParser
1414
{
15-
private const string MessageTagName = "message";
16-
private const string RoleAttributeName = "role";
17-
private const string ImageTagName = "image";
18-
private const string TextTagName = "text";
19-
2015
/// <summary>
2116
/// Parses a prompt for an XML representation of a <see cref="ChatHistory"/>.
2217
/// </summary>
@@ -73,20 +68,10 @@ private static ChatMessageContent ParseChatNode(PromptNode node)
7368
ChatMessageContentItemCollection items = [];
7469
foreach (var childNode in node.ChildNodes.Where(childNode => childNode.Content is not null))
7570
{
76-
if (childNode.TagName.Equals(ImageTagName, StringComparison.OrdinalIgnoreCase))
77-
{
78-
if (childNode.Content!.StartsWith("data:", StringComparison.OrdinalIgnoreCase))
79-
{
80-
items.Add(new ImageContent(childNode.Content));
81-
}
82-
else
83-
{
84-
items.Add(new ImageContent(new Uri(childNode.Content!)));
85-
}
86-
}
87-
else if (childNode.TagName.Equals(TextTagName, StringComparison.OrdinalIgnoreCase))
71+
if (s_contentFactoryMapping.TryGetValue(childNode.TagName.ToUpperInvariant(), out var createBinaryContent))
8872
{
89-
items.Add(new TextContent(childNode.Content));
73+
childNode.Attributes.TryGetValue("mimetype", out var mimeType);
74+
items.Add(createBinaryContent(childNode.Content!, mimeType));
9075
}
9176
}
9277

@@ -103,21 +88,55 @@ private static ChatMessageContent ParseChatNode(PromptNode node)
10388
: new ChatMessageContent(authorRole, node.Content);
10489
}
10590

91+
/// <summary>
92+
/// Creates a new instance of <typeparamref name="T"/> from a data URI.
93+
/// </summary>
94+
/// <typeparam name="T">Type of <see cref="BinaryContent"/> to create.</typeparam>
95+
/// <param name="content">Base64 encoded content or URI.</param>
96+
/// <param name="mimeType">Optional MIME type of the content.</param>
97+
/// <returns>A new instance of <typeparamref name="T"/> with <paramref name="content"/></returns>
98+
private static T CreateBinaryContent<T>(string content, string? mimeType) where T : BinaryContent, new()
99+
{
100+
return (content.StartsWith("data:", StringComparison.OrdinalIgnoreCase)) ? new T { DataUri = content } : new T { Uri = new Uri(content), MimeType = mimeType };
101+
}
102+
103+
/// <summary>
104+
/// Factory for creating a <see cref="KernelContent"/> instance based on the tag name.
105+
/// </summary>
106+
private static readonly Dictionary<string, Func<string, string?, KernelContent>> s_contentFactoryMapping = new()
107+
{
108+
{ TextTagName, (content, _) => new TextContent(content) },
109+
{ ImageTagName, CreateBinaryContent<ImageContent> },
110+
{ AudioTagName, CreateBinaryContent<AudioContent> },
111+
{ BinaryTagName, CreateBinaryContent<BinaryContent> }
112+
};
113+
106114
/// <summary>
107115
/// Checks if <see cref="PromptNode"/> is valid chat message.
108116
/// </summary>
109117
/// <param name="node">Instance of <see cref="PromptNode"/>.</param>
110118
/// <remarks>
111-
/// A valid chat message is a node with the following structure:<br/>
112-
/// TagName = "message"<br/>
113-
/// Attributes = { "role" : "..." }<br/>
114-
/// optional one or more child nodes <image>...</image><br/>
115-
/// optional one or more child nodes <text>...</text>
119+
/// A valid chat message is a node with the following structure:
120+
/// <list type="bullet">
121+
/// <item><description>TagName = "message"</description></item>
122+
/// <item><description>Attributes = { "role" : "..." }</description></item>
123+
/// <item><description>optional one or more child nodes &lt;image&gt;...&lt;/image&gt;</description></item>
124+
/// <item><description>optional one or more child nodes &lt;text&gt;...&lt;/text&gt;</description></item>
125+
/// <item><description>optional one or more child nodes &lt;audio&gt;...&lt;/audio&gt;</description></item>
126+
/// <item><description>optional one or more child nodes &lt;binary&gt;...&lt;/binary&gt;</description></item>
127+
/// </list>
116128
/// </remarks>
117129
private static bool IsValidChatMessage(PromptNode node)
118130
{
119131
return
120132
node.TagName.Equals(MessageTagName, StringComparison.OrdinalIgnoreCase) &&
121133
node.Attributes.ContainsKey(RoleAttributeName);
122134
}
135+
136+
private const string MessageTagName = "message";
137+
private const string RoleAttributeName = "role";
138+
private const string ImageTagName = "IMAGE";
139+
private const string TextTagName = "TEXT";
140+
private const string AudioTagName = "AUDIO";
141+
private const string BinaryTagName = "BINARY";
123142
}

0 commit comments

Comments
 (0)