.Net: Add AIContextProvider support to Semantic Kernel (#11689)

### Motivation and Context

Adding support for AI context behaviors to Semantic Kernel.
These allow creating plugins that are able to listen to messages being
added to the chat history and contribute to the AI context on each
invocation. This PR also integrates support for these with Agents and
AgentThreads.

#10100 
#10712

---------

Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>

Author: 3 people

docs/decisions/0072-agents-with-memory.md

@@ -0,0 +1,209 @@
+---
+# These are optional elements. Feel free to remove any of them.
+status: accepted
+contact: westey-m
+date: 2025-04-17
+deciders: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
+consulted: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
+informed: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
+---
+
+# Agents with Memory
+
+## What do we mean by Memory?
+
+By memory we mean the capability to remember information and skills that are learned during
+a conversation and re-use those later in the same conversation or later in a subsequent conversation.
+
+## Context and Problem Statement
+
+Today we support multiple agent types with different characteristics:
+
+1. In process vs remote.
+2. Remote agents that store and maintain conversation state in the service vs those that require the caller to provide conversation state on each invocation.
+
+We need to support advanced memory capabilities across this range of agent types.
+
+### Memory Scope
+
+Another aspect of memory that is important to consider is the scope of different memory types.
+Most agent implementations have instructions and skills but the agent is not tied to a single conversation.
+On each invocation of the agent, the agent is told which conversation to participate in, during that invocation.
+
+Memories about a user or about a conversation with a user is therefore extracted from one of these conversation and recalled
+during the same or another conversation with the same user.
+These memories will typically contain information that the user would not like to share with other users of the system.
+
+Other types of memories also exist which are not tied to a specific user or conversation.
+E.g. an Agent may learn how to do something and be able to do that in many conversations with different users.
+With these type of memories there is of cousrse risk in leaking personal information between different users which is important to guard against.
+
+### Packaging memory capabilities
+
+All of the above memory types can be supported for any agent by attaching software components to conversation threads.
+This is achieved via a simple mechanism of:
+
+1. Inspecting and using messages as they are passed to and from the agent.
+2. Passing additional context to the agent per invocation.
+
+With our current `AgentThread` implementation, when an agent is invoked, all input and output messages are already passed to the `AgentThread`
+and can be made available to any components attached to the `AgentThread`.
+Where agents are remote/external and manage conversation state in the service, passing the messages to the `AgentThread` may not have any
+affect on the thread in the service. This is OK, since the service will have already updated the thread during the remote invocation.
+It does however, still allow us to subscribe to messages in any attached components.
+
+For the second requirement of getting additional context per invocation, the agent may ask the thread passed to it, to in turn ask
+each of the components attached to it, to provide context to pass to the Agent.
+This enables the component to provide memories that it contains to the Agent as needed.
+
+Different memory capabilities can be built using separate components. Each component would have the following characteristics:
+
+1. May store some context that can be provided to the agent per invocation.
+2. May inspect messages from the conversation to learn from the conversation and build its context.
+3. May register plugins to allow the agent to directly store, retrieve, update or clear memories.
+
+### Suspend / Resume
+
+Building a service to host an agent comes with challenges.
+It's hard to build a stateful service, but service consumers expect an experience that looks stateful from the outside.
+E.g. on each invocation, the user expects that the service can continue a conversation they are having.
+
+This means that where the the service is exposing a local agent with local conversation state management (e.g. via `ChatHistory`)
+that conversation state needs to be loaded and persisted for each invocation of the service.
+
+It also means that any memory components that may have some in-memory state will need to be loaded and persisted too.
+
+For cases like this, the `OnSuspend` and `OnResume` methods allow notification of the components that they need to save or reload their state.
+It is up to each of these components to decide how and where to save state to or load state from.
+
+## Proposed interface for Memory Components
+
+The types of events that Memory Components require are not unique to memory, and can be used to package up other capabilities too.
+The suggestion is therefore to create a more generally named type that can be used for other scenarios as well and can even
+be used for non-agent scenarios too.
+
+This type should live in the `Microsoft.SemanticKernel.Abstractions` nuget, since these components can be used by systems other than just agents.
+
+```csharp
+namespace Microsoft.SemanticKernel;
+
+public abstract class AIContextBehavior
+{
+    public virtual IReadOnlyCollection<AIFunction> AIFunctions => Array.Empty<AIFunction>();
+
+    public virtual Task OnThreadCreatedAsync(string? threadId, CancellationToken cancellationToken = default);
+    public virtual Task OnThreadDeleteAsync(string? threadId, CancellationToken cancellationToken = default);
+
+    // OnThreadCheckpointAsync not included in initial release, maybe in future.
+    public virtual Task OnThreadCheckpointAsync(string? threadId, CancellationToken cancellationToken = default);
+
+    public virtual Task OnNewMessageAsync(string? threadId, ChatMessage newMessage, CancellationToken cancellationToken = default);
+    public abstract Task<string> OnModelInvokeAsync(ICollection<ChatMessage> newMessages, CancellationToken cancellationToken = default);
+
+    public virtual Task OnSuspendAsync(string? threadId, CancellationToken cancellationToken = default);
+    public virtual Task OnResumeAsync(string? threadId, CancellationToken cancellationToken = default);
+}
+```
+
+## Managing multiple components
+
+To manage multiple components I propose that we have a `AIContextBehavior`.
+This class allows registering components and delegating new message notifications, ai invocation calls, etc. to the contained components.
+
+## Integrating with agents
+
+I propose to add a `AIContextBehaviorManager` to the `AgentThread` class, allowing us to attach components to any `AgentThread`.
+
+When an `Agent` is invoked, we will call `OnModelInvokeAsync` on each component via the `AIContextBehaviorManager` to get
+a combined set of context to pass to the agent for this invocation. This will be internal to the `Agent` class and transparent to the user.
+
+```csharp
+var additionalInstructions = await currentAgentThread.OnModelInvokeAsync(messages, cancellationToken).ConfigureAwait(false);
+```
+
+## Usage examples
+
+### Multiple threads using the same memory component
+
+```csharp
+// Create a vector store for storing memories.
+var vectorStore = new InMemoryVectorStore();
+// Create a memory store that is tired to a "Memories" collection in the vector store and stores memories under the "user/12345" namespace.
+using var textMemoryStore = new VectorDataTextMemoryStore<string>(vectorStore, textEmbeddingService, "Memories", "user/12345", 1536);
+
+// Create a memory component to will pull user facts from the conversation, store them in the vector store
+// and pass them to the agent as additional instructions.
+var userFacts = new UserFactsMemoryComponent(this.Fixture.Agent.Kernel, textMemoryStore);
+
+// Create a thread and attach a Memory Component.
+var agentThread1 = new ChatHistoryAgentThread();
+agentThread1.ThreadExtensionsManager.Add(userFacts);
+var asyncResults1 = agent.InvokeAsync("Hello, my name is Caoimhe.", agentThread1);
+
+// Create a second thread and attach a Memory Component.
+var agentThread2 = new ChatHistoryAgentThread();
+agentThread2.ThreadExtensionsManager.Add(userFacts);
+var asyncResults2 = agent.InvokeAsync("What is my name?.", agentThread2);
+// Expected response contains Caoimhe.
+```
+
+### Using a RAG component
+
+```csharp
+// Create Vector Store and Rag Store/Component
+var vectorStore = new InMemoryVectorStore();
+using var ragStore = new TextRagStore<string>(vectorStore, textEmbeddingService, "Memories", 1536, "group/g2");
+var ragComponent = new TextRagComponent(ragStore, new TextRagComponentOptions());
+
+// Upsert docs into vector store.
+await ragStore.UpsertDocumentsAsync(
+[
+    new TextRagDocument("The financial results of Contoso Corp for 2023 is as follows:\nIncome EUR 174 000 000\nExpenses EUR 152 000 000")
+    {
+        SourceName = "Contoso 2023 Financial Report",
+        SourceReference = "https://www.consoso.com/reports/2023.pdf",
+        Namespaces = ["group/g2"]
+    }
+]);
+    
+// Create a new agent thread and register the Rag component
+var agentThread = new ChatHistoryAgentThread();
+agentThread.ThreadExtensionsManager.RegisterThreadExtension(ragComponent);
+
+// Inovke the agent.
+var asyncResults1 = agent.InvokeAsync("What was the income of Contoso for 2023", agentThread);
+// Expected response contains the 174M income from the document.
+```
+
+## Decisions to make
+
+### Extension base class name
+
+1. ConversationStateExtension
+
+    1.1. Long
+
+2. MemoryComponent
+
+    2.1. Too specific
+
+3. AIContextBehavior
+
+Decided 3. AIContextBehavior.
+
+### Location for abstractions
+
+1. Microsoft.SemanticKernel.<baseclass>
+2. Microsoft.SemanticKernel.Memory.<baseclass>
+3. Microsoft.SemanticKernel.Memory.<baseclass> (in separate nuget)
+
+Decided: 1. Microsoft.SemanticKernel.<baseclass>.
+
+### Location for memory components
+
+1. A nuget for each component
+2. Microsoft.SemanticKernel.Core nuget
+3. Microsoft.SemanticKernel.Memory nuget
+4. Microsoft.SemanticKernel.ConversationStateExtensions nuget
+
+Decided: 2. Microsoft.SemanticKernel.Core nuget

dotnet/docs/EXPERIMENTS.md

@@ -25,6 +25,7 @@ You can use the following diagnostic IDs to ignore warnings or errors for a part
 | SKEXP0100 | Advanced Semantic Kernel features |
 | SKEXP0110 | Semantic Kernel Agents |
 | SKEXP0120 | Native-AOT |
+| SKEXP0130 | AI Context Providers |
 | MEVD9000 | Microsoft.Extensions.VectorData experimental user-facing APIs |
 | MEVD9001 | Microsoft.Extensions.VectorData experimental connector-facing APIs |
 

dotnet/samples/Concepts/Agents/ChatCompletion_Mem0.cs

@@ -0,0 +1,119 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Net.Http.Headers;
+using Microsoft.SemanticKernel;
+using Microsoft.SemanticKernel.Agents;
+using Microsoft.SemanticKernel.Memory;
+
+namespace Agents;
+
+#pragma warning disable SKEXP0130 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
+
+/// <summary>
+/// Demonstrate creation of <see cref="ChatCompletionAgent"/> and
+/// adding memory capabilities to it using https://mem0.ai/.
+/// </summary>
+public class ChatCompletion_Mem0(ITestOutputHelper output) : BaseTest(output)
+{
+    private const string AgentName = "FriendlyAssistant";
+    private const string AgentInstructions = "You are a friendly assistant";
+
+    /// <summary>
+    /// Shows how to allow an agent to remember user preferences across multiple threads.
+    /// </summary>
+    [Fact]
+    private async Task UseMemoryAsync()
+    {
+        // Create a new HttpClient with the base address of the mem0 service and a token for authentication.
+        using var httpClient = new HttpClient()
+        {
+            BaseAddress = new Uri(TestConfiguration.Mem0.BaseAddress ?? "https://api.mem0.ai")
+        };
+        httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Token", TestConfiguration.Mem0.ApiKey);
+
+        // Create a mem0 component with the current user's id, so that it stores memories for that user.
+        var mem0Provider = new Mem0Provider(httpClient, new()
+        {
+            UserId = "U1"
+        });
+
+        // Clear out any memories from previous runs, if any, to demonstrate a first run experience.
+        await mem0Provider.ClearStoredMemoriesAsync();
+
+        // Create our agent and add our finance plugin with auto function invocation.
+        Kernel kernel = this.CreateKernelWithChatCompletion();
+        kernel.Plugins.AddFromType<FinancePlugin>();
+        ChatCompletionAgent agent =
+            new()
+            {
+                Name = AgentName,
+                Instructions = AgentInstructions,
+                Kernel = kernel,
+                Arguments = new KernelArguments(new PromptExecutionSettings { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() })
+            };
+
+        Console.WriteLine("----- First Conversation -----");
+
+        // Create a thread for the agent and add the mem0 component to it.
+        ChatHistoryAgentThread agentThread = new();
+        agentThread.AIContextProviders.Add(mem0Provider);
+
+        // First ask the agent to retrieve a company report with no previous context.
+        // The agent will not be able to invoke the plugin, since it doesn't know
+        // the company code or the report format, so it should ask for clarification.
+        string userMessage = "Please retrieve my company report";
+        Console.WriteLine($"User: {userMessage}");
+
+        ChatMessageContent message = await agent.InvokeAsync(userMessage, agentThread).FirstAsync();
+        Console.WriteLine($"Assistant:\n{message.Content}");
+
+        // Now tell the agent the company code and the report format that you want to use
+        // and it should be able to invoke the plugin and return the report.
+        userMessage = "I always work with CNTS and I always want a detailed report format";
+        Console.WriteLine($"User: {userMessage}");
+
+        message = await agent.InvokeAsync(userMessage, agentThread).FirstAsync();
+        Console.WriteLine($"Assistant:\n{message.Content}");
+
+        Console.WriteLine("----- Second Conversation -----");
+
+        // Create a new thread for the agent and add our mem0 component to it again
+        // The new thread has no context of the previous conversation.
+        agentThread = new();
+        agentThread.AIContextProviders.Add(mem0Provider);
+
+        // Since we have the mem0 component in the thread, the agent should be able to
+        // retrieve the company report without asking for clarification, as it will
+        // be able to remember the user preferences from the last thread.
+        userMessage = "Please retrieve my company report";
+        Console.WriteLine($"User: {userMessage}");
+
+        message = await agent.InvokeAsync(userMessage, agentThread).FirstAsync();
+        Console.WriteLine($"Assistant:\n{message.Content}");
+    }
+
+    private sealed class FinancePlugin
+    {
+        [KernelFunction]
+        public string RetrieveCompanyReport(string companyCode, ReportFormat reportFormat)
+        {
+            if (companyCode != "CNTS")
+            {
+                throw new ArgumentException("Company code not found");
+            }
+
+            return reportFormat switch
+            {
+                ReportFormat.Brief => "CNTS is a company that specializes in technology.",
+                ReportFormat.Detailed => "CNTS is a company that specializes in technology. It had a revenue of $10 million in 2022. It has 100 employees.",
+                _ => throw new ArgumentException("Report format not found")
+            };
+        }
+    }
+
+    private enum ReportFormat
+    {
+        Brief,
+        Detailed
+    }
+}