ChatCompletionCache cache key ignores tool_choice, serves stale cached response when only tool_choice differs

[ErenAta16]

What happened?

ChatCompletionCache's cache key doesn't include tool_choice. Two calls with identical messages/tools/json_output/extra_create_args but different tool_choice collide on the same cache key, so the second call silently gets served a stale, wrong-shaped cached response instead of hitting the underlying client.

tool_choice is a first-class parameter of ChatCompletionClient.create/create_stream (auto lets the model decide, required forces a tool call, none forbids tool calls, a specific Tool forces that exact tool), and it materially changes the shape of the response, in the real OpenAI client it's translated directly into the API's tool_choice field and changes whether the response is plain text or a list of FunctionCalls.

ChatCompletionCache._check_cache (python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py) builds its hash from messages, tools, json_output, and extra_create_args only:

data = {
    "messages": [message.model_dump() for message in messages],
    "tools": [(tool.schema if isinstance(tool, Tool) else tool) for tool in tools],
    "json_output": json_output_data,
    "extra_create_args": extra_create_args,
}
serialized_data = json.dumps(data, sort_keys=True)
cache_key = hashlib.sha256(serialized_data.encode()).hexdigest()

create() calls self._check_cache(messages, tools, json_output, extra_create_args), no tool_choice, even though it's an explicit parameter of create() itself and gets correctly forwarded to the underlying client a few lines later on cache miss. create_stream() has the identical gap.

This is exactly the scenario the codebase itself exercises internally: AssistantAgent._reflect_on_tool_use_flow explicitly passes tool_choice="none" to force a plain-text reflection response after tool execution, while the normal tool-call path uses the default tool_choice="auto". Any application/agent design that reuses one ChatCompletionCache-wrapped client across calls that vary only tool_choice for the same conversation prefix will silently get the wrong kind of answer.

What did you expect to happen?

A call that only differs from a previous one by tool_choice should be a cache miss and actually invoke the underlying client, not replay a response computed for a different tool_choice.

How can we reproduce it (as minimally and precisely as possible)?

import asyncio
from autogen_core import InMemoryStore
from autogen_core.models import UserMessage
from autogen_ext.models.cache import ChatCompletionCache
from autogen_ext.models.replay import ReplayChatCompletionClient

async def main():
    responses = ["This is dummy message number 0", "This is dummy message number 1"]
    replay_client = ReplayChatCompletionClient(responses)
    replay_client.set_cached_bool_value(False)
    cached_client = ChatCompletionCache(replay_client, InMemoryStore())

    message = [UserMessage(content="hi", source="user")]

    r0 = await cached_client.create(message, tool_choice="auto")
    print("call 1 (tool_choice=auto):", r0.cached, repr(r0.content))

    r1 = await cached_client.create(message, tool_choice="none")
    print("call 2 (tool_choice=none):", r1.cached, repr(r1.content))

asyncio.run(main())

Output:

call 1 (tool_choice=auto): False 'This is dummy message number 0'
call 2 (tool_choice=none): True 'This is dummy message number 0'

Call 2 reports cached=True and returns call 1's response, even though it was made with a different tool_choice and, with a real tool_choice-aware client, should produce a genuinely different (plain-text) response. I also reproduced this with a fake client whose response actually depends on tool_choice (mirroring a real provider), confirming the underlying client is never invoked for the second call at all (only tool_choice='auto' shows up in the fake client's call log).

AutoGen version

Reproduced against the current main branch source of _chat_completion_cache.py, using the installed autogen-ext/autogen-core packages from an editable install of the same source. Python 3.12.10, Windows.

Which package was this bug in?

Python autogen-ext

Model used

N/A, reproduced with ReplayChatCompletionClient and a fake client, no real model API needed.

Python version

3.12

Operating system

Windows

Additional Information

Opening a fix, will link the PR here.

[0xbrainkid]

Good catch. I would treat tool_choice as part of the agent's effective capability state, not just a model-call formatting option.

The risky case is any workflow that alternates between:

• a tool-enabled step where the assistant may return function calls
• a reflection / explanation / finalization step where tool calls are explicitly disabled

If those two calls share a cache key, the cache can replay a response produced under a different authority boundary. That is more serious than a stale text response because the caller may interpret the returned shape under the current step's policy, while it was generated under a previous step's policy.

A regression test I would want in the fix:

• same messages
• same tools
• same extra args
• first call with tool_choice="required" or a specific tool
• second call with tool_choice="none"
• assert the second call is not cached and the underlying client sees the second tool_choice

It may also be worth normalizing explicit tool_choice="auto" and omitted/default tool_choice deliberately, rather than letting those become accidental cache behavior. The important invariant is that any parameter changing whether an agent can call tools, which tool it can call, or what response shape is valid must be represented in the cache key.

This is the same class of issue we keep running into with multi-agent systems: identity and authorization are checked at the boundary, but the execution record needs to preserve the exact capability context that produced a result.

[ErenAta16]

That framing is better than mine, "effective capability state" names the actual invariant instead of just the symptom. My original report treated this as a cache-key completeness bug (tool_choice missing from the key), but you're right that the alternating tool-enabled/reflection-step pattern is the case that actually matters in practice, not just a theoretical hash-collision.

Agreed on normalizing "auto" vs. omitted tool_choice deliberately rather than letting that fall out of whatever the cache key construction happens to do with None vs the string "auto". If those two hash differently by accident, we'd have swapped one silent bug for a different, more confusing one (unnecessary cache misses that look like a partial fix).

I'll fold your regression test shape into the fix rather than the narrower one I had (same messages/tools/args, tool_choice="required" then tool_choice="none", assert no cache hit and the second call reaches the client with the right value), plus a case for "auto" vs. omitted to nail down the normalization question explicitly.

[IgorGanapolsky]

Agentic systems are turning into always-on factories: one request fans out into model calls, tool calls, policy checks, and storage.

The hard part is rarely tokens/sec — it is control-plane reliability:

1. Fan-out control — hard cap on tools/steps per request before external writes
2. Durable id + idempotency — one write per business object (ledger / preflight)
3. Observable long jobs — progress + cooperative cancel on multi-minute runs
4. Expected vs actual — durable artifact id before marking success
5. Policy before action — gate that can refuse known-bad side effects

If you share one sanitized failing run (expected vs actual + tool result), I can map which pillar is broken. Architecture only.