ToolScout

An intelligent agent system for dynamic tool discovery and execution using MCP (Model Context Protocol) servers with vector similarity search and conversation history optimization.

Overview

ToolScout is a demonstration project built on the Microsoft Agent Framework that addresses two key challenges when building AI agents with extensive capabilities:

1. Managing Large Tool Sets: As agents gain access to dozens or hundreds of tools through MCP servers, loading all tools into the agent's context becomes impractical due to token limits and increased latency. This also leads to context distraction and confusion, where irrelevant tool descriptions interfere with the agent's decision-making (see How Contexts Fail).

2. Long-Running Conversations: Extended multi-turn interactions accumulate conversation history that includes not just user queries and responses, but also tool invocation details that clutter the context window and waste tokens.

ToolScout tackles these challenges through:

• Vector Similarity Search: Dynamically discovers and loads only the 3-5 most relevant tools for each query, reducing tool context by 80-90%% (50 → 3.8 avg tools)
• Intelligent History Filtering: Automatically removes tool invocation and result messages from conversation history, keeping only user queries and assistant responses
• Dynamic Context Management: Combines both optimizations to enable agents to work effectively with 20-30+ tools over extended conversations without exhausting the context window

This approach significantly reduces token usage (~47% savings) while maintaining full functionality across complex multi-turn conversations.

Benchmarks

Model: GPT-5-Mini.
Scenario: Scenario_3 (Total iterations 312)

Configuration	ToolScout + Rolling Window + Message Purge	ToolScout + NO Rolling Window + Message Purge	All Tools + NO Rolling Window + NO Message Purge
Max input tokens	11332	45672	83307
Avg input tokens	7344	23972	45166
Total input token	2291272	7479400	14091813
Avg tool per turn	3.8	3.8	50
Duration (min)	51	65	76

Performance gains

• Max Input Token Reduction: ~45% (83K → 46K max input tokens)
• Total Token Reduction: ~47% (14M → 7.5M total input tokens)
• Time Reduction: ~14% (76 min → 65 min execution time)

Background

This investigation was inspired by Anthropic's research on advanced tool use patterns and embedding-based tool search:

• Advanced Tool Use - Techniques for optimizing tool calling in agentic systems
• Tool Search with Embeddings - Cookbook example demonstrating vector-based tool discovery

Features

• 🔍 Vector-based tool discovery using Azure OpenAI embeddings
• 🧠 Smart conversation history with automatic tool message filtering
• 🔧 Multi-server MCP support with concurrent server connections
• 📊 Optimization metrics showing token reduction and tool selection stats
• 🎯 Configurable thresholds for similarity and history limits
• 🐛 Debug mode for conversation history inspection

Built With

• Microsoft Agent Framework - Agentic AI application framework
• Azure OpenAI - LLM and embedding models
• FastMCP - Model Context Protocol server implementation
• uv - Fast Python package manager

Architecture

┌─────────────────┐
│  User Query     │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────┐
│  Vector Similarity Search   │  ← Embeddings
│  (Tool Registry)            │
└────────┬────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│  Selected Tools (3-5)       │
└────────┬────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│  Agent Execution            │  ← Filtered History
│  (Azure OpenAI Responses)   │
└────────┬────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│  Result + History Update    │
└─────────────────────────────┘

Setup

Prerequisites

• Python 3.13+
• uv package manager
• Azure OpenAI access with:
  • A chat model deployment (e.g., gpt-4o, gpt-4o-mini)
  • An embedding model deployment (e.g., text-embedding-3-small)
• Azure CLI for authentication

Installation

1. Clone the repository

git clone https://github.com/corradocavalli/toolscout.git
cd toolscout

2. Install dependencies

uv sync

3. Configure environment

cp .env.sample .env
# Edit .env with your Azure OpenAI details

4. Authenticate with Azure

az login

Configuration

Edit .env with your settings:

# Azure OpenAI Configuration
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AGENT_MODEL_DEPLOYMENT_NAME=gpt-4o-mini
AZURE_EMBEDDING_MODEL_DEPLOYMENT_NAME=text-embedding-3-small

# Tool Discovery Settings
TOP_K=3                      # Number of tools to select per query
SIMILARITY_THRESHOLD=0.3     # Minimum similarity score (0-1)

# History Management
MAX_HISTORY_MESSAGES=50      # Maximum messages in conversation history

Usage

Start MCP Servers

In a separate terminal:

cd mcp
./start_servers.sh

This starts 3 demo MCP servers:

• server1 (port 8000): User management, email, weather
• server2 (port 9000): Library management
• server3 (port 10000): Stock trading

Run Interactive Mode

uv run main.py

Run with Scenario

uv run main.py --scenario scenario_1

Debug Mode

View conversation history being passed to the agent:

uv run main.py --scenario scenario_1 --debug

Disable Optimization

Compare performance with all tools loaded:

uv run main.py --scenario scenario_1 --optimize false

Command Line Options

uv run main.py [OPTIONS]

Options:
  --scenario TEXT        Run a scenario file (e.g., scenario_1)
  --optimize BOOL        Enable tool selection optimization (default: true)
  --debug               Show conversation history in output
  --help                Show this message and exit

Project Structure

toolscout/
├── main.py                    # Main application entry point
├── tool_registry.py           # Vector search and tool indexing
├── tool_manager.py            # Tool discovery and loading
├── conversation_store.py      # Custom history with message filtering and lenght limitation
├── utils.py                   # Helper functions for display
├── mcp/                       # Demo MCP servers
│   ├── server1.py            # General tools
│   ├── server2.py            # Library management
│   ├── server3.py            # Stock trading
│   └── start_servers.sh      # Launch all servers
├── scenarios/                 # Test scenarios
│   ├── scenario_1.txt        # Simple multi-turn
│   ├── scenario_2.txt        # Complex interactions
│   └── scenario_3.txt        # Extended scenario
└── pyproject.toml            # Project dependencies

Key Concepts

Tool Discovery

ToolScout uses cosine similarity between query embeddings and tool description embeddings to select relevant tools:

1. At startup, all tool descriptions are embedded and indexed
2. For each query, compute query embedding
3. Find top-K tools with similarity above threshold
4. Load only selected tools into agent context

History Optimization

The custom LimitedHistoryChatMessageStore provides two modes of operation:

Optimization Mode (default):

• Filters out tool invocation messages
• Filters out tool result messages
• Keeps only user queries and assistant responses
• Trims to last N messages
• Reduces context size by ~50% in typical multi-tool scenarios

Full History Mode (--optimize false):

• Retains all conversation history including tool invocations and results
• No filtering or trimming applied
• Consumes significantly more tokens and context space
• Useful for comparing performance or debugging tool interactions

When to Use This Approach

This optimization strategy becomes beneficial when working with 20-30+ tools. Below this threshold, the overhead of vector similarity search and tool selection may outweigh the token savings from reduced context. For smaller tool sets (< 20 tools), loading all tools directly into the agent context is often simpler and more efficient.

Limitations and Future Improvements

Tool Selection Accuracy

Since tool selection relies on semantic similarity, some tools may not be correctly identified for certain queries. While adjusting the SIMILARITY_THRESHOLD parameter can help, a more effective solution involves a two-stage approach:

Stage 1: Action Extraction

Use an agent to extract required actions from the user's request before querying the tool registry:

User: "If it is raining, book a taxi"
  ↓
Agent extracts actions: "get_weather, book_taxi"
  ↓
Registry searches: ["get_weather", "book_transport", "fetch_weather", "book_driver", "book_taxi"]

Stage 2: Tool Refinement

Further refine the selection by having the agent choose from the candidate tools based on the original request:

Input: 
  - Original query: "If it is raining, book a taxi"
  - Candidates: ["get_weather", "book_transport", "fetch_weather", "book_driver", "book_taxi"]
  ↓
Agent selects: ["get_weather", "book_taxi"]

Performance Considerations

The primary challenge with this enhanced approach is the additional latency introduced by the scouting phase:

• Cloud LLM: Using a small cloud model for scouting approximately doubles the total response time
• Local LLM: Using a local model like TinyLlama-1.1B-Chat-v1.0 reduces the overhead to 15-20%, which may be acceptable when maximum tool selection accuracy is required

Recommendation: The current vector similarity approach provides a good balance between accuracy and latency for most use cases. Consider the two-stage approach only when tool selection accuracy is critical and the additional latency is acceptable.

Alternative Embedding Models

The current implementation uses Azure OpenAI's embedding models, which require API calls and incur costs. A potential improvement would be to evaluate performance using lightweight local embedding models such as all-MiniLM-L6-v2, which could:

• Eliminate embedding API costs and latency
• Enable fully offline operation
• Provide faster tool indexing and search

This would require benchmarking to ensure embedding quality remains sufficient for accurate tool selection.

Conversation Summarization

Beyond message filtering and length limits, another optimization for very long conversations would be to implement LLM-based history summarization. When the conversation exceeds a certain threshold (e.g., 100+ messages), an LLM could:

• Summarize older portions of the conversation into a concise context
• Preserve recent messages in full detail
• Replace the original messages with the summary to save tokens

This approach would enable maintaining context from extended sessions while keeping token usage manageable. The summarization could be triggered automatically at configurable intervals or message counts.

Contributing

This is a demonstration project. Feel free to fork and adapt for your needs!