AG-UI Protocol Research Benchmark

Comprehensive benchmark suite for the AG-UI (Agent-User Interaction) protocol across multiple AI agent frameworks. This is a research guide documenting protocol compliance, performance, and capabilities.

What is AG-UI?

AG-UI is an open, lightweight, event-based protocol for agent-user interaction created by CopilotKit. It enables framework-agnostic communication between AI agents and user interfaces through Server-Sent Events (SSE).

Research Goals

This benchmark provides rigorous testing and documentation of:

• Protocol Compliance - Which frameworks support which AG-UI events
• Framework Capabilities - What features each framework implements
• Performance Characteristics - Response times, throughput, tool calling
• HITL Implementation - How human-in-the-loop workflows work via the protocol

Documentation Structure

📖 Core Reference

• AGUI Spec Reference - Complete specification of all 26 AG-UI protocol events
• Framework Capabilities - Deep analysis of what each framework supports

🔬 Research Results

• HITL Validation Results - Testing human-in-the-loop implementation across frameworks
• Event Coverage Matrix - Which frameworks emit which events
• Framework Comparison - Performance and feature comparison
• Event Type Analysis - Event adoption statistics
• Benchmark Summary - Overall statistics and rankings

📚 Guides

• Report Generation - How to run benchmarks and auto-generate reports

Frameworks Tested

This benchmark covers 26 agent implementations across multiple frameworks:

Multi-Model Frameworks (Anthropic, OpenAI, Google):

• Agno, LangGraph, PydanticAI, LlamaIndex, Vercel AI SDK

Single-Model Frameworks:

• CrewAI (Anthropic), AG2 (OpenAI), Google ADK (Google)

Raw LLM APIs (baseline):

• Anthropic, OpenAI, Google Gemini

For detailed framework analysis, see Framework Capabilities.

Quick Start

Prerequisites

• Python 3.11+
• Node.js 18+ (for TypeScript agents)
• uv package manager
• API keys for OpenAI, Anthropic, and Google

Installation

# Clone the repository
git clone https://github.com/namastexlabs/agui-benchmark.git
cd agui-benchmark

# Install Python dependencies
uv sync

# Install TypeScript dependencies
cd ts-agents && npm install --legacy-peer-deps && cd ..

# Create .env file with your API keys
cat > .env << EOF
ANTHROPIC_API_KEY=your-anthropic-key
OPENAI_API_KEY=your-openai-key
GEMINI_API_KEY=your-gemini-key
EOF

Running Benchmarks

# Start all agents
./start_all.sh

# Wait for agents to initialize
sleep 10

# Run the benchmark
uv run python test_agents.py

# Stop all agents
./stop_all.sh

Benchmark Results

Test Summary

• Agents Tested: 26 implementations across 9 frameworks
• Tests Run: 702 total (27 per agent × 26 agents)
• Success Rate: 97.5% (686/702 passed)
• Models Tested: Claude, OpenAI GPT, Google Gemini, Cerebras Llama

Performance Rankings (Median Response Time)

Rank	Framework	Model	Time
🥇	Agno	Cerebras	262ms
🥈	LlamaIndex	Claude	1,728ms
🥉	PydanticAI	Claude	1,746ms
4️⃣	LangGraph	Claude	2,296ms
5️⃣	LlamaIndex	Gemini	2,768ms

Key Insights:

• Cerebras Llama 3.3-70b is dramatically faster than all others (262ms vs 1.7s+)
• PydanticAI offers best balance: fast (1.7s with Claude) + reliable (100% success)
• LangGraph has reliability issues (89% success rate on some models)
• Raw API wrappers are slower than framework abstractions (20-24s vs 1-8s)

Full Results

See Framework Comparison Matrix for complete rankings and metrics.

---

Key Findings

HITL Implementation

We validated that human-in-the-loop workflows can be fully implemented using the AG-UI protocol's existing TOOL_CALL_* events, without requiring special HITL-specific events. See HITL Validation Results.

Protocol Coverage

The AG-UI specification defines 26 events across 5 categories:

• Lifecycle: RUN_STARTED, RUN_FINISHED, RUN_ERROR
• Text Messages: TEXT_MESSAGE_START/CONTENT/END, THINKING_START/END, THINKING_TEXT_MESSAGE_*
• Tool Calls: TOOL_CALL_START/ARGS/END/RESULT
• State: STATE_SNAPSHOT, STATE_DELTA, MESSAGES_SNAPSHOT, ACTIVITY_SNAPSHOT, ACTIVITY_DELTA
• Custom: STEP_STARTED, STEP_FINISHED, RAW, CUSTOM

See AGUI Spec Reference for complete details.

Framework Comparison (Latest Benchmark)

Framework	Tests	Success	Median Time	Throughput	Tool Calls
agno-cerebras	27	100%	284ms	—	—
pydantic-anthropic	27	100%	1,771ms	13.4k c/s	20
agno-anthropic	27	100%	2,388ms	11.3k c/s	19
pydantic-gemini	27	100%	2,738ms	54k c/s	16
vercel-anthropic	27	100%	2,748ms	14.3k c/s	13
llamaindex-anthropic	27	89%	1,637ms	—	—
langgraph-anthropic	27	93%	2,295ms	6.6k c/s	—
openai-raw	27	100%	20,279ms	3.4k c/s	12

See full comparison: Framework Comparison Matrix

---

Benchmark Architecture

test_agents.py (Benchmark Runner)
    │
    ├─ Starts 26 agent implementations on various ports
    │
    └─ Runs 27 test scenarios per agent:
       • Simple prompt (no tools)
       • Tool calling (6 tools available)
       • Streaming performance
       • Error handling
       • State management
       └─ Collects timing, tool calls, response metrics
          Saves JSON results → generate_reports.py
                                     ↓
                          Auto-generates 4 markdown reports
                          in docs/reports/

Related Resources

• AG-UI Protocol - Official specification
• CopilotKit - AG-UI creators
• Framework repositories:
  • PydanticAI
  • LangGraph
  • Agno
  • LlamaIndex
  • Vercel AI SDK
  • CrewAI
  • AG2

License

MIT License - see LICENSE for details.