Skip to content

Latest commit

 

History

History
51 lines (39 loc) · 3.28 KB

large-language-model-performance-matrix.md

File metadata and controls

51 lines (39 loc) · 3.28 KB
mapped_pages applies_to
stack serverless
all
security
all

Large language model performance matrix

This page describes the performance of various large language models (LLMs) for different use cases in {{elastic-sec}}, based on our internal testing. To learn more about these use cases, refer to Attack discovery or AI Assistant.

::::{important} Excellent is the best rating, followed by Great, then by Good, and finally by Poor. Models rated Excellent or Great should produce quality results. Models rated Good or Poor are not recommended for that use case. ::::

Proprietary models [_proprietary_models]

Models from third-party LLM providers.

Feature - Assistant - General Assistant - {{esql}} generation Assistant - Alert questions Assistant - Knowledge retrieval Attack Discovery Automatic Migration
Model Claude 3.7: Sonnet Excellent Excellent Excellent Excellent Excellent Excellent
Claude 3.5: Sonnet v2 Excellent Excellent Excellent Excellent Great Excellent
Claude 3.5: Sonnet Excellent Excellent Excellent Excellent Excellent Excellent
Claude 3.5: Haiku Excellent Excellent Excellent Excellent Poor Poor
Claude 3: Haiku Excellent Excellent Excellent Excellent Poor Poor
GPT-4o Excellent Excellent Excellent Excellent Great Great
GPT-4o-mini Excellent Great Great Great Poor Good
GPT-4.1 Excellent Excellent Excellent Excellent Excellent Excellent
Gemini 1.5 Pro 002 Excellent Excellent Excellent Excellent Excellent Great
Gemini 1.5 Flash 002 Excellent Poor Good Excellent Poor Excellent
Gemini 2.0 Flash 001 Excellent Excellent Excellent Excellent Excellent Excellent
Gemini 2.5 Pro Excellent Excellent Excellent Excellent Excellent Excellent

Open-source models [_open_source_models]

Models you can deploy yourself.

| Feature | - | Assistant - General | Assistant - {{esql}} generation | Assistant - Alert questions | Assistant - Knowledge retrieval | Attack Discovery | Automatic Migration | --- | --- | --- | --- | --- | --- | --- | | Model | Mistral Nemo | Good | Good | Great | Good | Poor | Poor | | | Mistral-Small-3.1-24B-Instruct-2503 | Excellent | Poor | Excellent | Excellent | Good | N/A | | LLama 3.2 | Good | Poor | Good | Poor | Poor | Good | | | LLama 3.1 405b | Good | Great | Good | Good | Poor | Poor | | | LLama 3.1 70b | Good | Good | Poor | Poor | Poor | Good |