From 79a20743f9d51f78f64eddfd884903d2c146634f Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 23 Apr 2025 14:08:35 -0700 Subject: [PATCH 1/2] updates LLM performance matrix --- ...large-language-model-performance-matrix.md | 32 ++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/solutions/security/ai/large-language-model-performance-matrix.md b/solutions/security/ai/large-language-model-performance-matrix.md index c8f65804e..d053ae88a 100644 --- a/solutions/security/ai/large-language-model-performance-matrix.md +++ b/solutions/security/ai/large-language-model-performance-matrix.md @@ -24,17 +24,18 @@ Models from third-party LLM providers. | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | | --- | --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Claude 3: Opus** | Excellent | Excellent | Excellent | Good | Great | Good -| | **Claude 3.7: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude 3.5: Sonnet v2** | Excellent | Excellent | Excellent | Excellent | Great | Excellent -| | **Claude 3.5: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude 3.5: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | Poor -| | **Claude 3: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | Poor -| | **GPT-4o** | Excellent | Excellent | Excellent | Excellent | Great | Great -| | **GPT-4o-mini** | Excellent | Great | Great | Great | Poor | Good -| | **Gemini 1.5 Pro 002** | Excellent | Excellent | Excellent | Excellent | Excellent | Great -| | **Gemini 1.5 Flash 002** | Excellent | Poor | Good | Excellent | Poor | Excellent -| | **Gemini 2.0 Flash 001** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| **Model** | **Claude 3.7: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude 3.5: Sonnet v2** | Excellent | Excellent | Excellent | Excellent | Great | Excellent +| | **Claude 3.5: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude 3.5: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | Poor +| | **Claude 3: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | Poor +| | **GPT-4o** | Excellent | Excellent | Excellent | Excellent | Great | Great +| | **GPT-4o-mini** | Excellent | Great | Great | Great | Poor | Good +| | **GPT-4.1** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Gemini 1.5 Pro 002** | Excellent | Excellent | Excellent | Excellent | Excellent | Great +| | **Gemini 1.5 Flash 002** | Excellent | Poor | Good | Excellent | Poor | Excellent +| | **Gemini 2.0 Flash 001** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Gemini 2.5 Pro** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent ## Open-source models [_open_source_models] @@ -43,7 +44,8 @@ Models you can [deploy yourself](/solutions/security/ai/connect-to-own-local-llm | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Mistral Nemo** | Good | Good | Great | Good | Poor | Poor | -| | **LLama 3.2** | Good | Poor | Good | Poor | Poor | Good | -| | **LLama 3.1 405b** | Good | Great | Good | Good | Poor | Poor | -| | **LLama 3.1 70b** | Good | Good | Poor | Poor | Poor | Good | \ No newline at end of file +| **Model** | **Mistral Nemo** | Good | Good | Great | Good | Poor | Poor | +| | **Mistral-Small-3.1-24B-Instruct-2503** | Excellent | Poor | Excellent | Excellent | Good | Good +| | **LLama 3.2** | Good | Poor | Good | Poor | Poor | Good | +| | **LLama 3.1 405b** | Good | Great | Good | Good | Poor | Poor | +| | **LLama 3.1 70b** | Good | Good | Poor | Poor | Poor | Good | \ No newline at end of file From d7fb0af32e1eac039f93a2a0301bfae557f2498c Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Thu, 24 Apr 2025 09:09:11 -0700 Subject: [PATCH 2/2] Update large-language-model-performance-matrix.md --- .../security/ai/large-language-model-performance-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/security/ai/large-language-model-performance-matrix.md b/solutions/security/ai/large-language-model-performance-matrix.md index d053ae88a..dde574b63 100644 --- a/solutions/security/ai/large-language-model-performance-matrix.md +++ b/solutions/security/ai/large-language-model-performance-matrix.md @@ -45,7 +45,7 @@ Models you can [deploy yourself](/solutions/security/ai/connect-to-own-local-llm | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | --- | --- | --- | --- | --- | --- | --- | | **Model** | **Mistral Nemo** | Good | Good | Great | Good | Poor | Poor | -| | **Mistral-Small-3.1-24B-Instruct-2503** | Excellent | Poor | Excellent | Excellent | Good | Good +| | **Mistral-Small-3.1-24B-Instruct-2503** | Excellent | Poor | Excellent | Excellent | Good | N/A | | **LLama 3.2** | Good | Poor | Good | Poor | Poor | Good | | | **LLama 3.1 405b** | Good | Great | Good | Good | Poor | Poor | | | **LLama 3.1 70b** | Good | Good | Poor | Poor | Poor | Good | \ No newline at end of file