Skip to content

INTPYTHON-580 Add CrewAI Integration Tests #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .evergreen/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,20 @@ tasks:
- func: "setup remote atlas"
- func: "execute tests"

- name: test-crewai-tools-local
tags: [local]
commands:
- func: "fetch repo"
- func: "setup local atlas"
- func: "execute tests"

- name: test-crewai-tools-remote
tags: [remote]
commands:
- func: "fetch repo"
- func: "setup remote atlas"
- func: "execute tests"

- name: test-haystack-embeddings-local
tags: [local]
commands:
Expand Down Expand Up @@ -339,6 +353,16 @@ buildvariants:
- name: test-pymongo-voyageai-local
- name: test-pymongo-voyageai-remote

- name: test-crewai-tools-rhel
display_name: CrewAI-Tools Ubuntu
expansions:
DIR: crewai-tools
run_on:
- ubuntu2204-small
tasks:
- name: test-crewai-tools-local
- name: test-crewai-tools-remote

- name: test-haystack-embeddings-rhel
display_name: Haystack Embeddings RHEL
expansions:
Expand Down
3 changes: 3 additions & 0 deletions .evergreen/setup-remote.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ case $DIR in
pymongo-voyageai)
MONGODB_URI=$VOYAGEAI_MONGODB_URI
;;
crewai-tools)
MONGODB_URI=$CREWAI_TOOLS_URI
;;
*)
echo "Missing config in setup-remote.sh for DIR: $DIR"
exit 1
Expand Down
4 changes: 4 additions & 0 deletions crewai-tools/config.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
REPO_NAME=crewAI-tools
REPO_ORG=blink1073
DATABASE=crewai_test_db
REPO_BRANCH=INTPYTHON-580
26 changes: 26 additions & 0 deletions crewai-tools/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash

set -eu

# Get the MONGODB_URI and OPENAI_API_KEY.
SCRIPT_DIR=$(realpath "$(dirname ${BASH_SOURCE[0]})")
ROOT_DIR=$(dirname $SCRIPT_DIR)
. $ROOT_DIR/env.sh

. $ROOT_DIR/.evergreen/utils.sh

PYTHON_BINARY=$(find_python3)

$PYTHON_BINARY -m venv venv_pipeline
source venv_pipeline/bin/activate

pip install uv

uv sync --extra mongodb
uv run pytest -v tests/tools/test*mongodb*.py

export MONGODB_URI=$MONGODB_URI
export OPENAI_API_KEY=$OPENAI_API_KEY

mv ../test_mongodb_vector_search_tool.py .
uv run python test_mongodb_vector_search_tool.py
70 changes: 70 additions & 0 deletions crewai-tools/test_mongodb_vector_search_tool.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import os
from crewai import Agent
from crewai import Task
from crewai import Crew, Process
from crewai_tools import MongoDBVectorSearchTool
from langchain_community.document_loaders import PyPDFLoader
import time

# Pre-populate a collection and an index
print("Creating collection...")
conn_string = os.environ.get(
"MONGODB_URI", "mongodb://localhost:27017?directConnection=true"
)
database_name = "crewai_test_db"
collection_name = "vector_test"

tool = MongoDBVectorSearchTool(
connection_string=conn_string,
database_name=database_name,
collection_name=collection_name,
)
coll = tool._coll
coll.delete_many({})

# Insert documents from a pdf.
print("Loading documents...")
loader = PyPDFLoader("https://arxiv.org/pdf/2303.08774.pdf")
tool.add_texts([i.page_content for i in loader.load()])

print("Creating vector index...")
if not any([ix["name"] == "vector_index" for ix in coll.list_search_indexes()]):
tool.create_vector_search_index(dimensions=3072, auto_index_timeout=60)

# Create the MongoDB tool
print("Creating tool and waiting for index to be complete...")

# Wait for index to be complete.
n_docs = coll.count_documents({})
start = time.monotonic()
while time.monotonic() - start <= 60:
if len(tool._run(query="sandwich", limit=n_docs, oversampling_factor=1)) == n_docs:
break
else:
time.sleep(1)

# Assemble a crew
researcher = Agent(
role="AI Accuracy Researcher",
goal="Find and extract key information from a technical document",
backstory="You're specialized in analyzing technical content to extract insights and answers",
verbose=False,
tools=[tool],
)
research_task = Task(
description="Research information in a technical document",
expected_output="A summary of the accuracy of GPT-4",
agent=researcher,
)
crew = Crew(
agents=[researcher],
tasks=[research_task],
process=Process.sequential,
verbose=False,
)

# Get the result and assert something about the results
print("Running the crew...")
result = crew.kickoff()
assert "hallucinations" in result.raw.lower()
assert "GPT-4" in result.raw