notebooks/ingestion_api_usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a5726313-f5ab-48fb-b747-c790ebaafe48",
   "metadata": {},
   "source": [
    "# Ingestion API Usage\n",
    "\n",
    "- Replace `BASE_URL` with the actual server URL where the API is hosted.\n",
    "- Replace the directory path (`../data/dataset`) with the correct location of your dataset.\n",
    "- Ensure to follow the steps in the README under [Data Ingestion](../docs/quickstart.md#data-ingestion) to retrieve the dataset, including installing and pulling using Git LFS.\n",
    "- Replace file paths (`example_document.pdf`) with appropriate files for the Upload and Delete endpoints.\n",
    "- Modify the `collection_name` accordingly for the Upload and Delete\n",
    "- Ensure the server is running before executing the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e58505d5-5436-449a-b316-b943a1a57797",
   "metadata": {},
   "source": [
    "#### 1. Install Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78246c4e-040d-4e06-8ed3-edb88ca0c280",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a494be7-ffee-4dfb-968f-c5300f6ba0a2",
   "metadata": {},
   "source": [
    "#### 2. Setup Base Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2807ea21-f9b8-408b-b2ee-318bef308d1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "from typing import Dict, Any\n",
    "\n",
    "IPADDRESS = \"localhost\" #Replace this with the correct IP address\n",
    "RAG_PORT = \"8081\"\n",
    "BASE_URL = f\"http://{IPADDRESS}:{RAG_PORT}\"  # Replace with your server URL\n",
    "\n",
    "def print_response(response: requests.Response):\n",
    "    \"\"\"Helper to print API response.\"\"\"\n",
    "    print(f\"Status Code: {response.status_code}\")\n",
    "    try:\n",
    "        print(json.dumps(response.json(), indent=2))\n",
    "    except json.JSONDecodeError:\n",
    "        print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "677f85b3-767b-4e8f-82da-9d5c7069d609",
   "metadata": {},
   "source": [
    "#### 3. Health Check Endpoint\n",
    "\n",
    "**Purpose:**\n",
    "This endpoint performs a health check on the server. It returns a 200 status code if the server is operational."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f32e53c4-909b-4b04-975f-c598f33bd797",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /health\n",
    "url = f\"{BASE_URL}/health\"\n",
    "response = requests.get(url)\n",
    "print_response(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2334abfc-5832-4e39-8793-818b5265aa1d",
   "metadata": {},
   "source": [
    "#### 4. Upload Document Endpoint\n",
    "\n",
    "**Purpose:**\n",
    "This endpoint uploads a document to the vector store. You can specify the collection name where the document should be stored. To speed up the ingestion process, the code is parallelized using `concurrent.futures`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8070f61-cae8-4769-bb54-409868ec3fd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the Dataset\n",
    "!unzip ../data/dataset.zip -d ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff5fabd6-cd71-4643-9f71-45ed3ee416e1",
   "metadata": {},
   "source": [
    "##### Upload multiple files with `concurrent.futures`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8709ab2e-c1de-42d4-96b2-eb104f8bd6cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "\n",
    "def upload_file(file_path: str, collection_name: str):\n",
    "    url = f\"{BASE_URL}/documents\"\n",
    "    files = {\"file\": open(file_path, \"rb\")}\n",
    "    params = {\"collection_name\": collection_name}\n",
    "    response = requests.post(url, files=files, params=params)\n",
    "    print(f\"Uploading {os.path.basename(file_path)}...\")\n",
    "    print_response(response)\n",
    "\n",
    "directory_path = \"../data/dataset\"  # Replace with your directory path\n",
    "collection_name = \"nvidia_blogs\"\n",
    "\n",
    "file_paths = [os.path.join(directory_path, f) for f in os.listdir(directory_path) if os.path.isfile(os.path.join(directory_path, f))]\n",
    "\n",
    "with ThreadPoolExecutor() as executor:\n",
    "    executor.map(lambda file: upload_file(file, collection_name), file_paths)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3191cd7f-02d1-45c6-90a7-f3ca86c4046a",
   "metadata": {},
   "source": [
    "#### 5. Get Documents Endpoint\n",
    "\n",
    "**Purpose:**\n",
    "This endpoint retrieves a list of documents ingested into the vector store for a specified collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a250bcc6-2137-40d2-95ff-6202faff4fd1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# GET /documents\n",
    "url = f\"{BASE_URL}/documents\"\n",
    "params = {\"collection_name\": \"nvidia_blogs\"}\n",
    "\n",
    "response = requests.get(url, params=params)\n",
    "print_response(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be6b9e3c-2092-4d6b-9ff7-64d33730337b",
   "metadata": {},
   "source": [
    "#### 6. Delete Document Endpoint\n",
    "\n",
    "**Purpose:**\n",
    "This endpoint deletes a specified document from the vector store. The document is identified by its filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ae5ebcd-f741-491c-a797-ee4fd15c9060",
   "metadata": {},
   "outputs": [],
   "source": [
    "# DELETE /documents\n",
    "url = f\"{BASE_URL}/documents\"\n",
    "params = {\n",
    "    \"filename\": \"example_document.pdf\",  # Replace with the file you want to delete\n",
    "    \"collection_name\": \"nvidia_blogs\"\n",
    "}\n",
    "\n",
    "response = requests.delete(url, params=params)\n",
    "print_response(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec9bfd5-b943-4eed-874b-9067fdfe06ca",
   "metadata": {},
   "source": [
    "#### 7. Get Collections Endpoint\n",
    "\n",
    "**Purpose:**\n",
    "This endpoint retrieves a list of all collection names available on the server. Collections are used to organize documents in the vector store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6114bb1-f0f0-4444-859f-d4dc66fa9579",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET /collections\n",
    "url = f\"{BASE_URL}/collections\"\n",
    "response = requests.get(url)\n",
    "print_response(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "foundational_rag",
   "language": "python",
   "name": ".conda"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}