Merge pull request #10 from jasonacox/v0.15.0

Document Management - v0.15.0

Author: jasonacox

Diff for: .gitignore

@@ -175,3 +175,4 @@ agents/r
 chatbot/tlocal
 chatbot/tt
 lab/t
+chatbot/uploads/?*.*

Diff for: RELEASE.md

@@ -1,5 +1,10 @@
 # Releases
 
+## 0.15.0 - Document Manager
+
+* Chatbot: Using Document class for RAG functions.
+* DocMan: New web based UI for managing documents in the Weaviate vector database. Allows user to upload and embed content from URLs and uploaded files. Provides optional chunking and management of embedded documents.
+
 ## 0.14.13 - TPS Calculation
 
 * Chatbot: Fix a bug that was counting null tokens.

Diff for: chatbot/Dockerfile

@@ -27,11 +27,13 @@ WORKDIR /app
 
 # Install depencencies - Weaviate Vector Search
 RUN pip install fastapi uvicorn python-socketio jinja2 openai bs4 pypdf requests lxml aiohttp
-RUN pip install weaviate-client
+RUN pip install weaviate-client pdfreader pypandoc
 
 # Copy local files into container
 COPY server.py /app/server.py
 COPY templates /app/templates
+COPY documents.py /app/documents.py
+COPY version.py /app/version.py
 
 # Network
 EXPOSE $PORT

Diff for: chatbot/Dockerfile-docman

@@ -0,0 +1,35 @@
+# Dockerfile for docman - TinyLLM Document Manager
+#
+#   Author: Jason A. Cox
+#   22 Sept 2024
+#   https://github.com/jasonacox/TinyLLM
+
+# Use a base image
+FROM python:3.10-slim
+
+# Setting build related env vars
+ENV MAX_CHUNK_SIZE 1024
+ENV UPLOAD_FOLDER uploads
+ENV HOST localhost
+ENV PORT 5001
+ENV COLLECTIONS_ADMIN true
+
+# Set the working directory
+WORKDIR /app
+
+# Install depencencies
+RUN pip install fastapi uvicorn jinja2 bs4 pypdf requests lxml aiohttp
+RUN pip install weaviate-client pdfreader pypandoc
+RUN pip install python-multipart
+
+# Copy local files into container
+COPY docman.py /app/docman.py
+COPY documents.py /app/documents.py
+COPY version.py /app/version.py
+COPY docman /app/docman
+
+# Network
+EXPOSE $PORT
+
+# Run the server
+CMD uvicorn docman:app --host 0.0.0.0 --port $PORT

Diff for: chatbot/README.md

@@ -84,3 +84,42 @@ The `/news` command will fetch the latest news and have the LLM summarize the to
 * A Hacker’s Guide to Language Models - Jeremy Howard [[link](https://www.youtube.com/watch?v=jkrNMKz9pWU&ab_channel=JeremyHoward)]
 
 You are an autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers, and are brilliant at reasoning. If you think there might not be a correct answer, you say so. Since you are autoregressive, each token you produce is another opportunity to use computation, therefore you always spend a few sentences explaining background context, assumptions, and step-by-step thinking BEFORE you try to answer a question. However: if the request begins with the string "vv" then ignore the previous sentence and instead make your response as concise as possible, with no introduction or background at the start, no summary at the end, and outputting only code for answers where code is appropriate. Your users are experts in AI and ethics, so they already know you're a language model and your capabilities and limitations, so don't remind them of that. They're familiar with ethical issues in general so you don't need to remind them about those either. Don't be verbose in your answers, but do provide details and examples where it might help the explanation. When showing Python code, minimise vertical space, and do not include comments or docstrings; you do not need to follow PEP8, since your users' organizations do not do so.
+
+
+
+# TinyLLM Document Manager (Weaviate)
+
+The document manager allows you to manage the collections and documents in the Weaviate vector database. It provides an easy way for you to upload and ingest the content from files or URL. It performs simple chunking (if requested). The simple UI let's you navigate through the collections and documents.
+
+### Environment Variables
+
+- MAX_CHUNK_SIZE: Maximum size of a chunk in bytes (default 1024)
+- UPLOAD_FOLDER: Folder where uploaded files are stored (default uploads)
+- HOST: Weaviate host (default localhost)
+- COLLECTIONS: Comma separated list of collections allowed (default all)
+- PORT: Port for the web server (default 8000)
+- COLLECTIONS_ADMIN: Allow users to create and delete collections (default True)
+
+### Docker Setup
+
+```bash
+docker run \
+    -d \
+    -p 5001:5001 \
+    -e HOST="localhost" \
+    -e PORT="5001" \
+    -e MAX_CHUNK_SIZE="1024" \
+    -e UPLOAD_FOLDER="uploads" \
+    -e COLLECTIONS_ADMIN="true" \
+    --name docman \
+    --restart unless-stopped \
+    jasonacox/docman
+```
+Note - You can restrict collections by providing the environmental variable `COLLECTIONS` to a string of comma separated collection names.
+
+### Screenshots
+
+<img width="1035" alt="image" src="https://github.com/user-attachments/assets/544c75d4-a1a3-4c32-a95f-7f12ff11a450">
+
+<img width="1035" alt="image" src="https://github.com/user-attachments/assets/4b15ef87-8f25-4d29-9214-801a326b406f">
+