You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: experimental/oran-chatbot-multimodal/README.md
+13-41Lines changed: 13 additions & 41 deletions
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ This repository is designed to make it extremely easy to set up your own retriev
15
15
16
16
Before running the pipeline, please ensure that you have the following prerequisites:
17
17
- Python version 3.10.12
18
-
-<TODO: verify if GPU is necessary> GPU-enabled machine
18
+
- GPU-enabled machine
19
19
- NVIDIA API Key
20
20
21
21
If you do not have a NVIDIA API Key, please follow the steps 1-4 mentioned [here](https://github.com/NVIDIA/GenerativeAIExamples/blob/main/docs/rag/aiplayground.md#prepare-the-environment) to get your key.
@@ -31,53 +31,32 @@ The following describes how you can have this chatbot up-and-running in less tha
31
31
### Step 2. Clone this repository to a Linux machine
32
32
33
33
```
34
-
git clone <TODO: PUBLIC REPO LINK>
35
-
```
36
-
Once you have tested this on a local machine, you can [request a VM from IT](https://itss.nvidia.com/) and deploy it there so it can run all the time.
37
-
38
-
Currently, we are using the internal gitlab repository, which can be downloaded using following command:
git clone https://github.com/NVIDIA/GenerativeAIExamples.git && cd GenerativeAIExamples/experimental/oran-chatbot-multimodal
41
35
```
42
36
43
-
44
37
### Step 3. Install the requirements
45
38
Goto the root of the oran_chatbot repository, and install the requirements in the virtual environment.
46
39
```
47
40
pip install -r requirements.txt
48
41
```
49
42
50
-
<!-- #5. Follow instructions available [here](https://milvus.io/docs/install_standalone-docker.md#Install-Milvus-standalone-using-Docker-Compose). -->
51
-
52
43
### Step 4. Setup PowerPoint parsing
53
44
If you want to use the PowerPoint parsing feature, you will need LibreOffice. On Ubuntu Linux systems, use the following command to install it.
54
45
```
55
46
sudo apt install libreoffice
56
47
```
57
48
58
49
### Step 5. (Optional) Setup Feedback feature
59
-
If you would like to use the "Feedback" feature, you will need a service account for Google Sheets.
60
-
Save your service account credentials file as `service.json` inside the `oran_chatbot` folder.
50
+
If you would like to use the "Feedback" feature, you will need a service account for Google Sheets.
51
+
Save your service account credentials file as `service.json` inside the `oran-chatbot-multimodel` folder.
52
+
53
+
### Step 6. Setup your NVIDIA API key
61
54
62
-
### Step 6. Setup your NVIDIA API key
63
-
<!-- <TODO: Check if these API keys are being used internally.> -->
64
-
<!-- We needFollow the steps 1-4 mentioned [here](https://github.com/NVIDIA/GenerativeAIExamples/blob/main/docs/rag/aiplayground.md#prepare-the-environment) to get your NVIDIA API key. -->
65
55
To access NeMo services and language model, we will export the NVIDIA API key to the environment using the following command:
66
-
56
+
67
57
```
68
58
export NVIDIA_API_KEY="nvapi-b**************"
69
59
```
70
-
<!-- Now, we will generate an NVIDIA GPU Cloud (NGC) API Key for the NeMo service. You can request/get access for yourself or your team by reading instructions available [here](https://nvidia.sharepoint.com/sites/NemoService/SitePages/Team.aspx). For using this repository, you will need to export the NGC API KEY as an environment variable as shown below.
71
-
72
-
```
73
-
export NGC_API_KEY="Y2c2Nm**************"
74
-
``` -->
75
-
76
-
<!-- ### Step 7: Download ORAN documents
77
-
<!-- <TODO: Are we using google drive for this? update this section> -->
78
-
<!-- Download the official O-RAN documentations from [here](https://www.o-ran.org/specifications). These documents should be placed inside the `vectorstore/oran` folder. -->
79
-
80
-
<!-- Download these documents in the `vectorstore/oran` file. -->
81
60
82
61
### Step 7. Run the chatbot using streamlit
83
62
Go to the `oran_chatbot` folder to run the O-RAN RAG chatbot using streamlit.
@@ -96,20 +75,17 @@ Save your service account credentials file as `service.json` inside the `oran_ch
96
75
ssh -L PORT:IP_ADDR:PORT localhost
97
76
```
98
77
99
-
<!-- To setup the vector database, use the `Create vector DB` button under the `Evaluation Metrics` tab.
100
-
You can click on `Browse Files` and select your knowledge source. After selecting, click on `Upload!` button to complete the ingestion process. -->
101
-
102
78
### Step 8. Adding documents and creating vector database
103
79
104
80
Before you can run the chatbot, we need to upload our documents to the bot and create a vector database.
105
81
106
-
The official O-RAN documentations are available [here](https://www.o-ran.org/specifications).
82
+
The official O-RAN documentations are available [here](https://www.o-ran.org/specifications).
107
83
108
84
To upload these documents to O-RAN bot, you can use either of the following method:
109
85
- Using streamlit UI, you can navigate to the `Knowledge Base` tab. Select the files using the `Browse Files` button and click on `Upload!` button.
110
86
- Alternatively, you can upload the documents through the backend using following steps:
111
87
112
-
- Make a directory named `oran` inside the vectorstore folder using command
88
+
- Make a directory named `oran` inside the vectorstore folder using command
113
89
```mkdir vectorstore/oran```
114
90
- Move your O-RAN documents inside this folder using `cp /path/to/oran_document /vectorstore/oran`, or `cp /path/to/oran_folder/* /vectorstore/oran`. If you are running the bot through a remote server, you can use `scp` to upload the documents from your local machine to the server.
115
91
@@ -119,22 +95,20 @@ Save your service account credentials file as `service.json` inside the `oran_ch
119
95
120
96
#### Add or delete documents from knowledge base
121
97
122
-
All the documents in the knowledge base are shown in the drop-down menu of `Knowledge Base` tab.
123
-
98
+
All the documents in the knowledge base are shown in the drop-down menu of `Knowledge Base` tab.
99
+
124
100
If you want to add more documents to your existing knowledge base, navigate to the `Knowledge Base` tab, select the files using the `Browse Files` button and click on `Upload!` button. Finally, click on the `Re-train Multimodal ORAN Assistant' button to complete the ingestion process.
125
101
126
102
You can delete files from your knowledge base using the `Delete File` button. The default password to delete files is `avfleetengineering`, and can be changed using `1_Knowledge_Base.py` file inside `pages` folder.
127
-
103
+
128
104
**Warning:** The `Create vector DB` button overwrites any existing vector database, and creates a new vector database using documents available in the `vectorstore/oran` folder. To upload additional documents to your database, always use the `Re-train Multimodal ORAN Assistant` button.
129
105
130
106
131
107
### Step 9. Evaluating the chatbot
132
108
133
109
Once our chatbot is running, we can use the evaluation scripts to test its performance.
134
-
135
-
The `Run synthetic data generation` button in `Evaluation Metrics` tab generates a test dataset using the files in knowledge base. Once the synthetic test set is generated, click on the `Generate evaluation metrics` button to see a comprehensive evaluation report for the RAG performance.
136
-
<!-- <TODO: fill this. Mention about regenerating evals pkl file> -->
137
110
111
+
The `Run synthetic data generation` button in `Evaluation Metrics` tab generates a test dataset using the files in knowledge base. Once the synthetic test set is generated, click on the `Generate evaluation metrics` button to see a comprehensive evaluation report for the RAG performance.
138
112
139
113
## Architecture Diagram
140
114
@@ -165,7 +139,6 @@ This uses the NVIDIA NeMo Retriever model through NVIDIA AI Playground. This is
165
139
The vector database being used here is FAISS, a CPU-based embedding database. It can easily be swapped out for numerous other options like ChromaDB, Pinecone, Milvus and others. Some of the options are listed on the [LangChain docs here](https://python.langchain.com/docs/integrations/vectorstores/).
166
140
167
141
### Prompt Augmentation
168
-
<!-- <TODO: We can update this section> -->
169
142
Depending on the backend and model, you may need to modify the way in which you format your prompt and chat conversations to interact with the model. The current design considers each query independently. However, if you put the input as a set of user/assistant/user interactions, you can combine multi-turn conversations. This may also require periodic summarization of past context to ensure the chat does not exceed the context length of the model.
170
143
171
144
### Backend
@@ -174,7 +147,6 @@ Depending on the backend and model, you may need to modify the way in which you
174
147
175
148
## Pipeline Enhancement Opportunities:
176
149
177
-
<!-- <TODO: Do we want to keep this section? maybe reframe it better? maybe something like 'now that we have implemented this, you can try extending it's capabilities by xyz so that it looks more like learning opportunity for users rather than incomplete implementation> -->
178
150
### Multimodal Parsing:
179
151
Upgrade the current PyMuPDF-based PDF parsing with a more sophisticated parser for improved extraction of images and tables. Employ a high-quality Multimodal Language Model (MLLM) to enhance image descriptions and implement structured data analysis techniques like text2sql or text2pandas for efficient table summarization.
0 commit comments