Merge branch 'main' of https://github.com/powerli2002/CodeFuse-ModelCache into fix-mysql-delete

Author: powerli2002

.gitignore

@@ -135,5 +135,11 @@ dmypy.json
 /embedding_npy
 /flask_server
 *.bin
+**/maya_embedding_service
+
+*.ini
+
+**/multicache_serving.py
 **/modelcache_serving.py
-*.ini
+
+**/model/

README.md

@@ -40,20 +40,26 @@ The project's startup scripts are divided into flask4modelcache.py and flask4mod
 - Python version: 3.8 and above
 - Package Installation
 ```shell
-pip install requirements.txt 
+pip install -r requirements.txt 
 ```
 ### Service Startup
 #### Demo Service Startup
 1. Download the embedding model bin file from the following address: [https://huggingface.co/shibing624/text2vec-base-chinese/tree/main](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main). Place the downloaded bin file in the model/text2vec-base-chinese folder.
 2. Start the backend service using the flask4modelcache_dome.py script.
+```shell
+cd CodeFuse-ModelCache
+```
+```shell
+python flask4modelcache_demo.py
+```
 
 #### Normal Service Startup
 Before starting the service, the following environment configurations should be performed:
-1. Install the relational database MySQL and import the SQL file to create the data tables. The SQL file can be found at: reference_doc/create_table.sql
+1. Install the relational database MySQL and import the SQL file to create the data tables. The SQL file can be found at: ```reference_doc/create_table.sql```
 2. Install the vector database Milvus.
 3. Add the database access information to the configuration files: 
-   1. modelcache/config/milvus_config.ini 
-   2. modelcache/config/mysql_config.ini
+   1. ```modelcache/config/milvus_config.ini ```
+   2. ```modelcache/config/mysql_config.ini```
 4. Download the embedding model bin file from the following address: [https://huggingface.co/shibing624/text2vec-base-chinese/tree/main](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main). Place the downloaded bin file in the model/text2vec-base-chinese folder.
 5. Start the backend service using the flask4modelcache.py script.
 ## Service-Access
@@ -245,11 +251,23 @@ In ModelCache, we adopted the main idea of GPTCache,  includes core modules: ada
    - Asynchronous log write-back capability for data analysis and statistics. 
    - Added model field and data statistics field for feature expansion.
 
-Future Features Under Development: 
+## Todo List
+### Adapter
+- [ ] Register adapter for Milvus：Based on the "model" parameter in the scope, initialize the corresponding Collection and perform the load operation.
+### Embedding model&inference
+- [ ] Inference Optimization: Optimizing the speed of embedding inference, compatible with inference engines such as FasterTransformer, TurboTransformers, and ByteTransformer.
+- [ ] Compatibility with Hugging Face models and ModelScope models, offering more methods for model loading.
+### Scalar Storage
+- [ ] Support MongoDB
+- [ ] Support ElasticSearch
+### Vector Storage
+- [ ] Adapts Faiss storage in multimodal scenarios.
+### Rank能力
+- [ ] Add ranking model to refine the order of data after embedding recall.
+### Service
+- [ ] Supports FastAPI.
+- [ ] Add visual interface to offer a more direct user experience.
 
-- [ ] Data isolation based on hyperparameters. 
-- [ ] System prompt partitioning storage capability to enhance accuracy and efficiency of similarity matching.
-- [ ] More versatile embedding models and similarity evaluation algorithms.
 ## Acknowledgements
 This project has referenced the following open-source projects. We would like to express our gratitude to the projects and their developers for their contributions and research.<br />[GPTCache](https://github.com/zilliztech/GPTCache)
 

README_CN.md

@@ -37,24 +37,29 @@ Codefuse-ModelCache 是一个开源的大模型语义缓存系统，通过缓存
 - flask4modelcache_demo.py 为快速测试服务，内嵌了sqlite和faiss，用户无需关心数据库相关事宜。
 - flask4modelcache.py 为正常服务，需用户具备mysql和milvus等数据库服务。
 ### 环境依赖
-
 - python版本: 3.8及以上
 - 依赖包安装：
 ```shell
-pip install requirements.txt 
+pip install -r requirements.txt 
 ```
 ### 服务启动
 #### Demo服务启动
 - 离线模型bin文件下载， 参考地址：[https://huggingface.co/shibing624/text2vec-base-chinese/tree/main](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main)，并将下载的bin文件，放到 model/text2vec-base-chinese 文件夹中。
-- 执行flask4modelcache_demo.py脚本即可启动。
+- 执行flask4modelcache_demo.py启动服务。
+```shell
+cd CodeFuse-ModelCache
+```
+```shell
+python flask4modelcache_demo.py
+```
 
 #### 正常服务启动
 在启动服务前，应该进行如下环境配置：
-1. 安装关系数据库 mysql， 导入sql创建数据表，sql文件: reference_doc/create_table.sql
+1. 安装关系数据库 mysql， 导入sql创建数据表，sql文件:```reference_doc/create_table.sql```
 2. 安装向量数据库milvus
 3. 在配置文件中添加数据库访问信息，配置文件为：
-   1. modelcache/config/milvus_config.ini
-   2. modelcache/config/mysql_config.ini
+   1. ```modelcache/config/milvus_config.ini```
+   2. ```modelcache/config/mysql_config.ini```
 4. 离线模型bin文件下载， 参考地址：[https://huggingface.co/shibing624/text2vec-base-chinese/tree/main](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main)，并将下载的bin文件，放到 model/text2vec-base-chinese 文件夹中
 5. 通过flask4modelcache.py脚本启动后端服务。
 ## 服务访问
@@ -245,11 +250,23 @@ https://mp.weixin.qq.com/s/ExIRu2o7yvXa6nNLZcCfhQ
    - 异步日志回写能力，用于数据分析和统计
    - 增加model字段和数据统计字段，用于功能拓展。
 
-未来会持续建设的功能：
+## Todo List
+### Adapter
+- [ ] register adapter for Milvus：根据scope中的model参数，初始化对应Collection 并且执行load操作。
+### Embedding model&inference
+- [ ] inference优化：优化embedding推理速度，适配fastertransformer, TurboTransformers, ByteTransformer等推理引擎。
+- [ ] 兼容huggingface模型和modelscope模型，提供更多模型加载方式。
+### Scalar Storage
+- [ ] Support MongoDB。
+- [ ] Support ElasticSearch。
+### Vector Storage
+- [ ] 在多模态场景中适配faiss存储。
+### Ranking
+- [ ] 增加Rank模型，对embedding召回后的数据，进行精排。
+### Service
+- [ ] 支持fastapi。
+- [ ] 增加前端界面，用于测试。
 
-- [ ] 基于超参数的数据隔离
-- [ ] system promt分区存储能力，以提高相似度匹配的准确度和效率
-- [ ] 更通用的embedding模型和相似度评估算法
 ## 致谢
 本项目参考了以下开源项目，在此对相关项目和研究开发人员表示感谢。<br />[GPTCache](https://github.com/zilliztech/GPTCache)
 

examples/flask/llms_cache/__init__.py

@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-

examples/flask/data_insert.py renamed to examples/flask/llms_cache/data_insert.py

@@ -13,7 +13,6 @@ def run():
     headers = {"Content-Type": "application/json"}
     res = requests.post(url, headers=headers, json=json.dumps(data))
     res_text = res.text
-    print('res_text: {}'.format(res_text))
 
 
 if __name__ == '__main__':

examples/flask/data_query.py renamed to examples/flask/llms_cache/data_query.py

@@ -13,7 +13,6 @@ def run():
     headers = {"Content-Type": "application/json"}
     res = requests.post(url, headers=headers, json=json.dumps(data))
     res_text = res.text
-    print('res_text: {}'.format(res_text))
 
 
 if __name__ == '__main__':

examples/flask/data_query_long.py renamed to examples/flask/llms_cache/data_query_long.py

@@ -18,7 +18,6 @@ def run():
     headers = {"Content-Type": "application/json"}
     res = requests.post(url, headers=headers, json=json.dumps(data))
     res_text = res.text
-    print('res_text: {}'.format(res_text))
 
 
 if __name__ == '__main__':

examples/flask/register.py renamed to examples/flask/llms_cache/register.py

@@ -14,7 +14,6 @@ def run():
     headers = {"Content-Type": "application/json"}
     res = requests.post(url, headers=headers, json=json.dumps(data))
     res_text = res.text
-    print('res_text: {}'.format(res_text))
 
 
 if __name__ == '__main__':

examples/flask/multi_cache/__init__.py

@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-

examples/flask/multi_cache/data_insert.py

@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+import time
+import json
+import uuid
+import requests
+
+
+def run():
+    url = 'http://127.0.0.1:5000/multicache'
+
+    request_type = 'insert'
+    scope = {"model": "multimodal_test"}
+    # UUID = "820b0052-d9d8-11ee-95f1-52775e3e6fd1" + "==>" + str(time.time())
+    UUID = str(uuid.uuid1()) + "==>" + str(time.time())
+    img_data = "https://img0.baidu.com/it/u=1436460262,4166266890&fm=253&fmt=auto&app=138&f=JPEG?w=500&h=282"
+    query = {'text': ['父母带着孩子来这个地方可能会有什么顾虑'],
+             'imageRaw': '',
+             'imageUrl': img_data,
+             'imageId': 'ccc'}
+    answer = "应该注意小孩不要跑到铁轨上"
+    chat_info = [{"query": query, "answer": answer}]
+    data_dict = {'request_type': request_type, 'scope': scope, 'chat_info': chat_info, 'UUID': UUID}
+
+    headers = {"Content-Type": "application/json"}
+    res = requests.post(url, headers=headers, json=json.dumps(data_dict))
+    res_text = res.text
+    print('res_text: {}'.format(res_text))
+
+
+if __name__ == '__main__':
+    run()