Merge pull request #138 from gomate-community/pipeline

Pipeline

Author: yanqiangmiffy

README.md

@@ -254,9 +254,9 @@ bge_reranker = BgeReranker(reranker_config)
 <summary>PointWise-Rerank</summary>
 We have two pointwise methods so far:
 
-`relevance generation`: LLMs are prompted to judge whether the given query and document are relevant. Candidate documents are reranked based on the likelihood of generating a "yes" response by LLMs. It is the rerank method used in (https://arxiv.org/pdf/2211.09110).
+`relevance generation`: LLMs are prompted to judge whether the given query and document are relevant. Candidate documents are reranked based on the likelihood of generating a "yes" response by LLMs. It is the rerank method used in [Holistic Evaluation of Language Models](https://arxiv.org/pdf/2211.09110).
 
-`query generation`: LLMs are prompted to generate a pseudo-query based on the given document. Candidate documents are reranked based on the likelihood of generating the target query by LLMs. It is the rerank method used in (https://arxiv.org/pdf/2204.07496).
+`query generation`: LLMs are prompted to generate a pseudo-query based on the given document. Candidate documents are reranked based on the likelihood of generating the target query by LLMs. It is the rerank method used in [Improving Passage Retrieval with Zero-Shot Question Generation](https://arxiv.org/pdf/2204.07496).
 
 We have implemented [flan-t5](https://huggingface.co/docs/transformers/model_doc/flan-t5) as our pointwise reranker model.
 ```python
@@ -270,7 +270,19 @@ llm_reranker = PointWiseReranker(reranker_config)
 
 <details>
 <summary>PairWise-Rerank</summary>
-Waiting to implement...
+We have two pairwise method so far:
+
+`allpair`: LLMs are prompted to judge which document is the more relevant to the given query than the other. Candidate documents are based on the number of that they win. It is the rerank method used in [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://arxiv.org/pdf/2306.17563).
+
+`bubblesort`: LLMs are prompted to judge which document is the more relevant to the given query than the other. Candidate documents are reranked using bubblesort algorithm. It is the other rerank method used in [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://arxiv.org/pdf/2306.17563).
+
+```python
+from trustrag.modules.reranker.llm_reranker import LLMRerankerConfig, PairWiseReranker
+reranker_config = LLMRerankerConfig(
+    model_name_or_path="qwen2-7B-instruct"
+)
+llm_reranker = PairWiseReranker(reranker_config)
+```
 </details>
 
 <details>

config_online.json

@@ -7,7 +7,7 @@
         },
         "siliconflow": {
             "base_url": "https://api.siliconflow.cn/v1",
-            "api_key": "sk-yfgjndsavpwcnnedlhllyfunxwsckfguirokexokstbvwnjf",
+            "api_key": "sk-xxxx",
             "description": "SiliconFlow API 服务"
         },
         "rerank": {

docs/dify.md

@@ -1,7 +1,7 @@
 ## git常见命令
 - 合并分支
 ```bash
-git pull main
+git pull origin main
 git merge main
 ```
 

docs/git.md

@@ -0,0 +1 @@
+docker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v $(pwd):/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0

docs/xinference.md

@@ -1,10 +1,10 @@
 import re
-import numpy as np
 from sklearn.metrics.pairwise import cosine_similarity
 from trustrag.modules.document import rag_tokenizer
 from trustrag.modules.chunks.base import BaseChunker
 from sentence_transformers import SentenceTransformer
 from langchain.embeddings import OpenAIEmbeddings
+from langchain_experimental.text_splitter import SemanticChunker
 
 class SemanticChunker(BaseChunker):
     """
@@ -136,7 +136,7 @@ def get_chunks(self, paragraphs: list[str]) -> list[str]:
 
         # Determine breakpoints based on the similarity threshold
         breakpoint_indices = [i for i, distance in enumerate(distances) if distance > (1 - self.similarity_threshold)]
-
+        print(breakpoint_indices)
         # Combine sentences into chunks
         chunks = []
         start_index = 0
@@ -181,7 +181,7 @@ def process_text_chunks(self, chunks: list[str]) -> list[str]:
         return processed_chunks
 
 if __name__ == '__main__':
-    with open("../../../data/docs/news.txt", "r", encoding="utf-8") as f:
+    with open("../../../data/docs/伊朗总统罹难事件.txt", "r", encoding="utf-8") as f:
         content = f.read()
 
     # Example 1: Use SentenceTransformer