You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is the implementation of **Information-Retrieving Fact-Checker (InFact)**, the text-only claim verification system that won the [AVeriTeC fact-checking challenge](https://fever.ai/task.html), see also the [AVeriTeC Leaderboard](https://eval.ai/web/challenges/challenge-page/2285/leaderboard/5655).
4
5
@@ -7,7 +8,7 @@ This is the implementation of **Information-Retrieving Fact-Checker (InFact)**,
InFact is a text-only, 6-stage LLM pipeline for fact-checking with external evidence retrieval through Google Search. For evaluation on benchmarks that provide a knowledge store (like on AVeriTeC or FEVER), it supports the retrieval of relevant evidence by leveraging an embedding model to perform semantic search.
11
+
InFact is a text-only, 6-stage LLM pipeline for fact-checking with external evidence retrieval through Google Search. For evaluation on benchmarks that provide a knowledge store (like AVeriTeC or FEVER), it supports the retrieval of relevant evidence by leveraging an embedding model to perform semantic search.
11
12
12
13
> [!NOTE]
13
14
> The execution of the embedding model and the Llama LLMs requires GPUs.
@@ -40,6 +41,7 @@ You can also find the Docker image [here](https://hub.docker.com/r/tudamailab/in
40
41
pip install -r requirements.txt
41
42
```
42
43
44
+
43
45
## Configuration & Building
44
46
### 1. Configure API Keys
45
47
To this end, run the script `scripts/setup.py` with
@@ -52,7 +54,8 @@ python -m scripts.setup
52
54
Alternatively, you may manually insert all necessary API keys into the file `config/api_keys.yaml`.
In `config/globals.py`, insert the paths to the respective input/output directories.
57
+
In `config/globals.py`, insert the paths to the respective input/output directories. If you want to evaluate on AVeriTeC, you'll need to download the benchmark separately [here](https://huggingface.co/chenxwh/AVeriTeC/tree/main/data).
58
+
56
59
57
60
### 3. Optional: Build AVeriTeC Knowledge Base (KB)
58
61
Simply run the script `scripts/averitec/build.py`with
@@ -72,11 +75,15 @@ With `scripts/run.py`, you can fact-check your own claims. The command to run it
72
75
```bash
73
76
python -m scripts.run
74
77
```
75
-
If you provided a Serper API key, this will use Google Search to verify your claim.
78
+
If you configured a Serper API key, InFact will automatically use Google Search to verify your claim, otherwise it falls back to DuckDuckGo (which has strong rate limits).
76
79
77
80
### Reproduce InFact Paper Results
78
81
To reproduce InFact's result on AVeriTeC using its default hyperparameters, run
79
82
```bash
80
83
python -m scripts.averitec.evaluate
81
84
```
82
85
The number of workers executing the evaluation is based on the number of GPUs in your machine.
86
+
87
+
88
+
## License
89
+
This repository and all its contents are licensed under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0).
0 commit comments