Skip to content

Commit

Permalink
Added system requirements guesses
Browse files Browse the repository at this point in the history
  • Loading branch information
lunamidori5 committed Feb 5, 2024
1 parent 0412d18 commit e5377a4
Showing 1 changed file with 21 additions and 11 deletions.
32 changes: 21 additions & 11 deletions content/models/onsite_models/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@ weight = 154

All models are highly recommened for newer users as they are super easy to use and use the CHAT templ files from [Twinz](https://github.com/TwinFinz)

| Model Size | Description |
|---|---|
| 7b | CPU Friendly, small, okay quality |
| 2x7b | Normal sized, good quality |
| 8x7b | Big, great quality |
| 70b | Large, hard to run, significant quality |
| Model Size | Description | Links |
|---|---|---|
| 7b | CPU Friendly, small, okay quality | https://huggingface.co/TheBloke/dolphin-2.6-mistral-7B-GGUF |
| 2x7b | Normal sized, good quality | https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF |
| 8x7b | Big, great quality | https://huggingface.co/TheBloke/dolphin-2.7-mixtral-8x7b-GGUF |
| 70b | Large, hard to run, significant quality | https://huggingface.co/TheBloke/dolphin-2.2-70B-GGUF |

| Quant Mode | Description |
|---|---|
Expand All @@ -23,9 +23,19 @@ All models are highly recommened for newer users as they are super easy to use a
| Q8 | Extremely large, extremely low quality loss, hard to use - not recommended |
| None | Extremely large, No quality loss, super hard to use - really not recommended |

The models used by this program as of right now are:
The minimum RAM and VRAM requirements for each model size, as a rough estimate.
- 7b:
- System RAM: 10 GB
- VRAM: 2 GB

- 2x7b:
- System RAM: 25 GB
- V RAM: 8 GB

- 8x7b:
- System RAM: 55 GB
- VRAM: 28 GB

- 7b: [TheBloke/dolphin-2.6-mistral-7B-GGUF](https://huggingface.co/TheBloke/dolphin-2.6-mistral-7B-GGUF)
- 2x7b: [TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF](https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF)
- 8x7b: [TheBloke/dolphin-2.7-mixtral-8x7b-GGUF](https://huggingface.co/TheBloke/dolphin-2.7-mixtral-8x7b-GGUF)
- 70b: [TheBloke/dolphin-2.2-70B-GGUF](https://huggingface.co/TheBloke/dolphin-2.2-70B-GGUF)
- 70b:
- System RAM: 105 GB
- VRAM: AI Card or better

0 comments on commit e5377a4

Please sign in to comment.