Enhance the performance of quantized model in terms of accuracy

[hieuhthh]

After applying quantization with q4_0, I noticed that the performance of my generated results has declined. I appreciate the speed and reduced VRAM usage that this quantized model offers, but I am seeking ways to enhance its performance. Could you please suggest any solutions or improvements, such as calibration, LoRA, fine-tuning, or other techniques? Thank you for your assistance!

[FNsi]

Ah, me, searched 1.58bit in huggingface... there's a method related it.

But ...I compared a lot, and selected a smaller model.

[misutoneko]

Is there a reason to use q4_0 specifically? I mean, it's ok but it's old.
You'd most likely get improved results just by switching to iq4_xs.
This is model-specific though, AFAICR some of the new llama versions are more sensitive to quantizing than the old models.