Added quantization for OUTETTS #2662
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
@KodiaqQ , what results do you get with the quantized model vs original on your machine? |
FP model generate time: 3.926095366012305 upd. Recalculated times with ignored scope. |
| "hf_model = OVHFModel(model_dir, device.value).model\n", | ||
| "dataset = nncf.Dataset(libritts, partial(transform_fn, interface=interface))\n", | ||
| "\n", | ||
| "quantized_model = nncf.quantize(\n", |
There was a problem hiding this comment.
I would suggest to use INT4 weight compression with dynamic quantization (A8W4). @KodiaqQ claim that the performance of such model is equal to the performance of the quantized model, but compression reate is higher for A8W4 model.
cc' @MaximProshin
There was a problem hiding this comment.
Please share numbers for both cases. If int4 is better, I'm ok to use that method.
Ticket: 157133