Running demo code on V100

[mikeleatila]

Hi,

Congrats on the great work!

I only have V100 GPUs available to me.

Is there a way to run your inference/demo code and how (e.g. with no flash attention)?

Many thanks in advance!

[lxtGH]

@zhang-tao-whu Check the issues. It seems like you need to modify the code to remove the flash attention.

[mikeleatila]

@lxtGH @zhang-tao-whu Thanks a lot. Do you have some hints how to remove flash attention? Can it be done by "passing-a-parameter-kind-of-thing"? Been looking around and trying out but it still does not work.

[HarborYuan]

@mikeleatila

Maybe you can try:

use_flash_attn=False
to disable the flash attention?

Please let me know whether it works well on V100.

[Ruining0916]

Hi I have also tried to set use_flash_attn=False; but it seems not working for V100? as there is still error message saying the modeling file is requiring flash_attn

[HarborYuan]

I see. Let me check it.

[mikeleatila]

Yes, I did the same as per @HarborYuan comment and got the same error as @Ruining0916 described above.