Inquiry on Feasibility of Fine-tuning with 2-4 A800 GPUs

[c-z-zhao]

Hello, I think your work is excellent, and I'd like to apply it to a downstream project. I have approximately 10K training samples. I was wondering if fine-tuning with 2 to 4 A800 GPUs(80G) would be feasible?

[chenjoya]

Hi, thanks for your interests!

Yes, it is absolutely feasible. 80G GPUs support per_device_train_batch_size=1. Adjust gradient_accumulation_steps to set your total batch size!