Safetensor Support

[mert-kurttutan]

Describe the workflow you want to enable

Are models available via safetensor format?

Benefits:

• By default, it is safer than pickle format
• More standardized format
• Available to more other frameworks (not just pytorch).

Describe your proposed solution

Doing a PR to HF repo

Describe alternatives you've considered, if relevant

Right now I am using script to do conversion. It would be better to have an official support.

Additional context

No response

Impact

None

[alexvorobiev]

In many environments third party pickle files are not allowed for security reasons.

@mert-kurttutan Do you use this script for converting to safetensors format https://github.com/huggingface/safetensors/blob/main/bindings/python/convert.py? Did you have to modify the model loading code in TabPFN (https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/model/loading.py#L681)?

[klemens-floege]

Chatted with @oscarkey offline and this issue is still very much relevant!

[oscarkey]

#981 adds safetensor support, and a conversion script for people who need to use safetensor versions of the existing checkpoints. We'll also upload future model checkpoints as safetensor versions.