Download using xet

[coyotte508]

cc @hanouticelina @Wauplin @bpronan @assafvayner

downloadFile has a xet: true optional param, that when set will download the file with the xet protocol if possible.

Breaking changes

• downloadFile returns a Blob
• fileDownloadInfo's return format is changed:
  • downloadLink => url, and it's present every time
  • optional xet prop for xet files

Concerns

https://huggingface.co/spaces/coyotte508/hub-xet uses quite a bit of CPU when downloading a full xet file, maybe we need to introduce multi-threading (with workers?) or optimize perf

Especially since we use it to parse safetensors data now cc @Kakulukian

Performance work

Before releasing out of experimental, some things need to be addressed & tested on different engines (node, browser):

• CPU usage, eg how fast it is to de-chunk 10GB of local data. Can it be significantly improved with moving some of the code to WASM.
• Stream backpressure, how much RAM does get used, especially due to different handling of Web streams, and if we can make it consistent across
• Http calls are sequentially made, to save RAM, but it can hurt in high-ping high-bandwidth situations. Not a problem if we download multiple files in parallel, but when downloading one large file, we probably don't care about reading data on the wire and can just promise-queue a bunch of http calls and use some RAM. We can probably change the behavior of the lib depending on the downloaded file's size.

Being able to upload xet files would be nice too, experimentally as well (eg for hub-ci tests)

[assafvayner]

uses quite a bit of CPU when downloading a full xet file, maybe we need to introduce multi-threading (with workers?) or optimize perf

We (myself and @seanses) are also looking at introducing web-workers for multithreading (and general interface for processing uploads) wasm xet-core. Promising stuff so far I think.