Download using xet #1305
Merged
Download using xet #1305
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
coyotte508
commented
Mar 21, 2025
We (myself and @seanses) are also looking at introducing web-workers for multithreading (and general interface for processing uploads) wasm xet-core. Promising stuff so far I think. |
|
Merging this one, and major version! |
but for uploads from the Web we will/would use the Wasm version, is that correct? not the pure JS version? |
wasm for chunking |
|
^yes |
Deep-unlearning
pushed a commit
that referenced
this pull request
May 13, 2025
cc @hanouticelina @Wauplin @bpronan @assafvayner `downloadFile` has a `xet: true` optional param, that when set will download the file with the xet protocol if possible. ## Breaking changes - `downloadFile` returns a `Blob` - `fileDownloadInfo`'s return format is changed: - `downloadLink` => `url`, and it's present every time - optional `xet` prop for xet files ## Concerns https://huggingface.co/spaces/coyotte508/hub-xet uses quite a bit of CPU when downloading a full xet file, maybe we need to introduce multi-threading (with workers?) or optimize perf Especially since we use it to parse safetensors data now cc @Kakulukian ## Performance work Before releasing out of experimental, some things need to be addressed & tested on different engines (node, browser): - CPU usage, eg how fast it is to de-chunk 10GB of local data. Can it be significantly improved with moving some of the code to WASM. - Stream backpressure, how much RAM does get used, especially due to different handling of Web streams, and if we can make it consistent across - Http calls are sequentially made, to save RAM, but it can hurt in high-ping high-bandwidth situations. Not a problem if we download multiple files in parallel, but when downloading one large file, we probably don't care about reading data on the wire and can just promise-queue a bunch of http calls and use some RAM. We can probably change the behavior of the lib depending on the downloaded file's size. Being able to upload xet files would be nice too, experimentally as well (eg for hub-ci tests)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cc @hanouticelina @Wauplin @bpronan @assafvayner
downloadFilehas axet: trueoptional param, that when set will download the file with the xet protocol if possible.Breaking changes
downloadFilereturns aBlobfileDownloadInfo's return format is changed:downloadLink=>url, and it's present every timexetprop for xet filesConcerns
https://huggingface.co/spaces/coyotte508/hub-xet uses quite a bit of CPU when downloading a full xet file, maybe we need to introduce multi-threading (with workers?) or optimize perf
Especially since we use it to parse safetensors data now cc @Kakulukian
Performance work
Before releasing out of experimental, some things need to be addressed & tested on different engines (node, browser):
Being able to upload xet files would be nice too, experimentally as well (eg for hub-ci tests)