[Feature]: Optimize replay.json generation + presigning #2515
Assignees
Labels
enhancement
Requests a change to a feature
Comments
ikreymer
added a commit
that referenced
this issue
May 20, 2025
Fixes #2515. This PR introduces a significantly optimized logic for presigning URLs for crawls and collections. - For collections, the files needed from all crawls are looked up, and then the 'presign_urls' table is merged in one pass, resulting in a unified iterator containing files and presign urls for those files. - For crawls, the presign URLs are also looked up once, and the same iterator is used for a single crawl with passed in list of CrawlFiles - URLs that are already signed are added to the return list. - For any remaining URLs to be signed, a bulk presigning function is added, which shares an HTTP connection and signing 8 files in parallels (customizable via helm chart, though may not be needed). This function is used to call the presigning API in parallel.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What change would you like to see?
The replay.json for crawls and collections should be optimized to look up and presign bucket paths in bulk.
Context
Loading a crawl or collection with large number of WACZ files can be slow initially, since each file's path is looked up in the presign collection, and then if needed, each file is signed individually. This results in collections with hundereds of WACZ files to be slow on first load.
This can definitely be improved with batching querying and batch presigning.
The text was updated successfully, but these errors were encountered: