Skip to content

[Feature]: Optimize replay.json generation + presigning #2515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ikreymer opened this issue Mar 25, 2025 · 0 comments · May be fixed by #2516
Open

[Feature]: Optimize replay.json generation + presigning #2515

ikreymer opened this issue Mar 25, 2025 · 0 comments · May be fixed by #2516
Assignees
Labels
enhancement New feature or request

Comments

@ikreymer
Copy link
Member

What change would you like to see?

The replay.json for crawls and collections should be optimized to look up and presign bucket paths in bulk.

Context

Loading a crawl or collection with large number of WACZ files can be slow initially, since each file's path is looked up in the presign collection, and then if needed, each file is signed individually. This results in collections with hundereds of WACZ files to be slow on first load.

This can definitely be improved with batching querying and batch presigning.

@ikreymer ikreymer added the enhancement New feature or request label Mar 25, 2025
@ikreymer ikreymer self-assigned this Mar 25, 2025
@ikreymer ikreymer moved this from Triage to Todo in Webrecorder Projects Mar 25, 2025
@ikreymer ikreymer linked a pull request Mar 25, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

1 participant