Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create mirror queues and bucket #6859

Closed
hannes-ucsc opened this issue Jan 29, 2025 · 2 comments
Closed

Create mirror queues and bucket #6859

hannes-ucsc opened this issue Jan 29, 2025 · 2 comments
Assignees
Labels
+ [priority] High demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team mirror [subject] Code and infrastructure for mirroring data files orange [process] Done by the Azul team

Comments

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Jan 29, 2025

Create the cloud infrastructure needed for mirroring, excluding any Lambda functions, most prominently the mirror queue (FIFO), the corresponding fail queue and the actual mirror bucket. The bucket should be non-public for now, matching the default access for all of our buckets. Ensure existing functionality isn't impacted by that infrastructure. Extend the manage_queues.py script to be able to handle the new queue. Resetting the indexer shouldn't affect the mirror queues.

@github-actions github-actions bot added the orange [process] Done by the Azul team label Jan 29, 2025
@hannes-ucsc hannes-ucsc added mirror [subject] Code and infrastructure for mirroring data files + [priority] High enh labels Jan 29, 2025
@dsotirho-ucsc
Copy link
Contributor

@hannes-ucsc: "The specifics of the S3 mirror bucket are still being discussed. It is clear that it will have to reside in a separate AWS account, but ownership and control of that account is yet to be determined. In lower deployments we will have to use a non-ODP bucket anyways. Assignee to implement this issue for lower deployments where the mirror bucket resides in the same account as the reset of the infrastructure and the bucket should be part of the main component. Whether mirroring is enabled or not should be configurable. By default, it should be disabled. Assignee to also update the prototype (#6857) to mirror to that new bucket."

@achave11-ucsc achave11-ucsc removed the enh label Feb 11, 2025
@hannes-ucsc
Copy link
Member Author

hannes-ucsc commented Feb 25, 2025

For demo, show the new cloud resources in dev. Show that reindex.py is still functional by starting a reindex in a personal deployment, hitting Ctrl-C mid reindex and starting it again. While the reindex is running, show that the manage_queues.py script is functional.

@hannes-ucsc hannes-ucsc added the demo [process] To be demonstrated at the end of the sprint label Feb 25, 2025
@nadove-ucsc nadove-ucsc added the demoed [process] Successfully demonstrated to team label Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
+ [priority] High demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team mirror [subject] Code and infrastructure for mirroring data files orange [process] Done by the Azul team
Projects
None yet
Development

No branches or pull requests

4 participants