Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add optionality to run selectors/analysers on GCP #6

Closed
breezykermo opened this issue May 17, 2019 · 2 comments
Closed

add optionality to run selectors/analysers on GCP #6

breezykermo opened this issue May 17, 2019 · 2 comments
Labels
enhancement New feature or request

Comments

@breezykermo
Copy link
Member

this is fundamentally necessary so that MT can be run in the cloud without exhausting resources on a DO droplet or similarly contained instance.

When the 'cloud' flag for a selector or analyser is specified, the following should occur:

  • all raw data from the select pass should be saved in GCP buckets
  • all derived data should read from and write to GCP buckets.
  • there should be a boolean to switch between using GCP buckets and the local machine (we should definitely still support local file system so that google is not necessary to run).
  • ideally storage is abstracted as an interface so that other providers could also be used in the future.
@breezykermo breezykermo added the enhancement New feature or request label May 17, 2019
@JonathanReiss15
Copy link

Is there any specific reasoning behind choosing GCP over S3? I would propose that mtriage use S3 as there are multiple providers (AWS, DigitalOcean, Vultr, etc.) that provide S3 compatible services, while GCP buckets are GCP specific.

@breezykermo breezykermo added the notready There are other blocking issues before this can be fixed label Mar 17, 2020
@breezykermo breezykermo removed the notready There are other blocking issues before this can be fixed label Jul 23, 2020
@breezykermo
Copy link
Member Author

With #144, this is now supported in theory (documentation and more robust support is forthcoming).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants