Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP backend does not support multiple exports #48

Open
jhiemstrawisc opened this issue Oct 16, 2024 · 0 comments
Open

HTTP backend does not support multiple exports #48

jhiemstrawisc opened this issue Oct 16, 2024 · 0 comments

Comments

@jhiemstrawisc
Copy link
Member

In the S3 plugin, we developed the special s3.begin and s3.end config directives so that we could point various namespaces at different buckets, but I don't think we have a similar mechanism for the HTTP plugin.

Consider the scenario, where the HTTP server at https://data.lhncbc.nlm.nih.gov/ hosts two image datasets, one with path /public/Tuberculosis-Chest-X-ray-Datasets/Montgomery-County-CXR-Set/MontgomerySet/CXR_png and one with path /public/Tuberculosis-Chest-X-ray-Datasets/Shenzhen-Hospital-CXR-Set/CXR_png. It would be nice to add the config blocks to http to accomplish a config similar to:

httpserver.begin
httpserver.url_base https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Montgomery-County-CXR-Set/MontgomerySet/CXR_png
httpserver.storage_prefix /prefix-1
httpserver.token_file <some token file to use with requests to that path, if provided>
httpserver.end

httpserver.begin
httpserver.url_base https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Shenzhen-Hospital-CXR-Set/CXR_png
httpserver.storage_prefix prefix-2
httpserver.token_file <some token file to use with requests to that path, if provided
httpserver.token_file <some token file to use with requests to that path, if provided>
httpserver.end

This would let us scope these two paths under separate prefixes, so a request for /prefix-1/foo.png turns into an HTTP request for https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Montgomery-County-CXR-Set/MontgomerySet/CXR_png/foo.png, while a request for /prefix-2/foo.png turns into an HTTP request for https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Shenzhen-Hospital-CXR-Set/CXR_png/foo.png

Note that I've also moved the httpserver.token_file directive into the begin/end blocks with the assumption that different http paths could require different access tokens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant