Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only enable requester pays when necessary #400

Open
mbaumann-broad opened this issue Jan 17, 2023 · 0 comments
Open

Only enable requester pays when necessary #400

mbaumann-broad opened this issue Jan 17, 2023 · 0 comments

Comments

@mbaumann-broad
Copy link
Collaborator

Objective

Only perform the operation to enable requester pays when it is currently necessary.
The only case where it is currently necessary is when accessing AnVIL data hosted by Gen3.
This case can be identified by DRS URIs starting with "drs://dg.ANV0".
In all other cases, enabling requester pays adds unnecessary delays and, in some cases, as when a Terra user has an external identity linkage to Kids First, spurious errors.

Background

Accessing DRS data hosted by Gen3 in requester pays buckets using Google cloud-native access APIs required that the Gen3 user client service account be granted the serviceusage.services.use permission on the Terra user's Google project.

This privileged operation is performed by the Rawls API enableRequesterPaysForLinkedServiceAccounts.

Originally, this was needed for Terra DRS access to Gen3-hosted data for both Biodata Catalyst and AnVIL.
Terra DRS access to Gen3-hosted Biodata Catalyst data has since changed to using signed URLs, and enabling requester pays is no longer necessary for Biodata Catalyst.
Terra DRS access to Gen3-hosted data for NCI CRDC and Kids First has never been necessary.

When AnVIL data is fully hosted in Terra Data Repo (TDR), enabling requester pays will no longer be needed at all and should be disabled completely.

Acceptance Criteria

  • terra-notebook-utils only enables requester pays when DRS operations are performed on one or more DRS URIs starting with "drs://dg.ANV0"
  • A version of terra-notebook-utils with this functionality has been released and is available from PyPi.
mbaumann-broad pushed a commit that referenced this issue Jan 18, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 18, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 18, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 19, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 19, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 19, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 19, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 20, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 20, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 24, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad pushed a commit that referenced this issue Jan 25, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
mbaumann-broad added a commit that referenced this issue Jan 25, 2023
Only enable (Google) requester pays data access when the
given DRS URIs require it and the platform TNU is running
on supports it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant