Add new import logs senario #315
Conversation
scenarios/import_td_logs/README.md
Outdated
|
|
||
| # How to Run | ||
| ## Requirement | ||
| The workflow requires that Data Landing Areas feature is enabled in your Treasure Data account and thne you've got your User ID to access to it. |
| @@ -0,0 +1,36 @@ | |||
| # Workflow: Import Treasure Data Logs from Data Landing Area | |||
| This example shows how you can use workflow to ingest Treasure Data Logs From Data Landing Areas to your Treasure Data account. | |||
There was a problem hiding this comment.
Add a statement;
"This is Opt-in feature. Contact your Customer Success rep or Technical Support if you have an interest in this feature."
| host: ${dla_host} | ||
| user: ${user_id} | ||
| secret_key_file: {"content": "${secret:sftp.dla_secret_key_file}"} | ||
| path_prefix: "/treasure-data-logs/production/${account_id}/workflow_logs/v1/data.csv" |
There was a problem hiding this comment.
account id is possibly not unique across regions. Better to use account_key such a 'aws_1' Or 'production' can be production_us, production_tokyo, production_eu01, production_ap02
| host: ${dla_host} | ||
| user: ${user_id} | ||
| secret_key_file: {"content": "${secret:sftp.dla_secret_key_file}"} | ||
| path_prefix: "/treasure-data-logs/${site}/${account_id}/workflow_logs/v1/data.csv" |
There was a problem hiding this comment.
How do you block access to the other customer data?
There was a problem hiding this comment.
It's systematically separated on our Data Landing Areas feature that is using AWS Transfer Family.
So, a customer can only access to the space of the corresponding to the published user_id by TD.
https://treasure-data.atlassian.net/wiki/spaces/EN/pages/1740343892/ERD+Data+Landing+Areas+SFTP
https://docs.aws.amazon.com/ja_jp/transfer/latest/userguide/how-aws-transfer-works.html
| +users: | ||
| _env: | ||
| TD_API_KEY: ${secret:td.apikey} | ||
| py>: script.import_td_users.import_users | ||
| database: ${td.database} | ||
| table: ${users_table} | ||
| api_endpoint: ${api_endpoint} | ||
| docker: | ||
| image: "digdag/digdag-python:3.9" No newline at end of file |
There was a problem hiding this comment.
2 space
| +users: | |
| _env: | |
| TD_API_KEY: ${secret:td.apikey} | |
| py>: script.import_td_users.import_users | |
| database: ${td.database} | |
| table: ${users_table} | |
| api_endpoint: ${api_endpoint} | |
| docker: | |
| image: "digdag/digdag-python:3.9" | |
| +users: | |
| _env: | |
| TD_API_KEY: ${secret:td.apikey} | |
| py>: script.import_td_users.import_users | |
| database: ${td.database} | |
| table: ${users_table} | |
| api_endpoint: ${api_endpoint} | |
| docker: | |
| image: "digdag/digdag-python:3.9" |
There was a problem hiding this comment.
@satoshihirose I believe it's ok to merge this PR because @toru-takahashi's change request should have been fixed. (Sorry for the very very very late approval)
No description provided.