Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use Primary location unless location is set explicitly #172

Merged
merged 1 commit into from
Dec 16, 2024

Conversation

joker1007
Copy link
Contributor

BigQuery can switch dataset location.

Unless location field is set explicity, current embulk-output-bigquery use us/eu location implicitly.
But, BigQuery can insert to any location without location information, and return location information in job_reference field.
And so, I make embulk-output-bigquery enable to use returned location information, if location config is not set.

This patch eliminates the need to change embulk config when switching location

@joker1007
Copy link
Contributor Author

@hiroyuki-sato could you review this?

@hiroyuki-sato
Copy link
Member

@joker1007, I'm sorry for not getting back to you sooner.

Looking good to me. I want to ask some questions just in case.

  1. Does Primary locations (_response.job_reference.location ) mean it contains other places neither us nor eu ? (ex: asia-northeast1?) JobReference
  2. This change keeps the current behavior except the log message. correct?

Below is my memo.

This PR changes the log message from in us/eu to in Primary location.

2024-12-16 20:36:11.397 +0900 [INFO] (Ruby-0-Thread-2: /path/to/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery/bigquery_client.rb:153): embulk-output-bigquery: Load job starting... job_id:[embulk_load_job_d63fa73e-09b5-4a57-877e-bf934b3638d4] /var/folders/c4/n95yngmn1kz96x7jslsnqjr80000gn/T/embulk_output_bigquery_20241216-30614-i0qc0q.30614.4016.csv => bqtest2-xxxxx:mydataset.LOAD_TEMP_e58cad5e_7a7b_41bc_9194_ffe3e08d40e3_hello_embulk in us/eu

The variable @location_for_log is used in the logger only.

236:              raise Error, "failed to load #{path} to #{@destination_project}:#{@dataset}.#{table} in #{@location_for_log}, response:#{response}"
359:            Embulk.logger.info { "embulk-output-bigquery: Create dataset... #{@destination_project}:#{dataset} in #{@location_for_log}" }
375:            Embulk.logger.debug { "embulk-output-bigquery: insert_dataset(#{@project}, #{dataset}, #{@location_for_log}, #{body}, #{opts})" }
387:            raise Error, "failed to create dataset #{@destination_project}:#{dataset} in #{@location_for_log}, response:#{response}"
451:            Embulk.logger.debug { "embulk-output-bigquery: insert_table(#{@destination_project}, #{dataset}, #{@location_for_log}, #{body}, #{opts})" }
461:              "embulk-output-bigquery: insert_table(#{@destination_project}, #{dataset}, #{@location_for_log}, #{body}, #{opts}), response:#{response}"
463:            raise Error, "failed to create table #{@destination_project}:#{dataset}.#{table} in #{@location_for_log}, response:#{response}"

@joker1007
Copy link
Contributor Author

@hiroyuki-sato
Thank you for your comment.

  1. Does Primary locations (_response.job_reference.location ) mean it contains other places neither us nor eu ? (ex: asia-northeast1?) JobReference

Yes. For example a dataset location is asia-northeast1, insert_job can submit without location and returned response includes location field having "asia-northeast1".

  1. This change keeps the current behavior except the log message. correct?

Yes. at least I don't intend it.

Copy link
Member

@hiroyuki-sato hiroyuki-sato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM👍 Thanks!

@hiroyuki-sato hiroyuki-sato merged commit a84ca97 into master Dec 16, 2024
3 checks passed
@hiroyuki-sato hiroyuki-sato deleted the auto-location branch December 16, 2024 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants