DENG-6770: create table and view for fxqur_2024okr_desktop #7011

ogzillla · 2025-02-10T23:17:24Z

Description

Creates a new table and view for fxqur_2024okr_desktop_v

Related Tickets & Documents

DENG-6670

Reviewer, please follow this checklist

…Productionize-fx-user-research-residing-in-analysis-datasets

sql/moz-fx-data-shared-prod/fxresearch_derived/fxqur_2024okr_desktop_v1/metadata.yaml

alekhyamoz · 2025-02-11T01:22:19Z

@ogzillla let's name the derived dataset as fx_quant_user_research_derived

…Productionize-fx-user-research-residing-in-analysis-datasets

dataops-ci-bot · 2025-02-11T21:21:25Z

Integration report for "DENG-6770: update field type"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/dags/: bqetl_fivetran_alchemy.py
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_alchemy.py /tmp/workspace/generated-sql/dags/bqetl_fivetran_alchemy.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_alchemy.py	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fivetran_alchemy.py	2025-02-11 21:20:38.000000000 +0000
@@ -0,0 +1,63 @@
+# Generated via https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/query_scheduling/generate_airflow_dags.py
+
+from airflow import DAG
+from airflow.sensors.external_task import ExternalTaskMarker
+from airflow.sensors.external_task import ExternalTaskSensor
+from airflow.utils.task_group import TaskGroup
+import datetime
+from operators.gcp_container_operator import GKEPodOperator
+from utils.constants import ALLOWED_STATES, FAILED_STATES
+from utils.gcp import bigquery_etl_query, bigquery_dq_check, bigquery_bigeye_check
+
+docs = """
+### bqetl_fivetran_alchemy
+
+Built from bigquery-etl repo, [`dags/bqetl_fivetran_alchemy.py`](https://github.com/mozilla/bigquery-etl/blob/generated-sql/dags/bqetl_fivetran_alchemy.py)
+
+#### Description
+
+DAG for scheduling queries pulling survey data from Alchemy.
+
+#### Owner
+
+[email protected]
+
+#### Tags
+
+* impact/tier_3
+* repo/bigquery-etl
+"""
+
+
+default_args = {
+    "owner": "[email protected]",
+    "start_date": datetime.datetime(2025, 3, 1, 0, 0),
+    "end_date": None,
+    "email": ["[email protected]"],
+    "depends_on_past": False,
+    "retry_delay": datetime.timedelta(seconds=1500),
+    "email_on_failure": True,
+    "email_on_retry": False,
+    "retries": 1,
+}
+
+tags = ["impact/tier_3", "repo/bigquery-etl"]
+
+with DAG(
+    "bqetl_fivetran_alchemy",
+    default_args=default_args,
+    schedule_interval="0 12 * * *",
+    doc_md=docs,
+    tags=tags,
+) as dag:
+
+    fx_quant_user_research_derived__fxqur_2024okr_desktop__v1 = bigquery_etl_query(
+        task_id="fx_quant_user_research_derived__fxqur_2024okr_desktop__v1",
+        destination_table="fxqur_2024okr_desktop_v1",
+        dataset_id="fx_quant_user_research_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]"],
+        date_partition_parameter="submission_date",
+        depends_on_past=False,
+    )
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: fx_quant_user_research
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: fx_quant_user_research_derived
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/dataset_metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/dataset_metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/dataset_metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/dataset_metadata.yaml	2025-02-11 21:11:56.000000000 +0000
@@ -0,0 +1,18 @@
+friendly_name: Fx Quant User Research
+description: |-
+  Contains views for Fx Quant User Research.
+
+  JIRA: https://mozilla-hub.atlassian.net/browse/DENG-6770
+dataset_base_acl: view
+user_facing: false
+labels: {}
+default_table_workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+default_table_expiration_ms: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+syndication: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/metadata.yaml	2025-02-11 21:15:25.000000000 +0000
@@ -0,0 +1,13 @@
+friendly_name: Fxqur 2024okr Desktop
+description: |-
+  Please provide a description for the query
+owners: []
+labels: {}
+bigquery: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references:
+  view.sql:
+  - moz-fx-data-shared-prod.fx_quant_user_research_derived.fxqur_2024okr_desktop_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/view.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research/fxqur_2024okr_desktop/view.sql	2025-02-11 21:11:56.000000000 +0000
@@ -0,0 +1,7 @@
+CREATE OR REPLACE VIEW
+  `moz-fx-data-shared-prod.fx_quant_user_research.fxqur_2024okr_desktop`
+AS
+SELECT
+  *
+FROM
+  `moz-fx-data-shared-prod.fx_quant_user_research_derived.fxqur_2024okr_desktop_v1`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/dataset_metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/dataset_metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/dataset_metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/dataset_metadata.yaml	2025-02-11 21:11:56.000000000 +0000
@@ -0,0 +1,18 @@
+friendly_name: Fx Quant User Research Derived
+description: |-
+  Contains tables for Fx Quant User Research Derived.
+
+  JIRA: https://mozilla-hub.atlassian.net/browse/DENG-6770
+dataset_base_acl: derived
+user_facing: false
+labels: {}
+default_table_workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+default_table_expiration_ms: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+syndication: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/metadata.yaml	2025-02-11 21:15:25.000000000 +0000
@@ -0,0 +1,25 @@
+friendly_name: Fxqur 2024okr Desktop
+description: |-
+  Survey Results from Alchemy for Desktop
+owners:
+- [email protected]
+labels:
+  incremental: true
+  owner1: oanaekwe
+  dag: bqetl_fivetran_alchemy
+scheduling:
+  dag_name: bqetl_fivetran_alchemy
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: false
+    expiration_days: null
+  range_partitioning: null
+  clustering:
+    fields: []
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/query.sql	2025-02-11 21:11:56.000000000 +0000
@@ -0,0 +1,9 @@
+SELECT
+  month_year,
+  okr_groups,
+  csat_high,
+  csat_low,
+  month_tot,
+  csat_prop
+FROM
+  `mozdata.analysis.fxqur_2024okr_desktop`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fx_quant_user_research_derived/fxqur_2024okr_desktop_v1/schema.yaml	2025-02-11 21:11:56.000000000 +0000
@@ -0,0 +1,19 @@
+fields:
+- name: month_year
+  type: DATE
+  mode: NULLABLE
+- name: okr_groups
+  type: STRING
+  mode: NULLABLE
+- name: csat_high
+  type: FLOAT64
+  mode: NULLABLE
+- name: csat_low
+  type: FLOAT64
+  mode: NULLABLE
+- name: month_tot
+  type: INT64
+  mode: NULLABLE
+- name: csat_prop
+  type: FLOAT64
+  mode: NULLABLE

Link to full diff

Ogonna Anaekwe added 2 commits February 10, 2025 18:14

DENG-6770: create table and view for fxqur_2024okr_desktop

5be96b2

Merge branch 'main' of github.com:mozilla/bigquery-etl into DENG-6770-…

ed074f1

…Productionize-fx-user-research-residing-in-analysis-datasets

ogzillla requested review from alekhyamoz and kwindau February 10, 2025 23:27

ogzillla commented Feb 10, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fxresearch_derived/fxqur_2024okr_desktop_v1/metadata.yaml Outdated Show resolved Hide resolved

ogzillla commented Feb 10, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fxresearch_derived/fxqur_2024okr_desktop_v1/metadata.yaml Outdated Show resolved Hide resolved

DENG-6770: format queries with bqetl

4a0d8c8

This comment has been minimized.

Sign in to view

DENG-6770: format sql and yaml files

5fae8aa

This comment has been minimized.

Sign in to view

ogzillla mentioned this pull request Feb 11, 2025

DENG-7839 - TESTING - Backfill baseline_clients_last_seen #7010

Merged

Ogonna Anaekwe and others added 2 commits February 11, 2025 14:43

DENG-6770: add new DAG and update config values

fce1e18

Merge branch 'main' of github.com:mozilla/bigquery-etl into DENG-6770-…

ab722d8

…Productionize-fx-user-research-residing-in-analysis-datasets

This comment has been minimized.

Sign in to view

ogzillla added 3 commits February 11, 2025 16:07

DENG-6770: update dataset name and include schema

b53fc2a

DENG-6770: update dataset name and include schema

8074bc9

DENG-6770: update field type

640a567

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DENG-6770: create table and view for fxqur_2024okr_desktop #7011

DENG-6770: create table and view for fxqur_2024okr_desktop #7011

ogzillla commented Feb 10, 2025

alekhyamoz commented Feb 11, 2025 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented Feb 11, 2025

DENG-6770: create table and view for fxqur_2024okr_desktop #7011

Are you sure you want to change the base?

DENG-6770: create table and view for fxqur_2024okr_desktop #7011

Conversation

ogzillla commented Feb 10, 2025

Description

Related Tickets & Documents

alekhyamoz commented Feb 11, 2025 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented Feb 11, 2025

Integration report for "DENG-6770: update field type"

sql.diff

alekhyamoz commented Feb 11, 2025 •

edited

Loading

`sql.diff`