fix(ingest/snowflake): order queries for queries_v2 #12551
Conversation
github-actions
bot
added
the
ingestion
datahub-cyborg
bot
added
the
needs-review
Codecov ReportAll modified and coverable lines are covered by tests ✅ ✅ All tests successful. No failed tests found.
Continue to review full report in Codecov by Sentry.
|
| @@ -696,6 +696,9 @@ def _build_enriched_query_log_query( | |||
| JOIN filtered_access_history a USING (query_id) | |||
| ) | |||
| SELECT * FROM query_access_history | |||
| -- Our query aggregator expects the queries to be added in chronological order. | |||
| -- It's easier for us to push down the sorting to Snowflake/SQL instead of doing it in Python. | |||
| ORDER BY start_time ASC | |||
There was a problem hiding this comment.
I checked if we are doing the sorting in eg bigquery and yes, we do
Are there other sources to validate?
There was a problem hiding this comment.
Snowflake is getting fixed here, bq is all good. For redshift, we actually register temp tables as a preprocessing step, so it's fine that the other queries don't have an order by
There was a problem hiding this comment.
What if we add a validation in sql parsing aggregator? we may track start query time as queries are added to the aggregator and eg we may raise a warning if the start query time is not increasing for every addition
datahub-cyborg
bot
added
pending-submitter-response
datahub-cyborg
bot
added
pending-submitter-merge
and removed
pending-submitter-response
Checklist