Release 6.1.4 #1374
Merged
Release 6.1.4 #1374
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Scenario:
* Input batch contains data using schema, let’s say link_click
* link_click schema is used as a context AND as an entity/self describing event
* We have multiple versions of link_click schema
When translated to the content of shredding_complete JSON message, it would contain Iines like this:
```
"types": [
{
"schemaKey": "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-0",
"snowplowEntity": "SELF_DESCRIBING_EVENT"
},
....
{
"schemaKey": "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",
"snowplowEntity": "CONTEXT"
}
]
```
For such scenario it looks like we skip necessary warehouse migration for self describing column. We only execute migration for the context:
```
INFO Migration: Migrating contexts_com_snowplowanalytics_snowplow_link_click_1 AddColumn(Fragment("ALTER TABLE atomic.events ADD COLUMN contexts_com_snowplowanalytics_snowplow_link_click_1 ARRAY"),List()) (pre-transaction)
```
but never for unstruct_event_com_snowplowanalytics_snowplow_link_click_1, which results in an error when inserting data to the table:
```
ERROR Error executing transaction. Sleeping for 30 seconds for the first time
net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error: error line 1 at position 1,779
invalid identifier 'UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_LINK_CLICK_1'
```
It seems to be caused by this [line](https://github.com/snowplow/snowplow-rdb-loader/blob/fffcbe460d7960714116aa7fe606a5ffbb4fd31d/modules/loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/discovery/DataDiscovery.scala#L213) where we group incoming types by the name and find max schema key per group. But name doesn’t contain unstruct/context prefix, so it’s possible to “lose” type when schema is used as both context/unstruct + with older version.
I think the solution here could be using full type name instead of simple short name. Full name takes into account if it’s context/unstruct, still groups types, but with additional prefix.
|
Thanks for your pull request. Is this your first contribution to a Snowplow open source project? Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://docs.snowplowanalytics.com/docs/contributing/contributor-license-agreement/ to learn more and sign. Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks. |
pondzix
force-pushed
the
release/6.1.4
branch
from
62ceab3 to
0512ecb
Compare
istreeter
approved these changes
Jan 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira ref: PDP-1669
Created by Github Actions