[FLINK-35774][cdc-runtime] Fix the cache of transform is not updated after process schema change event #3455
Conversation
|
@yuxiqian @lvyanquan PTAL |
There was a problem hiding this comment.
I'm a bit unsure if transformProjectionProcessorMap and transformFilterProcessorMap will be flushed when upstream schema change event arrives at TransformDataOperator?
Got it, it will be refreshed by updating tableInfo.
|
I have rebased this branch onto master. |
…after process schema change event when merging with route
| "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 11, 1], after=[], op=DELETE, meta=()}", | ||
| "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, 22, ], after=[2, 22, x], op=UPDATE, meta=()}"); |
There was a problem hiding this comment.
Is transform semantic changed here? Seems after the schema evolution, upstream schema will be [col1, newCol3], and downstream schema should be inferred as [col1, newCol3, col12] instead of [col1, col12, newCol3].
There was a problem hiding this comment.
Yes. Transform semantic has changed. The previous discussion resulted in the current code architecture being unable to achieve the expected semantics.
Based on the following code, it may be possible to achieve.
#3285
| "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, 10], after=[], op=DELETE, meta=()}", | ||
| "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, , 20], after=[2, x, 20], op=UPDATE, meta=()}"); | ||
| "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 10, 1], after=[], op=DELETE, meta=()}", | ||
| "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, 20, ], after=[2, 20, x], op=UPDATE, meta=()}"); |
There was a problem hiding this comment.
Could you please also add tests to verify if downstream schema (could be accessed by ValuesDatabase#getTableSchema) is as expected?
There was a problem hiding this comment.
I will have a try tonight.
|
The issue has been fixed by #3285. |
This closes FLINK-35774.