[fix-issue-2676] repair a snapshot-split bug: #2968
Conversation
|
Hi @AidenPerce, could you please rebase this PR with latest |
I have rebase the branch of this pr, but there are some problems in test-case of db2-connector, that caused i can't |
|
Hi @AidenPerce, sorry about the inconvenience. I think it's related to a glitch in Db2 incremental connector and should be fixed by #3283. Will try to get it merged asap. |
|
Hi @AidenPerce, the Db2 CI fix has been merged into |
resolved the conflict of org.apache.flink.cdc.connectors.mysql.source.utils.ObjectUtilsTest
resolved the conflict of org.apache.flink.cdc.connectors.mysql.source.utils.ObjectUtilsTest
I get a new problem, there is a tcase of mongodb with a failure: But i re-run the test-case successfully. Are there any issues with this test case? |
|
@yuxiqian may help to merge it please ? |
AidenPerce
changed the title
There was a problem hiding this comment.
Thanks for @AidenPerce's great work, I left some comments. Seems MongoDB CI isn't stable recently, but it's clearly irrelevant to this PR, no worries.
| JdbcConnection jdbcConnection, | ||
| TableId tableId, | ||
| String splitColumn) | ||
| throws SQLException { |
There was a problem hiding this comment.
Should this function throw an SQLException since there's a fallback logic (normal compare(obj1, obj2)?
| * @throws ClassCastException if the compared objects are not instance of {@link Comparable} or | ||
| * not <i>mutually comparable</i> (for example, strings and integers). | ||
| */ | ||
| @SuppressWarnings("unchecked") |
There was a problem hiding this comment.
Any reason for this suppress?
| * Object)}, we whill get: split1: ['0000','1111'] split2: ['3333','4444'] split3: | ||
| * ['aaaa','bbbb','cccc','ZZZZ'....] the split3 whill contains all of the remain rows |
There was a problem hiding this comment.
| * Object)}, we whill get: split1: ['0000','1111'] split2: ['3333','4444'] split3: | |
| * ['aaaa','bbbb','cccc','ZZZZ'....] the split3 whill contains all of the remain rows | |
| * Object)}, we will get: split1: ['0000','1111'] split2: ['3333','4444'] split3: | |
| * ['aaaa','bbbb','cccc','ZZZZ'....] the split3 will contains all of the remain rows |
| String columnCollation = | ||
| StatementUtils.queryColumnCollation(jdbcConnection, tableId, splitColumn); |
There was a problem hiding this comment.
IIUC dbCompare might be called many times during the chunk splitting process, and my concern is performing a JDBC query execution every time might greatly deteriorate the performance. Is it possible to implement a native comparison compatible with MySQL?
There was a problem hiding this comment.
Yeah, you are right. during the chunk splitting process, db queries have been changed from once to three times. But at this stage, I cannot fully cover different MySQL versions as well as various database character sets and collations. So I temporarily chose a relatively simple approach to solve it.
There was a problem hiding this comment.
Got it, this solution looks reasonable.
|
Hi @AidenPerce, is there any updates on this PR? Feel free to comment here if you need any help. |
|
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs. |
|
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs. |
|
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request. |
when use splitOneUnevenlySizedChunk, sometimes(for example, a database or a table or a column use 'utf8mb4_general_ci') we will get a big chunk. For example when the value of primaryKey like ['0000','1111','2222','3333','4444','aaaa','bbbb','cccc','ZZZZ',...] almost all of the values which starts with a [a-zA-z] will be split into a big chunk;