[FLINK-35740][mysql] Allow column as chunk key even if not in Primary Keys #3448

SML0127 · 2024-07-02T09:24:50Z

Allow column as chunk key even if it is not in the primary keys.

There are cases where the primary key is not a numeric type, such as varchar or varbinary.
In this case, the distributed factor and chunk range may be calculated incorrectly, resulting in one chunk containing too many records.

But there was no conditions for restrict column type of primary keys or input chunk key, so this may cause out of memory in the flink task manager.

Actually, in our company, there was a mysql tables that PK is a varbinary column and the above situation occurred.

https://issues.apache.org/jira/browse/FLINK-35740

SML0127 · 2024-07-02T09:46:42Z

PTAL @leonardBang @lvyanquan @PatrickRen 🙇🏻‍♂️🙇🏻‍♂️

SML0127 · 2024-07-02T09:58:06Z

I was wondering whether to allow a column that is not a primary key as a chunk key.
But I think it would be better to relax the chunk key restrictions in the code, just whether or not input chunk key column is in the columns of table.
So It would be better to leave it up to the user to decide whether the correct chunk key has been passed.

yuxiqian

Perhaps testAssignCompositePkTableWithWrongChunkKeyColumn test case should be updated, too?

SML0127 · 2024-07-02T11:09:43Z

Perhaps testAssignCompositePkTableWithWrongChunkKeyColumn test case should be updated, too?

yes! I have to change those test code.

…WrongChunkKeyColumn

…lSnapshotSplitAssignerTest

…thChunkKeyColumn

…eWithoutPrimaryKeyWithChunkKeyColumn

SML0127 · 2024-07-02T15:56:05Z

Perhaps testAssignCompositePkTableWithWrongChunkKeyColumn test case should be updated, too?

yes! I have to change those test code.
@yuxiqian
I added some test code testAssignTableWithoutPrimaryKeyWithChunkKeyColumn

yuxiqian · 2024-07-04T02:48:14Z

Generally LGTM, but my concern is arbitrarily chosen columns can be risky, since chunking by columns without index is very slow.

I think we need to update related documentations and address warnings about this. Looking forward to thoughts from @SML0127.

SML0127 · 2024-07-04T09:33:27Z

Generally LGTM, but my concern is arbitrarily chosen columns can be risky, since chunking by columns without index is very slow.

I think we need to update related documentations and address warnings about this. Looking forward to thoughts from @SML0127.

From flink cdc v2.4.0, we have enabled to use incremental snapshots for tables without primary keys.
(https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/flink-sources/mysql-cdc/)
So I think it seems reasonable to allow column as chunk key not in primary keys.
But we also have to improve related documentations and address warnings as you suggested.

I will update documentations from this PR, too.

…heck splitKeyRange using chunkKeyStruct

…r table with primary key

SML0127 · 2024-07-09T03:30:25Z

@leonardBang PTAL

SML0127 · 2024-07-12T04:57:03Z

@leonardBang PTAL

@leonardBang @yuxiqian PTAL

yuxiqian

Looks good! Just left a minor comment about unit testing.

yuxiqian · 2024-07-12T05:18:00Z

...a/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssignerTest.java

+    @Test
+    public void testAssignTableWithoutPrimaryKeyWithChunkKeyColumn() {
+        String tableWithoutPrimaryKey = "customers_no_pk";
+        List<String> expected =
+                Arrays.asList(
+                        "customers_no_pk null [462]",
+                        "customers_no_pk [462] [823]",
+                        "customers_no_pk [823] [1184]",
+                        "customers_no_pk [1184] [1545]",
+                        "customers_no_pk [1545] [1906]",
+                        "customers_no_pk [1906] null");
+        List<String> splits =
+                getTestAssignSnapshotSplits(
+                        customerDatabase,
+                        4,
+                        CHUNK_KEY_EVEN_DISTRIBUTION_FACTOR_UPPER_BOUND.defaultValue(),
+                        CHUNK_KEY_EVEN_DISTRIBUTION_FACTOR_LOWER_BOUND.defaultValue(),
+                        new String[] {tableWithoutPrimaryKey},
+                        "id");
+        assertEquals(expected, splits);
+    }


It would be nice if we can also test using non-primary key columns as chunk keys for table with primary keys.

…yKey

…ColumnNotInPrimaryKey

…nNotInPrimaryKey

...a/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssignerTest.java

SML0127 · 2024-07-24T08:43:41Z

@leonardBang @ruanhang1993
PTAL~

leonardBang

@SML0127 Thanks for the update, looks generally to me, just one minor comment that could you update the doc under docs/content.zh directory as well?

docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md

SML0127 · 2024-07-24T10:05:19Z

@leonardBang
Thank you for comments. I modified documents with your suggestions!

leonardBang

Thanks @SML0127 for the update, LGTM

…'s not primary key This closes apache#3448.

EDIT: allowed column as chunk key even if not pk

94f6cc4

github-actions bot added the mysql-cdc-connector label Jul 2, 2024

SML0127 changed the title ~~Allow column as chunk key even if not in Primary Keys~~ [FLINK-35740][mysql] Allow column as chunk key even if not in Primary Keys Jul 2, 2024

yuxiqian reviewed Jul 2, 2024

View reviewed changes

louis.sml added 4 commits July 2, 2024 20:37

UPDATE: changed exception messsages in testAssignCompositePkTableWith…

3af8094

…WrongChunkKeyColumn

ADD: added testAssignTableWithoutPrimaryKeyWithChunkKeyColumn in MySq…

e4e6a2f

…lSnapshotSplitAssignerTest

FIX: passed chunk key column id in testAssignTableWithoutPrimaryKeyWi…

abfa237

…thChunkKeyColumn

FIX: passed database to getTestAssignSnapshotSplits in testAssignTabl…

3a0b0f2

…eWithoutPrimaryKeyWithChunkKeyColumn

FIX: fixed test code testAssignTableWithoutPrimaryKeyWithChunkKeyColumn

44db3a7

EDIT: pass binlogRecord.key() as keyStruct when hasPrimaryKey=true. c…

09e65e1

…heck splitKeyRange using chunkKeyStruct

SML0127 force-pushed the feature/allow_chunk_key_even_if_not_pk branch from 45106d4 to 09e65e1 Compare July 5, 2024 06:43

EDIT: removed comments that chunk key is restrcited to primary key fo…

3ac230f

…r table with primary key

github-actions bot added the base label Jul 5, 2024

Seung-Min Lee added 2 commits July 5, 2024 19:00

UPDATE: did spotlee:apply

0eb149f

UPDATE: added caution to use column not in priamry key as chunk key

8d5fb00

github-actions bot added the docs Improvements or additions to documentation label Jul 9, 2024

yuxiqian approved these changes Jul 12, 2024

View reviewed changes

Seung-Min Lee added 4 commits July 14, 2024 18:31

ADD: added testAssignTableWithPrimaryKeyWithChunkKeyColumnNotInPrimar…

f42f809

…yKey

EDIT: changed table name in testAssignTableWithPrimaryKeyWithChunkKey…

dc64490

…ColumnNotInPrimaryKey

EDIT: added @test

11c32b4

FIX: fixed expected in testAssignTableWithPrimaryKeyWithChunkKeyColum…

dfb036b

…nNotInPrimaryKey

SML0127 commented Jul 14, 2024

View reviewed changes

...a/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssignerTest.java Show resolved Hide resolved

leonardBang reviewed Jul 24, 2024

View reviewed changes

github-actions bot added the reviewed label Jul 24, 2024

EDIT: modified documents in content.zh

50bc0b7

SML0127 commented Jul 24, 2024

View reviewed changes

docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md Show resolved Hide resolved

leonardBang reviewed Jul 24, 2024

View reviewed changes

docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md Outdated Show resolved Hide resolved

docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md Show resolved Hide resolved

docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md Show resolved Hide resolved

Seung-Min Lee added 2 commits July 24, 2024 19:01

EDIT: applied comments to content.zh

941decb

EDIT: applied comments to content.zh

4332f21

leonardBang approved these changes Jul 24, 2024

View reviewed changes

github-actions bot added the approved label Jul 24, 2024

leonardBang merged commit 91fc7bb into apache:master Jul 24, 2024
19 checks passed

wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024

[FLINK-35740][cdc-connector][mysql] Allow column as chunk key even it…

cbf2b19

…'s not primary key This closes apache#3448.

qiaozongmi pushed a commit to qiaozongmi/flink-cdc that referenced this pull request Sep 23, 2024

[FLINK-35740][cdc-connector][mysql] Allow column as chunk key even it…

4496d31

…'s not primary key This closes apache#3448.

yuxiqian mentioned this pull request Nov 25, 2024

flink-cdc3.x版本不支持同步MySQL含有联合主键的表到doris中，任务会报异常并且停止 #3759

Closed

4 tasks

ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025

[FLINK-35740][cdc-connector][mysql] Allow column as chunk key even it…

d0b5136

…'s not primary key This closes apache#3448.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-35740][mysql] Allow column as chunk key even if not in Primary Keys #3448

[FLINK-35740][mysql] Allow column as chunk key even if not in Primary Keys #3448

SML0127 commented Jul 2, 2024 •

edited

Loading

SML0127 commented Jul 2, 2024 •

edited

Loading

SML0127 commented Jul 2, 2024

yuxiqian left a comment

SML0127 commented Jul 2, 2024

SML0127 commented Jul 2, 2024 •

edited

Loading

yuxiqian commented Jul 4, 2024

SML0127 commented Jul 4, 2024 •

edited

Loading

SML0127 commented Jul 9, 2024

SML0127 commented Jul 12, 2024

yuxiqian left a comment

yuxiqian Jul 12, 2024

SML0127 commented Jul 24, 2024

leonardBang left a comment

SML0127 commented Jul 24, 2024

leonardBang left a comment

[FLINK-35740][mysql] Allow column as chunk key even if not in Primary Keys #3448

[FLINK-35740][mysql] Allow column as chunk key even if not in Primary Keys #3448

Conversation

SML0127 commented Jul 2, 2024 • edited Loading

SML0127 commented Jul 2, 2024 • edited Loading

SML0127 commented Jul 2, 2024

yuxiqian left a comment

Choose a reason for hiding this comment

SML0127 commented Jul 2, 2024

SML0127 commented Jul 2, 2024 • edited Loading

yuxiqian commented Jul 4, 2024

SML0127 commented Jul 4, 2024 • edited Loading

SML0127 commented Jul 9, 2024

SML0127 commented Jul 12, 2024

yuxiqian left a comment

Choose a reason for hiding this comment

yuxiqian Jul 12, 2024

Choose a reason for hiding this comment

SML0127 commented Jul 24, 2024

leonardBang left a comment

Choose a reason for hiding this comment

SML0127 commented Jul 24, 2024

leonardBang left a comment

Choose a reason for hiding this comment

SML0127 commented Jul 2, 2024 •

edited

Loading

SML0127 commented Jul 2, 2024 •

edited

Loading

SML0127 commented Jul 2, 2024 •

edited

Loading

SML0127 commented Jul 4, 2024 •

edited

Loading