[FLINK-34689][MySQL][Feature] check binlog_row_value_options #3148

SML0127 · 2024-03-15T03:55:12Z

Implemented functino to check binlog_row_value_optoins within MySqlValidator.

When binlog_row_value_optoins is set to PARTIAL_JSON,
the update operator remains as Update_rows_partial.

Flink CDC does not parse this event because Update_row_partial binlog event is mapped to PARTIAL_UPDATE_ROWS_EVENT and Flink CDC do not handle that event type. Therefore, some update query-related binary logs are lost.

So, we have to check binlog_row_value_optoins before starting.

Implemented checkBinlogRowValueOptions

private final MySqlSourceConfig sourceConfig;
    public void validate() {
        checkVersion(connection);
        checkBinlogFormat(connection);
        checkBinlogRowImage(connection);
        checkBinlogRowValueOptions(connection);
        checkTimeZone(connection);
    } catch (SQLException ex) {
        throw new TableException(
    }
}

/** Check whether the binlog row value options is empty. */
private void checkBinlogRowValueOptions(JdbcConnection connection) throws SQLException {
    String rowValueOptions =
           connection
                    .queryAndMap(
                            "SHOW GLOBAL VARIABLES LIKE 'binlog_row_value_options'",
                            rs -> rs.next() ? rs.getString(2) : "")
                    .trim()
                    .toUpperCase();
    // This setting was introduced in MySQL 8.0+ with default of empty string ''
    // For older versions, assume empty string ''
    if (!rowValueOptions.equals(BINLOG_ROW_VALUE_OPTIONS)) {
        throw new ValidationException(
                String.format(
                       "The MySQL server is configured with binlog_row_value_options %s rather than %s, which is "
                                + "required for this connector to work properly. Change the MySQL configuration to use a "
                                + "binlog_row_image='' and restart the connector.",
                        rowValueOptions, BINLOG_ROW_VALUE_OPTIONS));
    }
}

SML0127 · 2024-03-15T03:59:11Z

jira issue: https://issues.apache.org/jira/browse/FLINK-34689

SML0127 · 2024-03-16T08:07:27Z

@leonardBang , @ruanhang1993 CC

leonardBang · 2024-03-18T05:40:17Z

Thanks @SML0127 for the contribution, @ruanhang1993 Would you like to help review this PR?

SML0127 · 2024-03-21T03:02:10Z

@ruanhang1993 PTAL

ruanhang1993

@SML0127 Hi. Sorry for the late review. I left a little comment. Thanks~

ruanhang1993 · 2024-04-10T07:43:37Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

+                            "The MySQL server is configured with binlog_row_value_options %s rather than %s, which is "
+                                    + "required for this connector to work properly. Change the MySQL configuration to use a "
+                                    + "binlog_row_image='' and restart the connector.",
+                            rowValueOptions, BINLOG_ROW_VALUE_OPTIONS));


I think this error message should be changed as follows.

The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events for the mysql cdc connector. Please remove the binlog_row_value_options setting in the MySQL server and rerun the job. See more details at https://dev.mysql.com/doc/refman/8.0/en/replication-features-json.html.

If I do not understand wrong, we do not need to change the binlog_row_image setting. Am I right ?

If I do not understand wrong, we do not need to change the binlog_row_image setting. Am I right ?

@ruanhang1993
I updated log message.

Yes. we do not need to change both settings binlog_row_image and binlog_row_value_options.

ruanhang1993

Hi, @SML0127.
I left some comments. Please take a look, thanks~

ruanhang1993 · 2024-04-11T02:55:30Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

@@ -50,6 +50,7 @@ public class MySqlValidator implements Validator {

    private static final String BINLOG_FORMAT_ROW = "ROW";
    private static final String BINLOG_FORMAT_IMAGE_FULL = "FULL";
+    private static final String BINLOG_ROW_VALUE_OPTIONS = "";


Suggested change

private static final String BINLOG_ROW_VALUE_OPTIONS = "";

private static final String DEFAULT_BINLOG_ROW_VALUE_OPTIONS = "";

ruanhang1993 · 2024-04-11T02:55:59Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

+                connection
+                        .queryAndMap(
+                                "SHOW GLOBAL VARIABLES LIKE 'binlog_row_value_options'",
+                                rs -> rs.next() ? rs.getString(2) : "")


Suggested change

rs -> rs.next() ? rs.getString(2) : "")

rs -> rs.next() ? rs.getString(2) : DEFAULT_BINLOG_ROW_VALUE_OPTIONS)

ruanhang1993 · 2024-04-11T02:57:32Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

+                        .toUpperCase();
+        // This setting was introduced in MySQL 8.0+ with default of empty string ''
+        // For older versions, assume empty string ''
+        if (!rowValueOptions.equals(BINLOG_ROW_VALUE_OPTIONS)) {


Suggested change

if (!rowValueOptions.equals(BINLOG_ROW_VALUE_OPTIONS)) {

if (!BINLOG_ROW_VALUE_OPTIONS.equals(rowValueOptions)) {

ruanhang1993 · 2024-04-11T02:57:47Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

+        if (!rowValueOptions.equals(BINLOG_ROW_VALUE_OPTIONS)) {
+            throw new ValidationException(
+                    String.format(
+                            "The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events"


Suggested change

"The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events"

"The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events "

ruanhang1993 · 2024-04-11T02:57:59Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

+                            "The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events"
+                                    + "for the mysql cdc connector. Please remove the binlog_row_value_options setting in the MySQL server and rerun the job."
+                                    + "See more details at https://dev.mysql.com/doc/refman/8.0/en/replication-features-json.html.",
+                            rowValueOptions, BINLOG_ROW_VALUE_OPTIONS));


Suggested change

rowValueOptions, BINLOG_ROW_VALUE_OPTIONS));

rowValueOptions));

…TIONS

SML0127 · 2024-04-11T04:35:28Z

Hi, @SML0127. I left some comments. Please take a look, thanks~

@ruanhang1993
Thank you for your comments!
I have applied all of your comments to the code.

SML0127 · 2024-04-11T04:44:41Z

...-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlValidator.java

    private static final String BINLOG_FORMAT_ROW = "ROW";
    private static final String BINLOG_FORMAT_IMAGE_FULL = "FULL";


@ruanhang1993
How about changing the variable names BINLOG_FORMAT_ROW, BINLOG_FORMAT_IMAGE_FULL to DEFAULT_BINLOG_FORMAT, DEFAULT_BINLOG_ROW_FORMAT?

Suggested change

private static final String BINLOG_FORMAT_ROW = "ROW";

private static final String BINLOG_FORMAT_IMAGE_FULL = "FULL";

private static final String DEFAULT_BINLOG_FORMAT = "ROW";

private static final String DEFAULT_BINLOG_ROW_FORMAT = "FULL";

In mysql 8, default values for the binlog_format and binlog_row_image options are ROW and full, respectively

https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_format

https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_image

Hi, @SML0127 .
I think we could not change these fields' names. These settings are checked by the ROW or FULL values.
But the new DEFAULT_BINLOG_ROW_VALUE_OPTIONS is used as the default value for mysql 5.7. And it is to check whether this setting is set.

Thank you for your explanation, @ruanhang1993
I undertand your responses. Do you have any other suggestions?

ruanhang1993

LGTM.

PatrickRen

@SML0127 Thanks for the contribution! LGTM

…3148)

ADD: implemented checkBinlogRowValueOptions within MySqlValidator

36eaa8f

github-actions bot added the mysql-cdc-connector label Mar 15, 2024

SML0127 changed the title ~~[MySQL][Feature] check binlog_row_value_optoins~~ [FLINK-34689][MySQL][Feature] check binlog_row_value_optoins Mar 15, 2024

ruanhang1993 reviewed Apr 10, 2024

View reviewed changes

EDIT: updated log message for checkBinlogRowValueOptions

ae1d27a

ruanhang1993 reviewed Apr 11, 2024

View reviewed changes

EDIT: renamed BINLOG_ROW_VALUE_OPTIONS -> DEFAULT_BINLOG_ROW_VALUE_OP…

7efd972

…TIONS

SML0127 commented Apr 11, 2024

View reviewed changes

ruanhang1993 approved these changes Apr 11, 2024

View reviewed changes

PatrickRen changed the title ~~[FLINK-34689][MySQL][Feature] check binlog_row_value_optoins~~ [FLINK-34689][MySQL][Feature] check binlog_row_value_options Apr 11, 2024

PatrickRen approved these changes Apr 12, 2024

View reviewed changes

PatrickRen merged commit af7665d into apache:master Apr 12, 2024
13 checks passed

wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024

[FLINK-34689][MySQL][Feature] check binlog_row_value_options (apache#…

ee8a2b8

…3148)

ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025

[FLINK-34689][MySQL][Feature] check binlog_row_value_options (apache#…

215c0fc

…3148)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-34689][MySQL][Feature] check binlog_row_value_options #3148

[FLINK-34689][MySQL][Feature] check binlog_row_value_options #3148

SML0127 commented Mar 15, 2024 •

edited

Loading

SML0127 commented Mar 15, 2024

SML0127 commented Mar 16, 2024

leonardBang commented Mar 18, 2024

SML0127 commented Mar 21, 2024

ruanhang1993 left a comment

ruanhang1993 Apr 10, 2024 •

edited

Loading

ruanhang1993 Apr 10, 2024

SML0127 Apr 10, 2024

ruanhang1993 left a comment

ruanhang1993 Apr 11, 2024

ruanhang1993 Apr 11, 2024

ruanhang1993 Apr 11, 2024

ruanhang1993 Apr 11, 2024

ruanhang1993 Apr 11, 2024

SML0127 commented Apr 11, 2024

SML0127 Apr 11, 2024 •

edited

Loading

ruanhang1993 Apr 11, 2024 •

edited

Loading

SML0127 Apr 11, 2024

ruanhang1993 left a comment

PatrickRen left a comment

	private static final String BINLOG_ROW_VALUE_OPTIONS = "";
	private static final String DEFAULT_BINLOG_ROW_VALUE_OPTIONS = "";

	rs -> rs.next() ? rs.getString(2) : "")
	rs -> rs.next() ? rs.getString(2) : DEFAULT_BINLOG_ROW_VALUE_OPTIONS)

	if (!rowValueOptions.equals(BINLOG_ROW_VALUE_OPTIONS)) {
	if (!BINLOG_ROW_VALUE_OPTIONS.equals(rowValueOptions)) {

	"The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events"
	"The MySQL server is configured with binlog_row_value_options=%s, which is possible to cause losing some binlog events "

	rowValueOptions, BINLOG_ROW_VALUE_OPTIONS));
	rowValueOptions));

		private static final String BINLOG_FORMAT_ROW = "ROW";
		private static final String BINLOG_FORMAT_IMAGE_FULL = "FULL";

[FLINK-34689][MySQL][Feature] check binlog_row_value_options #3148

[FLINK-34689][MySQL][Feature] check binlog_row_value_options #3148

Conversation

SML0127 commented Mar 15, 2024 • edited Loading

Implemented checkBinlogRowValueOptions

SML0127 commented Mar 15, 2024

SML0127 commented Mar 16, 2024

leonardBang commented Mar 18, 2024

SML0127 commented Mar 21, 2024

ruanhang1993 left a comment

Choose a reason for hiding this comment

ruanhang1993 Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruanhang1993 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SML0127 commented Apr 11, 2024

SML0127 Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

ruanhang1993 Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruanhang1993 left a comment

Choose a reason for hiding this comment

PatrickRen left a comment

Choose a reason for hiding this comment

SML0127 commented Mar 15, 2024 •

edited

Loading

ruanhang1993 Apr 10, 2024 •

edited

Loading

SML0127 Apr 11, 2024 •

edited

Loading

ruanhang1993 Apr 11, 2024 •

edited

Loading