diff --git a/docs/content.zh/docs/core-concept/data-pipeline.md b/docs/content.zh/docs/core-concept/data-pipeline.md index 7b286bfbf5d..a1f1e0f5139 100644 --- a/docs/content.zh/docs/core-concept/data-pipeline.md +++ b/docs/content.zh/docs/core-concept/data-pipeline.md @@ -24,23 +24,23 @@ specific language governing permissions and limitations under the License. --> -# Definition -Since events in Flink CDC flow from the upstream to the downstream in a pipeline manner, the whole ETL task is referred as a **Data Pipeline**. +# 定义 +由于在 Flink CDC 中，事件从上游流转到下游遵循 Pipeline 的模式，因此整个 ETL 作业也被称为 **Data Pipeline**。 -# Parameters -A pipeline corresponds to a chain of operators in Flink. -To describe a Data Pipeline, the following parts are required: +# 参数 +一个 pipeline 包含着 Flink 的一组算子链。 +为了描述 Data Pipeline，我们需要定义以下部分： - [source]({{< ref "docs/core-concept/data-source" >}}) - [sink]({{< ref "docs/core-concept/data-sink" >}}) - [pipeline](#pipeline-configurations) -the following parts are optional: +下面 是 Data Pipeline 的一些可选配置： - [route]({{< ref "docs/core-concept/route" >}}) - [transform]({{< ref "docs/core-concept/transform" >}}) -# Example -## Only required -We could use following yaml file to define a concise Data Pipeline describing synchronize all tables under MySQL app_db database to Doris : +# 示例 +## 只包含必须部分 +我们可以使用以下 yaml 文件来定义一个简单的 Data Pipeline 来同步 MySQL app_db 数据库下的所有表到 Doris： ```yaml source: @@ -62,8 +62,8 @@ We could use following yaml file to define a concise Data Pipeline describing sy parallelism: 2 ``` -## With optional -We could use following yaml file to define a complicated Data Pipeline describing synchronize all tables under MySQL app_db database to Doris and give specific target database name ods_db and specific target table name prefix ods_ : +## 包含可选部分 +我们可以使用以下 yaml 文件来定义一个复杂的 Data Pipeline 来同步 MySQL app_db 数据库下的所有表到 Doris，并给目标数据库名 ods_db 和目标表名前缀 ods_： ```yaml source: @@ -108,11 +108,11 @@ We could use following yaml file to define a complicated Data Pipeline describin classpath: com.example.functions.FormatFunctionClass ``` -# Pipeline Configurations -The following config options of Data Pipeline level are supported: +# Pipeline 配置 +下面 是 Data Pipeline 的一些可选配置： -| parameter | meaning | optional/required | -|-----------------|-----------------------------------------------------------------------------------------|-------------------| -| name | The name of the pipeline, which will be submitted to the Flink cluster as the job name. | optional | -| parallelism | The global parallelism of the pipeline. Defaults to 1. | optional | -| local-time-zone | The local time zone defines current session time zone id. | optional | \ No newline at end of file +| 参数 | 含义 | optional/required | +|-----------------|---------------------------------------|-------------------| +| name | 这个 pipeline 的名称，会用在 Flink 集群中作为作业的名称。 | optional | +| parallelism | pipeline的全局并发度，默认值是1。 | optional | +| local-time-zone | 作业级别的本地时区。 | optional | \ No newline at end of file diff --git a/docs/content.zh/docs/core-concept/data-sink.md b/docs/content.zh/docs/core-concept/data-sink.md index 2dab1dc4a70..6faf2ad7426 100644 --- a/docs/content.zh/docs/core-concept/data-sink.md +++ b/docs/content.zh/docs/core-concept/data-sink.md @@ -24,21 +24,21 @@ specific language governing permissions and limitations under the License. --> -# Definition -**Data Sink** is used to apply schema changes and write change data to external systems. -A Data Sink can write to multiple tables simultaneously. +# 定义 +**Data Sink** 是 用来应用 schema 变更并写入 change data 到外部系统的组件。 +一个 Data Sink 可以同时写入多个表。 -# Parameters -To describe a data sink, the follows are required: +# 参数 +为了定义一个 Data Sink，需要提供以下参数： -| parameter | meaning | optional/required | -|-----------------------------|-------------------------------------------------------------------------------------------------|-------------------| -| type | The type of the sink, such as doris or starrocks. | required | -| name | The name of the sink, which is user-defined (a default value provided). | optional | -| configurations of Data Sink | Configurations to build the Data Sink e.g. connection configurations and sink table properties. | optional | +| 参数 | 含义 | optional/required | +|-----------------------------|---------------------------------|-------------------| +| type | sink 的类型，例如 doris 或者 starrocks。 | required | +| name | sink 的名称，允许用户配置 (提供了一个默认值)。 | optional | +| configurations of Data Sink | 用于构建 sink 组件的配置，例如连接参数或者表属性的配置。 | optional | -# Example -We could use this yaml file to define a doris sink: +# 示例 +我们可以使用以下的 yaml 文件来定义一个 doris sink： ```yaml sink: type: doris diff --git a/docs/content.zh/docs/core-concept/data-source.md b/docs/content.zh/docs/core-concept/data-source.md index 5d6c33deb82..e18bf6e10be 100644 --- a/docs/content.zh/docs/core-concept/data-source.md +++ b/docs/content.zh/docs/core-concept/data-source.md @@ -24,21 +24,21 @@ specific language governing permissions and limitations under the License. --> -# Definition -**Data Source** is used to access metadata and read the changed data from external systems. -A Data Source can read data from multiple tables simultaneously. +# 定义 +**Data Source** 是 用来访问元数据以及从外部系统读取变更数据的组件。 +一个 Data Source 可以同时访问多个表。 -# Parameters -To describe a data source, the follows are required: +# 参数 +为了定义一个 Data Source，需要提供以下参数： -| parameter | meaning | optional/required | -|-------------------------------|-----------------------------------------------------------------------------------------------------|-------------------| -| type | The type of the source, such as mysql. | required | -| name | The name of the source, which is user-defined (a default value provided). | optional | -| configurations of Data Source | Configurations to build the Data Source e.g. connection configurations and source table properties. | optional | +| 参数 | 含义 | optional/required | +|-------------------------------|-----------------------------------|-------------------| +| type | source 的类型，例如 mysql。 | required | +| name | source 的名称，允许用户配置 (提供了一个默认值)。 | optional | +| configurations of Data Source | 用于构建 source 组件的配置，例如连接参数或者表属性的配置。 | optional | -# Example -We could use yaml files to define a mysql source: +# 示例 +我们可以使用yaml文件来定义一个mysql source： ```yaml source: type: mysql diff --git a/docs/content.zh/docs/core-concept/route.md b/docs/content.zh/docs/core-concept/route.md index ca5855d5861..fa1a9535309 100644 --- a/docs/content.zh/docs/core-concept/route.md +++ b/docs/content.zh/docs/core-concept/route.md @@ -24,24 +24,24 @@ specific language governing permissions and limitations under the License. --> -# Definition -**Route** specifies the rule of matching a list of source-table and mapping to sink-table. The most typical scenario is the merge of sub-databases and sub-tables, routing multiple upstream source tables to the same sink table. +# 定义 +**Route** 代表一个路由规则，用来匹配一个或多个source 表，并映射到 sink 表。最常见的场景是合并子数据库和子表，将多个上游源表路由到同一个目标表。 -# Parameters -To describe a route, the follows are required: +# 参数 +为了定义一个路由规则，需要提供以下参数： -| parameter | meaning | optional/required | -|----------------|---------------------------------------------------------------------------------------------|-------------------| -| source-table | Source table id, supports regular expressions | required | -| sink-table | Sink table id, supports symbol replacement | required | -| replace-symbol | Special symbol in sink-table for pattern replacing, will be replaced by original table name | optional | -| description | Routing rule description(a default value provided) | optional | +| 参数 | 含义 | optional/required | +|----------------|------------------------------------------|-------------------| +| source-table | Source 的 table id， 支持正则表达式 | required | +| sink-table | Sink 的 table id，支持符号替换 | required | +| replace-symbol | 用于在 sink-table 中进行模式替换的特殊字符串， 会被源表中的表名替换 | optional | +| description | Route 规则的描述(提供了一个默认描述) | optional | -A route module can contain a list of source-table/sink-table rules. +一个 Route 模块可以包含一个或多个 source-table/sink-table 规则。 -# Example -## Route one Data Source table to one Data Sink table -if synchronize the table `web_order` in the database `mydb` to a Doris table `ods_web_order`, we can use this yaml file to define this route： +# 示例 +## 路由一个 Data Source 表到一个 Data Sink 表 +如果同步一个 `mydb` 数据库中的 `web_order` 表到一个相同库的 `ods_web_order` 表，我们可以使用下面的 yaml 文件来定义这个路由： ```yaml route: @@ -50,17 +50,16 @@ route: description: sync table to one destination table with given prefix ods_ ``` -## Route multiple Data Source tables to one Data Sink table -What's more, if you want to synchronize the sharding tables in the database `mydb` to a Doris table `ods_web_order`, we can use this yaml file to define this route： +## 路由多个 Data Source 表到一个 Data Sink 表 +更进一步的，如果同步一个 `mydb` 数据库中的多个分表到一个相同库的 `ods_web_order` 表，我们可以使用下面的 yaml 文件来定义这个路由： ```yaml route: - source-table: mydb\.* sink-table: mydb.ods_web_order description: sync sharding tables to one destination table ``` - -## Complex Route via combining route rules -What's more, if you want to specify many different mapping rules, we can use this yaml file to define this route： +## 使用多个路由规则 +更进一步的，如果需要定义多个路由规则，我们可以使用下面的 yaml 文件来定义这个路由： ```yaml route: - source-table: mydb.orders @@ -74,9 +73,9 @@ route: description: sync products table to ods_products ``` -## Pattern Replacement in routing rules +## 包含符号替换的路由规则 -If you'd like to route source tables and rename them to sink tables with specific patterns, `replace-symbol` could be used to resemble source table names like this: +如果你想将源表路由到 sink 表，并使用特定的模式替换源表名，那么 `replace-symbol` 就可以做到这一点： ```yaml route: @@ -86,4 +85,4 @@ route: description: route all tables in source_db to sink_db ``` -Then, all tables including `source_db.XXX` will be routed to `sink_db.XXX` without hassle. \ No newline at end of file +然后，`source_db` 库下所有的表都会被同步到 `sink_db` 库下。 \ No newline at end of file diff --git a/docs/content.zh/docs/core-concept/table-id.md b/docs/content.zh/docs/core-concept/table-id.md index 261c8fd09b3..2908e12a607 100644 --- a/docs/content.zh/docs/core-concept/table-id.md +++ b/docs/content.zh/docs/core-concept/table-id.md @@ -24,17 +24,17 @@ specific language governing permissions and limitations under the License. --> -# Definition -When connecting to external systems, it is necessary to establish a mapping relationship with the storage objects of the external system. This is what **Table Id** refers to. +# 定义 +在连接外部系统时，有必要建立一个与外部系统存储对象（例如表）的映射关系。这就是 **Table Id** 所代表的含义。 -# Example -To be compatible with most external systems, the Table Id is represented by a 3-tuple : (namespace, schemaName, tableName). -Connectors should establish the mapping between Table Id and storage objects in external systems. +# 示例 +为了兼容大部分外部系统，Table Id 被表示为 3 元组：(namespace, schemaName, tableName)。 +连接器应该在连接外部系统时建立与外部系统存储对象的映射关系。 -The following table lists the parts in table Id of different data systems: +下面是不同数据系统对应的 tableId 的格式： -| data system | parts in tableId | String example | -|-----------------------|--------------------------|---------------------| -| Oracle/PostgreSQL | database, schema, table | mydb.default.orders | -| MySQL/Doris/StarRocks | database, table | mydb.orders | -| Kafka | topic | orders | +| 数据系统 | tableId 的组成 | 字符串示例 | +|-----------------------|-------------------------|---------------------| +| Oracle/PostgreSQL | database, schema, table | mydb.default.orders | +| MySQL/Doris/StarRocks | database, table | mydb.orders | +| Kafka | topic | orders | diff --git a/docs/content.zh/docs/core-concept/transform.md b/docs/content.zh/docs/core-concept/transform.md index 787edc7c206..190bcbd31fd 100644 --- a/docs/content.zh/docs/core-concept/transform.md +++ b/docs/content.zh/docs/core-concept/transform.md @@ -24,33 +24,33 @@ specific language governing permissions and limitations under the License. --> -# Definition -**Transform** module helps users delete and expand data columns based on the data columns in the table. -What's more, it also helps users filter some unnecessary data during the synchronization process. - -# Parameters -To describe a transform rule, the following parameters can be used: - -| Parameter | Meaning | Optional/Required | -|--------------|----------------------------------------------------|-------------------| -| source-table | Source table id, supports regular expressions | required | -| projection | Projection rule, supports syntax similar to the select clause in SQL | optional | -| filter | Filter rule, supports syntax similar to the where clause in SQL | optional | -| primary-keys | Sink table primary keys, separated by commas | optional | -| partition-keys | Sink table partition keys, separated by commas | optional | -| table-options | used to the configure table creation statement when automatically creating tables | optional | +# 定义 +**Transform** 模块允许用户根据数据列对数据列进行删除和扩展。 +更进一步的，这也也可以帮助用户在同步过程中过滤一些不需要的数据。 + +# 参数 +为了定义一个transform规则，可以使用以下参数： + +| 参数 | 含义 | Optional/Required | +|---------------------------|-----------------------------------------------------------------------------------|-------------------| +| source-table | Source table id, supports regular expressions | required | +| projection | Projection rule, supports syntax similar to the select clause in SQL | optional | +| filter | Filter rule, supports syntax similar to the where clause in SQL | optional | +| primary-keys | Sink table primary keys, separated by commas | optional | +| partition-keys | Sink table partition keys, separated by commas | optional | +| table-options | used to the configure table creation statement when automatically creating tables | optional | | converter-after-transform | used to add a converter to change DataChangeEvent after transform | optional | -| description | Transform rule description | optional | +| description | Transform rule description | optional | -Multiple rules can be declared in one single pipeline YAML file. +多个transform规则可以声明在一个pipeline YAML文件中。 ## converter-after-transform -`converter-after-transform` is used to change the DataChangeEvent after other transform. The available values of this options are as follows. +`converter-after-transform` 用于在其他 transform 处理之后再修改 DataChangeEvent。可用的值如下： -- SOFT_DELETE: The delete event will be converted as an insert event. This converter should be used together with the metadata `__data_event_type__`. Then you can implement the soft delete. +- SOFT_DELETE: 删除事件会被转换成插入事件。这个转换器应该与元数据 `__data_event_type__` 一起使用，然后你可以实现软删除。 -For example, the following transform will not delete data when the delete event happens. Instead it will update the column `op_type` to -D in sink and transform it to an insert record. +举个例子，下面的 transform 将不会删除数据，而是更新列 `op_type` 为 -D，然后转换成插入记录。 ```yaml transform: @@ -59,9 +59,9 @@ transform: converter-after-transform: SOFT_DELETE ``` -# Metadata Fields -## Fields definition -There are some hidden columns used to access metadata information. They will only take effect when explicitly referenced in the transform rules. +# 元数据字段 +## 字段定义 +有一些隐藏列可以用来访问元数据信息。它们仅在显式引用在 transform 规则中时才会生效。 | Field | Data Type | Description | |---------------------|-----------|----------------------------------------------| @@ -83,98 +83,98 @@ There are some hidden columns used to access metadata information. They will onl | StarRocks | Database | - | Table | | Doris | Database | - | Table | -# Functions -Flink CDC uses [Calcite](https://calcite.apache.org/) to parse expressions and [Janino script](https://www.janino.net/) to evaluate expressions with function call. - -## Comparison Functions - -| Function | Janino Code | Description | -|----------------------|-----------------------------|-----------------------------------------------------------------| -| value1 = value2 | valueEquals(value1, value2) | Returns TRUE if value1 is equal to value2; returns FALSE if value1 or value2 is NULL. | -| value1 <> value2 | !valueEquals(value1, value2) | Returns TRUE if value1 is not equal to value2; returns FALSE if value1 or value2 is NULL. | -| value1 > value2 | greaterThan(value1, value2) | Returns TRUE if value1 is greater than value2; returns FALSE if value1 or value2 is NULL. | -| value1 >= value2 | greaterThanOrEqual(value1, value2) | Returns TRUE if value1 is greater than or equal to value2; returns FALSE if value1 or value2 is NULL. | -| value1 < value2 | lessThan(value1, value2) | Returns TRUE if value1 is less than value2; returns FALSE if value1 or value2 is NULL. | -| value1 <= value2 | lessThanOrEqual(value1, value2) | Returns TRUE if value1 is less than or equal to value2; returns FALSE if value1 or value2 is NULL. | -| value IS NULL | null == value | Returns TRUE if value is NULL. | -| value IS NOT NULL | null != value | Returns TRUE if value is not NULL. | -| value1 BETWEEN value2 AND value3 | betweenAsymmetric(value1, value2, value3) | Returns TRUE if value1 is greater than or equal to value2 and less than or equal to value3. | -| value1 NOT BETWEEN value2 AND value3 | notBetweenAsymmetric(value1, value2, value3) | Returns TRUE if value1 is less than value2 or greater than value3. | -| string1 LIKE string2 | like(string1, string2) | Returns TRUE if string1 matches pattern string2. | -| string1 NOT LIKE string2 | notLike(string1, string2) | Returns TRUE if string1 does not match pattern string2. | -| value1 IN (value2 [, value3]* ) | in(value1, value2 [, value3]*) | Returns TRUE if value1 exists in the given list (value2, value3, …). | -| value1 NOT IN (value2 [, value3]* ) | notIn(value1, value2 [, value3]*) | Returns TRUE if value1 does not exist in the given list (value2, value3, …). | - -## Logical Functions - -| Function | Janino Code | Description | -|----------------------|-----------------------------|-----------------------------------------------------------------| -| boolean1 OR boolean2 | boolean1 || boolean2 | Returns TRUE if BOOLEAN1 is TRUE or BOOLEAN2 is TRUE. | -| boolean1 AND boolean2 | boolean1 && boolean2 | Returns TRUE if BOOLEAN1 and BOOLEAN2 are both TRUE. | -| NOT boolean | !boolean | Returns TRUE if boolean is FALSE; returns FALSE if boolean is TRUE. | -| boolean IS FALSE | false == boolean | Returns TRUE if boolean is FALSE; returns FALSE if boolean is TRUE. | -| boolean IS NOT FALSE | true == boolean | Returns TRUE if BOOLEAN is TRUE; returns FALSE if BOOLEAN is FALSE. | -| boolean IS TRUE | true == boolean | Returns TRUE if BOOLEAN is TRUE; returns FALSE if BOOLEAN is FALSE. | -| boolean IS NOT TRUE | false == boolean | Returns TRUE if boolean is FALSE; returns FALSE if boolean is TRUE. | - -## Arithmetic Functions - -| Function | Janino Code | Description | -|----------------------|-----------------------------|-----------------------------------------------------------------| -| numeric1 + numeric2 | numeric1 + numeric2 | Returns NUMERIC1 plus NUMERIC2. | -| numeric1 - numeric2 | numeric1 - numeric2 | Returns NUMERIC1 minus NUMERIC2. | -| numeric1 * numeric2 | numeric1 * numeric2 | Returns NUMERIC1 multiplied by NUMERIC2. | -| numeric1 / numeric2 | numeric1 / numeric2 | Returns NUMERIC1 divided by NUMERIC2. | -| numeric1 % numeric2 | numeric1 % numeric2 | Returns the remainder (modulus) of numeric1 divided by numeric2. | -| ABS(numeric) | abs(numeric) | Returns the absolute value of numeric. | -| CEIL(numeric) | ceil(numeric) | Rounds numeric up, and returns the smallest number that is greater than or equal to numeric. | -| FLOOR(numeric) | floor(numeric) | Rounds numeric down, and returns the largest number that is less than or equal to numeric. | -| ROUND(numeric, int) | round(numeric) | Returns a number rounded to INT decimal places for NUMERIC. | -| UUID() | uuid() | Returns an UUID (Universally Unique Identifier) string (e.g., "3d3c68f7-f608-473f-b60c-b0c44ad4cc4e") according to RFC 4122 type 4 (pseudo randomly generated) UUID. | - -## String Functions - -| Function | Janino Code | Description | -| -------------------- | ------------------------ | ------------------------------------------------- | -| string1 || string2 | concat(string1, string2) | Returns the concatenation of STRING1 and STRING2. | -| CHAR_LENGTH(string) | charLength(string) | Returns the number of characters in STRING. | -| UPPER(string) | upper(string) | Returns string in uppercase. | -| LOWER(string) | lower(string) | Returns string in lowercase. | -| TRIM(string1) | trim('BOTH',string1) | Returns a string that removes whitespaces at both sides. | -| REGEXP_REPLACE(string1, string2, string3) | regexpReplace(string1, string2, string3) | Returns a string from STRING1 with all the substrings that match a regular expression STRING2 consecutively being replaced with STRING3. E.g., 'foobar'.regexpReplace('oo\|ar', '') returns "fb". | -| SUBSTR(string, integer1[, integer2]) | substr(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). | -| SUBSTRING(string FROM integer1 [ FOR integer2 ]) | substring(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). | -| CONCAT(string1, string2,…) | concat(string1, string2,…) | Returns a string that concatenates string1, string2, …. E.g., CONCAT('AA', 'BB', 'CC') returns 'AABBCC'. | - -## Temporal Functions - -| Function | Janino Code | Description | -| -------------------- | ------------------------ | ------------------------------------------------- | -| LOCALTIME | localtime() | Returns the current SQL time in the local time zone, the return type is TIME(0). | -| LOCALTIMESTAMP | localtimestamp() | Returns the current SQL timestamp in local time zone, the return type is TIMESTAMP(3). | -| CURRENT_TIME | currentTime() | Returns the current SQL time in the local time zone, this is a synonym of LOCAL_TIME. | -| CURRENT_DATE | currentDate() | Returns the current SQL date in the local time zone. | -| CURRENT_TIMESTAMP | currentTimestamp() | Returns the current SQL timestamp in the local time zone, the return type is TIMESTAMP_LTZ(3). | -| NOW() | now() | Returns the current SQL timestamp in the local time zone, this is a synonym of CURRENT_TIMESTAMP. | -| DATE_FORMAT(timestamp, string) | dateFormat(timestamp, string) | Converts timestamp to a value of string in the format specified by the date format string. The format string is compatible with Java's SimpleDateFormat. | -| TIMESTAMPADD(timeintervalunit, interval, timepoint) | timestampadd(timeintervalunit, interval, timepoint) | Returns the timestamp of timepoint2 after timepoint added interval. The unit for the interval is given by the first argument, which should be one of the following values: SECOND, MINUTE, HOUR, DAY, MONTH, or YEAR. | -| TIMESTAMPDIFF(timepointunit, timepoint1, timepoint2) | timestampDiff(timepointunit, timepoint1, timepoint2) | Returns the (signed) number of timepointunit between timepoint1 and timepoint2. The unit for the interval is given by the first argument, which should be one of the following values: SECOND, MINUTE, HOUR, DAY, MONTH, or YEAR. | -| TO_DATE(string1[, string2]) | toDate(string1[, string2]) | Converts a date string string1 with format string2 (by default 'yyyy-MM-dd') to a date. | -| TO_TIMESTAMP(string1[, string2]) | toTimestamp(string1[, string2]) | Converts date time string string1 with format string2 (by default: 'yyyy-MM-dd HH:mm:ss') to a timestamp, without time zone. | -| FROM_UNIXTIME(numeric[, string]) | fromUnixtime(NUMERIC[, STRING]) | Returns a representation of the numeric argument as a value in string format (default is ‘yyyy-MM-dd HH:mm:ss’). numeric is an internal timestamp value representing seconds since ‘1970-01-01 00:00:00’ UTC, such as produced by the UNIX_TIMESTAMP() function. The return value is expressed in the session time zone (specified in TableConfig). E.g., FROM_UNIXTIME(44) returns ‘1970-01-01 00:00:44’ if in UTC time zone, but returns ‘1970-01-01 09:00:44’ if in ‘Asia/Tokyo’ time zone. | -| UNIX_TIMESTAMP() | unixTimestamp() | Gets current Unix timestamp in seconds. This function is not deterministic which means the value would be recalculated for each record. | -| UNIX_TIMESTAMP(string1[, string2]) | unixTimestamp(STRING1[, STRING2]) | Converts a date time string string1 with format string2 (by default: yyyy-MM-dd HH:mm:ss if not specified) to Unix timestamp (in seconds), using the specified timezone in table config.
If a time zone is specified in the date time string and parsed by UTC+X format such as “yyyy-MM-dd HH:mm:ss.SSS X”, this function will use the specified timezone in the date time string instead of the timezone in table config. If the date time string can not be parsed, the default value Long.MIN_VALUE(-9223372036854775808) will be returned.| - -## Conditional Functions - -| Function | Janino Code | Description | -| -------------------- | ------------------------ | ------------------------------------------------- | -| CASE value WHEN value1_1 [, value1_2]* THEN RESULT1 (WHEN value2_1 [, value2_2 ]* THEN result_2)* (ELSE result_z) END | Nested ternary expression | Returns resultX when the first time value is contained in (valueX_1, valueX_2, …). When no value matches, returns result_z if it is provided and returns NULL otherwise. | -| CASE WHEN condition1 THEN result1 (WHEN condition2 THEN result2)* (ELSE result_z) END | Nested ternary expression | Returns resultX when the first conditionX is met. When no condition is met, returns result_z if it is provided and returns NULL otherwise. | -| COALESCE(value1 [, value2]*) | coalesce(Object... objects) | Returns the first argument that is not NULL.If all arguments are NULL, it returns NULL as well. The return type is the least restrictive, common type of all of its arguments. The return type is nullable if all arguments are nullable as well. | -| IF(condition, true_value, false_value) | condition ? true_value : false_value | Returns the true_value if condition is met, otherwise false_value. E.g., IF(5 > 3, 5, 3) returns 5. | - -## Casting Functions +# 函数 +Flink CDC 使用 [Calcite](https://calcite.apache.org/) 来解析表达式并且使用 [Janino script](https://www.janino.net/) 来执行表达式。 + +## 比较函数 + +| Function | Janino Code | Description | +|--------------------------------------|----------------------------------------------|-------------------------------------------------------------------------------------------------------| +| value1 = value2 | valueEquals(value1, value2) | Returns TRUE if value1 is equal to value2; returns FALSE if value1 or value2 is NULL. | +| value1 <> value2 | !valueEquals(value1, value2) | Returns TRUE if value1 is not equal to value2; returns FALSE if value1 or value2 is NULL. | +| value1 > value2 | greaterThan(value1, value2) | Returns TRUE if value1 is greater than value2; returns FALSE if value1 or value2 is NULL. | +| value1 >= value2 | greaterThanOrEqual(value1, value2) | Returns TRUE if value1 is greater than or equal to value2; returns FALSE if value1 or value2 is NULL. | +| value1 < value2 | lessThan(value1, value2) | Returns TRUE if value1 is less than value2; returns FALSE if value1 or value2 is NULL. | +| value1 <= value2 | lessThanOrEqual(value1, value2) | Returns TRUE if value1 is less than or equal to value2; returns FALSE if value1 or value2 is NULL. | +| value IS NULL | null == value | Returns TRUE if value is NULL. | +| value IS NOT NULL | null != value | Returns TRUE if value is not NULL. | +| value1 BETWEEN value2 AND value3 | betweenAsymmetric(value1, value2, value3) | Returns TRUE if value1 is greater than or equal to value2 and less than or equal to value3. | +| value1 NOT BETWEEN value2 AND value3 | notBetweenAsymmetric(value1, value2, value3) | Returns TRUE if value1 is less than value2 or greater than value3. | +| string1 LIKE string2 | like(string1, string2) | Returns TRUE if string1 matches pattern string2. | +| string1 NOT LIKE string2 | notLike(string1, string2) | Returns TRUE if string1 does not match pattern string2. | +| value1 IN (value2 [, value3]* ) | in(value1, value2 [, value3]*) | Returns TRUE if value1 exists in the given list (value2, value3, …). | +| value1 NOT IN (value2 [, value3]* ) | notIn(value1, value2 [, value3]*) | Returns TRUE if value1 does not exist in the given list (value2, value3, …). | + +## 逻辑函数 + +| Function | Janino Code | Description | +|-----------------------|--------------------------------|---------------------------------------------------------------------| +| boolean1 OR boolean2 | boolean1 || boolean2 | Returns TRUE if BOOLEAN1 is TRUE or BOOLEAN2 is TRUE. | +| boolean1 AND boolean2 | boolean1 && boolean2 | Returns TRUE if BOOLEAN1 and BOOLEAN2 are both TRUE. | +| NOT boolean | !boolean | Returns TRUE if boolean is FALSE; returns FALSE if boolean is TRUE. | +| boolean IS FALSE | false == boolean | Returns TRUE if boolean is FALSE; returns FALSE if boolean is TRUE. | +| boolean IS NOT FALSE | true == boolean | Returns TRUE if BOOLEAN is TRUE; returns FALSE if BOOLEAN is FALSE. | +| boolean IS TRUE | true == boolean | Returns TRUE if BOOLEAN is TRUE; returns FALSE if BOOLEAN is FALSE. | +| boolean IS NOT TRUE | false == boolean | Returns TRUE if boolean is FALSE; returns FALSE if boolean is TRUE. | + +## 数学函数 + +| Function | Janino Code | Description | +|---------------------|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| numeric1 + numeric2 | numeric1 + numeric2 | Returns NUMERIC1 plus NUMERIC2. | +| numeric1 - numeric2 | numeric1 - numeric2 | Returns NUMERIC1 minus NUMERIC2. | +| numeric1 * numeric2 | numeric1 * numeric2 | Returns NUMERIC1 multiplied by NUMERIC2. | +| numeric1 / numeric2 | numeric1 / numeric2 | Returns NUMERIC1 divided by NUMERIC2. | +| numeric1 % numeric2 | numeric1 % numeric2 | Returns the remainder (modulus) of numeric1 divided by numeric2. | +| ABS(numeric) | abs(numeric) | Returns the absolute value of numeric. | +| CEIL(numeric) | ceil(numeric) | Rounds numeric up, and returns the smallest number that is greater than or equal to numeric. | +| FLOOR(numeric) | floor(numeric) | Rounds numeric down, and returns the largest number that is less than or equal to numeric. | +| ROUND(numeric, int) | round(numeric) | Returns a number rounded to INT decimal places for NUMERIC. | +| UUID() | uuid() | Returns an UUID (Universally Unique Identifier) string (e.g., "3d3c68f7-f608-473f-b60c-b0c44ad4cc4e") according to RFC 4122 type 4 (pseudo randomly generated) UUID. | + +## 字符串函数 + +| Function | Janino Code | Description | +|--------------------------------------------------|------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| string1 || string2 | concat(string1, string2) | Returns the concatenation of STRING1 and STRING2. | +| CHAR_LENGTH(string) | charLength(string) | Returns the number of characters in STRING. | +| UPPER(string) | upper(string) | Returns string in uppercase. | +| LOWER(string) | lower(string) | Returns string in lowercase. | +| TRIM(string1) | trim('BOTH',string1) | Returns a string that removes whitespaces at both sides. | +| REGEXP_REPLACE(string1, string2, string3) | regexpReplace(string1, string2, string3) | Returns a string from STRING1 with all the substrings that match a regular expression STRING2 consecutively being replaced with STRING3. E.g., 'foobar'.regexpReplace('oo\|ar', '') returns "fb". | +| SUBSTR(string, integer1[, integer2]) | substr(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). | +| SUBSTRING(string FROM integer1 [ FOR integer2 ]) | substring(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). | +| CONCAT(string1, string2,…) | concat(string1, string2,…) | Returns a string that concatenates string1, string2, …. E.g., CONCAT('AA', 'BB', 'CC') returns 'AABBCC'. | + +## 时间函数 + +| Function | Janino Code | Description | +|------------------------------------------------------|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| LOCALTIME | localtime() | Returns the current SQL time in the local time zone, the return type is TIME(0). | +| LOCALTIMESTAMP | localtimestamp() | Returns the current SQL timestamp in local time zone, the return type is TIMESTAMP(3). | +| CURRENT_TIME | currentTime() | Returns the current SQL time in the local time zone, this is a synonym of LOCAL_TIME. | +| CURRENT_DATE | currentDate() | Returns the current SQL date in the local time zone. | +| CURRENT_TIMESTAMP | currentTimestamp() | Returns the current SQL timestamp in the local time zone, the return type is TIMESTAMP_LTZ(3). | +| NOW() | now() | Returns the current SQL timestamp in the local time zone, this is a synonym of CURRENT_TIMESTAMP. | +| DATE_FORMAT(timestamp, string) | dateFormat(timestamp, string) | Converts timestamp to a value of string in the format specified by the date format string. The format string is compatible with Java's SimpleDateFormat. | +| TIMESTAMPADD(timeintervalunit, interval, timepoint) | timestampadd(timeintervalunit, interval, timepoint) | Returns the timestamp of timepoint2 after timepoint added interval. The unit for the interval is given by the first argument, which should be one of the following values: SECOND, MINUTE, HOUR, DAY, MONTH, or YEAR. | +| TIMESTAMPDIFF(timepointunit, timepoint1, timepoint2) | timestampDiff(timepointunit, timepoint1, timepoint2) | Returns the (signed) number of timepointunit between timepoint1 and timepoint2. The unit for the interval is given by the first argument, which should be one of the following values: SECOND, MINUTE, HOUR, DAY, MONTH, or YEAR. | +| TO_DATE(string1[, string2]) | toDate(string1[, string2]) | Converts a date string string1 with format string2 (by default 'yyyy-MM-dd') to a date. | +| TO_TIMESTAMP(string1[, string2]) | toTimestamp(string1[, string2]) | Converts date time string string1 with format string2 (by default: 'yyyy-MM-dd HH:mm:ss') to a timestamp, without time zone. | +| FROM_UNIXTIME(numeric[, string]) | fromUnixtime(NUMERIC[, STRING]) | Returns a representation of the numeric argument as a value in string format (default is ‘yyyy-MM-dd HH:mm:ss’). numeric is an internal timestamp value representing seconds since ‘1970-01-01 00:00:00’ UTC, such as produced by the UNIX_TIMESTAMP() function. The return value is expressed in the session time zone (specified in TableConfig). E.g., FROM_UNIXTIME(44) returns ‘1970-01-01 00:00:44’ if in UTC time zone, but returns ‘1970-01-01 09:00:44’ if in ‘Asia/Tokyo’ time zone. | +| UNIX_TIMESTAMP() | unixTimestamp() | Gets current Unix timestamp in seconds. This function is not deterministic which means the value would be recalculated for each record. | +| UNIX_TIMESTAMP(string1[, string2]) | unixTimestamp(STRING1[, STRING2]) | Converts a date time string string1 with format string2 (by default: yyyy-MM-dd HH:mm:ss if not specified) to Unix timestamp (in seconds), using the specified timezone in table config.
If a time zone is specified in the date time string and parsed by UTC+X format such as “yyyy-MM-dd HH:mm:ss.SSS X”, this function will use the specified timezone in the date time string instead of the timezone in table config. If the date time string can not be parsed, the default value Long.MIN_VALUE(-9223372036854775808) will be returned. | + +## 条件函数 + +| Function | Janino Code | Description | +|-----------------------------------------------------------------------------------------------------------------------|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| CASE value WHEN value1_1 [, value1_2]* THEN RESULT1 (WHEN value2_1 [, value2_2 ]* THEN result_2)* (ELSE result_z) END | Nested ternary expression | Returns resultX when the first time value is contained in (valueX_1, valueX_2, …). When no value matches, returns result_z if it is provided and returns NULL otherwise. | +| CASE WHEN condition1 THEN result1 (WHEN condition2 THEN result2)* (ELSE result_z) END | Nested ternary expression | Returns resultX when the first conditionX is met. When no condition is met, returns result_z if it is provided and returns NULL otherwise. | +| COALESCE(value1 [, value2]*) | coalesce(Object... objects) | Returns the first argument that is not NULL.If all arguments are NULL, it returns NULL as well. The return type is the least restrictive, common type of all of its arguments. The return type is nullable if all arguments are nullable as well. | +| IF(condition, true_value, false_value) | condition ? true_value : false_value | Returns the true_value if condition is met, otherwise false_value. E.g., IF(5 > 3, 5, 3) returns 5. | + +## 转换函数 You can use `CAST( AS )` syntax to convert any valid expression `` to a specific type ``. Possible conversion paths are: @@ -191,9 +191,9 @@ You can use `CAST( AS )` syntax to convert any valid expression `