[FLINK-34680][cdc][docs] Add overview page for CDC connectors and optimize connector docs

PatrickRen · web-flow · commit 8fd28f1788f3 · 2024-03-18T19:06:08.000+08:00
This closes #3160.
diff --git a/docs/content/docs/connectors/doris.md b/docs/content/docs/connectors/doris.md
@@ -24,13 +24,12 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# Doris Pipeline Connector
-
-This article introduces of Doris Pipeline Connector
+# Doris Connector
 
+This article introduces of Doris Connector
 
 ## Example
-----------------
+
 
 ```yaml
 source:
@@ -50,8 +49,7 @@ pipeline:
 
 ```
 
-## Pipeline options
-----------------
+## Connector Options
 
 <div class="highlight">
 <table class="colwidths-auto docutils">
@@ -180,17 +178,16 @@ pipeline:
      </tbody>
 </table>
 </div>
-## Data Type Mapping
 
-----------------
+## Data Type Mapping
 
 <div class="wy-table-responsive">
 <table class="colwidths-auto docutils">
     <thead>
       <tr>
-        <th class="text-left" style="width:10%;">CDC type</th>
-        <th class="text-left" style="width:30%;">Doris type<a href="https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Types/BOOLEAN/"></a></th>
-        <th class="text-left" style="width:60%;">NOTE</th>
+        <th class="text-left" style="width:10%;">Flink CDC Type</th>
+        <th class="text-left" style="width:30%;"><a href="https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Types/BOOLEAN/">Doris Type</a></th>
+        <th class="text-left" style="width:60%;">Note</th>
       </tr>
     </thead>
     <tbody>
diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md b/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
@@ -24,10 +24,12 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# CDC Connectors for Apache Flink
+# Legacy CDC sources for Apache Flink
 
-CDC Connectors for Apache Flink<sup>®</sup> is a set of source connectors for <a href="https://flink.apache.org/">Apache Flink<sup>®</sup></a>, ingesting changes from different databases using change data capture (CDC).
-The CDC Connectors for Apache Flink<sup>®</sup> integrate Debezium as the engine to capture data changes. So it can fully leverage the ability of Debezium. See more about what is [Debezium](https://github.com/debezium/debezium).
+Flink CDC sources is a set of source connectors for <a href="https://flink.apache.org/">Apache Flink<sup>®</sup></a>, ingesting changes from different databases using change data capture (CDC).
+Some CDC sources integrate Debezium as the engine to capture data changes. So it can fully leverage the ability of Debezium. See more about what is [Debezium](https://github.com/debezium/debezium).
+
+You can also read [tutorials]({{< ref "docs/connectors/legacy-flink-cdc-sources/tutorials/mysql-postgres-tutorial" >}}) about how to use these sources.
 
 {{< img src="/fig/cdc-flow.png" width="600px" alt="Flink CDC" >}}
 
diff --git a/docs/content/docs/connectors/mysql.md b/docs/content/docs/connectors/mysql.md
@@ -24,16 +24,15 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# MySQL CDC Pipeline Connector
+# MySQL Connector
 
-The MySQL CDC Pipeline Connector allows for reading snapshot data and incremental data from MySQL database and provides end-to-end full-database data synchronization capabilities.
-This document describes how to setup the MySQL CDC Pipeline connector.
+MySQL connector allows reading snapshot data and incremental data from MySQL database and provides end-to-end full-database data synchronization capabilities.
+This document describes how to setup the MySQL connector.
 
 
-How to create Pipeline
-----------------
+## Example
 
-The pipeline for reading data from MySQL and sink to Doris can be defined as follows:
+An example of the pipeline for reading data from MySQL and sink to Doris can be defined as follows:
 
 ```yaml
 source:
@@ -58,8 +57,7 @@ pipeline:
    parallelism: 4
 ```
 
-Pipeline Connector Options
-----------------
+## Connector Options
 
 <div class="highlight">
 <table class="colwidths-auto docutils">
@@ -245,8 +243,7 @@ Pipeline Connector Options
 </table>
 </div>
 
-Startup Reading Position
---------
+## Startup Reading Position
 
 The config option `scan.startup.mode` specifies the startup mode for MySQL CDC consumer. The valid enumerations are:
 
@@ -259,16 +256,15 @@ The config option `scan.startup.mode` specifies the startup mode for MySQL CDC c
 - `timestamp`: Skip snapshot phase and start reading binlog events from a specific timestamp.
 
 
-Data Type Mapping
-----------------
+## Data Type Mapping
 
 <div class="wy-table-responsive">
 <table class="colwidths-auto docutils">
     <thead>
       <tr>
-        <th class="text-left" style="width:30%;">MySQL type<a href="https://dev.mysql.com/doc/man/8.0/en/data-types.html"></a></th>
-        <th class="text-left" style="width:10%;">CDC type</th>
-        <th class="text-left" style="width:60%;">NOTE</th>
+        <th class="text-left" style="width:30%;"><a href="https://dev.mysql.com/doc/man/8.0/en/data-types.html">MySQL type</a></th>
+        <th class="text-left" style="width:10%;">Flink CDC type</th>
+        <th class="text-left" style="width:60%;">Note</th>
       </tr>
     </thead>
     <tbody>
diff --git a/docs/content/docs/connectors/overview.md b/docs/content/docs/connectors/overview.md
@@ -24,21 +24,33 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# Pipeline Connectors Of CDC Streaming ELT Framework
+# Connectors
+
+Flink CDC provides several source and sink connectors to interact with external
+systems. You can use these connectors out-of-box, by adding released JARs to
+your Flink CDC environment, and specifying the connector in your YAML pipeline
+definition.
 
 ## Supported Connectors
 
-| Connector                                   | Database                                                                                                                                                                                                                                                                                                                                                                                               | 
-|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [doris-pipeline](doris-pipeline.md)         | <li> [Doris](https://doris.apache.org/): 1.2.x, 2.x.x                                                                                                                                                                                                                                                                                                                                                  | 
-| [mysql-pipeline](mysql-pipeline.md)         | <li> [MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li> [MariaDB](https://mariadb.org): 10.x <li> [PolarDB X](https://github.com/ApsaraDB/galaxysql): 2.0.1 | 
-| [starrocks-pipeline](starrocks-pipeline.md) | <li> [StarRocks](https://www.starrocks.io/): 2.x, 3.x                                                                                                                                                                                                                                                                                                                                                  | 
+| Connector                                            | Supported Type | External System                                                                                                                                                                                                                                                                                                                                                                                       | 
+|------------------------------------------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Apache Doris]({{< ref "docs/connectors/doris" >}})  | Sink           | <li> [Apache Doris](https://doris.apache.org/): 1.2.x, 2.x.x                                                                                                                                                                                                                                                                                                                                          | 
+| [MySQL]({{< ref "docs/connectors/mysql" >}})         | Source         | <li> [MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li> [MariaDB](https://mariadb.org): 10.x <li> [PolarDB X](https://github.com/ApsaraDB/galaxysql): 2.0.1 | 
+| [StarRocks]({{< ref "docs/connectors/starrocks" >}}) | Sink           | <li> [StarRocks](https://www.starrocks.io/): 2.x, 3.x                                                                                                                                                                                                                                                                                                                                                 |
+
+## Develop Your Own Connector
+
+If provided connectors cannot fulfill your requirement, you can always develop
+your own connector to get your external system involved in Flink CDC pipelines.
+Check out [Flink CDC APIs]({{< ref "docs/developer-guide/understand-flink-cdc-api" >}})
+to learn how to develop your own connectors.
 
-## Supported Flink Versions
-The following table shows the version mapping between Flink<sup>®</sup> CDC Pipeline and Flink<sup>®</sup>:
+## Legacy Flink CDC Sources
 
-|    Flink<sup>®</sup> CDC Version    |                                                                                                      Flink<sup>®</sup> Version                                                                                                       |
-|:-----------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
-| <font color="DarkCyan">3.0.*</font> | <font color="MediumVioletRed">1.14.\*</font>, <font color="MediumVioletRed">1.15.\*</font>, <font color="MediumVioletRed">1.16.\*</font>, <font color="MediumVioletRed">1.17.\*</font>, <font color="MediumVioletRed">1.18.\*</font> |
+Flink CDC sources introduces before 3.0 are still available as normal Flink 
+connector sources. You can find more details in the 
+[overview page]({{< ref "docs/connectors/legacy-flink-cdc-sources/overview" >}})
+of legacy Flink CDC sources.
 
 {{< top >}}
diff --git a/docs/content/docs/connectors/starrocks.md b/docs/content/docs/connectors/starrocks.md
@@ -24,17 +24,16 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# StarRocks Pipeline Connector
+# StarRocks Connector
 
-The StarRocks Pipeline connector can be used as the *Data Sink* of the pipeline, and write data to [StarRocks](https://github.com/StarRocks/starrocks). This document describes how to set up the StarRocks Pipeline connector.
+StarRocks connector can be used as the *Data Sink* of the pipeline, and write data to [StarRocks](https://github.com/StarRocks/starrocks). This document describes how to set up the StarRocks connector.
 
 ## What can the connector do?
 * Create table automatically if not exist
 * Schema change synchronization
 * Data synchronization
 
-How to create Pipeline
-----------------
+## Example
 
 The pipeline for reading data from MySQL and sink to StarRocks can be defined as follows:
 
@@ -62,8 +61,8 @@ pipeline:
    parallelism: 2
 ```
 
-Pipeline Connector Options
-----------------
+## Connector Options
+
 <div class="highlight">
 <table class="colwidths-auto docutils">
    <thead>
@@ -220,8 +219,8 @@ Pipeline Connector Options
 </table>    
 </div>
 
-Usage Notes
---------
+
+## Usage Notes
 
 * Only support StarRocks primary key table, so the source table must have primary keys.
 
@@ -244,15 +243,14 @@ Usage Notes
   to write data to StarRocks. You can see [sink documentation](https://github.com/StarRocks/starrocks-connector-for-apache-flink/blob/main/docs/content/connector-sink.md)
   for how it works. 
 
-Data Type Mapping
-----------------
+## Data Type Mapping
 <div class="wy-table-responsive">
 <table class="colwidths-auto docutils">
     <thead>
       <tr>
-        <th class="text-left">CDC type</th>
+        <th class="text-left">Flink CDC type</th>
         <th class="text-left">StarRocks type</th>
-        <th class="text-left" style="width:60%;">NOTE</th>
+        <th class="text-left" style="width:60%;">Note</th>
       </tr>
     </thead>
     <tbody>
