[FLINK-36794] [cdc-composer/cli] pipeline cdc connector support multiple data sources

[linjianchang]

pipeline cdc connector support multiple data sources

[ChaomingZhangCN]

I think multi data sources should be regarded as parallelized.

[linjianchang]

Already modified

[ChaomingZhangCN]

Should we use a new key like 'sources' to describe multiple sources? The '_multiple' suffix in value seems a bit odd. Because the YAML content does not correspond one-to-one with the PipelineDef.

[linjianchang]

Already modified

[ChaomingZhangCN]

sources:
 - type: mysql
    name: mysql-instance-00
    hostname: localhost
    port: 3306
    ....
 - type: mysql
    name: mysql-instance-01
    hostname: localhost
    port: 3307
    ....

And the corresponding PipelineDef looks like this:

public class PipelineDef {
    @Nullable private List<SourceDef> sources;
    private final SourceDef source;
    ...
}

If the sources is not null then we use these data sources, otherwise we use source to build up DataStream. In this way, the previous usage will not be affected.
I want to hear your opinion. @yuxiqian

[yuxiqian]

I do like @ChaomingZhangCN's proposed syntax for a fully multiple data source, they're intuitive and expressive, but might be a chore if users just want to connect to a MySQL cluster with multiple servers, as they have to copy all identical configurations to both source definition.

@linjianchang's solution for now seems like MySQL specific, especially for multi-host clusters. It could not be extended for hetero-sources (like concatenating data from different DBMS), or when one wants to use different configs for each node. These cases don't exist for now since all we have is MySQL source connector, but as we're modifying composer and YAML API (instead of MySQL connector itself), such possibility should be discussed more carefully.

As for multiple sources in pipeline itself, I remembered the idea has been informally discussed with @leonardBang and @PatrickRen long time ago, and the conclusion was running multiple sources in one single job actually makes the pipeline more fragile, since any single-point failure would easily escalate and cause a global failover. Things might have changed since then, still needs hearing from senior developers on this.

[linjianchang]

sources:
 - type: mysql
    name: mysql-instance-00
    hostname: localhost
    port: 3306
    ....
 - type: mysql
    name: mysql-instance-01
    hostname: localhost
    port: 3307
    ....

And the corresponding PipelineDef looks like this:

public class PipelineDef {
    @Nullable private List<SourceDef> sources;
    private final SourceDef source;
    ...
}

If the sources is not null then we use these data sources, otherwise we use source to build up DataStream. In this way, the previous usage will not be affected. I want to hear your opinion. @yuxiqian @ChaomingZhangCN

It has been modified according to comments, please review it again, thanks!

[ChaomingZhangCN]

Should be _multiple.

[linjianchang]

Already modified