mysql做为数据源批采集无法正常checkpoint #5924
Unanswered
suifeng08371
asked this question in
Q&A
Replies: 1 comment
-
Hello, is this what happens when you try to remove the speed limit? mysql data volume How much DDL? Can you provide it? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
通过seatunnel进行mysql到hive的数据同步,mysql作为数据源,通过jdbc的方式进行连接。指定了条数的限流"read_limit.rows_per_second" : 100000,导致checkpoint没法正常成功。导到checkpoint超时时间后,checkpoint直接报错。
批cai采集不支持checkpoint,还是我配置的问题?请帮忙解答。
对应的报错截图信息如下:

checpoint出错:
Exceptions信息:


对应的seatunnel配置如下:
{
"env" : {
"job.name" : "sigle_test_diaodu",
"job.mode" : "BATCH",
"execution.checkpoint.cleanup-mode" : false,
"execution.checkpoint.interval" : 10000,
"execution.checkpoint.data-uri" : "hdfs://ns1/seatunnel/checkpoint/xxxx",
"parallelism" : 1,
"read_limit.rows_per_second" : 100000
},
"source" : [ {
"url" : "jdbc:mysql://10.10.10.10:3306/dbName",
"driver" : "com.mysql.jdbc.Driver",
"user" : "username",
"password" : "password",
"query" : "SELECT DETAIL_ADRESS,MSISDN,USER_ID,USER_NAME,STATUS,ID FROM dbName.table_name",
"plugin_name" : "Jdbc",
"partition_column" : "ID",
"result_table_name" : "_seatunnel_table_5678"
} ],
"transform" : [ {
"query" : "SELECT DETAIL_ADRESS AS TEST1,MSISDN AS TEST4,USER_ID AS TEST3,USER_NAME AS TEST5,STATUS AS TEST6,'1' AS TEST2 FROM _seatunnel_table_5678",
"field_name" : "TEST1,TEST4,TEST3,TEST5,TEST6,TEST2",
"plugin_name" : "Sql",
"source_table_name" : "_seatunnel_table_5678",
"result_table_name" : "_seatunnel_table_5680"
} ],
"sink" : [ {
"table_name" : "prod.hive_table_name",
"hive_site_path" : "/hive/conf/hive-site.xml",
"metastore_uri" : "thrift://11.1.1.1:9083,thrift://11.1.1.2:9083",
"plugin_name" : "Hive",
"source_table_name" : "_seatunnel_table_5680"
} ]
}
Beta Was this translation helpful? Give feedback.
All reactions