How to set the inserts frequency?

[askmyhat]

There is an option to set the batch size:
https://clickhouse.com/docs/en/integrations/kafka#i-would-like-to-adjust-the-batch-size-for-the-sink-connector

However, I would to like to set the ingestion interval instead so that Clickhouse Sink generates at most one insert per second regardless of the data size.
I was not able to find such a parameter in the Kafka connect documentation:
https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html
https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html

Currently I have 30-100 inserts per second which leads to high CPU and IO utilization.
Do I miss something?

[Paultagoras]

However, I would to like to set the ingestion interval instead so that Clickhouse Sink generates at most one insert per second regardless of the data size.

https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html

https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html

Currently I have 30-100 inserts per second which leads to high CPU and IO utilization.

At this moment, we don't have a setting for that - the connector just tries to insert whatever records Kafka Connect sends to it.

There are a few creative options available (but nothing exactly matching):

• Number of Workers: If you were to reduce the number of workers, that could group up more messages. The general suggestion is 1 Worker for 1 Partition, but you could reduce below this number if your data volume is lower.

• fetch.min.bytes: This should allow for larger batches of data to be accumulated before sending. Paired with fetch.max.wait.ms you may be able to set the byte size so large it almost always hits that max wait. You would also need to combine with this to make sure you pull enough records to make up for the "slowdown".

Hope those help!