You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello,
I have KafkaConnect with 2 workers and FilePulse connector with 4 tasks. During the rolling restart of the workers, connector stops cleaning up files from S3. At the same time once the restart is finished, new files are processed and cleaned up successfully.
All not-cleaned files have the status "COMMITTED" in the status topic.
here is my connector config:
There are no errors in the logs.
Just a thought, is it possible/makes sense to check file status by AmazonS3MoveCleanupPolicy retrospectively and if it has "COMMITTED" status clean it up?
The text was updated successfully, but these errors were encountered:
arnitolog
changed the title
AmazonS3MoveCleanupPolicy doesn't cleanup after restart
AmazonS3MoveCleanupPolicy doesn't cleanup files during KafkaConnect workers restart
Sep 23, 2024
Hey @arnitolog , I know it's old question and I don't know if you succeed to find a solution for your problem but here are some details about the cleanup policy.
The cleanup process is handle by the connector thread. Files can be cleanup either on the COMPLETED (default) or COMMITTED status depending on the value of the fs.cleanup.policy.triggered.on property.
However, you may have encountered a bug. If the connector thread is killed or restarted while the file status changes from COMPLETED to COMMITTED, once restarted, the connector thread missed the state transition and did not clean up the file.
One solution is to configure the property fs.cleanup.policy.triggered.on=COMMITTED then doing a rolling-restart of Kafka Connect should trigger the cleanup of files stuck in COMMITTED state
Hello,
I have KafkaConnect with 2 workers and FilePulse connector with 4 tasks. During the rolling restart of the workers, connector stops cleaning up files from S3. At the same time once the restart is finished, new files are processed and cleaned up successfully.
All not-cleaned files have the status "COMMITTED" in the status topic.
here is my connector config:
There are no errors in the logs.
Just a thought, is it possible/makes sense to check file status by AmazonS3MoveCleanupPolicy retrospectively and if it has "COMMITTED" status clean it up?
The text was updated successfully, but these errors were encountered: