Detect Topic not present in metadata after 60sec error early

[rajjaiswalsaumya]

I am using Spring's Kafka Template to send data to a Kafka Cluster. The topic in Kafka is created dynamically on the fly by another microservice. We want to detect a case where the event is posted on that topic before its creation in Kafka, which would lead to the topic not being present in metadata after a 60-second error. This error happens on a separate thread and therefore we get a timeout Exception from the LisenableFuture returned by KafkaTemplate. Is there a smart way to detect this error so that we can ignore the message and drop it? Currently due to this issue our pipeline is getting blocked as we cannot ack to those messages and call punctuate till retry is not expired.

Also, the TimoutError thrown does not have any traces of TopicOrPartitionUnknown error in stack trace to be looked at.

[artembilan]

This has nothing to do with Spring. The behavior you are facing is just standard and built-in Apache Kafka client: https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update.

So, consider to adjust that max.block.ms for your requirements.

Not sure about UnknownTopicOrPartitionException: probably the question for Apache Kafka by itself, too.

You may consider to use KafkaAdminOperations.describeTopics() API to check topic presence before sending messages.

[rajjaiswalsaumya]

Yeah, I thought to do the same, but just want to understand if the admin client throws that error, then why not KafkaTemplate?

[artembilan]

Because the KafkaProducer that KafkaTemplate is based on, does not throw that exception.
As I said: the question is more for Apache Kafka by itself: nothing what Spring can do for us if underlying library doesn't expose some hooks on the matter.

[rajjaiswalsaumya]

Can I request you to suggest what should be the approach to detect the above use case? And will setting max.block.ms help or do i still need to go to KafkaAdminOperations.describeTopics() to get topic details?

[artembilan]

I would go with KafkaAdminOperations.describeTopics(), because it is a straightforward call with an immediate answer.
That metadata request from Kafka producer is a bit out of our control, there I'd prefer something what I can handle.
This KafkaAdminOperations.describeTopics() call could also be @Cacheable to avoid extra calls to broker for those topics which have been created and resolved.
But again: this is my opinion since I'm not an expert in Kafka internals and how it manages that metadata and its caching.

[Wzy19930507]

Hi, Can provide a tool, like AbstractMessageListenerContainer.checkTopics().
We can call it before application start.