Make the topic name configurable in storage paths

[Hariprasad Kuppuswamy (hariprasad-k)]

Improvements to provide ability to exclude topic name in partition by configuration

[Deleted user (ghost)]

Confluent Inc. (@confluentinc) It looks like Hariprasad Kuppuswamy (@hariprasad-k) just signed our Contributor License Agreement. 👍

Always at your service,

clabot

[Hariprasad Kuppuswamy (hariprasad-k)]

Fixed #126

[Jordan Moore (OneCricketeer)]

Question: What happens when you have topics as a list or topics.regex capture more than one topic?

Then all files in the directory will be of multiple topics?

[Jordan Moore (OneCricketeer)]

I suggest you break this out as a separate PR

[Hariprasad Kuppuswamy (hariprasad-k)]

Jordan Moore (@OneCricketeer) Sure, I will break into separate PR.

[Jordan Moore (OneCricketeer)]

The patch also fixes #121 too

So does #122

[Hariprasad Kuppuswamy (hariprasad-k)]

Question: What happens when you have topics as a list or topics.regex capture more than one topic?

Then all files in the directory will be of multiple topics?

Jordan Moore (@OneCricketeer) Yes, all files of multiple topics will be in the same directory in this case.

But, this behavior will only happen if you use topics as a list or topics.regex in combination with path.include.topicname set as false. Further, as path.include.topicname is by default true, it will preserve the existing behavior by default.

[Hariprasad Kuppuswamy (hariprasad-k)]

Jordan Moore (@OneCricketeer) This PR does not have any changes related to Timestamp scaling factor anymore, which is covered by #160

Can you please review this changes ?

[Jordan Moore (OneCricketeer)]

I'm not a Confluent employee, so my reviews don't do anything

[Daniel (dosvath)]

Thanks Hariprasad Kuppuswamy (@hariprasad-k) for the contribution! This change will add flexibility to our connectors that use storage common to optionally exclude the topic name from the path (seems to be a popular request confluentinc/kafka-connect-hdfs#544). I'd like to get second view from Lev Zemlyanov (@levzem) before proceeding.

[Hariprasad Kuppuswamy (hariprasad-k)]

Daniel (@dosvath) Lev Zemlyanov (@levzem) Any thoughts, or plans to review this contribution ?

[Jeremy Holland (awebneck)]

Would be very much interested in having this feature integrated. The topic name being forcefully appended to the path limits us severely when sinking to already-established directory structures.