Add Kafka Support

[Ahmad Marzouq (ahmad-marzouq)]

No description provided.

[Snowplow CLA bot (snowplowcla)]

Thanks for your pull request. Is this your first contribution to a Snowplow open source project? Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://docs.snowplowanalytics.com/docs/contributing/contributor-license-agreement/ to learn more and sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

[Ahmad Marzouq (ahmad-marzouq)]

I signed it!

[Snowplow CLA bot (snowplowcla)]

Confirmed! Ahmad Marzouq (@ahmad-marzouq) has signed the Contributor License Agreement. Thanks so much.

[Paul Boocock (paulboocock)]

Sorry for going silent on this issue! We've not looked at this yet as running the Snowplow pipeline on top of Kafka isn't currently a high priority for us, and being transparent it likely won't be in the near future. We do have some plans to revist our Kafka support as we know how popular it is amongst our Open Source users, but this is going to be a quite a while away, I anticipate late 2022 at the earliest.

Unfortunately, as it stands we don't have a particularly good and repeatable way to test pipelines running on top of Kafka which has prevented us merging new and updated Kafka features, once it's merged we need to be able to maintain it which is why this is at a standstill. We'll update this issue should anything change and we become in a better position to thing about improving our Kafka support across the platform.