v2.3.2

Noticeable changes

1. Update Scala3 from 3.2.2 to 3.3.0
2. Update kafka-clients from 3.4.0 to 3.4.1
3. New helper functions in the testkit. See #893 for details.
4. We now have an Adopters list. Don't hesitate to put the name of your company there if you use zio-kafka! 🔥
5. New setting in ConsumerSettings: ConsumerSettings::enableOptimisticResume, true by default.

This new setting allows you to disable an internal optimisation we call "optimistic resume".
This optimisation has been introduced in v2.3.0.
Some users have reported very high latencies when this optimisation is enabled.
Disabling it seems to resolve this latency issue. (More details, see #903)

Our benchmarks also show that this optimisation can negatively impact performance if you use manual subscriptions (Subscription.manual(...)).
Therefore, If you use manual subscriptions, you might want to try to disable the "optimistic resume" optimisation and see if it improves the throughput and/or latency you observe in production.

Please, report any improvement or issue observed with and without this optimisation activated in our ZIO Discord channel so that we better learn about the positive and negative impacts this optimisation can have on real-world workloads.

Thanks! ❤️

All Changes

• Fix: Runloop thread name is always zio-kafka-runloop-thread-0 @guizmaii (#909)
• Update Scala3 to v3.3.0 @guizmaii (#873)
• Update kafka-clients to 3.4.1 @github-actions (#906)
• Update zio-streams, zio-test-sbt to 2.0.15 @github-actions (#905)
• Update kafka-clients to 3.4.1 @scala-steward (#908)
• Update zio, zio-streams, zio-test, ... to 2.0.15 @github-actions (#904)
• Extract all code refactoring from #587 @guizmaii (#901)
• [Testkit] Allow to customise broker properties @V-Lamp (#893)
• Document optimistic resume @erikvanoosten (#894)
• Update CI definition after merge of #896 @guizmaii (#897)
• Update README.md @zio-assistant (#896)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.10 @github-actions (#895)
• Add new ConsumerSettings::enableOptimisticResume setting (true by default) to allow users to disable the "optimistic resume" optimisation && Write benchmarks where this optimisation is disabled @guizmaii (#889)
• Update sbt to 1.9.0 @github-actions (#891)
• Update zio-streams, zio-test-sbt to 2.0.14 @scala-steward (#888)
• Add Rocker as adopter @loathwine (#885)
• Update zio, zio-streams, zio-test, ... to 2.0.14 @github-actions (#886)
• Update scalafmt-core to 3.7.4 @github-actions (#887)
• Update jackson-databind to 2.15.2 @github-actions (#882)
• Update README.md @zio-assistant (#878)
• Update README.md @zio-assistant (#877)
• Update README @guizmaii (#876)
• docs: add KelkooGroup as adopter @gaeljw (#875)
• Initiate an "Adopters" list @guizmaii (#872)
• [Internal] Clean some code @guizmaii (#871)
• [Internal] Simplify code: Remove useless doPoll method @guizmaii (#870)
• [Internal] Extract Command from the Runloop file and rename it to RunloopCommand @guizmaii (#867)
• [Internal] Improve function name @guizmaii (#869)
• [Internal] Reduce timeouts in tests from 5 to 2 minutes @guizmaii (#866)
• [Internal] Fix tests logging configuration @guizmaii (#865)
• [Internal] Use SLF4J/Logback to configure logging in tests @guizmaii (#858)
• [Internal] Simplify Diagnostics interface @guizmaii (#861)
• Update zio-kafka version in README && Update zio-kafka-example project @guizmaii (#855)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.52.0 to 2.54.0 @dependabot (#874)

v2.3.1

zio-kafka-testkit library

In this version, we renamed the zio-kafka-test-utils library to zio-kafka-testkit, we changed its base package from zio.kafka to zio.kafka.testkit, and we re-wrote the documentation on how to write tests when using zio-kafka and zio-kafka-testkit.

To migrate, you will have to:

1. Change in your build.sbt:

// Before
libraryDependencies += "dev.zio"  %% "zio-kafka-test-utils" % "2.3.0" % Test

// After
libraryDependencies += "dev.zio"  %% "zio-kafka-testkit" % "2.3.1" % Test

2. Fix your imports

// Before
import zio.kafka.ZIOSpecWithKafka
import zio.kafka.KafkaTestUtils
import zio.kafka.KafkaRandom
import zio.kafka.embedded.Kafka

// After
import zio.kafka.testkit.ZIOSpecWithKafka
import zio.kafka.testkit.KafkaTestUtils
import zio.kafka.testkit.KafkaRandom
import zio.kafka.testkit.Kafka

You can find the new documentation here: https://zio.dev/zio-kafka/writing-tests

Other changes

• We un-deprecated the ZIOSpecWithKafka trait, and we added some documentation about its purpose and usage in the documentation.
  See https://zio.dev/zio-kafka/writing-tests#ziospecwithkafka-trait

All Changes

• Write zio-kafka-testkit documentation @guizmaii (#846)
• Rename the zio-kafka-test-utils module to zio-kafka-testkit && change its base package from zio.kafka to zio.kafka.testkit @guizmaii (#845)
• Update zio-logging-slf4j2 to 2.1.13 @github-actions (#854)
• Un-deprecate ZIOSpecWithKafka && Add doc explaining when to use it @guizmaii (#851)
• Update jackson-databind to 2.15.1 @github-actions (#850)
• Add minimal "manual consumer" benchmarks @guizmaii (#849)
• Update sbt to 1.8.3 @github-actions (#847)
• Remove all rights reserved @erikvanoosten (#843)
• Update zio-kafka to 2.3.0 @github-actions (#842)
• Update README.md @zio-assistant (#841)
• Bump version, remove all rights reserved @erikvanoosten (#840)

v2.3.0

Consumer’s performances

This release brings many performance improvements to zio-kafka’s Consumer, getting closer to the performances of the native Java kafka-clients’ Consumer.

Because of these changes, the broker will be polled more often which can cause higher CPU and memory consumption. You may need to re-tune the consumer settings. For example, you can lower max.poll.records when there is high memory pressure, and you can increase pollTimeout when you observe high CPU usage and latency is not a concern.

Our benchmarks are showing that with this new version of zio-kafka, to consume a million messages, zio-kafka’s Consumer now takes ~737.614 ± 19.178 ms while the native Java kafka-clients one takes ~640.530 ± 1.855 ms

For comparison, the Consumer of zio-kafka v2.2 was taking ~1950.181 ± 590.420 ms with very high variability in the benchmark results:

// v2.2
Result "zio.kafka.bench.ConsumersComparisonBenchmark.zioKafka":
1950.181 ±(99.9%) 590.420 ms/op [Average]
(min, avg, max) = (1015.872, 1950.181, 4967.793), stdev = 1192.678
CI (99.9%): [1359.761, 2540.602] (assumes normal distribution)

while now, the results are much more stable:

// v2.3.0
Result "zio.kafka.bench.ConsumersComparisonBenchmark.zioKafka":
737.614 ±(99.9%) 19.178 ms/op [Average]
(min, avg, max) = (677.381, 737.614, 820.314), stdev = 38.740
CI (99.9%): [718.437, 756.792] (assumes normal distribution)

See details of the current performance results here: #809 (comment)

Note that these benchmarks were run on my local machine, a MacBook Pro 2019 Intel 2.4Ghz 8 cores 64GB 2667 MHz DDR4.
To reproduce these results, you can run this sbt command on any version of zio-kafka containing these benchmarks:

sbt "clean;Test/compile;zioKafkaBench/Jmh/run -wi 10 -i 10 -r 1 -w 1 -t 1 -f 5 -foe true .*ConsumersComparisonBenchmark*"

If you monitor your Consumers' performances in production, don't hesitate to come to the zio-kafka channel of the ZIO Discord to share the changes you could have observed with this new version! ❤️

Breaking changes

• The AdminClient::listConsumerGroupOffsets(groupSpecs: Map[String, ListConsumerGroupOffsetsSpec]) and AdminClient::listConsumerGroupOffsets(groupSpecs: Map[String, ListConsumerGroupOffsetsSpec], options: ListConsumerGroupOffsetsOptions) methods were returning a Task[Map[TopicPartition, OffsetAndMetadata]] and are now returning a Task[Map[String, Map[TopicPartition, OffsetAndMetadata]]]

The String in the key of the Map being the Consumer Group IDs (same ones as the ones you passed in parameters)

• The AdminClient.fromManagedJavaClient has been renamed AdminClient.fromScopedJavaClient

• The ConsumerSettings::perPartitionChunkPrefetch Consumer configuration has been removed

Deprecations

• The ZIOSpecWithKafka trait in zio-kafka-test-utils is deprecated

See https://github.com/zio/zio-kafka/blob/master/zio-kafka-test-utils/src/main/scala/zio/kafka/ZIOSpecWithKafka.scala#L7-L23 to see how to remove it from your code.

Other noticeable changes

• When instantiating a client (AdminClient, Producer or Consumer) zio-kafka is now checking that your client SSL configuration is valid for your server SSL configuration, failing if the client configuration is incorrect.

This check prevents your app from crashing with an OOM a few minutes after you start the client.
This check protects you against a long-standing bug present in kafka-clients, see https://issues.apache.org/jira/browse/KAFKA-4090

This check is provided to you by Conduktor, backported from their internal fork of zio-kafka.

For more details about this check, you can have a look at its code here: https://github.com/zio/zio-kafka/blob/master/zio-kafka/src/main/scala/zio/kafka/utils/SslHelper.scala

All changes

• Optimistically resume partitions early @erikvanoosten (#809)
• Performances: Remove Scala-- code @guizmaii (#834)
• Update scala-collection-compat to 2.10.0 @github-actions (#833)
• Backport Conduktor's improvements @guizmaii (#781)
• Update jackson-databind to 2.15.0 @scala-steward (#831)
• Small improvements taken from #788 @erikvanoosten (#829)
• Update logback-classic to 1.3.7 @scala-steward (#827)
• Fix Scala3 build @guizmaii (#824)
• Update embedded-kafka to 3.4.0.1 @github-actions (#823)
• Update zio-sbt-ci and CI definition @guizmaii (#822)
• Add a minimal example app, useful to tests things manually if necessary @guizmaii (#807)
• Fix CI @guizmaii (#820)
• Update README.md @guizmaii (#819)
• Update zio, zio-streams, zio-test, ... to 2.0.13 @github-actions (#818)
• New code owners part 2 @erikvanoosten (#812)
• Use finalizer to ensure seeing de-register in unit test @erikvanoosten (#813)
• Remove unnecessary nowarn @erikvanoosten (#810)
• New code owners @erikvanoosten (#811)
• ConsumerAccess::runloopAccess should be private @guizmaii (#808)
• Backport improvement made in zio/zio#7996 @guizmaii (#805)
• Run the Runloop on a dedicated single-threaded thread pool @guizmaii (#802)
• Optimize offerRecordsToStreams code @guizmaii (#791)
• Update zio, zio-streams, zio-test, ... to 2.0.12 @github-actions (#793)
• Fix CI @guizmaii (#792)
• Update ZIO @guizmaii (#787)
• Simplify CI definition @guizmaii (#786)
• Improve stream restart in restartStreamsOnRebalancing mode @erikvanoosten (#785)
• Update README.md @github-actions (#782)
• Simpler shutdown handling @svroonland (#778)

🌱 Dependency Updates

• Bump peter-evans/create-pull-request from 5.0.0 to 5.0.1 @dependabot (#837)
• Bump benchmark-action/github-action-benchmark from 1.16.2 to 1.17.0 @dependabot (#835)
• Bump scala-steward-org/scala-steward-action from 2.51.0 to 2.52.0 @dependabot (#814)
• Bump peter-evans/create-pull-request from 4.2.3 to 5.0.0 @dependabot (#816)
• Bump peter-evans/create-pull-request from 4.2.3 to 5.0.0 @dependabot (#789)
• Bump actions/setup-java from 3.10.0 to 3.11.0 @dependabot (#790)
• Bump actions/setup-java from 3.10.0 to 3.11.0 @dependabot (#779)

v2.2

This release contains many performance enhancements and internal code optimizations to the consumer side of zio-kafka. The Runloop, the heart of the Consumer, has been optimized to reduce the polling and streaming latency and be able to provide higher throughput, as well as less GC pressure. This results in a throughput that is 2 to 5 times higher.

The Runloop polls the underlying Kafka client for new data. It also applies backpressure for streams that do not require data by pausing partitions. When the stream needs more data, the partition is resumed. Previously the Runloop would poll every pollInterval. Now it polls continuously. When all streams stop processing, data polling stops so that the broker can detect that this Kafka client is stalled.

Several other changes to be aware of:

• OffsetBatch.merge was deprecated in favour of add.
• Consumer setting pollInterval was removed because polling is now continuous. We recommend you use ConsumerSettings(bootstrapServers) and the with* methods to build consumer settings to prevent future source incompatibilities.
• Consumer setting runloopTimeout was added. When the runloop has nothing to do for this period, it will fail.

Changes

• Increase default runloop timeout @erikvanoosten (#775)
• Give ConsumerSettings.runloopTimeout a default @erikvanoosten (#774)
• Fix restartStreamsOnRebalancing @erikvanoosten (#772)
• Improve readability @erikvanoosten (#773)
• Update scalafmt-core to 3.7.3 @github-actions (#769)
• Add better-monadic-for scalac plugin @guizmaii (#765)
• Improve readability @guizmaii (#763)
• Remove unused offsetRetrieval field from ChangeSubscription command @guizmaii (#764)
• Fix getConsumerGroupMetadataIfAny: It shouldn't access the low-level consumer directly @guizmaii (#762)
• Optimize offerRecordsToStreams code @guizmaii (#761)
• Optimise handlePoll code: Ensure that both runningStreams and newAssignedStreams are Chunks to take advantage of the optimised version of ++ when computing the updatedStreams @guizmaii (#760)
• Divide allocations: Instantiate the OffsetImpl as late as possible to avoid allocatiing one OffsetImpl and one TopicPartition for each consumed Record @guizmaii (#759)
• add test case for the runloop polling behavior @tnielens (#684)
• Optimise offerRecordsToStreams code @guizmaii (#756)
• Tune benchmark parameters @svroonland (#729)
• Review of PR #727: Remove unused Request::dataQueue @guizmaii (#747)
• Improve performances: Remove the aggregateAsync call @guizmaii (#751)
• Improve Logback configuration in benchmarks @guizmaii (#750)
• Review of PR #727 @guizmaii (#745)
• Fix ConsumersComparisonBenchmark: Avoid booting the Consumer instance in the bench, which could add noise in the bench results @guizmaii (#748)
• Remove buffered records @erikvanoosten (#744)
• ZStream based implementation of poll loop @svroonland (#736)
• Feed streams through queue instead of promise @erikvanoosten (#727)
• First seek then use for manual assignments @erikvanoosten (#726)
• Stabilize benchmark @svroonland (#714)
• Fix CI @guizmaii (#723)
• Optimise Runloop::run code @guizmaii (#721)
• Add profiling in the CI @guizmaii (#696)
• Init comparison benchmarks: compare kafka-clients vs zio-kafka consumers @guizmaii (#695)
• Rollbacks @guizmaii (#722)
• Update logback-classic to 1.4.6 @github-actions (#717)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.3.10+94-cbe49d51-SNAPSHOT @github-actions (#718)
• Make as much computation as possible in fulfillRequests code @guizmaii (#720)
• Rename OffsetBatch::merge(Offset) to OffsetBatch::add(Offset) to improve readability @guizmaii (#700)
• Apply ZIO SBT Ecosystem and CI Plugins @khajavi (#702)
• Improve polling behavior @svroonland (#683)
• Update logback-classic to 1.3.6 @github-actions (#709)
• Update README.md @github-actions (#693)
• Consumer benchmark @svroonland (#689)

🌱 Dependency Updates

• Bump hmarr/auto-approve-action from 3.2.0 to 3.2.1 @dependabot (#734)
• Bump hmarr/auto-approve-action from 3.1.0 to 3.2.0 @dependabot (#705)
• Bump benchmark-action/github-action-benchmark from 1.16.0 to 1.16.2 @dependabot (#704)

v2.1.3

Changes

• doc: remove extra clock from consumer documentation @ouertani (#685)
• Improve docs for partitionedStream @svroonland (#682)

🌱 Dependency Updates

• Update zio, zio-streams, zio-test, ... to 2.0.10 @github-actions (#690)
• Bump benchmark-action/github-action-benchmark from 1.15.0 to 1.16.0 @dependabot (#687)
• Bump scala-steward-org/scala-steward-action from 2.50.0 to 2.51.0 @dependabot (#688)

v2.1.2

Changes

• Review usage of lazy values @guizmaii (#680)
• Change level of internal to zio-kafka logs from INFO to DEBUG @guizmaii (#681)
• Fix flaky RunloopSpec @guizmaii (#678)
• Update README.md @github-actions (#677)

v2.1.1

(Same as 2.1.0 with some artifact publishing fixes. Release notes for 2.1.0 are repeated below)

Note: this release contains some breaking changes for Consumer's plainStream, partitionedStream, partitionedAssignmentStream and consumeWith, as well as some of the companion objects on parameters used for the AdminClient methods. It should be relatively easy to upgrade though.

To upgrade, calls to Consumer.subscribeAnd(subscription).plainStream(keySerde, valueSerde) should be replaced with Consumer.plainStream(subscription, keySerde, valueSerde), likewise for partitionedStream.

See #517, #620 and #539 for more details.

Changes

• Support for multiple subscriptions on the same Consumer @svroonland (#517)
• Replace key&value tuple with ConsumerRecord in consumeWith @duxet (#620)
• Avoid using a Task when code is pure. Prefer to use Try @guizmaii (#539)
• Avoid extra latency potentially happening between Poll calls @guizmaii (#668)
• Optimize Runloop::endRevoked, Runloop::fulfillRequests, and Runloop::bufferRecordsForUnrequestedPartitions code @guizmaii (#666)
• Optimize Runloop::handleRequests code @guizmaii (#667)
• No need to instantiate more than one State.initial instance @guizmaii (#657)
• Clean some code @guizmaii (#652)
• Add transactional test @erikvanoosten (#644)
• added missing type annotations @aashish2054 (#636)
• Update README.md @github-actions (#635)

🌱 Dependency Updates

• Upgrade to kafka 3.4.0 @svroonland (#650)
• Update zio, zio-streams, zio-test, ... to 2.0.9 @github-actions (#641)
• Update zio, zio-streams, zio-test, ... to 2.0.8 @github-actions (#634)
• Update scalafmt-core to 3.7.2 @github-actions (#659)
• Bump scala-steward-org/scala-steward-action from 2.49.0 to 2.50.0 @dependabot (#647)
• Bump actions/setup-java from 3.9.0 to 3.10.0 @dependabot (#637)

v2.1.0

Note: this release contains some breaking changes for Consumer's plainStream, partitionedStream, partitionedAssignmentStream and consumeWith, as well as some of the companion objects on parameters used for the AdminClient methods. It should be relatively easy to upgrade though.

To upgrade, calls to Consumer.subscribeAnd(subscription).plainStream(keySerde, valueSerde) should be replaced with Consumer.plainStream(subscription, keySerde, valueSerde), likewise for partitionedStream.

See #517, #620 and #539 for more details.

Changes

• Support for multiple subscriptions on the same Consumer @svroonland (#517)
• Replace key&value tuple with ConsumerRecord in consumeWith @duxet (#620)
• Avoid using a Task when code is pure. Prefer to use Try @guizmaii (#539)
• Avoid extra latency potentially happening between Poll calls @guizmaii (#668)
• Optimize Runloop::endRevoked, Runloop::fulfillRequests, and Runloop::bufferRecordsForUnrequestedPartitions code @guizmaii (#666)
• Optimize Runloop::handleRequests code @guizmaii (#667)
• No need to instantiate more than one State.initial instance @guizmaii (#657)
• Clean some code @guizmaii (#652)
• Add transactional test @erikvanoosten (#644)
• added missing type annotations @aashish2054 (#636)
• Update README.md @github-actions (#635)

🌱 Dependency Updates

• Upgrade to kafka 3.4.0 @svroonland (#650)
• Update zio, zio-streams, zio-test, ... to 2.0.9 @github-actions (#641)
• Update zio, zio-streams, zio-test, ... to 2.0.8 @github-actions (#634)
• Update scalafmt-core to 3.7.2 @github-actions (#659)
• Bump scala-steward-org/scala-steward-action from 2.49.0 to 2.50.0 @dependabot (#647)
• Bump actions/setup-java from 3.9.0 to 3.10.0 @dependabot (#637)

v0.17.8

What's Changed

• update zio to 1.0.18 by @tellnobody1 in #633

New Contributors

• @tellnobody1 made their first contribution in #633

Full Changelog: v0.17.7...v0.17.8

v2.0.7

Changes

• Remove redundant collection conversion @rituraj2342 (#632)
• Clean some code @aashish2054 (#628)
• SAM conversion @rituraj2342 (#626)
• SpecSmell @rituraj2342 (#624)
• Configure benchmarks module and CI @guizmaii (#614)
• Ease code reading @guizmaii (#606)
• Prefer case object Poll @guizmaii (#608)
• Pass the ZIO Runtime in the RunLoop @guizmaii (#607)
• Update README.md @github-actions (#605)
• Add missing final @guizmaii (#603)

Dependency Updates

• Update zio, zio-streams, zio-test, ... to 2.0.7 @github-actions (#631)
• Update sbt-jmh to 0.4.4 @github-actions (#623)
• Update scala3-library to 3.2.2 @github-actions (#621)
• Update jackson-databind to 2.14.2 @github-actions (#615)
• Update embedded-kafka to 3.3.2 @github-actions (#613)
• Update zio-sbt-website to 0.3.10 @github-actions (#611)
• Update scalafmt-core to 3.7.1 @github-actions (#612)
• Update zio-sbt-website to 0.3.9 @github-actions (#610)
• Bump scala-steward-org/scala-steward-action from 2.47.0 to 2.49.0 @dependabot (#627)
• Bump actions/cache from 1 to 3 @dependabot (#619)

CI

• Fix mergify configuration @guizmaii (#625)
• Speed up CI: Do not scalafmt the codebase on compilation in CI @guizmaii (#618)
• Fix Benchmark CI configuration @guizmaii (#617)
• Fix Benchmark CI configuration @guizmaii (#616)