v2.7.3

Noteworthy Changes

• Collect consumer metrics @erikvanoosten (#1143)

Zio-kafka now passes a lot of metrics to zio. Use any zio metrics backend to expose them! See the issue #1143 for more information on which metrics are collected.

Changes

• Update zio, zio-streams, zio-test, ... to 2.0.21 @zio-scala-steward (#1144)
• Add scala steward badge (2nd attempt) @erikvanoosten (#1157)
• Update README.md @zio-assistant (#1155)
• Ensure that fibers forked in Runloop.make are interruptible @svroonland (#1154)
• Add scala steward badge @erikvanoosten (#1145)
• Collect consumer metrics @erikvanoosten (#1143)
• Update zio-logging-slf4j, ... to 2.2.0 @zio-scala-steward (#1152)
• Update zio-logging-slf4j, ... to 2.1.17 @zio-scala-steward (#1151)
• Update README.md @zio-assistant (#1150)
• Update zio-kafka, zio-kafka-testkit to 2.7.2 @zio-scala-steward (#1149)

v2.7.2

Note worthy changes

• Add partitionsFor method to producer @kyri-petrou (#1147)
• retry Consumer.CommitTimeout exception @myazinn (#1140)

Other changes

• Use Semaphore @adamgfraser (#1142)
• Make tests faster @erikvanoosten (#1139)
• Delete RunloopTimeout and small improvements @erikvanoosten (#1141)
• Update kafka-clients to 3.6.1 @zio-scala-steward (#1131)
• Improve support and documentation for manual offsets @erikvanoosten (#1135)
• Update sbt to 1.9.8 @zio-scala-steward (#1138)
• Update zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1137)
• Update zio, zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1136)
• Update logback-classic to 1.4.14 @zio-scala-steward (#1129)
• Update logback-classic to 1.3.14 @zio-scala-steward (#1128)
• Update zio-kafka, zio-kafka-testkit to 2.7.1 @zio-scala-steward (#1127)
• Remove unused config for Mergify @erikvanoosten (#1124)
• Adds test for stopping consumption while doing async commits @erikvanoosten (#1123)
• Update README.md @zio-assistant (#1125)
• Update logback-classic to 1.4.13 @zio-scala-steward (#1122)

v2.7.1

Introducing Rebalance Safe Commits

This release introduces a new experimental mode that prevent duplicate message processing during a rebalance. This mode has been in the making for 8 months and has been implemented over 9 different PRs.

We have marked Rebalance Safe Commits as an experimental feature because zio-kafka is used in many different and sometimes surprising ways. Some of those are not compatible with the new mode.

RebalanceSafeCommits can be enabled via ConsumerSettings.withRebalanceSafeCommits. Please read the scaladocs to see if this mode is applicable to how you use zio-kafka.

Much more information on the new mode (motivation, why it works, the challenges, etc.), are available in the descriptions of PR Await commits during a rebalance. A manual is forthcoming.

Changes

• Make maxRebalanceDuration configurable @erikvanoosten (#1118)
• Update zio-logging-slf4j, ... to 2.1.16 @zio-scala-steward (#1114)
• Automatically add new flame graphs to gh_pages, prune old @erikvanoosten (#1105)
• Await commit during a rebalance @erikvanoosten (#1098)
• Update README.md @zio-assistant (#1113)
• Update zio-kafka, zio-kafka-testkit to 2.7.0 @zio-scala-steward (#1112)

v2.7.0

Improvements

Zio-kafka became a little bit faster by doing less work when there is no rebalance.

Really large and tiny messages are better described in the tuning documentation, and ConsumerSettings.withoutPartitionPreFetching was introduced to make it easier to disable pre-fetching.

Previously, in some use cases, you could end up with a live-lock trying to get access to the underlying java consumer. This should no longer happen.

Breaking changes

Zio-kafka no longer supports scala 2.12. Scala 2.13 has been available for 4½ year so everyone has had plenty of time to move to a more modern scala version. (#1104)

This release has some some minor breaking changes:

• The diagnostics events for rebalances have been improved in an incompatible way (#1102)
• Jackson was removed as a dependency (#1096)
• partitionPreFetchBufferLimit is now 0-based instead of 1-based (#1091)

Changes

• Update scalafmt-core to 3.7.17 @zio-scala-steward (#1111)
• Make access to the java consumer fair @flavienbert (#1109)
• Extend tuning docs @erikvanoosten (#1110)
• Protect agains user diagnostics, better rebalance events @erikvanoosten (#1102)
• Drop support for scala 2.12 @erikvanoosten (#1104)
• Update zio-streams, zio-test-sbt to 2.0.19 @zio-scala-steward (#1101)
• Update zio, zio-streams, zio-test, ... to 2.0.19 @zio-scala-steward (#1100)
• Update zio-logging-slf4j, ... to 2.1.15 @zio-scala-steward (#1099)
• Track latest completed commit offset per partition @erikvanoosten (#1097)
• Always end streams in rebalance listener, support lost partitions @erikvanoosten (#1089)
• Remove jackson dependency @erikvanoosten (#1096)
• Make partitionPreFetchBufferLimit value 0 disable prefetching @erikvanoosten (#1091)
• Update zio-kafka, zio-kafka-testkit to 2.6.0 @zio-scala-steward (#1092)
• Update README.md @zio-assistant (#1090)
• Prevent double logging in tests @erikvanoosten (#1088)

v2.6.0

Noticeable Changes

• Automatically merge commits by @erikvanoosten in #1073
  In this PR zio-kafka automatically merges all commits that were generated in the course of a single run of the runloop. This frees users from having to merge streams and do the commit merging themselves.
• Add processing timeout at the stream level by @erikvanoosten in #1009
  Say good bye to the runloop-timeout! From this version on we respect Kafka configuration max.poll.interval.ms. When a stream does not consume any records for max.poll.interval.ms or longer (while records are available), the stream will be interrupted and the consumer closed. This allows another member of the consumer group to take over the work.
• Add tuning options and docs by @erikvanoosten in #1081
  A new tuning guide on https://zio.dev/zio-kafka/.
• Enable the consumer to commit an offset with metadata by @flavienbert in #1067

See the PR descriptions for more information.

All Changes

• Update zio-kafka, zio-kafka-testkit to 2.5.0 by @zio-scala-steward in #1070
• Update README.md by @zio-assistant in #1071
• Gitignore cleanup by @erikvanoosten in #1072
• Small cleanup by @erikvanoosten in #1065
• Do not run CI and benches for gh-pages branch by @erikvanoosten in #1075
• Add instructions for pruning the benchmark history by @erikvanoosten in #1077
• Automatically merge commits by @erikvanoosten in #1073
• Update zio-sbt to 0.4.0-alpha.22 by @erikvanoosten in #1078
• Update jackson-databind to 2.15.3 by @zio-scala-steward in #1080
• Add processing timeout at the stream level by @erikvanoosten in #1009
• Fix flaky test by @erikvanoosten in #1082
• Fix flaky test again by @erikvanoosten in #1083
• Add tuning options and docs by @erikvanoosten in #1081
• Add unit test for QueueSizeBasedFetchStrategy by @erikvanoosten in #1017
• Update sbt to 1.9.7 by @zio-scala-steward in #1084
• Update scalafmt-core to 3.7.15 by @zio-scala-steward in #1085
• Enable the consumer to commit an offset with metadata by @flavienbert in #1067
• Keep track of last pulled offset by @erikvanoosten in #1086

New Contributors

• @flavienbert made their first contribution in #1067

Full Changelog: v2.5.0...v2.6.0

v2.5.0

Noticeable Changes

• Update kafka-clients to 3.6.0 @zio-scala-steward (#1068)
• Drop support of Java 8, 9 and 10. Minimal supported version is Java 11 (#1059)

All Changes

• Update kafka-clients to 3.6.0 @zio-scala-steward (#1068)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.21 @zio-scala-steward (#1059)
• Update README.md @zio-assistant (#1062)
• Update README.md @zio-assistant (#1061)
• Update zio-streams, zio-test-sbt to 2.0.18 @zio-scala-steward (#1060)
• Update zio-kafka, zio-kafka-testkit to 2.4.3 @zio-scala-steward (#1058)

v2.4.3

Noticeable Changes

• [Breaking] Bootstrap servers is regular config @lukestephenson (#1038)

  • AdminClientSettings::bootstrapServers field has been removed and replaced with the AdminClientSettings::withBootstrapServers method.
  • ConsumerSettings::bootstrapServers field has been removed. Use ConsumerSettings::withBootstrapServers method instead.
  • ProducerSettings::bootstrapServers field has been removed. Use ProducerSettings::withBootstrapServers method instead.

• Fix Consumer.fromJavaConsumer and Producer.fromJavaProducer type signatures @devsprint (#1033)

All Changes

• Put back the ProducerSettings::driverSettings and AdminClientSettings::driverSettings methods to avoid a useless breaking change and to provide a more unified interface through all the settings case classes @guizmaii (#1057)
• Update zio, zio-streams, zio-test, ... to 2.0.18 @zio-scala-steward (#1056)
• Update zio-streams, zio-test-sbt to 2.0.17 @zio-scala-steward (#1053)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.19 @zio-scala-steward (#1054)
• Fix Scala Steward configuration @guizmaii (#1052)
• Use ZIO Scala Steward Github app with the scala-steward CI @guizmaii (#1051)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.18 @github-actions (#1044)
• Update zio, zio-streams, zio-test, ... to 2.0.17 @github-actions (#1049)
• Update sbt to 1.9.6 @github-actions (#1050)
• Update scala-library to 2.13.12 @github-actions (#1047)
• Update sbt-scalafix to 0.11.1 @github-actions (#1046)
• Bootstrap servers is regular config @lukestephenson (#1038)
• [Internal] Micro-optimisation: Only compute the requestedPartitions Set if necessary @guizmaii (#1021)
• Update scala3-library to 3.3.1 @github-actions (#1037)
• Update scalafmt-core to 3.7.14 @github-actions (#1034)
• Fix Consumer.fromJavaConsumer and Producer.fromJavaProducer type signatures @devsprint (#1033)
• [Internals] Fix build @guizmaii (#1032)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.17 @github-actions (#1031)
• Update scalafmt-core to 3.7.13 @github-actions (#1030)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.14 @github-actions (#1029)
• Update sbt to 1.9.4 @github-actions (#1027)
• Update zio-streams, zio-test-sbt to 2.0.16 @github-actions (#1026)
• Update zio, zio-streams, zio-test, ... to 2.0.16 @github-actions (#1025)
• Update logback-classic to 1.4.11 @github-actions (#1020)
• Update logback-classic to 1.3.11 @github-actions (#1019)
• Update README.md @zio-assistant (#1016)
• Update README.md @zio-assistant (#1015)
• Update zio-kafka, zio-kafka-testkit to 2.4.2 @github-actions (#1013)
• Update zio-logging-slf4j, ... to 2.1.14 @github-actions (#1014)

v2.4.2

Noticeable Changes

• Update kafka-clients to 3.5.1 @github-actions (#994)

• Add note about partition reassignment with multiple subscriptions @svroonland (#1010)

• Document pollTimeout setting @erikvanoosten (#995)

• new ConsumerSettings::commitTimeout configuration. Default value: 15 seconds
  This timeout is a workaround for a deadlock that can occur when committing. Details #852

• Consumer:

  • new Consumer.fromJavaConsumer function to make a zio-kafka Consumer

• Producer:

  • new Producer.fromJavaProducer function to make a zio-kafka Producer
  • Fix signature of Producer#produceAll @vladimirkl (#993)
  • Enriched interface with the following additional new methods:

  trait Producer {
  
    def produce(
      record: ProducerRecord[Array[Byte], Array[Byte]]
    ): Task[RecordMetadata]
  
    def produceAsync(
      record: ProducerRecord[Array[Byte], Array[Byte]]
    ): Task[Task[RecordMetadata]]
  
    def produceChunk(
      records: Chunk[ProducerRecord[Array[Byte], Array[Byte]]]
    ): Task[Chunk[RecordMetadata]]
    
    def produceChunkAsync(
      records: Chunk[ProducerRecord[Array[Byte], Array[Byte]]]
    ): Task[Task[Chunk[RecordMetadata]]]
  
    def produceChunkAsyncWithFailures(
      records: Chunk[ProducerRecord[Array[Byte], Array[Byte]]]
    ): UIO[UIO[Chunk[Either[Throwable, RecordMetadata]]]]
  
  }

• Improvement of the SslHelper.validateEndpoint mechanism to make it as resilient as Kafka: If some Kafka bootstrap servers are down, that shouldn't affect the result of this function. Details #950

All Changes

• [Internal] Don't run benchmarks on forks @guizmaii (#989)
• Producer.produceChunkAsyncWithFailures @lukestephenson (#879)
• Update embedded-kafka to 3.5.1 @github-actions (#1012)
• Add note about partition reassignment with multiple subscriptions @svroonland (#1010)
• Create Consumer and Producer from existing apache kafka Consumer/Producer @svroonland (#1011)
• Document setting pollTimeout @erikvanoosten (#995)
• Update logback-classic to 1.4.9 @github-actions (#1008)
• Update logback-classic to 1.3.9 @github-actions (#1007)
• [Internal] Reorganize Consumer code @guizmaii (#999)
• Update scalafmt-core to 3.7.12 @github-actions (#1006)
• Update scalafmt-core to 3.7.11 @github-actions (#1002)
• Update sbt-tpolecat to 0.5.0 @github-actions (#1001)
• Update sbt to 1.9.3 @github-actions (#1000)
• [Internal] Re-organise Producer code @guizmaii (#998)
• Enrich Producer interface @guizmaii (#990)
• Fix signature of Producer#produceAll @vladimirkl (#993)
• Update kafka-clients to 3.5.1 @github-actions (#994)
• Fix: Runloop might be suspended when assigned with many partitions @yaarix (#986)
• Fix: Make the SslHelper.validateEndpoint as resilient as Kafka. If some Kafka bootstraps servers are down, that shouldn't affect the result of the validation @guizmaii (#950)
• Update scalafmt-core to 3.7.10 @github-actions (#985)
• Improve Scalafix configuration @guizmaii (#984)
• Implement default commit timeout @vladimirkl (#982)
• Prevent asInstanceOf @erikvanoosten (#963)
• Fix CI @guizmaii (#981)
• Update scalafmt-core to 3.7.9 @github-actions (#979)
• Update sbt-tpolecat to 0.4.4 @github-actions (#978)
• Update scalafmt-core to 3.7.8 @github-actions (#976)
• Update sbt to 1.9.2 @github-actions (#973)
• Update sbt-tpolecat to 0.4.3 @github-actions (#972)
• Update scalafmt-core to 3.7.7 @github-actions (#971)
• Fix scala 2.12 scaladoc @erikvanoosten (#969)
• Introduce FetchStrategy to allow custom fetch strategies @erikvanoosten (#953)
• Update scalafmt-core to 3.7.6 @github-actions (#968)
• Update zio-kafka, zio-kafka-testkit to 2.4.1 @github-actions (#967)
• Update README.md @zio-assistant (#965)
• Update README.md @zio-assistant (#964)
• Only calculate initial state once @erikvanoosten (#962)
• Document ConsumerSettings better @erikvanoosten (#961)

🌱 Dependency Updates

• Bump actions/setup-java from 3.11.0 to 3.12.0 @dependabot (#1005)
• Bump scala-steward-org/scala-steward-action from 2.59.0 to 2.61.0 @dependabot (#997)
• Bump benchmark-action/github-action-benchmark from 1.17.0 to 1.18.0 @dependabot (#974)
• Bump scala-steward-org/scala-steward-action from 2.56.0 to 2.59.0 @dependabot (#975)

v2.4.1

Noticeable Changes

• [Fix] Issue #856 is now fixed
• [Fix] After a call to consumer.stopConsumption, starting a new consumption with the same Consumer instance was not doing anything (fixed in #857)
• [Breaking] The Producer.withProducerService(f) function has been removed. You need to replace it with ZIO.serviceWithZIO[Producer](f)

All Changes

• Micro-optimisation for manual consumption mode: Use only 1 IO to execute all the c.seek calls in doSeekForNewPartitions @guizmaii (#900)
• Using a Consumer without consuming shouldn't throw a RunloopTimeout exception @guizmaii (#857)
• Update scalafmt-core to 3.7.5 @github-actions (#952)
• Update sbt to 1.9.1 @github-actions (#948)
• Clean some code of LiveTransactionalProducer and Producer @guizmaii (#947)
• Update zio-kafka, zio-kafka-testkit to 2.4.0 @github-actions (#946)
• Update README.md @zio-assistant (#945)

v2.4.0

Noticeable Changes

1. Update kafka-clients from 3.4.1 to 3.5.0

2. Update Scala 2.12 to 2.12.8 and Scala 2.13 to 2.13.11

3. The optimistic prefetch mechanism, which we called "optimistic resume" in previous releases, introduced in v2.3.0 has been removed because of several performance issues.
   Instead, the prefetch mechanism, as present in v2.2 and earlier, has been put back and optimized by fixing a race condition in the enqueueing of internal messages to fetch more data and the next poll cycle starting.

   The prefetch amount is now controlled with a ConsumerSettings::maxPartitionQueueSize setting instead of the old ConsumerSettings::perPartitionChunkPrefetch setting.
   See ConsumerSettings::maxPartitionQueueSize documentation: #943

   The ConsumerSettings::enableOptimisticResume setting introduced in v2.3.2 has therefore been removed.

All Changes

• Document maxPartitionQueueSize setting @svroonland (#943)
• Review #803: Optimize partitionsToFetch computation @guizmaii (#942)
• Improve prefetch @svroonland (#803)
• Use less heap by calling offer directly @erikvanoosten (#940)
• Fix Consumer.consumeWith interface @guizmaii (#939)
• updating kafka-clients to version 3.5.0 @Casper12345 (#936)
• Revert "Optimistically resume partitions early" @svroonland (#932)
• [Internal] Only evaluate if (hasGroupId) once in getConsumerGroupMetadataIfAny @guizmaii (#931)
• Update scala-collection-compat to 2.11.0 @github-actions (#928)
• Revert "Update kafka-clients to 3.5.0 (#925)" @guizmaii (#927)
• Update embedded-kafka to 3.4.1 @github-actions (#926)
• Update kafka-clients to 3.5.0 @github-actions (#925)
• Update logback-classic to 1.3.8 @github-actions (#922)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.12 @github-actions (#924)
• Avoid using ZIO.succeed when wrapping Kafka code @guizmaii (#921)
• [DX] Fix ConsumerSettings interface @guizmaii (#919)
• Fix project compilation @guizmaii (#916)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.11 @github-actions (#912)
• Update README.md @zio-assistant (#915)
• Update scala-library to 2.12.18 @github-actions (#913)
• Update scala-library to 2.13.11 @github-actions (#914)
• Update zio-kafka, zio-kafka-testkit to 2.3.2 @github-actions (#911)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.54.0 to 2.56.0 @dependabot (#934)