v1.7.0

[Magnus Edenhill (edenhill)]

librdkafka v1.7.0

librdkafka v1.7.0 is feature release:

• KIP-360 - Improve reliability of transactional producer.
  Requires Apache Kafka 2.5 or later.
• OpenSSL Engine support (ssl.engine.location) by AdiNigam (@adinigam) and ajbarb.

Enhancements

• Added connections.max.idle.ms to automatically close idle broker
  connections.
  This feature is disabled by default unless bootstrap.servers contains
  the string azure in which case the default is set to <4 minutes to improve
  connection reliability and circumvent limitations with the Azure load
  balancers (see Robustness and resiliency on Azure #3109 for more information).
• Bumped to OpenSSL 1.1.1k in binary librdkafka artifacts.
• The binary librdkafka artifacts for Alpine are now using Alpine 3.12.
• Improved static librdkafka Windows builds using MinGW (Bill Rose (@neptoess), Build static librdkafka_windows.a with travis #3130).

Upgrade considerations

• The C++ oauthbearer_token_refresh_cb() was missing a Handle *
  argument that has now been added. This is a breaking change but the original
  function signature is considered a bug.
  This change only affects C++ OAuth developers.
• KIP-735 The consumer session.timeout.ms
  default was changed from 10 to 45 seconds to make consumer groups more
  robust and less sensitive to temporary network and cluster issues.
• Statistics: consumer_lag is now using the committed_offset,
  while the new consumer_lag_stored is using stored_offset
  (offset to be committed).
  This is more correct than the previous consumer_lag which was using
  either committed_offset or app_offset (last message passed
  to application).

Fixes

General fixes

• Fix accesses to freed metadata cache mutexes on client termination (ThreadSanitizer: use of an invalid mutex (e.g. uninitialized or destroyed) (pid=7) #3279)
• There was a race condition on receiving updated metadata where a broker id
  update (such as bootstrap to proper broker transformation) could finish after
  the topic metadata cache was updated, leading to existing brokers seemingly
  being not available.
  One occurrence of this issue was query_watermark_offsets() that could return
  ERR__UNKNOWN_PARTITION for existing partitions shortly after the
  client instance was created.
• The OpenSSL context is now initialized with TLS_client_method()
  (on OpenSSL >= 1.1.0) instead of the deprecated and outdated
  SSLv23_client_method().
• The initial cluster connection on client instance creation could sometimes
  be delayed up to 1 second if a group.id or transactional.id
  was configured (Metadata requests on startup are slow (regression) #3305).
• Speed up triggering of new broker connections in certain cases by exiting
  the broker thread io/op poll loop when a wakeup op is received.
• SASL GSSAPI: The Kerberos kinit refresh command was triggered from
  rd_kafka_new() which made this call blocking if the refresh command
  was taking long. The refresh is now performed by the background rdkafka
  main thread.
• Fix busy-loop (100% CPU on the broker threads) during the handshake phase
  of an SSL connection.
• Disconnects during SSL handshake are now propagated as transport errors
  rather than SSL errors, since these disconnects are at the transport level
  (e.g., incorrect listener, flaky load balancer, etc) and not due to SSL
  issues.
• Increment metadata fast refresh interval backoff exponentially (ajbarb, Increment the backoff exponentially #3237).
• Unthrottled requests are no longer counted in the brokers[].throttle
  statistics object.
• Log CONFWARN warning when global topic configuration properties
  are overwritten by explicitly setting a default_topic_conf.

Consumer fixes

• If a rebalance happened during a consume_batch..() call the already
  accumulated messages for revoked partitions were not purged, which would
  pass messages to the application for partitions that were no longer owned
  by the consumer. Fixed by Jing Liu (@jliunyu). Don't call kinit cmd from rd_kafka_new() #3340.
• Fix balancing and reassignment issues with the cooperative-sticky assignor.
  Issues with "partition.assignment.strategy=cooperative-sticky" #3306.
• Fix incorrect detection of first rebalance in sticky assignor (Taylor Foxhall (@hallfox)).
• Aborted transactions with no messages produced to a partition could
  cause further successfully committed messages in the same Fetch response to
  be ignored, resulting in consumer-side message loss.
  A log message along the lines Abort txn ctrl msg bad order at offset 7501: expected before or at 7702: messages in aborted transactions may be delivered to the application
  would be seen.
  This is a rare occurrence where a transactional producer would register with
  the partition but not produce any messages before aborting the transaction.
• The consumer group deemed cached metadata up to date by checking
  topic.metadata.refresh.interval.ms: if this property was set too low
  it would cause cached metadata to be unusable and new metadata to be fetched,
  which could delay the time it took for a rebalance to settle.
  It now correctly uses metadata.max.age.ms instead.
• The consumer group timed auto commit would attempt commits during rebalances,
  which could result in "Illegal generation" errors. This is now fixed, the
  timed auto committer is only employed in the steady state when no rebalances
  are taking places. Offsets are still auto committed when partitions are
  revoked.
• Retriable FindCoordinatorRequest errors are no longer propagated to
  the application as they are retried automatically.
• Fix rare crash (assert rktp_started) on consumer termination
  (introduced in v1.6.0).
• Fix unaligned access and possibly corrupted snappy decompression when
  building with MSVC (Azat Khuzhin (@azat))
• A consumer configured with the cooperative-sticky assignor did
  not actively Leave the group on unsubscribe(). This delayed the
  rebalance for the remaining group members by up to session.timeout.ms.
• The current subscription list was sometimes leaked when unsubscribing.

Producer fixes

• The timeout value of flush() was not respected when delivery reports
  were scheduled as events (such as for confluent-kafka-go) rather than
  callbacks.
• There was a race conditition in purge() which could cause newly
  created partition objects, or partitions that were changing leaders, to
  not have their message queues purged. This could cause
  abort_transaction() to time out. This issue is now fixed.
• In certain high-thruput produce rate patterns producing could stall for
  1 second, regardless of linger.ms, due to rate-limiting of internal
  queue wakeups. This is now fixed by not rate-limiting queue wakeups but
  instead limiting them to one wakeup per queue reader poll. Random latency spikes when producing messages #2912.

Transactional Producer fixes

• KIP-360: Fatal Idempotent producer errors are now recoverable by the
  transactional producer and will raise a txn_requires_abort() error.
• If the cluster went down between produce() and commit_transaction()
  and before any partitions had been registered with the coordinator, the
  messages would time out but the commit would succeed because nothing
  had been sent to the coordinator. This is now fixed.
• If the current transaction failed while commit_transaction() was
  checking the current transaction state an invalid state transaction could
  occur which in turn would trigger a assertion crash.
  This issue showed up as "Invalid txn state transition: .." crashes, and is
  now fixed by properly synchronizing both checking and transition of state.

---

This discussion was created from the release v1.7.0.