Skip to content

Support jruby #516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mensfeld opened this issue Oct 7, 2024 · 11 comments · Fixed by #571
Closed

Support jruby #516

mensfeld opened this issue Oct 7, 2024 · 11 comments · Fixed by #571
Assignees

Comments

@mensfeld
Copy link
Member

mensfeld commented Oct 7, 2024

I run tests on most recent jruby stable and FFI aside from one case (fixable) works ok. We could rollout the jruby support easily.

@mensfeld mensfeld self-assigned this Oct 7, 2024
@headius
Copy link

headius commented Oct 7, 2024

I look forward to playing with benchmarks and seeing how we can make the JRuby support superfast!

@mensfeld
Copy link
Member Author

mensfeld commented Oct 8, 2024

Things we need to do to make this happen:

  • Fix segfaults related to oauth bearear callback assignments
  • Ignore fork specs for jruby
  • Benchmark performance and check if it makes sense (at least 50k msg/s passthrough, ideally more than 100k)

@headius
Copy link

headius commented Oct 8, 2024

oauth bearear callback assignments

FFI? There's at least one known JRuby issue with FFI callbacks on Apple Silicon that requires some low-level C work (probably by me 😭).

@mensfeld
Copy link
Member Author

@headius, the thing is, other callbacks work as expected, so I think it's rather a misuse/misconfiguration. I'll get back to you next week when I look into this with more info.

@headius
Copy link

headius commented Mar 12, 2025

Revisiting this because it came up today...

Still crashes on my M1 MBA during specs:

...
  oauthbearer set token
    without args
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000012a612ccc, pid=44035, tid=9219
#
# JRE version: OpenJDK Runtime Environment Zulu21.38+21-CA (21.0.5+11) (build 21.0.5+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.38+21-CA (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  [librdkafka.dylib+0x11eccc]  rd_kafka_oauthbearer_set_token0+0x28

Weirdly enough, if I force JIT in JRuby, it does not crash. There are some peculiar warnings, though:

  oauthbearer set token
    without args
      should raise argument error
    with args
%5|1741803010.316|CONFWARN|rdkafka#consumer-51| [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster
      should set token or capture failure
  oauthbearer set token failure
    without args
      should fail
    with args
%5|1741803010.322|CONFWARN|rdkafka#consumer-52| [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster
      should succeed
  oauthbearer callback
    without an oauthbearer callback
      should do nothing
    with an oauthbearer callback
%5|1741803010.326|CONFWARN|rdkafka#consumer-53| [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster
      should call the oauth bearer callback and receive config and client name

Final result of this forced-JIT run:

Failures:

  1) Rdkafka::Admin when operating from a fork expect to be able to create topics and run other admin operations without hanging
     Failure/Error:
       pid = fork do
         admin
           .create_topic(topic_name, topic_partition_count, topic_replication_factor)
           .wait
       end
     
     NotImplementedError:
       fork is not available on this platform
     # ./spec/rdkafka/admin_spec.rb:745:in 'block in <main>'
     # ./spec/spec_helper.rb:154:in 'block in <main>'
     # ./spec/spec_helper.rb:153:in 'block in <main>'

  2) Rdkafka::Config logger expect to start new logger thread after fork and work
     Failure/Error:
       pid = fork do
         $stdout.reopen(writer)
         Rdkafka::Config.logger = Logger.new($stdout)
         reader.close
         producer = rdkafka_producer_config(debug: 'all').producer
         producer.close
         writer.close
         sleep(1)
       end
     
     NotImplementedError:
       fork is not available on this platform
     # ./spec/rdkafka/config_spec.rb:39:in 'block in <main>'
     # ./spec/spec_helper.rb:154:in 'block in <main>'
     # ./spec/spec_helper.rb:153:in 'block in <main>'

Finished in 5 minutes 18 seconds (files took 3.66 seconds to load)
362 examples, 2 failures, 1 pending

Not bad (and obviously fork specs should just be skipped), but we obviously need to figure out that crash.

@mensfeld
Copy link
Member Author

Yeah I do recall this crash on oauth. WHen you disable oauth bearer, is the rest of the specs working as expected?

@mensfeld
Copy link
Member Author

Btw @headius we can work on this somewhere around RubyKaigi if you fancy.

@mensfeld
Copy link
Member Author

I can reproduce:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007e3fee1c9d32, pid=15776, tid=15807
#
# JRE version: OpenJDK Runtime Environment (21.0.6+7) (build 21.0.6+7-Ubuntu-124.04.1)
# Java VM: OpenJDK 64-Bit Server VM (21.0.6+7-Ubuntu-124.04.1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [librdkafka.so+0x1c9d32]  rd_kafka_oauthbearer_set_token0+0x12
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/mencio/Software/Karafka/rdkafka-ruby/core.15776)
#
# An error report file with more information is saved as:
# /home/mencio/Software/Karafka/rdkafka-ruby/hs_err_pid15776.log
[43.535s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-21
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#



@mensfeld
Copy link
Member Author

mensfeld commented Apr 16, 2025

bundle exec rspec spec/rdkafka/bindings_spec.rb:152

@mensfeld
Copy link
Member Author

@mensfeld
Copy link
Member Author

ref karafka/karafka#2549

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

2 participants