Having issue running on server versus locally; TLS SSL_HANDSHAKE PLAINTEXT conflict?

[bhugh-chs]

I'm using almost the identical settings locally and on a server in EC2. Local instance is using Ubuntu. EC2 instance using RHEL.

• Same approach used for the certificates.
• Using kcat to verify connectivity. My local install works as expected.
• We are using Confluent Cloud with Private networking. Both local and EC2 instances are on the private network.
• We have been able to deploy Kroxylicious with the approach mentioned above and it works. So there must be some nuance between the configuration and deployment that is causing a conflict.
• Using kafka-proxy version 0.3.12

Only thing different is where we are running the kafka proxy from. The rest is the same.
Certs are specific to the server as well.

Appreciate if anyone can see what might be causing the below issue

Here are the configurations for kcat I'm using (only change the bootstrap server)
kcat commands
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
api.version.request=true

Localhost

#bootstrap.servers=localhost:9092

grepplabs

bootstrap.servers=redacted.aws.redacted.ds:9092
sasl.username=[redacted]
sasl.password=[redacted]

Below are some of the errors we are seeing.

Client running kcat
Client Errors
root@514e95fdb5b1:/opt/grepplabs# kcat -F kcat.conf -L
% Reading configuration from file kcat.conf
%3|1737476147.481|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://redacted.aws.redacted.ds:9092/bootstrap]: sasl_ssl://redacted.aws.redacted.ds:9092/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 105ms in state SSL_HANDSHAKE)
%3|1737476147.688|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://redacted.aws.redacted.ds:9092/bootstrap]: sasl_ssl://redacted.aws.redacted.ds:9092/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 105ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m ?)
root@514e95fdb5b1:/opt/grepplabs#

Kafka Proxy
Proxy Errors
INFO[2025-01-21T16:14:53Z] Ready for new connections
INFO[2025-01-21T16:15:47Z] New connection for lkc-redacted.redacted.us-east-1.aws.confluent.cloud:9092
INFO[2025-01-21T16:15:47Z] Reading data from local connection on 10.30.112.244:9092 from 10.99.93.212:53972 (lkc-zno6z7.redacted.us-east-1.aws.confluent.cloud:9092) had error: api key 16897 is invalid, possible cause: using plain connection instead of TLS
INFO[2025-01-21T16:15:47Z] New connection for lkc-redacted.redacted.us-east-1.aws.confluent.cloud:9092
INFO[2025-01-21T16:15:47Z] Reading data from local connection on 10.30.112.244:9092 from 10.99.93.212:53973 (lkc-zno6z7.redacted.us-east-1.aws.confluent.cloud:9092) had error: api key 16897 is invalid, possible cause: using plain connection instead of TLS
INFO[2025-01-21T16:15:47Z] New connection for lkc-redacted.redacted.us-east-1.aws.confluent.cloud:9092
INFO[2025-01-21T16:15:47Z] Reading data from local connection on 10.30.112.244:9092 from 10.99.93.212:53974 (lkc-zno6z7.redacted.us-east-1.aws.confluent.cloud:9092) had error: api key 16897 is invalid, possible cause: using plain connection instead of TLS
INFO[2025-01-21T16:15:48Z] New connection for lkc-redacted.redacted.us-east-1.aws.confluent.cloud:9092
INFO[2025-01-21T16:15:48Z] Reading data from local connection on 10.30.112.244:9092 from 10.99.93.212:53975 (lkc-zno6z7.redacted.us-east-1.aws.confluent.cloud:9092) had error: api key 16897 is invalid, possible cause: using plain connection instead of TLS
INFO[2025-01-21T16:15:48Z] New connection for lkc-redacted.redacted.us-east-1.aws.confluent.cloud:9092
INFO[2025-01-21T16:15:48Z] Reading data from local connection on 10.30.112.244:9092 from 10.99.93.212:53976 (lkc-zno6z7.redacted.us-east-1.aws.confluent.cloud:9092) had error: api key 16897 is invalid, possible cause: using plain connection instead of TLS
INFO[2025-01-21T16:15:48Z] New connection for lkc-redacted.redacted.us-east-1.aws.confluent.cloud:9092

[bhugh-chs]

We figured this issue out. Turned out to be a number of conflicts that we later resolved.