Fix containers name in docker compose files.

[afsg77]

Rename the containers to match the host name in the properties. Otherwise it fails to connect to the containers from the applications. Remove the 'version: 3' to match the main docker compose file.

Signed-off-by: Alejandro Fabian Silva Grife [email protected]

[onobc]

Hi @afsg77

Thanks for the contribution! I have a few questions/comments to clarify why this is needed etc..

🔹 What behavior exactly are you seeing when you say

Otherwise it fails to connect to the containers from the applications"?

A stacktrace may be helpful. Also what is "it" in the sentence above?

🔹 Can you clarify where the host name is currently being set when you say "Rename the containers to match the host name in the properties"?

🔹 If this change is required then it is required in all of the DC files in this directory as all of the container names are prefixed with dataflow-.

🔹 Can you please adjust your commit according to the DCO - there is something it is complaining about currently.

[afsg77]

Hi

Sure, gladly will clarify it!

I was trying guide:

https://dataflow.spring.io/docs/installation/local/docker/

With Kafka and Postgres files:

wget -O docker-compose.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/main/src/docker-compose/docker-compose.yml;
wget -O docker-compose-kafka.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/main/src/docker-compose/docker-compose-kafka.yml;
wget -O docker-compose-postgres.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/main/src/docker-compose/docker-compose-postgres.yml;

SCDF starts fine with:

docker-compose -f docker-compose.yml -f docker-compose-kafka.yml -f docker-compose-postgres.yml up

But when creating simple pipeline stream create --name simple-stream --definition "time | log" --deploy. It was showing in the logs (for log-sink application) the following stacktrace:

2025-02-06 16:52:18.510  WARN [log-sink,,] 1511 --- [           main] org.apache.kafka.clients.ClientUtils     : Couldn't resolve server PLAINTEXT://kafka-broker:9092 from bootstrap.servers as DNS resolution failed for kafka-broker
2025-02-06 16:52:18.535 ERROR [log-sink,,] 1511 --- [           main] o.s.cloud.stream.binding.BindingService  : Failed to create consumer binding; retrying in 30 seconds

org.springframework.cloud.stream.binder.BinderException: Exception thrown while starting consumer: 
	at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindConsumer(AbstractMessageChannelBinder.java:480) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindConsumer(AbstractMessageChannelBinder.java:92) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binder.AbstractBinder.bindConsumer(AbstractBinder.java:143) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binding.BindingService.doBindConsumer(BindingService.java:184) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binding.BindingService.bindConsumer(BindingService.java:137) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binding.AbstractBindableProxyFactory.createAndBindInputs(AbstractBindableProxyFactory.java:118) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binding.InputBindingLifecycle.doStartWithBindable(InputBindingLifecycle.java:58) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at java.base/java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source) ~[na:na]
	at org.springframework.cloud.stream.binding.AbstractBindingLifecycle.start(AbstractBindingLifecycle.java:57) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binding.InputBindingLifecycle.start(InputBindingLifecycle.java:34) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.20.jar!/:5.3.20]
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.20.jar!/:5.3.20]
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.20.jar!/:5.3.20]
	at java.base/java.lang.Iterable.forEach(Unknown Source) ~[na:na]
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.20.jar!/:5.3.20]
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.20.jar!/:5.3.20]
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.20.jar!/:5.3.20]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.20.jar!/:5.3.20]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.8.jar!/:2.6.8]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) ~[spring-boot-2.6.8.jar!/:2.6.8]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:420) ~[spring-boot-2.6.8.jar!/:2.6.8]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) ~[spring-boot-2.6.8.jar!/:2.6.8]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) ~[spring-boot-2.6.8.jar!/:2.6.8]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.6.8.jar!/:2.6.8]
	at org.springframework.cloud.stream.app.log.sink.kafka.LogSinkKafkaApplication.main(LogSinkKafkaApplication.java:29) ~[classes!/:3.2.1]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[log-sink-kafka-3.2.1.jar:3.2.1]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[log-sink-kafka-3.2.1.jar:3.2.1]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[log-sink-kafka-3.2.1.jar:3.2.1]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[log-sink-kafka-3.2.1.jar:3.2.1]
Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:537) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:143) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49) ~[kafka-clients-3.0.1.jar!/:na]
	at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createAdminClient(KafkaTopicProvisioner.java:260) ~[spring-cloud-stream-binder-kafka-core-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.doProvisionConsumerDestination(KafkaTopicProvisioner.java:230) ~[spring-cloud-stream-binder-kafka-core-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.provisionConsumerDestination(KafkaTopicProvisioner.java:197) ~[spring-cloud-stream-binder-kafka-core-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.provisionConsumerDestination(KafkaTopicProvisioner.java:86) ~[spring-cloud-stream-binder-kafka-core-3.2.4.jar!/:3.2.4]
	at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindConsumer(AbstractMessageChannelBinder.java:421) ~[spring-cloud-stream-3.2.4.jar!/:3.2.4]
	... 32 common frames omitted
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:489) ~[kafka-clients-3.0.1.jar!/:na]
	... 39 common frames omitted

It should be able to resolve from the service name. But it was not.

These are the properties that specified the kafka host:

• KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-broker:9092
• KAFKA_ADVERTISED_HOST_NAME=kafka-broker
• SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=PLAINTEXT://kafka-broker:9092
• SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_STREAMS_BINDER_BROKERS=PLAINTEXT://kafka-broker:9092
• SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=PLAINTEXT://kafka-broker:9092
• SPRING_CLOUD_DATAFLOW_APPLICATIONPROPERTIES_STREAM_SPRING_CLOUD_STREAM_KAFKA_STREAMS_BINDER_BROKERS=PLAINTEXT://kafka-broker:9092

I tried to update the properties to use the container name, but got similar stacktrace exception:

2025-02-06 17:04:13.569  WARN [log-sink,,] 183 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Error connecting to node dataflow-kafka:9092 (id: 1001 rack: null)

java.net.UnknownHostException: dataflow-kafka: Name or service not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:na]
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source) ~[na:na]
	at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source) ~[na:na]
	at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName0(Unknown Source) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[na:na]
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:979) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1127) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1387) ~[kafka-clients-3.0.1.jar!/:na]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1330) ~[kafka-clients-3.0.1.jar!/:na]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

Renaming the kafka container the same as the service works fine and the log-sink application was able to connect and run the simple pipeline.

This might indicate some issue with Docker’s built-in service discovery assigning the hostname to each service.
But I didn't see any customization in the files or hint of what was wrong. So, I just took the easiest path of renaming the containers to get it working.

For postgres and other services it seems to be working fine. So, not sure it will require this change in all DC files.
The version: '3' in the DC files is certainly not needed if it already has been removed from the main docker-compose.yml file.

The DCO was complaining since I signed with my full name and personal email and my git author had only my single name / last name with my work email. I have amended this in the commit.

Please let me know if I'm missing something or if you want me close this PR, create a ticket and then create a PR for the ticket with a different resolution. Or what are the next steps.

Thank you for your time!

[onobc]

Hi @afsg77 ,

I am unable to reproduce - this is working for me.

Please run these commands and report the result:

docker version
docker-compose --version

Also, here are the exact steps I took:

Get the compose files...

wget -O docker-compose.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/main/src/docker-compose/docker-compose.yml;

wget -O docker-compose-kafka.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/main/src/docker-compose/docker-compose-kafka.yml;

wget -O docker-compose-postgres.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/main/src/docker-compose/docker-compose-postgres.yml;

Start the stack...

export DATAFLOW_VERSION=2.11.5
export SKIPPER_VERSION=2.11.5

docker-compose -f docker-compose.yml -f docker-compose-kafka.yml -f docker-compose-postgres.yml up

Create and deploy stream in the SCDF UI...

• Navigate to the SCDF UI to create the stream via url http://localhost:9393/dashboard/index.html#/streams/list/create

• Enter time | log in "Stream Definition" panel

• Click "CREATE STREAM(S)"

• Name the stream "timeLogger1" (or whatever)

• Click "CREATE THE STREAM(S)"

• Select newly created stream by clicking its name

• Click "DEPLOY STREAM"

• Click "DEPLOY THE STREAM"

• Select newly created stream by clicking its name

• From the "RUNTIME" panel click "VIEW LOG" for the log sink app

At the very end of the log output you should see repeated entries like:

2025-02-09 00:58:45.596  INFO [log-sink,3f2f2a8cdd5ae764,e29d319003b4da07] 199 --- [container-0-C-1] log-sink                                 : 02/09/25 00:58:40
2025-02-09 00:58:45.599  INFO [log-sink,bed283792ca294e9,4ab6b798b8a2f389] 199 --- [container-0-C-1] log-sink                                 : 02/09/25 00:58:41
2

If you don't mind running the request version commands above and then trying the exact steps I posted above that would be helpful.

Thank you

[onobc]

Thats great news @afsg77 !

You are more than welcome. Docker can be flakey wrt to networking etc.. so who knows what it was? Glad its working now though.

I still think the version: '3' in the DC files is not needed.

Agreed. The best for that would be just to open a separate PR that just removes the version from all of the DC files. We would be most thankful if you could do that. If you don't get to it though - no worries.

[afsg77]

Hi @onobc ,

Sorry! I deleted the previous message before seeing your reply. I just added some more information.

Thanks heaps!

I tried the same steps you mentioned and wasn't able to reproduce it.

I had some additional configurations:

• export BP_JVM_VERSION=-jdk17
• Some volumes in the postgres service.

But even with those changes it's not showing in the simple stream.

I do see it sometimes when I create a simple cdc-debezium stream:

stream create --name cdc-stream --definition "cdc-debezium \
  --cdc.connector=postgres \
  --cdc.offset.storage=memory \
  --cdc.driverClassName=org.postgresql.Driver \
  --cdc.name=postgres-connector \
  --cdc.config.database.user=root \
  --cdc.config.database.password=rootpw \
  --cdc.config.database.dbname=dataflow \
  --cdc.config.database.hostname=postgres \
  --cdc.config.database.server.name=postgres \
  --cdc.config.database.port=5432 \
  --cdc.config.plugin.name=pgoutput \
  --cdc.schema=true | log" --deploy

But I'm not able to reliably reproduce it. Once it happens it also shows in the simple stream.
Not sure why it happens and renaming the containers works for me. Maybe some issues with my docker...

Anyway thanks!

I still think the version: '3' in the DC files is not needed.

I can create a different PR if you agree and want me to do that.

Really appreciate your time and response.

[onobc]

Hi @onobc ,

Sorry! I deleted the previous message before seeing your reply. I just added some more information.

Thanks heaps!

No worries. You are more than welcome.

I do see it sometimes when I create a simple cdc-debezium stream:

stream create --name cdc-stream --definition "cdc-debezium \
  --cdc.connector=postgres \
  --cdc.offset.storage=memory \
  --cdc.driverClassName=org.postgresql.Driver \
  --cdc.name=postgres-connector \
  --cdc.config.database.user=root \
  --cdc.config.database.password=rootpw \
  --cdc.config.database.dbname=dataflow \
  --cdc.config.database.hostname=postgres \
  --cdc.config.database.server.name=postgres \
  --cdc.config.database.port=5432 \
  --cdc.config.plugin.name=pgoutput \
  --cdc.schema=true | log" --deploy

But I'm not able to reliably reproduce it. Once it happens it also shows in the simple stream. Not sure why it happens and renaming the containers works for me.

Does it go away if you set hostname: kafka-broker rather than adjusting the container name? I think that may be what is needed.

Anyway thanks!

I still think the version: '3' in the DC files is not needed.

I can create a different PR if you agree and want me to do that.

Agreed. The best for that would be just to open a separate PR that just removes the version from all of the DC files. We would be most thankful if you could do that. If you don't get to it though - no worries.

Really appreciate your time and response.

Likewise. It was fun. Take care.