How to use Microcks Docker with Apicurio Schema Registry

[CROprogrammer]

Is it possbile, and how to use Microcks installed using Docker in combination with Apicurio Schema Registry?

https://microcks.io/documentation/guides/avro-messaging/#1-setup-schema-registry is not useful to me because I'm trying to achive this without Helm Char or Operator based installation

[lbroudoux]

Hey! Currently, we do not provide docker-compose file for deploying this config.

However this should be possible simply adding an addon docker-compose file as we've done with async part. Something that is just doing the equivalent of docker run -it -p 8888:8080 apicurio/apicurio-registry-mem:latest

You'll also need to adjust properties in application.properties, something like:

kafka.schema.registry.url=http://localhost:8888/api
kafka.schema.registry.confluent=false

minion.default-avro-encoding=REGISTRY

[CROprogrammer]

Okay, I've put it there. I can run Microcks via Docker, and I get access to its UI. I've used User signed-up API example for my testing purposes and event of that type arrive to created topic. Only problem I'm facing now is that the schema of User signed-up API was not automatically registered in my schema registry. Any idea why is that?

[lbroudoux]

Schema should have been written in registry on first mock message publication. Any relevant logs on the server side? Check the microcks-async-minion container for that.

[CROprogrammer]

nothing relevant, still haven't found a way to do this

[lbroudoux]

Can't you see an info message saying Publishing on topic {%s}, record: %s ? If not present, could you try turning on the DEBUG level for microcks-async-minion container. For that you'll need to add those 2 properties to application.properties in /config folder:

quarkus.log.level=DEBUG
quarkus.log.console.level=DEBUG

[CROprogrammer]

Yes I can see that message

[CROprogrammer]

Hi, I'm trying this again, but again no luck :(

I've added this to application.properties:

# Schema registry config
kafka.schema.registry.url=http://localhost:8888
kafka.schema.registry.confluent=true

minion.default-avro-encoding=REGISTRY

and this to docker-compose-async-addon.yml:

...
apicurio:
    image: apicurio/apicurio-registry-mem:2.2.3.Final
    container_name: apicurio-schema-registry
    ports:
      - "8888:8080"

  app:
    depends_on:
      - mongo
      - keycloak
      - postman
      - apicurio
...

Then I've run: docker compose -f docker-compose.yml -f docker-compose-async-addon.yml -f kafdrop-addon.yml up

And Microcks started sending messages (User signed-up API - 0.1.1) to created Kafka topic, but it did not register schema in the schema registry. Is there a way to fix that?

[lbroudoux]

Hey there! Just had a look at it and looks like we've got a regression on this feature due to #782 ... Will have a look at it during the afternoon to see how we can quickly produce a patch.

[CROprogrammer]

Okay, thanks :D

[lbroudoux]

Created #862 and #863 to track these ones. Sorry for inconvience...

[lbroudoux]

Hey there!

Following our discussion and the fixes produced for #862 and #863, I ran a successful test this morning. Would you check that everything is now ok on your side before the patch release?

In order to do so, I replaced the microcks:1.7.1 and microcks-async-minion:1.7.1 images by microcks:maintenance and microcks-async-minion:maintenance in the 2 docker compose files. Then I updated the config/application.properties file with %docker-compose prefixed properties to enabled connecting to a locally deployed Apicurio Registry. Here's a copy of my config file:

# Async mocking support.
async-api.enabled=true

# Access to Microcks API server.
%docker-compose.io.github.microcks.minion.async.client.MicrocksAPIConnector/mp-rest/url=http://microcks:8080

# Access to Keycloak through docker network
%docker-compose.keycloak.auth.url=http://keycloak:8080

# Access to Kafka broker.
%docker-compose.kafka.bootstrap.servers=kafka:19092

# Do not save any consumer-offset on the broker as there's a re-sync on each minion startup.
%docker-compose.mp.messaging.incoming.microcks-services-updates.enable.auto.commit=false
%docker-compose.mp.messaging.incoming.microcks-services-updates.bootstrap.servers=kafka:19092

# Explicitly telling the minion the protocols we want to support
%docker-compose.minion.supported-bindings=KAFKA,WS

# %docker-compose.minion.supported-bindings=KAFKA,WS,MQTT,AMQP,NATS

# %docker-compose.mqtt.server=localhost:1883
# %docker-compose.mqtt.username=microcks
# %docker-compose.mqtt.password=microcks

# %docker-compose.amqp.server=localhost:5672
# %docker-compose.amqp.username=microcks
# %docker-compose.amqp.password=microcks

%docker-compose.kafka.schema.registry.url=http://host.docker.internal:8888
%docker-compose.kafka.schema.registry.confluent=false

%docker-compose.minion.default-avro-encoding=REGISTRY

%docker-compose.quarkus.log.level=DEBUG
%docker-compose.quarkus.log.console.level=DEBUG

If reusing a non empty instance, you'll have to first delete and then reimport the AsyncAPI+Avro file.

Thanks for feedback!

[CROprogrammer]

I'm still not getting my schemas registered in registry :/

Here are relevant files:

docker-compose.yml:

version: '2'

services:
  mongo:
    image: mongo:3.4.23
    container_name: microcks-db
    volumes:
      - "~/tmp/microcks-data:/data/db"

  keycloak:
    image: quay.io/keycloak/keycloak:20.0.2
    container_name: microcks-sso
    ports:
      - "18080:8080"
    environment:
      KEYCLOAK_ADMIN: "admin"
      KEYCLOAK_ADMIN_PASSWORD: "admin"
      KC_HOSTNAME_ADMIN_URL: "http://localhost:18080"
      KC_HOSTNAME_URL: "http://localhost:18080"
    volumes:
      - "./keycloak-realm/microcks-realm-sample.json:/opt/keycloak/data/import/microcks-realm.json"
    command:
      - start-dev --import-realm

  postman:
    image: quay.io/microcks/microcks-postman-runtime:latest
    container_name: microcks-postman-runtime

  apicurio:
    image: apicurio/apicurio-registry-mem:2.2.3.Final
    container_name: apicurio-schema-registry
    ports:
      - "8888:8080"

  app:
    depends_on:
      - mongo
      - keycloak
      - postman
      - apicurio
#    image: quay.io/microcks/microcks:1.7.1
    image: quay.io/microcks/microcks:maintenance
    container_name: microcks
    ports:
      - "8080:8080"
      - "9090:9090"
    environment:
      - SPRING_PROFILES_ACTIVE=prod
      - SPRING_DATA_MONGODB_URI=mongodb://mongo:27017
      - SPRING_DATA_MONGODB_DATABASE=microcks
      - POSTMAN_RUNNER_URL=http://postman:3000
      - TEST_CALLBACK_URL=http://microcks:8080
      - SERVICES_UPDATE_INTERVAL=0 0 0/2 * * *
      - KEYCLOAK_URL=http://keycloak:8080
      - KEYCLOAK_PUBLIC_URL=http://localhost:18080
      #- MAX_UPLOAD_FILE_SIZE=3MB

docker-compose-async-addon.yml:

version: '2'

services:
  zookeeper:
    image: quay.io/strimzi/kafka:0.30.0-kafka-3.1.0
    container_name: microcks-zookeeper
    command: [
      "sh", "-c", "bin/zookeeper-server-start.sh config/zookeeper.properties"
    ]
    ports:
      - "2181:2181"
    environment:
      LOG_DIR: /tmp/logs

  kafka:
    depends_on:
      - zookeeper
    image: quay.io/strimzi/kafka:0.30.0-kafka-3.1.0
    container_name: microcks-kafka
    command: [
      "sh", "-c",
      "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override listener.security.protocol.map=$${KAFKA_LISTENER_SECURITY_PROTOCOL_MAP} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
    ]
    ports:
      - "9092:9092"
      - "19092:19092"
    environment:
      LOG_DIR: "/tmp/logs"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

  app:
    depends_on:
      - mongo
      - keycloak
      - postman
      - apicurio
#    image: quay.io/microcks/microcks:1.7.1
    image: quay.io/microcks/microcks:maintenance
    container_name: microcks
    volumes:
      - "./config:/deployments/config"
    ports:
      - "8080:8080"
      - "9090:9090"
    environment:
      - SPRING_PROFILES_ACTIVE=prod
      - SPRING_DATA_MONGODB_URI=mongodb://mongo:27017
      - SPRING_DATA_MONGODB_DATABASE=microcks
      - POSTMAN_RUNNER_URL=http://postman:3000
      - TEST_CALLBACK_URL=http://microcks:8080
      - SERVICES_UPDATE_INTERVAL=0 0 0/2 * * *
      - KEYCLOAK_URL=http://keycloak:8080
      - KEYCLOAK_PUBLIC_URL=http://localhost:18080
      - ASYNC_MINION_URL=http://microcks-async-minion:8081
      - KAFKA_BOOTSTRAP_SERVER=kafka:19092

  async-minion:
    depends_on:
      - app
    ports:
      - "8081:8081"
#    image: quay.io/microcks/microcks-async-minion:1.7.1
    image: quay.io/microcks/microcks-async-minion:maintenance
    container_name: microcks-async-minion
    restart: on-failure
    volumes:
      - "./config:/deployments/config"
    environment:
      - QUARKUS_PROFILE=docker-compose

application.properties:

async-api.enabled=true

# Access to Microcks API server.
%docker-compose.io.github.microcks.minion.async.client.MicrocksAPIConnector/mp-rest/url=http://microcks:8080

# Access to Keycloak through docker network
%docker-compose.keycloak.auth.url=http://keycloak:8080

# Access to Kafka broker.
%docker-compose.kafka.bootstrap.servers=kafka:19092

# Do not save any consumer-offset on the broker as there's a re-sync on each minion startup.
%docker-compose.mp.messaging.incoming.microcks-services-updates.enable.auto.commit=false
%docker-compose.mp.messaging.incoming.microcks-services-updates.bootstrap.servers=kafka:19092

# Explicitly telling the minion the protocols we want to support
%docker-compose.minion.supported-bindings=KAFKA,WS

# %docker-compose.minion.supported-bindings=KAFKA,WS,MQTT,AMQP,NATS

# %docker-compose.mqtt.server=localhost:1883
# %docker-compose.mqtt.username=microcks
# %docker-compose.mqtt.password=microcks

# %docker-compose.amqp.server=localhost:5672
# %docker-compose.amqp.username=microcks
# %docker-compose.amqp.password=microcks

# Schema Registry
%docker-compose.kafka.schema.registry.url=http://host.docker.internal:8888
%docker-compose.kafka.schema.registry.confluent=false

%docker-compose.minion.default-avro-encoding=REGISTRY

# Logging
%docker-compose.quarkus.log.level=DEBUG
%docker-compose.quarkus.log.console.level=DEBUG

Any idea why it does not work? I use the newest version from master branch.

[CROprogrammer]

What did you mean by ´If reusing a non empty instance, you'll have to first delete and then reimport the AsyncAPI+Avro file.´
Do I have to somehow upload .asvc Avro schema file separately from AsyncAPI for it to be registered in registry?

[CROprogrammer]

I found out that I was using wrong AsyncAPI schema. I should be using: https://github.com/microcks/microcks/blob/master/webapp/src/test/resources/io/github/microcks/util/asyncapi/user-signedup-avro-ref-asyncapi.yaml

But when I use: https://github.com/microcks/microcks/blob/master/webapp/src/test/resources/io/github/microcks/util/asyncapi/user-signedup-avro-asyncapi.yaml
or
https://github.com/microcks/microcks/blob/master/webapp/src/test/resources/io/github/microcks/util/asyncapi/user-signedup-avro-ref-asyncapi.yaml
I don't get any messages in my topic. I get an error message Failed finding a suitable Avro schema for the 'SUBSCRIBE user/signedup' operation. No publication done.
Any idea why?

[CROprogrammer]

I've achived that messages are written to the Kafka topic and schemas registered in schema registry.
This issue: #782 helped me. I had to import AsyncAPI schema that has reference to Avro schema using raw link to schema from GitHub: https://raw.githubusercontent.com/microcks/microcks/master/webapp/src/test/resources/io/github/microcks/util/asyncapi/user-signedup-avro-ref-asyncapi.yaml.