Microcks + Copilot, Samples Generation

[der-equinix]

Hi,

I'm using Microcks and Copilot to generate samples. When I tested with example APIs from Microcks, like the Pastry API, it generated samples successfully. However, after importing our API and testing it, the samples were not generated properly.

Any one has tried similar?

[lbroudoux]

Hi @der-equinix,
Is it a matter of exception being raised, timeout or maximum number of tokens being reached? Could you share some logs or a reproducer so that we can try to tune this?

[yada]

Your settings seem to be good, but it all depends on your openAPI specs... You can check your Spec and get some input using our Spectral rules. See: https://github.com/microcks/microcks-spectral-ruleset

I guess it can help. Debug mode and logs can provide more details as well but try also using openAI; see my question above...
"Can you try using OpenAI first to simplify it before using Ollama and on-prem LLM?"

[der-equinix]

I tried importing my API using specs, that went fine and my examples showed as samples. Later I tried to generate few mare samples using copilot and it was generating empty samples with only sample name and endpoint, without Header and response.

Is this what you want me to try basically?

If yes, I am good till here. Now please help me to enable logs and to get more details to know where it is failing to generate samples.

[lbroudoux]

Hey @der-equinix
I'm back from a few days off. Sorry for the delay in answer. To help you troubleshoot this, I have initialized a guide with how to change the log level: https://microcks.io/documentation/guides/troubleshoot/#changing-the-log-level
Let me know if it helps you.

[der-equinix]

Yes it helped to configure.

Please check these are changes done and reflecting in docker

Post this tried with my steps to generate the samples.

At console no Debug logs found

[lbroudoux]

It looks like your compose file is not correct. I put # [...] as an ellipsis to represent prexisting content.

On my machine, the complete file is:

version: '2'

services:
  mongo:
    image: mongo:4.4.29
    container_name: microcks-db
    volumes:
      - "~/tmp/microcks-data:/data/db"

  keycloak:
    image: quay.io/keycloak/keycloak:26.0.0
    container_name: microcks-sso
    ports:
      - "18080:8080"
    environment:
      KEYCLOAK_ADMIN: "admin"
      KEYCLOAK_ADMIN_PASSWORD: "admin"
    volumes:
      - "./keycloak-realm/microcks-realm-sample.json:/opt/keycloak/data/import/microcks-realm.json"
    command: ["start-dev", "--hostname=http://localhost:18080", "--import-realm"]

  postman:
    image: quay.io/microcks/microcks-postman-runtime:0.6.0
    container_name: microcks-postman-runtime

  app:
    depends_on:
      - mongo
      - keycloak
      - postman
    image: quay.io/microcks/microcks:nightly
    container_name: microcks
    volumes:
      - "./config-logs:/deployments/config"
    ports:
      - "8080:8080"
      - "9090:9090"
    environment:
      - SPRING_PROFILES_ACTIVE=prod
      - SPRING_DATA_MONGODB_URI=mongodb://mongo:27017
      - SPRING_DATA_MONGODB_DATABASE=microcks
      - POSTMAN_RUNNER_URL=http://postman:3000
      - TEST_CALLBACK_URL=http://microcks:8080
      - SERVICES_UPDATE_INTERVAL=0 0 0/2 * * *
      - KEYCLOAK_URL=http://keycloak:8080
      - KEYCLOAK_PUBLIC_URL=http://localhost:18080
      - JAVA_OPTIONS=-Dspring.security.oauth2.resourceserver.jwt.issuer-uri=http://localhost:18080/realms/microcks -Dspring.security.oauth2.resourceserver.jwt.jwk-set-uri=http://keycloak:8080/realms/microcks/protocol/openid-connect/certs
      #- MAX_UPLOAD_FILE_SIZE=3MB

[yada]

Hi @der-equinix ,

Could you please copy/paste your logs (see below) in comments instead of screenshots to facilitate the conversation and analysis ;-)

I do not see any DEBUG logs from the AICopilotController!? You should see some logs from the docker-compose stdout, like:

12:02:09.519 DEBUG   --- [handler-42] i.g.microcks.web.AICopilotController     : Retrieving service with id 6788ef34b0259d3a04f75dab
12:02:09.537 DEBUG   --- [handler-42] i.g.microcks.web.AICopilotController     : We found service, now looking for required contract...
12:02:09.552 DEBUG   --- [handler-42] i.g.microcks.util.ai.OpenAICopilot       : Asking OpenAI to suggest samples for this prompt: Given the OpenAPI specification below, generate 2 full examples (request and response) for operation 'POST /check' strictly (no sub-path).

From the DEBUG logs above, you will see the prompt request and more details on what is going on, using your openAPI specification to help you understand and debug.

Here is an example of openAI errors log due to my account quota limits:

12:00:15.835 ERROR   --- [handler-41] i.g.microcks.web.AICopilotController     : Caught and exception while generating samples
microcks-camara           |
microcks-camara           | org.springframework.web.client.HttpClientErrorException$TooManyRequests: 429 Too Many Requests: "{<EOL>    "error": {<EOL>        "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",<EOL>        "type": "insufficient_quota",<EOL>        "param": null,<EOL>        "code": "insufficient_quota"<EOL>    }<EOL>}<EOL>"

BTW, could you please answer the previous questions as both can certainly help you to troubleshoot IMHO:

1. "Can you try using OpenAI first to simplify it before using Ollama and on-prem LLM?"
2. " You can check your Spec and get some input using our Spectral rules. See: https://github.com/microcks/microcks-spectral-ruleset"

[der-equinix]

Yes I am not getting DEBUG logs after doing the mentioned setting, this is what help I need first.

[yada]

I guess you did not add the log (debug) and AIcopilot config correctly.

Here is a config ex to help you; this is working fine for me (using openAI):
1/
cat config/logback.xml

Warning

The logback.xml indentation in the document may cause an issue (I will check first and reproduce but take care of the indentation of the XML file).

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true">
  <statusListener class="ch.qos.logback.core.status.NopStatusListener" />

  <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter" />
  <conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
  <conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />

  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <charset>utf-8</charset>
      <pattern>%clr(%d{HH:mm:ss.SSS}){faint} %clr(%5p) %clr( ){magenta} %clr(---){faint} %clr([%10.10t]){faint} %clr(%-40.40logger{36}){cyan} %clr(:){faint} %m%n%wEx</pattern>
    </encoder>
  </appender>

  <logger name="io.github.microcks" level="DEBUG"/>

  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
  </root>
</configuration>

2/
cat config/application.properties

Note

Logging debug at the end of the file, please note I use the config directory not config-log

# Async mocking support.
async-api.enabled=true

# AI Copilot configuration properties
ai-copilot.enabled=true
ai-copilot.implementation=openai
ai-copilot.openai.api-key=YOUR-KEY-HERE

# AI Copilot tuning by Yacine
ai-copilot.openai.model=gpt-4-turbo-preview
ai-copilot.openai.timeout=360
ai-copilot.openai.maxTokens=4096

# Access to Microcks API server.
%docker-compose.io.github.microcks.minion.async.client.MicrocksAPIConnector/mp-rest/url=http://microcks:8080

# Access to Keycloak through docker network
%docker-compose.keycloak.auth.url=http://keycloak:8080

# Access to Kafka broker.
%docker-compose.kafka.bootstrap.servers=kafka:19092

# Do not save any consumer-offset on the broker as there's a re-sync on each minion startup.
%docker-compose.mp.messaging.incoming.microcks-services-updates.enable.auto.commit=false
%docker-compose.mp.messaging.incoming.microcks-services-updates.bootstrap.servers=kafka:19092

# Explicitly telling the minion the protocols we want to support
%docker-compose.minion.supported-bindings=KAFKA,WS

# %docker-compose.minion.supported-bindings=KAFKA,WS,MQTT,AMQP,NATS

# %docker-compose.mqtt.server=localhost:1883
# %docker-compose.mqtt.username=microcks
# %docker-compose.mqtt.password=microcks

# %docker-compose.amqp.server=localhost:5672
# %docker-compose.amqp.username=microcks
# %docker-compose.amqp.password=microcks

# Logging configuration properties
logging.config=/deployments/config/logback.xml

%docker-compose.quarkus.log.level=DEBUG
%docker-compose.quarkus.log.console.level=DEBUG

3/
cat docker-compose-camara.yml

Note

I removed Async for my needs again please note I use the config directory, not config-log

services:
  mongo:
    image: mongo:4.4.29
    container_name: microcks-db
    volumes:
      - "~/tmp/microcks-data:/data/db"

  postman:
    image: quay.io/microcks/microcks-postman-runtime:0.6.0
    container_name: microcks-postman-runtime

  app:
    depends_on:
      - mongo
      - postman
    image: quay.io/microcks/microcks:1.11.0
    container_name: microcks-camara
    volumes:
      - "./config:/deployments/config"
    ports:
      - "8080:8080"
      - "9090:9090"
    environment:
      - SPRING_PROFILES_ACTIVE=prod
      - SPRING_DATA_MONGODB_URI=mongodb://mongo:27017
      - SPRING_DATA_MONGODB_DATABASE=microcks
      - POSTMAN_RUNNER_URL=http://postman:3000
      - TEST_CALLBACK_URL=http://microcks:8080
      - SERVICES_UPDATE_INTERVAL=0 0 0/2 * * *
      - KEYCLOAK_ENABLED=false
      #- MAX_UPLOAD_FILE_SIZE=3MB

Let us know if it helps...