Expected behaviour with Consul Service Discovery and queryPassing=true

[liammacisaac]

When using a basic configuration to support consul service discovery, we're seeing some inconsistent behaviour when using the queryPassing setting. An example configuration is shown below.

spring:
    cloud:
        consul:
            config:
                failFast: false
            enabled: true
            host: localhost
            port: 8500
            discovery:
                register: false
                enabled: true
                registerHealthCheck: false
                instanceId: ${spring.application.name}:${random.value}
                query-passing: true       
        gateway:     
            discovery:
                locator:
                    enabled: true
                    predicates:
                    - name: Path
                      args:
                        pattern: "'/services/' + serviceId + '/**'"
                    filters:
                    -   name: RewritePath
                        args:
                            regexp: "'/services/' + serviceId + '/(?<remaining>.*)'"
                            replacement: "'/${remaining}'"

When queryPassing is false, we observe:

• Service starts and registers with consul - endpoints not yet started.
• Route is registered in gateway as expected.
• Service health check goes to passing.
• If multiple instances are started, the load balancer will occasionally choose one that is not passing healthchecks.

When queryPassing is true and the service is up before gateway, we observe:

• Gateway starts, detects new service and registers route.
• If multiple instances are running, load balancer will only choose ones with healthchecks passing.

When queryPassing is true and the service is started after gateway, we observe:

• Service starts and registers with consul - endpoints not yet started.
• Route is not registered with gateway because /actuator/health not yet up on the service.
• Service health check goes to passing.
• Service route returns 404.
• Another service starts and registers with consul, same thing happens, but first service now has its route registered because healthcheck is passing.

My initial thoughts on this are that we should be doing one of the following:

• Don't include queryPassing when establishing routes and only use it for the load balancer.
• Add some logic to the Consul catalog scan so that it will refresh routes when a service that was failing goes to passing. I can't see a way of doing this without making significantly more requests to Consul.

[spencergibb]

There used to be filters for load balancing with ribbon. We should add those back so health check can be only used for loadbalancing. https://github.com/spring-cloud/spring-cloud-consul/tree/2.2.x/spring-cloud-consul-discovery/src/main/java/org/springframework/cloud/consul/discovery/filters

See also #712

[nkvaratskhelia]

@spencergibb As explained above, the spring.cloud.consul.discovery.query-passing=true parameter works inconsistently. Is there currently any other way to exclude unhealthy services from gateway load balancing?