9

So we have Java microservices written with Spring-Boot, using Consul for service discovery and config management and running in Docker containers. All of it is working, but when a container dies or a service restarts the old service-id never goes away in Consul and the service forever after shows as "Failing" in the Consul UI, even though the new container has registered and shows all Green.

We are not using heartbeat - but I cannot find much documentation on what the difference between heartbeat and healthcheck are for Consul.

Here's my bootstrp.yml

spring:
  application:
    name: my-service
  cloud:
    config:
      enabled: false
    consul:
      host: ${discovery.host:localhost}
      port: ${discovery.port:8500}
      config:
        watch:
          wait-time: 30
          delay: 10000 
        profile-separator: "-"
        format: FILES
      discovery:
        prefer-ip-address: true
        instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}

There are other settings to enable heartbeat, but the docs say something about this putting more stress on the Consul cluster.

Has anyone managed to get Consul and Spring Boot/Docker services to actually de-register automatically? It actually doesn't cause any real problems, but it makes the Consul UI pretty useless to actually monitor for up/down services.

Gandalf
  • 9,648
  • 8
  • 53
  • 88
  • Maybe this one helps: http://stackoverflow.com/questions/32259323/consul-not-deregistering-zombie-services – gesellix Apr 19 '17 at 18:14

2 Answers2

1

Consul doesn't automatically deregister services.

See https://groups.google.com/forum/#!topic/consul-tool/slV5xfWRpEE for the hint about the same question. According to that thread you need to either update the config or perform an Agent API call. Since the agent is the source of truth, you shouldn't try to update via Catalog API. See GitHub for details. They also mention at the Google group that you don't necessarily have to deregister services if the node goes down gracefully, but that doesn't seem to be your use case.

Please have a look at Consul not deregistering zombie services for hints about automating the service de-registration using either the api or tools like registrator.

Community
  • 1
  • 1
gesellix
  • 3,024
  • 28
  • 31
1

You have mentioned you are using a docker container to run the microservice. Are you trapping the SIGTERM in your entrypoint script in docker container ? If a SIGTERM is sent, the boot application will get it and you will see the below log showing that the microservice is deregistering with Consul.

2017-04-27 09:20:19.854  INFO 6852 --- [on(6)-127.0.0.1] inMXBeanRegistrar$SpringApplicationAdmin : Application shutdown requested.
2017-04-27 09:20:19.857  INFO 6852 --- [on(6)-127.0.0.1] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@afb5821: startup date [Thu Apr 27 09:20:00 EDT 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@130c12b7
2017-04-27 09:20:19.859  INFO 6852 --- [on(6)-127.0.0.1] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 2147483647
2017-04-27 09:20:19.863  INFO 6852 --- [on(6)-127.0.0.1] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2017-04-27 09:20:19.863  INFO 6852 --- [on(6)-127.0.0.1] o.s.c.c.s.ConsulServiceRegistry          : Deregistering service with consul: xxxxxxxxxxxxx

This blog post discusses this.

Tito
  • 8,894
  • 12
  • 52
  • 86
  • Yup, I was thinking along those lines as well, but in my logs I do see the "Deregistering with Consul...." INFO message. – Gandalf Apr 28 '17 at 16:27
  • @Gandalf Are you using this form in your Dockerfile ? { CMD exec java -jar microservice.jar } In my consul/docker/spring-boot setup, the services are properly deregistered. The difference is, I don't have a consul agent. My micro service directly talks to the consul server which is in bootstrap mode. – Tito May 03 '17 at 01:59