AutoKuma not adding monitors with Service labels and Swarm

[nomad64]

I deployed AutoKuma to my Docker Swarm and it wasn't creating any monitors in Uptime Kuma. It wouldn't even add a test static monitor I had configured. After enabling trace logging, I was able to see that it was logging into Uptime Kuma successfully, but wasn't doing anything else. There are no errors in the log file.

After troubleshooting this for some time, I eventually tried removing the AUTOKUMA__DOCKER__SOURCE variable, and it started adding monitors via the container labels. Manually setting AUTOKUMA__DOCKER__SOURCE to Services or Both seems to break AutoKuma.

I also tried deploying it on a standalone Docker host. Setting AUTOKUMA__DOCKER__SOURCE to Services or Both causes similar behavior. Removing the variable, or setting it to 'Containers', allows AutoKuma to work correctly.

Note: This was done against a fresh install of Uptime Kuma v1 (initial testing with v2 showed similar results)

Stack YAML:

version: "3.7"
services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    environment:
      PUID: 1001
      PGID: 1001
      TZ: America/Chicago
      UPTIME_KUMA_CLOUDFLARED_TOKEN: "{{ app.uptime_kuma.cloudflared.token }}"
      UPTIME_KUMA_DB_TYPE: "mariadb"
      UPTIME_KUMA_DB_HOSTNAME: "{{ app.uptime_kuma.db.hostname }}"
      UPTIME_KUMA_DB_NAME: "{{ app.uptime_kuma.db.name }}"
      UPTIME_KUMA_DB_USERNAME: "{{ app.uptime_kuma.db.username }}"
      UPTIME_KUMA_DB_PASSWORD: "{{ app.uptime_kuma.db.password }}"
    volumes:
      - "/home/synohomes/docker/uptime-kuma/data:/app/data"
    ports:
      - "3001:3001"
    deploy:
      labels:
        shepard_autoupdate: "true"
    networks:
      plexreq_backend:
        aliases:
          - uptime-kuma
      socket_proxy_net:

  autokuma:
    image: ghcr.io/bigboot/autokuma:master
    depends_on:
      - uptime-kuma
    environment:
      AUTOKUMA__KUMA__URL: http://uptime-kuma:3001
      AUTOKUMA__KUMA__USERNAME: {{ app.autokuma.kuma.username }}
      AUTOKUMA__KUMA__PASSWORD: {{ app.autokuma.kuma.password }}
      AUTOKUMA__TAG_NAME: AutoKuma
      AUTOKUMA__SYNC_INTERVAL: "30.0"
      AUTOKUMA__STATIC_MONITORS: "/data/static"
      AUTOKUMA__ON_DELETE: delete
      AUTOKUMA__DOCKER__HOSTS: "tcp://socket-proxy-dokpv01:2375;tcp://socket-proxy-dokpv02:2375;tcp://socket-proxy-dokpv03:2375"
      AUTOKUMA__DOCKER__LABEL_PREFIX: kuma
      AUTOKUMA__DOCKER__SOURCE: Both
      # Enable the below for debugging purposes
      RUST_LOG: "kuma_client=trace"
      AUTOKUMA__LOG_DIR: "/data/logs"
    volumes:
      - "/home/synohomes/docker/autokuma/data:/data"
      - "/home/synohomes/docker/autokuma/config/autokuma.toml:/autokuma.toml:ro"
    deploy:
      placement:
        constraints:
          - "node.role==manager"
    networks:
      plexreq_backend:
        aliases:
          - autokuma
      socket_proxy_net:

networks:
  plexreq_backend:
    external: true
  socket_proxy_net:
    external: true

Autokuma docker log:

                                                        
                .:::.                                      .:::.                
              .===-====:                                :-===--==:              
             .==.    .:==-.        ..........         :==-.    .==:             
             -=-        :===--====================---==:        -==             
             -=-          :===-..              ..:===-          :==             
             -=-            ::                    .-.           -==             
             :==                                                ==-             
              ==.                                              .==.             
             :==-                                              -==-             
            .====.                                             ====-            
            ==-                                                  .==:           
           :==                                                    ===           
           -==                                                    -==           
           -==                                                    :==           
           -==               ..        ...       ..               -==           
           .==.             :===     -=====.    ====              ==-           
            ===              .:.   :==-  :==-    ::              :==.           
            .==:                  :==:    .==-                  .==:            
             .==.                :==:      .==-                .==-             
              .==:              .==:        .==:              .==:              
               .==-             ==-          :==             :==:               
                .-==:          :==            -=-          .-==.                
                  .===.        ==.   .::::..   ==.       .-==:                  
                    :===.     :=-  ==========. :==     .-==:                    
                      .===:   ==.  -=========  .==.  .-==:                      
                        .-==-:==    .======:    ==-:===:                        
                           :-===:      ...     .====:.                          
                              :==-.          .-==:                              
                                :====---:--====:                                
                                   .::----::.                                   
                            _           _  __                         
              /\           | |         | |/ /                         
             /  \    _   _ | |_   ___  | ' /  _   _  _ __ ___    __ _ 
            / /\ \  | | | || __| / _ \ |  <  | | | || '_ ` _ \  / _` |
           / ____ \ | |_| || |_ | (_) || . \ | |_| || | | | | || (_| |
          /_/    \_\ \__,_| \__| \___/ |_|\_\ \__,_||_| |_| |_| \__,_|  
                                                       v0.8.0-3236fb93
DEBUG [kuma_client::client] Waiting for connection
DEBUG [kuma_client::client] Connection opened!
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00")}]))
DEBUG [kuma_client::client] call login -> Text([Array [Object {"ok": Bool(true), "token": String("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imt1bWFhZG1pbiIsImgiOiJiZDgwZDdjYzhhOTI3ZTJlOGQ1ZTIxYTgzZTY0YjVlNyIsImlhdCI6MTczODUwNTQ2OH0.ANI0aaMGukUl6-XmGtmpSkdBk52rOoA8ZG1Z8jCs2Ow")}]])
DEBUG [kuma_client::client] Logged in as kumaadmin!
TRACE [kuma_client::client] Client::on_any(Custom("monitorList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("maintenanceList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"isContainer": Bool(true), "latestVersion": String("1.23.15"), "primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00"), "version": String("1.23.16")}]))
TRACE [kuma_client::client] Client::on_any(Custom("notificationList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("proxyList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("dockerHostList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("apiKeyList"), Text([Array []]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("statusPageList"), Text([Object {}]))
DEBUG [kuma_client::client] Connected!
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] Connection closed!
DEBUG [kuma_client::client] Waiting for connection
DEBUG [kuma_client::client] Connection opened!
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00")}]))
DEBUG [kuma_client::client] call login -> Text([Array [Object {"ok": Bool(true), "token": String("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imt1bWFhZG1pbiIsImgiOiJiZDgwZDdjYzhhOTI3ZTJlOGQ1ZTIxYTgzZTY0YjVlNyIsImlhdCI6MTczODUwNTUwMH0.WkrB14D-AEocIyXOVhNWyabVwB7Hp9pp5Vp4_oofxtU")}]])
DEBUG [kuma_client::client] Logged in as kumaadmin!
TRACE [kuma_client::client] Client::on_any(Custom("monitorList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("maintenanceList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"isContainer": Bool(true), "latestVersion": String("1.23.15"), "primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00"), "version": String("1.23.16")}]))
TRACE [kuma_client::client] Client::on_any(Custom("notificationList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("proxyList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("dockerHostList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("apiKeyList"), Text([Array []]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("statusPageList"), Text([Object {}]))
DEBUG [kuma_client::client] Connected!
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] Connection closed!
DEBUG [kuma_client::client] Waiting for connection
DEBUG [kuma_client::client] Connection opened!
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00")}]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] call login -> Text([Array [Object {"ok": Bool(true), "token": String("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imt1bWFhZG1pbiIsImgiOiJiZDgwZDdjYzhhOTI3ZTJlOGQ1ZTIxYTgzZTY0YjVlNyIsImlhdCI6MTczODUwNTUzMX0.dSaRwAnh_yHgnE6iSJEWkrX7OjExeRBfPOktxYu85zw")}]])
DEBUG [kuma_client::client] Logged in as kumaadmin!
TRACE [kuma_client::client] Client::on_any(Custom("monitorList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("maintenanceList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"isContainer": Bool(true), "latestVersion": String("1.23.15"), "primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00"), "version": String("1.23.16")}]))
TRACE [kuma_client::client] Client::on_any(Custom("notificationList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("proxyList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("dockerHostList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("apiKeyList"), Text([Array []]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("statusPageList"), Text([Object {}]))
DEBUG [kuma_client::client] Connected!
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] Connection closed!

The configured autokuma.toml file is empty.

Docker version:

Docker version 24.0.2, build cb74dfc

It appears that AutoKuma fails at some point when trying to read the Service labels for some unknown reason. At which point it won't add any monitors, whether configured via static files or container labels.

Could you please add some additional logging to help with troubleshooting issues like this? AutoKuma doesn't seem to log anything about what it was trying to do, what labels it is interpreting, whether it can connect to the Docker sockets, whether the snippet I put in there was understood (I eventually just removed it), etc. The only logs I ever saw were about its connection to Uptime Kuma which, while useful, weren't helpful during the hours of troubleshooting.

Please let me know if you need more info or would like me to test something, I would be happy to help!