certificate verify failed error when stopping and restarting server #2396
Description
I have been experimenting with sccache-dist and have a set-up with one client, one scheduler, and two servers. Each is running on its on virtual machine within the same private network, therefore all requests are internally through the network. I have noticed that sometimes when stopping a server then restarting it, if I try to run a distributed build I observe the following error in the scheduler logs:
[2025-05-19T14:51:57Z WARN sccache::dist::http::server] Res 317 error: assign job failed, job un-assigned from the server, caused by: POST to scheduler assign_job failed, caused by: error sending request for url (https://172.16.0.5:10501/api/v1/distserver/assign_job/110), caused by: client error (Connect), caused by: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091: (self-signed certificate), caused by: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:
As far as I am aware, the two server instances are configured identically, except for their private IP addresses. I am using the following configuration files:
~/.config/sccache/config (Client)
[dist]
scheduler_url = "http://172.16.0.4:10600"
toolchains = []
toolchain_cache_size = 5368709120
[dist.auth]
type = "token"
token = "<CLIENT_TOKEN>"
scheduler.conf (Scheduler)
public_addr = "172.16.0.4:10600"
[client_auth]
type = "token"
token = "<CLIENT_TOKEN>"
[server_auth]
type = "token"
token = "<SERVER_TOKEN>"server.conf (Server A)
cache_dir = "/tmp/toolchains"
public_addr = "172.16.0.6:10501"
scheduler_url = "http://172.16.0.4:10600"
[builder]
type = "overlay"
build_dir = "/tmp/build"
bwrap_path = "/usr/bin/bwrap"
[scheduler_auth]
type = "token"
token = "<SERVER_TOKEN>"server.conf (Server B)
cache_dir = "/tmp/toolchains"
public_addr = "172.16.0.5:10501"
scheduler_url = "http://172.16.0.4:10600"
[builder]
type = "overlay"
build_dir = "/tmp/build"
bwrap_path = "/usr/bin/bwrap"
[scheduler_auth]
type = "token"
token = "<SERVER_TOKEN>"