`certificate verify failed` error when stopping and restarting server

[cecheta]

I have been experimenting with sccache-dist and have a set-up with one client, one scheduler, and two servers. Each is running on its on virtual machine within the same private network, therefore all requests are internally through the network. I have noticed that sometimes when stopping a server then restarting it, if I try to run a distributed build I observe the following error in the scheduler logs:

[2025-05-19T14:51:57Z WARN  sccache::dist::http::server] Res 317 error: assign job failed, job un-assigned from the server, caused by: POST to scheduler assign_job failed, caused by: error sending request for url (https://172.16.0.5:10501/api/v1/distserver/assign_job/110), caused by: client error (Connect), caused by: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091: (self-signed certificate), caused by: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:

As far as I am aware, the two server instances are configured identically, except for their private IP addresses. I am using the following configuration files:

~/.config/sccache/config (Client)

[dist]
scheduler_url = "http://172.16.0.4:10600"
toolchains = []
toolchain_cache_size = 5368709120

[dist.auth]
type = "token"
token = "<CLIENT_TOKEN>"

scheduler.conf (Scheduler)

public_addr = "172.16.0.4:10600"

[client_auth]
type = "token"
token = "<CLIENT_TOKEN>"

[server_auth]
type = "token"
token = "<SERVER_TOKEN>"

server.conf (Server A)

cache_dir = "/tmp/toolchains"
public_addr = "172.16.0.6:10501"
scheduler_url = "http://172.16.0.4:10600"

[builder]
type = "overlay"
build_dir = "/tmp/build"
bwrap_path = "/usr/bin/bwrap"

[scheduler_auth]
type = "token"
token = "<SERVER_TOKEN>"

server.conf (Server B)

cache_dir = "/tmp/toolchains"
public_addr = "172.16.0.5:10501"
scheduler_url = "http://172.16.0.4:10600"

[builder]
type = "overlay"
build_dir = "/tmp/build"
bwrap_path = "/usr/bin/bwrap"

[scheduler_auth]
type = "token"
token = "<SERVER_TOKEN>"

[cecheta]

Seems like it's caused by seanmonstar/reqwest#1260

[cecheta]

A workaround is to restart the server on a different port