Use bootstrap tokens provisioned by ansible for K3s instead of persistent tokens in cloud-init metadata

[wtripp180901]

Server now generates a short lived bootstrap token which nodes use for initial connection/auth. Performed in site.yml rather than ansible-init. Tokens no longer needs to be generated by ansible, but consistent node passwords are needed (previously we were using the ansible-generated token for this) so added to the secret store.

[wtripp180901]

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/13409545021

[sd109]

@wtripp180901 Do we also need to update this table in the compute-init README as a result of this change?

Do we have a plan yet for how to get nodes to rejoin the k3s cluster after a compute-init driven rebuild? Is it even possible with this bootstrap token approach?

[sjpb]

@sd109

Do we have a plan yet for how to get nodes to rejoin the k3s cluster after a compute-init driven rebuild? Is it even possible with this bootstrap token approach?

Yeah I think so, we discussed it the other day. The RebootProgram will have to generate a new token, that will be distributed by 🪄 (i.e. however we distribute other secrets), then the node picks it up on boot.

A failure to provision a new token which results in RebootProgram failing and node being marked down (unless we do something to handle that) so I think that will work.

[wtripp180901]

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/13453455728