Kubernetes

[janosh]

has anyone used jobflow-remote with cloud computing (AWS, GCP, Azure, IBM Cloud, etc.). is jobflow-remote the right tool for that (also asking about future plans)? just curious if people have advice/experiences to share?

[gpetretto]

Hi, this is an interesting point to discuss.
At the moment we have only made some tests with Azure cyclecloud. In that case, once the cyclecloud cluster is started, the setup of jobflow-remote is similar to the one of a standard cluster. DFT codes can be compiled there and jobflow-remote can go through the standard steps of uploading input files and submitting the jobs to the queue. In this case I would say that the tricky part is setting up the cyclecloud cluster.
At the moment it would not be possible to let jobflow-remote autonomously start a container in a cloud environment and run the job there. It could be possible to implement the option to do this as well, but I think it would be good to work on a real test case. Do you have something more specific in mind?

[janosh]

thanks @gpetretto for the quick reply. at Radical, we've settled on what i think is a nice multi-cloud backend where multiple compute providers (AWS, GCP, ect.) all contribute nodes to the same slurm queue. that way, the underlying complexity that comes with multiple providers is abstracted away before it even reaches jf-remote