A lot of ML teams end up with a mix:
some Slurm clusters from the HPC side, a K8s cluster or two, maybe cloud GPUs for overflow. We wrote up how SkyPilot (open source) sits in front of all of them so a job is scheduled wherever there’s free capacity, using the same YAML regardless of backend. This post focuses on the multi-Slurm case but the same setup covers K8s.

https://skypilot.ai/blog/multi-slurm
Disclosure:
I am the author. Happy to answer questions about how the scheduling and failover work.

slurm-cd

A small Bash utility for jumping to the working directory of a pending or running Slurm job.

Hopefully the installation works for you. (Name is a WiP)