Add slurm-gke blueprint

[ACW101]

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[ACW101]

This PR include using the existing NFS server on the controller to distribute slurm key to nodeset running on GKE. If this is accepted, we can close #4562 as it's no longer needed.

[nick-stroud]

/gcbrun

[nick-stroud]

Only partial review. I will try to send more by 6pm PT.

[ACW101]

/gcbrun

[pawloch00]

I deployed the blueprint from this PR. After a few minutes, gke based nodes went to DOWN state. The slurmd log shows:

CPUs=8 Boards=1 Sockets=1 Cores=8 Threads=1 Memory=64309 TmpDisk=96515 Uptime=67206 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2025-09-25T08:43:49.012] error: Security violation, ping RPC from uid 981
[2025-09-25T08:43:49.012] error: Do you have SlurmUser configured as uid 981?

If we have nodes going down for such reason, than we need our own build of slurmd container with SlurmUser=401

[ACW101]

I deployed the blueprint from this PR. After a few minutes, gke based nodes went to DOWN state. The slurmd log shows:

CPUs=8 Boards=1 Sockets=1 Cores=8 Threads=1 Memory=64309 TmpDisk=96515 Uptime=67206 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2025-09-25T08:43:49.012] error: Security violation, ping RPC from uid 981
[2025-09-25T08:43:49.012] error: Do you have SlurmUser configured as uid 981?

If we have nodes going down for such reason, than we need our own build of slurmd container with SlurmUser=401

This was caused by uid mismatch in the public slinky image. A custom-built image with the correct UID is required to fix this. I will open a follow-up PR with instructions for building this image.

[pawloch00]

From time to time, deployment is failing, pods are stuck in init stage with message:

MountVolume.MountDevice failed for volume "slurm-key-pv" : rpc error: code = DeadlineExceeded desc = context deadline exceeded 
<br class="Apple-interchange-newline">

Also, sometimes ./gcluster deploy has to be run twice, since for the first time below error message appears:

for: "/tmp/608228131kubectl_manifest.yaml": error when patching "/tmp/608228131kubectl_manifest.yaml": PersistentVolume "slurm-key-pv" is invalid: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
  core.PersistentVolumeSource{

[ACW101]

/gcbrun

[ACW101]

/gcbrun