[k8s] GPU Feature discovery label formatter

[asaiacai]

Resolves #2460

This allows k8s to consume the node label nvidia.com/gpu.product created by GPU feature discovery which is commonly deployed through the NVIDIA GPU operator

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual test: test against GKE labels (tested against T4)
• Manual test: test against skypilot labeler script labels on EKS deployed via eks_test_cluster.yaml
• Manual tests: deploy k3s with gpu-operator using deploy_k3s.sh modified to exclude the skypilot k8s labeler, ensure the following can run

# check nvidia-smi and nvidia.com/gpu.product info
nvidia-smi --query-gpu=name --format=csv,noheader,nounits
kubectl describe node | grep nvidia.com/gpu.product
# test skypilot against gpu type
sky show-gpus --cloud kubernetes
sky launch --cloud kubernetes --gpus <GPU_TYPE>

• A100-80GB
• A100
• H100
• T4
• V100
• A10G
• P100
• P4
• L4

[Michaelvll]

This is awesome @asaiacai! It looks very reasonable to me. @romilbhardwaj for another look to make sure it does not break our other formatters : )

[romilbhardwaj]

Thanks @asaiacai!

[romilbhardwaj]

Do we also need to implement get_label_value?

[asaiacai]

I don't think we need to. It's also tricky to create since the set of accelerator types can have one -> many mappings to GFD label values. One that I've run into is PCIE vs SXM variants of the A100 and H100 gpus. This is the main reason I changed the logic to check by going from (acc_type --> label value) to (label value --> acc_type). Outside of there, get_label_value is only used for autoscaling which GFDLabeler wouldn't be used.