Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[k8s] Identifying worker id from pods #3510

Open
romilbhardwaj opened this issue May 6, 2024 · 1 comment
Open

[k8s] Identifying worker id from pods #3510

romilbhardwaj opened this issue May 6, 2024 · 1 comment
Labels
k8s Kubernetes related items

Comments

@romilbhardwaj
Copy link
Collaborator

From user:

 there an easy way to tell which pod correspond to which worker? e.g. right now I have
NAME                                      READY   STATUS        RESTARTS      AGE
myclus-39f7-1e3c-worker        1/1     Running       0             10h
myclus-39f7-aaba-worker        1/1     Running       0             10h
myclus-39f7-b0f5-worker        1/1     Running       0             10h
myclus-39f7-d100-worker        1/1     Running       0             10h
myclus-39f7-d8ed-worker        1/1     Running       0             10h
myclus-39f7-d966-worker        1/1     Running       0             10h
myclus-39f7-head               1/1     Running       0             10h
I want to know which is worker 1 etc

We should look into naming the pod same as the worker id, and if that's not possible, patch a label on to the pod to identify worker id once the cluster is ready.

@romilbhardwaj romilbhardwaj added the k8s Kubernetes related items label May 6, 2024
@romilbhardwaj
Copy link
Collaborator Author

Current workaround - use sky exec to get a mapping of worker id to pod name:

sky exec clus --num-nodes n -- 'echo worker$SKYPILOT_NODE_RANK: $HOSTNAME'
...
(worker2, rank=2, pid=1040, ip=10.244.0.38) worker2: clus-2ea4-c095-worker
(head, rank=0, pid=2370) worker0: clus-2ea4-head
(worker1, rank=1, pid=994, ip=10.244.0.37) worker1: clus-2ea4-92c1-worker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
k8s Kubernetes related items
Projects
None yet
Development

No branches or pull requests

1 participant