SageMaker Hyperpod "Target not connected"

[sean-smith]

If you're trying to connect to your SageMaker Hyperpod cluster and you see an error "An error occurred (TargetNotConnected)", there's a couple of common causes:

An error occurred (TargetNotConnected) when calling the StartSession operation: sagemaker-cluster:..._controller-machine-i-... is not connected.
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

To troubleshoot do a few things:

1. Check your aws credentials are configured for the right account:

aws sts get-caller-identity --query Account --output text

2. Check to see the region is correct:

aws configure get region

If those don't work, try and ssm into a compute node, you'll need the cluster-id, worker-group name and instance-id which you can get from the aws sagemaker list-cluster-nodes --cluster-name <cluster-name> CLI call.

aws ssm start-session \
    --target sagemaker-cluster:<cluster-id>_worker-group-1-<instance-id>

Once you're there you can get the ip address of the controller node by running:

sudo cat /opt/ml/config/resource_config.json | jq | grep -5 controller-machine

That'll show:

      "Name": "controller-machine",
      "InstanceType": "ml.m5.12xlarge",
      "Instances": [
        {
          "InstanceName": "controller-machine-1",
          "AgentIpAddress": "172.16.90.220",
          "CustomerIpAddress": "10.1.39.83",
          "InstanceId": "i-0defeb24a1f5dfe85"
        }
      ]

Use the CustomerIpAddress 10.1.39.83 to SSH into headnode from that compute node:

ssh 10.1.39.83