Unexpected error handling request: inability to access platform

[sammigachuhi]

Description
Hello, I am trying to sync some work into my online databricks by running a .sh file. called source. So whenever I run the source sync.sh file, I get the below error:

Reproduction

Error: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

GET /api/2.0/preview/scim/v2/Me

• Host:
• Accept: application/json
• Authorization: REDACTED
• User-Agent: cli/0.212.0 databricks-sdk-go/0.29.0 go/1.21.0 os/linux cmd/current-user_me auth/pat
  < HTTP/2.0 403 Forbidden
  < * Content-Length: 70
  < * Content-Type: application/json; charset=utf-8
  < * Date: Wed, 13 Mar 2024 06:06:36 GMT
  < * Server: databricks
  < * Vary: Accept-Encoding
  < * X-Databricks-Reason-Phrase: Invalid access to Org: xxxxx
  < {
  < "error_code": 403,
  < "message": "Invalid access to Org: xxxxxx"
  < }

Error: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

POST /api/2.0/dbfs/mkdirs

• Host:
• Accept: application/json
• Authorization: REDACTED
• Content-Type: application/json
• User-Agent: cli/0.212.0 databricks-sdk-go/0.29.0 go/1.21.0 os/linux cmd/fs_cp auth/pat
  {
  "path": "/User"
  }
  < HTTP/2.0 403 Forbidden
  < * Access-Control-Allow-Headers: Authorization, X-Databricks-Azure-Workspace-Resource-Id, X-Databricks-Org-Id, Content-Type
  < * Access-Control-Allow-Origin: *
  < * Cache-Control: no-cache, no-store, must-revalidate
  < * Content-Length: 70
  < * Content-Type: application/json; charset=utf-8
  < * Date: Wed, 13 Mar 2024 06:06:37 GMT
  < * Expires: 0
  < * Pragma: no-cache
  < * Server: databricks
  < * Vary: Accept-Encoding
  < * X-Databricks-Reason-Phrase: Invalid access to Org: xxxxx
  < {
  < "error_code": 403,
  < "message": "Invalid access to Org: xxxxx"
  < }

I have my environment variables such as token number etc. in a file called .env.example which is super fine, I have crosschecked numerous times and it is all good. However, running source sync.sh no longer works.

Expected behavior
Running source sync.sh should start syncing some of my files to databricks online. This additional step is required outside the normal databricks extension.

[mgyucht]

Do you have an IP access list enabled for your workspace per chance?

[sammigachuhi]

Hello if you could tell me where to access it, I will look it up and get back to you. No idea if that IP access list exists and just heard about it now

[sammigachuhi]

As best as I can tell no IP Access restrictions have been made

[mgyucht]

See this doc page for an overview of IP access lists: https://docs.databricks.com/en/security/network/front-end/ip-access-list.html

[mgyucht]

There are two things going on here:

1. The API response includes a number for "error_code" where the SDK expects a string. This is causing a failure when deserializing the API response, resulting in this error message. We want to follow up with the backend team to make sure their error responses are consistent with other Databricks services.
2. Still, your request is being rejected due to some authorization problem. IP Access lists can be one cause of this. Generally, we don't include information about why your request was rejected in the error, as it would allow an illegitimate user to gain knowledge about your workspace/account.

[sammigachuhi]

Hello @mgyucht I checked if my workspace could have any allowed or blocked IP address lists through the code databricks ip-access-lists list as seen here . However, there are no IP Access lists for my account or workspace so the matter of IP address blocking is ruled out. This was the error I got in return after executing the aforementioned code: Error: default auth: metadata-service: token request: {"not_found":true}. Config: host=https:xxxxx.net, cluster_id=xxxxx, metadata_service_url=***. Env: DATABRICKS_HOST, DATABRICKS_CLUSTER_ID, DATABRICKS_METADATA_SERVICE_URL. Furthermore, when I ran the source sync.sh some days ago outside the office it worked perfectly. It was only two days ago this error came up after executing outside the office. Before it was all running fine, be it in or out of the office.

[mgyucht]

Ah interesting! Are you using the VS Code Extension? cc @fjakobs maybe there is a strange interaction going on here with metadata service.

Can you try running this in another terminal outside of VS Code?

[shadfdz]

I had the same issue @mgyucht, what worked was rerunning the command on a new terminal instance on VSCode. I am also using the dbx extension