Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cannot create mws workspaces: cannot create token: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API #759

Open
SabyasachiNayak opened this issue Jan 7, 2024 · 3 comments
Open

cannot create mws workspaces: cannot create token: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API #759

SabyasachiNayak opened this issue Jan 7, 2024 · 3 comments
Assignees
@mgyucht

Comments

@SabyasachiNayak
Copy link

SabyasachiNayak commented Jan 7, 2024 •
edited

Description
Not able to create Databricks workspace using terraform in GCP Databricks.

Reproduction

provider "databricks" {
  alias                   = "accounts"
  host                   = var.databricks_account_url
  account_id        = var.databricks_account_id
  google_service_account = var.google_service_account_email
}
provider "databricks" {
  alias                  = "workspace"
  host                  = databricks_mws_workspaces.db_workspace.workspace_url
  token                = databricks_mws_workspaces.db_workspace.token[0].token_value
}

resource "databricks_mws_workspaces" "db_workspace" {
  provider       = databricks.accounts
  account_id     = var.databricks_account_id
  workspace_name = var.workspace_name
  location       = var.google_region
 
  cloud_resource_container {
    gcp {
      project_id = var.google_project_name
    }
  }

  network_id = databricks_mws_networks.db_network.network_id
  private_access_settings_id = var.private_access_settings_id[var.google_region]
  
  gke_config {
    connectivity_type = "PRIVATE_NODE_PUBLIC_MASTER"
    master_ip_range   = "10.0.0.0/28" #dummy value
  }
 token{}
}

Expected behavior
DB PAT token should be created which can be used to create other entities in the workspace.

Debug Logs

 GET /login.html?error=private-link-validation-error:7023241863962948
│ > * Host: 
│ > * Accept: application/json
│ > * Authorization: REDACTED
│ > * Content-Type: application/json
│ > * Referer: https://7023241863962948.8.gcp.databricks.com/api/2.0/token/create
│ > * User-Agent: databricks-tf-provider/1.33.0 databricks-sdk-go/0.28.1 go/1.20.12 os/darwin terraform/1.6.6 resource/mws_workspaces auth/google-id
│ > {
│ >   "comment": "Terraform PAT",
│ >   "lifetime_seconds": 2592000
│ > }
│ < HTTP/2.0 200 OK
│ < * Cache-Control: no-cache, no-store, must-revalidate
│ < * Content-Security-Policy: default-src *; font-src * data:; frame-src * blob:; img-src * blob: data:; media-src * data:; object-src 'none'; style-src * 'unsafe-inline'; worker-src * blob:; script-src 'self' 'unsafe-eval' 'unsafe-hashes' 'report-sample' https://*.databricks.com https://databricks.github.io/debug-bookmarklet/ https://widget.intercom.io https://js.intercomcdn.com https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js https://databricks-ui-assets.azureedge.net https://ui-serving-cdn-testing.azureedge.net https://uiserviceprodwestus-cdn-endpoint.azureedge.net https://databricks-ui-infra.s3.us-west-2.amazonaws.com 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-+rWPVvhj3ywy9UdFC6+/89wsdWmA3rTCNgFaY2dro70=' 'sha256-2LEXJFgmmXOWz6lxNPAJ6NCOzQ2hV+e7m8HOos34Dc8=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-cnI3vvN7Fw9I4ceZbMNHX+pm+FtUn156ckGfpwLqdQQ='; report-uri /ui-csp-reports; frame-ancestors *.vocareum.com *.docebosaas.com *.edx.org *.deloitte.com *.cloudlabs.ai *.databricks.com
│ < * Content-Type: text/html; charset=utf-8
│ < * Date: Sun, 7 Jan 2024 23:44:50 GMT
│ < * Server: databricks
│ < * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
│ < * X-Content-Type-Options: nosniff
│ < * X-Ui-Svc: true
│ < * X-Xss-Protection: 1; mode=block
│ < <!doctype html>
│ < <html>
│ <  <head>
│ <   <meta charset="utf-8">
│ <   <meta http-equiv="Content-Language" content="en">
│ <   <title>Databricks - Sign In</title>
│ <   <meta name="viewport" content="width=960">
│ <   <link rel="icon" type="image/png" href="https://ui-assets.gcp.databricks.com/favicon.ico">
│ <   <meta http-equiv="content-type" content="text/html; charset=UTF8">
│ <   <script id="__databricks_react_script"></script>
│ <   <script>window.__DATABRICKS_SAFE_FLAGS__={"databricks.infra.showErrorModalOnFetchError":true,"databricks.fe.infra.useReact18":false},window.__DATABRICKS_CONFIG__={"publicPath":{"mlflow":"https://ui-assets.gcp.databricks.com/","dbsql":"https://ui-assets.gcp.databricks.com/","feature-store":"https://ui-assets.gcp.databricks.com/","monolith":"https://ui-assets.gcp.databricks.com/","jaws":"https://ui-assets.gcp.databricks.com/"}}</script>
│ <   <link rel="icon" href="https://ui-assets.gcp.databricks.com/favicon.ico">
│ <   <script>
│ <   function setNoCdnAndReload() {
│ <       const secIn7Days = 60 * 60 * 24 * 7;
│ <       document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=${secIn7Days}`;
│ <       const metric = 'cdnFallbackOccurred';
│ <       const browserUserAgent = navigator.userAgent;
│ <       const browserTabId = window.browserTabId;
│ <       const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]
│ <       sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));
│ <       window.location.reload();
│ <   }
│ < </script>
│ <   <script defer src="https://ui-assets.gcp.databricks.com/static/js/login/login.7b0da44f.js" onerror="setNoCdnAndReload()"></script>
│ <  </head>
│ <  <body class="light-mode">
│ <   <uses-legacy-bootstrap>
│ <    <div id="login-page"></div>
│ <   </uses-legacy-bootstrap>
│ <  </body>
│ < </html>
│ 
│ 
│   with databricks_mws_workspaces.db_workspace,
│   on workspace.tf line 20, in resource "databricks_mws_workspaces" "db_workspace":
│   20: resource "databricks_mws_workspaces" "db_workspace" {
│

Other Information

  • OS: macOS
  • Version: Ventura 13.6
  • Terraform version: 1.33.0
@SabyasachiNayak SabyasachiNayak changed the title cannot create token: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API cannot create mws workspaces: cannot create token: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API Jan 8, 2024
@mgyucht
Copy link
Contributor

mgyucht commented Jan 8, 2024

For some reason, the REST API is responding to this API call with a 301, redirecting you to the login page. I've raised a ticket with the team responsible for this API to investigate.

There is a private link validation error referenced in the redirect. Does it work if you create the workspace without private link at first?

@mgyucht mgyucht self-assigned this Jan 8, 2024
@SabyasachiNayak
Copy link
Author

I am testing this in GCP. So there should not be any private link error. We are using Private Service Connect(PSC) equivalent of private link in AWS.

@alexott
Copy link
Contributor

alexott commented Jan 11, 2024

It looks like traffic isn't going via PSC but via public internet. Check DNS & routing...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mgyucht mgyucht

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alexott @mgyucht @SabyasachiNayak