Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: dbx.api.adjuster.mixins.sql_properties.DashboardsList() argument after ** must be a mapping, not list #825

Open
lubrunn opened this issue Aug 4, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@lubrunn
Copy link

lubrunn commented Aug 4, 2023

Expected Behavior

When defining an Alert SQL task through the deployment.yml like this:

environments:
  default:
    workflows:
      - name: "alert-job"
          - task_key: "alert_task"
            sql_task:
              alert:
                alert_id: "alert://alert1"
              warehouse_id: "warehouse://sql-warehouse"

it should automatically determine the alert_id through the alert name like it does for warehouses and queries etc.

Current Behavior

Currently I get the following error in my azure pipeline:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/commands/deploy.py:154 in  │
│ deploy                                                                       │
│                                                                              │
│   151 │   │   if pipelines:                                                  │
│   152 │   │   │   dbx_echo("Found DLT pipelines definition, applying them fi │
│   153 │   │   │   for elements in [pipelines, workflows]:                    │
│ ❱ 154 │   │   │   │   adjuster.traverse(elements)                            │
│   155 │   │   │   │   wf_manager = WorkflowDeploymentManager(api_client, ele │
│   156 │   │   │   │   wf_manager.apply()                                     │
│   157 │   │   else:                                                          │
│                                                                              │
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/api/adjuster/adjuster.py:1 │
│ 87 in traverse                                                               │
│                                                                              │
│   184 │   │   dbx_echo("Starting the traversal process")                     │
│   185 │   │   self.property_adjuster.library_traverse(workflows, self.additi │
│   186 │   │   self.property_adjuster.file_traverse(workflows, self.file_adju │
│ ❱ 187 │   │   self.property_adjuster.property_traverse(workflows)            │
│   188 │   │   self.property_adjuster.cluster_policy_traverse(workflows)      │
│   189 │   │   dbx_echo("Traversal process finished, all provided references  │
│   190                                                                        │
│                                                                              │
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/api/adjuster/adjuster.py:1 │
│ 45 in property_traverse                                                      │
│                                                                              │
│   142 │   │   │   │   │   self._adjust_dashboard_ref(element, parent, index) │
│   143 │   │   │   │                                                          │
│   144 │   │   │   │   elif element.startswith("alert://"):                   │
│ ❱ 145 │   │   │   │   │   self._adjust_alert_ref(element, parent, index)     │
│   146 │                                                                      │
│   147 │   def cluster_policy_traverse(self, workflows: WorkflowList):        │
│   148 │   │   """                                                            │
│                                                                              │
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/api/adjuster/mixins/sql_pr │
│ operties.py:116 in _adjust_alert_ref                                         │
│                                                                              │
│   113 │                                                                      │
│   114 │   def _adjust_alert_ref(self, element: str, parent: Any, index: Any) │
│   115 │   │   alert_name = element.replace("alert://", "")                   │
│ ❱ 116 │   │   _relevant = DashboardsList(                                    │
│   117 │   │   │   **self.api_client.perform_query("GET", path="/preview/sql/ │
│   118 │   │   )                                                              │
│   119 │   │   _id = _relevant.get(alert_name).id                             │

TypeError: dbx.api.adjuster.mixins.sql_properties.DashboardsList() argument 
after ** must be a mapping, not list

Steps to Reproduce (for bugs)

  1. Create deployment.yml with the following:
environments:
  default:
    workflows:
      - name: "alert-job"
          - task_key: "alert_task"
            sql_task:
              alert:
                alert_id: "alert://alert1"
              warehouse_id: "warehouse://sql-warehouse"
  1. Deploy.

Context

Your Environment

  • dbx version used: 0.8.18

On a first glance I think the reason for the error might be that in the _adjust_alert_refmethod of the SqlPropertiesAdjusterclass (line 116) in dbx/api/adjuster/mixins/sql_properties.py it should reference the class AlertsList instead of DashboardsList:

.
.
.

class DashboardsList(FlexibleModel, ResultsListGetterMixin):
    results: List[DashboardInfo] = []

    @property
    def object_type(self) -> str:
        return "dashboard"


class AlertsList(FlexibleModel, ResultsListGetterMixin):
    results: List[AlertInfo] = []

    @property
    def object_type(self) -> str:
        return "alert"

class SqlPropertiesAdjuster(ApiClientMixin, ElementSetterMixin):
    # TODO: design of this class is a terrible copy-paste. It must be rewritten.

    @functools.cached_property
    def _warehouses(self) -> WarehousesList:
        return WarehousesList(**self.api_client.perform_query("GET", path="/sql/warehouses/"))

    def _adjust_warehouse_ref(self, element: str, parent: Any, index: Any):
        _id = self._warehouses.get(element.replace("warehouse://", "")).id
        self.set_element_at_parent(_id, parent, index)

    def _adjust_query_ref(self, element: str, parent: Any, index: Any):
        query_name = element.replace("query://", "")
        _relevant = QueriesList(
            **self.api_client.perform_query("GET", path="/preview/sql/queries", data={"q": query_name})
        )
        _id = _relevant.get(query_name).id
        self.set_element_at_parent(_id, parent, index)

    def _adjust_dashboard_ref(self, element: str, parent: Any, index: Any):
        dashboard_name = element.replace("dashboard://", "")
        _relevant = DashboardsList(
            **self.api_client.perform_query("GET", path="/preview/sql/dashboards", data={"q": dashboard_name})
        )
        _id = _relevant.get(dashboard_name).id
        self.set_element_at_parent(_id, parent, index)

    def _adjust_alert_ref(self, element: str, parent: Any, index: Any):
        alert_name = element.replace("alert://", "")
        _relevant = AlertsList(
            **self.api_client.perform_query("GET", path="/preview/sql/alerts", data={"q": alert_name})
        )
        _id = _relevant.get(alert_name).id
        self.set_element_at_parent(_id, parent, index)

However, I haven't confirmed this.
(When I provide the explicit alert_id the deployment goes through without any problems btw.)

@renardeinside renardeinside added the bug Something isn't working label Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants