New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add deferrable support to DatabricksNotebookOperator
#39295
base: main
Are you sure you want to change the base?
Conversation
…db-notebook-deferrable-support Signed-off-by: kalyanr <kalyan.ben10@live.com>
…db-notebook-deferrable-support
…db-notebook-deferrable-support
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
implementation looks good, but it would be better if we can add some test
…db-notebook-deferrable-support
…db-notebook-deferrable-support
"Task failed. Final state %s. Reason: %s", | ||
run_state.result_state, | ||
run_state.state_message, | ||
f"Task failed. Final state {run_state.result_state}. Reason: {run_state.state_message}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pankajkoti , I updated this exception call. The reason is that AirflowException
just extend the Base Exception class and all the extra arguments we send are not actually used to parse the final exception message.
example:
>>> from airflow.exceptions import AirflowException
>>> raise AirflowException("This is %s", "Hello")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
airflow.exceptions.AirflowException: ('This is %s', 'Hello')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One nitpick. Everything else looks good to me
) | ||
operator.databricks_run_id = 12345 | ||
|
||
with pytest.raises(TaskDeferred): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
assert isinstance(exc.value.trigger, HttpTrigger), "Trigger is not a HttpTrigger" |
let's extend the test like this one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added it. Also added one more assert to check the method
…db-notebook-deferrable-support
related: #39178
This PR intends to make
DatabricksNotebookOperator
deferrable