dbx config should support job_tasks

[ericfeunekes]

Workflow tasks should support job_tasks to enable the latest behaviour in Databricks. Named properties will also need to support job://.

[stephen-bowser]

This would be very helpful for me too. Is this on the roadmap at all?

[jford-ctm]

This would be great if dbx supported the run_job_task type, and it should be able to perform the lookup by Job Name rather than supplying the job_id as per the API.

[nathnx]

This would be great if dbx supported the run_job_task type, and it should be able to perform the lookup by Job Name rather than supplying the job_id as per the API.

To add, this means the job_id value has to be initialised on the workflow prior to retrieval step as the id cannot be pre-defined. In other word, the child job needs to be deployed before the parent job.