Versatile Data Kit 0.14

[antoniivanov]

Major features include:

VDK DAG plugin release

VDK DAG (previously vdk-meta-jobs) is the official name of the plugin allowing users to express dependencies between data jobs and is released as Beta with more stability and usability and documentation improvements.

Check out for more in the plugin page.

Versatile Data Kit UI Shareable Web links

Now users can share links with filters applied:

• Data Jobs list (Manage and Explore screen) are shareable through URL, as every applied filter is persisted to URL and vice-versa
• Data Job Executions screen filters and sort parameters are shareable through URL, as every applied filter or sort is persisted to URL and vice-versa

VDK UI configuration improvements and easy to get started by using quickstart-vdk

Users can now access VDK UI using quickstart-vdk. VDK UI is made to be much more configurable:

• Toggleable authentication (default: enabled) using the 'skipAuth' flag.
• Configuration of authentication parameters.
• Ability to specify visual elements displayed, e.g., navigation button to the Explore page.

VDK Control CLI supports python version

People now can specify the python version they need their job to run when deployed in VDK Control Service runtime:

vdk deploy --python-version 3.7 ..

Or in job config.ini

[job]
python_version = 3.7

Users can also see which version of python is VDK Control Service supporting currently:

vdk info

What's Changed

• control-service: Clean up old data job configurations by @doks5 in control-service: Clean up old data job configurations #2075
• control-service: Fix backwards-compatibility issues by @doks5 in control-service: Fix backwards-compatibility issues #2022
• control-service: Only CLI executions are "Manual" by @gageorgiev in control-service: Only CLI executions are "Manual" #1763
• control-service: Rework supported python version logic by @doks5 in control-service: Rework supported python version logic #1992
• control-service: Swagger UI quickstart-vdk server config by @ivakoleva in control-service: Swagger UI quickstart-vdk server config #2062
• control-service: [Bug fix] Fix supported python versions helm configuration by @doks5 in control-service: [Bug fix] Fix supported python versions helm configuration #1964
• control-service: a clear error message on how to handle the failed pipeline by @murphp15 in control-service: a clear error message on how to handle the failed pipeline #2127
• control-service: add ability to check if docker image exists in ecr by @mrMoZ1 in control-service: add ability to check if docker image exists in ecr #1977
• control-service: allow more time to reach a complete state by @murphp15 in control-service: allow more time to reach a complete state #2143
• control-service: append integration test name to job name by @mivanov1988 in control-service: append integration test name to job name #2093
• control-service: better error logging and pull private image in private test by @murphp15 in control-service: better error logging and pull private image in private test #2156
• control-service: better error message by @murphp15 in control-service: better error message #2094
• control-service: better error message from throwable by @murphp15 in control-service: better error message from throwable #2157
• control-service: clarify build steps by @dakodakov in control-service: clarify build steps #1959
• control-service: code expected to run in transaction now runs in transaction by @murphp15 in control-service: code expected to run in transaction now runs in transaction #2117
• control-service: custom app config values can we passed to helm. by @murphp15 in control-service: custom app config values can we passed to helm. #2004
• control-service: delete unused method by @murphp15 in control-service: delete unused method #2038
• control-service: disable authorization on test/cicd deployment by @tozka in control-service: disable authorization on test/cicd deployment #2129
• control-service: disable failing test by @murphp15 in control-service: disable failing test #2086
• control-service: fail tests fast by @murphp15 in control-service: fail tests fast #2137
• control-service: fix api declaration by @murphp15 in control-service: fix api declaration #1974
• control-service: fix oom tests by @murphp15 in control-service: fix oom tests #2028
• control-service: handle null started by value by @murphp15 in control-service: handle null started by value #2151
• control-service: if a test is in a bad state it fails straight away by @murphp15 in control-service: if a test is in a bad state it fails straight away #2098
• control-service: include details in error message by @murphp15 in control-service: include details in error message #2122
• control-service: increase CICD deployment resources by @tozka in control-service: increase CICD deployment resources #2130
• control-service: killed job was shown as successful by @mivanov1988 in control-service: killed job was shown as successful #2116
• control-service: latest version of gradle and spring /remove old comment by @murphp15 in control-service: latest version of gradle and spring /remove old comment #1976
• control-service: logs url can include team name by @murphp15 in control-service: logs url can include team name #2013
• control-service: new python client. by @murphp15 in control-service: new python client. #1983
• control-service: print response body on error by @murphp15 in control-service: print response body on error #2113
• control-service: remove a test that is testing behaviour that doesn't exist by @murphp15 in control-service: remove a test that is testing behaviour that doesn't exist #2031
• control-service: remove unused parameter by @murphp15 in control-service: remove unused parameter #2027
• control-service: remove unused parameters by @murphp15 in control-service: remove unused parameters #2016
• control-service: see more details when there is an error by @murphp15 in control-service: see more details when there is an error #2050
• control-service: update ecr credentials integration test by @mrMoZ1 in control-service: update ecr credentials integration test #2079
• control-service: upgrade python client by @murphp15 in control-service: upgrade python client #2076
• control-service: use git for images by @murphp15 in control-service: use git for images #2097
• docs: add getting started section for quickstart-vdk and ui by @DeltaMichael in docs: add getting started section for quickstart-vdk and ui #2019
• frontend: Bugfix in e2e plugins function and bump major versions for UI libs by @gorankokin in frontend: Bugfix in e2e plugins function and bump major versions for UI libs #1994
• frontend: Fix for e2e tests by @gorankokin in frontend: Fix for e2e tests #2030
• frontend: Implement executions list enhacements by @gorankokin in frontend: Implement executions list enhacements #2126
• frontend: Improve visibility of 'User error' messages by @hzhristova in frontend: Improve visibility of 'User error' messages #1960
• frontend: Job sharable executions filter and sort by @gorankokin in frontend: Job sharable executions filter and sort #2072
• frontend: Toggleable auth by @ivakoleva in frontend: Toggleable auth #1958
• frontend: Upgrade lineage to beta version by @hzhristova in frontend: Upgrade lineage to beta version #1991
• frontend: shareable links with query params for Data Jobs grids by @hzhristova in frontend: shareable links with query params for Data Jobs grids #2049
• frontend: visibility of app components is configurable by @DeltaMichael in frontend: visibility of app components is configurable #1978
• quickstart-vdk: ignore explore page and widgets in frontend by @DeltaMichael in quickstart-vdk: ignore explore page and widgets in frontend #2073
• vdk-airflow: fix failing tests by @murphp15 in vdk-airflow: fix failing tests #2078
• vdk-control-cli: Add support for python_version by @doks5 in vdk-control-cli: Add support for python_version #2002
• vdk-control-cli: Add support for python_version in config by @doks5 in vdk-control-cli: Add support for python_version in config #2023
• vdk-control-cli: add vdk info command to list of cli commands by @dakodakov in vdk-control-cli: add vdk info command to list of cli commands #2069
• vdk-control-cli: import the latest version of the client into cli by @murphp15 in vdk-control-cli: import the latest version of the client into cli #1969
• vdk-control-cli: upgrade python client by @murphp15 in vdk-control-cli: upgrade python client #2077
• vdk-control-cli: use explicit parameter names by @murphp15 in vdk-control-cli: use explicit parameter names #1975
• vdk-dag: DAGs propagate their execution type to their component jobs by @yonitoo in vdk-dag: DAGs propagate their execution type to their component jobs #2080
• vdk-dag: Drop deprecation warnings by @gageorgiev in vdk-dag: Drop deprecation warnings #2012
• vdk-dag: Fix config bug by @gageorgiev in vdk-dag: Fix config bug #2029
• vdk-dag: Rename vdk-meta-jobs to vdk-dag by @gageorgiev in vdk-dag: Rename vdk-meta-jobs to vdk-dag #1831
• vdk-dag: fix plugin name of DAGs example README.md by @yonitoo in vdk-dag: fix plugin name of DAGs example README.md #1945
• vdk-dag: improve DAGs docs and example by @yonitoo in vdk-dag: improve DAGs docs and example #1984
• vdk-dag: update VEP about the execution type propagation by @yonitoo in vdk-dag: update VEP about the execution type propagation #2095
• vdk-examples: Change Meta Jobs to DAGs in examples by @gageorgiev in vdk-examples: Change Meta Jobs to DAGs in examples #2024
• vdk-gdp-execution-id: example added by @ivakoleva in vdk-gdp-execution-id: example added #1962
• vdk-heartbeat: add test that pings the frontend url by @DeltaMichael in heartbeat: add test that pings the frontend url #2039
• vdk-heartbeat: fix python3.7 issue by @DeltaMichael in vdk-heartbeat: fix python3.7 issue #2104
• vdk-impala: add udf hive2server user error classification by @mrMoZ1 in vdk-impala: add udf hive2server user error classification #2056
• vdk-jupyter: Drop faulty statement by @gageorgiev in vdk-jupyter: Drop faulty statement #2160
• vdk-jupyter: Pick up REST API URL from the env by @gageorgiev in vdk-jupyter: Pick up REST API URL from the env #2142
• vdk-jupyter: UI failing cell marks by @duyguHsnHsn in vdk-jupyter: UI failing cell marks #2057
• vdk-jupyter: UI failing cell marks enhancement by @duyguHsnHsn in vdk-jupyter: UI failing cell marks enhancement #2136
• vdk-jupyter: add separator to the VDK menu by @duyguHsnHsn in vdk-jupyter: add separator to the VDK menu #2114
• vdk-jupyter: fix broken server in one use case by @duyguHsnHsn in vdk-jupyter: fix broken server in one use case #2170
• vdk-jupyter: pin lumino/widgets package to version 1 by @duyguHsnHsn in vdk-jupyter: pin lumino/widgets package to version 1 #2099
• vdk-jupyter: pin npm packages to previous version by @duyguHsnHsn in vdk-jupyter: pin npm packages to previous version #2088
• vdk-meta-jobs: Undelete the package for potential future fixes by @gageorgiev in vdk-meta-jobs: Undelete the package for potential future fixes #2106
• vdk-server: logs url in quickstart vdk by @murphp15 in vdk-server: logs url in quickstart vdk #2018
• vdk-server: trigger package release by @murphp15 in vdk-server: trigger package release #2058
• versatile-data-kit: Update CONTRIBUTING.md by @tozka in versatile-data-kit: Update CONTRIBUTING.md #1988

New Contributors

• @hzhristova made their first contribution in frontend: Improve visibility of 'User error' messages #1960

Full Changelog: v0.13...v0.14

---

This discussion was created from the release Versatile Data Kit 0.14.