Add data quality part for parking_sensor_synapse using great expectations library #641
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adding linkedService: sywsdev77-WorkspaceDefaultSqlServer Adding integrationRuntime: AutoResolveIntegrationRuntime Adding linkedService: sywsdev77-WorkspaceDefaultStorage Adding linkedService: Ls_KeyVault_01 Adding linkedService: Ls_AdlsGen2_01 Adding linkedService: Ls_Rest_MelParkSensors_01 Adding pipeline: P_Ingest_MelbParkingData Adding dataset: Ds_REST_MelbParkingData Adding dataset: Ds_AdlsGen2_MelbParkingData Adding notebook: 02_standardize Adding notebook: 03_transform Adding trigger: T_Sched Adding notebook: 00_setup Adding notebook: 01a_explore Adding notebook: 01b_explore_sqlserverless
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of PR
Purpose
0. Create mount point path for spark job
1. Configure DataContext
2. Create a BatchRequest based on dataframe
3. Define Expecation Suite and corresponding Data Expectations
4. Configure a checkpoint and run Expectation suite using checkpoint
Does this introduce a breaking change? If yes, details on what can break
No
Author pre-publish checklist
Validation steps
Issues Closed or Referenced