[Dataset Quality]Add logic to display degraded fields in DQ Flyout #183934

achyutjhunjhunwala · 2024-05-21T15:16:27Z

Summary

Description

This PR adds the Degraded Fields Table to the Dataset Quality Flyout for individual Data Stream. Following tasks were done as part of this PR

A new server side endpoint created which queries the datastream directly and aggregates _ignored fields for that data stream during the given time range and also adds a sub aggregation for last occurence.
On the UI Side, the table was added with 3 columns as mentioned in the Original ticket - Field, Count and Last Occurrence
The UI currently supports clients side sorting and pagination. We can move this to server side pagination sorting if required in the future.
The Flyout also supports sync with the URL which means user can navigate to the Dataset Quality page where the flyout would be open and Sorting and Pagination would be pre-applied
API Tests
Stateful and Serverless FTR tests

Screenshot

Scenario

Spin up a 8.14-snapshot instance
Ingest degraded docs
Upgrade to 8.15-snapshot
Open Data Set Quality Flyout and see how the page looks like

apmmachine · 2024-05-21T15:16:39Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…hareable URLs

achyutjhunjhunwala · 2024-05-24T13:08:37Z

/ci

elasticmachine · 2024-05-24T13:14:59Z

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

x-pack/plugins/observability_solution/observability/server/utils/queries.ts

elasticmachine · 2024-05-24T13:54:15Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

tonyghiani

I did a first pass and left some notes, I think the new API is not versioned and it requires to update the implementation to comply with the specs, good job overall 👌

x-pack/plugins/observability_solution/dataset_quality/common/api_types.ts

.../observability_solution/dataset_quality/public/components/flyout/degraded_fields/columns.tsx

...bility_solution/dataset_quality/public/components/flyout/degraded_fields/degraded_fields.tsx

...ns/observability_solution/dataset_quality/public/hooks/use_dataset_quality_degraded_field.ts

...lution/dataset_quality/public/state_machines/dataset_quality_controller/src/state_machine.ts

x-pack/plugins/observability_solution/dataset_quality/server/routes/data_streams/routes.ts

flash1293 · 2024-05-28T12:25:36Z

It seems like the "last occurrence" is not formatted properly:

achyutjhunjhunwala · 2024-05-29T08:07:19Z

@flash1293 Thank you for pointing it out. I have fixed it now

flash1293 · 2024-05-29T08:15:14Z

Thanks @achyutjhunjhunwala ! One other note about the user experience - should we indicate somehow if the list of ignored fields is not exhaustive?

In the simplest way it would just be a mention in the tooltip: https://github.com/elastic/kibana/pull/183934/files#diff-e14f87ebe930d154e4d8be9da8d99811c8a756fca476f339edaaa62009c5c257R147

achyutjhunjhunwala · 2024-05-29T09:00:39Z

Thanks @achyutjhunjhunwala ! One other note about the user experience - should we indicate somehow if the list of ignored fields is not exhaustive?

In the simplest way it would just be a mention in the tooltip: https://github.com/elastic/kibana/pull/183934/files#diff-e14f87ebe930d154e4d8be9da8d99811c8a756fca476f339edaaa62009c5c257R147

@flash1293 Sounds good to me. I will proceed with something like this "Set of degraded fields in the dataset. Please not that this list may not be exhaustive."

@mdbirnstiehl Can you help me here with a better copy ?

tonyghiani

LGTM, thanks for addressing the changes!

kibana-ci · 2024-05-29T11:47:55Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 6f06c20
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183934-6f06c20a1d51
Observability Deployment

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`datasetQuality`	187	193	+6

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`observability`	689	691	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`datasetQuality`	179.4KB	183.5KB	+4.1KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
`module count`	-	5412	+5412
`total size`	-	8.8MB	+8.8MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`datasetQuality`	36.7KB	38.1KB	+1.4KB
`observabilityLogsExplorer`	16.3KB	16.3KB	+49.0B
total			+1.5KB

Unknown metric groups

API count

id	before	after	diff
`observability`	698	700	+2

History

💚 Build #212768 succeeded 10abcbc
💛 Build #212597 was flaky e24a79c
💛 Build #212504 was flaky 75113c7
💔 Build #212428 failed 803f067

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @achyutjhunjhunwala

mdbirnstiehl · 2024-05-29T13:55:21Z

x-pack/plugins/observability_solution/dataset_quality/common/translations.ts

+  'xpack.datasetQuality.flyoutDegradedFieldsSectionTooltip',
+  {
+    defaultMessage:
+      'Set of degraded fields in the dataset. Please not that this list may not be exhaustive.',


Looks like there's a typo here. "Please note..." instead of "Please not..." You can probably remove that part anyway and just say "This list may not be exhaustive." We could also change the tooltip a little more and say, "A partial list of degraded fields found in your dataset." to imply that it's not exhaustive.

Thank @mdbirnstiehl I will go with "A partial list of degraded fields found in your dataset." and update the tooltip in my next PR

…lastic#183934) ## Summary Closes elastic#172265 ## Description This PR adds the Degraded Fields Table to the Dataset Quality Flyout for individual Data Stream. Following tasks were done as part of this PR 1. A new server side endpoint created which queries the `datastream` directly and aggregates `_ignored` fields for that data stream during the given time range and also adds a sub aggregation for last occurence. 2. On the UI Side, the table was added with 3 columns as mentioned in the Original ticket - Field, Count and Last Occurrence 3. The UI currently supports clients side sorting and pagination. We can move this to server side pagination sorting if required in the future. 4. The Flyout also supports sync with the URL which means user can navigate to the Dataset Quality page where the flyout would be open and Sorting and Pagination would be pre-applied 5. API Tests 6. Stateful and Serverless FTR tests ## Screenshot <img width="1278" alt="image" src="https://github.com/elastic/kibana/assets/7416358/36a9b5cd-de05-4d17-99a2-cc08ec4583dd"> ## Scenario 1. Spin up a 8.14-snapshot instance 2. Ingest degraded docs 3. Upgrade to 8.15-snapshot 4. Open Data Set Quality Flyout and see how the page looks like <img width="1286" alt="image" src="https://github.com/elastic/kibana/assets/7416358/100f3c8c-b697-4f81-ac7e-427d0f468407">

achyutjhunjhunwala added 2 commits May 21, 2024 17:07

Add logic to display degraded fields in DQ Flyout

919b207

Revert synthtrace changes

23fa2e2

achyutjhunjhunwala self-assigned this May 21, 2024

achyutjhunjhunwala added 5 commits May 22, 2024 18:06

Move pagination logic to state machine and added URL sync logic for s…

07c0fc6

…hareable URLs

Fix logic to only call specific datastream

f38aec5

Add API tests

592c396

Merge branch 'main' into logic-to-display-degraded-fields

aed4b24

Add FTR tests

0402cb9

achyutjhunjhunwala marked this pull request as ready for review May 24, 2024 13:14

achyutjhunjhunwala requested review from a team as code owners May 24, 2024 13:14

shahzad31 approved these changes May 24, 2024

View reviewed changes

x-pack/plugins/observability_solution/observability/server/utils/queries.ts Show resolved Hide resolved

botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-management Observability Management User Experience Team labels May 24, 2024

achyutjhunjhunwala added 4 commits May 27, 2024 12:08

Fix checktype issue on Public State

00563cd

Merge branch 'main' into logic-to-display-degraded-fields

add5206

Fix type issue with url schema

803f067

Fix ts issue with synthtrace

75113c7

tonyghiani self-requested a review May 28, 2024 09:22

tonyghiani requested changes May 28, 2024

View reviewed changes

Fix review comments iteration 1

e24a79c

tonyghiani approved these changes May 29, 2024

View reviewed changes

achyutjhunjhunwala added 2 commits May 29, 2024 12:45

Change tooltip message for degraded fields

10abcbc

Merge branch 'main' into logic-to-display-degraded-fields

6f06c20

achyutjhunjhunwala enabled auto-merge (squash) May 29, 2024 10:46

achyutjhunjhunwala merged commit 6f7f133 into elastic:main May 29, 2024
21 of 22 checks passed

kibanamachine added the v8.15.0 label May 29, 2024

mdbirnstiehl reviewed May 29, 2024

View reviewed changes

achyutjhunjhunwala deleted the logic-to-display-degraded-fields branch May 30, 2024 07:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dataset Quality]Add logic to display degraded fields in DQ Flyout #183934

[Dataset Quality]Add logic to display degraded fields in DQ Flyout #183934

achyutjhunjhunwala commented May 21, 2024 •

edited

apmmachine commented May 21, 2024

achyutjhunjhunwala commented May 24, 2024

elasticmachine commented May 24, 2024

elasticmachine commented May 24, 2024

tonyghiani left a comment

flash1293 commented May 28, 2024

achyutjhunjhunwala commented May 29, 2024

flash1293 commented May 29, 2024

achyutjhunjhunwala commented May 29, 2024 •

edited

tonyghiani left a comment

kibana-ci commented May 29, 2024 •

edited

API count

mdbirnstiehl May 29, 2024

achyutjhunjhunwala May 30, 2024

[Dataset Quality]Add logic to display degraded fields in DQ Flyout #183934

[Dataset Quality]Add logic to display degraded fields in DQ Flyout #183934

Conversation

achyutjhunjhunwala commented May 21, 2024 • edited

Summary

Description

Screenshot

Scenario

apmmachine commented May 21, 2024

🤖 GitHub comments

achyutjhunjhunwala commented May 24, 2024

elasticmachine commented May 24, 2024

elasticmachine commented May 24, 2024

tonyghiani left a comment

Choose a reason for hiding this comment

flash1293 commented May 28, 2024

achyutjhunjhunwala commented May 29, 2024

flash1293 commented May 29, 2024

achyutjhunjhunwala commented May 29, 2024 • edited

tonyghiani left a comment

Choose a reason for hiding this comment

kibana-ci commented May 29, 2024 • edited

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Canvas Sharable Runtime

Page load bundle

API count

History

mdbirnstiehl May 29, 2024

Choose a reason for hiding this comment

achyutjhunjhunwala May 30, 2024

Choose a reason for hiding this comment

achyutjhunjhunwala commented May 21, 2024 •

edited

achyutjhunjhunwala commented May 29, 2024 •

edited

kibana-ci commented May 29, 2024 •

edited