Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

I think we should display the number of times a specific field has been ignored. For example Like (42) message. As we don't necessarily know the type of the ignored field, I don't think it makes much sense to show the type indicators, such as [k] message.

We should also cater for the situation where there are a lot of distinct ignored fields so that we can't necessarily list them all.

I think it would be important to add a definition of what ignored fields are, so I'm calling for @mdbirnstiehl's input as well.

Can we also add here why we are adding this section to the flyout and what is the value for the user, for future reference? I'd love for our issues descriptions to always include the reasoning behind it, so I can come in with the design/UX solution for discussion and the objectives are clear for everyone who reads this. cc @gbamparop @ruflin

Question: Should users be able to take actions from these fields? Filtering by, for instance?

The _ignored fields are the fields that were not indexed because of different reasons. At first, these fields are informative for users to be able to go into the dataset and apply changes to the mapping / pipelines themselfs to fix it.

Actions on these fields are needed to jump to Logs Explorer and filter down on it, to see the exact documents that are affected. Later on, actions around jumping into a "fixing" flow will exist. It could go so far, that we directly offer them already a fix eventually 🪄 .

There are different reasons why a field can end up in _ignore. We initially wanted to build each reason into Elasticsearch directly but to reduce scope, for now we are munging it all together into a single field. @felixbarny has some ideas how we could still take it apart. Assuming we can take it apart, it would be great to show a list for each reason:

• Field limit reached
• Mapping conflict

How the different problems are solved is also different, so separating these is important.

For the dataset quality page, we won't be able to determine the reason why a field has been ignored on aggregate. But I don't think that's a big issue. What matters most is which fields have been ignored, and how often. When clicking on an ignored field, it should take the user do the Logs explorer with a filter (_ignored: "<field>") so that it shows all documents that have a specific ignored field (such as message). The next step in the journey is for the user to open the flyout for a specific log event.

What's happening there should be defined in this issue:

• [Logs Explorer] Add _ignored dedicated section to Doc Flyout #172272

In short, we should list all ignored fields, their values, and the reason for why they have been ignored.

But I don't think that's a big issue. What matters most is which fields have been ignored, and how often.

If we would have the field reason, we could directly offer quick actions to the user directly from the dataset quality page fly out. For example, we see 25 of the ignored fields hit the field limit. We can group it and tell them a single fix to get rid of all of it. In the Log Explorere, users have to go through each document and figure out the reason per document and don't see the bigger picture.

For now, all we can do is what Felix suggests but I think we put an unecessary burden on the user and should eventually improve it.